Data Science Engineer at J&F · Remote, Bengaluru (Bangalore), Noida, Chennai · 3 - 10 years · Bootstrapped · Remote friendly · Posted 12 Aug 2026
Role Summary
We are hiring a Data Engineer / ML Data Pipeline Engineer to build and operate the data backbone of the Enterprise AI platform:
What You'll Own
- Ingestion & ETL/ELT pipelines for heterogeneous project folders (PDF drawings, SVG files, IFC models, BBS.json bar-bending-schedule data, Excel exports, and AI agent output JSON).
- AWS-based data architecture: S3 raw/staging/curated/outputs structuring, partitioning, versioning, and lifecycle management; querying via Athena/Glue and warehousing via Redshift or Snowflake as needed.
- Data validation frameworks: GUID cross-referencing between SVG and BBS data, schema enforcement, duplicate/orphan detection, reference integrity checks, and structured validation reporting.
- Agent run logging & observability: designing the database schema and pipelines that track every AI agent run (inputs, outputs, status, errors, cost, retries, reviewer feedback).
- AI Factory monitoring dashboards: operational dashboards (failure rates, retries, latency, data quality) and business dashboards (throughput, cost per run, rework rate) for Power BI/QuickSight or equivalent.
- ML data pipeline support: dataset preparation, labeling/annotation workflows, human-in-the-loop review tooling, and dataset versioning for models that classify or QC drawing issues.
- APIs: designing and building FastAPI/Flask endpoints to trigger validation runs and expose agent processing status to internal tools.
- Data quality & testing discipline: idempotent pipelines, quarantine/reject handling, regression and reconciliation testing, and root-cause debugging when pipelines or query performance degrade in production.
Key Skills — Non-Negotiable (Must-Have, Strong Level)
- Python — production-grade scripting: file/folder handling, JSON/schema processing, clean error handling, not just notebook-level scripting.
- SQL — strong hands-on ability, including GROUP BY/HAVING for duplicate detection, window functions, and daily aggregate/rate calculations (e.g., success-rate queries).
- AWS S3 data handling — practical experience structuring buckets for raw/staging/curated data, versioning, and avoiding overwrite issues at scale.
- Data validation — demonstrable experience building validation logic (set comparisons, duplicate/missing detection, structured pass/fail reporting), not just "I write assertions."
- ETL/ELT pipeline design — end-to-end ownership of at least one pipeline: source → transform → storage → validation → monitoring → business outcome, with clear articulation of what they personally built.
- Query/warehouse engine judgment — working knowledge of when to use Athena vs. Redshift vs. Snowflake (or equivalent), partitioning, clustering, sort/distribution keys, and storage format trade-offs (Parquet vs. JSON vs. CSV).
Key Skills — Good to Have
- Dashboarding — Power BI / QuickSight (or equivalent) fact/dimension table design, KPI cards, drill-downs; medium-to-strong level is a plus but trainable.
- FastAPI / Flask — building real endpoints with request/response schemas and basic error handling; especially valuable for validation-trigger and agent-status APIs.
- ML data pipeline experience — dataset labeling, annotation platform design, train/test/validation splitting, dataset versioning; strong on the pipeline/data side rather than model training itself.
- Human-in-the-loop / review tooling — experience building or contributing to browser-based labeling/review platforms (session persistence, label schema, export formats).
- Large-scale metadata querying — experience making file discovery fast across large volumes (1,000+ projects, thousands of files each) via metadata index tables, event-based ingestion, or catalog tools like AWS Glue.
Shubham Vishwakarma
Full Stack Developer - Averlon
About J&F
About
At J&F, we are a globally established engineering consultancy with over 500 engineers and detailers across five countries, delivering high-quality engineering solutions for buildings and infrastructure projects. We specialize in BIM-enabled Structural and MEP engineering, combining deep technical expertise with digital innovation to help clients execute complex projects efficiently, accurately, and at scale.
Through our two core business lines—Virtual Technical Office and End-to-End Engineering Solutions—we provide tailored engineering support as well as fully integrated project planning and supervision. By partnering closely with our clients, we deliver sustainable, cost-effective, and technology-driven solutions across a wide range of sectors, ensuring engineering excellence throughout every stage of the project lifecycle.
Company social profiles
Similar jobs (10)
PythonJob Description
We are looking for an experienced Data Engineer with strong expertise in Python, ETL, SQL, CI/CD, and DevOps to design, develop, and maintain scalable data pipelines and data processing solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines.
- Develop data processing solutions using Python.
- Write complex and optimized SQL queries, stored procedures, and data transformations.
- Build and maintain data ingestion and integration workflows.
- Implement data quality, validation, monitoring, and error-handling processes.
- Develop and maintain CI/CD pipelines for data engineering applications.
- Work with DevOps tools and practices for automated build, deployment, and infrastructure management.
- Collaborate with data analysts, data scientists, software engineers, and business teams.
- Optimize data pipelines for performance, reliability, and scalability.
- Troubleshoot production data issues and ensure timely resolution.
- Follow best practices for version control, code quality, testing, and deployment.
Mandatory Skills
- Python
- ETL
- SQL
- CI/CD
- DevOps
- Git / Version Control
- Strong problem-solving and debugging skills
PythonJob Summary
Role Overview
We are looking for an experienced Data Engineer with strong expertise in Python, ETL, Advanced SQL, CI/CD, DevOps, and Data Analytics. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, transforming large datasets, and supporting data-driven applications.
Experience with Google Cloud Platform (GCP) will be an added advantage.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL.
- Develop complex and optimized SQL queries, stored procedures, and data transformations.
- Build and maintain reliable data integration workflows across multiple data sources.
- Perform data cleansing, validation, transformation, and quality checks.
- Analyze data and provide insights to support business and technical requirements.
- Implement and maintain CI/CD pipelines for data engineering applications.
- Work with DevOps practices and tools to automate deployments, monitoring, and infrastructure processes.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Optimize data processing workflows for performance, scalability, and reliability.
- Collaborate with Data Analysts, Data Scientists, Developers, and other stakeholders.
- Follow best practices for version control, testing, documentation, and deployment.
- Contribute to cloud-based data engineering initiatives, preferably on GCP.
Required Skills
- 5–7 years of hands-on experience in Data Engineering.
- Strong programming skills in Python.
- Strong expertise in Advanced SQL and database concepts.
- Hands-on experience with ETL/ELT processes and data pipelines.
- Good understanding of Data Warehousing and Data Modeling concepts.
- Experience with CI/CD practices and tools.
- Strong understanding of DevOps principles, automation, and deployment processes.
- Strong data analytics and problem-solving skills.
- Experience working with large datasets and performance optimization.
- Good understanding of Git/version control and software development best practices.
Good to Have
- Hands-on experience with Google Cloud Platform (GCP).
- Exposure to GCP data services such as BigQuery, Cloud Storage, Dataflow, Composer, or Pub/Sub.
- Experience with containerization/orchestration technologies such as Docker/Kubernetes.
- Experience with workflow orchestration tools such as Airflow.
- Knowledge of cloud-based data architecture and distributed data processing.
Preferred Candidate Profile
- Strong analytical and problem-solving abilities.
- Good communication and stakeholder management skills.
- Ability to work independently as well as in a collaborative team environment.
- Strong ownership of data pipelines and production systems.
- Candidates who can join at short notice are preferred.
Mandatory Skills
Data Engineer, Python , ETL, GCP, Advanced SQL, Strong Data Analytics skills, CICD, Devops
PythonUrgent Hiring – Senior Data Engineer
We are hiring for a Senior Data Engineer for a reputed product-based company in Pune.
Location: Pune – Magarpatta / Baner
Experience: 7–10 Years
Work Mode: 5 Days WFO
Notice Period: Immediate to 30 Days preferred
What We're Looking For
- 7+ years of hands-on experience in Data Engineering
- 5+ years of experience in Python
- 4+ years of experience in Snowflake
- 5+ years of experience in SQL
- 5+ years of experience with ADF / Fivetran / Matillion or equivalent Data Integration tools
- Hands-on experience with AWS / Azure
- Experience with Airflow or equivalent orchestration tools
- Strong experience in ETL/ELT and Data Pipelines
- Experience with APIs, JSON, XML and Webhooks
- Knowledge of CI/CD, Git and automated testing
- Exposure to dbt or similar transformation tools
- Experience in data pipeline monitoring, troubleshooting and performance optimization
Key Responsibilities
- Design, develop and maintain scalable ETL/ELT data pipelines
- Build batch, real-time and on-demand data processing workflows
- Integrate data from cloud and on-premise sources
- Ensure data quality, reliability, performance and SLA adherence
- Work closely with Data Scientists, Analysts, DevOps and Business teams
- Implement data engineering best practices, CI/CD and automated testing
- Troubleshoot pipeline issues and perform root cause analysis
- Optimize data pipelines and SQL queries for performance and cost efficiency
Why Join?
- Opportunity to work with a reputed product-based organization
- Work on modern data engineering and cloud technologies
- Exposure to large-scale data platforms and business-critical data solutions
- Collaborative and technically strong environment
Interested candidates can share their updated CV for immediate consideration.
PythonData Engineer Short Hiring Post
🚨 Hiring: Data Engineer
🔹 Experience: 5–9 Years
🔹 Location: Bangalore / Hyderabad
🔹 Skills: PySpark, Python, SQL, ETL, CI/CD, Data Modeling
🔹 Process: L1 Virtual → L2 F2F Karat Test
🔹 F2F: Bangalore / Hyderabad Location
🔹 Positions: Immediate requirement
⚠️ Note: Candidates must be available for F2F Karat immediately after L1.
#Hiring #DataEngineer #PySpark #Python #SQL #BangaloreJobs #HyderabadJobs #Mphasis #ImmediateJoiners
PythonJob Summary
We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure. The ideal candidate should have strong expertise in SQL, Python, Linux, and modern data engineering practices to support data integration, transformation, and analytics.
Key Responsibilities
- Design, develop, and maintain ETL/ELT data pipelines.
- Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
- Develop automation scripts using Python for data processing and workflow optimization.
- Work with Linux environments for deployment, monitoring, and troubleshooting.
- Ensure data quality, integrity, and reliability across data platforms.
- Collaborate with data analysts, software engineers, and business stakeholders to deliver data solutions.
- Monitor, troubleshoot, and optimize data pipelines for performance and scalability.
- Implement best practices for data security, governance, and documentation.
Required Skills
- Strong experience in Data Engineering concepts and ETL/ELT processes.
- Proficiency in SQL, including query optimization and database design.
- Strong programming skills in Python.
- Hands-on experience with Linux commands, shell scripting, and system administration basics.
- Experience with relational databases such as PostgreSQL, MySQL, SQL Server, or Oracle.
- Familiarity with Git/version control.
- Strong analytical and problem-solving skills.
Preferred Skills
- Experience with cloud platforms (AWS, Azure, or GCP).
- Knowledge of Apache Spark, Airflow, Kafka, or similar data engineering tools.
- Experience with data warehousing solutions and big data technologies.
- Understanding of CI/CD pipelines and containerization (Docker/Kubernetes).
Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
- Relevant certifications in cloud or data engineering are an added advantage.
PythonWe are hiring a Data Engineer to build reliable pipelines and data platforms.
Responsibilities
- Build and maintain batch and streaming data pipelines
- Orchestrate workflows with Apache Airflow
- Model and optimise data in the warehouse
- Monitor data quality and pipeline health
Requirements
- 1+ years of data engineering
- Strong Python and SQL skills
- Experience with Airflow or a similar orchestrator
We are looking for an AWS Data Engineer to build reliable, scalable data pipelines on AWS.
Responsibilities
- Build ETL and ELT pipelines with AWS Glue and dbt
- Model and optimise data warehouses in Redshift and Snowflake
- Orchestrate workflows with Apache Airflow
- Build streaming ingestion with Amazon Kinesis
- Ensure data quality, monitoring and cost efficiency
Requirements
- 2+ years of data engineering on AWS
- Hands-on with Glue, Redshift or Snowflake, and Airflow
- Strong data modelling and warehousing skills
Amazon Web Services (AWS)PythonData Engineer
Location: Bengaluru, India (Hybrid)
Employment Type: Full-time
Experience: 3-5 years
Role Overview
What We’re Looking For:
- Bachelor’s degree in Computer Science/Engineering or equivalent experience required.
- Experience designing and shipping cloud services products.
- Experience driving and managing technical and architectural dependencies on AWS Cloud.
- A firm understanding of system architecture, cloud computing, PaaS/SaaS design principles, S3, DynamoDB, RDS mandatory.
- Experience in building or maintaining ETL processes and tools, i.e., AWS Glue or any open-source tool.
- Proven system-level design contribution to a current “Live” (in production / under daily high load) multi-region SaaS or PaaS offering.
- Proven experience with S3, DynamoDB, SQL, and AWS RDS services.
- Proficiency in programming languages such as Python.
- Strong analytical and problem-solving skills.
Required Skills & Experience
- Experience with Python, SQL, and data visualization/exploration tools.
- Familiarity with the AWS ecosystem, specifically S3, DynamoDB, and RDS.
- Communication skills, especially for explaining technical concepts to nontechnical business leaders.
- Ability to work on a dynamic, research-oriented team that has concurrent projects.
- Experience in AWS cost optimization (Savings Plans, Reserved Instances, Spot Instances) and governance frameworks.
- Experience developing solutions using infrastructure orchestration tools (SSM, automation account, Ansible, etc.).
- Excellent leadership, stakeholder management, and communication skills.
What We Offer
- Work with some of the brightest minds in the emerging EV industry.
- Make a tangible impact in reducing carbon emissions and enabling sustainable energy.
- Freedom to suggest, implement, and innovate on systems, processes, and technologies.
- Daily ownership in a high-growth, challenging environment.
- Flexible work environment with hybrid schedules and virtualization options.
- Competitive pay and benefits including health coverage, innovative PTO program, and performance bonuses.
Python
GitData Engineer - Remote
Nearshore engineer on a team converting SAS code to Python and SQL on Databricks using generative AI. You will work with the existing accelerators, own deliverables end to end, and communicate directly with client and partner stakeholders.
Required for both:
4+ years of professional data or software engineering experience
Strong Python and SQL
Hands-on Databricks (Unity Catalog, Workflows, Databricks Asset Bundles)
GitLab CI/CD: pipelines, merge request workflows, automated testing
Git branching and code review discipline
Clear written and spoken English with client-facing partners
Demonstrated ownership: scoping, delivering, and flagging risk without prompting
Focus: Pipeline reliability, validation, and delivery of converted code.
Responsibilities:
Build and run the pipelines that process SAS inventories and converted outputs
Validate converted code for parity against SAS outputs (row counts, checksums, schema, data types)
Own deployment through DABs and GitLab CI/CD
Manage Unity Catalog objects, permissions, and environment promotion
Troubleshoot job failures and performance issues
Required:
Spark and Delta Lake performance tuning
Data validation and reconciliation experience
Infrastructure as code or DAB-based deployment experience
Nice to have:
SAS reading ability, healthcare data exposure, Azure.
PythonAmazon Web Services (AWS)About Us:
The QX Impact was launched with a mission to make A.I accessible and affordable and deliver AI Products/Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation. We believe without insights; businesses will continue to face challenges to better understand their customers and even lose them. Secondly, without insights businesses won't’ be able to deliver differentiated products/services; and finally, without insights, businesses can’t achieve a new level of “Operational Excellence” is crucial to remain competitive, meeting rising customer expectations, expanding markets, and digitalization.
Job Summary:
We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions.
As a Senior Data Engineer, you will work with internal and external data, collaborate with data scientists, and contribute to the design, development, and deployment of innovative solutions.
Key Responsibilities:
- Design, develop, test, and maintain optimal data pipeline and ETL architectures.
- Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
- Prepare and optimize data for predictive and prescriptive modeling.
- Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
- Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
- Utilize big data tools and frameworks to optimize data acquisition and preparation.
- Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
- Develop and curate data models for analytics, dashboards, and reports.
- Conduct code reviews, maintain production-level code, and implement testing approaches.
- Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
- Drive innovation and implement efficient new approaches to data engineering tasks.
Must-Have Skills:
- Bachelor’s degree in Computer Science, Mathematics, Engineering, or a related field.
- 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
- 3–5 years of experience designing and implementing data warehouse solutions.
- Expertise in SQL, including developing stored procedures (SP) and applying advanced data design concepts.
- Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
- Experience with AWS or Azure services (e.g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
- Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
- Hands-on experience with orchestration tools (e.g., Apache Airflow, Prefect).
- Knowledge of CI/CD processes, version control (e.g., Git, Jenkins), and deployment automation.
- Strong problem-solving, communication, and collaboration skills.
Good-to-Have Skills:
- Experience in integrating ERP data into data lakes.
- Experience with traditional ETL tools (e.g., Talend, Pentaho).
Competencies:
- Tech Savvy - Anticipating and adopting innovations in business-building digital and technology applications.
- Self-Development - Actively seeking new ways to grow and be challenged using both formal and informal development channels.
- Action Oriented - Taking on new opportunities and tough challenges with a sense of urgency, high energy, and enthusiasm.
- Customer Focus - Building strong customer relationships and delivering customer-centric solutions.
- Optimize Work Processes - Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.
Why Join Us?
- Be part of a collaborative and agile team driving cutting-edge AI and data engineering solutions.
- Work on impactful projects that make a difference across industries.
- Opportunities for professional growth and continuous learning.
- Competitive salary and benefits package.
Application Details
Ready to make an impact? Apply today and become part of the QX Impact team!
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon