ETL Developer

Dear Candidate,

Thanks for showing interest in the opportunity!!!

As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.

Job Description:

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.

Primary & Mandatory Skills:

• ETL Development
• Hadoop / HDFS
• PySpark / Apache Spark
• Data Extraction, Transformation & Loading
• SQL and Data Processing

Good to Have:

• Hive / Spark SQL
• Data Pipeline Development
• Big Data Processing
• Python Programming

Company: Mphasis

Role: ETL Developer – Hadoop & PySpark

Experience: As per requirement

Location: Bangalore

Employment Type: Permanent

Data Engineer Hiring Post

🚨 Hiring: Data Engineer | PySpark + Python + SQL

We are looking for experienced Data Engineers to join our team!

🔹 Experience: 5 to 9 Years

🔹 Locations: Bangalore / Hyderabad

🔹 Interview Process:

• 1st Round – Virtual

• 2nd Round – Face-to-Face (Karat Test)

🔑 Key Skills:

✅ PySpark

✅ SQL

✅ Python

✅ ETL

📩 Interested candidates can share their updated resume.

#Hiring #DataEngineer #PySpark #Python #SQL #ETL #BangaloreJobs #HyderabadJobs #TechHiring #ImmediateHiring

Data Engineer Short Hiring Post

🚨 Hiring: Data Engineer

🔹 Experience: 5–9 Years

🔹 Location: Bangalore / Hyderabad

🔹 Skills: PySpark, Python, SQL, ETL, CI/CD, Data Modeling

🔹 Process: L1 Virtual → L2 F2F Karat Test

🔹 F2F: Bangalore / Hyderabad Location

🔹 Positions: Immediate requirement

⚠️ Note: Candidates must be available for F2F Karat immediately after L1.

#Hiring #DataEngineer #PySpark #Python #SQL #BangaloreJobs #HyderabadJobs #Mphasis #ImmediateJoiners

Job Title : AWS Data Engineer

Experience : 4+ Years

Location : Bengaluru (HSR – Hybrid, 3 Days WFO)

Notice Period : Immediate Joiner

💡 Role Overview :

We are looking for a skilled AWS Data Engineer to design, build, and scale modern data platforms. The role involves working with AWS-native services, Python, Spark, and DBT to deliver secure, scalable, and high-performance data solutions in an Agile environment.

🔥 Mandatory Skills :

Python, SQL, Spark, AWS (S3, Glue, EMR, Redshift, Athena, Lambda), DBT, ETL/ELT pipeline development, Airflow/Step Functions, Data Lake (Parquet/ORC/Iceberg), Terraform & CI/CD, Data Governance & Security

🚀 Key Responsibilities :

• Design, build, and optimize ETL/ELT pipelines using Python, DBT, and AWS services
• Develop and manage scalable data lakes on S3 using formats like Parquet, ORC, and Iceberg
• Build end-to-end data solutions using Glue, EMR, Lambda, Redshift, and Athena
• Implement data governance, security, and metadata management using Glue Data Catalog, Lake Formation, IAM, and KMS
• Orchestrate workflows using Airflow, Step Functions, or AWS-native tools
• Ensure reliability and automation via CloudWatch, CloudTrail, CodePipeline, and Terraform
• Collaborate with data analysts and data scientists to deliver actionable insights
• Work in an Agile environment to deliver high-quality data solutions

✅ Mandatory Skills :

• Strong Python (including AWS SDKs), SQL, Spark
• Hands-on experience with AWS data stack (S3, Glue, EMR, Redshift, Athena, Lambda)
• Experience with DBT and ETL/ELT pipeline development
• Workflow orchestration using Airflow / Step Functions
• Knowledge of data lake formats (Parquet, ORC, Iceberg)
• Exposure to DevOps practices (Terraform, CI/CD)
• Strong understanding of data governance and security best practices
• Minimum 4–7 years in Data Engineering (3+ years on AWS)

➕ Good to Have :

• Understanding of Data Mesh architecture
• Experience with platforms like Data.World
• Exposure to Hadoop / HDFS ecosystems

🤝 What We’re Looking For :

• Strong problem-solving and analytical skills
• Ability to work in a collaborative, cross-functional environment
• Good communication and stakeholder management skills
• Self-driven and adaptable to fast-paced environments

📝 Interview Process :

1. Online Assessment
2. Technical Interview
3. Fitment Round
4. Client Round

🚀 Job Title : Data Engineer – AWS, PySpark & Airflow

Experience : 5+ Years

Industry : Health Insurance / BFSI

Location : Sector 43, Gurugram, Haryana

Work Mode : Work From Office (5 Days a Week)

Employment Type : Full-Time

Role Overview :

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS Data Engineering, Data Lake Architecture, Python, SQL, PySpark, and Apache Airflow.

The candidate will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data pipelines using AWS cloud technologies. The role involves building end-to-end ETL / ELT workflows, implementing data transformation and processing solutions, optimizing pipeline performance, and ensuring the reliability of production data systems.

Mandatory Skills :

AWS Data Engineering | Data Lake Architecture | Amazon S3 | AWS Glue | Amazon EMR | Athena | Redshift | Python | SQL | PySpark | Apache Spark | Apache Airflow | ETL/ELT | PostgreSQL | Data Modelling | Pipeline Optimization | Production Support | Root Cause Analysis

Key Responsibilities :

• Design, develop, and maintain end-to-end data pipelines for data ingestion, staging, transformation, curation, and consumption.
• Build and manage AWS-based data lake solutions using Amazon S3, AWS Glue, Amazon EMR, Athena, and Redshift.
• Develop, maintain, and optimize ETL / ELT pipelines using Python, SQL, Apache Spark, and PySpark.
• Implement and manage workflow orchestration using Apache Airflow, including scheduling, dependency management, retries, monitoring, and failure handling.
• Integrate data from multiple structured and semi-structured sources.
• Design and maintain data models, dimensional models, and analytical data marts.
• Work with relational databases, including PostgreSQL and Amazon RDS.
• Optimize Spark jobs, SQL queries, data transformations, and cloud resource utilization.
• Implement data quality checks, validation rules, and error-handling mechanisms.
• Monitor production data pipelines, troubleshoot failures, and perform root cause analysis (RCA).
• Ensure pipeline reliability, scalability, performance, and compliance with service-level agreements (SLAs).
• Follow data security, access management, governance, and sensitive-data protection best practices.
• Collaborate with business, analytics, and application teams to deliver effective data solutions.
• Maintain technical documentation and follow established data engineering standards.

Required Technical Skills :

• AWS Data Engineering : Strong hands-on experience with AWS-based data engineering and data lake architecture.
• AWS Services : Amazon S3, AWS Glue, Glue Data Catalog, Amazon EMR, Athena, Redshift, AWS Lambda, and IAM.
• Programming : Strong proficiency in Python and SQL.
• Big Data Processing : Hands-on experience with Apache Spark and PySpark.
• Workflow Orchestration : Experience developing and managing workflows using Apache Airflow.
• ETL/ELT Development : Experience building scalable data ingestion, transformation, and processing pipelines.
• Database Technologies : Good understanding of relational databases and PostgreSQL concepts.
• Data Modelling : Experience with data modelling, data transformation, and analytical data structures.
• Production Support : Ability to monitor pipelines, troubleshoot failures, perform RCA, and resolve production issues.
• Performance Optimization : Experience optimizing SQL queries, Spark jobs, and data processing workflows.

Good-to-Have Skills :

• Experience with AWS Database Migration Service (DMS) and Amazon RDS.
• Familiarity with dbt for data transformation and modelling.
• Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.
• Exposure to Apache Kafka and real-time data streaming.
• Familiarity with GraphQL, Hasura, or similar data-serving technologies.
• Understanding of data governance and data quality frameworks.
• Experience in the BFSI, healthcare, or health insurance domain.

Candidate Requirements :

• 5+ years of relevant experience in Data Engineering.
• Proven hands-on experience building and maintaining production-grade data pipelines on AWS.
• Strong understanding of data lake architecture and large-scale data processing.
• Practical experience with Apache Airflow and Spark / PySpark-based transformations.
• Strong programming, analytical, debugging, and problem-solving skills.
• Experience handling production incidents, pipeline failures, and performance bottlenecks.
• Ability to take ownership of deliverables and work effectively with cross-functional teams.
• Good communication, collaboration, and technical documentation skills.
• Willingness to work from the Gurugram office five days a week.

Ideal Candidate :

The ideal candidate is a hands-on Data Engineer who can independently build, optimize, and maintain AWS-based data pipelines, work with large datasets using PySpark, orchestrate workflows through Airflow, and resolve production issues in a business-critical environment.

Job Description

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines.
• Work with Hadoop and PySpark to process and transform large volumes of data.
• Develop and optimize Python scripts for data processing and automation.
• Write complex SQL queries for data extraction, transformation, and validation.
• Perform data cleansing, integration, transformation, and validation.
• Work with data warehousing concepts and large-scale data processing systems.
• Optimize ETL workflows for performance, scalability, and reliability.
• Troubleshoot data pipeline failures and resolve data quality issues.
• Collaborate with data engineers, analysts, and other technical teams.
• Follow data governance, quality, and development best practices.

Required Skills

• Strong experience in ETL Development – Mandatory
• Hands-on experience with Hadoop – Mandatory
• Strong experience in PySpark – Mandatory
• Good programming skills in Python – Mandatory
• Strong knowledge of SQL – Mandatory
• Experience in Data Processing and Transformation – Mandatory
• Knowledge of Data Warehousing – Mandatory
• Experience with Big Data Technologies – Mandatory

Opportunity Details

Role: ETL Developer – Hadoop & PySpark

Experience: 6–10 Years

Location: Bangalore

Mandatory Skills

ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work extensively with Databricks and Delta Lake, including Delta tables, MERGE operations, schema evolution, Time Travel, and data optimization.
• Develop, schedule, and monitor Apache Airflow DAGs for workflow orchestration and dependency management.
• Work with Kubernetes for containerized application deployment, orchestration, monitoring, and troubleshooting.
• Develop reusable and optimized Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Use Git for version control, branching, pull requests, code reviews, and collaborative development.
• Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.
• Collaborate with cross-functional teams to understand business requirements and deliver efficient data solutions.

Required Technical Skills

• Python: Strong programming skills, including data manipulation, exception handling, modular programming, and debugging.
• PySpark: Hands-on experience with Spark DataFrames, Spark SQL, joins, transformations, partitioning, and performance tuning.
• Databricks: Strong understanding of Databricks architecture, notebooks, workflows, compute management, and job execution.
• Delta Lake: Experience with Delta tables, ACID transactions, MERGE/UPSERT, schema enforcement, schema evolution, VACUUM, and Time Travel.
• Apache Airflow: Experience developing DAGs, managing task dependencies, configuring retries, scheduling workflows, and troubleshooting failures.
• Kubernetes: Practical knowledge of pods, deployments, services, ConfigMaps, Secrets, resource management, and troubleshooting containerized workloads.
• Git: Proficiency in branching strategies, pull requests, merging, and resolving conflicts.
• SQL: Strong SQL skills, including complex joins, CTEs, window functions, and query optimization.

Good to Have

• Experience with Docker and containerization best practices.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of CI/CD pipelines and automated deployments.
• Experience deploying and managing Airflow workloads on Kubernetes.
• Familiarity with Kubernetes monitoring, autoscaling, and resource optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of data lakehouse architecture and data modeling.
• Experience with batch and incremental data processing.
• Familiarity with data quality frameworks and automated testing.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently design, develop, deploy, and troubleshoot data pipelines.
• Good understanding of distributed data processing and container orchestration.
• Experience working in an Agile development environment.
• Strong communication and collaboration skills.
• 
  

Hiring: ETL Developer – Hadoop & PySpark

Location: Bangalore

Experience: 6–10 Years

Budget: 25–30 LPA

Work Mode: Weekdays

Job Description

We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Develop data transformation workflows using PySpark and Python.
• Work with Hadoop and other big data technologies to process and manage large datasets.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Implement data cleansing, data quality checks, and data validation processes.
• Develop and optimize data integration workflows for data warehousing solutions.
• Troubleshoot ETL failures, performance issues, and data discrepancies.
• Optimize PySpark jobs and SQL queries for performance and scalability.
• Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
• Ensure data accuracy, consistency, and adherence to established data governance standards.

Mandatory Skills

• Strong hands-on experience in ETL development.
• Good experience with Hadoop and big data technologies.
• Proficiency in PySpark and Python programming.
• Strong SQL skills, including complex queries and data manipulation.
• Experience in data processing, data transformation, and data integration.
• Understanding of data warehousing concepts and architecture.
• Experience in developing, debugging, and optimizing ETL pipelines.

Required Experience

• 6–10 years of relevant experience in ETL development and data engineering.
• Practical experience handling large datasets using Hadoop and PySpark.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently and collaborate effectively within a team.

Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.

Strong Data Engineer (AWS) Profile

2

Mandatory (Experience 1) – Must have minimum 6+ years of experience in Data Engineering, with strong experience designing, developing, and maintaining scalable data pipelines and ETL processes.

3

Mandatory (Experience 2) – Must have 3+ years of hands-on experience with AWS data engineering services, particularly S3, AWS Glue, EMR, Lambda, and Redshift.

4

Mandatory (Experience 3) – Must have strong programming experience in Python, Java, or Scala, with hands-on development of data processing and pipeline solutions.

5

Mandatory (Experience 4) – Must have strong expertise in SQL, data warehousing, and database technologies, including experience with both SQL and NoSQL databases.

6

Mandatory (Experience 5) – Must have hands-on experience working with Big Data technologies and distributed data processing, including designing and optimizing large-scale data pipelines.

7

Mandatory (Experience 6) – Must have experience with data pipeline troubleshooting, monitoring, data quality, governance, and security best practices.

8

Mandatory (Education) – B.Tech / B.E./ M.Tech

Strong Data Engineer (AWS) Profile

2

Mandatory (Experience 1) – Must have minimum 6+ years of experience in Data Engineering, with strong experience designing, developing, and maintaining scalable data pipelines and ETL processes.

3

Mandatory (Experience 2) – Must have 3+ years of hands-on experience with AWS data engineering services, particularly S3, AWS Glue, EMR, Lambda, and Redshift.

4

Mandatory (Experience 3) – Must have strong programming experience in Python, Java, or Scala, with hands-on development of data processing and pipeline solutions.

5

Mandatory (Experience 4) – Must have strong expertise in SQL, data warehousing, and database technologies, including experience with both SQL and NoSQL databases.

6

Mandatory (Experience 5) – Must have hands-on experience working with Big Data technologies and distributed data processing, including designing and optimizing large-scale data pipelines.

7

Mandatory (Experience 6) – Must have experience with data pipeline troubleshooting, monitoring, data quality, governance, and security best practices.

8

Mandatory (Education) – B.Tech / B.E./ M.Tech