Big Data Engineer

Job Description

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines.
• Work with Hadoop and PySpark to process and transform large volumes of data.
• Develop and optimize Python scripts for data processing and automation.
• Write complex SQL queries for data extraction, transformation, and validation.
• Perform data cleansing, integration, transformation, and validation.
• Work with data warehousing concepts and large-scale data processing systems.
• Optimize ETL workflows for performance, scalability, and reliability.
• Troubleshoot data pipeline failures and resolve data quality issues.
• Collaborate with data engineers, analysts, and other technical teams.
• Follow data governance, quality, and development best practices.

Required Skills

• Strong experience in ETL Development – Mandatory
• Hands-on experience with Hadoop – Mandatory
• Strong experience in PySpark – Mandatory
• Good programming skills in Python – Mandatory
• Strong knowledge of SQL – Mandatory
• Experience in Data Processing and Transformation – Mandatory
• Knowledge of Data Warehousing – Mandatory
• Experience with Big Data Technologies – Mandatory

Opportunity Details

Role: ETL Developer – Hadoop & PySpark

Experience: 6–10 Years

Location: Bangalore

Mandatory Skills

ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies

Dear Candidate,

Thanks for showing interest in the opportunity!!!

As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.

Job Description:

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.

Primary & Mandatory Skills:

• ETL Development
• Hadoop / HDFS
• PySpark / Apache Spark
• Data Extraction, Transformation & Loading
• SQL and Data Processing

Good to Have:

• Hive / Spark SQL
• Data Pipeline Development
• Big Data Processing
• Python Programming

Company: Mphasis

Role: ETL Developer – Hadoop & PySpark

Experience: As per requirement

Location: Bangalore

Employment Type: Permanent

Hiring: ETL Developer – Hadoop & PySpark

Location: Bangalore

Experience: 6–10 Years

Budget: 25–30 LPA

Work Mode: Weekdays

Job Description

We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Develop data transformation workflows using PySpark and Python.
• Work with Hadoop and other big data technologies to process and manage large datasets.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Implement data cleansing, data quality checks, and data validation processes.
• Develop and optimize data integration workflows for data warehousing solutions.
• Troubleshoot ETL failures, performance issues, and data discrepancies.
• Optimize PySpark jobs and SQL queries for performance and scalability.
• Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
• Ensure data accuracy, consistency, and adherence to established data governance standards.

Mandatory Skills

• Strong hands-on experience in ETL development.
• Good experience with Hadoop and big data technologies.
• Proficiency in PySpark and Python programming.
• Strong SQL skills, including complex queries and data manipulation.
• Experience in data processing, data transformation, and data integration.
• Understanding of data warehousing concepts and architecture.
• Experience in developing, debugging, and optimizing ETL pipelines.

Required Experience

• 6–10 years of relevant experience in ETL development and data engineering.
• Practical experience handling large datasets using Hadoop and PySpark.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently and collaborate effectively within a team.

Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.

Position: Backup Data ETL Developer

Experience: 6–9 Years

Location:Mumbai

Employment Type: Full-time

Job Description:

We are looking for a Backup Data ETL Developer with 6–9 years of experience in designing, developing, and supporting data extraction, transformation, and loading (ETL) processes.

The candidate should have strong hands-on experience with Hadoop, Spark, Informatica, and Snowflake, along with expertise in large-scale data processing and data integration technologies.

Key Skills:

Hadoop

Spark

Informatica

Snowflake

ETL Development

Data Integration

Data Transformation

Large-Scale Data Processing

ETL Workflows and Data Pipelines

Troubleshooting and Problem-Solving

Strong analytical and collaboration skills

We are hiring experienced Data Engineers for a permanent opportunity in Bangalore.

Job Description

Role: Data Engineer

Location: Bangalore

Experience: 6+ Years

Key Responsibilities:

• Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
• Develop data engineering solutions using Python and Scala.
• Process and transform large volumes of structured and unstructured data.
• Build and optimize distributed data processing workflows for performance and scalability.
• Work with big data technologies and cloud-based data processing environments.
• Perform data integration, validation, transformation, and quality checks.
• Troubleshoot data pipeline issues and optimize processing performance.
• Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.

Required Skills:

• Strong hands-on experience with Amazon EMR.
• Proficiency in Python and Scala.
• Experience in big data processing and data pipeline development.
• Good understanding of distributed computing and data transformation techniques.
• Experience in performance tuning and troubleshooting data processing jobs.
• Strong analytical, problem-solving, and communication skills.

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
• Build and optimize distributed data applications using Apache Spark and Python Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Design and manage data workflows using Apache Airflow.
• Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
• Work with large datasets to ensure data quality, consistency, reliability, and performance.
• Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, performance, and cost efficiency.
• Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
• Strong hands-on experience with Apache Spark and Scala.
• Experience designing, building, and maintaining large-scale ETL pipelines.
• Strong hands-on experience with AWS, particularly Amazon S3.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong SQL skills and a solid understanding of distributed data processing concepts.
• Experience working with batch and/or streaming data pipelines.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with Databricks and the broader Databricks data platform.
• Familiarity with streaming technologies such as Apache Kafka.
• Experience working on large-scale data platforms handling high-volume data workloads.
• Exposure to additional AWS data services and cloud-native data architectures.

🚀 Job Title : Data Engineer – AWS, PySpark & Airflow

Experience : 5+ Years

Industry : Health Insurance / BFSI

Location : Sector 43, Gurugram, Haryana

Work Mode : Work From Office (5 Days a Week)

Employment Type : Full-Time

Role Overview :

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS Data Engineering, Data Lake Architecture, Python, SQL, PySpark, and Apache Airflow.

The candidate will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data pipelines using AWS cloud technologies. The role involves building end-to-end ETL / ELT workflows, implementing data transformation and processing solutions, optimizing pipeline performance, and ensuring the reliability of production data systems.

Mandatory Skills :

AWS Data Engineering | Data Lake Architecture | Amazon S3 | AWS Glue | Amazon EMR | Athena | Redshift | Python | SQL | PySpark | Apache Spark | Apache Airflow | ETL/ELT | PostgreSQL | Data Modelling | Pipeline Optimization | Production Support | Root Cause Analysis

Key Responsibilities :

• Design, develop, and maintain end-to-end data pipelines for data ingestion, staging, transformation, curation, and consumption.
• Build and manage AWS-based data lake solutions using Amazon S3, AWS Glue, Amazon EMR, Athena, and Redshift.
• Develop, maintain, and optimize ETL / ELT pipelines using Python, SQL, Apache Spark, and PySpark.
• Implement and manage workflow orchestration using Apache Airflow, including scheduling, dependency management, retries, monitoring, and failure handling.
• Integrate data from multiple structured and semi-structured sources.
• Design and maintain data models, dimensional models, and analytical data marts.
• Work with relational databases, including PostgreSQL and Amazon RDS.
• Optimize Spark jobs, SQL queries, data transformations, and cloud resource utilization.
• Implement data quality checks, validation rules, and error-handling mechanisms.
• Monitor production data pipelines, troubleshoot failures, and perform root cause analysis (RCA).
• Ensure pipeline reliability, scalability, performance, and compliance with service-level agreements (SLAs).
• Follow data security, access management, governance, and sensitive-data protection best practices.
• Collaborate with business, analytics, and application teams to deliver effective data solutions.
• Maintain technical documentation and follow established data engineering standards.

Required Technical Skills :

• AWS Data Engineering : Strong hands-on experience with AWS-based data engineering and data lake architecture.
• AWS Services : Amazon S3, AWS Glue, Glue Data Catalog, Amazon EMR, Athena, Redshift, AWS Lambda, and IAM.
• Programming : Strong proficiency in Python and SQL.
• Big Data Processing : Hands-on experience with Apache Spark and PySpark.
• Workflow Orchestration : Experience developing and managing workflows using Apache Airflow.
• ETL/ELT Development : Experience building scalable data ingestion, transformation, and processing pipelines.
• Database Technologies : Good understanding of relational databases and PostgreSQL concepts.
• Data Modelling : Experience with data modelling, data transformation, and analytical data structures.
• Production Support : Ability to monitor pipelines, troubleshoot failures, perform RCA, and resolve production issues.
• Performance Optimization : Experience optimizing SQL queries, Spark jobs, and data processing workflows.

Good-to-Have Skills :

• Experience with AWS Database Migration Service (DMS) and Amazon RDS.
• Familiarity with dbt for data transformation and modelling.
• Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.
• Exposure to Apache Kafka and real-time data streaming.
• Familiarity with GraphQL, Hasura, or similar data-serving technologies.
• Understanding of data governance and data quality frameworks.
• Experience in the BFSI, healthcare, or health insurance domain.

Candidate Requirements :

• 5+ years of relevant experience in Data Engineering.
• Proven hands-on experience building and maintaining production-grade data pipelines on AWS.
• Strong understanding of data lake architecture and large-scale data processing.
• Practical experience with Apache Airflow and Spark / PySpark-based transformations.
• Strong programming, analytical, debugging, and problem-solving skills.
• Experience handling production incidents, pipeline failures, and performance bottlenecks.
• Ability to take ownership of deliverables and work effectively with cross-functional teams.
• Good communication, collaboration, and technical documentation skills.
• Willingness to work from the Gurugram office five days a week.

Ideal Candidate :

The ideal candidate is a hands-on Data Engineer who can independently build, optimize, and maintain AWS-based data pipelines, work with large datasets using PySpark, orchestrate workflows through Airflow, and resolve production issues in a business-critical environment.

Description

We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance data platforms that power advertising insights and analytics. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Spark and Scala.

You will work on designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Build and optimize distributed data applications using Spark and Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Work with large datasets to ensure data quality, consistency, and performance.
• Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, and cost efficiency.
• Deploy and manage data workloads in cloud and containerized environments.
• Troubleshoot production issues and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering or Big Data Engineering.
• Strong hands-on experience with Apache Spark and Scala.
• Experience building and maintaining ETL pipelines.
• Familiarity with Google Cloud Storage (GCS).
• Experience with Kubernetes (K8s).
• Strong SQL skills and understanding of distributed data processing.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with AWS and cloud-native data services.
• Familiarity with streaming technologies such as Kafka.
• Experience working on large-scale data platforms or AdTech systems.
• Exposure to orchestration tools such as Airflow.

Benefits

• Best-in-class salary: We hire strong talent and compensate accordingly.
• Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
• Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
• High-impact work: Build AI-first systems and products used at scale by global clients.

About Us

Proximity is the trusted technology, design, and consulting partner for some of the biggest Sports, Media, and Entertainment companies in the world. We’re headquartered in San Francisco and have offices in Palo Alto, Dubai, Mumbai, and Bangalore.

Since 2019, Proximity has built high-impact, scalable products used by millions of users every day. Today, we are a global team of engineers, designers, product managers, and experts solving complex problems and building cutting-edge technology at scale.

Job Summary

We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.

Technical Skills

• Strong hands-on experience in Python and PySpark development.
• Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
• Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Power BI Data Modeling and Semantic Layer development.
• Proficiency in DAX (Data Analysis Expressions).
• Experience designing and managing Semantic Models in Power BI.
• Strong SQL skills and experience working with large datasets.
• Knowledge of data warehousing concepts and best practices.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Exposure to modern data platforms like Databricks.
• Understanding of data governance and data quality frameworks.

Job Title : Tech Lead – Data Lake Platform

Number of Positions : 2

Experience : 7+ Years

Role Type : Technical Lead / Data Platform Lead

Domain : Data Engineering / Data Platform / AWS

About the Role :

We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.

The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.

As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.

Core Tech Stack :

AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka

Key Responsibilities :

• Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
• Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
• Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
• Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
• Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
• Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
• Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
• Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
• Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
• Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
• Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
• Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.

Must-Have Skills :

• 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
• Strong hands-on experience with AWS Data Services, including:
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Athena
• Amazon Redshift
• AWS DMS
• AWS Lambda
• Amazon RDS
• AWS IAM
• Strong production experience with Apache Airflow.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
• Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
• Strong understanding of data lake architecture and enterprise data platforms.
• Proven experience leading engineers and driving technical decisions.
• Hands-on experience with production support, troubleshooting, incident management, and RCA.
• Strong understanding of data platform performance, scalability, reliability, and SLA management.

Good-to-Have Skills :

• Experience with DBT and modern data transformation practices.
• Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
• Strong knowledge of Amazon Redshift workload optimization, including :
• Distribution keys
• Sort keys
• Spectrum
• External tables
• Experience with Kafka or other streaming/data ingestion technologies.
• Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
• Experience with Lambda and API Gateway for API-based data serving.
• Experience with enterprise data governance, data quality, security, and compliance.
• Experience in BFSI / Banking / Financial Services / Insurance domain.

Ideal Candidate :

The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.