Sr Hadoop Operations Engineer

We are hiring experienced Data Engineers for a permanent opportunity in Bangalore.

Job Description

Role: Data Engineer

Location: Bangalore

Experience: 6+ Years

Key Responsibilities:

• Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
• Develop data engineering solutions using Python and Scala.
• Process and transform large volumes of structured and unstructured data.
• Build and optimize distributed data processing workflows for performance and scalability.
• Work with big data technologies and cloud-based data processing environments.
• Perform data integration, validation, transformation, and quality checks.
• Troubleshoot data pipeline issues and optimize processing performance.
• Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.

Required Skills:

• Strong hands-on experience with Amazon EMR.
• Proficiency in Python and Scala.
• Experience in big data processing and data pipeline development.
• Good understanding of distributed computing and data transformation techniques.
• Experience in performance tuning and troubleshooting data processing jobs.
• Strong analytical, problem-solving, and communication skills.

Job Title : Senior AWS Data Engineer – Data Lake Platform

Experience : 5+ Years

Location: Gurugram Sector 43, Haryana

Work Mode : Work From Office (5 Days a Week – Monday to Friday)

Open Positions : 8+

About the Role :

We are looking for a skilled Senior AWS Data Engineer with strong hands-on experience in building, managing, and optimizing enterprise-scale data lake platforms on AWS.

The ideal candidate will have expertise in AWS data services, PySpark, Apache Airflow, and large-scale data pipeline development. The role involves designing end-to-end data ingestion and processing workflows, maintaining data quality, optimizing platform performance, and supporting analytics and downstream data consumption requirements.

Mandatory skills :

AWS Data Lake, Amazon S3, EMR, Glue, Athena, Redshift, DMS, Lambda, RDS, IAM, PySpark, Apache Spark, Apache Airflow, Python, Advanced SQL, Data Modeling, Production Data Pipelines, Hasura/GraphQL, PostgreSQL, and Production Support.

Key Responsibilities :

• Design, develop, and maintain scalable AWS-based data lake platforms and end-to-end data pipelines.
• Build data ingestion, staging, transformation, curated, and consumption layers using AWS services.
• Develop and optimize data processing pipelines using PySpark, Apache Spark, and AWS EMR.
• Implement workflow orchestration and scheduling using Apache Airflow.
• Work with Amazon S3, AWS Glue, Glue Data Catalog, Amazon Athena, and Amazon Redshift for data storage, cataloging, processing, and analytics.
• Develop efficient SQL queries and implement data modeling strategies for enterprise reporting and analytics.
• Build and support data-serving solutions using Amazon RDS, AWS Lambda, API Gateway, and PostgreSQL-based platforms.
• Monitor pipeline performance, troubleshoot production issues, perform root-cause analysis, and ensure data pipeline SLAs.
• Implement data security, access controls, PII protection, and governance best practices across the platform.
• Optimize data processing performance, query execution, storage utilization, and AWS infrastructure costs.
• Collaborate with application, analytics, and business teams to deliver reliable and scalable data solutions.

Mandatory Skills :

• AWS Data Services : Amazon S3, EMR, Glue, Athena, Redshift, DMS, Lambda, RDS, and IAM.
• Data Engineering : Strong hands-on experience building and maintaining enterprise data lake platforms.
• Programming : Python, PySpark, and Apache Spark.
• Orchestration : Production experience with Apache Airflow.
• Database & SQL : Advanced SQL, data modeling, and query optimization.
• Data Architecture : Experience with data ingestion, staging, transformation, curated layers, and consumption architecture.
• Production Support : Pipeline monitoring, troubleshooting, root-cause analysis, and SLA management.
• API / Data Serving : Experience with Hasura or a similar GraphQL-based data-serving layer integrated with PostgreSQL.
• Platform Ownership : Ability to independently manage data pipelines, improve platform reliability, and collaborate with cross-functional teams.

Good-to-Have Skills :

• Experience with dbt for data transformation and modeling.
• Knowledge of data lakehouse architectures and table formats such as Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with Amazon Redshift workload optimization, distribution keys, sort keys, and Spectrum.
• Knowledge of Kafka and real-time or streaming data ingestion.
• Experience with PostgreSQL or Amazon Aurora as a data-serving layer.
• Familiarity with data governance, compliance, and sensitive-data management.
• Experience in the BFSI or health insurance domain.

Ideal Candidate Profile :

• 5+ years of hands-on experience in AWS Data Engineering.
• Strong experience building and supporting production-grade AWS data lake platforms.
• Practical expertise in PySpark, Apache Airflow, SQL, and AWS data services.
• Ability to troubleshoot complex data pipeline issues and optimize performance.
• Strong understanding of data architecture, data modeling, and production support practices.
• Candidates with experience in enterprise data platforms and insurance or BFSI environments will be preferred.

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
• Build and optimize distributed data applications using Apache Spark and Python Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Design and manage data workflows using Apache Airflow.
• Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
• Work with large datasets to ensure data quality, consistency, reliability, and performance.
• Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, performance, and cost efficiency.
• Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
• Strong hands-on experience with Apache Spark and Scala.
• Experience designing, building, and maintaining large-scale ETL pipelines.
• Strong hands-on experience with AWS, particularly Amazon S3.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong SQL skills and a solid understanding of distributed data processing concepts.
• Experience working with batch and/or streaming data pipelines.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with Databricks and the broader Databricks data platform.
• Familiarity with streaming technologies such as Apache Kafka.
• Experience working on large-scale data platforms handling high-volume data workloads.
• Exposure to additional AWS data services and cloud-native data architectures.

Job Title : Tech Lead – Data Lake Platform

Number of Positions : 2

Experience : 7+ Years

Role Type : Technical Lead / Data Platform Lead

Domain : Data Engineering / Data Platform / AWS

About the Role :

We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.

The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.

As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.

Core Tech Stack :

AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka

Key Responsibilities :

• Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
• Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
• Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
• Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
• Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
• Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
• Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
• Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
• Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
• Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
• Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
• Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.

Must-Have Skills :

• 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
• Strong hands-on experience with AWS Data Services, including:
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Athena
• Amazon Redshift
• AWS DMS
• AWS Lambda
• Amazon RDS
• AWS IAM
• Strong production experience with Apache Airflow.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
• Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
• Strong understanding of data lake architecture and enterprise data platforms.
• Proven experience leading engineers and driving technical decisions.
• Hands-on experience with production support, troubleshooting, incident management, and RCA.
• Strong understanding of data platform performance, scalability, reliability, and SLA management.

Good-to-Have Skills :

• Experience with DBT and modern data transformation practices.
• Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
• Strong knowledge of Amazon Redshift workload optimization, including :
• Distribution keys
• Sort keys
• Spectrum
• External tables
• Experience with Kafka or other streaming/data ingestion technologies.
• Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
• Experience with Lambda and API Gateway for API-based data serving.
• Experience with enterprise data governance, data quality, security, and compliance.
• Experience in BFSI / Banking / Financial Services / Insurance domain.

Ideal Candidate :

The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.

🚨 HIRING | Data Engineer – Amazon EMR 🔥

We are looking for experienced Data Engineers with strong hands-on expertise in Amazon EMR, Python, and Scala.

📌 JOB DETAILS:

💼 Experience: 6+ Years

💰 CTC: Up to 30 LPA

📍 Location: Bangalore

🔑 MANDATORY SKILLS:

• Amazon EMR

• Python

• Scala

• Data Engineering

📋 KEY REQUIREMENTS:

• 6+ years of relevant experience in Data Engineering.

• Strong hands-on experience with Amazon EMR.

• Strong programming skills in Python and Scala.

• Experience with large-scale data processing and data engineering solutions.

• Strong problem-solving and communication skills.

📩 Interested candidates can share their updated resumes:

📧 pavithra.e @vysystems. com

📞 8 9 2 5 8 0 2 4 5 7

🔄 Know someone who would be a good fit? Tag them or share this post!

#Hiring #DataEngineer #DataEngineering #AmazonEMR #Python #Scala #AWSJobs #BangaloreJobs #ITJobs #TechHiring #HiringNow