Senior Data Architect

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
• Build and optimize distributed data applications using Apache Spark and Python Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Design and manage data workflows using Apache Airflow.
• Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
• Work with large datasets to ensure data quality, consistency, reliability, and performance.
• Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, performance, and cost efficiency.
• Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
• Strong hands-on experience with Apache Spark and Scala.
• Experience designing, building, and maintaining large-scale ETL pipelines.
• Strong hands-on experience with AWS, particularly Amazon S3.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong SQL skills and a solid understanding of distributed data processing concepts.
• Experience working with batch and/or streaming data pipelines.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with Databricks and the broader Databricks data platform.
• Familiarity with streaming technologies such as Apache Kafka.
• Experience working on large-scale data platforms handling high-volume data workloads.
• Exposure to additional AWS data services and cloud-native data architectures.

Job Title : Tech Lead – Data Lake Platform

Number of Positions : 2

Experience : 7+ Years

Role Type : Technical Lead / Data Platform Lead

Domain : Data Engineering / Data Platform / AWS

About the Role :

We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.

The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.

As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.

Core Tech Stack :

AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka

Key Responsibilities :

• Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
• Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
• Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
• Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
• Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
• Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
• Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
• Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
• Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
• Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
• Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
• Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.

Must-Have Skills :

• 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
• Strong hands-on experience with AWS Data Services, including:
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Athena
• Amazon Redshift
• AWS DMS
• AWS Lambda
• Amazon RDS
• AWS IAM
• Strong production experience with Apache Airflow.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
• Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
• Strong understanding of data lake architecture and enterprise data platforms.
• Proven experience leading engineers and driving technical decisions.
• Hands-on experience with production support, troubleshooting, incident management, and RCA.
• Strong understanding of data platform performance, scalability, reliability, and SLA management.

Good-to-Have Skills :

• Experience with DBT and modern data transformation practices.
• Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
• Strong knowledge of Amazon Redshift workload optimization, including :
• Distribution keys
• Sort keys
• Spectrum
• External tables
• Experience with Kafka or other streaming/data ingestion technologies.
• Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
• Experience with Lambda and API Gateway for API-based data serving.
• Experience with enterprise data governance, data quality, security, and compliance.
• Experience in BFSI / Banking / Financial Services / Insurance domain.

Ideal Candidate :

The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.

4 - 10 years of experience in designing and buildingarchitecting highly resilient data platforms 

∙Strong knowledge of data engineering, architecture and data modeling 

∙Experience in platforms like Databricks and Snowflake 

∙Experience on building applications on cloud (AWS or Azure or Google Cloud) 

∙Strong analytical and problem-solving skills 

∙Prior experience in developing data or computation intensive (e.g. grid based) backend applications is an 

advantage 

∙OOP design skills with an understanding or at least personal interest towards the concepts of Functional 

Programming  

∙Willingness to understand and enhance other people’s code, being able to work in an environment where 

developers will oversee and work on wider components also dealing with older “legacy” code 

∙Strong programming skills (Java/ Scala / Python) skills with the willingness to pick up the other language if not 

already mastered at a sufficient level is important 

∙Spring knowledge is an advantage, but in general willingness to learn, work with and even enhance in-house 

developed frameworks is a must 

∙Prior experience in working with Git, Bitbucket, Jenkins, working with PR-s, using JIRA, following the Scrum Agile 

methodology is an advantage 

∙Prior knowledge of financial products is an advantage 

∙Bachelors or Masters in any relevant field of IT/Engineering area is an advantage  

Tech Lead – Data Lake Platform

Recruitment & Payroll Company: Neogencode Technologies

Job Location: Sector 43, Gurugram, Haryana

Work Mode: On-site (Work from Office)

Working Days: 5 days a week

Experience Required: 5+ years

Employment Type: Full-time

About the Opportunity

We are hiring a Tech Lead – Data Lake Platform. This opportunity involves working on an enterprise-grade AWS data lake platform supporting data engineering, analytics, reporting, and business applications.

The candidate will be employed on the payroll of Neogencode Technologies and will work at the client location in Sector 43, Gurugram.

Key Responsibilities

• Own and design end-to-end data lake architecture, including data ingestion, staging, curated layers, and data consumption.
• Lead data engineering pipelines using AWS services, Apache Airflow, Spark/PySpark, and dbt.
• Manage AWS data services such as S3, EMR, Glue, Athena, Redshift, Lambda, RDS, IAM, and DMS.
• Design and support API/data-serving solutions using PostgreSQL, Hasura GraphQL, or similar technologies.
• Improve pipeline performance, platform reliability, monitoring, and service-level agreements (SLAs).
• Ensure data security, access control, personally identifiable information (PII) protection, and governance.
• Mentor data engineers, conduct code reviews, and handle production incidents and root-cause analysis.
• Collaborate with business and application teams to deliver data and reporting solutions.

Required Skills

• 5+ years of experience in data engineering or platform engineering, including technical leadership.
• Strong hands-on experience with AWS data services and enterprise data lake architecture.
• Production experience with Apache Airflow, Spark/PySpark, and SQL.
• Strong understanding of data modeling and layered data architecture.
• Experience with PostgreSQL and Hasura GraphQL or a similar API/data-serving layer.
• Proven experience in technical leadership, production support, and troubleshooting.

Good to Have

• Experience with dbt and lakehouse table formats.
• Knowledge of Kafka or streaming data ingestion.
• Experience with Redshift optimization and workload design.

Data Architect – Databricks & AWS

• Strong experience in Data Architecture, Data Engineering, Databricks, Apache Spark/PySpark, Python, and Advanced SQL.
• Design and implement scalable ETL/ELT pipelines, data platforms, and Lakehouse architectures using Medallion Architecture.
• Experience with Databricks, Databricks Workflows, Unity Catalog OR Databricks Jobs
• Strong knowledge of Delta Lake, Databricks Workflows, Delta Live Tables (DLT), and dimensional data modeling.
• Hands-on experience with AWS services such as S3, Glue, IAM, Lambda, and CloudWatch.
• Experience with Apache Airflow, Data Warehouse concepts, Git/CI-CD, performance optimization, and data quality.
• Good to have exposure to Kafka/Structured Streaming, Unity Catalog, and modern data governance.

Description

We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance data platforms that power advertising insights and analytics. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Spark and Scala.

You will work on designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Build and optimize distributed data applications using Spark and Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Work with large datasets to ensure data quality, consistency, and performance.
• Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, and cost efficiency.
• Deploy and manage data workloads in cloud and containerized environments.
• Troubleshoot production issues and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering or Big Data Engineering.
• Strong hands-on experience with Apache Spark and Scala.
• Experience building and maintaining ETL pipelines.
• Familiarity with Google Cloud Storage (GCS).
• Experience with Kubernetes (K8s).
• Strong SQL skills and understanding of distributed data processing.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with AWS and cloud-native data services.
• Familiarity with streaming technologies such as Kafka.
• Experience working on large-scale data platforms or AdTech systems.
• Exposure to orchestration tools such as Airflow.

Benefits

• Best-in-class salary: We hire strong talent and compensate accordingly.
• Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
• Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
• High-impact work: Build AI-first systems and products used at scale by global clients.

About Us

Proximity is the trusted technology, design, and consulting partner for some of the biggest Sports, Media, and Entertainment companies in the world. We’re headquartered in San Francisco and have offices in Palo Alto, Dubai, Mumbai, and Bangalore.

Since 2019, Proximity has built high-impact, scalable products used by millions of users every day. Today, we are a global team of engineers, designers, product managers, and experts solving complex problems and building cutting-edge technology at scale.

🚨 HIRING | Data Engineer – Amazon EMR 🔥

We are looking for experienced Data Engineers with strong hands-on expertise in Amazon EMR, Python, and Scala.

📌 JOB DETAILS:

💼 Experience: 6+ Years

💰 CTC: Up to 30 LPA

📍 Location: Bangalore

🔑 MANDATORY SKILLS:

• Amazon EMR

• Python

• Scala

• Data Engineering

📋 KEY REQUIREMENTS:

• 6+ years of relevant experience in Data Engineering.

• Strong hands-on experience with Amazon EMR.

• Strong programming skills in Python and Scala.

• Experience with large-scale data processing and data engineering solutions.

• Strong problem-solving and communication skills.

📩 Interested candidates can share their updated resumes:

📧 pavithra.e @vysystems. com

📞 8 9 2 5 8 0 2 4 5 7

🔄 Know someone who would be a good fit? Tag them or share this post!

#Hiring #DataEngineer #DataEngineering #AmazonEMR #Python #Scala #AWSJobs #BangaloreJobs #ITJobs #TechHiring #HiringNow

role: data engineer

Python pyspark, SQL, data engineer

5+yrs

Bangalore/Hyderabad

immediate to 15days.

🚀 Job Title : Data Engineer – AWS, PySpark & Airflow

Experience : 5+ Years

Industry : Health Insurance / BFSI

Location : Sector 43, Gurugram, Haryana

Work Mode : Work From Office (5 Days a Week)

Employment Type : Full-Time

Role Overview :

We are looking for an experienced Data Engineer with strong hands-on expertise in AWS Data Engineering, Data Lake Architecture, Python, SQL, PySpark, and Apache Airflow.

The candidate will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data pipelines using AWS cloud technologies. The role involves building end-to-end ETL / ELT workflows, implementing data transformation and processing solutions, optimizing pipeline performance, and ensuring the reliability of production data systems.

Mandatory Skills :

AWS Data Engineering | Data Lake Architecture | Amazon S3 | AWS Glue | Amazon EMR | Athena | Redshift | Python | SQL | PySpark | Apache Spark | Apache Airflow | ETL/ELT | PostgreSQL | Data Modelling | Pipeline Optimization | Production Support | Root Cause Analysis

Key Responsibilities :

• Design, develop, and maintain end-to-end data pipelines for data ingestion, staging, transformation, curation, and consumption.
• Build and manage AWS-based data lake solutions using Amazon S3, AWS Glue, Amazon EMR, Athena, and Redshift.
• Develop, maintain, and optimize ETL / ELT pipelines using Python, SQL, Apache Spark, and PySpark.
• Implement and manage workflow orchestration using Apache Airflow, including scheduling, dependency management, retries, monitoring, and failure handling.
• Integrate data from multiple structured and semi-structured sources.
• Design and maintain data models, dimensional models, and analytical data marts.
• Work with relational databases, including PostgreSQL and Amazon RDS.
• Optimize Spark jobs, SQL queries, data transformations, and cloud resource utilization.
• Implement data quality checks, validation rules, and error-handling mechanisms.
• Monitor production data pipelines, troubleshoot failures, and perform root cause analysis (RCA).
• Ensure pipeline reliability, scalability, performance, and compliance with service-level agreements (SLAs).
• Follow data security, access management, governance, and sensitive-data protection best practices.
• Collaborate with business, analytics, and application teams to deliver effective data solutions.
• Maintain technical documentation and follow established data engineering standards.

Required Technical Skills :

• AWS Data Engineering : Strong hands-on experience with AWS-based data engineering and data lake architecture.
• AWS Services : Amazon S3, AWS Glue, Glue Data Catalog, Amazon EMR, Athena, Redshift, AWS Lambda, and IAM.
• Programming : Strong proficiency in Python and SQL.
• Big Data Processing : Hands-on experience with Apache Spark and PySpark.
• Workflow Orchestration : Experience developing and managing workflows using Apache Airflow.
• ETL/ELT Development : Experience building scalable data ingestion, transformation, and processing pipelines.
• Database Technologies : Good understanding of relational databases and PostgreSQL concepts.
• Data Modelling : Experience with data modelling, data transformation, and analytical data structures.
• Production Support : Ability to monitor pipelines, troubleshoot failures, perform RCA, and resolve production issues.
• Performance Optimization : Experience optimizing SQL queries, Spark jobs, and data processing workflows.

Good-to-Have Skills :

• Experience with AWS Database Migration Service (DMS) and Amazon RDS.
• Familiarity with dbt for data transformation and modelling.
• Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.
• Exposure to Apache Kafka and real-time data streaming.
• Familiarity with GraphQL, Hasura, or similar data-serving technologies.
• Understanding of data governance and data quality frameworks.
• Experience in the BFSI, healthcare, or health insurance domain.

Candidate Requirements :

• 5+ years of relevant experience in Data Engineering.
• Proven hands-on experience building and maintaining production-grade data pipelines on AWS.
• Strong understanding of data lake architecture and large-scale data processing.
• Practical experience with Apache Airflow and Spark / PySpark-based transformations.
• Strong programming, analytical, debugging, and problem-solving skills.
• Experience handling production incidents, pipeline failures, and performance bottlenecks.
• Ability to take ownership of deliverables and work effectively with cross-functional teams.
• Good communication, collaboration, and technical documentation skills.
• Willingness to work from the Gurugram office five days a week.

Ideal Candidate :

The ideal candidate is a hands-on Data Engineer who can independently build, optimize, and maintain AWS-based data pipelines, work with large datasets using PySpark, orchestrate workflows through Airflow, and resolve production issues in a business-critical environment.