Data Engineer (Databricks, Python)

Job description: Data Architect – Databricks / AWS

Job Summary

We are looking for an experienced Data Architect to define and drive the target data architecture for the Horizon MVP and its future evolution. The role will be responsible for designing a scalable, secure, governed cloud data platform covering ingestion, storage, processing, analytics, APIs, and downstream data consumption.

The architect will work closely with Data Engineering, Backend, DevOps, QA, and business stakeholders to establish architecture standards and ensure the platform is ready for advanced analytics, AI/ML, vector storage, and future LLM-based capabilities.

• Job Title: Data Architect
• Experience: 8+ Years
• Relevant Architecture Experience: 3+ Years in Data Architecture
• Location: Chennai / Pune
• Work Mode: Hybrid – 3 Days WFO
• Budget: Up to 24 LPA
• Payroll: Haparz
• Notice Period: Immediate Preferred

Key Responsibilities

• Define the target data architecture for the Horizon MVP and establish an architecture roadmap for future scalability.
• Design end-to-end architecture covering data ingestion, storage, processing, serving, reporting, APIs, and downstream applications.
• Establish canonical data models and schemas for travel signals, corridors, sources, evidence, scores, and generated insights.
• Define data normalization strategies for structured, semi-structured, and unstructured data from multiple external sources.
• Design and govern the Databricks platform architecture, including Unity Catalog, data schemas, access controls, and governance standards.
• Establish data-retention, lineage, data-quality, security, privacy, and compliance controls.
• Define secure integration patterns between Databricks, AWS PRODIGY, SharePoint, external APIs, and downstream applications.
• Design scalable data processing for 30-day signal windows, convergence/divergence scoring, corridor ranking, and spike detection.
• Define architecture patterns that support future vector storage, embeddings, LLM integration, and multi-year analytics.
• Design reliable batch and API-driven ingestion frameworks for structured and unstructured data.
• Review technical designs, identify architectural risks, and provide technical direction to engineering teams.
• Guide backend, data engineering, DevOps, and QA teams in implementing architecture standards.
• Ensure architecture decisions align with enterprise security, RBAC, PII handling, privacy, and operational requirements.
• Communicate architecture decisions, trade-offs, and technical recommendations effectively to technical and business stakeholders.
• 
  

What We’re Looking For

• 8+ years of experience in data engineering, data platforms, or data architecture, with at least 3+ years in a Data Architect capacity.
• Strong hands-on experience designing cloud-based data platforms, lakehouses, or analytical platforms.
• Advanced knowledge of Databricks, Apache Spark/PySpark, Delta Lake, and Unity Catalog.
• Strong understanding of AWS data services, IAM, networking, and secure cloud integration patterns.
• Strong expertise in data modelling, metadata management, data lineage, data quality, retention, and governance.
• Experience architecting batch and API-based ingestion pipelines for structured, semi-structured, and unstructured data.
• Understanding of AI/ML workloads, feature pipelines, vector databases, embeddings, and LLM integration patterns.
• Experience designing APIs and downstream data-serving architectures.
• Strong knowledge of PII protection, RBAC, data privacy, and enterprise security controls.
• Excellent architectural communication and stakeholder-management skills.

Hiring : Senior Databricks AI Architect

Exp : 15 - 18 yrs

Work Location : Pune WFO

Skills :

10 +years of experience in Data Engineering, Data Architecture, Analytics, or Software Engineering.

Minimum 5 years of hands-on experience with Databricks (Mandatory).

Strong expertise in designing and implementing enterprise-scale data platforms on Databricks.

Hands-on experience with AI-powered engineering tools such as Databricks Genie, Cursor, GitHub Copilot, or similar AI platforms.

Strong proficiency in Python, SQL, Spark, Delta Lake, and Databricks notebooks.

Excellent communication, stakeholder management

Job Description:

Position: Lead / Senior Data Engineer

Location: Chennai

Shift: US Eastern Time ( 5:00 PM – 2:00 AM )

Experience : 8+ years

Notice Period: Immediate Joiner only

Roles and Responsibilities:

Role Overview

The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities:

Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

Define technical strategy, architecture, and best practices for development and data engineering.

Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

Construct infrastructure for efficient ETL processes from various sources and storage systems.

Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

Lead the implementation of algorithms and prototypes to transform raw data into useful information.

Review code for quality, scalability, and performance.

Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

Develop analytical tools, programs, and reporting mechanisms.

Create data validation methods and data analysis tools.

Interpret data trends and patterns to establish operational alerts.

Conduct complex data analysis and present results effectively.

Prepare data for prescriptive and predictive modeling.

Ensure compliance with data governance and security policies.

Troubleshoot, debug, and resolve complex technical issues.

Drive continuous improvement in software and data development processes, tools, and methodologies.

Mentor and guide engineers through code reviews, technical discussions, and training.

Ensure timely delivery of projects while maintaining high engineering standards.

Continuously explore opportunities to enhance data quality and reliability.

Apply strong programming and problem-solving skills to develop scalable solutions.

Demonstrate passion for testing strategy, problem-solving, and continuous learning.

Willingness to acquire new skills and knowledge.

Possess a product/engineering mindset to drive impactful data solutions.

Experience working in distributed environments with global teams.

Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

High proficiency in Python, Scala, and Spark for applied large-scale data processing.

Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

Solid understanding of batch and streaming data processing techniques.

Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

Familiarity with Agile methodologies.

Obsession for service observability, instrumentation, monitoring, and alerting.

Knowledge or experience in architectural best practices for building data lakes.

Qualifications - Bachelor’s degree in computer science, Engineering, Information Systems, or related field.

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.

Data Engineer

Data Lakehouse & Platform Engineering 

About the Role

We are hiring Data Engineer to own the lifecycle of our enterprise Data Lakehouse platform. We are looking for engineers who think in systems, make platform-level design decisions, and can build and operate a production-grade, multi-source lakehouse from the ground up, covering ingestion through consumption across a complex, multi-cloud source landscape.

You will be the technical authority for a platform that consolidates data from 18+ enterprise products (Costpoint, GovWin, Specpoint, Vantagepoint, and others) into a governed, medallion-architected data lake on AWS S3 with Apache Iceberg table format, orchestrated via AWS Step Functions, and queryable through AWS Athena and Trino. This role is end-to-end: you own ingestion, transformation, quality, orchestration, ML data supply, and BI consumption.

Key Responsibilities

•      Architect and evolve the full medallion lakehouse — Bronze, Silver, and Gold layers — on AWS S3 with Apache Iceberg; own schema design, partitioning, compaction, and retention policies.

•      Design and implement scalable Glue ETL (PySpark) pipelines for bronze_to_silver and silver_to_gold transformations, incorporating dbt for SQL-layer transformations where appropriate.

•      Own and extend CDC ingestion via Fivetran; manage schema evolution, connector health, and sync reliability across 18+ source products.

•      Build and maintain AWS Step Functions state machines and EventBridge schedules for end-to-end pipeline orchestration; implement Lambda-based quality and drift monitors.

•      Govern the Glue Catalog and Lake Formation policies; enforce column-level security, row-level access controls, and audit logging to meet SOC2 and regulatory requirements.

•      Architect the query layer — optimize Athena workgroups and partition pruning; plan and execute Trino-on-EKS deployment for sub-second analytics workloads.

•      Partner with data science teams on SageMaker data supply: feature engineering pipelines, training dataset preparation, and model registry integration.

•      Implement real-time and near-real-time streaming solutions using Kafka or Kinesis where sub-13-minute latency is required.

•      Lead platform modernization initiatives: evaluate emerging formats (Iceberg vs. Delta Lake vs. Hudi), tooling, and cost optimization strategies.

•      Establish and enforce data engineering best practices: code reviews, CI/CD for pipeline code, IaC (Terraform / CloudFormation), and incident response runbooks.

•      Mentor and level up junior and mid-level data engineers; define team standards for pipeline design, testing, and documentation.

Required Qualifications

•      Software or data engineering experience, with at least 4 years in an architect or technical lead capacity designing large-scale cloud data platforms.

•      Deep, hands-on expertise with AWS data services: S3, Glue (PySpark ETL), Athena, Step Functions, Lambda, EventBridge, Lake Formation, SageMaker, and CloudWatch.

•      Production experience with Apache Iceberg (or Delta Lake / Hudi) table formats — compaction, snapshot management, schema evolution, and time travel.

•      Strong PySpark and Python skills; ability to write, review, and optimize distributed data processing jobs at scale.

•      Hands-on experience with CDC-based ingestion platforms (Fivetran, Debezium, or equivalent) across heterogeneous source systems.

•      Proven experience designing and implementing medallion (Bronze/Silver/Gold) or equivalent multi-hop lakehouse architectures.

•      Experience with data pipeline orchestration: AWS Step Functions, Apache Airflow, or equivalent; event-driven pipeline design patterns.

•      Strong SQL skills; experience with Athena, Trino, Presto, or equivalent query engines for large-scale analytical workloads.

•      Familiarity with data governance tooling: catalog management (Glue Catalog, Apache Polaris/Iceberg REST), data lineage, access controls, and audit frameworks.

•      Experience with Infrastructure as Code (Terraform or CloudFormation) for data platform provisioning and drift management.

•      Solid understanding of dimensional modeling, schema design (star/snowflake), and data normalization for BI and analytics workloads.

•      Bachelor's degree in Computer Science, Engineering, or a related field; or equivalent professional experience.

Preferred Qualifications

•      Experience operating Trino or PrestoDB on Kubernetes (EKS); tuning for sub-second query latency and multi-tenant workloads.

•      Familiarity with streaming platforms (Kafka, Kinesis, or Pub/Sub) and real-time lakehouse patterns.

•      Experience with Apache Polaris or other Iceberg REST catalog implementations.

•      Exposure to SageMaker MLOps pipelines, Model Registry, and feature store patterns for ML data supply.

•      Experience with dbt (data build tool) for SQL-layer transformation and documentation in lakehouse environments.

•      Government contracting or ERP domain knowledge (Costpoint, Deltek, Oracle, or similar enterprise platforms) is a strong plus.

•      AWS certifications: Data Engineer Associate, Solutions Architect Professional, or equivalent.

What You Will Build

You will be a founding architect of a strategic, cross-product data platform that serves 18+ enterprise applications and their analytics, ML, and AI workloads. The decisions you make on schema, storage format, query layer, governance, and orchestration will shape the data foundation of the company for years. This is a high-impact, high-ownership role with direct visibility to senior leadership.

Solution Architect – AZURE Data Engineering

Job Overview

We are looking for an experienced Solution Architect – Data Engineering with strong expertise in designing data solutions and hands-on experience with Azure, Synapse, PySpark, Data Warehousing, and Data Lakes. The ideal candidate should have strong architectural and data engineering knowledge.

Key Responsibilities

• Design and implement scalable data architecture and solutions.
• Develop and manage Data Warehouse and Data Lake architectures.
• Design data platforms using Medallion Architecture.
• Lead Data Engineering and ETL activities.
• Work with Azure Synapse Analytics for data processing and analytics.
• Develop data solutions using PySpark / Apache Spark.
• Define and implement data validation and data quality processes.
• Collaborate with business, data, and technology teams to deliver effective data solutions.

Required Skills

• Strong experience in Solution Architecture / Data Architecture.
• Strong knowledge of Data Warehouse and Data Lake architecture.
• Good understanding of Medallion Architecture.
• Strong experience in Data Engineering and ETL.
• Hands-on experience with Microsoft Azure and Azure Synapse Analytics.
• Strong knowledge of PySpark / Apache Spark.
• Experience with Data Validation and Data Quality.
• Good communication and stakeholder management skills.
• 
  

Experience

8+ Years

Hi 

Position: Senior Data Engineer 

Experience: 5–7Years

Location: Bangalore / Chennai / Hyderabad / Pune

Work Model: Hybrid (Mandatory 2 Days Work From Office)

Job Description

• Design and build Bronze, Silver, and Gold data pipelines using Microsoft Fabric and Databricks.
• Implement Cascade 1.0 & 2.0 merge logic in the Silver layer, including handling new and nulled columns.
• Migrate legacy SQL Server tables, views, and stored procedures into Microsoft Fabric Notebooks and Pipelines.
• Develop scalable PySpark transformations and Delta Lake models while optimizing performance and cost.
• Establish data quality checks, lineage, and source-to-target reconciliation.
• Collaborate with Report Engineers and QA teams to deliver governed semantic models.

Mandatory Skills

• Microsoft Fabric (Lakehouse, OneLake, Pipelines)
• PySpark
• SQL / T-SQL
• Delta Lake
• Azure Data Factory (ADF)
• Power BI (Datasets / Semantic Models)
• Databricks
• Medallion Architecture (Bronze / Silver / Gold)

Preferred Skills

• SQL Server
• SSIS / SQL Server ETL Migration
• Lakehouse Architecture
• Data Quality & Validation
• Performance Tuning

Candidate Requirements

• 5-7 years of overall IT experience with a minimum of 5+ years in Data Engineering.
• Hands-on expertise in Microsoft Fabric and/or Databricks.
• Strong experience in PySpark, SQL/T-SQL, Delta Lake, and Azure Data Factory.
• Experience migrating legacy SQL Server/SSIS ETL workloads to modern Lakehouse platforms is highly preferred.
• Good understanding of Power BI datasets and semantic models.
• Strong communication and stakeholder management skills.

Interview & Work Model

• Face-to-Face HR interview is mandatory.
• Candidate should be flexible to travel to the Chennai office whenever required for project-related work.

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.

Must-Have Skills :

• 14 to 20 years of Data Engineering experience
• Databricks & Apache Spark / PySpark
• Python & SQL
• AWS Cloud
• Lakehouse Architecture
• ETL / ELT & Distributed Data Processing
• Batch & Streaming Pipelines
• Data Pipeline Optimization & Data Modeling
• CDC & Incremental Processing
• Git, CI/CD & Testing
• Data Quality, Monitoring & Observability
• Technical Leadership & Stakeholder Management

Key Responsibilities :

• Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
• Own data products from design through production.
• Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
• Optimize pipelines for performance, scalability, reliability, and cost.
• Design scalable data architectures and data models.
• Implement data quality, monitoring, lineage, and CI/CD practices.
• Lead technical discussions and mentor engineering teams.
• Collaborate with business stakeholders, architects, product owners, and engineering teams.
• Remain hands-on while providing technical leadership.

Ideal Candidate :

A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.

🔴 Super Urgent : Only Bangalore-based immediate joiners.

Roles & Responsibilities

• Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

2Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

4Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

5Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

6Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

8Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Azure Data Factory and Azure Databricks, processing 5 million+ records/week from 4+ source systems into a governed Lakehouse.