Lead II - Data Engineering -Python - Databricks, PySpark, Python at Global digital transformation solutions provider. · Bengaluru (Bangalore) · 7 - 9 years · ₹15L - ₹28L / yr · Posted 15 Dec 2025

Lead II - Data Engineering -Python - Databricks, PySpark, Python
at Global digital transformation solutions provider.
Role Proficiency:
This role requires proficiency in developing data pipelines including coding and testing for ingesting wrangling transforming and joining data from various sources. The ideal candidate should be adept in ETL tools like Informatica Glue Databricks and DataProc with strong coding skills in Python PySpark and SQL. This position demands independence and proficiency across various data domains. Expertise in data warehousing solutions such as Snowflake BigQuery Lakehouse and Delta Lake is essential including the ability to calculate processing costs and address performance issues. A solid understanding of DevOps and infrastructure needs is also required.
Skill Examples:
- Proficiency in SQL Python or other programming languages used for data manipulation.
- Experience with ETL tools such as Apache Airflow Talend Informatica AWS Glue Dataproc and Azure ADF.
- Hands-on experience with cloud platforms like AWS Azure or Google Cloud particularly with data-related services (e.g. AWS Glue BigQuery).
- Conduct tests on data pipelines and evaluate results against data quality and performance specifications.
- Experience in performance tuning.
- Experience in data warehouse design and cost improvements.
- Apply and optimize data models for efficient storage retrieval and processing of large datasets.
- Communicate and explain design/development aspects to customers.
- Estimate time and resource requirements for developing/debugging features/components.
- Participate in RFP responses and solutioning.
- Mentor team members and guide them in relevant upskilling and certification.
Knowledge Examples:
- Knowledge of various ETL services used by cloud providers including Apache PySpark AWS Glue GCP DataProc/Dataflow Azure ADF and ADLF.
- Proficient in SQL for analytics and windowing functions.
- Understanding of data schemas and models.
- Familiarity with domain-related data.
- Knowledge of data warehouse optimization techniques.
- Understanding of data security concepts.
- Awareness of patterns frameworks and automation practices.
Additional Comments:
# of Resources: 22 Role(s): Technical Role Location(s): India Planned Start Date: 1/1/2026 Planned End Date: 6/30/2026
Project Overview:
Role Scope / Deliverables: We are seeking highly skilled Data Engineer with strong experience in Databricks, PySpark, Python, SQL, and AWS to join our data engineering team on or before 1st week of Dec, 2025.
The candidate will be responsible for designing, developing, and optimizing large-scale data pipelines and analytics solutions that drive business insights and operational efficiency.
Design, build, and maintain scalable data pipelines using Databricks and PySpark.
Develop and optimize complex SQL queries for data extraction, transformation, and analysis.
Implement data integration solutions across multiple AWS services (S3, Glue, Lambda, Redshift, EMR, etc.).
Collaborate with analytics, data science, and business teams to deliver clean, reliable, and timely datasets.
Ensure data quality, performance, and reliability across data workflows.
Participate in code reviews, data architecture discussions, and performance optimization initiatives.
Support migration and modernization efforts for legacy data systems to modern cloud-based solutions.
Key Skills:
Hands-on experience with Databricks, PySpark & Python for building ETL/ELT pipelines.
Proficiency in SQL (performance tuning, complex joins, CTEs, window functions).
Strong understanding of AWS services (S3, Glue, Lambda, Redshift, CloudWatch, etc.).
Experience with data modeling, schema design, and performance optimization.
Familiarity with CI/CD pipelines, version control (Git), and workflow orchestration (Airflow preferred).
Excellent problem-solving, communication, and collaboration skills.
Skills: Databricks, Pyspark & Python, Sql, Aws Services
Must-Haves
Python/PySpark (5+ years), SQL (5+ years), Databricks (3+ years), AWS Services (3+ years), ETL tools (Informatica, Glue, DataProc) (3+ years)
Hands-on experience with Databricks, PySpark & Python for ETL/ELT pipelines.
Proficiency in SQL (performance tuning, complex joins, CTEs, window functions).
Strong understanding of AWS services (S3, Glue, Lambda, Redshift, CloudWatch, etc.).
Experience with data modeling, schema design, and performance optimization.
Familiarity with CI/CD pipelines, Git, and workflow orchestration (Airflow preferred).
******
Notice period - Immediate to 15 days
Location: Bangalore

Similar jobs (10)

SENIOR DATA ENGINEER – DATABRICKS & SNOWFLAKE
Experience: 7+ Years
Location: Remote
Employment Type: Full-Time
Budget: Up to ₹30 LPA
Time Zone: US Eastern Time (ET)
Shift: US ET Night Shift
Job Description
We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Snowflake, Apache Spark, and AWS to design, develop, and maintain scalable data engineering solutions.
The candidate will work on integrating data from multiple sources, building modern data pipelines, developing enterprise data models, and preparing data platforms for AI/ML applications.
Experience in healthcare, scientific data, or the histopathology domain will be an added advantage.
Key Responsibilities
- Design, build, and maintain scalable data pipelines to acquire, integrate, and manage data from diverse sources such as laboratory systems, MES, clinical supply, quality systems, and external partners.
- Develop and optimize data workflows for structured and unstructured data using Python, PySpark, R, SQL, Databricks, and Snowflake.
- Design and maintain enterprise-level data models and data repositories.
- Perform ETL development, data warehousing, database design, and data integration.
- Ensure data quality, versioning, traceability, and consistency to support AI/ML readiness.
- Collaborate with stakeholders and technical teams to translate business requirements into technical solutions.
- Manage multiple projects while ensuring scalability, performance, and reliability.
Required Skills
- 7+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks and Snowflake.
- Excellent knowledge of Apache Spark / PySpark.
- Strong experience with AWS cloud services and AWS RDS.
- Experience with Oracle and Microsoft SQL Server (MSSQL).
- Proficiency in Python, R, and SQL.
- Strong understanding of ETL, Data Warehousing (DWH), data modeling, and database design.
- Experience with dbt and cloud-based data architectures.
- Knowledge of NoSQL and graph databases.
- Strong analytical, problem-solving, and stakeholder-management skills.
- Good English communication skills (B2 level or above).
- Willingness to work in the US Eastern Time (ET) night shift.
Preferred / Good to Have
- Experience in histopathology, healthcare, life sciences, or scientific data environments.
- Knowledge of CDISC, HL7, FHIR, OMOP, DICOM, or other regulated data standards.
- Experience handling high-dimensional data, including imaging and sensor data.
- Familiarity with MLOps and machine learning model deployment workflows.
- Knowledge of Manufacturing Execution Systems (MES) and Laboratory Information Management Systems (LIMS).
- Exposure to knowledge graphs and ontology-driven architectures.
Education
Bachelor's degree in Engineering, Data Science, Life Sciences, Computer Science, or a related field. A master's degree is preferred.
Interested candidates can apply with their updated resumes.

Job Title : Senior Data Engineer – Databricks
Experience : 14 to 20 Years
Location : HSR Layout, Bangalore
Work Mode : Hybrid – 3 Days WFO
Shift : 11:30 AM – 07:30 PM IST
Positions : 2
Notice Period : Immediate Joiners Only
Interview : 1 Technical Round + 2 Client Rounds
Role Overview :
We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.
The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.
Must-Have Skills :
- 14 to 20 years of Data Engineering experience
- Databricks & Apache Spark / PySpark
- Python & SQL
- AWS Cloud
- Lakehouse Architecture
- ETL / ELT & Distributed Data Processing
- Batch & Streaming Pipelines
- Data Pipeline Optimization & Data Modeling
- CDC & Incremental Processing
- Git, CI/CD & Testing
- Data Quality, Monitoring & Observability
- Technical Leadership & Stakeholder Management
Key Responsibilities :
- Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
- Own data products from design through production.
- Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
- Optimize pipelines for performance, scalability, reliability, and cost.
- Design scalable data architectures and data models.
- Implement data quality, monitoring, lineage, and CI/CD practices.
- Lead technical discussions and mentor engineering teams.
- Collaborate with business stakeholders, architects, product owners, and engineering teams.
- Remain hands-on while providing technical leadership.
Ideal Candidate :
A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.
🔴 Super Urgent : Only Bangalore-based immediate joiners.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
About the Role
We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.
Key Responsibilities
- Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
- Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
- Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
- Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
- Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
- Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.
Required Skills
- Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
- Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
- Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
- Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
- Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
- Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
- Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
- Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
- Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
- Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.
Nice to Have
- Hands-on exposure to Microsoft Fabric for data integration and analytics.
- Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
- Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
- Domain expertise in financial services.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Title : Data / AI / GenAI – Engineering & Analytics
Experience : 4+ Years
Location : Noida / Gurugram / Ahmedabad
Employment Type : Full-time | On-site at client location
Working Days : 5 days per week
Working Hours : Standard business hours
Position Overview : We are seeking professionals to deliver modern data engineering, business intelligence, cloud analytics and AI-powered applications, according to their specialization.
Mandatory Skills :
Python, SQL, ETL/ELT, Data Engineering, Power BI, Azure Data Factory, Databricks, PySpark, Data Modeling, GenAI/LLMs, RAG, Vector Databases, AI Agents (as per role specialization).
Key Roles / Specializations :
- Power BI Developers : Dashboards, DAX, Power Query, reporting and data modeling
- Azure Data Engineers : Azure Data Factory, Data Lake, Azure SQL and cloud pipelines
- Databricks Engineers : Apache Spark, PySpark, Delta Lake and scalable processing
- Data Engineers : ETL/ELT, SQL, Python, data warehousing and quality
- GenAI / RAG Engineers : LLMs, prompts, embeddings, vector databases and retrieval pipelines
- Agentic AI Engineers : AI agents, tool calling, orchestration and multi-agent workflows
Key Responsibilities :
- Design, develop and maintain data pipelines, analytics solutions or AI applications.
- Build and optimize data integration, transformation and processing workflows.
- Develop dashboards and analytical models for business decision-making.
- Use cloud data platforms and modern engineering frameworks.
- Develop and evaluate LLM applications, RAG solutions or AI agents as applicable.
- Ensure data quality, security, performance and reliability.
- Collaborate with engineering teams and stakeholders on testing, deployment and documentation.
Required Skills & Qualifications :
- Bachelor’s degree in Computer Science, IT, Data Science, Engineering or related discipline.
- 4–5 years of relevant experience in data, analytics, AI or software engineering.
- Hands-on expertise in at least one listed specialization; not all technologies are required.
- Relevant Python, SQL, DAX or other technical skills depending on the position.
- Understanding of cloud platforms, APIs, integration and software practices.
Preferred Qualifications :
- Azure, Databricks or Microsoft Fabric experience.
- For GenAI roles : Azure OpenAI, OpenAI APIs, LangChain, LlamaIndex or LangGraph.
- For relevant roles : vector databases, CI/CD, Docker, MLOps or LLMOps.
Note : Candidates are expected to have expertise in their relevant specialization, not in every listed technology or role.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Summary
We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.
Technical Skills
- Strong hands-on experience in Python and PySpark development.
- Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
- Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
- Experience with Power BI Data Modeling and Semantic Layer development.
- Proficiency in DAX (Data Analysis Expressions).
- Experience designing and managing Semantic Models in Power BI.
- Strong SQL skills and experience working with large datasets.
- Knowledge of data warehousing concepts and best practices.
Preferred Skills
- Experience with cloud platforms such as Azure, AWS, or GCP.
- Exposure to modern data platforms like Databricks.
- Understanding of data governance and data quality frameworks.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Description:
We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.
Key Responsibilities
· 5-8 years of professional work experience in a relevant field
· Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.
· Good Experience on Microsoft Dynamics 365
· Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.
· Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.
· Good knowledge of SQL and Python for data manipulation, transformation, and analysis
· Understand business requirements to set functional specifications for reporting applications
- Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.
- Data Integration: Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.
- Power BI Integration: Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.
- Performance Optimization: Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.
- Data Modeling: Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.
- Cloud Integration: Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.
- Security & Compliance: Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.
- Technical Skills:
- Proficiency in Databricks, including Delta Lake, Spark SQL.
- Strong programming skills in Python, Scala, or Java.
- Experience with Apache Spark for big data processing.
- Knowledge of SQL for querying and transforming data.
- Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.
Preferred Qualifications
- Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience with real-time data processing and streaming
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Summary
Role Overview
We are looking for an experienced Data Engineer with strong expertise in Python, ETL, Advanced SQL, CI/CD, DevOps, and Data Analytics. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, transforming large datasets, and supporting data-driven applications.
Experience with Google Cloud Platform (GCP) will be an added advantage.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL.
- Develop complex and optimized SQL queries, stored procedures, and data transformations.
- Build and maintain reliable data integration workflows across multiple data sources.
- Perform data cleansing, validation, transformation, and quality checks.
- Analyze data and provide insights to support business and technical requirements.
- Implement and maintain CI/CD pipelines for data engineering applications.
- Work with DevOps practices and tools to automate deployments, monitoring, and infrastructure processes.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Optimize data processing workflows for performance, scalability, and reliability.
- Collaborate with Data Analysts, Data Scientists, Developers, and other stakeholders.
- Follow best practices for version control, testing, documentation, and deployment.
- Contribute to cloud-based data engineering initiatives, preferably on GCP.
Required Skills
- 5–7 years of hands-on experience in Data Engineering.
- Strong programming skills in Python.
- Strong expertise in Advanced SQL and database concepts.
- Hands-on experience with ETL/ELT processes and data pipelines.
- Good understanding of Data Warehousing and Data Modeling concepts.
- Experience with CI/CD practices and tools.
- Strong understanding of DevOps principles, automation, and deployment processes.
- Strong data analytics and problem-solving skills.
- Experience working with large datasets and performance optimization.
- Good understanding of Git/version control and software development best practices.
Good to Have
- Hands-on experience with Google Cloud Platform (GCP).
- Exposure to GCP data services such as BigQuery, Cloud Storage, Dataflow, Composer, or Pub/Sub.
- Experience with containerization/orchestration technologies such as Docker/Kubernetes.
- Experience with workflow orchestration tools such as Airflow.
- Knowledge of cloud-based data architecture and distributed data processing.
Preferred Candidate Profile
- Strong analytical and problem-solving abilities.
- Good communication and stakeholder management skills.
- Ability to work independently as well as in a collaborative team environment.
- Strong ownership of data pipelines and production systems.
- Candidates who can join at short notice are preferred.
Mandatory Skills
Data Engineer, Python , ETL, GCP, Advanced SQL, Strong Data Analytics skills, CICD, Devops
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Responsibilities and JD
Job Description: We are looking for a Senior Developer with strong expertise in PySpark, Databricks, and Snowflake to build scalable data engineering solutions and enterprise data platforms.
Key Responsibilities:
- Design, develop, and maintain ETL/ELT pipelines using PySpark, Databricks, and Snowflake.
- Develop batch and real-time data processing solutions for structured and semi-structured data.
- Build and optimize Databricks notebooks, workflows, and Delta Lake solutions.
- Design and implement Snowflake databases, schemas, views, stored procedures, tasks, and streams.
- Develop scalable data models, data marts, and data warehouse solutions.
- Optimize PySpark jobs, Databricks workloads, and Snowflake queries for performance and cost efficiency.
- Implement data quality, validation, governance, and security controls.
- Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.
- Manage source control and CI/CD deployments using Git and Azure DevOps.
- Troubleshoot production issues, perform root cause analysis, and ensure pipeline reliability.
- Mentor junior team members and participate in code reviews and technical design discussions.
Required Skills: PySpark, Databricks, Snowflake, Python, SQL.
Experience: 5+ years of Data Engineering experience with strong hands-on expertise in PySpark, Databricks, and Snowflake.

Senior Data Engineer Databricks and Snowflake
Exp- 7+ years
Rate: 30 LPA
Location: Remote
Time zone: ET (US Eastern time)
Shift: Night Shift
Type: Fulltime
Main tech requirements: Databricks, Snowflake, Spark, AWS, RDS, Oracle, MSSQL, domain of histopathology
Responsibilities:
Design, build, and maintain scalable data pipelines for acquiring, integrating, and managing data from diverse data generation sources and systems (e.g., lab systems, MES, clinical supply, quality systems, external partners).
Create and optimize data flows for structured and unstructured data using Python (PySpark), R, SQL, Databricks, Snowflake, and other modern engineering tools.
Develop and maintain specific data repositories, implementing enterprise-level data models, and creating new models as needed.
Enable AI/ML readiness by ensuring data is well-structured, versioned, traceable, and semantically aligned with enterprise data standards.
What we expect:
Bachelor’s degree in Engineering, Data Science, Life Sciences, Computer Science, or related field; advanced degree preferred.
6+ years of experience in data engineering, including data modeling and database design, preferably in a scientific, manufacturing, or healthcare environment.
Proficiency with Python, R, SQL, and cloud-based architectures (AWS services, Snowflake, Databricks, Redshift) along with Spark and dbt.
Proficiency in RDS
Expertise in ETL and DWH.
Experience with NoSQL and graph databases.
English language proficiency of B2+
Strong analytical, problem-solving, and stakeholder-management skills, with the ability to translate discussions into actionable requirements.
Ability to drive multiple exciting projects simultaneously with strong organizational skills and adaptability.
Nice to have:
Experience working with the histopathology domain
Experience with regulated or standards-driven data environments, such as CDISC, HL7, FHIR, OMOP, DICOM, or manufacturing/quality data standards.
Familiarity with high dimensional data (e.g., imaging, sensor data, etc).
Experience with principles connecting to or feeding MLOps and model deployment workflows.
Knowledge of manufacturing systems (MES), laboratory information systems, or industrial data systems.
Exposure to knowledge graph or ontology-driven architectures.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
1st virtual , 2nd round F2F
Python pyspark, SQL, data engineer
5+yrs
9+yrs
Bangalore/Hyderabad
immediate to 15days.








