Senior Data Engineer

Senior Data Engineer at Astegic · Remote only · 10 - 13 years · ₹30L - ₹50L / yr · Bootstrapped · Remote only · Posted 27 May 2025

Senior Data Engineer

at Astegic

Agency job

via Hunarstreet Technologies Pvt Ltd

10 - 13 yrs

₹30L - ₹50L / yr

Remote only

Skills

Scala

Apache Spark

Big Data

Python

Java

SQL

MySQL

databricks

POSITION:

Senior Data Engineer

The Senior Data Engineer will be responsible for building and extending our data pipeline architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys working with big data and building systems from the ground up.

You will collaborate with our software engineers, database architects, data analysts and data scientists to ensure our data delivery architecture is consistent throughout the platform. You must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

What You’ll Be Doing:

● Design and build parts of our data pipeline architecture for extraction, transformation, and loading of data from a wide variety of data sources using the latest Big Data technologies.

● Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

● Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.

● Work with machine learning, data, and analytics experts to drive innovation, accuracy and greater functionality in our data system.

Qualifications:

● Bachelor's degree in Engineering, Computer Science, or relevant

field.

● 10+ years of relevant and recent experience in a Data Engineer role.

● 5+ years recent experience with Apache Spark and solid understanding of the fundamentals.

● Deep understanding of Big Data concepts and distributed systems.

● Strong coding skills with Scala, Python, Java and/or other languages and the ability to quickly switch between them with ease.

● Advanced working SQL knowledge and experience working with a variety of relational databases such as Postgres and/or MySQL.

● Cloud Experience with DataBricks

● Experience working with data stored in many formats including Delta Tables, Parquet, CSV and JSON.

● Comfortable working in a linux shell environment and writing scripts as needed.

● Comfortable working in an Agile environment

● Machine Learning knowledge is a plus.

● Must be capable of working independently and delivering stable,

efficient and reliable software.

● Excellent written and verbal communication skills in English.

● Experience supporting and working with cross-functional teams in a dynamic environment.

REPORTING: This position will report to our CEO or any other Lead as assigned by Management.

EMPLOYMENT TYPE: Full-Time, Permanent LOCATION: Remote (Pan India) SHIFT TIMINGS: 2.00 pm-11:00pm IST

WHO WE ARE:

SalesIntel is the top revenue intelligence platform on the market. Our combination of automation and researchers allows us to reach 95% data accuracy for all our published contact data, while continuing to scale up our number of contacts. We currently have more than 5 million human-verifi ed contacts, another 70 million plus machine processed contacts, and the highest number of direct dial contacts in the industry. We guarantee our accuracy with our well-trained research team that re-verifi es every direct dial number, email, and contact every 90 days. With the most comprehensive contact and company data and our excellent customer service, SalesIntel has the best B2B data available. For more information, please visit – www.salesintel.io

WHAT WE OFFER: SalesIntel’s workplace is all about diversity. Different countries and cultures are represented in our workforce. We are growing at a fast pace and our work environment is constantly evolving with changing times. We motivate our team to better themselves by offering all the good stuff you’d expect like Holidays, Paid Leaves, Bonuses, Incentives, Medical Policy and company paid Training Programs.

SalesIntel is an Equal Opportunity Employer. We prohibit discrimination and harassment of any type and offer equal employment opportunities to employees and applicants without regard to race, color, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

POSITION:

Senior Data Engineer

What You’ll Be Doing:

● Design and build parts of our data pipeline architecture for extraction, transformation, and loading of data from a wide variety of data sources using the latest Big Data technologies.

● Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

● Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.

● Work with machine learning, data, and analytics experts to drive innovation, accuracy and greater functionality in our data system.

Qualifications:

● Bachelor's degree in Engineering, Computer Science, or relevant

field.

● 10+ years of relevant and recent experience in a Data Engineer role.

● 5+ years recent experience with Apache Spark and solid understanding of the fundamentals.

● Deep understanding of Big Data concepts and distributed systems.

● Strong coding skills with Scala, Python, Java and/or other languages and the ability to quickly switch between them with ease.

● Advanced working SQL knowledge and experience working with a variety of relational databases such as Postgres and/or MySQL.

● Cloud Experience with DataBricks

● Experience working with data stored in many formats including Delta Tables, Parquet, CSV and JSON.

● Comfortable working in a linux shell environment and writing scripts as needed.

● Comfortable working in an Agile environment

● Machine Learning knowledge is a plus.

● Must be capable of working independently and delivering stable,

efficient and reliable software.

● Excellent written and verbal communication skills in English.

● Experience supporting and working with cross-functional teams in a dynamic environment.

REPORTING: This position will report to our CEO or any other Lead as assigned by Management.

EMPLOYMENT TYPE: Full-Time, Permanent LOCATION: Remote (Pan India) SHIFT TIMINGS: 2.00 pm-11:00pm IST

WHO WE ARE:

Users love Cutshort

Read about what our users have to say about finding their next opportunity on Cutshort.

Shubham Vishwakarma

Full Stack Developer - Averlon

I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.

Companies hiring on Cutshort

About Astegic

Founded :

2002

Type :

Products & Services

Size :

100-1000

Stage :

Bootstrapped

About

Astegic offers a full range of mobile and IT services, from mobile app proof-of-concept development through full enterprise, cloud deployment strategies, implementation, and operational support. Our team is composed of experts in various areas including mobile solutions, enterprise IT services & operations, and UX & design. Click on each of our services below to learn…

Nikita Pasricha

Connect

Tanu Priya

Connect

Pooja Sapra

Connect

Company social profiles

Similar jobs (10)

Data Engineer - Data Platform Team

Posted by Tushar Vaghela

Bengaluru (Bangalore)

5 - 10 yrs

Best in industry

Python

Scala

Apache Spark

Apache Kafka

databricks

+1 more

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
Build and optimize distributed data applications using Apache Spark and Python Scala.
Develop reliable, high-performance data pipelines for batch and streaming workloads.
Design and manage data workflows using Apache Airflow.
Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
Work with large datasets to ensure data quality, consistency, reliability, and performance.
Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
Optimize data workflows for scalability, reliability, performance, and cost efficiency.
Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
Strong hands-on experience with Apache Spark and Scala.
Experience designing, building, and maintaining large-scale ETL pipelines.
Strong hands-on experience with AWS, particularly Amazon S3.
Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
Strong SQL skills and a solid understanding of distributed data processing concepts.
Experience working with batch and/or streaming data pipelines.
Excellent debugging, problem-solving, and performance optimization skills.
Strong communication and collaboration skills.

Good to Have

Experience with Databricks and the broader Databricks data platform.
Familiarity with streaming technologies such as Apache Kafka.
Experience working on large-scale data platforms handling high-volume data workloads.
Exposure to additional AWS data services and cloud-native data architectures.

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

Key Responsibilities

Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
Build and optimize distributed data applications using Apache Spark and Python Scala.
Develop reliable, high-performance data pipelines for batch and streaming workloads.
Design and manage data workflows using Apache Airflow.
Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
Work with large datasets to ensure data quality, consistency, reliability, and performance.
Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
Optimize data workflows for scalability, reliability, performance, and cost efficiency.
Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
Strong hands-on experience with Apache Spark and Scala.
Experience designing, building, and maintaining large-scale ETL pipelines.
Strong hands-on experience with AWS, particularly Amazon S3.
Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
Strong SQL skills and a solid understanding of distributed data processing concepts.
Experience working with batch and/or streaming data pipelines.
Excellent debugging, problem-solving, and performance optimization skills.
Strong communication and collaboration skills.

Good to Have

Experience with Databricks and the broader Databricks data platform.
Familiarity with streaming technologies such as Apache Kafka.
Experience working on large-scale data platforms handling high-volume data workloads.
Exposure to additional AWS data services and cloud-native data architectures.

Data Engineer Databricks/Snowflake

Posted by Robin Silverster

Bengaluru (Bangalore)

7 - 10 yrs

₹15L - ₹40L / yr

Data engineering

Python

PySpark

Data Transformation Tool (DBT)

Apache Airflow

+2 more

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

Hands-on exposure to Microsoft Fabric for data integration and analytics.
Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
Domain expertise in financial services.

About the Role

Key Responsibilities

Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

Hands-on exposure to Microsoft Fabric for data integration and analytics.
Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
Domain expertise in financial services.

Lead/Senior Data Engineer

Posted by Mamta K

Chennai

8 - 15 yrs

₹18L - ₹24L / yr

Data engineering

Python

Scala

Apache Hive

Delta Lake

+3 more

Job Description:

Position: Lead / Senior Data Engineer

Location: Chennai

Shift: US Eastern Time ( 5:00 PM – 2:00 AM )

Experience : 8+ years

Notice Period: Immediate Joiner only

Roles and Responsibilities:

Role Overview

The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities:

Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

Define technical strategy, architecture, and best practices for development and data engineering.

Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

Construct infrastructure for efficient ETL processes from various sources and storage systems.

Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

Lead the implementation of algorithms and prototypes to transform raw data into useful information.

Review code for quality, scalability, and performance.

Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

Develop analytical tools, programs, and reporting mechanisms.

Create data validation methods and data analysis tools.

Interpret data trends and patterns to establish operational alerts.

Conduct complex data analysis and present results effectively.

Prepare data for prescriptive and predictive modeling.

Ensure compliance with data governance and security policies.

Troubleshoot, debug, and resolve complex technical issues.

Drive continuous improvement in software and data development processes, tools, and methodologies.

Mentor and guide engineers through code reviews, technical discussions, and training.

Ensure timely delivery of projects while maintaining high engineering standards.

Continuously explore opportunities to enhance data quality and reliability.

Apply strong programming and problem-solving skills to develop scalable solutions.

Demonstrate passion for testing strategy, problem-solving, and continuous learning.

Willingness to acquire new skills and knowledge.

Possess a product/engineering mindset to drive impactful data solutions.

Experience working in distributed environments with global teams.

Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

High proficiency in Python, Scala, and Spark for applied large-scale data processing.

Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

Solid understanding of batch and streaming data processing techniques.

Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

Familiarity with Agile methodologies.

Obsession for service observability, instrumentation, monitoring, and alerting.

Knowledge or experience in architectural best practices for building data lakes.

Qualifications - Bachelor’s degree in computer science, Engineering, Information Systems, or related field.

Job Description:

Position: Lead / Senior Data Engineer

Location: Chennai

Shift: US Eastern Time ( 5:00 PM – 2:00 AM )

Experience : 8+ years

Notice Period: Immediate Joiner only

Roles and Responsibilities:

Role Overview

Key Responsibilities:

Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

Define technical strategy, architecture, and best practices for development and data engineering.

Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

Construct infrastructure for efficient ETL processes from various sources and storage systems.

Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

Lead the implementation of algorithms and prototypes to transform raw data into useful information.

Review code for quality, scalability, and performance.

Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

Develop analytical tools, programs, and reporting mechanisms.

Create data validation methods and data analysis tools.

Interpret data trends and patterns to establish operational alerts.

Conduct complex data analysis and present results effectively.

Prepare data for prescriptive and predictive modeling.

Ensure compliance with data governance and security policies.

Troubleshoot, debug, and resolve complex technical issues.

Drive continuous improvement in software and data development processes, tools, and methodologies.

Mentor and guide engineers through code reviews, technical discussions, and training.

Ensure timely delivery of projects while maintaining high engineering standards.

Continuously explore opportunities to enhance data quality and reliability.

Apply strong programming and problem-solving skills to develop scalable solutions.

Demonstrate passion for testing strategy, problem-solving, and continuous learning.

Willingness to acquire new skills and knowledge.

Possess a product/engineering mindset to drive impactful data solutions.

Experience working in distributed environments with global teams.

Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

High proficiency in Python, Scala, and Spark for applied large-scale data processing.

Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

Solid understanding of batch and streaming data processing techniques.

Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

Familiarity with Agile methodologies.

Obsession for service observability, instrumentation, monitoring, and alerting.

Knowledge or experience in architectural best practices for building data lakes.

Qualifications - Bachelor’s degree in computer science, Engineering, Information Systems, or related field.

Senior Data Engineer – Databricks

Posted by Akshay Patil

Bengaluru (Bangalore)

14 - 25 yrs

₹50L - ₹70L / yr

Data engineering

databricks

Apache Spark

PySpark

Python

+19 more

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.

Must-Have Skills :

14 to 20 years of Data Engineering experience
Databricks & Apache Spark / PySpark
Python & SQL
AWS Cloud
Lakehouse Architecture
ETL / ELT & Distributed Data Processing
Batch & Streaming Pipelines
Data Pipeline Optimization & Data Modeling
CDC & Incremental Processing
Git, CI/CD & Testing
Data Quality, Monitoring & Observability
Technical Leadership & Stakeholder Management

Key Responsibilities :

Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
Own data products from design through production.
Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
Optimize pipelines for performance, scalability, reliability, and cost.
Design scalable data architectures and data models.
Implement data quality, monitoring, lineage, and CI/CD practices.
Lead technical discussions and mentor engineering teams.
Collaborate with business stakeholders, architects, product owners, and engineering teams.
Remain hands-on while providing technical leadership.

Ideal Candidate :

A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.

🔴 Super Urgent : Only Bangalore-based immediate joiners.

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

Must-Have Skills :

14 to 20 years of Data Engineering experience
Databricks & Apache Spark / PySpark
Python & SQL
AWS Cloud
Lakehouse Architecture
ETL / ELT & Distributed Data Processing
Batch & Streaming Pipelines
Data Pipeline Optimization & Data Modeling
CDC & Incremental Processing
Git, CI/CD & Testing
Data Quality, Monitoring & Observability
Technical Leadership & Stakeholder Management

Key Responsibilities :

Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
Own data products from design through production.
Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
Optimize pipelines for performance, scalability, reliability, and cost.
Design scalable data architectures and data models.
Implement data quality, monitoring, lineage, and CI/CD practices.
Lead technical discussions and mentor engineering teams.
Collaborate with business stakeholders, architects, product owners, and engineering teams.
Remain hands-on while providing technical leadership.

Ideal Candidate :

🔴 Super Urgent : Only Bangalore-based immediate joiners.

Java + Spark + ETL

Posted by Chaitanya Rajadnya

Pune

5 - 12 yrs

Best in industry

Java

Apache Spark

ETL

Amazon Web Services (AWS)

Key Responsibilities

Build and maintain data transformation pipelines using java Spark

Develop and optimize large-scale/CPU intensive data processing using Apache Spark

Orchestrate workflows using Airflow

Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage

Support schema evolution, backfills, and incremental processing

Ensure pipelines meet SLAs for freshness, reliability, and performance

Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with Apache Spark

Experience with HBase/SQL or similar lakehouse query engines

Airflow

Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)

Proficiency in Java

Experience with Git-based development and CI/CD

Key Responsibilities

Build and maintain data transformation pipelines using java Spark

Develop and optimize large-scale/CPU intensive data processing using Apache Spark

Orchestrate workflows using Airflow

Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage

Support schema evolution, backfills, and incremental processing

Ensure pipelines meet SLAs for freshness, reliability, and performance

Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with Apache Spark

Experience with HBase/SQL or similar lakehouse query engines

Airflow

Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)

Proficiency in Java

Experience with Git-based development and CI/CD

Senior Data Engineer - AdTech (6-Month Contract)

Posted by Tushar Vaghela

Remote only

5 - 10 yrs

Best in industry

ETL

Google Cloud Platform (GCP)

Kubernetes

Scala

Apache Spark

Description

We are looking for Senior Data Engineers to join our AdTech team and build scalable, high-performance data platforms that power advertising insights and analytics. The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Spark and Scala.

You will work on designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
Build and optimize distributed data applications using Spark and Scala.
Develop reliable, high-performance data pipelines for batch and streaming workloads.
Work with large datasets to ensure data quality, consistency, and performance.
Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
Optimize data workflows for scalability, reliability, and cost efficiency.
Deploy and manage data workloads in cloud and containerized environments.
Troubleshoot production issues and continuously improve platform performance.

Requirements

Candidates who demonstrate:

5+ years of experience in Data Engineering or Big Data Engineering.
Strong hands-on experience with Apache Spark and Scala.
Experience building and maintaining ETL pipelines.
Familiarity with Google Cloud Storage (GCS).
Experience with Kubernetes (K8s).
Strong SQL skills and understanding of distributed data processing.
Excellent debugging, problem-solving, and performance optimization skills.
Strong communication and collaboration skills.

Good to Have

Experience with AWS and cloud-native data services.
Familiarity with streaming technologies such as Kafka.
Experience working on large-scale data platforms or AdTech systems.
Exposure to orchestration tools such as Airflow.

Benefits

Best-in-class salary: We hire strong talent and compensate accordingly.
Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
High-impact work: Build AI-first systems and products used at scale by global clients.

About Us

Proximity is the trusted technology, design, and consulting partner for some of the biggest Sports, Media, and Entertainment companies in the world. We’re headquartered in San Francisco and have offices in Palo Alto, Dubai, Mumbai, and Bangalore.

Since 2019, Proximity has built high-impact, scalable products used by millions of users every day. Today, we are a global team of engineers, designers, product managers, and experts solving complex problems and building cutting-edge technology at scale.

Description

Key Responsibilities

Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
Build and optimize distributed data applications using Spark and Scala.
Develop reliable, high-performance data pipelines for batch and streaming workloads.
Work with large datasets to ensure data quality, consistency, and performance.
Collaborate with engineering, product, and analytics teams to deliver robust data solutions.
Optimize data workflows for scalability, reliability, and cost efficiency.
Deploy and manage data workloads in cloud and containerized environments.
Troubleshoot production issues and continuously improve platform performance.

Requirements

Candidates who demonstrate:

5+ years of experience in Data Engineering or Big Data Engineering.
Strong hands-on experience with Apache Spark and Scala.
Experience building and maintaining ETL pipelines.
Familiarity with Google Cloud Storage (GCS).
Experience with Kubernetes (K8s).
Strong SQL skills and understanding of distributed data processing.
Excellent debugging, problem-solving, and performance optimization skills.
Strong communication and collaboration skills.

Good to Have

Experience with AWS and cloud-native data services.
Familiarity with streaming technologies such as Kafka.
Experience working on large-scale data platforms or AdTech systems.
Exposure to orchestration tools such as Airflow.

Benefits

Best-in-class salary: We hire strong talent and compensate accordingly.
Proximity Talks: Meet and learn from designers, engineers, product leaders, and AI practitioners.
Continuous learning: Work with a world-class team and stay close to the latest in AI, engineering, and product development.
High-impact work: Build AI-first systems and products used at scale by global clients.

About Us

Data Engineer_Java+Spark

Posted by Vishakha Walunj

Pune

3 - 7 yrs

Best in industry

Java

Apache Spark

Apache Airflow

SQL

Amazon Web Services (AWS)

Key Responsibilities

Build and maintain data transformation pipelines using java Spark

Develop and optimize large-scale/CPU intensive data processing using Apache Spark

Orchestrate workflows using Airflow

Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage

Support schema evolution, backfills, and incremental processing

Ensure pipelines meet SLAs for freshness, reliability, and performance

Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with

HBase

Apache Spark

Experience with HBase or similar lakehouse query engines

Airflow

Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)

Proficiency in Java

Experience with Git-based development and CI/CD

Nice-to-Have Skills

OpenTable format/Iceberg ,Apache Arrow

CDC-based analytics pipelines

Cloud platforms (AWS)

Kubernetes-based data platforms

Key Responsibilities

Build and maintain data transformation pipelines using java Spark

Develop and optimize large-scale/CPU intensive data processing using Apache Spark

Orchestrate workflows using Airflow

Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage

Support schema evolution, backfills, and incremental processing

Ensure pipelines meet SLAs for freshness, reliability, and performance

Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with

HBase

Apache Spark

Experience with HBase or similar lakehouse query engines

Airflow

Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)

Proficiency in Java

Experience with Git-based development and CI/CD

Nice-to-Have Skills

OpenTable format/Iceberg ,Apache Arrow

CDC-based analytics pipelines

Cloud platforms (AWS)

Kubernetes-based data platforms

Java Spark Data Engineer

Posted by Meghana Shinde

Pune

2.5 - 5 yrs

Best in industry

Java

Spark

Hadoop

Apache HBase

SQL

+1 more

Company Name – Wissen Technology

Group of companies in India – Wissen Technology & Wissen Infotech

Work Location – Whitefield, Bangalore

Website and Company profile:

www.wissen.com

LinkedIn Page:

https://www.linkedin.com/company/wissen-technology/

While you may already know about Wissen and the company history, here is a quick rundown for you.

About Wissen Technology:

· The Wissen Group was founded in the year 2000. Wissen Technology, a part of Wissen Group, was established in the year 2015.

· Wissen Technology is a specialized technology company that delivers high-end consulting for organizations in the Banking & Finance, Telecom, and Healthcare domains. We help clients build world class products.

· Our workforce has highly skilled professionals, with leadership and senior management executives who have graduated from Ivy League Universities like Wharton, MIT, IITs, IIMs, and NITs and with rich work experience in some of the biggest companies in the world.

· Wissen Technology has grown its revenues by 400% in these five years without any external funding or investments.

· Globally present with offices US, India, UK, Australia, Mexico, and Canada.

· We offer an array of services including Application Development, Artificial Intelligence & Machine Learning, Big Data & Analytics, Visualization & Business Intelligence, Robotic Process Automation, Cloud, Mobility, Agile & DevOps, Quality Assurance & Test Automation.

· Wissen Technology has been certified as a Great Place to Work®.

· Wissen Technology has been voted as the Top 20 AI/ML vendor by CIO Insider in 2020.

· Over the years, Wissen Group has successfully delivered $650 million worth of projects for more than 20 of the Fortune 500 companies.

· We have served client across sectors like Banking, Telecom, Healthcare, Manufacturing, and Energy. They include likes of Morgan Stanley, Goldman Sachs, MSCI, StateStreet, Flipkart, Swiggy, Trafigura, GE to name a few.

About Role :

Key Responsibilities

Build and maintain data transformation pipelines using java Spark
Develop and optimize large-scale/CPU intensive data processing using Apache Spark
Orchestrate workflows using Airflow
Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
Support schema evolution, backfills, and incremental processing
Ensure pipelines meet SLAs for freshness, reliability, and performance
Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with
HBase
Apache Spark
Experience with HBase or similar lakehouse query engines
Airflow
Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
Proficiency in Java
Experience with Git-based development and CI/CD

Nice-to-Have Skills

OpenTable format/Iceberg ,Apache Arrow
CDC-based analytics pipelines
Cloud platforms (AWS)
Kubernetes-based data platforms

Company Name – Wissen Technology

Group of companies in India – Wissen Technology & Wissen Infotech

Work Location – Whitefield, Bangalore

Website and Company profile:

www.wissen.com

LinkedIn Page:

https://www.linkedin.com/company/wissen-technology/

While you may already know about Wissen and the company history, here is a quick rundown for you.

About Wissen Technology:

· The Wissen Group was founded in the year 2000. Wissen Technology, a part of Wissen Group, was established in the year 2015.

· Wissen Technology has grown its revenues by 400% in these five years without any external funding or investments.

· Globally present with offices US, India, UK, Australia, Mexico, and Canada.

· Wissen Technology has been certified as a Great Place to Work®.

· Wissen Technology has been voted as the Top 20 AI/ML vendor by CIO Insider in 2020.

· Over the years, Wissen Group has successfully delivered $650 million worth of projects for more than 20 of the Fortune 500 companies.

About Role :

Key Responsibilities

Build and maintain data transformation pipelines using java Spark
Develop and optimize large-scale/CPU intensive data processing using Apache Spark
Orchestrate workflows using Airflow
Implement data quality checks, testing, and monitoring for pipeline. Good to have exposer into managing metadata, cataloguing, and lineage
Support schema evolution, backfills, and incremental processing
Ensure pipelines meet SLAs for freshness, reliability, and performance
Expertise/working knowledge in Spark and HBase(semantic layer, virtual datasets, Reflections)

Required Skills & Qualifications

Strong hands-on experience with
HBase
Apache Spark
Experience with HBase or similar lakehouse query engines
Airflow
Understanding of data catalogs and lineage (e.g., OpenLineage, DataHub, Apache Polaris , openlineage)
Proficiency in Java
Experience with Git-based development and CI/CD

Nice-to-Have Skills

OpenTable format/Iceberg ,Apache Arrow
CDC-based analytics pipelines
Cloud platforms (AWS)
Kubernetes-based data platforms

Azure Databricks Developers (Pyspark, Databricks)

Posted by tabbasum shaikh

Bengaluru (Bangalore), Mumbai, Pune, Hyderabad, Noida, Kolkata

8 - 15 yrs

₹13L - ₹20L / yr

Azure Data Factory

Azure Databricks

PySpark

Python

SQL

+4 more

Roles & Responsibilities

Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

2Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

4Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

5Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

6Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

8Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Roles & Responsibilities

Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Azure Databricks Developers (Pyspark, Databricks)

Posted by Mayank Choudhary

Bengaluru (Bangalore), Mumbai, Pune, Noida, Hyderabad, Kolkata, Gurugram, Chennai

8 - 15 yrs

₹17L - ₹22L / yr

Azure

Python

Strong Azure Databricks Engineer / Senior Data Engineer Profile

Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Strong Azure Databricks Engineer / Senior Data Engineer Profile

Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Why apply to jobs via Cutshort