Senior Software Engineer - Data

Data Engineer Short Hiring Post

🚨 Hiring: Data Engineer

🔹 Experience: 5–9 Years

🔹 Location: Bangalore / Hyderabad

🔹 Skills: PySpark, Python, SQL, ETL, CI/CD, Data Modeling

🔹 Process: L1 Virtual → L2 F2F Karat Test

🔹 F2F: Bangalore / Hyderabad Location

🔹 Positions: Immediate requirement

⚠️ Note: Candidates must be available for F2F Karat immediately after L1.

#Hiring #DataEngineer #PySpark #Python #SQL #BangaloreJobs #HyderabadJobs #Mphasis #ImmediateJoiners

Job Summary

We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure. The ideal candidate should have strong expertise in SQL, Python, Linux, and modern data engineering practices to support data integration, transformation, and analytics.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines.
• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
• Develop automation scripts using Python for data processing and workflow optimization.
• Work with Linux environments for deployment, monitoring, and troubleshooting.
• Ensure data quality, integrity, and reliability across data platforms.
• Collaborate with data analysts, software engineers, and business stakeholders to deliver data solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance and scalability.
• Implement best practices for data security, governance, and documentation.

Required Skills

• Strong experience in Data Engineering concepts and ETL/ELT processes.
• Proficiency in SQL, including query optimization and database design.
• Strong programming skills in Python.
• Hands-on experience with Linux commands, shell scripting, and system administration basics.
• Experience with relational databases such as PostgreSQL, MySQL, SQL Server, or Oracle.
• Familiarity with Git/version control.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with cloud platforms (AWS, Azure, or GCP).
• Knowledge of Apache Spark, Airflow, Kafka, or similar data engineering tools.
• Experience with data warehousing solutions and big data technologies.
• Understanding of CI/CD pipelines and containerization (Docker/Kubernetes).

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Relevant certifications in cloud or data engineering are an added advantage.

Data Engineer Hiring Post

🚨 Hiring: Data Engineer | PySpark + Python + SQL

We are looking for experienced Data Engineers to join our team!

🔹 Experience: 5 to 9 Years

🔹 Locations: Bangalore / Hyderabad

🔹 Interview Process:

• 1st Round – Virtual

• 2nd Round – Face-to-Face (Karat Test)

🔑 Key Skills:

✅ PySpark

✅ SQL

✅ Python

✅ ETL

📩 Interested candidates can share their updated resume.

#Hiring #DataEngineer #PySpark #Python #SQL #ETL #BangaloreJobs #HyderabadJobs #TechHiring #ImmediateHiring

Skills Referential (Required knowledge, skills and abilities)

Technical Skills:

Python

Pyspark

SQL

ETL Aws, Azure, gcp

Job Summary

We are seeking a motivated Data Engineer with strong skills in SQL, Python, and Linux to design, build, and maintain scalable data pipelines and support data-driven decision-making. The ideal candidate should have experience working with large datasets, ETL processes, and relational databases while ensuring data quality and performance.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines.
• Write optimized SQL queries, stored procedures, and database objects.
• Develop Python scripts for data extraction, transformation, and automation.
• Work in Linux environments to manage scripts, cron jobs, and system processes.
• Monitor and troubleshoot data pipeline failures.
• Ensure data integrity, consistency, and quality across systems.
• Collaborate with data analysts, software engineers, and business stakeholders.
• Optimize database performance and query execution.
• Participate in code reviews and follow best engineering practices.

Required Skills

• Strong proficiency in SQL (joins, subqueries, window functions, CTEs, indexing, query optimization).
• Good programming experience in Python.
• Hands-on experience with Linux commands and shell scripting.
• Understanding of ETL/ELT concepts and data warehousing.
• Knowledge of relational databases such as PostgreSQL, MySQL, Oracle, or SQL Server.
• Familiarity with Git for version control.
• Strong problem-solving and analytical skills.

We are looking for a Data Engineer with at least 1 year of hands-on experience building solutions on Snowflake. The candidate should be comfortable designing, building, and managing reliable data pipelines that move data from multiple sources into a central data platform.

Responsibilities

• Build and maintain data pipelines for ingesting, transforming, and loading data into Snowflake
• Design scalable data models, schemas, tables, and views in Snowflake
• Develop ETL/ELT workflows using SQL, Python, or data orchestration tools
• Integrate data from APIs, databases, files, and third-party platforms
• Monitor pipeline performance, failures, data quality, and freshness
• Optimize Snowflake queries, warehouses, storage, and compute usage
• Implement incremental loads, change data capture, and scheduled workflows
• Work with engineering and business teams to understand data requirements
• Maintain documentation for pipelines, datasets, and data transformations

Requirements

• 1+ year of hands-on experience working with Snowflake
• Strong SQL skills and experience writing complex queries
• Experience building and managing ETL or ELT data pipelines
• Knowledge of data warehousing concepts, dimensional modelling, and data quality
• Experience with Python or another scripting language
• Familiarity with orchestration tools such as Airflow, Dagster, Prefect, dbt, or similar
• Understanding of APIs, relational databases, file formats, and cloud storage
• Ability to troubleshoot pipeline failures and performance issues
• Strong analytical, problem-solving, and communication skills

Good to Have

• Experience with dbt and Snowflake Tasks, Streams, Snowpipe, or Dynamic Tables
• Knowledge of AWS, Azure, or Google Cloud
• Experience with Kafka or other streaming platforms
• Familiarity with CI/CD, Git, monitoring, and data governance practices
• Experience integrating ERP, finance, or operational systems

Description

We are looking for Senior Data Engineers to join our Data Platform team and build scalable, high-performance data platforms that power data processing, analytics, and downstream applications.

The ideal candidate will have strong experience in distributed data processing, ETL pipelines, and Big Data technologies, with hands-on expertise in Apache Spark and Python Scala.

You will be responsible for designing, developing, and optimizing large-scale data pipelines while collaborating closely with cross-functional engineering teams to build reliable, production-grade data solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL and data processing pipelines for large-scale datasets.
• Build and optimize distributed data applications using Apache Spark and Python Scala.
• Develop reliable, high-performance data pipelines for batch and streaming workloads.
• Design and manage data workflows using Apache Airflow.
• Build and operate data workloads on AWS, with strong usage of Amazon S3 for large-scale data storage.
• Work with large datasets to ensure data quality, consistency, reliability, and performance.
• Collaborate with engineering, product, analytics, and other platform teams to deliver robust data solutions.
• Optimize data workflows for scalability, reliability, performance, and cost efficiency.
• Troubleshoot production issues, identify bottlenecks, and continuously improve platform performance.

Requirements

Candidates who demonstrate:

• 5+ years of experience in Data Engineering, Big Data Engineering, or a similar role.
• Strong hands-on experience with Apache Spark and Scala.
• Experience designing, building, and maintaining large-scale ETL pipelines.
• Strong hands-on experience with AWS, particularly Amazon S3.
• Hands-on experience with Apache Airflow for workflow orchestration and scheduling.
• Strong SQL skills and a solid understanding of distributed data processing concepts.
• Experience working with batch and/or streaming data pipelines.
• Excellent debugging, problem-solving, and performance optimization skills.
• Strong communication and collaboration skills.

Good to Have

• Experience with Databricks and the broader Databricks data platform.
• Familiarity with streaming technologies such as Apache Kafka.
• Experience working on large-scale data platforms handling high-volume data workloads.
• Exposure to additional AWS data services and cloud-native data architectures.

About Us

We believe the future of software development is AI-native — where engineers operate at a higher level of abstraction and quality remains non-negotiable. 

Incubyte is a software craft consultancy where the “how” of building software matters as much as the “what”.  

We partner with companies of all sizes, from helping enterprises build, scale, and modernize to early-stage founders bring their ideas to life. 

Our engineers operate in an AI-native development model, using AI as a collaborator across the SDLC to accelerate development while upholding the discipline of software craftsmanship. Guided by Software Craftsmanship and Extreme Programming practices, we build reliable, maintainable, and scalable systems with speed, without compromising quality. If this way of building software resonates with you, we’d like to talk. 

Our Guiding Principles 

These principles define how we work at Incubyte. They are non-negotiable. 

Relentless Pursuit of Quality with Pragmatism 

  We build high-quality systems without losing sight of delivery. 

Extreme Ownership 

  We take responsibility end-to-end for decisions, execution, and outcomes. 

Proactive Collaboration 

  We collaborate closely, challenge each other, and solve problems together. 

Active Pursuit of Mastery 

  We continuously improve our craft and raise our bar. 

Invite, Give, and Act on Feedback 

We seek, give, and act on feedback to get better every day. 

Ensuring Client Success 

We act as trusted partners and focus on real outcomes, not just output. 

Job Description

This is a remote position.

Experience Level

2+ years of experience in SQL, Python, and Snowflake (or equivalent cloud data warehouse), Azure Cloud services.

Role Overview

If you're a Data Craftsperson who takes pride in clean, well-tested data solutions and believes in the principles of Extreme Programming, we'd love to meet you. At Incubyte, we're a DevOps organization where developers own the entire release cycle — you'll get hands-on experience across data engineering, analytics, cloud infrastructure, and direct client communication. This role sits primarily in data engineering (80%) with a meaningful analytics component (20%), supporting our client's data systems end-to-end.

What You'll Do

• Design, build, and maintain data pipelines and infrastructure using SQL and Python
• Work within Snowflake to build and optimize data models supporting business use cases
• Parse and process structured and semi-structured data (JSON, XML) from varied sources
• Diagnose issues across raw, intermediate, and summary tables
• Build SQL queries to support repeatable analytics use cases based on stakeholder requirements
• Investigate and resolve data quality issues, including time-sensitive or urgent ones
• Identify opportunities to consolidate models and maintain a single source of truth (SSOT)

Requirements

What We're Looking For

• 2+ years of experience with SQL and relational databases, with the ability to understand complex data relationships and transformations (required)
• 2+ years of experience with Python for data engineering tasks (required)
• Experience with Snowflake or an equivalent cloud data warehouse (required)
• Experience parsing JSON and XML data (a plus)
• A strong eye for data quality and attention to detail
• Knowledge of Git (required)
• Knowledge of Azure cloud services such as Azure Data Factory, Azure Blob Storage, and Azure SQL Database (required)
• Knowledge of data infrastructure/modeling tools like DBT, Fivetran (a plus)
• Experience with BI tools like Power BI(a plus, not core to this role)
• Knowledge of Docker, Linux, Shell/Bash, and virtualization technologies (a plus)
• Knowledge of SSIS packages (a plus)
• Familiarity with CI/CD methodologies

Benefits

Life at Incubyte  

We are a remote-first company with structured flexibility. Teams commit to shared rhythms during core hours, ensuring smooth collaboration while maintaining autonomy. Twice a year, we come together in person for a co-working sprint and once a year for a retreat - with all travel expenses covered. 

Our environment is built for crafters: pairing, refactoring, experimenting with AI, and pushing the boundaries of software excellence. We are all lifelong learners, and our work is our passion. 

Perks

• Dedicated learning & development budget. 
• Sponsorship for conference talks. 
• Comprehensive medical & term insurance.
• Employee-friendly leave policies. 
• Home Office fund 
• Medical Insurance

Job Description:

Position: Lead / Senior Data Engineer

Location: Chennai

Shift: US Eastern Time ( 5:00 PM – 2:00 AM )

Experience : 8+ years

Notice Period: Immediate Joiner only

Roles and Responsibilities:

Role Overview

The Lead Data Engineer will be responsible for designing, developing, and delivering high-quality software and data solutions while leading a team of engineers. The role involves hands-on technical work, architectural decision-making, mentoring junior developers, and collaborating with cross-functional teams to ensure successful delivery of scalable data platforms and analytical solutions.

Key Responsibilities:

Lead the end-to-end design, development, and delivery of software systems, data pipelines, and components.

Define technical strategy, architecture, and best practices for development and data engineering.

Design and build optimized data pipelines using cutting-edge technologies in a cloud environment.

Construct infrastructure for efficient ETL processes from various sources and storage systems.

Architect, design, and maintain database pipeline architectures, ensuring readiness for AI/ML transformations.

Lead the implementation of algorithms and prototypes to transform raw data into useful information.

Review code for quality, scalability, and performance.

Collaborate with Product Managers, Business Managers, Designers, and QA teams to translate business requirements into technical solutions.

Develop analytical tools, programs, and reporting mechanisms.

Create data validation methods and data analysis tools.

Interpret data trends and patterns to establish operational alerts.

Conduct complex data analysis and present results effectively.

Prepare data for prescriptive and predictive modeling.

Ensure compliance with data governance and security policies.

Troubleshoot, debug, and resolve complex technical issues.

Drive continuous improvement in software and data development processes, tools, and methodologies.

Mentor and guide engineers through code reviews, technical discussions, and training.

Ensure timely delivery of projects while maintaining high engineering standards.

Continuously explore opportunities to enhance data quality and reliability.

Apply strong programming and problem-solving skills to develop scalable solutions.

Demonstrate passion for testing strategy, problem-solving, and continuous learning.

Willingness to acquire new skills and knowledge.

Possess a product/engineering mindset to drive impactful data solutions.

Experience working in distributed environments with global teams.

Stay current with emerging technologies and industry trends to propose innovative solutions.

Technical Skills and Experience Requirements

Minimum 8+ years of hands-on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.

High proficiency in Python, Scala, and Spark for applied large-scale data processing.

Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.

Solid understanding of batch and streaming data processing techniques.

Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.

Expert-level ability to write complex, optimized SQL queries across extensive data volumes.

Experience with RDBMS and OLAP databases such as MySQL and Snowflake.

Familiarity with Agile methodologies.

Obsession for service observability, instrumentation, monitoring, and alerting.

Knowledge or experience in architectural best practices for building data lakes.

Qualifications - Bachelor’s degree in computer science, Engineering, Information Systems, or related field.

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.