Data Engineer

Job Summary

Role Overview

We are looking for an experienced Data Engineer with strong expertise in Python, ETL, Advanced SQL, CI/CD, DevOps, and Data Analytics. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, transforming large datasets, and supporting data-driven applications.

Experience with Google Cloud Platform (GCP) will be an added advantage.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL.
• Develop complex and optimized SQL queries, stored procedures, and data transformations.
• Build and maintain reliable data integration workflows across multiple data sources.
• Perform data cleansing, validation, transformation, and quality checks.
• Analyze data and provide insights to support business and technical requirements.
• Implement and maintain CI/CD pipelines for data engineering applications.
• Work with DevOps practices and tools to automate deployments, monitoring, and infrastructure processes.
• Troubleshoot data pipeline failures, performance issues, and production incidents.
• Optimize data processing workflows for performance, scalability, and reliability.
• Collaborate with Data Analysts, Data Scientists, Developers, and other stakeholders.
• Follow best practices for version control, testing, documentation, and deployment.
• Contribute to cloud-based data engineering initiatives, preferably on GCP.

Required Skills

• 5–7 years of hands-on experience in Data Engineering.
• Strong programming skills in Python.
• Strong expertise in Advanced SQL and database concepts.
• Hands-on experience with ETL/ELT processes and data pipelines.
• Good understanding of Data Warehousing and Data Modeling concepts.
• Experience with CI/CD practices and tools.
• Strong understanding of DevOps principles, automation, and deployment processes.
• Strong data analytics and problem-solving skills.
• Experience working with large datasets and performance optimization.
• Good understanding of Git/version control and software development best practices.

Good to Have

• Hands-on experience with Google Cloud Platform (GCP).
• Exposure to GCP data services such as BigQuery, Cloud Storage, Dataflow, Composer, or Pub/Sub.
• Experience with containerization/orchestration technologies such as Docker/Kubernetes.
• Experience with workflow orchestration tools such as Airflow.
• Knowledge of cloud-based data architecture and distributed data processing.

Preferred Candidate Profile

• Strong analytical and problem-solving abilities.
• Good communication and stakeholder management skills.
• Ability to work independently as well as in a collaborative team environment.
• Strong ownership of data pipelines and production systems.
• Candidates who can join at short notice are preferred.

Mandatory Skills

 Data Engineer, Python , ETL, GCP, Advanced SQL, Strong Data Analytics skills, CICD, Devops

Must-Have Skills

• Minimum 3 years of experience in Data Engineering / Analytics Engineering / Fintech Data roles
• Must have worked on SMS Parsing, intelligent platform, converting RAW customer SMS data into structured actionable financial signals and enabling downstream usage of SMS derived variables
• Must have established a continuous learning cycle to expand parser coverage
• Experience in Lending / NBFC / Fintech domain
• Experience working with Bureau, SMS, Device, or Banking data
• Strong Python and SQL (production level)
• Experience handling unstructured data (SMS, logs, JSON, APIs)
• Experience building data pipelines, schedulers, and cron jobs
• Strong database design and data modelling skills
• Ability to work in a startup environment with high ownership
• Familiarity with modern platforms like AWS, Snowflake, Google BigQuery, Redshift

Good to Have

• Experience in STPL, especially less than 25K ticket size
• Experience with streaming (Kafka/Kinesis) and orchestration (Airflow or Step Functions)
• Experience with feature stores and risk analytics datasets
• Knowledge of regex, NLP basics for SMS parsing
• Experience supporting real-time decision engines/underwriting systems

Role Summary

This role will be responsible for owning the end-to-end data-structuring layer across the organisation. The individual will transform large volumes of raw, unstructured, and semi-structured data (such as SMS, device, bureau, and app data) into clean, standardised, and analysis-ready datasets. These structured datasets will directly power risk analytics, fraud detection, marketing insights, collections strategy, and policy decisioning.

Key Objective of the Role

Ensure all raw lending data (SMS, Bureau, Device, AA, App logs) is captured, parsed, structured, and stored in a clean analytics-ready format inside databases (PostgreSQL, DynamoDB, AWS stack) so that the Risk and Data Science team can directly use it for feature creation, policy building, and portfolio monitoring.

Core Responsibilities

1. End-to-End Data Ownership

• Design, build, and maintain end-to-end data pipelines (batch + streaming) using AWS native services (Glue, Lambda, Step Functions, Kinesis, S3, Athena, Redshift, EMR/Spark, etc.): ingestion

→ parsing → structuring → storage

• Work closely with Tech, Product, and Data Science to define what data should be captured
• Maintain data documentation, data dictionaries, and schema governance
• Ensure data quality, consistency, and version control

1. Unstructured Data Processing (Highest Priority)

• Parse raw SMS dumps and categorise into salary, EMI, loan apps, collections, credits, debits, OTP, etc.
• Process device fingerprint, behavioural logs, and vendor data (FinBox, AA, Bureau APIs)
• 
  
• 
  
• Convert JSON, logs, and raw API responses into structured feature tables
• Build regex/keyword-based parsers for financial SMS classification

1. Feature Implementation (From Risk & Data Science Team)

• Implement feature creation logic provided by Risk/Data Science team
• Translate business and policy logic into SQL/Python pipelines
• Create reusable feature layers for underwriting, fraud, collections, and monitoring
• Maintain a feature store for consistent model and policy usage

1. Lending Data Understanding (Domain-Specific Requirement)

• Work with Bureau data
• Structure SMS-derived financial variables (income, stress, EMI signals)
• Work with Account Aggregator and bank transaction datasets
• Understand fintech alternate data used in underwriting and fraud detection

1. Data Pipelines & Automation

• Build and maintain ETL/ELT pipelines using Python & SQL
• Create cron jobs for automated data ingestion and feature refresh
• Automate vendor data pulls (Bureau, SMS SDK, AA, device data)
• Ensure low-latency pipelines for real-time underwriting use cases

1. Database Structuring & Storage Architecture

• Structure clean datasets in PostgreSQL (analytics layer)
• Manage raw data storage in DynamoDB / S3 data lake
• Design normalized and denormalised tables for risk analytics
• Optimise database performance for large-scale query workloads

1. Dashboards & Readable Data Layer

• Create analytics-ready datasets, implement & write Metabase queries and convert into dashboards (Metabase / Power BI)
• Enable self-serve data access for Risk, Business, and Founders
• Support ad-hoc analysis requirements from leadership

1. Cross-Functional Collaboration (Very Important)

• The role requires close collaboration with data science, tech, product, and business teams to ensure reliable data pipelines, well-defined schemas, API integrations, logging architecture and high data quality, enabling faster and more accurate decision-making across lending workflows.

Tech Stack (Current Environment)

• AWS Services
• PostgreSQL (Primary analytics DB)
• DynamoDB (Raw/NoSQL storage)
• Python (Pandas, NumPy, ETL frameworks)
• Advanced SQL
• APIs, JSON, and Log Data Handling

About Us

We believe the future of software development is AI-native — where engineers operate at a higher level of abstraction and quality remains non-negotiable. 

Incubyte is a software craft consultancy where the “how” of building software matters as much as the “what”.  

We partner with companies of all sizes, from helping enterprises build, scale, and modernize to early-stage founders bring their ideas to life. 

Our engineers operate in an AI-native development model, using AI as a collaborator across the SDLC to accelerate development while upholding the discipline of software craftsmanship. Guided by Software Craftsmanship and Extreme Programming practices, we build reliable, maintainable, and scalable systems with speed, without compromising quality. If this way of building software resonates with you, we’d like to talk. 

Our Guiding Principles 

These principles define how we work at Incubyte. They are non-negotiable. 

Relentless Pursuit of Quality with Pragmatism 

  We build high-quality systems without losing sight of delivery. 

Extreme Ownership 

  We take responsibility end-to-end for decisions, execution, and outcomes. 

Proactive Collaboration 

  We collaborate closely, challenge each other, and solve problems together. 

Active Pursuit of Mastery 

  We continuously improve our craft and raise our bar. 

Invite, Give, and Act on Feedback 

We seek, give, and act on feedback to get better every day. 

Ensuring Client Success 

We act as trusted partners and focus on real outcomes, not just output. 

Job Description

This is a remote position.

Experience Level

2+ years of experience in SQL, Python, and Snowflake (or equivalent cloud data warehouse), Azure Cloud services.

Role Overview

If you're a Data Craftsperson who takes pride in clean, well-tested data solutions and believes in the principles of Extreme Programming, we'd love to meet you. At Incubyte, we're a DevOps organization where developers own the entire release cycle — you'll get hands-on experience across data engineering, analytics, cloud infrastructure, and direct client communication. This role sits primarily in data engineering (80%) with a meaningful analytics component (20%), supporting our client's data systems end-to-end.

What You'll Do

• Design, build, and maintain data pipelines and infrastructure using SQL and Python
• Work within Snowflake to build and optimize data models supporting business use cases
• Parse and process structured and semi-structured data (JSON, XML) from varied sources
• Diagnose issues across raw, intermediate, and summary tables
• Build SQL queries to support repeatable analytics use cases based on stakeholder requirements
• Investigate and resolve data quality issues, including time-sensitive or urgent ones
• Identify opportunities to consolidate models and maintain a single source of truth (SSOT)

Requirements

What We're Looking For

• 2+ years of experience with SQL and relational databases, with the ability to understand complex data relationships and transformations (required)
• 2+ years of experience with Python for data engineering tasks (required)
• Experience with Snowflake or an equivalent cloud data warehouse (required)
• Experience working with Snowflake Coco or any other AI tools(required) 
• Experience parsing JSON and XML data (a plus)
• A strong eye for data quality and attention to detail
• Knowledge of Git (required)
• Knowledge of Azure cloud services such as Azure Data Factory, Azure Blob Storage, and Azure SQL Database (required)
• Knowledge of data infrastructure/modeling tools like DBT, Fivetran (a plus)
• Experience with BI tools like Power BI(a plus, not core to this role)
• Knowledge of Docker, Linux, Shell/Bash, and virtualization technologies (a plus)
• Knowledge of SSIS packages (a plus)
• Familiarity with CI/CD methodologies

Benefits

Life at Incubyte  

We are a remote-first company with structured flexibility. Teams commit to shared rhythms during core hours, ensuring smooth collaboration while maintaining autonomy. Twice a year, we come together in person for a co-working sprint and once a year for a retreat - with all travel expenses covered. 

Our environment is built for crafters: pairing, refactoring, experimenting with AI, and pushing the boundaries of software excellence. We are all lifelong learners, and our work is our passion. 

Perks

• Dedicated learning & development budget. 
• Sponsorship for conference talks. 
• Comprehensive medical & term insurance.
• Employee-friendly leave policies. 
• Home Office fund 
• Medical Insurance

Job Title : Data Engineer – Databricks

Experience : 6+ Years

Location : Noida / Hyderabad / Chennai / Pune / Bengaluru (Hybrid)

Shift : IST (Normal Shift)

Job Summary :

We are seeking an experienced Data Engineer with strong expertise in Databricks, Snowflake, Python, and Spark to build and optimize scalable data pipelines and support AI/ML model deployments. The ideal candidate should have experience working with cloud-based data platforms and preferably possess exposure to the Healthcare domain.

Required Skills :

• Databricks (Preferred)
• Snowflake
• Python
• Apache Spark
• SQL
• Azure Cloud
• Kubernetes
• Apache Airflow
• GitHub & CI/CD Pipelines
• AI/ML Model Deployment
• Data Analytics

Preferred :

• Experience in the Healthcare domain.
• Strong understanding of scalable data engineering architectures and best practices.

Job Description

We are looking for an experienced Data Engineer with strong expertise in Python, ETL, SQL, CI/CD, and DevOps to design, develop, and maintain scalable data pipelines and data processing solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Develop data processing solutions using Python.
• Write complex and optimized SQL queries, stored procedures, and data transformations.
• Build and maintain data ingestion and integration workflows.
• Implement data quality, validation, monitoring, and error-handling processes.
• Develop and maintain CI/CD pipelines for data engineering applications.
• Work with DevOps tools and practices for automated build, deployment, and infrastructure management.
• Collaborate with data analysts, data scientists, software engineers, and business teams.
• Optimize data pipelines for performance, reliability, and scalability.
• Troubleshoot production data issues and ensure timely resolution.
• Follow best practices for version control, code quality, testing, and deployment.

Mandatory Skills

• Python
• ETL
• SQL
• CI/CD
• DevOps
• Git / Version Control
• Strong problem-solving and debugging skills

Job Summary

We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.

Technical Skills

• Strong hands-on experience in Python and PySpark development.
• Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
• Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Power BI Data Modeling and Semantic Layer development.
• Proficiency in DAX (Data Analysis Expressions).
• Experience designing and managing Semantic Models in Power BI.
• Strong SQL skills and experience working with large datasets.
• Knowledge of data warehousing concepts and best practices.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Exposure to modern data platforms like Databricks.
• Understanding of data governance and data quality frameworks.

About the Role

We are seeking motivated Data Engineering Interns to join our team remotely for a 3-month internship. This role is designed for students or recent graduates interested in working with data pipelines, ETL processes, and big data tools. You will gain practical experience in building scalable data solutions. While this is an unpaid internship, interns who successfully complete the program will receive a Completion Certificate and a Letter of Recommendation.

Responsibilities

• Assist in designing and building data pipelines for structured and unstructured data.
• Support ETL (Extract, Transform, Load) processes to prepare data for analytics.
• Work with databases (SQL/NoSQL) for data storage and retrieval.
• Help optimize data workflows for performance and scalability.
• Collaborate with data scientists and analysts to ensure data quality and consistency.
• Document workflows, schemas, and technical processes.

Requirements

• Strong interest in data engineering, databases, and big data systems.
• Basic knowledge of SQL and relational database concepts.
• Familiarity with Python, Java, or Scala for data processing.
• Understanding of ETL concepts and data pipelines.
• Exposure to cloud platforms (AWS, Azure, or GCP) is a plus.
• Familiarity with big data frameworks (Hadoop, Spark, Kafka) is an advantage.
• Good problem-solving skills and ability to work independently in a remote setup.

What You’ll Gain

• Hands-on experience in data engineering and ETL pipelines.
• Exposure to real-world data workflows.
• Mentorship and guidance from experienced engineers.
• Completion Certificate upon successful completion.
• Letter of Recommendation based on performance.

Internship Details

• Duration: 3 months
• Location: Remote (Work from Home)
• Stipend: Unpaid
• Perks: Completion Certificate + Letter of Recommendation

Role Overview

We are looking for a GCP Data Engineer with 10+ years of experience to design, develop, and optimize scalable cloud-based data solutions. The ideal candidate will have strong hands-on expertise in GCP, BigQuery, and advanced SQL, with experience building data pipelines and working with large-scale datasets.

Key Responsibilities

• Design and develop scalable data pipelines and ETL/ELT processes on GCP.
• Build, optimize, and maintain data solutions using Google BigQuery.
• Develop complex SQL queries for data transformation, aggregation, and analysis.
• Design efficient data models and optimize pipelines for performance, scalability, and cost.
• Integrate data from multiple sources and ensure data quality, reliability, and availability.
• Troubleshoot pipeline and data issues and drive continuous improvement.
• Collaborate with data architects, analysts, application teams, and business stakeholders.
• Follow best practices for cloud security, data governance, testing, and documentation.

Required Skills

• 8+ years of Data Engineering experience
• Strong hands-on experience with GCP, Django, and MongoDB
• Extensive experience with BigQuery
• Advanced SQL skills
• Strong understanding of ETL/ELT and data pipeline development
• Data modeling and data warehousing experience
• Experience handling large-scale datasets and performance optimization
• Strong problem-solving and communication skills

Good to Have

• GCP services such as Cloud Storage, Dataflow, Pub/Sub, Cloud Composer, or Cloud Functions
• Python or other data engineering languages
• Experience with data governance and security
• Agile development experience

About Us:

The QX Impact was launched with a mission to make A.I accessible and affordable and deliver AI Products/Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation. We believe without insights; businesses will continue to face challenges to better understand their customers and even lose them. Secondly, without insights businesses won't’ be able to deliver differentiated products/services; and finally, without insights, businesses can’t achieve a new level of “Operational Excellence” is crucial to remain competitive, meeting rising customer expectations, expanding markets, and digitalization.

Job Summary:

We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions.

As a Senior Data Engineer, you will work with internal and external data, collaborate with data scientists, and contribute to the design, development, and deployment of innovative solutions.

Key Responsibilities:

• Design, develop, test, and maintain optimal data pipeline and ETL architectures.
• Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
• Prepare and optimize data for predictive and prescriptive modeling.
• Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
• Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
• Utilize big data tools and frameworks to optimize data acquisition and preparation.
• Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
• Develop and curate data models for analytics, dashboards, and reports.
• Conduct code reviews, maintain production-level code, and implement testing approaches.
• Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
• Drive innovation and implement efficient new approaches to data engineering tasks.

Must-Have Skills:

• Bachelor’s degree in Computer Science, Mathematics, Engineering, or a related field.
• 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
• 3–5 years of experience designing and implementing data warehouse solutions.
• Expertise in SQL, including developing stored procedures (SP) and applying advanced data design concepts.
• Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
• Experience with AWS or Azure services (e.g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
• Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
• Hands-on experience with orchestration tools (e.g., Apache Airflow, Prefect).
• Knowledge of CI/CD processes, version control (e.g., Git, Jenkins), and deployment automation.
• Strong problem-solving, communication, and collaboration skills.

Good-to-Have Skills:

• Experience in integrating ERP data into data lakes.
• Experience with traditional ETL tools (e.g., Talend, Pentaho).

Competencies:

• Tech Savvy - Anticipating and adopting innovations in business-building digital and technology applications.
• Self-Development - Actively seeking new ways to grow and be challenged using both formal and informal development channels.
• Action Oriented - Taking on new opportunities and tough challenges with a sense of urgency, high energy, and enthusiasm.
• Customer Focus - Building strong customer relationships and delivering customer-centric solutions.
• Optimize Work Processes - Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.

Why Join Us?

• Be part of a collaborative and agile team driving cutting-edge AI and data engineering solutions.
• Work on impactful projects that make a difference across industries.
• Opportunities for professional growth and continuous learning.
• Competitive salary and benefits package.

Application Details

Ready to make an impact? Apply today and become part of the QX Impact team!

Job Summary

We are seeking a skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure. The ideal candidate should have strong expertise in SQL, Python, Linux, and modern data engineering practices to support data integration, transformation, and analytics.

Key Responsibilities

• Design, develop, and maintain ETL/ELT data pipelines.
• Write efficient and optimized SQL queries for data extraction, transformation, and reporting.
• Develop automation scripts using Python for data processing and workflow optimization.
• Work with Linux environments for deployment, monitoring, and troubleshooting.
• Ensure data quality, integrity, and reliability across data platforms.
• Collaborate with data analysts, software engineers, and business stakeholders to deliver data solutions.
• Monitor, troubleshoot, and optimize data pipelines for performance and scalability.
• Implement best practices for data security, governance, and documentation.

Required Skills

• Strong experience in Data Engineering concepts and ETL/ELT processes.
• Proficiency in SQL, including query optimization and database design.
• Strong programming skills in Python.
• Hands-on experience with Linux commands, shell scripting, and system administration basics.
• Experience with relational databases such as PostgreSQL, MySQL, SQL Server, or Oracle.
• Familiarity with Git/version control.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with cloud platforms (AWS, Azure, or GCP).
• Knowledge of Apache Spark, Airflow, Kafka, or similar data engineering tools.
• Experience with data warehousing solutions and big data technologies.
• Understanding of CI/CD pipelines and containerization (Docker/Kubernetes).

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Relevant certifications in cloud or data engineering are an added advantage.