248 PySpark Jobs in India

GCP Data engineer

Exp 9+yrs

Bangalore/Hyderabad

Lead exp+ Python + pyspark + sql or plsql

immediate to 15days joiners

Call or WhatsApp: 6 3 8 5 0 7 6 1 4 2

GCP Data engineer (L1 virtual, L2 or DM F2F )

GCP + Python + Pyspark + sql + Lead

8+yrs

Bangalore/hyderabad location ( relocation profiles also from Chennai pune location)

immediate to 15days

🚀 Hiring: Senior GCP Data Engineer

We are looking for experienced Senior Data Engineers with strong hands-on expertise in Google Cloud Platform (GCP), Python, PySpark, and SQL. Candidates with team lead experience will be preferred.

📌 Job Details:

• Role: Senior GCP Data Engineer
• Experience: 8+ Years
• Budget: Up to ₹30 LPA
• Location: Bangalore / Hyderabad
• Relocation: Candidates from Chennai and Pune willing to relocate are welcome.
• Notice Period: Immediate to 15 days
• Interview Process: L1 – Virtual | L2 / DM – Face-to-Face (F2F)

💻 Required Technical Skills:

• Strong hands-on experience with GCP data engineering.
• Excellent proficiency in Python, PySpark, and SQL.
• Experience in designing, developing, and optimizing data pipelines.
• Good understanding of ETL/ELT processes and large-scale data processing.
• Experience working with cloud-based data platforms and data warehousing.
• Strong analytical, problem-solving, and communication skills.

⭐ Preferred Skills:

• Previous experience as a Technical Lead or leading a data engineering team.
• Knowledge of GCP services such as BigQuery, Dataproc, Dataflow, and Cloud Storage.

Job Title : Data / AI / GenAI – Engineering & Analytics

Experience : 4+ Years

Location : Noida / Gurugram / Ahmedabad

Employment Type : Full-time | On-site at client location

Working Days : 5 days per week

Working Hours : Standard business hours

Position Overview : We are seeking professionals to deliver modern data engineering, business intelligence, cloud analytics and AI-powered applications, according to their specialization.

Mandatory Skills :

Python, SQL, ETL/ELT, Data Engineering, Power BI, Azure Data Factory, Databricks, PySpark, Data Modeling, GenAI/LLMs, RAG, Vector Databases, AI Agents (as per role specialization).

Key Roles / Specializations :

1. Power BI Developers : Dashboards, DAX, Power Query, reporting and data modeling
2. Azure Data Engineers : Azure Data Factory, Data Lake, Azure SQL and cloud pipelines
3. Databricks Engineers : Apache Spark, PySpark, Delta Lake and scalable processing
4. Data Engineers : ETL/ELT, SQL, Python, data warehousing and quality
5. GenAI / RAG Engineers : LLMs, prompts, embeddings, vector databases and retrieval pipelines
6. Agentic AI Engineers : AI agents, tool calling, orchestration and multi-agent workflows

Key Responsibilities :

• Design, develop and maintain data pipelines, analytics solutions or AI applications.
• Build and optimize data integration, transformation and processing workflows.
• Develop dashboards and analytical models for business decision-making.
• Use cloud data platforms and modern engineering frameworks.
• Develop and evaluate LLM applications, RAG solutions or AI agents as applicable.
• Ensure data quality, security, performance and reliability.
• Collaborate with engineering teams and stakeholders on testing, deployment and documentation.

Required Skills & Qualifications :

• Bachelor’s degree in Computer Science, IT, Data Science, Engineering or related discipline.
• 4–5 years of relevant experience in data, analytics, AI or software engineering.
• Hands-on expertise in at least one listed specialization; not all technologies are required.
• Relevant Python, SQL, DAX or other technical skills depending on the position.
• Understanding of cloud platforms, APIs, integration and software practices.

Preferred Qualifications :

• Azure, Databricks or Microsoft Fabric experience.
• For GenAI roles : Azure OpenAI, OpenAI APIs, LangChain, LlamaIndex or LangGraph.
• For relevant roles : vector databases, CI/CD, Docker, MLOps or LLMOps.

Note : Candidates are expected to have expertise in their relevant specialization, not in every listed technology or role.

Job Title : Senior Data Engineer – GCP & Apache Spark

Job Details :

• Number of Positions : 3
• Employment Type : Contract-to-Hire (C2H)
• Experience Required : 5 to 8 Years
• Location : Manyata Tech Park, Bengaluru
• Work Mode : Hybrid – 3 Days in Office per Week
• Preferred Notice Period : Immediate to 15–20 Days
• Project Duration : Initially 6–8 months, with a strong possibility of extension for long-term projects
• Relocation : Candidates from outside Bengaluru can apply if they are willing to relocate.

Role Overview :

We are looking for experienced Senior Data Engineers with strong expertise in Google Cloud Platform (GCP), Apache Spark, BigQuery, and SQL to design, develop, and maintain scalable data pipelines and distributed data processing solutions.

The ideal candidate should have hands-on experience with cloud-based data engineering, end-to-end pipeline architecture, system design, performance optimization, debugging, and CI/CD practices.

Mandatory skills :

Apache Spark (PySpark / Scala), SQL, GCP, BigQuery, GCP Dataproc, Data Pipelines, System Design, CI/CD, Debugging & Problem-Solving.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and/or Scala.
• Build and optimize batch and distributed data processing workflows on GCP.
• Work extensively with BigQuery for data transformation, validation, analytics, and query optimization.
• Develop and maintain data workflows using GCP services, including Dataproc, BigQuery, and Cloud Storage.
• Design end-to-end data pipeline architectures with a focus on scalability, reliability, and performance.
• Write and optimize complex SQL queries for large-scale datasets.
• Troubleshoot and resolve data processing issues, pipeline failures, and performance bottlenecks.
• Implement CI/CD practices for data engineering workflows, testing, and deployments.
• Monitor pipeline performance and identify opportunities for continuous improvement.
• Collaborate with cross-functional teams to understand requirements and deliver reliable data solutions.
• Participate in code reviews, testing, production deployments, and production support.

Mandatory Skills :

• Strong hands-on experience with Apache Spark, including PySpark and/or Scala.
• Strong SQL skills and experience working with large-scale datasets.
• Hands-on experience with Google Cloud Platform (GCP).
• Practical experience with BigQuery and data warehousing concepts.
• Working knowledge of GCP Dataproc.
• Strong understanding of data pipelines, end-to-end workflows, and system design.
• Practical experience with CI/CD and deployment processes.
• Excellent debugging, troubleshooting, and problem-solving skills.

Preferred Skills :

• Experience with Apache Airflow, Google Cloud Composer, or similar orchestration tools.
• Knowledge of Kafka or other streaming and messaging technologies.
• Proficiency in Python for data engineering and automation.
• Experience with Git-based development workflows.
• Knowledge of Linux/Unix environments and shell scripting.
• Familiarity with monitoring, logging, data quality, and governance practices.

Required Qualifications :

• 5 to 8 years of experience in Data Engineering, Big Data Engineering, or a related field.
• Proven experience developing and maintaining scalable data pipelines.
• Strong understanding of distributed data processing and cloud-based data architecture.
• Ability to troubleshoot complex technical issues and optimize pipeline performance.
• Strong communication, collaboration, and ownership skills.

Interview Process :

Round 1 : AI-Based Interview

• Candidates must complete the AI interview as soon as possible after receiving the invitation.

Round 2 : Virtual Interview Drive

• Interviews are planned for the upcoming drive; exact dates will be communicated separately.

Round 3 : Online Client Interview

• Shortlisted candidates will attend the online interview.
• Candidates must be available throughout the day on the scheduled interview date.

1st virtual , 2nd round F2F -

Python pyspark, SQL, data engineer

5+yrs

Bang/hyderabad

immediate to 15days.

Job Description

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines.
• Work with Hadoop and PySpark to process and transform large volumes of data.
• Develop and optimize Python scripts for data processing and automation.
• Write complex SQL queries for data extraction, transformation, and validation.
• Perform data cleansing, integration, transformation, and validation.
• Work with data warehousing concepts and large-scale data processing systems.
• Optimize ETL workflows for performance, scalability, and reliability.
• Troubleshoot data pipeline failures and resolve data quality issues.
• Collaborate with data engineers, analysts, and other technical teams.
• Follow data governance, quality, and development best practices.

Required Skills

• Strong experience in ETL Development – Mandatory
• Hands-on experience with Hadoop – Mandatory
• Strong experience in PySpark – Mandatory
• Good programming skills in Python – Mandatory
• Strong knowledge of SQL – Mandatory
• Experience in Data Processing and Transformation – Mandatory
• Knowledge of Data Warehousing – Mandatory
• Experience with Big Data Technologies – Mandatory

Opportunity Details

Role: ETL Developer – Hadoop & PySpark

Experience: 6–10 Years

Location: Bangalore

Mandatory Skills

ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies

Hiring: ETL Developer – Hadoop & PySpark

Location: Bangalore

Experience: 6–10 Years

Budget: 25–30 LPA

Work Mode: Weekdays

Job Description

We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Develop data transformation workflows using PySpark and Python.
• Work with Hadoop and other big data technologies to process and manage large datasets.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Implement data cleansing, data quality checks, and data validation processes.
• Develop and optimize data integration workflows for data warehousing solutions.
• Troubleshoot ETL failures, performance issues, and data discrepancies.
• Optimize PySpark jobs and SQL queries for performance and scalability.
• Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
• Ensure data accuracy, consistency, and adherence to established data governance standards.

Mandatory Skills

• Strong hands-on experience in ETL development.
• Good experience with Hadoop and big data technologies.
• Proficiency in PySpark and Python programming.
• Strong SQL skills, including complex queries and data manipulation.
• Experience in data processing, data transformation, and data integration.
• Understanding of data warehousing concepts and architecture.
• Experience in developing, debugging, and optimizing ETL pipelines.

Required Experience

• 6–10 years of relevant experience in ETL development and data engineering.
• Practical experience handling large datasets using Hadoop and PySpark.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently and collaborate effectively within a team.

Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.

Dear Candidate,

We have an immediate opening with Mphasis for Senior Full Stack Engineer (Python AI).

📍 Location: Hyderabad / Bangalore

💼 Experience: 5+ Years

🏢 Employment: Permanent

⚡ Immediate Joiners Preferred

Mandatory Skills:

• Python
• Java + Spring Boot
• React.js
• MongoDB

GenAI

Dear Candidate,

Thanks for showing interest in the opportunity!!!

As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.

Job Description:

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.

Primary & Mandatory Skills:

• ETL Development
• Hadoop / HDFS
• PySpark / Apache Spark
• Data Extraction, Transformation & Loading
• SQL and Data Processing

Good to Have:

• Hive / Spark SQL
• Data Pipeline Development
• Big Data Processing
• Python Programming

Company: Mphasis

Role: ETL Developer – Hadoop & PySpark

Experience: As per requirement

Location: Bangalore

Employment Type: Permanent

role: data engineer

Python pyspark, SQL, data engineer

5+yrs

Bangalore/Hyderabad

immediate to 15days.

Company: Wissen Technology

Position: Databricks Engineer

Experience: 6-10 Years

Location: Bengaluru/Mumbai

Employment Type: Full-time

About Wissen Technology

Wissen Technology is a global technology services company focused on delivering innovative software engineering, data, and technology solutions to leading enterprises. The company works with clients across industries to build scalable, high-performance technology platforms and solve complex business and technology challenges.

With a strong focus on engineering excellence, innovation, and collaboration, Wissen Technology brings together skilled technology professionals across areas such as software engineering, data engineering, cloud, analytics, and digital transformation.

At Wissen Technology, employees have the opportunity to work on challenging technology projects, collaborate with experienced engineering teams, and contribute to solutions that create measurable business impact.

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to understand requirements and deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement appropriate data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands-on expertise in Apache Spark and PySpark.
• Strong programming skills in Python; experience with Scala is an advantage.
• Strong proficiency in SQL and hands-on experience with relational databases.
• Practical experience developing and optimizing data pipelines and data processing applications.
• Familiarity with Git and version control systems.
• Strong understanding of data engineering concepts, data transformation, and data validation.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Job Title : Tech Lead – Data Lake Platform

Number of Positions : 2

Experience : 7+ Years

Role Type : Technical Lead / Data Platform Lead

Domain : Data Engineering / Data Platform / AWS

About the Role :

We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.

The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.

As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.

Core Tech Stack :

AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka

Key Responsibilities :

• Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
• Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
• Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
• Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
• Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
• Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
• Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
• Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
• Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
• Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
• Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
• Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.

Must-Have Skills :

• 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
• Strong hands-on experience with AWS Data Services, including:
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Athena
• Amazon Redshift
• AWS DMS
• AWS Lambda
• Amazon RDS
• AWS IAM
• Strong production experience with Apache Airflow.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
• Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
• Strong understanding of data lake architecture and enterprise data platforms.
• Proven experience leading engineers and driving technical decisions.
• Hands-on experience with production support, troubleshooting, incident management, and RCA.
• Strong understanding of data platform performance, scalability, reliability, and SLA management.

Good-to-Have Skills :

• Experience with DBT and modern data transformation practices.
• Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
• Strong knowledge of Amazon Redshift workload optimization, including :
• Distribution keys
• Sort keys
• Spectrum
• External tables
• Experience with Kafka or other streaming/data ingestion technologies.
• Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
• Experience with Lambda and API Gateway for API-based data serving.
• Experience with enterprise data governance, data quality, security, and compliance.
• Experience in BFSI / Banking / Financial Services / Insurance domain.

Ideal Candidate :

The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.

Data Architect – Databricks & AWS

• Strong experience in Data Architecture, Data Engineering, Databricks, Apache Spark/PySpark, Python, and Advanced SQL.
• Design and implement scalable ETL/ELT pipelines, data platforms, and Lakehouse architectures using Medallion Architecture.
• Experience with Databricks, Databricks Workflows, Unity Catalog OR Databricks Jobs
• Strong knowledge of Delta Lake, Databricks Workflows, Delta Live Tables (DLT), and dimensional data modeling.
• Hands-on experience with AWS services such as S3, Glue, IAM, Lambda, and CloudWatch.
• Experience with Apache Airflow, Data Warehouse concepts, Git/CI-CD, performance optimization, and data quality.
• Good to have exposure to Kafka/Structured Streaming, Unity Catalog, and modern data governance.

Role Overview

We are looking for an experienced Data Scientist – Agentic AI with strong expertise in Python, Machine Learning, Generative AI, Large Language Models (LLMs), RAG and Agentic AI.

The ideal candidate should have hands-on experience in developing, fine-tuning, evaluating and deploying machine learning and GenAI solutions. The candidate should be comfortable working with open-source LLMs, LangChain/LangGraph, PySpark and AI observability/tracing frameworks.

The role involves building intelligent AI systems that can reason, use tools, retrieve information and execute multi-step tasks using Agentic AI architectures.

Mandatory Technical Skills

1. Data Science & Python

• 5+ years of experience in Data Science / Machine Learning / AI.
• Strong programming experience in Python.
• Strong understanding of data analysis, feature engineering and statistical techniques.
• Experience with Python ML and data science libraries such as:
• NumPy
• Pandas
• Scikit-learn
• Matplotlib / Seaborn
• Good understanding of data preprocessing, exploratory data analysis and experimentation.

2. Machine Learning & Statistics

• Strong understanding of Machine Learning fundamentals.
• Experience with supervised and unsupervised learning techniques.
• Knowledge of:
• Regression
• Classification
• Clustering
• Feature Engineering
• Model Selection
• Hyperparameter Tuning
• Cross-validation
• Strong understanding of Statistics / ML fundamentals.
• Ability to interpret model performance and statistical results.

3. Generative AI / LLM

• Strong hands-on experience with Generative AI and Large Language Models (LLMs).
• Understanding of Transformer architecture and modern LLM-based applications.
• Experience working with commercial or open-source LLMs.
• Strong understanding of:
• Prompt Engineering
• Context Management
• Embeddings
• Tokenization
• LLM inference
• Hallucination mitigation

4. RAG – Retrieval Augmented Generation

• Strong hands-on experience developing RAG applications.
• Experience with:
• Document ingestion
• Chunking
• Embeddings
• Vector search
• Semantic search
• Retrieval pipelines
• Context retrieval
• Reranking
• Ability to optimize RAG pipelines for relevance, accuracy and latency.
• Experience integrating LLMs with enterprise knowledge sources.

5. Agentic AI

• Hands-on experience building Agentic AI / AI Agent solutions.
• Understanding of agent architecture and multi-step reasoning workflows.
• Experience with:
• AI Agents
• Multi-Agent systems
• Tool Calling
• Function Calling
• Agent orchestration
• Planning and reasoning workflows
• Memory
• Workflow automation
• Ability to build agents that can interact with tools, APIs, databases and external systems.

6. LangChain / LangGraph

• Strong hands-on experience with LangChain and/or LangGraph.
• Experience building LLM workflows and agent-based applications.
• Understanding of:
• Chains
• Agents
• Tools
• State management
• Graph-based workflows
• Agent orchestration
• Retrieval workflows
• Experience designing scalable Agentic AI workflows.

7. LLM Fine-Tuning

• Hands-on experience with LLM fine-tuning.
• Understanding of techniques such as:
• Supervised Fine-Tuning (SFT)
• Parameter-Efficient Fine-Tuning
• LoRA
• QLoRA
• Experience preparing datasets for fine-tuning.
• Ability to evaluate fine-tuned models against baseline models.
• Understanding of model optimization and inference considerations.

8. BERT / LLaMA / Open-Source LLMs

Experience working with one or more open-source / transformer-based models such as:

• BERT
• LLaMA / Llama
• Mistral
• Gemma
• Qwen
• Other open-source LLMs

Candidate should understand model loading, inference, fine-tuning and evaluation.

9. PySpark

• Strong experience with PySpark for large-scale data processing.
• Experience working with large datasets and distributed data processing.
• Knowledge of:
• Data transformations
• Data cleaning
• Aggregations
• Joins
• Spark SQL
• Performance optimization
• Ability to build scalable data processing pipelines.

10. Model Validation & Evaluation

• Experience validating and evaluating ML and GenAI models.
• Understanding of traditional ML evaluation metrics.
• Experience evaluating LLM/RAG applications using relevant quality metrics.
• Ability to compare model performance and identify areas for improvement.
• Experience with:
• Accuracy
• Precision
• Recall
• F1 Score
• ROC-AUC
• Retrieval metrics
• LLM response quality
• Groundedness / relevance
• Experience designing evaluation datasets and test cases is preferred.

11. AI Tracing / Observability

• Experience with AI/LLM tracing and observability.
• Ability to monitor AI applications in production.
• Experience tracking:
• LLM requests/responses
• Latency
• Token usage
• Errors
• Retrieval performance
• Agent/tool execution
• Model performance
• Exposure to tools/frameworks such as LangSmith, OpenTelemetry, Arize Phoenix, MLflow or similar is preferred.

12. Model Deployment

• Experience deploying ML/LLM/GenAI solutions into production.
• Exposure to cloud and/or on-premise model deployment.
• Experience with model serving, APIs and production inference.
• Knowledge of deployment environments such as:
• AWS
• Azure
• GCP
• On-premise infrastructure
• Experience with Docker, APIs and CI/CD is an advantage.

Key Responsibilities

• Design, develop and deploy Data Science, Machine Learning and GenAI solutions.
• Build production-ready RAG and Agentic AI applications.
• Develop intelligent agents capable of tool calling, reasoning and multi-step task execution.
• Build LLM-powered applications using LangChain/LangGraph.
• Work with open-source LLMs including BERT, LLaMA and other transformer-based models.
• Fine-tune LLMs for specific business use cases.
• Develop scalable data processing pipelines using PySpark.
• Perform data analysis, feature engineering and statistical modeling.
• Develop and maintain model validation and evaluation frameworks.
• Evaluate ML and LLM models using appropriate performance and quality metrics.
• Implement AI tracing, monitoring and observability for production GenAI systems.
• Deploy models and AI applications in cloud or on-premise environments.
• Optimize model performance, response quality, latency and cost.
• Troubleshoot issues related to model inference, retrieval, agents and LLM workflows.
• Collaborate with Data Scientists, ML Engineers, Software Engineers and business stakeholders.
• Convert business requirements into scalable AI/ML solutions.

Good to Have

• Experience with Vector Databases such as:
• FAISS
• Pinecone
• Weaviate
• Milvus
• Chroma
• Azure AI Search
• Experience with MLflow or similar ML lifecycle tools.
• Experience with Docker/Kubernetes.
• Experience with REST APIs / FastAPI.
• Knowledge of cloud AI/ML services.
• Experience with MLOps / LLMOps.
• Experience with multi-agent frameworks other than LangChain/LangGraph.
• Experience working with enterprise GenAI applications.

Ideal Candidate Profile

The ideal candidate should be a Data Scientist / ML Engineer with strong GenAI and Agentic AI experience, rather than a pure Python developer.

A strong candidate would typically have:

Data Science + Python + ML + Statistics + GenAI/LLM + RAG + Agentic AI + LangChain/LangGraph + LLM Fine-Tuning + Open-Source LLMs + PySpark + Model Evaluation + AI Observability + Model Deployment.

Core Mandatory Skills

Data Science, Python, Machine Learning, Statistics/ML Fundamentals, GenAI/LLM, RAG, Agentic AI, LangChain/LangGraph, LLM Fine-Tuning, BERT/LLaMA/Open-Source LLMs, PySpark, Model Validation/Evaluation, AI Tracing/Observability, Cloud/On-Prem Model Deployment.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

9+yrs

Bangalore/Hyderabad

immediate to 15days.

Job Description – Data Engineer

Role: Data Engineer

Experience: 9+ Years

Location: Bangalore / Hyderabad

Notice Period: Immediate to 15 Days

Interview Process: 1st Round – Virtual | 2nd Round – F2F

Mandatory Skills

• Python
• PySpark
• SQL
• Azure Data Engineering

Job Description

We are looking for an experienced Azure Data Engineer with 9+ years of experience and strong hands-on expertise in Python, PySpark, SQL, and Azure Data Engineering.

Key Responsibilities

• Develop and maintain scalable data engineering solutions using Azure.
• Build and optimize data processing pipelines using PySpark and Python.
• Write complex SQL queries for data extraction and transformation.
• Work with Azure data services and cloud-based data platforms.
• Perform data processing, transformation, and integration.
• Troubleshoot data pipeline and production issues.
• Collaborate with technical and business teams to deliver data solutions.

Preferred: Immediate to 15 Days joiners.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

Bang/hyderabad

immediate to 15days.

Responsibilities and JD

Job Description: We are looking for a Senior Developer with strong expertise in PySpark, Databricks, and Snowflake to build scalable data engineering solutions and enterprise data platforms.

Key Responsibilities:

1. Design, develop, and maintain ETL/ELT pipelines using PySpark, Databricks, and Snowflake.
2. Develop batch and real-time data processing solutions for structured and semi-structured data.
3. Build and optimize Databricks notebooks, workflows, and Delta Lake solutions.
4. Design and implement Snowflake databases, schemas, views, stored procedures, tasks, and streams.
5. Develop scalable data models, data marts, and data warehouse solutions.
6. Optimize PySpark jobs, Databricks workloads, and Snowflake queries for performance and cost efficiency.
7. Implement data quality, validation, governance, and security controls.
8. Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.
9. Manage source control and CI/CD deployments using Git and Azure DevOps.
10. Troubleshoot production issues, perform root cause analysis, and ensure pipeline reliability.
11. Mentor junior team members and participate in code reviews and technical design discussions.

Required Skills: PySpark, Databricks, Snowflake, Python, SQL.

Experience: 5+ years of Data Engineering experience with strong hands-on expertise in PySpark, Databricks, and Snowflake.

Job Description – Senior GCP Data Engineer

Position: Senior GCP Data Engineer

Experience: 8+ Years

Budget: Up to 30 LPA

Location: Bangalore / Hyderabad

Relocation: Candidates from Chennai / Pune willing to relocate are also welcome.

Notice Period: Immediate to 15 Days

Interview Process:

• Round 1: Virtual Interview
• Round 2 / DM Round: Face-to-Face Interview

Required Skills

• Strong hands-on experience in GCP Data Engineering.
• Good programming experience in Python.
• Strong experience with PySpark / Apache Spark.
• Proficiency in SQL and database concepts.
• Experience in developing and maintaining data pipelines.
• Good understanding of data processing, transformation, and integration.
• Experience in handling large-scale data processing.
• Strong problem-solving and debugging skills.
• Good understanding of cloud-based data engineering concepts.
• Experience in optimizing data pipelines for performance and scalability.

Preferred Skills

• Previous experience in a Lead Data Engineer role.
• Experience in technical leadership, mentoring, and guiding team members.
• Ability to collaborate with cross-functional teams and deliver data engineering solutions.

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines using GCP.
• Develop data processing and transformation workflows using Python and PySpark.
• Write and optimize SQL queries for efficient data processing.
• Build and maintain reliable data engineering solutions.
• Troubleshoot data pipeline issues and optimize performance.
• Ensure data quality, reliability, and scalability.
• Collaborate with stakeholders and team members to deliver project requirements.
• Lead and guide team members where applicable.
• Important: Candidates must have strong Senior Data Engineer experience. Lead experience is preferred but not mandatory.

Senior Data Engineer – PySpark & Oracle

Experience: 7+ Years

Location: Bangalore

Notice Period: Immediate to 10 Days

Key Skills:

• Strong expertise in Data Modeling, Data Design & Modernization
• Primary skills: PySpark, Oracle SQL/PLSQL
• Secondary skills: Python, ETL & Data Pipelines
• Experience with Kafka and Hadoop
• Exposure to AWS / Azure / GCP
• Good knowledge of Git and JIRA

Roles & Responsibilities:

• Design, develop, and modernize scalable data models and data architecture.
• Develop and optimize data processing solutions using PySpark and Oracle SQL/PLSQL.
• Build and maintain robust ETL workflows and data pipelines.
• Work with Kafka, Hadoop, and cloud platforms for data processing and integration.
• Perform data transformation, optimization, and performance tuning.
• Collaborate with technical teams on data design, development, testing, and deployment.

🚨 WE ARE HIRING – SENIOR DATA ENGINEER | BANGALORE 🚨

Looking for experienced Senior Data Engineers with strong expertise in PySpark, Oracle SQL/PLSQL, and Data Modeling!

🔹 Role: Senior Data Engineer

🔹 Experience: 7+ Years

🔹 Location: Bangalore

🔹 CTC: Up to 26 LPA

🔹 Notice Period: Immediate to 10 Days Preferred

💻 Mandatory Skills

✅ PySpark

✅ Oracle SQL / PL/SQL

✅ Data Modeling & Design / Modernization

✅ Python

✅ ETL

✅ Data Pipelines

⚙️ Good to Have / Ecosystem Skills

✅ Kafka

✅ Hadoop

✅ AWS / Azure / GCP

✅ Git

✅ JIRA

📌 Key Responsibilities

• Develop and maintain scalable data pipelines using PySpark

• Work extensively with Oracle SQL/PLSQL for data processing and transformation

• Design and implement data models and data architecture

• Work on data design and modernization initiatives

• Develop and optimize ETL processes and data pipelines

• Handle large volumes of data using PySpark and Python

• Work with technologies such as Kafka, Hadoop, and Cloud platforms

• Collaborate with cross-functional teams to deliver scalable data solutions

• Use Git and JIRA for version control and project tracking

📩 Interested candidates can share their updated CV along with:

Total Experience:

Relevant PySpark Experience:

Relevant Oracle SQL/PLSQL Experience:

Data Modeling Experience:

Current Location:

Notice Period:

Current CTC:

Expected CTC:

#Hiring #DataEngineer #SeniorDataEngineer #PySpark #Oracle #PLSQL #DataModeling #Python #ETL #DataPipelines #Kafka #Hadoop #AWS #Azure #GCP #BangaloreJobs #ITJobs #TechJobs #ImmediateJoiners

🚨 Hiring: GCP Data Engineer

We are looking for experienced GCP Data Engineers to join our team!

🔹 Experience: 9+ Years

🔹 Relevant Experience: 4+ Years in GCP Data Engineering

🔹 Required Skills: GCP, Oracle PL/SQL, Python, PySpark

🔹 Location: Bangalore / Hyderabad

🔹 Notice Period: Immediate to 10 Days Preferred

Key Skills:

🔹 Strong hands-on experience in GCP Data Engineering

🔹 Good experience with PySpark & Python

🔹 Strong knowledge of Oracle PL/SQL

🔹 Experience in data processing, ETL, and data pipelines

🔹 Good understanding of cloud-based data engineering

📩 Interested candidates can share their updated CV via DM.

#Hiring #GCPDataEngineer #GCP #DataEngineering #PySpark #Python #OraclePLSQL #DataEngineer #BangaloreJobs #HyderabadJobs #ImmediateJoiner #TechJobs #ITJobs #HiringNow

GCP Data Engineering Lead

Experience: 9+ Years

Lead Experience: 2+ Years

Location: Bangalore / Hyderabad

Key Skills:

• Strong experience in GCP Data Engineering and BigQuery
• Hands-on experience with Oracle Exadata / PL-SQL
• Strong knowledge of PySpark / Scala and Python
• Experience with GoldenGate, Kafka and CDC
• Hands-on experience with Apache Airflow
• Good experience in CI/CD and DevOps practices
• Experience with Terraform / Infrastructure as Code
• Exposure to AI/LLM technologies and GenAI solutions
• Strong understanding of data architecture, ETL/ELT and data pipelines

Roles & Responsibilities:

• Lead the design and development of scalable GCP data engineering solutions.
• Design and implement batch and real-time data pipelines using BigQuery, PySpark, Kafka/CDC and Airflow.
• Work with Oracle Exadata/PL-SQL and GoldenGate for data integration and migration.
• Implement CI/CD pipelines and infrastructure automation using Terraform.
• Explore and integrate AI/LLM capabilities into data engineering solutions.
• Lead technical discussions, code reviews, solution design and mentor team members.
• Collaborate with business and technical teams to deliver high-quality data solutions.

Job Description: Python + AI

Company: Wissen Technology

Location: Bangalore, India

Experience: 5+Years

Employment Type: Full-Time

Role: Python + AI / Data Engineer

About the Role

Wissen Technology is looking for experienced Python + AI / Data Engineering professionals to join our technology team in Bangalore. The ideal candidate will have strong hands-on experience in Python, Artificial Intelligence, Generative AI, PySpark, Snowflake, and data pipeline development.

The candidate should be capable of designing and developing scalable data and AI solutions, building robust ETL/ELT pipelines, working with large datasets, and integrating AI/ML capabilities into enterprise applications.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Develop robust ETL/ELT pipelines for processing large volumes of structured and unstructured data.
• Build and optimize data processing solutions using Apache Spark / PySpark.
• Develop data ingestion and transformation pipelines into Snowflake.
• Design and implement scalable Snowflake data models, tables, views, and SQL transformations.
• Work with batch and, where applicable, real-time data processing pipelines.
• Build and integrate AI and Generative AI solutions using Python.
• Develop LLM-based applications, RAG solutions, AI agents, and AI-powered services.
• Integrate AI models with enterprise data platforms and data pipelines.
• Develop REST APIs and microservices using FastAPI, Flask, or Django.
• Perform data cleansing, transformation, validation, and quality checks.
• Optimize PySpark jobs, SQL queries, Snowflake workloads, and data pipelines for performance and scalability.
• Implement data pipeline monitoring, logging, error handling, and alerting.
• Work with cloud platforms such as AWS, Azure, or GCP.
• Collaborate with Data Scientists, Data Engineers, Software Engineers, Architects, and business stakeholders.
• Participate in technical design, architecture, code reviews, and production support.
• Mentor junior engineers and contribute to engineering best practices.

Preferred Qualifications

• Bachelor's or master's degree in computer science, Engineering, Data Science, Artificial Intelligence, or a related field.
• Experience working on enterprise-scale AI and data engineering projects.
• Experience combining Python + PySpark + Snowflake + AI/GenAI in production environments.
• Experience with Databricks is an advantage.
• Experience with AI Agents / Agentic AI and tool/function calling.
• Knowledge of distributed systems and cloud-native architecture.
• Experience leading technical initiatives or mentoring engineering teams.

Data Engineer Hiring Post

🚨 Hiring: Data Engineer | PySpark + Python + SQL

We are looking for experienced Data Engineers to join our team!

🔹 Experience: 5 to 9 Years

🔹 Locations: Bangalore / Hyderabad

🔹 Interview Process:

• 1st Round – Virtual

• 2nd Round – Face-to-Face (Karat Test)

🔑 Key Skills:

✅ PySpark

✅ SQL

✅ Python

✅ ETL

📩 Interested candidates can share their updated resume.

#Hiring #DataEngineer #PySpark #Python #SQL #ETL #BangaloreJobs #HyderabadJobs #TechHiring #ImmediateHiring

Data Engineer Short Hiring Post

🚨 Hiring: Data Engineer

🔹 Experience: 5–9 Years

🔹 Location: Bangalore / Hyderabad

🔹 Skills: PySpark, Python, SQL, ETL, CI/CD, Data Modeling

🔹 Process: L1 Virtual → L2 F2F Karat Test

🔹 F2F: Bangalore / Hyderabad Location

🔹 Positions: Immediate requirement

⚠️ Note: Candidates must be available for F2F Karat immediately after L1.

#Hiring #DataEngineer #PySpark #Python #SQL #BangaloreJobs #HyderabadJobs #Mphasis #ImmediateJoiners

Job Description:

We are looking for a skilled Python Developer with 2+ years of hands-on experience in backend development and data processing. The ideal candidate should be proficient in Python and have working experience with web frameworks like Flask and FastAPI, along with exposure to data engineering and machine learning workflows.

Key Responsibilities:

Design, develop, and maintain scalable Python applications and APIs using Flask and FastAPI

Work with PySpark and Kafka for real-time data processing

Perform data manipulation and analysis using Pandas and NumPy

Develop and maintain modular, reusable, and testable code following OOP principles

Collaborate with data scientists to integrate ML models (TensorFlow, PyTorch, Scikit-learn) into production

Participate in code reviews, design discussions, and contribute to best practices

Write and maintain documentation for developed modules and workflows

Required Skills:

Strong proficiency in Python programming

Hands-on experience with Flask and/or FastAPI

Experience working with PySpark and Kafka

Solid understanding of Pandas, NumPy, and data handling in Python

Familiarity with TensorFlow, PyTorch, and scikit-learn

Good grasp of Object-Oriented Programming (OOP) concepts

Ability to write modular and maintainable code

Nice to Have:

Understanding of Machine Learning concepts and pipelines

Exposure to Generative AI (GenAI) technologies and tools

Experience with CI/CD tools, Docker, or cloud environments (AWS, GCP, etc.)

Qualifications:

Bachelor's degree in Computer Science, Engineering, or a related field

Data Engineer

Location: Bengaluru, India (Hybrid)

Employment Type: Full-time

Experience: 3-5 years

Role Overview  

What We’re Looking For:

• Bachelor’s degree in Computer Science/Engineering or equivalent experience required.
• Experience designing and shipping cloud services products.
• Experience driving and managing technical and architectural dependencies on AWS Cloud.
• A firm understanding of system architecture, cloud computing, PaaS/SaaS design principles, S3, DynamoDB, RDS mandatory.
• Experience in building or maintaining ETL processes and tools, i.e., AWS Glue or any open-source tool.
• Proven system-level design contribution to a current “Live” (in production / under daily high load) multi-region SaaS or PaaS offering.
• Proven experience with S3, DynamoDB, SQL, and AWS RDS services.
• Proficiency in programming languages such as Python.
• Strong analytical and problem-solving skills.
• 
  

Required Skills & Experience

• Experience with Python, SQL, and data visualization/exploration tools.
• Familiarity with the AWS ecosystem, specifically S3, DynamoDB, and RDS.
• Communication skills, especially for explaining technical concepts to nontechnical business leaders.
• Ability to work on a dynamic, research-oriented team that has concurrent projects.
• Experience in AWS cost optimization (Savings Plans, Reserved Instances, Spot Instances) and governance frameworks.
• Experience developing solutions using infrastructure orchestration tools (SSM, automation account, Ansible, etc.).
• Excellent leadership, stakeholder management, and communication skills.

What We Offer

• Work with some of the brightest minds in the emerging EV industry.
• Make a tangible impact in reducing carbon emissions and enabling sustainable energy.
• Freedom to suggest, implement, and innovate on systems, processes, and technologies.
• Daily ownership in a high-growth, challenging environment.
• Flexible work environment with hybrid schedules and virtualization options.
• Competitive pay and benefits including health coverage, innovative PTO program, and performance bonuses.

Role Descriptions: Azure data engineer

SN Required Information Details

1 Role** Azure Cosmos DB Developer

2 Required Technical Skill Set** Primary - PySpark Azure Cosmos DB

Secondary -Python, Microsoft Fabric

3 No of Requirements** 4

4 Desired Experience Range** 8 to 12

5 Location of Requirement Pune

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

1. Deep hands-on experience with Python,Pyspark,Spark batch,notebook in Azure

2. Experience in Cosmos DB – including data modeling, indexing, partitioning, consistency levels, and performance tuning.

3. Strong understanding of Cosmos DB APIs (Core SQL API, MongoDB API, etc.) and integration patterns.

4. Proficiency in query optimization and troubleshooting Cosmos DB performance issues.

5. Experience in data processing using PySpark and Python.

6. Familiarity with Microsoft Fabric for data engineering and analytics workflows.

7. Proficient using source code management tools such as Git or GitHub

8. Experience with Test Driven Development and / or Behavior Driven Development.

Good-to-Have 1. Exposure to data governance tools like Azure Purview.

2. Familiar with various design patterns

3. Familiar with Azure: SQL Managed Instance, Cosmos DB, Storage Services, Azure Functions

4. Global project experience, and excellent communication skills, verbal and written, and soft skills in agile projects

SN Responsibility of / Expectations from the Role

1 Design and implement scalable and high-performance solutions using Azure Cosmos DB.

2 Develop data ingestion and transformation pipelines using PySpark and Azure Data Factory.

3 Optimize Cosmos DB performance through indexing, partitioning, and query tuning.

4 Collaborate with architects and data engineers to ensure best practices in data modeling and cloud architecture.

5 Implement and maintain CI/CD pipelines for automated deployments and testing.

6 Troubleshoot and resolve technical issues related to Cosmos DB and data pipelines.

Desirable Skills:

Keyword:

Skills: Digital : Python~Digital : Databricks~Digital : PySpark~Microsoft Fabric~MySQL

Experience Required: 8-10.

Job Description – QA & Data Validation Engineer

Experience: 5–6 Years

Location: Pan India

Employment Type: Full-Time

Work Mode: Pan India / Remote or Hybrid as applicable

About the Role

We are looking for an experienced QA & Data Validation Engineer with 5–6 years of hands-on experience in data quality assurance, solution analysis, data validation, SQL, Python, PySpark, Azure Data Factory, Azure Synapse Analytics, and Power BI validation.

The ideal candidate will be responsible for validating large-scale data pipelines, performing source-to-target reconciliation, analyzing business rules, investigating data defects, and ensuring the accuracy, completeness, and consistency of data across source, staging, intermediate, and target systems.

The role requires strong analytical and problem-solving skills along with the ability to work closely with development, data engineering, business, and other stakeholders in an Agile delivery environment.

You will also contribute to the design, development, and maintenance of automated validation frameworks and utilities using Python, SQL, PySpark, Azure Data Factory, and Azure Synapse.

---

Key Responsibilities

1. QA & Solution Analysis

- Analyze business and technical requirements to understand data processing and validation needs.

- Participate in requirement analysis sessions and clarify functional and technical requirements with stakeholders.

- Review solution designs, data flows, mapping documents, interface specifications, and business rules.

- Validate that implemented solutions meet defined business and technical requirements.

- Identify gaps, inconsistencies, ambiguities, and potential data quality issues during requirement and solution analysis.

- Translate business requirements into detailed test scenarios, test cases, and validation conditions.

- Perform end-to-end validation of data processing workflows.

- Ensure data is accurately processed from source systems through intermediate layers to final outputs.

- Validate business rules and transformation logic implemented within data pipelines.

2. Test Planning & Execution

- Prepare comprehensive test strategies, test plans, test scenarios, and test cases for data-intensive applications.

- Execute functional, integration, regression, system, and data validation testing.

- Perform positive and negative testing for different data processing scenarios.

- Validate data pipelines across multiple environments, including staging, testing, and production.

- Identify test data requirements and prepare appropriate datasets for validation.

- Execute SQL queries to validate data processing and transformation results.

- Document test results, observations, defects, and validation evidence.

- Track testing progress and communicate status, risks, issues, and dependencies to stakeholders.

3. Data Validation & Reconciliation

- Perform detailed source-to-target data validation and reconciliation.

- Validate source, intermediate, staging, and output datasets.

- Perform record count validation between source and target systems.

- Verify data completeness, consistency, accuracy, and integrity.

- Validate data transformations against defined business rules.

- Perform field-level and record-level comparisons.

- Validate data types, formats, precision, scale, and null handling.

- Verify schema structure, layout, column names, and column sequence.

- Validate mandatory and optional fields.

- Identify missing, duplicate, truncated, or incorrectly transformed records.

- Analyze invalid records, rejected records, and exception datasets.

- Verify exception and reject-handling mechanisms.

- Compare production and staging data to identify discrepancies.

- Perform reconciliation between files, databases, and reporting layers.

- Validate data across different processing stages and identify the root cause of discrepancies.

4. File & Data Processing Validation

- Validate large-scale datasets across multiple file formats.

- Perform validation of:

 - CSV files

 - Delimited files

 - Fixed-width files

 - Excel files

 - Database tables

 - Structured and semi-structured datasets

- Validate file layouts, headers, delimiters, record formats, and column sequences.

- Verify file-level and record-level counts.

- Analyze source, intermediate, and final output files.

- Validate file-to-database and database-to-file reconciliation.

- Identify incomplete, corrupted, malformed, or invalid records.

- Verify data movement and transformation between different storage locations.

- Validate Azure-to-AWS file transfer processes.

- Ensure transferred files are complete and match the expected source datasets.

---

5. Defect Investigation & Root Cause Analysis

- Investigate data discrepancies and application/data pipeline defects.

- Perform detailed root cause analysis for data quality and validation failures.

- Analyze source data, transformation logic, pipeline execution, database records, and output datasets to identify defects.

- Collaborate with developers and data engineers to resolve identified issues.

- Reproduce defects and provide detailed technical evidence.

- Perform defect impact analysis.

- Conduct retesting and regression testing after defect resolution.

- Monitor recurring data quality issues and recommend preventive solutions.

- Maintain detailed defect documentation and validation results.

---

6. Python Development & Automation

- Develop Python scripts and utilities for data validation and reconciliation.

- Design, develop, and maintain reusable data validation frameworks.

- Automate repetitive data comparison and validation activities.

- Build automated utilities for:

 - Record count validation

 - Data completeness checks

 - Schema validation

 - Column sequence validation

 - Source-to-target comparison

 - Duplicate detection

 - Exception identification

 - Data quality checks

 - Automated reporting

- Develop Python-based validation and reporting utilities.

- Optimize Python scripts for processing large datasets.

- Maintain and enhance existing automation frameworks.

- Implement reusable validation components to improve testing efficiency and coverage.

---

7. SQL Development & Data Analysis

- Write complex SQL queries for data analysis and validation.

- Perform data extraction and comparison using SQL Server / SSMS.

- Validate source and target database records.

- Perform joins, aggregations, subqueries, CTEs, and analytical queries as required.

- Develop SQL queries to identify data mismatches, duplicates, missing records, and transformation issues.

- Validate database tables, schemas, columns, constraints, and relationships.

- Perform record count and reconciliation checks using SQL.

- Analyze SQL Server metrics databases.

- Validate data processing results against expected business rules.

- Troubleshoot data discrepancies using SQL queries.

---

8. PySpark & Large-Scale Data Processing

- Develop and execute PySpark notebooks for large-scale dataset processing and validation.

- Analyze large volumes of structured and semi-structured data.

- Perform data transformation and validation using PySpark.

- Compare large source and target datasets efficiently.

- Implement data quality and reconciliation checks using PySpark.

- Analyze exception, reject, and invalid datasets.

- Optimize data validation processes for large datasets.

- Work with Azure Synapse notebooks and data processing environments.

---

9. Azure Data Factory & Pipeline Testing

- Design and execute validation scenarios for Azure Data Factory (ADF) pipelines.

- Validate pipeline execution, data movement, transformations, and dependencies.

- Monitor pipeline runs and investigate failures.

- Validate source-to-target data movement through ADF.

- Develop and maintain test pipelines using Azure Data Factory.

- Verify pipeline parameters, triggers, activities, and execution results.

- Validate file ingestion and processing workflows.

- Perform end-to-end testing of data pipelines.

- Investigate pipeline-related data discrepancies and failures.

---

10. Azure Synapse Analytics

- Work with Azure Synapse Analytics for data validation and analysis.

- Develop and execute Synapse notebooks using PySpark.

- Validate datasets processed through Synapse pipelines and notebooks.

- Perform data quality and reconciliation checks within Synapse environments.

- Analyze large-scale datasets and processing results.

- Validate data movement between Azure storage, Synapse, databases, and reporting systems.

---

11. Azure Storage & Cosmos DB

- Validate data stored in Azure Storage Accounts and Containers.

- Verify file ingestion, processing, and output data.

- Perform file-level and content-level validation within Azure storage.

- Validate data processing workflows involving Azure Storage.

- Perform data validation in Azure Cosmos DB.

- Verify records, fields, formats, and data completeness within Cosmos DB.

- Investigate discrepancies between source files, Azure storage, databases, and Cosmos DB.

---

12. AWS S3 & Azure-to-AWS Validation

- Validate files stored in AWS S3.

- Perform source-to-target validation for files transferred between Azure and AWS.

- Verify file counts, file names, sizes, formats, and record counts.

- Compare source files with transferred S3 files.

- Validate data integrity after cloud-to-cloud file transfers.

- Investigate missing, incomplete, duplicate, or corrupted files.

- Support end-to-end validation of Azure-to-AWS data movement processes.

---

13. Metrics, Reporting & Power BI Validation

- Extract and validate source system metrics.

- Validate metrics stored in SQL Server databases.

- Perform reconciliation between source metrics, database metrics, and reporting outputs.

- Validate Power BI dashboards and reports against underlying source data.

- Verify report calculations, KPIs, measures, filters, and aggregations.

- Perform file-to-database-to-Power BI reconciliation.

- Validate data displayed in Power BI against SQL Server and source datasets.

- Identify discrepancies between backend data and dashboard results.

- Support reporting and analytics teams with data validation and troubleshooting.

---

14. Production Support & Job Monitoring

- Monitor scheduled data processing jobs and pipelines.

- Perform production validation and health checks.

- Analyze production failures and data discrepancies.

- Support incident investigation and resolution.

- Compare production and staging environments to identify differences.

- Validate production data after deployments and pipeline executions.

- Monitor ECG jobs and provide support for job execution and data processing issues.

- Perform post-production validation and reconciliation.

- Communicate critical production issues and risks to relevant stakeholders.

---

15. Agile Delivery & Stakeholder Collaboration

- Work effectively within an Agile/Scrum delivery environment.

- Participate in sprint planning, daily stand-ups, backlog refinement, sprint reviews, and retrospectives.

- Collaborate with Business Analysts, Developers, Data Engineers, DevOps teams, Product Owners, and other stakeholders.

- Provide timely updates on testing progress and issues.

- Participate in requirement clarification and solution discussions.

- Support release planning and production deployment activities.

- Track work items and defects using Rally.

- Ensure testing activities are aligned with sprint and release timelines.

---

Required Technical Skills

Mandatory Skills

- 5–6 years of experience in QA / Data Validation / Data Testing / Data Quality Engineering.

- Strong experience in SQL and data analysis.

- Hands-on experience with Python development and automation.

- Experience with PySpark and large-scale data processing.

- Strong experience with Azure Data Factory (ADF).

- Experience with Azure Synapse Analytics / Synapse Pipelines / Notebooks.

- Strong understanding of source-to-target data validation and reconciliation.

- Experience in data completeness, record count, schema, layout, and column validation.

- Experience in defect investigation and root cause analysis.

- Experience validating large datasets and multiple file formats.

- Experience with SQL Server / SSMS.

- Experience with Power BI dashboard/report validation.

- Strong understanding of data pipelines and ETL/ELT processes.

Cloud & Data Platform Experience

- Azure Data Factory

- Azure Synapse Analytics

- Azure Synapse Pipelines

- Azure Synapse Notebooks

- Azure Storage Accounts

- Azure Storage Containers

- Azure Cosmos DB

- Azure Privileged Identity Management (PIM)

- AWS S3

- Azure-to-AWS file transfer validation

---

Preferred Skills

- Experience developing automated data validation frameworks.

- Experience building automated reporting and reconciliation utilities.

- Knowledge of ETL/ELT testing methodologies.

- Experience working with very large datasets.

- Experience in production data validation and support.

- Knowledge of cloud-based data platforms.

- Experience with Power BI data reconciliation.

- Experience working in Agile environments.

- Experience with Rally or similar Agile project management tools.

- Familiarity with Microsoft Copilot and AI-assisted productivity/automation tools.

---

Key Responsibilities at a Glance

The successful candidate will be responsible for:

- Requirement analysis and clarification

- Business rule validation

- Test planning and execution

- Data quality and data validation

- Source-to-target reconciliation

- Record count and completeness validation

- Schema and layout validation

- Column sequence validation

- Exception and reject data analysis

- Production vs. staging comparison

- SQL-based data analysis

- Python automation

- PySpark development

- Azure Data Factory pipeline testing

- Azure Synapse validation

- Azure Storage validation

- Cosmos DB validation

- AWS S3 validation

- Azure-to-AWS file transfer validation

- Power BI dashboard validation

- SQL Server metrics validation

- Automated reporting

- Defect investigation and root cause analysis

- Production job monitoring and support

- Agile delivery and stakeholder collaboration

---

Candidate Profile

We are looking for a detail-oriented, analytical, and technically strong QA/Data Validation professional who can work independently on complex data validation assignments.

The candidate should be comfortable working with large datasets, writing SQL queries, developing Python automation, analyzing PySpark datasets, validating cloud-based data pipelines, and troubleshooting data discrepancies across multiple systems.

Strong communication and stakeholder management skills are essential, as the role requires regular collaboration with technical and business teams.

---

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field is preferred.

Experience

5–6 years of relevant professional experience in QA, Data Testing, Data Validation, ETL Testing, Data Quality, Data Engineering QA, or a similar role.

Location

Pan India

Employment Type

Full-Time

Keywords

QA Engineer, Data QA, Data Validation, Data Testing, ETL Testing, Data Quality, SQL, Python, PySpark, Azure Data Factory, ADF, Azure Synapse, Synapse Analytics, Synapse Pipelines, Azure Storage, Cosmos DB, AWS S3, Power BI, SQL Server, SSMS, Data Reconciliation, Source-to-Target Validation, Data Pipeline Testing, ETL QA, Automation Testing, Data Analytics, Root Cause Analysis, Agile, Rally, Cloud Data Testing, Data Engineering QA.

Job Description

We are looking for an experienced Data Engineer with strong expertise in Python, ETL, SQL, CI/CD, and DevOps to design, develop, and maintain scalable data pipelines and data processing solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Develop data processing solutions using Python.
• Write complex and optimized SQL queries, stored procedures, and data transformations.
• Build and maintain data ingestion and integration workflows.
• Implement data quality, validation, monitoring, and error-handling processes.
• Develop and maintain CI/CD pipelines for data engineering applications.
• Work with DevOps tools and practices for automated build, deployment, and infrastructure management.
• Collaborate with data analysts, data scientists, software engineers, and business teams.
• Optimize data pipelines for performance, reliability, and scalability.
• Troubleshoot production data issues and ensure timely resolution.
• Follow best practices for version control, code quality, testing, and deployment.

Mandatory Skills

• Python
• ETL
• SQL
• CI/CD
• DevOps
• Git / Version Control
• Strong problem-solving and debugging skills

Job Description:

Experience: 10+ Years

Job Summary

We are looking for an experienced Azure Fabric Data Architect to lead the design and implementation of an enterprise data platform on Microsoft Fabric. The role involves architecting scalable data solutions, defining data governance, and enabling AI-driven analytics for a global financial services client.

Key Responsibilities

• Design end-to-end data architecture using Microsoft Fabric.
• Build enterprise Lakehouse, Data Warehouse, and OneLake solutions.
• Define data ingestion, ETL/ELT, governance, security, and performance strategies.
• Lead architecture for AI-powered analytics, AI Agents, and enterprise chatbots using Azure AI services.
• Work with business stakeholders to translate requirements into technical solutions.
• Mentor engineering teams and provide technical leadership.

Required Skills

• Microsoft Fabric (Data Factory, Lakehouse, Data Warehouse, OneLake)
• Azure Data Engineering
• Power BI
• Azure AI Services / Azure OpenAI
• Data Architecture & Data Modeling
• SQL, Python
• Azure DevOps, CI/CD
• Strong stakeholder management and solution design experience
• 
  

Preferred: Experience in Capital Markets or Financial Services and Microsoft Azure/Fabric certifications.

NOTE: One technical round is mandatory to be taken F2F from office.

CAP-190 : Senior Data Analyst

📍Location : Bangalore / Hyderabad / Chennai / Pune / NCR

🧠Experience : 7 - 15 Years

🆔Job Code : CAP-190

🏢Work Type : Hybrid - 3 Days in a Week

About the Client (CODE: CAP)

CAP operates at the forefront of the financial services industry, providing consulting, technology, and digital transformation solutions to leading organizations worldwide. With a focus on innovation, CAP empowers clients to navigate complex regulatory landscapes, optimize operations, and drive business growth. The company fosters a collaborative and agile culture, encouraging continuous learning and excellence.

Key Responsibilities

● Define and obtain source data required to deliver insights and use cases.

● Determine data mapping and join multiple data sets across various sources.

● Develop methods to highlight and report data inconsistencies for user review.

● Propose and assist with suitable data migration sets for stakeholders.

● Support teams in processing data migration sets and coordinating migration activities.

● Plan, track, and coordinate the data migration team and migration run-book.

● Collaborate with stakeholders to avoid negative customer and business impacts.

● Ensure robust communication and escalation mechanisms across project portfolios.

● Implement strategic solutions and avoid short-term workarounds.

● Maintain strong control and compliance standards in data handling.

Required Skills

● Minimum 7+ years of experience as a Data Analyst, preferably in financial services.

● Strong expertise in Pyspark, Python, and SQL.

● Experience with big data programs and data models in banking or financial markets.

● Ability to write SQL queries and navigate databases such as Hive, CMD, Putty, and Note++.

● Excellent analytical skills and commercial acumen.

● Strong verbal and written communication skills.

● Proven ability to manage multiple priorities and deliver within tight deadlines.

● Business analysis skills, including defining and understanding requirements.

● Familiarity with SDLC, Agile processes, and a bias towards TDD.

● Attention to detail and a proactive, problem-solving mindset.

Nice to Have

● Knowledge and experience in Data Quality & Governance.

● Working experience with Spark Scala or Java for Spark.

● Proven track record of managing small, delivery-focused data teams (for senior roles).

● Experience with market data vendors and domains such as Party/Client, Trade, Settlements, Payments, Instrument and Pricing, Market and/or Credit Risk.

Why Join CAP (Code Name)

Join CAP to work on impactful data initiatives within the financial services sector, tackling complex technical challenges and driving meaningful business outcomes. You'll collaborate with talented professionals in a dynamic, agile environment that values innovation and continuous improvement. CAP offers opportunities for professional growth, skill development, and the chance to contribute to high-visibility projects that shape the future of financial technology.

About the Employment Model

Direct Hire (Client Payroll) : For this role, you’ll be hired directly by the client and be part of their internal team. Straatix supports the hiring process, but your employment, payroll, and benefits are all managed by the client.

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.

Must-Have Skills :

• 14 to 20 years of Data Engineering experience
• Databricks & Apache Spark / PySpark
• Python & SQL
• AWS Cloud
• Lakehouse Architecture
• ETL / ELT & Distributed Data Processing
• Batch & Streaming Pipelines
• Data Pipeline Optimization & Data Modeling
• CDC & Incremental Processing
• Git, CI/CD & Testing
• Data Quality, Monitoring & Observability
• Technical Leadership & Stakeholder Management

Key Responsibilities :

• Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
• Own data products from design through production.
• Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
• Optimize pipelines for performance, scalability, reliability, and cost.
• Design scalable data architectures and data models.
• Implement data quality, monitoring, lineage, and CI/CD practices.
• Lead technical discussions and mentor engineering teams.
• Collaborate with business stakeholders, architects, product owners, and engineering teams.
• Remain hands-on while providing technical leadership.

Ideal Candidate :

A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.

🔴 Super Urgent : Only Bangalore-based immediate joiners.

Location – Hyderabad (Hybrid)

Work Experience – 5 to 7 years

CTC – upto 20 LPA

Roles & Responsibilities:

· We are looking for a Senior Data Engineering who will be majorly responsible for designing, building and maintaining ETL/ ELT pipelines.

· Integration of data from multiple sources or vendors to provide the holistic insights from data.

· You are expected to build and manage Data warehouse solutions, designing data models, creating ETL processes, implementing data quality mechanisms etc.

· Performs EDA (exploratory data analysis) required to troubleshoot data related issues and assist in the resolution of data issues.

· Should have experience in client interaction.

· Experience in mentoring juniors and providing required guidance.

Required Technical Skills

· Extensive hands on experience in Python, Pyspark, SQL, Dataiku.

· Strong experience in Data Warehouse, ETL, Data Modelling, building ETL Pipelines, Snowflake database.

· Working knowledge in Databricks, Redshift, ADF etc.

· Hands-on experience in cloud services like Azure, AWS- S3, Glue, Lambda, CloudWatch, Athena.

· Sound knowledge in end-to-end Data management, Data ops, quality and data governance.

· Familiar with SFDC, Waterfall/ Agile methodology.

· Strong domain knowledge in Pharma domain/ life sciences commercial data operations.

Qualifications

· Bachelor’s or master’s Engineering/ MCA or equivalent degree.

· 5-7 years of relevant industry experience as Data Engineer.

· Experience working on Pharma syndicated data such as IQVIA, Veeva, Symphony; Claims, CRM, Sales etc.

· High motivation, good work ethic, maturity, self-organized and personal initiative.

· Ability to work collaboratively and providing the support to the team.

· Excellent written and verbal communication skills.

· Strong analytical and problem-solving skills. 

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.

Define and obtain source data required to deliver insights and use cases.

● Determine data mapping and join multiple data sets across various sources.

● Develop methods to highlight and report data inconsistencies for user review.

● Propose and assist with suitable data migration sets for stakeholders.

● Support teams in processing data migration sets and coordinating migration activities.

● Plan, track, and coordinate the data migration team and migration run-book.

● Collaborate with stakeholders to avoid negative customer and business impacts.

● Ensure robust communication and escalation mechanisms across project portfolios.

● Implement strategic solutions and avoid short-term workarounds.

● Maintain strong control and compliance standards in data handling.

Required Skills

● Minimum 7+ years of experience as a Data Analyst, preferably in financial services.

● Strong expertise in Pyspark, Python, and SQL.

● Experience with big data programs and data models in banking or financial markets.

● Ability to write SQL queries and navigate databases such as Hive, CMD, Putty, and Note++.

● Excellent analytical skills and commercial acumen.

● Strong verbal and written communication skills.

● Proven ability to manage multiple priorities and deliver within tight deadlines.

● Business analysis skills, including defining and understanding requirements.

● Familiarity with SDLC, Agile processes, and a bias towards TDD.

● Attention to detail and a proactive, problem-solving mindset.

Nice to Have

● Knowledge and experience in Data Quality & Governance.

● Working experience with Spark Scala or Java for Spark.

● Proven track record of managing small, delivery-focused data teams (for senior roles).

● Experience with market data vendors and domains such as Party/Client, Trade, Settlements, Payments, Instrument and Pricing, Market and/or Credit Risk.

Design, develop, and maintain ETL pipelines involving large-scale data.

Develop data processing and analytics applications primarily using PySpark and Python.

Build scalable and distributed data processing solutions using Apache Spark.

Develop and deploy data applications on AWS cloud.

Work with AWS services related to storage, compute, ETL, data warehousing, analytics, and streaming.

Implement distributed storage and processing solutions capable of handling high-volume datasets.

Design data processing applications with a focus on performance, scalability, reliability, and optimization.

Work with both SQL and NoSQL databases for data storage, processing, and analytics.

Write, optimize, and analyze SQL, HQL, and NoSQL queries.

Troubleshoot data pipeline and processing issues and ensure data quality and reliability.

Collaborate with data engineers, analysts, architects, and other technical teams to deliver data-driven solutions.

Job Summary

We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.

Technical Skills

• Strong hands-on experience in Python and PySpark development.
• Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
• Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Power BI Data Modeling and Semantic Layer development.
• Proficiency in DAX (Data Analysis Expressions).
• Experience designing and managing Semantic Models in Power BI.
• Strong SQL skills and experience working with large datasets.
• Knowledge of data warehousing concepts and best practices.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Exposure to modern data platforms like Databricks.
• Understanding of data governance and data quality frameworks.

Roles & Responsibilities

• Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

2Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

4Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

5Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

6Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

8Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Skills Referential (Required knowledge, skills and abilities)

Technical Skills:

Python

Pyspark

SQL

ETL Aws, Azure, gcp

Key Responsibilities  

• Design, build, and optimize scalable data pipelines for AI/ML applications.
• Develop, train, evaluate, and deploy Machine Learning and Deep Learning models.
• Build production-ready LLM applications using Retrieval-Augmented Generation (RAG), prompt engineering, and vector databases.
• Fine-tune open-source and foundation models using domain-specific datasets.
• Develop and maintain end-to-end MLOps pipelines for model deployment, monitoring, and lifecycle management.
• Perform data preprocessing, feature engineering, exploratory data analysis (EDA), and model evaluation.
• Develop APIs and AI services for production deployment.
• Collaborate with cross-functional teams to deliver scalable AI-driven solutions.
• Monitor model performance, troubleshoot production issues, and maintain technical documentation.

Required Skills  

Mandatory  

• 1–3 years of experience in Data Science, Data Engineering, or AI/ML development.
• Strong programming skills in Python and SQL.
• Hands-on experience with Machine Learning frameworks such as PyTorch, TensorFlow, or Scikit-learn.
• Experience building LLM-powered applications using RAG, Prompt Engineering, and Embeddings.
• Hands-on experience with LangChain, LlamaIndex, CrewAI, or n8n for LLM orchestration and AI workflow automation.
• Experience in LLM fine-tuning and working with Hugging Face models.
• Knowledge of MLOps concepts including model deployment, monitoring, versioning, and CI/CD.
• Experience with Git, REST APIs, Linux environments, and data processing libraries.

Preferred  

• Experience with vector databases such as Pinecone, Chroma, Milvus, or Weaviate.
• Familiarity with Docker, Kubernetes, and MLflow.
• Exposure to Apache Spark or Airflow for data engineering workflows.
• Experience with cloud platforms (AWS, Azure, or GCP).

Primary Technology Stack  

• Languages & Data Processing: Python, SQL, Pandas, NumPy, Apache Spark
• AI & Machine Learning: PyTorch, TensorFlow, Scikit-learn
• Application Frameworks: LangChain, LlamaIndex, CrewAI, n8n
• Core Methodologies: Retrieval-Augmented Generation (RAG), Model Fine-Tuning, Prompt Engineering, Embeddings
• Models & Infrastructure: OpenAI APIs, Hugging Face Ecosystem, Embedding Models
• Vector Databases: Pinecone, Chroma, Milvus, Weaviate
• Databases: PostgreSQL, MongoDB
• MLOps & DevOps: Docker, Kubernetes, MLflow, CI/CD, Git
• Cloud Platforms: AWS, Azure, GCP

Experience: 1–3 Years

Domain: Data Science | Data Engineering | Machine Learning | Generative AI | MLOps

Job Title: Data Engineer – PySpark | Oracle | GCP

Experience: 5–7 Years

Location: Hyderabad

Notice Period: Immediate Joiners Preferred

Job Summary

We are seeking an experienced Data Engineer with strong expertise in PySpark, Oracle, and Google Cloud Platform (GCP) to design, develop, and optimize scalable data pipelines. The ideal candidate should have hands-on experience in ETL development, data integration, and cloud-based data engineering solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/data pipelines using PySpark.
• Extract, transform, and load data from Oracle databases into GCP environments.
• Build and optimize batch data processing workflows for high performance and reliability.
• Develop data engineering solutions using GCP services.
• Ensure data quality through validation, monitoring, and troubleshooting.
• Optimize SQL queries and ETL jobs for performance and scalability.

Required Skills

• 5–7 years of experience as a Data Engineer.
• Strong hands-on experience with PySpark.
• Solid experience with Oracle Database and advanced SQL.
• Hands-on experience with Google Cloud Platform (GCP).
• Strong understanding of ETL processes and data warehousing concepts.

Work Location: Hyderabad

Notice Period: Immediate Joiners Preferred

Job Summary

We are seeking a skilled Azure Data Engineer with hands-on experience in Azure Data Services, Azure Databricks, Python, PySpark, and SQL. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions to support business intelligence, analytics, and reporting requirements.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Build and optimize data processing workflows using Python and SQL.
• Develop and manage data ingestion pipelines from multiple structured and unstructured data sources.
• Work with Azure Data Factory (ADF) to orchestrate and schedule data pipelines.
• Implement data transformation and cleansing logic using PySpark.
• Optimize SQL queries and Spark jobs for performance and scalability.
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and governance across the data platform.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Follow coding standards, version control, and CI/CD best practices.

Required Skills

• Strong experience with Microsoft Azure cloud services.
• Hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Expertise in PySpark for large-scale data processing.
• Strong SQL skills, including query optimization and performance tuning.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake, Spark SQL, and Databricks notebooks.
• Experience with Git or Azure DevOps for source code management.
• Understanding of data warehousing concepts and ETL/ELT processes.

Preferred Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Delta Live Tables (DLT).
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Familiarity with Unity Catalog and data governance.
• Exposure to CI/CD pipelines and infrastructure-as-code.
• Experience working in Agile/Scrum environments.

Job Description: Software Engineer / Core Engineer - Java Full Stack (GCP)

Location: Sholinganallur, Chennai - Work From Office (WFO)only chennai candidate

Experience: 6+ years of IT experience with 4+ years in Software Development [Ideal: 6-10 years]

Education: Bachelor's Degree [B.E./B.Tech or equivalent]

Employment Type: Full Time

About the Role:

This is a Java Full Stack Developer role with a strong GCP requirement. We are looking for a hands-on Software Engineer / Core Engineer to design, build, and scale cloud-native full-stack applications. Java, Spring Boot, Angular, and GCP are mandatory skills for this position.

Mandatory Skills:

• Full Stack Java Development
• Java
• Spring Boot
• Angular
• Google Cloud Platform (GCP)
• REST APIs & Microservices
• Agile Software Development

Preferred Skills:

• Data & Analytics on GCP: BigQuery, Dataflow, Dataproc, Data Fusion, Cloud SQL, PostgreSQL
• Big Data & Orchestration: Apache Airflow, PySpark, Python
• IaC & DevOps: Terraform, Tekton, Maven, Gradle, SonarQube
• Frameworks & Tools: Spring Cloud, APIGEE, Swagger/OpenAPI
• UI/UX Collaboration

Roles & Responsibilities:

1. Design, develop, test, and deploy robust full-stack applications using Java, Spring Boot, and Angular.
2. Build and maintain scalable REST APIs and microservices-based architecture.
3. Develop responsive, high-performance Angular-based UI applications and collaborate with UI/UX teams for seamless user experience.
4. Design software architecture and end-to-end cloud-native solutions on Google Cloud Platform.
5. Work on GCP deployments, infrastructure setup, and cloud migration initiatives.
6. Follow and advocate Agile, TDD, CI/CD, and DevOps best practices throughout the SDLC.
7. Collaborate effectively with Product Owners, Architects, UI/UX teams, and cross-functional stakeholders.
8. Ensure application security, performance optimization, scalability, and high code quality with SonarQube, code reviews, and testing.

Ideal Candidate Profile:

• 6 to 10 years of total IT experience with strong hands-on development background.
• Strong expertise in Java, Spring Boot, Angular, and GCP - must have built production-grade applications.
• Hands-on experience with Microservices, REST APIs, CI/CD pipelines, and Cloud Deployment.
• Experience with Cloud SQL, PostgreSQL, and GCP data services is a plus.
• Good understanding of software architecture, distributed systems, and performance tuning.
• Excellent communication skills and proven experience working in Agile teams.
• Willingness to work from Sholinganallur, Chennai office (WFO).

What We Offer:

• Opportunity to work on cloud-native, large-scale platforms
• Agile and collaborative work culture
• Exposure to modern GCP data stack and DevOps ecosystem

• Job Title: Lead Data Engineer
• Location: Marathahalli, Bangalore 
• Job Mode: Hybrid

Key Responsibilities:

·   Create and maintain data sources inside our data lake on Azure

·   Assemble and analyze large, complex data sets that meet both functional and non-functional business requirements

·   Identify, design, and implement internal process improvements: automating manual processes; optimizing data delivery; re-designing infrastructure for greater scalability.

·   Work with stakeholders including the Executive, Tech, Product, Data and Design teams to optimize data driven decisions

·   Keep our data separated and secure by understanding and applying best practice access policies

·   Create tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader

·   Ensure data integrity with thorough data quality test coverage and data validation

·   Improve, create and implement CI/CD processes and tooling inside the data lake

Required Qualifications

-     Join immediately

-     Databricks

-     Databricks hands on experience (esp. Pipelines PySpark & SQL, Delta Lake, Unity Catalog, DLT*)

-     Ingesting & transforming data using Databricks - Min 3 years continuous hands-on

-     Advanced SQL and Python knowledge

-     Data Lake housing and Modelling experience (e.g. medallion architecture & query performance optimization)

-     Experience with backend systems ERP, CRM (e.g. SAP, Oracle, Salesforce)

-     Exposure of working with Azure (ADLS, ADF)

-     Good Communication skills

Job Summary

We are seeking a highly skilled GCP Data Engineer with strong expertise in Google Cloud Platform (GCP), Python, ETL, and modern data engineering technologies. The ideal candidate should have hands-on experience designing and building scalable data pipelines using BigQuery, Dataflow, Pub/Sub, Airflow, and modern data lake technologies such as Apache Iceberg or Delta Lake.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines on Google Cloud Platform.
• Build and optimize data processing workflows using Python and Google Cloud Dataflow (Apache Beam).
• Develop and manage large-scale analytical data models in BigQuery.
• Implement event-driven data ingestion using Google Cloud Pub/Sub.
• Create, schedule, and monitor workflows using Apache Airflow and Autosys.
• Design and implement modern data lake architectures using Apache Iceberg or Delta Lake.
• Optimize query performance, storage, and compute costs in GCP.
• Ensure data quality, governance, security, and compliance across data platforms.
• Collaborate with Data Scientists, Analysts, and Application teams to deliver scalable data solutions.
• Troubleshoot production issues and continuously improve pipeline reliability and performance.

Mandatory Skills

• Strong hands-on experience with Google Cloud Platform (GCP).
• Proficiency in Python programming.
• Experience in designing and implementing ETL/ELT pipelines.
• Strong knowledge of BigQuery.
• Experience with Google Cloud Dataflow (Apache Beam).
• Experience with Google Cloud Pub/Sub.
• Hands-on experience with Apache Airflow.
• Experience in job scheduling using Autosys.
• Experience with modern table formats such as Apache Iceberg or Delta Lake.
• Strong SQL and data modeling skills.

Preferred Skills

• Experience with Cloud Storage, Dataproc, Cloud Composer, and Cloud Functions.
• Knowledge of CI/CD pipelines and DevOps practices.
• Experience with Docker and Kubernetes.
• Familiarity with Git and Agile/Scrum methodologies.
• Knowledge of data warehousing and dimensional modeling.
• Exposure to streaming and real-time data processing.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 4–8+ years of experience in Data Engineering with hands-on expertise in GCP technologies.

Required Experience

• Strong experience in developing enterprise-grade data pipelines using Python and GCP.
• Hands-on experience with BigQuery, Dataflow, Pub/Sub, and Airflow.
• Experience scheduling and monitoring batch workflows using Autosys.
• Experience implementing modern data lake architectures using Apache Iceberg or Delta Lake.
• Strong understanding of ETL best practices, performance tuning, and data optimization.
• Excellent analytical, troubleshooting, and problem-solving skills.

Mandatory Skills

• Google Cloud Platform (GCP)
• Python
• ETL
• BigQuery
• Autosys
• Apache Airflow
• Google Cloud Pub/Sub
• Google Cloud Dataflow (Apache Beam)
• Apache Iceberg / Delta Lake
• SQL & Data Modeling

Job Summary

We are looking for an experienced AI Data Architect to design and build an enterprise AI-ready data platform that serves as the single source of truth for AI applications, including RAG, Agentic AI, Conversational AI, ML models, and analytics.

Key Responsibilities

• Design enterprise AI data platform and Lakehouse architecture.
• Build batch & real-time data pipelines.
• Develop semantic models, knowledge graphs, and vector databases.
• Architect RAG and LLMOps infrastructure.
• Implement data governance, security, and AI observability.
• Modernize legacy data platforms to cloud-native architectures.

Required Skills

• Python, SQL, PySpark
• Databricks, Delta Lake, Snowflake
• Kafka, Spark Structured Streaming
• AWS / Azure
• LangChain, LlamaIndex
• OpenAI, Claude, Bedrock
• Pinecone, FAISS, ChromaDB, Neo4j
• MLflow, Docker, Kubernetes, Terraform
• FastAPI, GitHub Actions, Jenkins
• Data Governance, RBAC, CI/CD

Requirements

• 12+ years in Data Engineering/Data Architecture.
• Experience with AI/ML, RAG, LLMOps, and enterprise AI platforms.
• Strong expertise in Lakehouse, Data Mesh, Cloud, and Vector Databases.
• Hands-on experience with enterprise-scale AI data architecture and governance. 

We are looking for a hands-on Data Engineer to help build and manage our data platform for reporting, analytics, and future data science use cases.

The role will involve working with SQL, Python, PySpark, AWS, ETL/ELT pipelines, data warehouses, and BI/reporting tools. Our architecture may use technologies such as ClickHouse, Redshift, AWS Glue, Airflow, Step Functions, Lambda, S3, and CDC-based replication tools based on scale, cost, and operational needs.

Key Responsibilities

·     Design, build, and maintain scalable ETL/ELT data pipelines from multiple databases, applications, and external systems.

·     Build raw, cleaned, and business-ready data layers to support reporting, analytics, and future data science use cases.

·     Write efficient SQL, Python, and PySpark jobs for data ingestion, transformation, validation, and processing.

·     Implement workflow orchestration using Apache Airflow, AWS Glue, Step Functions, or similar tools.

·     Work with data warehouses such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.

·     Support cross-service reporting as the architecture moves towards independent microservice databases.

·     Build reusable reporting tables, aggregates, summaries, and basic data marts.

·     Monitor, troubleshoot, and optimize data pipelines, warehouse queries, and processing jobs.

·     Implement data quality checks for freshness, completeness, consistency, duplicates, and reconciliation.

·     Support BI/reporting needs through tools such as Power BI, Metabase, Superset, Redash, or similar.

·     Apply data governance, access control, security, and PII-handling best practices.

·     Collaborate with engineering, DevOps, product, business, finance, risk, and support teams.

Required Skills

·     Strong expertise in SQL for joins, aggregations, window functions, query optimization, and analytical reporting.

·     Hands-on experience with Python for data processing, automation, validation, and scripting.

·     Working experience with PySpark / Apache Spark for processing large datasets.

·     Good understanding of ETL/ELT pipelines, data warehousing, and data modelling concepts.

·     Experience with workflow orchestration using Apache Airflow, AWS Step Functions, AWS Glue, or similar tools.

·     Experience with AWS data services such as S3, Glue, Lambda, Step Functions, Redshift, DMS, CloudWatch, or similar.

·     Experience with any data warehouse such as ClickHouse, Redshift, Snowflake, BigQuery, or similar.

·     Understanding of relational databases, preferably PostgreSQL.

·     Ability to debug data mismatches, failed pipelines, slow queries, and data quality issues.

·     Exposure to BI tools such as Power BI, Metabase, Superset, Redash, or similar.

Good ownership, problem-solving, communication, and collaboration skills.