141 PySpark Jobs in Bangalore (Bengaluru) | PySpark Job openings in Bangalore (Bengaluru)

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python and PySpark.
• Work extensively with Databricks and Delta Lake, including Delta tables, MERGE operations, schema evolution, Time Travel, and data optimization.
• Develop, schedule, and monitor Apache Airflow DAGs for workflow orchestration and dependency management.
• Work with Kubernetes for containerized application deployment, orchestration, monitoring, and troubleshooting.
• Develop reusable and optimized Python code following software engineering best practices.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, error handling, logging, and monitoring mechanisms.
• Use Git for version control, branching, pull requests, code reviews, and collaborative development.
• Troubleshoot pipeline failures, investigate data issues, and implement reliable solutions.
• Collaborate with cross-functional teams to understand business requirements and deliver efficient data solutions.

Required Technical Skills

• Python: Strong programming skills, including data manipulation, exception handling, modular programming, and debugging.
• PySpark: Hands-on experience with Spark DataFrames, Spark SQL, joins, transformations, partitioning, and performance tuning.
• Databricks: Strong understanding of Databricks architecture, notebooks, workflows, compute management, and job execution.
• Delta Lake: Experience with Delta tables, ACID transactions, MERGE/UPSERT, schema enforcement, schema evolution, VACUUM, and Time Travel.
• Apache Airflow: Experience developing DAGs, managing task dependencies, configuring retries, scheduling workflows, and troubleshooting failures.
• Kubernetes: Practical knowledge of pods, deployments, services, ConfigMaps, Secrets, resource management, and troubleshooting containerized workloads.
• Git: Proficiency in branching strategies, pull requests, merging, and resolving conflicts.
• SQL: Strong SQL skills, including complex joins, CTEs, window functions, and query optimization.

Good to Have

• Experience with Docker and containerization best practices.
• Experience with Databricks Unity Catalog and data governance.
• Knowledge of CI/CD pipelines and automated deployments.
• Experience deploying and managing Airflow workloads on Kubernetes.
• Familiarity with Kubernetes monitoring, autoscaling, and resource optimization.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Understanding of data lakehouse architecture and data modeling.
• Experience with batch and incremental data processing.
• Familiarity with data quality frameworks and automated testing.

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently design, develop, deploy, and troubleshoot data pipelines.
• Good understanding of distributed data processing and container orchestration.
• Experience working in an Agile development environment.
• Strong communication and collaboration skills.
• 
  

Job Details:

• Role: Python Developer
• Location: Bangalore
• Experience: 7+ Years
• Required Skills:
• Python Development
• MySQL
• REST API Development
• Django Framework
• PySpark

INTERESTED CANDIDATE SEND YOUR RESUME IN MAIL OR WHATSAPP :

Email: revathi.r @vysystems. com

WhatsApp / Call: 6 3 8 5 0 7 6 1 4 2

Job Summary

We are looking for a skilled Python Backend Developer with strong hands-on programming experience and a good understanding of SQL, databases, REST APIs, and backend development. The ideal candidate should have experience developing, maintaining, and optimizing backend applications, along with strong debugging and problem-solving skills.

Key Responsibilities

• Develop, maintain, and enhance backend applications using Python.
• Write, optimize, and manage SQL queries, including joins, subqueries, and complex queries.
• Work with relational databases such as PostgreSQL, MySQL, or SQL Server.
• Develop and integrate RESTful APIs for backend services.
• Perform debugging, troubleshooting, and performance optimization.
• Write unit tests and ensure code quality and application reliability.
• Use Git for version control and collaborative development.
• Collaborate with frontend developers, QA teams, and other stakeholders.
• Support data processing and ETL-related activities.

Required Skills

• Strong hands-on experience in Python programming.
• Good knowledge of SQL, joins, subqueries, and query optimization.
• Experience working with PostgreSQL, MySQL, or SQL Server.
• Knowledge of REST APIs and backend development concepts.
• Understanding of debugging, testing, and troubleshooting.
• Hands-on experience with Git and version control.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Python frameworks such as Django, Flask, or FastAPI.
• Knowledge of data processing and ETL pipelines.
• Familiarity with cloud platforms such as AWS or Microsoft Azure.
• Understanding of application deployment and backend architecture.

​Kindly fill the Details :-

GCP Data engineer

Exp 9+yrs

Bangalore/Hyderabad

Lead exp+ Python + pyspark + sql or plsql

immediate to 15days joiners

Call or WhatsApp: 6 3 8 5 0 7 6 1 4 2

GCP Data engineer (L1 virtual, L2 or DM F2F )

GCP + Python + Pyspark + sql + Lead

8+yrs

Bangalore/hyderabad location ( relocation profiles also from Chennai pune location)

immediate to 15days

🚀 Hiring: Senior GCP Data Engineer

We are looking for experienced Senior Data Engineers with strong hands-on expertise in Google Cloud Platform (GCP), Python, PySpark, and SQL. Candidates with team lead experience will be preferred.

📌 Job Details:

• Role: Senior GCP Data Engineer
• Experience: 8+ Years
• Budget: Up to ₹30 LPA
• Location: Bangalore / Hyderabad
• Relocation: Candidates from Chennai and Pune willing to relocate are welcome.
• Notice Period: Immediate to 15 days
• Interview Process: L1 – Virtual | L2 / DM – Face-to-Face (F2F)

💻 Required Technical Skills:

• Strong hands-on experience with GCP data engineering.
• Excellent proficiency in Python, PySpark, and SQL.
• Experience in designing, developing, and optimizing data pipelines.
• Good understanding of ETL/ELT processes and large-scale data processing.
• Experience working with cloud-based data platforms and data warehousing.
• Strong analytical, problem-solving, and communication skills.

⭐ Preferred Skills:

• Previous experience as a Technical Lead or leading a data engineering team.
• Knowledge of GCP services such as BigQuery, Dataproc, Dataflow, and Cloud Storage.

Job Title : Senior Data Engineer – GCP & Apache Spark

Job Details :

• Number of Positions : 3
• Employment Type : Contract-to-Hire (C2H)
• Experience Required : 5 to 8 Years
• Location : Manyata Tech Park, Bengaluru
• Work Mode : Hybrid – 3 Days in Office per Week
• Preferred Notice Period : Immediate to 15–20 Days
• Project Duration : Initially 6–8 months, with a strong possibility of extension for long-term projects
• Relocation : Candidates from outside Bengaluru can apply if they are willing to relocate.

Role Overview :

We are looking for experienced Senior Data Engineers with strong expertise in Google Cloud Platform (GCP), Apache Spark, BigQuery, and SQL to design, develop, and maintain scalable data pipelines and distributed data processing solutions.

The ideal candidate should have hands-on experience with cloud-based data engineering, end-to-end pipeline architecture, system design, performance optimization, debugging, and CI/CD practices.

Mandatory skills :

Apache Spark (PySpark / Scala), SQL, GCP, BigQuery, GCP Dataproc, Data Pipelines, System Design, CI/CD, Debugging & Problem-Solving.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and/or Scala.
• Build and optimize batch and distributed data processing workflows on GCP.
• Work extensively with BigQuery for data transformation, validation, analytics, and query optimization.
• Develop and maintain data workflows using GCP services, including Dataproc, BigQuery, and Cloud Storage.
• Design end-to-end data pipeline architectures with a focus on scalability, reliability, and performance.
• Write and optimize complex SQL queries for large-scale datasets.
• Troubleshoot and resolve data processing issues, pipeline failures, and performance bottlenecks.
• Implement CI/CD practices for data engineering workflows, testing, and deployments.
• Monitor pipeline performance and identify opportunities for continuous improvement.
• Collaborate with cross-functional teams to understand requirements and deliver reliable data solutions.
• Participate in code reviews, testing, production deployments, and production support.

Mandatory Skills :

• Strong hands-on experience with Apache Spark, including PySpark and/or Scala.
• Strong SQL skills and experience working with large-scale datasets.
• Hands-on experience with Google Cloud Platform (GCP).
• Practical experience with BigQuery and data warehousing concepts.
• Working knowledge of GCP Dataproc.
• Strong understanding of data pipelines, end-to-end workflows, and system design.
• Practical experience with CI/CD and deployment processes.
• Excellent debugging, troubleshooting, and problem-solving skills.

Preferred Skills :

• Experience with Apache Airflow, Google Cloud Composer, or similar orchestration tools.
• Knowledge of Kafka or other streaming and messaging technologies.
• Proficiency in Python for data engineering and automation.
• Experience with Git-based development workflows.
• Knowledge of Linux/Unix environments and shell scripting.
• Familiarity with monitoring, logging, data quality, and governance practices.

Required Qualifications :

• 5 to 8 years of experience in Data Engineering, Big Data Engineering, or a related field.
• Proven experience developing and maintaining scalable data pipelines.
• Strong understanding of distributed data processing and cloud-based data architecture.
• Ability to troubleshoot complex technical issues and optimize pipeline performance.
• Strong communication, collaboration, and ownership skills.

Interview Process :

Round 1 : AI-Based Interview

• Candidates must complete the AI interview as soon as possible after receiving the invitation.

Round 2 : Virtual Interview Drive

• Interviews are planned for the upcoming drive; exact dates will be communicated separately.

Round 3 : Online Client Interview

• Shortlisted candidates will attend the online interview.
• Candidates must be available throughout the day on the scheduled interview date.

1st virtual , 2nd round F2F -

Python pyspark, SQL, data engineer

5+yrs

Bang/hyderabad

immediate to 15days.

Job Description

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.

Key Responsibilities

• Design, develop, and maintain ETL processes and data pipelines.
• Work with Hadoop and PySpark to process and transform large volumes of data.
• Develop and optimize Python scripts for data processing and automation.
• Write complex SQL queries for data extraction, transformation, and validation.
• Perform data cleansing, integration, transformation, and validation.
• Work with data warehousing concepts and large-scale data processing systems.
• Optimize ETL workflows for performance, scalability, and reliability.
• Troubleshoot data pipeline failures and resolve data quality issues.
• Collaborate with data engineers, analysts, and other technical teams.
• Follow data governance, quality, and development best practices.

Required Skills

• Strong experience in ETL Development – Mandatory
• Hands-on experience with Hadoop – Mandatory
• Strong experience in PySpark – Mandatory
• Good programming skills in Python – Mandatory
• Strong knowledge of SQL – Mandatory
• Experience in Data Processing and Transformation – Mandatory
• Knowledge of Data Warehousing – Mandatory
• Experience with Big Data Technologies – Mandatory

Opportunity Details

Role: ETL Developer – Hadoop & PySpark

Experience: 6–10 Years

Location: Bangalore

Mandatory Skills

ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies

Hiring: ETL Developer – Hadoop & PySpark

Location: Bangalore

Experience: 6–10 Years

Budget: 25–30 LPA

Work Mode: Weekdays

Job Description

We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.

Key Responsibilities

• Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
• Develop data transformation workflows using PySpark and Python.
• Work with Hadoop and other big data technologies to process and manage large datasets.
• Write complex SQL queries for data extraction, transformation, validation, and analysis.
• Implement data cleansing, data quality checks, and data validation processes.
• Develop and optimize data integration workflows for data warehousing solutions.
• Troubleshoot ETL failures, performance issues, and data discrepancies.
• Optimize PySpark jobs and SQL queries for performance and scalability.
• Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
• Ensure data accuracy, consistency, and adherence to established data governance standards.

Mandatory Skills

• Strong hands-on experience in ETL development.
• Good experience with Hadoop and big data technologies.
• Proficiency in PySpark and Python programming.
• Strong SQL skills, including complex queries and data manipulation.
• Experience in data processing, data transformation, and data integration.
• Understanding of data warehousing concepts and architecture.
• Experience in developing, debugging, and optimizing ETL pipelines.

Required Experience

• 6–10 years of relevant experience in ETL development and data engineering.
• Practical experience handling large datasets using Hadoop and PySpark.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently and collaborate effectively within a team.

Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.

Dear Candidate,

We have an immediate opening with Mphasis for Senior Full Stack Engineer (Python AI).

📍 Location: Hyderabad / Bangalore

💼 Experience: 5+ Years

🏢 Employment: Permanent

⚡ Immediate Joiners Preferred

Mandatory Skills:

• Python
• Java + Spring Boot
• React.js
• MongoDB

GenAI

Dear Candidate,

Thanks for showing interest in the opportunity!!!

As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.

Job Description:

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.

Primary & Mandatory Skills:

• ETL Development
• Hadoop / HDFS
• PySpark / Apache Spark
• Data Extraction, Transformation & Loading
• SQL and Data Processing

Good to Have:

• Hive / Spark SQL
• Data Pipeline Development
• Big Data Processing
• Python Programming

Company: Mphasis

Role: ETL Developer – Hadoop & PySpark

Experience: As per requirement

Location: Bangalore

Employment Type: Permanent

role: data engineer

Python pyspark, SQL, data engineer

5+yrs

Bangalore/Hyderabad

immediate to 15days.

Company: Wissen Technology

Position: Databricks Engineer

Experience: 6-10 Years

Location: Bengaluru/Mumbai

Employment Type: Full-time

About Wissen Technology

Wissen Technology is a global technology services company focused on delivering innovative software engineering, data, and technology solutions to leading enterprises. The company works with clients across industries to build scalable, high-performance technology platforms and solve complex business and technology challenges.

With a strong focus on engineering excellence, innovation, and collaboration, Wissen Technology brings together skilled technology professionals across areas such as software engineering, data engineering, cloud, analytics, and digital transformation.

At Wissen Technology, employees have the opportunity to work on challenging technology projects, collaborate with experienced engineering teams, and contribute to solutions that create measurable business impact.

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to understand requirements and deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement appropriate data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands-on expertise in Apache Spark and PySpark.
• Strong programming skills in Python; experience with Scala is an advantage.
• Strong proficiency in SQL and hands-on experience with relational databases.
• Practical experience developing and optimizing data pipelines and data processing applications.
• Familiarity with Git and version control systems.
• Strong understanding of data engineering concepts, data transformation, and data validation.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Role Overview

We are looking for an experienced Data Scientist – Agentic AI with strong expertise in Python, Machine Learning, Generative AI, Large Language Models (LLMs), RAG and Agentic AI.

The ideal candidate should have hands-on experience in developing, fine-tuning, evaluating and deploying machine learning and GenAI solutions. The candidate should be comfortable working with open-source LLMs, LangChain/LangGraph, PySpark and AI observability/tracing frameworks.

The role involves building intelligent AI systems that can reason, use tools, retrieve information and execute multi-step tasks using Agentic AI architectures.

Mandatory Technical Skills

1. Data Science & Python

• 5+ years of experience in Data Science / Machine Learning / AI.
• Strong programming experience in Python.
• Strong understanding of data analysis, feature engineering and statistical techniques.
• Experience with Python ML and data science libraries such as:
• NumPy
• Pandas
• Scikit-learn
• Matplotlib / Seaborn
• Good understanding of data preprocessing, exploratory data analysis and experimentation.

2. Machine Learning & Statistics

• Strong understanding of Machine Learning fundamentals.
• Experience with supervised and unsupervised learning techniques.
• Knowledge of:
• Regression
• Classification
• Clustering
• Feature Engineering
• Model Selection
• Hyperparameter Tuning
• Cross-validation
• Strong understanding of Statistics / ML fundamentals.
• Ability to interpret model performance and statistical results.

3. Generative AI / LLM

• Strong hands-on experience with Generative AI and Large Language Models (LLMs).
• Understanding of Transformer architecture and modern LLM-based applications.
• Experience working with commercial or open-source LLMs.
• Strong understanding of:
• Prompt Engineering
• Context Management
• Embeddings
• Tokenization
• LLM inference
• Hallucination mitigation

4. RAG – Retrieval Augmented Generation

• Strong hands-on experience developing RAG applications.
• Experience with:
• Document ingestion
• Chunking
• Embeddings
• Vector search
• Semantic search
• Retrieval pipelines
• Context retrieval
• Reranking
• Ability to optimize RAG pipelines for relevance, accuracy and latency.
• Experience integrating LLMs with enterprise knowledge sources.

5. Agentic AI

• Hands-on experience building Agentic AI / AI Agent solutions.
• Understanding of agent architecture and multi-step reasoning workflows.
• Experience with:
• AI Agents
• Multi-Agent systems
• Tool Calling
• Function Calling
• Agent orchestration
• Planning and reasoning workflows
• Memory
• Workflow automation
• Ability to build agents that can interact with tools, APIs, databases and external systems.

6. LangChain / LangGraph

• Strong hands-on experience with LangChain and/or LangGraph.
• Experience building LLM workflows and agent-based applications.
• Understanding of:
• Chains
• Agents
• Tools
• State management
• Graph-based workflows
• Agent orchestration
• Retrieval workflows
• Experience designing scalable Agentic AI workflows.

7. LLM Fine-Tuning

• Hands-on experience with LLM fine-tuning.
• Understanding of techniques such as:
• Supervised Fine-Tuning (SFT)
• Parameter-Efficient Fine-Tuning
• LoRA
• QLoRA
• Experience preparing datasets for fine-tuning.
• Ability to evaluate fine-tuned models against baseline models.
• Understanding of model optimization and inference considerations.

8. BERT / LLaMA / Open-Source LLMs

Experience working with one or more open-source / transformer-based models such as:

• BERT
• LLaMA / Llama
• Mistral
• Gemma
• Qwen
• Other open-source LLMs

Candidate should understand model loading, inference, fine-tuning and evaluation.

9. PySpark

• Strong experience with PySpark for large-scale data processing.
• Experience working with large datasets and distributed data processing.
• Knowledge of:
• Data transformations
• Data cleaning
• Aggregations
• Joins
• Spark SQL
• Performance optimization
• Ability to build scalable data processing pipelines.

10. Model Validation & Evaluation

• Experience validating and evaluating ML and GenAI models.
• Understanding of traditional ML evaluation metrics.
• Experience evaluating LLM/RAG applications using relevant quality metrics.
• Ability to compare model performance and identify areas for improvement.
• Experience with:
• Accuracy
• Precision
• Recall
• F1 Score
• ROC-AUC
• Retrieval metrics
• LLM response quality
• Groundedness / relevance
• Experience designing evaluation datasets and test cases is preferred.

11. AI Tracing / Observability

• Experience with AI/LLM tracing and observability.
• Ability to monitor AI applications in production.
• Experience tracking:
• LLM requests/responses
• Latency
• Token usage
• Errors
• Retrieval performance
• Agent/tool execution
• Model performance
• Exposure to tools/frameworks such as LangSmith, OpenTelemetry, Arize Phoenix, MLflow or similar is preferred.

12. Model Deployment

• Experience deploying ML/LLM/GenAI solutions into production.
• Exposure to cloud and/or on-premise model deployment.
• Experience with model serving, APIs and production inference.
• Knowledge of deployment environments such as:
• AWS
• Azure
• GCP
• On-premise infrastructure
• Experience with Docker, APIs and CI/CD is an advantage.

Key Responsibilities

• Design, develop and deploy Data Science, Machine Learning and GenAI solutions.
• Build production-ready RAG and Agentic AI applications.
• Develop intelligent agents capable of tool calling, reasoning and multi-step task execution.
• Build LLM-powered applications using LangChain/LangGraph.
• Work with open-source LLMs including BERT, LLaMA and other transformer-based models.
• Fine-tune LLMs for specific business use cases.
• Develop scalable data processing pipelines using PySpark.
• Perform data analysis, feature engineering and statistical modeling.
• Develop and maintain model validation and evaluation frameworks.
• Evaluate ML and LLM models using appropriate performance and quality metrics.
• Implement AI tracing, monitoring and observability for production GenAI systems.
• Deploy models and AI applications in cloud or on-premise environments.
• Optimize model performance, response quality, latency and cost.
• Troubleshoot issues related to model inference, retrieval, agents and LLM workflows.
• Collaborate with Data Scientists, ML Engineers, Software Engineers and business stakeholders.
• Convert business requirements into scalable AI/ML solutions.

Good to Have

• Experience with Vector Databases such as:
• FAISS
• Pinecone
• Weaviate
• Milvus
• Chroma
• Azure AI Search
• Experience with MLflow or similar ML lifecycle tools.
• Experience with Docker/Kubernetes.
• Experience with REST APIs / FastAPI.
• Knowledge of cloud AI/ML services.
• Experience with MLOps / LLMOps.
• Experience with multi-agent frameworks other than LangChain/LangGraph.
• Experience working with enterprise GenAI applications.

Ideal Candidate Profile

The ideal candidate should be a Data Scientist / ML Engineer with strong GenAI and Agentic AI experience, rather than a pure Python developer.

A strong candidate would typically have:

Data Science + Python + ML + Statistics + GenAI/LLM + RAG + Agentic AI + LangChain/LangGraph + LLM Fine-Tuning + Open-Source LLMs + PySpark + Model Evaluation + AI Observability + Model Deployment.

Core Mandatory Skills

Data Science, Python, Machine Learning, Statistics/ML Fundamentals, GenAI/LLM, RAG, Agentic AI, LangChain/LangGraph, LLM Fine-Tuning, BERT/LLaMA/Open-Source LLMs, PySpark, Model Validation/Evaluation, AI Tracing/Observability, Cloud/On-Prem Model Deployment.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

9+yrs

Bangalore/Hyderabad

immediate to 15days.

Job Description – Data Engineer

Role: Data Engineer

Experience: 9+ Years

Location: Bangalore / Hyderabad

Notice Period: Immediate to 15 Days

Interview Process: 1st Round – Virtual | 2nd Round – F2F

Mandatory Skills

• Python
• PySpark
• SQL
• Azure Data Engineering

Job Description

We are looking for an experienced Azure Data Engineer with 9+ years of experience and strong hands-on expertise in Python, PySpark, SQL, and Azure Data Engineering.

Key Responsibilities

• Develop and maintain scalable data engineering solutions using Azure.
• Build and optimize data processing pipelines using PySpark and Python.
• Write complex SQL queries for data extraction and transformation.
• Work with Azure data services and cloud-based data platforms.
• Perform data processing, transformation, and integration.
• Troubleshoot data pipeline and production issues.
• Collaborate with technical and business teams to deliver data solutions.

Preferred: Immediate to 15 Days joiners.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

Bang/hyderabad

immediate to 15days.

Job Description – Senior GCP Data Engineer

Position: Senior GCP Data Engineer

Experience: 8+ Years

Budget: Up to 30 LPA

Location: Bangalore / Hyderabad

Relocation: Candidates from Chennai / Pune willing to relocate are also welcome.

Notice Period: Immediate to 15 Days

Interview Process:

• Round 1: Virtual Interview
• Round 2 / DM Round: Face-to-Face Interview

Required Skills

• Strong hands-on experience in GCP Data Engineering.
• Good programming experience in Python.
• Strong experience with PySpark / Apache Spark.
• Proficiency in SQL and database concepts.
• Experience in developing and maintaining data pipelines.
• Good understanding of data processing, transformation, and integration.
• Experience in handling large-scale data processing.
• Strong problem-solving and debugging skills.
• Good understanding of cloud-based data engineering concepts.
• Experience in optimizing data pipelines for performance and scalability.

Preferred Skills

• Previous experience in a Lead Data Engineer role.
• Experience in technical leadership, mentoring, and guiding team members.
• Ability to collaborate with cross-functional teams and deliver data engineering solutions.

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines using GCP.
• Develop data processing and transformation workflows using Python and PySpark.
• Write and optimize SQL queries for efficient data processing.
• Build and maintain reliable data engineering solutions.
• Troubleshoot data pipeline issues and optimize performance.
• Ensure data quality, reliability, and scalability.
• Collaborate with stakeholders and team members to deliver project requirements.
• Lead and guide team members where applicable.
• Important: Candidates must have strong Senior Data Engineer experience. Lead experience is preferred but not mandatory.

Senior Data Engineer – PySpark & Oracle

Experience: 7+ Years

Location: Bangalore

Notice Period: Immediate to 10 Days

Key Skills:

• Strong expertise in Data Modeling, Data Design & Modernization
• Primary skills: PySpark, Oracle SQL/PLSQL
• Secondary skills: Python, ETL & Data Pipelines
• Experience with Kafka and Hadoop
• Exposure to AWS / Azure / GCP
• Good knowledge of Git and JIRA

Roles & Responsibilities:

• Design, develop, and modernize scalable data models and data architecture.
• Develop and optimize data processing solutions using PySpark and Oracle SQL/PLSQL.
• Build and maintain robust ETL workflows and data pipelines.
• Work with Kafka, Hadoop, and cloud platforms for data processing and integration.
• Perform data transformation, optimization, and performance tuning.
• Collaborate with technical teams on data design, development, testing, and deployment.

🚨 WE ARE HIRING – SENIOR DATA ENGINEER | BANGALORE 🚨

Looking for experienced Senior Data Engineers with strong expertise in PySpark, Oracle SQL/PLSQL, and Data Modeling!

🔹 Role: Senior Data Engineer

🔹 Experience: 7+ Years

🔹 Location: Bangalore

🔹 CTC: Up to 26 LPA

🔹 Notice Period: Immediate to 10 Days Preferred

💻 Mandatory Skills

✅ PySpark

✅ Oracle SQL / PL/SQL

✅ Data Modeling & Design / Modernization

✅ Python

✅ ETL

✅ Data Pipelines

⚙️ Good to Have / Ecosystem Skills

✅ Kafka

✅ Hadoop

✅ AWS / Azure / GCP

✅ Git

✅ JIRA

📌 Key Responsibilities

• Develop and maintain scalable data pipelines using PySpark

• Work extensively with Oracle SQL/PLSQL for data processing and transformation

• Design and implement data models and data architecture

• Work on data design and modernization initiatives

• Develop and optimize ETL processes and data pipelines

• Handle large volumes of data using PySpark and Python

• Work with technologies such as Kafka, Hadoop, and Cloud platforms

• Collaborate with cross-functional teams to deliver scalable data solutions

• Use Git and JIRA for version control and project tracking

📩 Interested candidates can share their updated CV along with:

Total Experience:

Relevant PySpark Experience:

Relevant Oracle SQL/PLSQL Experience:

Data Modeling Experience:

Current Location:

Notice Period:

Current CTC:

Expected CTC:

#Hiring #DataEngineer #SeniorDataEngineer #PySpark #Oracle #PLSQL #DataModeling #Python #ETL #DataPipelines #Kafka #Hadoop #AWS #Azure #GCP #BangaloreJobs #ITJobs #TechJobs #ImmediateJoiners

🚨 Hiring: GCP Data Engineer

We are looking for experienced GCP Data Engineers to join our team!

🔹 Experience: 9+ Years

🔹 Relevant Experience: 4+ Years in GCP Data Engineering

🔹 Required Skills: GCP, Oracle PL/SQL, Python, PySpark

🔹 Location: Bangalore / Hyderabad

🔹 Notice Period: Immediate to 10 Days Preferred

Key Skills:

🔹 Strong hands-on experience in GCP Data Engineering

🔹 Good experience with PySpark & Python

🔹 Strong knowledge of Oracle PL/SQL

🔹 Experience in data processing, ETL, and data pipelines

🔹 Good understanding of cloud-based data engineering

📩 Interested candidates can share their updated CV via DM.

#Hiring #GCPDataEngineer #GCP #DataEngineering #PySpark #Python #OraclePLSQL #DataEngineer #BangaloreJobs #HyderabadJobs #ImmediateJoiner #TechJobs #ITJobs #HiringNow

GCP Data Engineering Lead

Experience: 9+ Years

Lead Experience: 2+ Years

Location: Bangalore / Hyderabad

Key Skills:

• Strong experience in GCP Data Engineering and BigQuery
• Hands-on experience with Oracle Exadata / PL-SQL
• Strong knowledge of PySpark / Scala and Python
• Experience with GoldenGate, Kafka and CDC
• Hands-on experience with Apache Airflow
• Good experience in CI/CD and DevOps practices
• Experience with Terraform / Infrastructure as Code
• Exposure to AI/LLM technologies and GenAI solutions
• Strong understanding of data architecture, ETL/ELT and data pipelines

Roles & Responsibilities:

• Lead the design and development of scalable GCP data engineering solutions.
• Design and implement batch and real-time data pipelines using BigQuery, PySpark, Kafka/CDC and Airflow.
• Work with Oracle Exadata/PL-SQL and GoldenGate for data integration and migration.
• Implement CI/CD pipelines and infrastructure automation using Terraform.
• Explore and integrate AI/LLM capabilities into data engineering solutions.
• Lead technical discussions, code reviews, solution design and mentor team members.
• Collaborate with business and technical teams to deliver high-quality data solutions.

Job Description: Python + AI

Company: Wissen Technology

Location: Bangalore, India

Experience: 5+Years

Employment Type: Full-Time

Role: Python + AI / Data Engineer

About the Role

Wissen Technology is looking for experienced Python + AI / Data Engineering professionals to join our technology team in Bangalore. The ideal candidate will have strong hands-on experience in Python, Artificial Intelligence, Generative AI, PySpark, Snowflake, and data pipeline development.

The candidate should be capable of designing and developing scalable data and AI solutions, building robust ETL/ELT pipelines, working with large datasets, and integrating AI/ML capabilities into enterprise applications.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python and PySpark.
• Develop robust ETL/ELT pipelines for processing large volumes of structured and unstructured data.
• Build and optimize data processing solutions using Apache Spark / PySpark.
• Develop data ingestion and transformation pipelines into Snowflake.
• Design and implement scalable Snowflake data models, tables, views, and SQL transformations.
• Work with batch and, where applicable, real-time data processing pipelines.
• Build and integrate AI and Generative AI solutions using Python.
• Develop LLM-based applications, RAG solutions, AI agents, and AI-powered services.
• Integrate AI models with enterprise data platforms and data pipelines.
• Develop REST APIs and microservices using FastAPI, Flask, or Django.
• Perform data cleansing, transformation, validation, and quality checks.
• Optimize PySpark jobs, SQL queries, Snowflake workloads, and data pipelines for performance and scalability.
• Implement data pipeline monitoring, logging, error handling, and alerting.
• Work with cloud platforms such as AWS, Azure, or GCP.
• Collaborate with Data Scientists, Data Engineers, Software Engineers, Architects, and business stakeholders.
• Participate in technical design, architecture, code reviews, and production support.
• Mentor junior engineers and contribute to engineering best practices.

Preferred Qualifications

• Bachelor's or master's degree in computer science, Engineering, Data Science, Artificial Intelligence, or a related field.
• Experience working on enterprise-scale AI and data engineering projects.
• Experience combining Python + PySpark + Snowflake + AI/GenAI in production environments.
• Experience with Databricks is an advantage.
• Experience with AI Agents / Agentic AI and tool/function calling.
• Knowledge of distributed systems and cloud-native architecture.
• Experience leading technical initiatives or mentoring engineering teams.

Data Engineer Hiring Post

🚨 Hiring: Data Engineer | PySpark + Python + SQL

We are looking for experienced Data Engineers to join our team!

🔹 Experience: 5 to 9 Years

🔹 Locations: Bangalore / Hyderabad

🔹 Interview Process:

• 1st Round – Virtual

• 2nd Round – Face-to-Face (Karat Test)

🔑 Key Skills:

✅ PySpark

✅ SQL

✅ Python

✅ ETL

📩 Interested candidates can share their updated resume.

#Hiring #DataEngineer #PySpark #Python #SQL #ETL #BangaloreJobs #HyderabadJobs #TechHiring #ImmediateHiring

Data Engineer

Location: Bengaluru, India (Hybrid)

Employment Type: Full-time

Experience: 3-5 years

Role Overview  

What We’re Looking For:

• Bachelor’s degree in Computer Science/Engineering or equivalent experience required.
• Experience designing and shipping cloud services products.
• Experience driving and managing technical and architectural dependencies on AWS Cloud.
• A firm understanding of system architecture, cloud computing, PaaS/SaaS design principles, S3, DynamoDB, RDS mandatory.
• Experience in building or maintaining ETL processes and tools, i.e., AWS Glue or any open-source tool.
• Proven system-level design contribution to a current “Live” (in production / under daily high load) multi-region SaaS or PaaS offering.
• Proven experience with S3, DynamoDB, SQL, and AWS RDS services.
• Proficiency in programming languages such as Python.
• Strong analytical and problem-solving skills.
• 
  

Required Skills & Experience

• Experience with Python, SQL, and data visualization/exploration tools.
• Familiarity with the AWS ecosystem, specifically S3, DynamoDB, and RDS.
• Communication skills, especially for explaining technical concepts to nontechnical business leaders.
• Ability to work on a dynamic, research-oriented team that has concurrent projects.
• Experience in AWS cost optimization (Savings Plans, Reserved Instances, Spot Instances) and governance frameworks.
• Experience developing solutions using infrastructure orchestration tools (SSM, automation account, Ansible, etc.).
• Excellent leadership, stakeholder management, and communication skills.

What We Offer

• Work with some of the brightest minds in the emerging EV industry.
• Make a tangible impact in reducing carbon emissions and enabling sustainable energy.
• Freedom to suggest, implement, and innovate on systems, processes, and technologies.
• Daily ownership in a high-growth, challenging environment.
• Flexible work environment with hybrid schedules and virtualization options.
• Competitive pay and benefits including health coverage, innovative PTO program, and performance bonuses.

Job Description

We are looking for an experienced Data Engineer with strong expertise in Python, ETL, SQL, CI/CD, and DevOps to design, develop, and maintain scalable data pipelines and data processing solutions.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Develop data processing solutions using Python.
• Write complex and optimized SQL queries, stored procedures, and data transformations.
• Build and maintain data ingestion and integration workflows.
• Implement data quality, validation, monitoring, and error-handling processes.
• Develop and maintain CI/CD pipelines for data engineering applications.
• Work with DevOps tools and practices for automated build, deployment, and infrastructure management.
• Collaborate with data analysts, data scientists, software engineers, and business teams.
• Optimize data pipelines for performance, reliability, and scalability.
• Troubleshoot production data issues and ensure timely resolution.
• Follow best practices for version control, code quality, testing, and deployment.

Mandatory Skills

• Python
• ETL
• SQL
• CI/CD
• DevOps
• Git / Version Control
• Strong problem-solving and debugging skills

Job Description:

Experience: 10+ Years

Job Summary

We are looking for an experienced Azure Fabric Data Architect to lead the design and implementation of an enterprise data platform on Microsoft Fabric. The role involves architecting scalable data solutions, defining data governance, and enabling AI-driven analytics for a global financial services client.

Key Responsibilities

• Design end-to-end data architecture using Microsoft Fabric.
• Build enterprise Lakehouse, Data Warehouse, and OneLake solutions.
• Define data ingestion, ETL/ELT, governance, security, and performance strategies.
• Lead architecture for AI-powered analytics, AI Agents, and enterprise chatbots using Azure AI services.
• Work with business stakeholders to translate requirements into technical solutions.
• Mentor engineering teams and provide technical leadership.

Required Skills

• Microsoft Fabric (Data Factory, Lakehouse, Data Warehouse, OneLake)
• Azure Data Engineering
• Power BI
• Azure AI Services / Azure OpenAI
• Data Architecture & Data Modeling
• SQL, Python
• Azure DevOps, CI/CD
• Strong stakeholder management and solution design experience
• 
  

Preferred: Experience in Capital Markets or Financial Services and Microsoft Azure/Fabric certifications.

NOTE: One technical round is mandatory to be taken F2F from office.

CAP-190 : Senior Data Analyst

📍Location : Bangalore / Hyderabad / Chennai / Pune / NCR

🧠Experience : 7 - 15 Years

🆔Job Code : CAP-190

🏢Work Type : Hybrid - 3 Days in a Week

About the Client (CODE: CAP)

CAP operates at the forefront of the financial services industry, providing consulting, technology, and digital transformation solutions to leading organizations worldwide. With a focus on innovation, CAP empowers clients to navigate complex regulatory landscapes, optimize operations, and drive business growth. The company fosters a collaborative and agile culture, encouraging continuous learning and excellence.

Key Responsibilities

● Define and obtain source data required to deliver insights and use cases.

● Determine data mapping and join multiple data sets across various sources.

● Develop methods to highlight and report data inconsistencies for user review.

● Propose and assist with suitable data migration sets for stakeholders.

● Support teams in processing data migration sets and coordinating migration activities.

● Plan, track, and coordinate the data migration team and migration run-book.

● Collaborate with stakeholders to avoid negative customer and business impacts.

● Ensure robust communication and escalation mechanisms across project portfolios.

● Implement strategic solutions and avoid short-term workarounds.

● Maintain strong control and compliance standards in data handling.

Required Skills

● Minimum 7+ years of experience as a Data Analyst, preferably in financial services.

● Strong expertise in Pyspark, Python, and SQL.

● Experience with big data programs and data models in banking or financial markets.

● Ability to write SQL queries and navigate databases such as Hive, CMD, Putty, and Note++.

● Excellent analytical skills and commercial acumen.

● Strong verbal and written communication skills.

● Proven ability to manage multiple priorities and deliver within tight deadlines.

● Business analysis skills, including defining and understanding requirements.

● Familiarity with SDLC, Agile processes, and a bias towards TDD.

● Attention to detail and a proactive, problem-solving mindset.

Nice to Have

● Knowledge and experience in Data Quality & Governance.

● Working experience with Spark Scala or Java for Spark.

● Proven track record of managing small, delivery-focused data teams (for senior roles).

● Experience with market data vendors and domains such as Party/Client, Trade, Settlements, Payments, Instrument and Pricing, Market and/or Credit Risk.

Why Join CAP (Code Name)

Join CAP to work on impactful data initiatives within the financial services sector, tackling complex technical challenges and driving meaningful business outcomes. You'll collaborate with talented professionals in a dynamic, agile environment that values innovation and continuous improvement. CAP offers opportunities for professional growth, skill development, and the chance to contribute to high-visibility projects that shape the future of financial technology.

About the Employment Model

Direct Hire (Client Payroll) : For this role, you’ll be hired directly by the client and be part of their internal team. Straatix supports the hiring process, but your employment, payroll, and benefits are all managed by the client.

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.

Must-Have Skills :

• 14 to 20 years of Data Engineering experience
• Databricks & Apache Spark / PySpark
• Python & SQL
• AWS Cloud
• Lakehouse Architecture
• ETL / ELT & Distributed Data Processing
• Batch & Streaming Pipelines
• Data Pipeline Optimization & Data Modeling
• CDC & Incremental Processing
• Git, CI/CD & Testing
• Data Quality, Monitoring & Observability
• Technical Leadership & Stakeholder Management

Key Responsibilities :

• Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
• Own data products from design through production.
• Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
• Optimize pipelines for performance, scalability, reliability, and cost.
• Design scalable data architectures and data models.
• Implement data quality, monitoring, lineage, and CI/CD practices.
• Lead technical discussions and mentor engineering teams.
• Collaborate with business stakeholders, architects, product owners, and engineering teams.
• Remain hands-on while providing technical leadership.

Ideal Candidate :

A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.

🔴 Super Urgent : Only Bangalore-based immediate joiners.

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.

Define and obtain source data required to deliver insights and use cases.

● Determine data mapping and join multiple data sets across various sources.

● Develop methods to highlight and report data inconsistencies for user review.

● Propose and assist with suitable data migration sets for stakeholders.

● Support teams in processing data migration sets and coordinating migration activities.

● Plan, track, and coordinate the data migration team and migration run-book.

● Collaborate with stakeholders to avoid negative customer and business impacts.

● Ensure robust communication and escalation mechanisms across project portfolios.

● Implement strategic solutions and avoid short-term workarounds.

● Maintain strong control and compliance standards in data handling.

Required Skills

● Minimum 7+ years of experience as a Data Analyst, preferably in financial services.

● Strong expertise in Pyspark, Python, and SQL.

● Experience with big data programs and data models in banking or financial markets.

● Ability to write SQL queries and navigate databases such as Hive, CMD, Putty, and Note++.

● Excellent analytical skills and commercial acumen.

● Strong verbal and written communication skills.

● Proven ability to manage multiple priorities and deliver within tight deadlines.

● Business analysis skills, including defining and understanding requirements.

● Familiarity with SDLC, Agile processes, and a bias towards TDD.

● Attention to detail and a proactive, problem-solving mindset.

Nice to Have

● Knowledge and experience in Data Quality & Governance.

● Working experience with Spark Scala or Java for Spark.

● Proven track record of managing small, delivery-focused data teams (for senior roles).

● Experience with market data vendors and domains such as Party/Client, Trade, Settlements, Payments, Instrument and Pricing, Market and/or Credit Risk.

Design, develop, and maintain ETL pipelines involving large-scale data.

Develop data processing and analytics applications primarily using PySpark and Python.

Build scalable and distributed data processing solutions using Apache Spark.

Develop and deploy data applications on AWS cloud.

Work with AWS services related to storage, compute, ETL, data warehousing, analytics, and streaming.

Implement distributed storage and processing solutions capable of handling high-volume datasets.

Design data processing applications with a focus on performance, scalability, reliability, and optimization.

Work with both SQL and NoSQL databases for data storage, processing, and analytics.

Write, optimize, and analyze SQL, HQL, and NoSQL queries.

Troubleshoot data pipeline and processing issues and ensure data quality and reliability.

Collaborate with data engineers, analysts, architects, and other technical teams to deliver data-driven solutions.

Job Summary

We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.

Technical Skills

• Strong hands-on experience in Python and PySpark development.
• Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
• Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Power BI Data Modeling and Semantic Layer development.
• Proficiency in DAX (Data Analysis Expressions).
• Experience designing and managing Semantic Models in Power BI.
• Strong SQL skills and experience working with large datasets.
• Knowledge of data warehousing concepts and best practices.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Exposure to modern data platforms like Databricks.
• Understanding of data governance and data quality frameworks.

Roles & Responsibilities

• Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

2Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

4Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

5Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

6Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

8Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Skills Referential (Required knowledge, skills and abilities)

Technical Skills:

Python

Pyspark

SQL

ETL Aws, Azure, gcp

Key Responsibilities  

• Design, build, and optimize scalable data pipelines for AI/ML applications.
• Develop, train, evaluate, and deploy Machine Learning and Deep Learning models.
• Build production-ready LLM applications using Retrieval-Augmented Generation (RAG), prompt engineering, and vector databases.
• Fine-tune open-source and foundation models using domain-specific datasets.
• Develop and maintain end-to-end MLOps pipelines for model deployment, monitoring, and lifecycle management.
• Perform data preprocessing, feature engineering, exploratory data analysis (EDA), and model evaluation.
• Develop APIs and AI services for production deployment.
• Collaborate with cross-functional teams to deliver scalable AI-driven solutions.
• Monitor model performance, troubleshoot production issues, and maintain technical documentation.

Required Skills  

Mandatory  

• 1–3 years of experience in Data Science, Data Engineering, or AI/ML development.
• Strong programming skills in Python and SQL.
• Hands-on experience with Machine Learning frameworks such as PyTorch, TensorFlow, or Scikit-learn.
• Experience building LLM-powered applications using RAG, Prompt Engineering, and Embeddings.
• Hands-on experience with LangChain, LlamaIndex, CrewAI, or n8n for LLM orchestration and AI workflow automation.
• Experience in LLM fine-tuning and working with Hugging Face models.
• Knowledge of MLOps concepts including model deployment, monitoring, versioning, and CI/CD.
• Experience with Git, REST APIs, Linux environments, and data processing libraries.

Preferred  

• Experience with vector databases such as Pinecone, Chroma, Milvus, or Weaviate.
• Familiarity with Docker, Kubernetes, and MLflow.
• Exposure to Apache Spark or Airflow for data engineering workflows.
• Experience with cloud platforms (AWS, Azure, or GCP).

Primary Technology Stack  

• Languages & Data Processing: Python, SQL, Pandas, NumPy, Apache Spark
• AI & Machine Learning: PyTorch, TensorFlow, Scikit-learn
• Application Frameworks: LangChain, LlamaIndex, CrewAI, n8n
• Core Methodologies: Retrieval-Augmented Generation (RAG), Model Fine-Tuning, Prompt Engineering, Embeddings
• Models & Infrastructure: OpenAI APIs, Hugging Face Ecosystem, Embedding Models
• Vector Databases: Pinecone, Chroma, Milvus, Weaviate
• Databases: PostgreSQL, MongoDB
• MLOps & DevOps: Docker, Kubernetes, MLflow, CI/CD, Git
• Cloud Platforms: AWS, Azure, GCP

Experience: 1–3 Years

Domain: Data Science | Data Engineering | Machine Learning | Generative AI | MLOps

Job Summary

We are seeking a skilled Azure Data Engineer with hands-on experience in Azure Data Services, Azure Databricks, Python, PySpark, and SQL. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and data processing solutions to support business intelligence, analytics, and reporting requirements.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks and PySpark.
• Build and optimize data processing workflows using Python and SQL.
• Develop and manage data ingestion pipelines from multiple structured and unstructured data sources.
• Work with Azure Data Factory (ADF) to orchestrate and schedule data pipelines.
• Implement data transformation and cleansing logic using PySpark.
• Optimize SQL queries and Spark jobs for performance and scalability.
• Collaborate with data architects, analysts, and business stakeholders to understand data requirements.
• Ensure data quality, integrity, and governance across the data platform.
• Monitor, troubleshoot, and resolve production data pipeline issues.
• Follow coding standards, version control, and CI/CD best practices.

Required Skills

• Strong experience with Microsoft Azure cloud services.
• Hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Expertise in PySpark for large-scale data processing.
• Strong SQL skills, including query optimization and performance tuning.
• Experience with Azure Data Factory (ADF).
• Knowledge of Delta Lake, Spark SQL, and Databricks notebooks.
• Experience with Git or Azure DevOps for source code management.
• Understanding of data warehousing concepts and ETL/ELT processes.

Preferred Skills

• Experience with Azure Synapse Analytics.
• Knowledge of Delta Live Tables (DLT).
• Experience with Azure Data Lake Storage (ADLS Gen2).
• Familiarity with Unity Catalog and data governance.
• Exposure to CI/CD pipelines and infrastructure-as-code.
• Experience working in Agile/Scrum environments.

• Job Title: Lead Data Engineer
• Location: Marathahalli, Bangalore 
• Job Mode: Hybrid

Key Responsibilities:

·   Create and maintain data sources inside our data lake on Azure

·   Assemble and analyze large, complex data sets that meet both functional and non-functional business requirements

·   Identify, design, and implement internal process improvements: automating manual processes; optimizing data delivery; re-designing infrastructure for greater scalability.

·   Work with stakeholders including the Executive, Tech, Product, Data and Design teams to optimize data driven decisions

·   Keep our data separated and secure by understanding and applying best practice access policies

·   Create tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader

·   Ensure data integrity with thorough data quality test coverage and data validation

·   Improve, create and implement CI/CD processes and tooling inside the data lake

Required Qualifications

-     Join immediately

-     Databricks

-     Databricks hands on experience (esp. Pipelines PySpark & SQL, Delta Lake, Unity Catalog, DLT*)

-     Ingesting & transforming data using Databricks - Min 3 years continuous hands-on

-     Advanced SQL and Python knowledge

-     Data Lake housing and Modelling experience (e.g. medallion architecture & query performance optimization)

-     Experience with backend systems ERP, CRM (e.g. SAP, Oracle, Salesforce)

-     Exposure of working with Azure (ADLS, ADF)

-     Good Communication skills

Job Summary

We are seeking a highly skilled GCP Data Engineer with strong expertise in Google Cloud Platform (GCP), Python, ETL, and modern data engineering technologies. The ideal candidate should have hands-on experience designing and building scalable data pipelines using BigQuery, Dataflow, Pub/Sub, Airflow, and modern data lake technologies such as Apache Iceberg or Delta Lake.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines on Google Cloud Platform.
• Build and optimize data processing workflows using Python and Google Cloud Dataflow (Apache Beam).
• Develop and manage large-scale analytical data models in BigQuery.
• Implement event-driven data ingestion using Google Cloud Pub/Sub.
• Create, schedule, and monitor workflows using Apache Airflow and Autosys.
• Design and implement modern data lake architectures using Apache Iceberg or Delta Lake.
• Optimize query performance, storage, and compute costs in GCP.
• Ensure data quality, governance, security, and compliance across data platforms.
• Collaborate with Data Scientists, Analysts, and Application teams to deliver scalable data solutions.
• Troubleshoot production issues and continuously improve pipeline reliability and performance.

Mandatory Skills

• Strong hands-on experience with Google Cloud Platform (GCP).
• Proficiency in Python programming.
• Experience in designing and implementing ETL/ELT pipelines.
• Strong knowledge of BigQuery.
• Experience with Google Cloud Dataflow (Apache Beam).
• Experience with Google Cloud Pub/Sub.
• Hands-on experience with Apache Airflow.
• Experience in job scheduling using Autosys.
• Experience with modern table formats such as Apache Iceberg or Delta Lake.
• Strong SQL and data modeling skills.

Preferred Skills

• Experience with Cloud Storage, Dataproc, Cloud Composer, and Cloud Functions.
• Knowledge of CI/CD pipelines and DevOps practices.
• Experience with Docker and Kubernetes.
• Familiarity with Git and Agile/Scrum methodologies.
• Knowledge of data warehousing and dimensional modeling.
• Exposure to streaming and real-time data processing.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
• 4–8+ years of experience in Data Engineering with hands-on expertise in GCP technologies.

Required Experience

• Strong experience in developing enterprise-grade data pipelines using Python and GCP.
• Hands-on experience with BigQuery, Dataflow, Pub/Sub, and Airflow.
• Experience scheduling and monitoring batch workflows using Autosys.
• Experience implementing modern data lake architectures using Apache Iceberg or Delta Lake.
• Strong understanding of ETL best practices, performance tuning, and data optimization.
• Excellent analytical, troubleshooting, and problem-solving skills.

Mandatory Skills

• Google Cloud Platform (GCP)
• Python
• ETL
• BigQuery
• Autosys
• Apache Airflow
• Google Cloud Pub/Sub
• Google Cloud Dataflow (Apache Beam)
• Apache Iceberg / Delta Lake
• SQL & Data Modeling

Job Summary

We are looking for an experienced AI Data Architect to design and build an enterprise AI-ready data platform that serves as the single source of truth for AI applications, including RAG, Agentic AI, Conversational AI, ML models, and analytics.

Key Responsibilities

• Design enterprise AI data platform and Lakehouse architecture.
• Build batch & real-time data pipelines.
• Develop semantic models, knowledge graphs, and vector databases.
• Architect RAG and LLMOps infrastructure.
• Implement data governance, security, and AI observability.
• Modernize legacy data platforms to cloud-native architectures.

Required Skills

• Python, SQL, PySpark
• Databricks, Delta Lake, Snowflake
• Kafka, Spark Structured Streaming
• AWS / Azure
• LangChain, LlamaIndex
• OpenAI, Claude, Bedrock
• Pinecone, FAISS, ChromaDB, Neo4j
• MLflow, Docker, Kubernetes, Terraform
• FastAPI, GitHub Actions, Jenkins
• Data Governance, RBAC, CI/CD

Requirements

• 12+ years in Data Engineering/Data Architecture.
• Experience with AI/ML, RAG, LLMOps, and enterprise AI platforms.
• Strong expertise in Lakehouse, Data Mesh, Cloud, and Vector Databases.
• Hands-on experience with enterprise-scale AI data architecture and governance. 

Senior Data Scientist (ML) – 5+ Years

Required Skills:

• 5+ years of experience in Data Science/ML
• Strong knowledge of Probability & Statistics
• Strong educational background in Statistics or Mathematics (degree/course specialization). 
• Good understanding of core Machine Learning algorithms
• Proficiency in Python, PySpark, and SQL
• Experience with Databricks, Azure ML, SageMaker, or Vertex AI
• Hands-on experience in Natural Language Processing (NLP)
• Understanding of Transformer-based models and architectures
• Ability to build, deploy, and optimize ML solutions at scale

About the Role

The Smart Operations (Smart Ops) division is seeking a seasoned Senior Data Engineer / Lead Data Engineer to design and scale our next-generation industrial data platform. In this role, you will architect the real-time data backbone that processes high-frequency iHistorian telemetry, IoT sensor feeds, and complex manufacturing data across global operations.

You will be the core technical owner of our Databricks Lakehouse ecosystem, responsible for building optimized, cost-efficient streaming and batch pipelines that deliver high-trust data to our Advanced Analytics, MLOps, and Business Intelligence teams.

Key Responsibilities

• End-to-End Lakehouse Architecture: Design, implement, and scale a robust Medallion Architecture (Bronze-Silver-Gold) on Azure Databricks and Delta Lake.
• Real-Time IoT Ingestion: Build fault-tolerant, low-latency streaming pipelines to ingest high-frequency telemetry and machine sensor data.
• Enterprise Governance: Enforce data security, fine-grained access controls, and comprehensive lineage tracking using Unity Catalog.
• Performance Tuning & Cost Optimization: Actively profile, tune, and optimize Spark workloads (utilizing partitioning, liquid clustering, AQE, and Photon) to drastically minimize cloud compute costs and pipeline runtimes.
• CI/CD Automation: Modernize data platform deployments across Dev/QA/Prod environments using Infrastructure as Code (IaC) and Databricks Asset Bundles (DABs).
• Cross-Functional Collaboration: Partner closely with manufacturing stakeholders, supply chain analysts, and data scientists to provide curated datasets for predictive maintenance, anomaly detection, and operational KPI reporting.

Technical Skills Breakdown

Mandatory Skills (Must-Haves)

• Domain Context: Minimum 5+ years of Data Engineering experience with direct exposure to Heavy Manufacturing, Automotive/Commercial Vehicles, Industrial Automation, or Global Logistics data environments.
• Databricks Ecosystem: Deep architectural mastery of Azure Databricks, PySpark, Spark SQL, Delta Lake, and Unity Catalog governance.
• Streaming & CDC: Proven hands-on experience engineering real-time data pipelines using Apache Kafka, Azure Event Hubs, or Spark Structured Streaming, alongside Change Data Capture (CDC) tools (e.g., Debezium).
• Cloud Infrastructure: Strong proficiency within the Microsoft Azure stack—specifically Azure Data Factory (ADF), Azure Data Lake Storage Gen2 (ADLS Cores), Azure Key Vault, and Azure Monitor.
• Advanced SQL & Python: Expert-level Python programming and complex SQL optimisation (advanced indexing, partition pruning, performance tuning of massive datasets).
• DevOps Automation: Hands-on experience setting up production CI/CD pipelines using Git, Azure DevOps, GitHub Actions, or Databricks Asset Bundles (DABs).

Good to Have Skills (Nice-to-Haves)

• Professional Certifications: Databricks Certified Data Engineer Professional (highly preferred), Databricks Certified Associate Developer for Apache Spark, or Microsoft Certified: Azure Data Engineer Associate (DP-203).
• Industrial & ERP Systems: Familiarity with industrial data systems (SCADA, PLC, iHistorian), heavy vehicle telemetry standards (MDF/MF4 files), or deep integration experience with enterprise ERPs (SAP S/4HANA, SAP BW).
• MLOps & GenAI: Exposure to working alongside Machine Learning pipelines, dataset feature engineering, or implementing Generative AI frameworks (RAG, Vector Databases like FAISS/Weaviate) for semantic query self-service.
• Infrastructure as Code (IaC): Basic proficiency with Terraform for provisioning cloud data infrastructure.

Education & Qualifications

• Bachelor’s or Master’s Degree in Computer Science, Information Technology, Electrical/Electronics Engineering, or a closely related technical field.

Location: Bangalore (Hybrid/Onsite)

Experience: 5–8 Years

Work location -Manyata Tech park

Job Description

We are seeking a skilled GCP Data Engineer with 5–8 years of experience in designing, developing, and maintaining scalable data pipelines and cloud-based data solutions. The ideal candidate should have strong expertise in Google Cloud Platform (GCP), Python, PySpark, SQL, and Data Engineering concepts.

Key Responsibilities

• Design, build, and optimize scalable ETL/ELT data pipelines.
• Develop and maintain data processing solutions using Python and PySpark.
• Work with large-scale structured and unstructured datasets.
• Implement data ingestion, transformation, and data quality frameworks.
• Build and manage data solutions on Google Cloud Platform (GCP).
• Develop and optimize complex SQL queries, stored procedures, and data models.
• Collaborate with business stakeholders, data analysts, and cross-functional teams to understand data requirements.
• Monitor, troubleshoot, and improve data pipeline performance and reliability.
• Ensure data governance, security, and compliance standards are followed.
• Support data warehousing and analytics initiatives.

Required Skills

• 5–8 years of experience in Data Engineering.
• Strong programming experience in Python.
• Hands-on experience with PySpark and distributed data processing.
• Strong expertise in SQL and database performance tuning.
• Experience with Google Cloud Platform (GCP) services such as:
• BigQuery
• Cloud Storage
• Dataflow
• Dataproc
• Cloud Composer
• Pub/Sub
• Experience in designing ETL/ELT workflows.
• Knowledge of data warehousing concepts and dimensional modeling.
• Experience with version control tools such as Git.
• Strong problem-solving and analytical skills.

Preferred Skills

• Experience with CI/CD pipelines and DevOps practices.
• Exposure to orchestration tools like Airflow/Cloud Composer.
• Experience working in Agile/Scrum environments.
• Knowledge of streaming data processing and real-time data pipelines.

Educational Qualification

• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

Hiring: GCP Data Engineer (FTE)

📍 Location: Bangalore | Chennai | Pune | Gurgaon | Kolkata

💼 Employment Type: Full-Time

Notice Period - Immediate Joiner ( Serving Notice Period )

Work Mode - Hybrid

We are looking for experienced GCP Data Engineers with strong expertise in building scalable cloud data solutions.

Required Skills:

✔ GCP Data Engineering Experience

✔ BigQuery

✔ SQL & Python

✔ PySpark / Apache Spark

✔ Apache Beam / Dataflow

✔ ETL / ELT Pipeline Development

✔ Airflow / Cloud Composer

Responsibilities:

• Design and develop scalable ETL/ELT pipelines on GCP
• Build and optimize BigQuery solutions
• Process large-scale structured/unstructured data using Spark
• Develop automated workflows and cloud-native data pipelines
• Work with GCP services like Dataflow, Dataproc, Cloud Storage, etc.
• 
  

Good to Have:

• Google Cloud Certifications (Professional Data Engineer / Solution Architect)
• Experience with BigTable, Cloud SQL, Spanner, NoSQL databases

🎓 Qualification: Bachelor’s / Master’s in CS, Engineering, or related field

Experience: 5 to 8 years 

Job Summary: 

• We are looking for an experienced Databricks Developer with a strong background in Spark/pyspark to design, develop, and optimize data processing applications. The ideal candidate will build scalable data engineering solutions and ensure seamless data flow across platforms. 
• Key Responsibilities: 
• Design, develop, and maintain scalable data processing applications using Python, PySpark and Spark. 
• Collaborate with data engineers, data scientists, and other stakeholders to understand business requirements and deliver high-quality solutions. 
• Ensure data integrity, performance, and reliability across all data processing pipelines. 
• Write clean, maintainable, and efficient code following best practices and coding standards. 
• Perform data analysis and implement data validation to ensure quality. 
• Monitor and troubleshoot performance issues in data workflows and pipelines. 
• Implement and manage CI/CD pipelines for automated testing, integration, and deployment. 

• Required Skills & Qualifications: 
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. 
• Proven experience as a Databricks Developer, with strong expertise in Spark/Pyspark. 
• Proficiency in SQL and hands-on experience with relational databases. 
• Strong programming experience in Python and PySpark. 
• Familiarity with version control systems such as Git. 
• Strong analytical and problem-solving abilities. 
• Excellent communication and collaboration skills. 

            Azure Database Developer

Mandatory Skills: Data Bricks (Pyspark/Spark), Unity Catalog, T-SQL, Azure Data Factory, Power BI.

Power BI, & Azure Devops

·        Very strong understanding on ETL and ELT

·        Very strong understanding on Medallion architecture.

·        Very strong knowledge in Pyspark and Spark architecture.

·        Very strong knowledge in Unity Catalog architecture using databricks.

·        Very strong knowledge in T-SQL 

·        Good knowledge in Azure data lake architecture and access controls.

·        Good knowledge in data pipeline creation and monitoring using Azure data factory

·        Good knowledge in CI /CD process using Azure devops

·        Power BI

Profile - Databricks Developer

Experience- 5+ years

Location- Bangalore (On site)

PF & BGV is Mandatory

Job Description: -

* Design, build, and optimize data pipelines and ETL/ELT workflows using Databricks and

Apache Spark (PySpark).

* Develop scalable, high performance data solutions using Spark distributed processing.

* Lead engineering initiatives focused on automation, performance tuning, and platform

modernization.

* Implement and manage CI/CD pipelines using Git-based workflows and tools such as

GitHub Actions or Jenkins.

* Collaborate with cross-functional teams to translate business needs into technical

solutions.

* Ensure data quality, governance, and security across all processes.

* Troubleshoot and optimize Spark jobs, Databricks clusters, and workflows.

* Participate in code reviews and develop reusable engineering frameworks.

* Should have knowledge of utilizing AI tools to improve productivity and support daily

engineering activities.

* Strong knowledge and hands-on experience in Databricks Genie, including prompt

engineering, workspace usage, and automation.

Required Skills & Experience:

* 5+ years of experience in Data Engineering or related fields.

* Strong hands-on expertise in Databricks (notebooks, Delta Lake, job orchestration).

* Deep knowledge of Apache Spark (PySpark, Spark SQL, optimization techniques).

* Strong proficiency in Python for data processing, automation, and framework

development.

* Strong proficiency in SQL, including complex queries, performance tuning, and analytical

functions.

* Strong knowledge of Databricks Genie and leveraging it for engineering workflows.

* Strong experience with CI/CD and Git-based development workflows.

* Proficiency in data modeling and ETL/ELT pipeline design.

* Experience with automation frameworks and scheduling tools.

* Solid understanding of distributed systems and big data concepts

Want to work on exciting GenAI projects for Fortune 500 companies across multiple sectors? Then read on..

About Company:

CSG is a multi-national company having a presence in 20 countries with 1600+ Engineers. Company works with more than 40 Fortune 500 customers such as Sony, Samsung, ABB, Thyssenkrup, Toyota, Mitsubishi and many more.

Job Description:

We are looking for a talented Generative AI Developer to join our dynamic AI/ML team. This position offers an exciting opportunity to leverage cutting-edge Generative AI (GenAI) technologies to drive innovation to solve real world problems. You will be responsible for developing and optimizing GenAI-based applications, implementing advanced techniques like Retrieval-Augmented Generation (RAG), RIG (Retrieval Interleaved Generation), Agentic Frameworks and vector databases. This is a collaborative role where you will work directly with customers cross-functional teams to design, implement, and optimize AI-driven solutions. Exposure to cloud-native AI platforms such as Amazon Bedrock and Microsoft Azure OpenAI is highly desirable.

Key Responsibilities

Generative AI Application Development:

Design, develop, and deploy GenAI-driven applications to address complex industrial challenges.

Implement Retrieval-Augmented Generation (RAG) and Agentic frameworks

Data Management & Optimization:

Design and optimize document chunking strategies tailored to specific datasets and use cases.

Build, manage, and optimize data embeddings for high-performance similarity searches across vector databases.

Collaboration & Integration:

Work closely with data engineers and scientists to integrate AI solutions into existing pipelines.

Collaborate with cross-functional teams to ensure seamless AI implementation.

Cloud & AI Platform Utilization:

Explore and implement best practices for utilizing cloud-native AI platforms, such as Amazon Bedrock and Azure OpenAI, to enhance solution delivery.

Continuous Learning & Innovation:

Stay updated with the latest trends and emerging technologies in the GenAI and AI/ML fields, ensuring our solutions remain cutting-edge.

Requirements:

The ideal candidate will have strong experience in Generative AI technologies, particularly in the areas of RAG, document chunking, and vector database management. They will be able to quickly adapt to evolving AI frameworks and leverage cloud-native platforms to create efficient, scalable solutions. You will be working in a fast-paced and collaborative environment, where innovation and the ability to learn and grow are key to success.

- 3 to 5 years of overall experience in software development, with 3 years focused on AI/ML.

- Minimum 2 years of experience specifically working with Generative AI (GenAI) technologies.

- Python, PySpark and SQL knowledge is necessary for tasks

- Proven ability to work in a collaborative, fast-paced, and innovative environment.

Technical Skills:

- Generative AI Frameworks & Technologies:

- Expertise in Generative AI frameworks, including prompt engineering, fine-tuning, and few-shot learning.

- Familiarity with frameworks such as T5 (Text-to-Text Transfer Transformation), LangChain, Lang Graph, Open-source tech stalk Ollama, Mistral, DeepSeek.

- Strong knowledge of Retrieval-Augmented Generation (RAG) for combining LLMs with external data retrieval systems.

Data Management:

- Experience in designing chunking strategies for different datasets.

- Expertise in data embedding techniques and experience with vector databases like Pinecone, ChromaDB etc

- Programming & AI/ML Libraries:

- Strong programming skills in Python.

- Experience with AI/ML libraries such as TensorFlow, PyTorch, and Hugging Face Transformers.

Cloud Platforms & Integration:

- Familiarity with cloud services for AI/ML workloads (AWS, Azure).

- Experience with API integration for AI services and building scalable applications.

- Certifications (Optional but Desirable):

- Certification in AI/ML (e.g., TensorFlow, AWS Certified Machine Learning Specialty).

- Certification or coursework in Generative AI or related technologies.

Profile - Databricks Developer

Experience- 5+ years

Location- Bangalore (On site)

PF & BGV is Mandatory

Job Description: -

* Design, build, and optimize data pipelines and ETL/ELT workflows using Databricks and Apache Spark (PySpark).

* Develop scalable, high performance data solutions using Spark distributed processing.

* Lead engineering initiatives focused on automation, performance tuning, and platform modernization.

* Implement and manage CI/CD pipelines using Git-based workflows and tools such as GitHub Actions or Jenkins.

* Collaborate with cross-functional teams to translate business needs into technical solutions.

* Ensure data quality, governance, and security across all processes.

* Troubleshoot and optimize Spark jobs, Databricks clusters, and workflows.

* Participate in code reviews and develop reusable engineering frameworks.

* Should have knowledge of utilizing AI tools to improve productivity and support daily engineering activities.

* Strong knowledge and hands-on experience in Databricks Genie, including prompt engineering, workspace usage, and automation

. Required Skills & Experience:

* 5+ years of experience in Data Engineering or related fields.

* Strong hands-on expertise in Databricks (notebooks, Delta Lake, job orchestration).

* Deep knowledge of Apache Spark (PySpark, Spark SQL, optimization techniques).

* Strong proficiency in Python for data processing, automation, and framework development.

* Strong proficiency in SQL, including complex queries, performance tuning, and analytical functions.

* Strong knowledge of Databricks Genie and leveraging it for engineering workflows.

* Strong experience with CI/CD and Git-based development workflows. * Proficiency in data modeling and ETL/ELT pipeline design.

* Experience with automation frameworks and scheduling tools.

* Solid understanding of distributed systems and big data concepts