50+ PySpark Jobs in India

Job Title: PySpark/Scala Developer

Functional Skills: Experience in Credit Risk/Regulatory risk domain

Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting

Good to Have Skills: Exposure to Machine Learning Techniques

Job Description:

5+ Years of experience with Developing/Fine tuning and implementing programs/applications

Using Python/PySpark/Scala on Big Data/Hadoop Platform.

Roles and Responsibilities:

a)     Work with a Leading Bank’s Risk Management team on specific projects/requirements pertaining to risk Models in

 consumer and wholesale banking

b)     Enhance Machine Learning Models using PySpark or Scala

c)     Work with Data Scientists to Build ML Models based on Business Requirements and Follow ML Cycle to Deploy them all

the way to Production Environment

d)     Participate Feature Engineering, Training Models, Scoring and retraining

e)     Architect Data Pipeline and Automate Data Ingestion and Model Jobs

Skills and competencies:

Required:

·       Strong analytical skills in conducting sophisticated statistical analysis using bureau/vendor data, customer performance

Data and macro-economic data to solve business problems.

·       Working experience in languages PySpark & Scala to develop code to validate and implement models and codes in

Credit Risk/Banking

·       Experience with distributed systems such as Hadoop/MapReduce, Spark, streaming data processing, cloud architecture.

• Familiarity with machine learning frameworks and libraries (like scikit-learn, SparkML, tensorflow, pytorch etc.
• Experience in systems integration, web services, batch processing
• Experience in migrating codes to PySpark/Scala is big Plus
• The ability to act as liaison conveying information needs of the business to IT and data constraints to the business

applies equal conveyance regarding business strategy and IT strategy, business processes and work flow

·       Flexibility in approach and thought process

·       Attitude to learn and comprehend the periodical changes in the regulatory requirement as per FED

Skills and competencies:

Required:

·        Strong analytical skills in conducting sophisticated statistical analysis using bureau/vendor data, customer performance

Data and macro-economic data to solve business problems.

·        Working experience in languages PySpark & Scala to develop code to validate and implement models and codes in

Credit Risk/Banking

·        Experience with distributed systems such as Hadoop/MapReduce, Spark, streaming data processing, cloud architecture.

• Familiarity with machine learning frameworks and libraries (like scikit-learn, SparkML, tensorflow, pytorch etc.
• Experience in systems integration, web services, batch processing
• Experience in migrating codes to PySpark/Scala is big Plus
• The ability to act as liaison conveying information needs of the business to IT and data constraints to the business

applies equal conveyance regarding business strategy and IT strategy, business processes and work flow

·        Flexibility in approach and thought process

·        Attitude to learn and comprehend the periodical changes in the regulatory requirement as per FED

Profile: AWS Data Engineer

Mandate skills :AWS + Databricks + Pyspark + SQL role

Location: Bangalore/Pune/Hyderabad/Chennai/Gurgaon:

Notice Period: Immediate

Key Requirements :

• Design, build, and maintain scalable data pipelines to collect, process, and store from multiple datasets.
• Optimize data storage solutions for better performance, scalability, and cost-efficiency.
• Develop and manage ETL/ELT processes to transform data as per schema definitions, apply slicing and dicing, and make it available for downstream jobs and other teams.
• Collaborate closely with cross-functional teams to understand system and product functionalities, pace up feature development, and capture evolving data requirements.
• Engage with stakeholders to gather requirements and create curated datasets for downstream consumption and end-user reporting.
• Automate deployment and CI/CD processes using GitHub workflows, identifying areas to reduce manual, repetitive work.
• Ensure compliance with data governance policies, privacy regulations, and security protocols.
• Utilize cloud platforms like AWS and work on Databricks for data processing with S3 Storage.
• Work with distributed systems and big data technologies such as Spark, SQL, and Delta Lake.
• Integrate with SFTP to push data securely from Databricks to remote locations.
• Analyze and interpret spark query execution plans to fine-tune queries for faster and more efficient processing.
• Strong problem-solving and troubleshooting skills in large-scale distributed systems.

• Data Pipeline Development: Design and implement scalable data pipelines using PySpark and Databricks on AWS cloud infrastructure

• ETL/ELT Operations: Extract, transform, and load data from various sources using Python, SQL, and PySpark for batch and streaming data processing

• Databricks Platform Management: Develop and maintain data workflows, notebooks, and clusters in Databricks environment for efficient data processing

• AWS Cloud Services: Utilize AWS services including S3, Glue, EMR, Redshift, Kinesis, and Lambda for comprehensive data solutions

• Data Transformation: Write efficient PySpark scripts and SQL queries to process large-scale datasets and implement complex business logic

• Data Quality & Monitoring: Implement data validation, quality checks, and monitoring solutions to ensure data integrity across pipelines

• Collaboration: Work closely with data scientists, analysts, and other engineering teams to support analytics and machine learning initiatives

• Performance Optimization: Monitor and optimize data pipeline performance, query efficiency, and resource utilization in Databricks and AWS environments

Required Qualifications:

• Experience: 3+ years of hands-on experience in data engineering, ETL development, or related field

• PySpark Expertise: Strong proficiency in PySpark for large-scale data processing and transformations

• Python Programming: Solid Python programming skills with experience in data manipulation libraries (pandas etc)

• SQL Proficiency: Advanced SQL skills including complex queries, window functions, and performance optimization

• Databricks Experience: Hands-on experience with Databricks platform, including notebook development, cluster management, and job scheduling

• AWS Cloud Services: Working knowledge of core AWS services (S3, Glue, EMR, Redshift, IAM, Lambda)

• Data Modeling: Understanding of dimensional modeling, data warehousing concepts, and ETL best practices

• Version Control: Experience with Git and collaborative development workflows

Preferred Qualifications:

• Education: Bachelor's degree in Computer Science, Engineering, Mathematics, or related technical field

• Advanced AWS: Experience with additional AWS services like Athena, QuickSight, Step Functions, and CloudWatch

• Data Formats: Experience working with various data formats (JSON, Parquet, Avro, Delta Lake)

• Containerization: Basic knowledge of Docker and container orchestration

• Agile Methodology: Experience working in Agile/Scrum development environments

• Business Intelligence Tools: Exposure to BI tools like Tableau, Power BI, or Databricks SQL Analytics

Technical Skills Summary:

Core Technologies:

• PySpark & Spark SQL
• Python (pandas, boto3)
• SQL (PostgreSQL, MySQL, Redshift)
• Databricks (notebooks, clusters, jobs, Delta Lake)

AWS Services:

• S3, Glue, EMR, Redshift
• Lambda, Athena
• IAM, CloudWatch

Development Tools:

• Git/GitHub
• CI/CD pipelines, Docker
• Linux/Unix command line

About Data Axle:

Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for over 50 years in the USA. Data Axle now as an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging proprietary business & consumer databases.

Data Axle Pune is pleased to have achieved certification as a Great Place to Work!

Roles & Responsibilities:

We are looking for a Data Scientist to join the Data Science Client Services team to continue our success of identifying high quality target audiences that generate profitable marketing return for our clients. We are looking for experienced data science, machine learning and MLOps practitioners to design, build and deploy impactful predictive marketing solutions that serve a wide range of verticals and clients. The right candidate will enjoy contributing to and learning from a highly talented team and working on a variety of projects.

We are looking for a Senior Data Scientist who will be responsible for:

1. Ownership of design, implementation, and deployment of machine learning algorithms in a modern Python-based cloud architecture
2. Design or enhance ML workflows for data ingestion, model design, model inference and scoring
3. Oversight on team project execution and delivery
4. If senior, establish peer review guidelines for high quality coding to help develop junior team members’ skill set growth, cross-training, and team efficiencies
5. Visualize and publish model performance results and insights to internal and external audiences

Qualifications:

1. Masters in a relevant quantitative, applied field (Statistics, Econometrics, Computer Science, Mathematics, Engineering)
2. Minimum of 3.5 years of work experience in the end-to-end lifecycle of ML model development and deployment into production within a cloud infrastructure (Databricks is highly preferred)
3. Proven ability to manage the output of a small team in a fast-paced environment and to lead by example in the fulfilment of client requests
4. Exhibit deep knowledge of core mathematical principles relating to data science and machine learning (ML Theory + Best Practices, Feature Engineering and Selection, Supervised and Unsupervised ML, A/B Testing, etc.)
5. Proficiency in Python and SQL required; PySpark/Spark experience a plus
6. Ability to conduct a productive peer review and proper code structure in Github
7. Proven experience developing, testing, and deploying various ML algorithms (neural networks, XGBoost, Bayes, and the like)
8. Working knowledge of modern CI/CD methods This position description is intended to describe the duties most frequently performed by an individual in this position.

It is not intended to be a complete list of assigned duties but to describe a position level.

About Oneture Technologies:

Founded in 2016, Oneture is a cloud-first, full-service digital solutions company, helping clients harness the power of Digital Technologies and Data to drive transformations and turning ideas into business realities. Our team is full of curious, full-stack, innovative thought leaders who are dedicated to providing outstanding customer experiences and building authentic relationships. We are compelled by our core values to drive transformational results from Ideas to Reality for clients across all company sizes, geographies, and industries.

Oneture team delivers full lifecycle solutions—from ideation, project inception, planning through deployment to ongoing support and maintenance. Our core competencies and technical expertise includes Cloud powered: Product Engineering, Big Data and AI ML. Our deep commitment to value creation for our clients and partners and “Startups-like agility with Enterprises-like maturity” philosophy has helped us establish long-term relationships with our clients and enabled us to build and manage mission-critical platforms for them.

About the Role

We are seeking a Data Engineer to design and maintain scalable data pipelines that power finance-related analytics and reporting. This role involves partnering closely with finance, product, and business teams to translate requirements into robust technical solutions while ensuring data accuracy, consistency, and performance.

You will work with modern data technologies—including SQL,Python, Spark, Airflow, and AWS cloud services—to build efficient workflows, optimize large-scale datasets, and deliver business-critical insights.

Role & Responsibilities

• Design, build, and maintain scalable data pipelines for finance-related datasets.
• Partner with finance stakeholders (product & business teams) to understand requirements and translate them into robust technical solutions.
• Ensure data accuracy, consistency, and timeliness across reporting layers and downstream systems.
• Optimize SQL queries and Spark jobs for large-scale datasets to improve performance and cost efficiency.
• Implement scheduling, orchestration, and monitoring of workflows using Airflow (or similar tools).
• Collaborate with cross-functional teams (Finance, Product, Analytics, Engineering) to deliver business-critical solutions.
• Adhere to best practices for coding, testing, and deployment in AWS cloud environments. 

Key Skills & Requirements

• 1.5–3 years of experience as a Data Engineer or in a related role.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Spark (PySpark preferred).
• Experience with Airflow (or other orchestration tools).
• Working knowledge of AWS services (S3, EMR, Glue, Redshift, etc.).
• Strong problem-solving skills with the ability to convert business needs into technical solutions.
• Effective communication skills to collaborate with both technical and non-technical stakeholders.

Good to Have

• Experience with Delta Lake / Data Lake architectures.
• Knowledge of CI/CD, DevOps practices, and observability tools.
• Exposure to SAP RISE environments.

Technical Architect (Databricks)

• 10+ Years Data Engineering Experience with expertise in Databricks
• 3+ years of consulting experience
• Completed Data Engineering Professional certification & required classes
• Minimum 2-3 projects delivered with hands-on experience in Databricks
• Completed Apache Spark Programming with Databricks, Data Engineering with Databricks, Optimizing Apache Spark™ on Databricks
• Experience in Spark and/or Hadoop, Flink, Presto, other popular big data engines
• Familiarity with Databricks multi-hop pipeline architecture

Sr. Data Engineer (Databricks)

• 5+ Years Data Engineering Experience with expertise in Databricks
• Completed Data Engineering Associate certification & required classes
• Minimum 1 project delivered with hands-on experience in development on Databricks
• Completed Apache Spark Programming with Databricks, Data Engineering with Databricks, Optimizing Apache Spark™ on Databricks
• SQL delivery experience, and familiarity with Bigquery, Synapse or Redshift
• Proficient in Python, knowledge of additional databricks programming languages (Scala)

We are seeking a highly skilled Fabric Data Engineer with strong expertise in Azure ecosystem to design, build, and maintain scalable data solutions. The ideal candidate will have hands-on experience with Microsoft Fabric, Databricks, Azure Data Factory, PySpark, SQL, and other Azure services to support advanced analytics and data-driven decision-making.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Microsoft Fabric and Azure data services.
• Implement data integration, transformation, and orchestration workflows with Azure Data Factory, Databricks, and PySpark.
• Work with stakeholders to understand business requirements and translate them into robust data solutions.
• Optimize performance and ensure data quality, reliability, and security across all layers.
• Develop and maintain data models, metadata, and documentation to support analytics and reporting.
• Collaborate with data scientists, analysts, and business teams to deliver insights-driven solutions.
• Stay updated with emerging Azure and Fabric technologies to recommend best practices and innovations.
• Required Skills & Experience
• Proven experience as a Data Engineer with strong expertise in the Azure cloud ecosystem.

Hands-on experience with:

• Microsoft Fabric
• Azure Databricks
• Azure Data Factory (ADF)
• PySpark & Python
• SQL (T-SQL/PL-SQL)
• Solid understanding of data warehousing, ETL/ELT processes, and big data architectures.
• Knowledge of data governance, security, and compliance within Azure.
• Strong problem-solving, debugging, and performance tuning skills.
• Excellent communication and collaboration abilities.

Preferred Qualifications

• Microsoft Certified: Fabric Analytics Engineer Associate / Azure Data Engineer Associate.
• Experience with Power BI, Delta Lake, and Lakehouse architecture.
• Exposure to DevOps, CI/CD pipelines, and Git-based version control.

We are hiring freelancers to work on advanced Data & AI projects using Databricks. If you are passionate about cloud platforms, machine learning, data engineering, or architecture, and want to work with cutting-edge tools on real-world challenges, this is the opportunity for you!

✅ Key Details

• Work Type: Freelance / Contract
• Location: Remote
• Time Zones: IST / EST only
• Domain: Data & AI, Cloud, Big Data, Machine Learning
• Collaboration: Work with industry leaders on innovative projects

🔹 Open Roles

1. Databricks – Senior Consultant

• Skills: Data Warehousing, Python, Java, Scala, ETL, SQL, AWS, GCP, Azure
• Experience: 6+ years

2. Databricks – ML Engineer

• Skills: CI/CD, MLOps, Machine Learning, Spark, Hadoop
• Experience: 4+ years

3. Databricks – Solution Architect

• Skills: Azure, GCP, AWS, CI/CD, MLOps
• Experience: 7+ years

4. Databricks – Solution Consultant

• Skills: SQL, Spark, BigQuery, Python, Scala
• Experience: 2+ years

✅ What We Offer

• Opportunity to work with top-tier professionals and clients
• Exposure to cutting-edge technologies and real-world data challenges
• Flexible remote work environment aligned with IST / EST time zones
• Competitive compensation and growth opportunities

📌 Skills We Value

Cloud Computing | Data Warehousing | Python | Java | Scala | ETL | SQL | AWS | GCP | Azure | CI/CD | MLOps | Machine Learning | Spark |

Job Summary:

We are looking for a highly skilled and experienced Data Engineer with deep expertise in Airflow, dbt, Python, and Snowflake. The ideal candidate will be responsible for designing, building, and managing scalable data pipelines and transformation frameworks to enable robust data workflows across the organization.

Key Responsibilities:

• Design and implement scalable ETL/ELT pipelines using Apache Airflow for orchestration.
• Develop modular and maintainable data transformation models using dbt.
• Write high-performance data processing scripts and automation using Python.
• Build and maintain data models and pipelines on Snowflake.
• Collaborate with data analysts, data scientists, and business teams to deliver clean, reliable, and timely data.
• Monitor and optimize pipeline performance and troubleshoot issues proactively.
• Follow best practices in version control, testing, and CI/CD for data projects.

Must-Have Skills:

• Strong hands-on experience with Apache Airflow for scheduling and orchestrating data workflows.
• Proficiency in dbt (data build tool) for building scalable and testable data models.
• Expert-level skills in Python for data processing and automation.
• Solid experience with Snowflake, including SQL performance tuning, data modeling, and warehouse management.
• Strong understanding of data engineering best practices including modularity, testing, and deployment.

Good to Have:

• Experience working with cloud platforms (AWS/GCP/Azure).
• Familiarity with CI/CD pipelines for data (e.g., GitHub Actions, GitLab CI).
• Exposure to modern data stack tools (e.g., Fivetran, Stitch, Looker).
• Knowledge of data security and governance best practices.

Note : One face-to-face (F2F) round is mandatory, and as per the process, you will need to visit the office for this.

Job Description: Data Engineer

Location: Ahmedabad

Experience: 1+

Employment Type: Full-Time

We are looking for a highly motivated and experienced Data Engineer to join our team. As a Data Engineer, you will play a critical role in designing, building, and optimizing data pipelines that ensure the availability, reliability, and performance of our data infrastructure. You will collaborate closely with data scientists, analysts, and cross-functional teams to provide timely and efficient data solutions.

Responsibilities

● Design and optimize data pipelines for various data sources

● Design and implement efficient data storage and retrieval mechanisms

● Develop data modelling solutions and data validation mechanisms

● Troubleshoot data-related issues and recommend process improvements

● Collaborate with data scientists and stakeholders to provide data-driven insights and solutions

● Coach and mentor junior data engineers in the team

Skills Required: 

● Minimum 1 year of experience in data engineering or related field

● Proficient in designing and optimizing data pipelines and data modeling

● Strong programming expertise in Python

● Hands-on experience with big data technologies such as Hadoop, Spark, and Hive

● Extensive experience with cloud data services such as AWS, Azure, and GCP

● Advanced knowledge of database technologies like SQL, NoSQL, and data warehousing

● Knowledge of distributed computing and storage systems

● Familiarity with DevOps practices and power automate and Microsoft Fabric will be an added advantage

● Strong analytical and problem-solving skills with outstanding communication and collaboration abilities

Qualifications

Bachelor's degree in Computer Science, Data Science, or a Computer related field

Job Description: 

• 4+ years of experience in a Data Engineer role,
• Experience with object-oriented/object function scripting languages: Python, Scala, Golang, Java, etc.
• Experience with Big data tools such as Spark, Hadoop/ Kafka/ Airflow/Hive
• Experience with Streaming data: Spark/Kinesis/Kafka/Pubsub/Event Hub
• Experience with GCP/Azure data factory/AWS
• Strong in SQL Scripting
• Experience with ETL tools
• Knowledge of Snowflake Data Warehouse
• Knowledge of Orchestration frameworks: Airflow/Luig
• Good to have knowledge of Data Quality Management frameworks
• Good to have knowledge of Master Data Management
• Self-learning abilities are a must
• Familiarity with upcoming new technologies is a strong plus.
• Should have a bachelor's degree in big data analytics, computer engineering, or a related field

Personal Competency:

• Strong communication skills is a MUST
• Self-motivated, detail-oriented
• Strong organizational skills
• Ability to prioritize workloads and meet deadlines

Job Description

Overview:

We are seeking an experienced Azure Data Engineer to join our team in a hybrid Developer/Support capacity. This role focuses on enhancing and supporting existing Data & Analytics solutions by leveraging Azure Data Engineering technologies. The engineer will work on developing, maintaining, and deploying IT products and solutions that serve various business users, with a strong emphasis on performance, scalability, and reliability.

Must-Have Skills:

Azure Databricks

PySpark

Azure Synapse Analytics

Key Responsibilities:

• Incident classification and prioritization
• Log analysis and trend identification
• Coordination with Subject Matter Experts (SMEs)
• Escalation of unresolved or complex issues
• Root cause analysis and permanent resolution implementation
• Stakeholder communication and status updates
• Resolution of complex and major incidents
• Code reviews (Per week 2 per individual) to ensure adherence to standards and optimize performance
• Bug fixing of recurring or critical issues identified during operations
• Gold layer tasks, including enhancements and performance tuning.
• Design, develop, and support data pipelines and solutions using Azure data engineering services.
• Implement data flow and ETL techniques leveraging Azure Data Factory, Databricks, and Synapse.
• Cleanse, transform, and enrich datasets using Databricks notebooks and PySpark.

• Orchestrate and automate workflows across services and systems.
• Collaborate with business and technical teams to deliver robust and scalable data solutions.
• Work in a support role to resolve incidents, handle change/service requests, and monitor performance.
• Contribute to CI/CD pipeline implementation using Azure DevOps.

Technical Requirements:

• 4 to 6 years of experience in IT and Azure data engineering technologies.
• Strong experience in Azure Databricks, Azure Synapse, and ADLS Gen2.
• Proficient in Python, PySpark, and SQL.
• Experience with file formats such as JSON and Parquet.
• Working knowledge of database systems, with a preference for Teradata and Snowflake.
• Hands-on experience with Azure DevOps and CI/CD pipeline deployments.
• Understanding of Data Warehousing concepts and data modeling best practices.
• Familiarity with SNOW (ServiceNow) for incident and change management.

Non-Technical Requirements:

• Ability to work independently and collaboratively in virtual teams across geographies.
• Strong analytical and problem-solving skills.
• Experience in Agile development practices, including estimation, testing, and deployment.
• Effective task and time management with the ability to prioritize under pressure.
• Clear communication and documentation skills for project updates and technical processes.

Technologies:

• Azure Data Factory
• Azure Databricks
• Azure Synapse Analytics
• PySpark / SQL
• Azure Data Lake Storage (ADLS), Blob Storage
• Azure DevOps (CI/CD pipelines)

Nice-to-Have:

• Experience with Business Intelligence tools, preferably Power BI
• DP-203 certification (Azure Data Engineer Associate) 

NOTE -

Weekly rotational shifts -

11am to 8pm

2pm to 11pm

5pm to 2 am

P.S. - In any one weekend they should be available in call. If there is any issues alone they should work on that. there will be on call support monthly once.

 Key Responsibilities

• Design and implement ETL/ELT pipelines using Databricks, PySpark, and AWS Glue
• Develop and maintain scalable data architectures on AWS (S3, EMR, Lambda, Redshift, RDS)
• Perform data wrangling, cleansing, and transformation using Python and SQL
• Collaborate with data scientists to integrate Generative AI models into analytics workflows
• Build dashboards and reports to visualize insights using tools like Power BI or Tableau
• Ensure data quality, governance, and security across all data assets
• Optimize performance of data pipelines and troubleshoot bottlenecks
• Work closely with stakeholders to understand data requirements and deliver actionable insights

🧪 Required Skills

Skill AreaTools & TechnologiesCloud PlatformsAWS (S3, Lambda, Glue, EMR, Redshift)Big DataDatabricks, Apache Spark, PySparkProgrammingPython, SQLData EngineeringETL/ELT, Data Lakes, Data WarehousingAnalyticsData Modeling, Visualization, BI ReportingGen AI IntegrationOpenAI, Hugging Face, LangChain (preferred)DevOps (Bonus)Git, Jenkins, Terraform, Docker

📚 Qualifications

• Bachelor's or Master’s degree in Computer Science, Data Science, or related field
• 3+ years of experience in data engineering or data analytics
• Hands-on experience with Databricks, PySpark, and AWS
• Familiarity with Generative AI tools and frameworks is a strong plus
• Strong problem-solving and communication skills

🌟 Preferred Traits

• Analytical mindset with attention to detail
• Passion for data and emerging technologies
• Ability to work independently and in cross-functional teams
• Eagerness to learn and adapt in a fast-paced environment

Job Title: Python Developer

Location: Bangalore

Experience: 5–7 Years

Employment Type: Full-Time

Job Description:

We are seeking an experienced Python Developer with strong proficiency in data analysis tools and PySpark, along with a solid understanding of SQL syntax. The ideal candidate will work on large-scale data processing and analysis tasks within a fast-paced environment.

Key Requirements:

Python: Hands-on experience with Python, specifically in data analysis using libraries such as pandas, numpy, etc.

PySpark: Proficiency in writing efficient PySpark code for distributed data processing.

SQL: Strong knowledge of SQL syntax and experience in writing optimized queries.

Ability to work independently and collaborate effectively with cross-functional teams.

🔍 Job Description:

We are looking for an experienced and highly skilled Technical Lead to guide the development and enhancement of a large-scale Data Observability solution built on AWS. This platform is pivotal in delivering monitoring, reporting, and actionable insights across the client's data landscape.

The Technical Lead will drive end-to-end feature delivery, mentor junior engineers, and uphold engineering best practices. The position reports to the Programme Technical Lead / Architect and involves close collaboration to align on platform vision, technical priorities, and success KPIs.

🎯 Key Responsibilities:

• Lead the design, development, and delivery of features for the data observability solution.
• Mentor and guide junior engineers, promoting technical growth and engineering excellence.
• Collaborate with the architect to align on platform roadmap, vision, and success metrics.
• Ensure high quality, scalability, and performance in data engineering solutions.
• Contribute to code reviews, architecture discussions, and operational readiness.

🔧 Primary Must-Have Skills (Non-Negotiable):

• 5+ years in Data Engineering or Software Engineering roles.
• 3+ years in a technical team or squad leadership capacity.
• Deep expertise in AWS Data Services: Glue, EMR, Kinesis, Lambda, Athena, S3.
• Advanced programming experience with PySpark, Python, and SQL.
• Proven experience in building scalable, production-grade data pipelines on cloud platforms.

Job Title : Data Engineer – GCP + Spark + DBT

Location : Bengaluru (On-site at Client Location | 3 Days WFO)

Experience : 8 to 12 Years

Level : Associate Architect

Type : Full-time

Job Overview :

We are looking for a seasoned Data Engineer to join the Data Platform Engineering team supporting a Unified Data Platform (UDP). This role requires hands-on expertise in DBT, GCP, BigQuery, and PySpark, with a solid foundation in CI/CD, data pipeline optimization, and agile delivery.

Mandatory Skills : GCP, DBT, Google Dataform, BigQuery, PySpark/Spark SQL, Advanced SQL, CI/CD, Git, Agile Methodologies.

Key Responsibilities :

• Design, build, and optimize scalable data pipelines using BigQuery, DBT, and PySpark.
• Leverage GCP-native services like Cloud Storage, Pub/Sub, Dataproc, Cloud Functions, and Composer for ETL/ELT workflows.
• Implement and maintain CI/CD for data engineering projects with Git-based version control.
• Collaborate with cross-functional teams including Infra, Security, and DataOps for reliable, secure, and high-quality data delivery.
• Lead code reviews, mentor junior engineers, and enforce best practices in data engineering.
• Participate in Agile sprints, backlog grooming, and Jira-based project tracking.

Must-Have Skills :

• Strong experience with DBT, Google Dataform, and BigQuery
• Hands-on expertise with PySpark/Spark SQL
• Proficient in GCP for data engineering workflows
• Solid knowledge of SQL optimization, Git, and CI/CD pipelines
• Agile team experience and strong problem-solving abilities

Nice-to-Have Skills :

• Familiarity with Databricks, Delta Lake, or Kafka
• Exposure to data observability and quality frameworks (e.g., Great Expectations, Soda)
• Knowledge of MDM patterns, Terraform, or IaC is a plus

1. Solid Databricks & pyspark experience 

2. Must have worked in projects dealing with data at terabyte scale

3. Must have knowledge of spark optimization techniques 

4. Must have experience setting up job pipelines in Databricks 

5. Basic knowledge of gcp and big query is required 

6. Understanding LLMs and vector db

Job title - Python developer

Exp – 4 to 6 years

Location – Pune/Mum/B’lore

PFB JD

Requirements:

• Proven experience as a Python Developer
• Strong knowledge of core Python and Pyspark concepts
• Experience with web frameworks such as Django or Flask
• Good exposure to any cloud platform (GCP Preferred)
• CI/CD exposure required
• Solid understanding of RESTful APIs and how to build them
• Experience working with databases like Oracle DB and MySQL
• Ability to write efficient SQL queries and optimize database performance
• Strong problem-solving skills and attention to detail
• Strong SQL programing (stored procedure, functions)
• Excellent communication and interpersonal skill

Roles and Responsibilities

• Design, develop, and maintain data pipelines and ETL processes using pyspark
• Work closely with data scientists and analysts to provide them with clean, structured data.
• Optimize data storage and retrieval for performance and scalability.
• Collaborate with cross-functional teams to gather data requirements.
• Ensure data quality and integrity through data validation and cleansing processes.
• Monitor and troubleshoot data-related issues to ensure data pipeline reliability.
• Stay up to date with industry best practices and emerging technologies in data engineering.

🚀 We Are Hiring: Data Engineer | 4+ Years Experience 🚀

Job description

🔍 Job Title: Data Engineer

📍 Location: Ahmedabad

🚀 Work Mode: On-Site Opportunity

📅 Experience: 4+ Years

🕒 Employment Type: Full-Time

⏱️ Availability : Immediate Joiner Preferred

Join Our Team as a Data Engineer

We are seeking a passionate and experienced Data Engineer to be a part of our dynamic and forward-thinking team in Ahmedabad. This is an exciting opportunity for someone who thrives on transforming raw data into powerful insights and building scalable, high-performance data infrastructure.

As a Data Engineer, you will work closely with data scientists, analysts, and cross-functional teams to design robust data pipelines, optimize data systems, and enable data-driven decision-making across the organization.

Your Key Responsibilities

Architect, build, and maintain scalable and reliable data pipelines from diverse data sources.

Design effective data storage, retrieval mechanisms, and data models to support analytics and business needs.

Implement data validation, transformation, and quality monitoring processes.

Collaborate with cross-functional teams to deliver impactful, data-driven solutions.

Proactively identify bottlenecks and optimize existing workflows and processes.

Provide guidance and mentorship to junior engineers in the team.

Skills & Expertise We’re Looking For

3+ years of hands-on experience in Data Engineering or related roles.

Strong expertise in Python and data pipeline design.

Experience working with Big Data tools like Hadoop, Spark, Hive.

Proficiency with SQL, NoSQL databases, and data warehousing solutions.

Solid experience in cloud platforms - Azure

Familiar with distributed computing, data modeling, and performance tuning.

Understanding of DevOps, Power Automate, and Microsoft Fabric is a plus.

Strong analytical thinking, collaboration skills, Excellent Communication Skill and the ability to work independently or as part of a team.

Qualifications

Bachelor’s degree in Computer Science, Data Science, or a related field.

Company Name – Wissen Technology

Group of companies in India – Wissen Technology & Wissen Infotech

Work Location - Senior Backend Developer – Java (with Python Exposure)- Mumbai

Experience - 4 to 10 years

Kindly revert over mail if you are interested.

Java Developer – Job Description

We are seeking a Senior Backend Developer with strong expertise in Java (Spring Boot) and working knowledge of Python. In this role, Java will be your primary development language, with Python used for scripting, automation, or selected service modules. You’ll be part of a collaborative backend team building scalable and high-performance systems.

Key Responsibilities

• Design and develop robust backend services and APIs primarily using Java (Spring Boot)
• Contribute to Python-based components where needed for automation, scripting, or lightweight services
• Build, integrate, and optimize RESTful APIs and microservices
• Work with relational and NoSQL databases
• Write unit and integration tests (JUnit, PyTest)
• Collaborate closely with DevOps, QA, and product teams
• Participate in architecture reviews and design discussions
• Help maintain code quality, organization, and automation

Required Skills & Qualifications

• 4 to 10 years of hands-on Java development experience
• Strong experience with Spring Boot, JPA/Hibernate, and REST APIs
• At least 1–2 years of hands-on experience with Python (e.g., for scripting, automation, or small services)
• Familiarity with Python frameworks like Flask or FastAPI is a plus
• Experience with SQL/NoSQL databases (e.g., PostgreSQL, MongoDB)
• Good understanding of OOP, design patterns, and software engineering best practices
• Familiarity with Docker, Git, and CI/CD pipelines

Job Summary:

As an AWS Data Engineer, you will be responsible for designing, developing, and maintaining scalable, high-performance data pipelines using AWS services. With 6+ years of experience, you’ll collaborate closely with data architects, analysts, and business stakeholders to build reliable, secure, and cost-efficient data infrastructure across the organization.

Key Responsibilities:

• Design, develop, and manage scalable data pipelines using AWS Glue, Lambda, and other serverless technologies
• Implement ETL workflows and transformation logic using PySpark and Python on AWS Glue
• Leverage AWS Redshift for warehousing, performance tuning, and large-scale data queries
• Work with AWS DMS and RDS for database integration and migration
• Optimize data flows and system performance for speed and cost-effectiveness
• Deploy and manage infrastructure using AWS CloudFormation templates
• Collaborate with cross-functional teams to gather requirements and build robust data solutions
• Ensure data integrity, quality, and security across all systems and processes

Required Skills & Experience:

• 6+ years of experience in Data Engineering with strong AWS expertise
• Proficient in Python and PySpark for data processing and ETL development
• Hands-on experience with AWS Glue, Lambda, DMS, RDS, and Redshift
• Strong SQL skills for building complex queries and performing data analysis
• Familiarity with AWS CloudFormation and infrastructure as code principles
• Good understanding of serverless architecture and cost-optimized design
• Ability to write clean, modular, and maintainable code
• Strong analytical thinking and problem-solving skills

🚀 Hiring: Data Engineer | GCP + Spark + Python + .NET |

| 6–10 Yrs | Gurugram (Hybrid)

We’re looking for a skilled Data Engineer with strong hands-on experience in GCP, Spark-Scala, Python, and .NET.

📍 Location: Suncity, Sector 54, Gurugram (Hybrid – 3 days onsite)

💼 Experience: 6–10 Years

⏱️ Notice Period :- Immediate Joiner

Required Skills:

• 5+ years of experience in distributed computing (Spark) and software development.
• 3+ years of experience in Spark-Scala
• 5+ years of experience in Data Engineering.
• 5+ years of experience in Python.
• Fluency in working with databases (preferably Postgres).
• Have a sound understanding of object-oriented programming and development principles.
• Experience working in an Agile Scrum or Kanban development environment.
• Experience working with version control software (preferably Git).
• Experience with CI/CD pipelines.
• Experience with automated testing, including integration/delta, Load, and Performance

About the Role:

We are seeking a talented Lead Data Engineer to join our team and play a pivotal role in transforming raw data into valuable insights. As a Data Engineer, you will design, develop, and maintain robust data pipelines and infrastructure to support our organization's analytics and decision-making processes.

Responsibilities:

• Data Pipeline Development: Build and maintain scalable data pipelines to extract, transform, and load (ETL) data from various sources (e.g., databases, APIs, files) into data warehouses or data lakes.
• Data Infrastructure: Design, implement, and manage data infrastructure components, including data warehouses, data lakes, and data marts.
• Data Quality: Ensure data quality by implementing data validation, cleansing, and standardization processes.
• Team Management: Able to handle team.
• Performance Optimization: Optimize data pipelines and infrastructure for performance and efficiency.
• Collaboration: Collaborate with data analysts, scientists, and business stakeholders to understand their data needs and translate them into technical requirements.
• Tool and Technology Selection: Evaluate and select appropriate data engineering tools and technologies (e.g., SQL, Python, Spark, Hadoop, cloud platforms).
• Documentation: Create and maintain clear and comprehensive documentation for data pipelines, infrastructure, and processes.

Skills:

• Strong proficiency in SQL and at least one programming language (e.g., Python, Java).
• Experience with data warehousing and data lake technologies (e.g., Snowflake, AWS Redshift, Databricks).
• Knowledge of cloud platforms (e.g., AWS, GCP, Azure) and cloud-based data services.
• Understanding of data modeling and data architecture concepts.
• Experience with ETL/ELT tools and frameworks.
• Excellent problem-solving and analytical skills.
• Ability to work independently and as part of a team.

Preferred Qualifications:

• Experience with real-time data processing and streaming technologies (e.g., Kafka, Flink).
• Knowledge of machine learning and artificial intelligence concepts.
• Experience with data visualization tools (e.g., Tableau, Power BI).
• Certification in cloud platforms or data engineering.

Skill Name: ETL Automation Testing

Location: Bangalore, Chennai and Pune

Experience: 5+ Years

Required:

Experience in ETL Automation Testing

Strong experience in Pyspark.

Required Skills:

• Hands-on experience with Databricks, PySpark
• Proficiency in SQL, Python, and Spark.
• Understanding of data warehousing concepts and data modeling.
• Experience with CI/CD pipelines and version control (e.g., Git).
• Fundamental knowledge of any cloud services, preferably Azure or GCP.

Good to Have:

• Bigquery
• Experience with performance tuning and data governance.

About the Role:

We are seeking a talented Lead Data Engineer to join our team and play a pivotal role in transforming raw data into valuable insights. As a Data Engineer, you will design, develop, and maintain robust data pipelines and infrastructure to support our organization's analytics and decision-making processes.

Responsibilities:

• Data Pipeline Development: Build and maintain scalable data pipelines to extract, transform, and load (ETL) data from various sources (e.g., databases, APIs, files) into data warehouses or data lakes.
• Data Infrastructure: Design, implement, and manage data infrastructure components, including data warehouses, data lakes, and data marts.
• Data Quality: Ensure data quality by implementing data validation, cleansing, and standardization processes.
• Team Management: Able to handle team.
• Performance Optimization: Optimize data pipelines and infrastructure for performance and efficiency.
• Collaboration: Collaborate with data analysts, scientists, and business stakeholders to understand their data needs and translate them into technical requirements.
• Tool and Technology Selection: Evaluate and select appropriate data engineering tools and technologies (e.g., SQL, Python, Spark, Hadoop, cloud platforms).
• Documentation: Create and maintain clear and comprehensive documentation for data pipelines, infrastructure, and processes.

 Skills:

• Strong proficiency in SQL and at least one programming language (e.g., Python, Java).
• Experience with data warehousing and data lake technologies (e.g., Snowflake, AWS Redshift, Databricks).
• Knowledge of cloud platforms (e.g., AWS, GCP, Azure) and cloud-based data services.
• Understanding of data modeling and data architecture concepts.
• Experience with ETL/ELT tools and frameworks.
• Excellent problem-solving and analytical skills.
• Ability to work independently and as part of a team.

Preferred Qualifications:

• Experience with real-time data processing and streaming technologies (e.g., Kafka, Flink).
• Knowledge of machine learning and artificial intelligence concepts.
• Experience with data visualization tools (e.g., Tableau, Power BI).
• Certification in cloud platforms or data engineering.

Position: AWS Data Engineer

Experience: 5 to 7 Years

Location: Bengaluru, Pune, Chennai, Mumbai, Gurugram

Work Mode: Hybrid (3 days work from office per week)

Employment Type: Full-time

About the Role:

We are seeking a highly skilled and motivated AWS Data Engineer with 5–7 years of experience in building and optimizing data pipelines, architectures, and data sets. The ideal candidate will have strong experience with AWS services including Glue, Athena, Redshift, Lambda, DMS, RDS, and CloudFormation. You will be responsible for managing the full data lifecycle from ingestion to transformation and storage, ensuring efficiency and performance.

Key Responsibilities:

• Design, develop, and optimize scalable ETL pipelines using AWS Glue, Python/PySpark, and SQL.
• Work extensively with AWS services such as Glue, Athena, Lambda, DMS, RDS, Redshift, CloudFormation, and other serverless technologies.
• Implement and manage data lake and warehouse solutions using AWS Redshift and S3.
• Optimize data models and storage for cost-efficiency and performance.
• Write advanced SQL queries to support complex data analysis and reporting requirements.
• Collaborate with stakeholders to understand data requirements and translate them into scalable solutions.
• Ensure high data quality and integrity across platforms and processes.
• Implement CI/CD pipelines and best practices for infrastructure as code using CloudFormation or similar tools.

Required Skills & Experience:

• Strong hands-on experience with Python or PySpark for data processing.
• Deep knowledge of AWS Glue, Athena, Lambda, Redshift, RDS, DMS, and CloudFormation.
• Proficiency in writing complex SQL queries and optimizing them for performance.
• Familiarity with serverless architectures and AWS best practices.
• Experience in designing and maintaining robust data architectures and data lakes.
• Ability to troubleshoot and resolve data pipeline issues efficiently.
• Strong communication and stakeholder management skills.

Position: AWS Data Engineer

Experience: 5 to 7 Years

Location: Bengaluru, Pune, Chennai, Mumbai, Gurugram

Work Mode: Hybrid (3 days work from office per week)

Employment Type: Full-time

About the Role:

We are seeking a highly skilled and motivated AWS Data Engineer with 5–7 years of experience in building and optimizing data pipelines, architectures, and data sets. The ideal candidate will have strong experience with AWS services including Glue, Athena, Redshift, Lambda, DMS, RDS, and CloudFormation. You will be responsible for managing the full data lifecycle from ingestion to transformation and storage, ensuring efficiency and performance.

Key Responsibilities:

• Design, develop, and optimize scalable ETL pipelines using AWS Glue, Python/PySpark, and SQL.
• Work extensively with AWS services such as Glue, Athena, Lambda, DMS, RDS, Redshift, CloudFormation, and other serverless technologies.
• Implement and manage data lake and warehouse solutions using AWS Redshift and S3.
• Optimize data models and storage for cost-efficiency and performance.
• Write advanced SQL queries to support complex data analysis and reporting requirements.
• Collaborate with stakeholders to understand data requirements and translate them into scalable solutions.
• Ensure high data quality and integrity across platforms and processes.
• Implement CI/CD pipelines and best practices for infrastructure as code using CloudFormation or similar tools.

Required Skills & Experience:

• Strong hands-on experience with Python or PySpark for data processing.
• Deep knowledge of AWS Glue, Athena, Lambda, Redshift, RDS, DMS, and CloudFormation.
• Proficiency in writing complex SQL queries and optimizing them for performance.
• Familiarity with serverless architectures and AWS best practices.
• Experience in designing and maintaining robust data architectures and data lakes.
• Ability to troubleshoot and resolve data pipeline issues efficiently.
• Strong communication and stakeholder management skills.

Job Summary:

Seeking a seasoned SQL + ETL Developer with 4+ years of experience in managing large-scale datasets and cloud-based data pipelines. The ideal candidate is hands-on with MySQL, PySpark, AWS Glue, and ETL workflows, with proven expertise in AWS migration and performance optimization.

Key Responsibilities:

• Develop and optimize complex SQL queries and stored procedures to handle large datasets (100+ million records).
• Build and maintain scalable ETL pipelines using AWS Glue and PySpark.
• Work on data migration tasks in AWS environments.
• Monitor and improve database performance; automate key performance indicators and reports.
• Collaborate with cross-functional teams to support data integration and delivery requirements.
• Write shell scripts for automation and manage ETL jobs efficiently.

Required Skills:

• Strong experience with MySQL, complex SQL queries, and stored procedures.
• Hands-on experience with AWS Glue, PySpark, and ETL processes.
• Good understanding of AWS ecosystem and migration strategies.
• Proficiency in shell scripting.
• Strong communication and collaboration skills.

Nice to Have:

• Working knowledge of Python.
• Experience with AWS RDS.

Profile: AWS Data Engineer

Mode- Hybrid

Experience- 5+7 years

Locations - Bengaluru, Pune, Chennai, Mumbai, Gurugram

Roles and Responsibilities

• Design and maintain ETL pipelines using AWS Glue and Python/PySpark
• Optimize SQL queries for Redshift and Athena
• Develop Lambda functions for serverless data processing
• Configure AWS DMS for database migration and replication
• Implement infrastructure as code with CloudFormation
• Build optimized data models for performance
• Manage RDS databases and AWS service integrations
• Troubleshoot and improve data processing efficiency
• Gather requirements from business stakeholders
• Implement data quality checks and validation
• Document data pipelines and architecture
• Monitor workflows and implement alerting
• Keep current with AWS services and best practices

Required Technical Expertise:

• Python/PySpark for data processing
• AWS Glue for ETL operations
• Redshift and Athena for data querying
• AWS Lambda and serverless architecture
• AWS DMS and RDS management
• CloudFormation for infrastructure
• SQL optimization and performance tuning

Job Overview:

We are seeking an experienced AWS Data Engineer to join our growing data team. The ideal candidate will have hands-on experience with AWS Glue, Redshift, PySpark, and other AWS services to build robust, scalable data pipelines. This role is perfect for someone passionate about data engineering, automation, and cloud-native development.

Key Responsibilities:

• Design, build, and maintain scalable and efficient ETL pipelines using AWS Glue, PySpark, and related tools.
• Integrate data from diverse sources and ensure its quality, consistency, and reliability.
• Work with large datasets in structured and semi-structured formats across cloud-based data lakes and warehouses.
• Optimize and maintain data infrastructure, including Amazon Redshift, for high performance.
• Collaborate with data analysts, data scientists, and product teams to understand data requirements and deliver solutions.
• Automate data validation, transformation, and loading processes to support real-time and batch data processing.
• Monitor and troubleshoot data pipeline issues and ensure smooth operations in production environments.

Required Skills:

• 5 to 7 years of hands-on experience in data engineering roles.
• Strong proficiency in Python and PySpark for data transformation and scripting.
• Deep understanding and practical experience with AWS Glue, AWS Redshift, S3, and other AWS data services.
• Solid understanding of SQL and database optimization techniques.
• Experience working with large-scale data pipelines and high-volume data environments.
• Good knowledge of data modeling, warehousing, and performance tuning.

Preferred/Good to Have:

• Experience with workflow orchestration tools like Airflow or Step Functions.
• Familiarity with CI/CD for data pipelines.
• Knowledge of data governance and security best practices on AWS.

Role - ETL Developer

Work Mode - Hybrid

Experience- 4+ years

Location - Pune, Gurgaon, Bengaluru, Mumbai

Required Skills - AWS, AWS Glue, Pyspark, ETL, SQL

Required Skills:

• 4+ years of hands-on experience in MySQL, including SQL queries and procedure development
• Experience in Pyspark, AWS, AWS Glue
• Experience in AWS ,Migration
• Experience with automated scripting and tracking KPIs/metrics for database performance
• Proficiency in shell scripting and ETL.
• Strong communication skills and a collaborative team player
• Knowledge of Python and AWS RDS is a plus

Job Description: Data Engineer 

Position Overview:

Role Overview

We are seeking a skilled Python Data Engineer with expertise in designing and implementing data solutions using the AWS cloud platform. The ideal candidate will be responsible for building and maintaining scalable, efficient, and secure data pipelines while leveraging Python and AWS services to enable robust data analytics and decision-making processes.

Key Responsibilities

· Design, develop, and optimize data pipelines using Python and AWS services such as Glue, Lambda, S3, EMR, Redshift, Athena, and Kinesis.

· Implement ETL/ELT processes to extract, transform, and load data from various sources into centralized repositories (e.g., data lakes or data warehouses).

· Collaborate with cross-functional teams to understand business requirements and translate them into scalable data solutions.

· Monitor, troubleshoot, and enhance data workflows for performance and cost optimization.

· Ensure data quality and consistency by implementing validation and governance practices.

· Work on data security best practices in compliance with organizational policies and regulations.

· Automate repetitive data engineering tasks using Python scripts and frameworks.

· Leverage CI/CD pipelines for deployment of data workflows on AWS.

Role: GCP Data Engineer

Notice Period: Immediate Joiners

Experience: 5+ years

Location: Remote

Company: Deqode

About Deqode

At Deqode, we work with next-gen technologies to help businesses solve complex data challenges. Our collaborative teams build reliable, scalable systems that power smarter decisions and real-time analytics.

Key Responsibilities

• Build and maintain scalable, automated data pipelines using Python, PySpark, and SQL.
• Work on cloud-native data infrastructure using Google Cloud Platform (BigQuery, Cloud Storage, Dataflow).
• Implement clean, reusable transformations using DBT and Databricks.
• Design and schedule workflows using Apache Airflow.
• Collaborate with data scientists and analysts to ensure downstream data usability.
• Optimize pipelines and systems for performance and cost-efficiency.
• Follow best software engineering practices: version control, unit testing, code reviews, CI/CD.
• Manage and troubleshoot data workflows in Linux environments.
• Apply data governance and access control via Unity Catalog or similar tools.

Required Skills & Experience

• Strong hands-on experience with PySpark, Spark SQL, and Databricks.
• Solid understanding of GCP services (BigQuery, Cloud Functions, Dataflow, Cloud Storage).
• Proficiency in Python for scripting and automation.
• Expertise in SQL and data modeling.
• Experience with DBT for data transformations.
• Working knowledge of Airflow for workflow orchestration.
• Comfortable with Linux-based systems for deployment and troubleshooting.
• Familiar with Git for version control and collaborative development.
• Understanding of data pipeline optimization, monitoring, and debugging.

Work Mode: Hybrid

Need B.Tech, BE, M.Tech, ME candidates - Mandatory

Must-Have Skills:

● Educational Qualification :- B.Tech, BE, M.Tech, ME in any field.

● Minimum of 3 years of proven experience as a Data Engineer.

● Strong proficiency in Python programming language and SQL.

● Experience in DataBricks and setting up and managing data pipelines, data warehouses/lakes.

● Good comprehension and critical thinking skills.

● Kindly note Salary bracket will vary according to the exp. of the candidate - 

- Experience from 4 yrs to 6 yrs - Salary upto 22 LPA

- Experience from 5 yrs to 8 yrs - Salary upto 30 LPA

- Experience more than 8 yrs - Salary upto 40 LPA

We are looking for a skilled and passionate Data Engineers with a strong foundation in Python programming and hands-on experience working with APIs, AWS cloud, and modern development practices. The ideal candidate will have a keen interest in building scalable backend systems and working with big data tools like PySpark.

Key Responsibilities:

• Write clean, scalable, and efficient Python code.
• Work with Python frameworks such as PySpark for data processing.
• Design, develop, update, and maintain APIs (RESTful).
• Deploy and manage code using GitHub CI/CD pipelines.
• Collaborate with cross-functional teams to define, design, and ship new features.
• Work on AWS cloud services for application deployment and infrastructure.
• Basic database design and interaction with MySQL or DynamoDB.
• Debugging and troubleshooting application issues and performance bottlenecks.

Required Skills & Qualifications:

• 4+ years of hands-on experience with Python development.
• Proficient in Python basics with a strong problem-solving approach.
• Experience with AWS Cloud services (EC2, Lambda, S3, etc.).
• Good understanding of API development and integration.
• Knowledge of GitHub and CI/CD workflows.
• Experience in working with PySpark or similar big data frameworks.
• Basic knowledge of MySQL or DynamoDB.
• Excellent communication skills and a team-oriented mindset.

Nice to Have:

• Experience in containerization (Docker/Kubernetes).
• Familiarity with Agile/Scrum methodologies.

Azure DE

Primary Responsibilities -

• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure Create data models for analytics purposes
• Utilizing Azure Data Factory or comparable technologies, create and maintain ETL (Extract, Transform, Load) operations
• Use Azure Data Factory and Databricks to assemble large, complex data sets
• Implementing data validation and cleansing procedures will ensure the quality, integrity, and dependability of the data.
• Ensure data security and compliance
• Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures

Required skills:

• Blend of technical expertise, analytical problem-solving, and collaboration with cross-functional teams
• Azure DevOps
• Apache Spark, Python
• SQL proficiency
• Azure Databricks knowledge
• Big data technologies

The DEs should be well versed in coding, spark core and data ingestion using Azure. Moreover, they need to be decent in terms of communication skills. They should also have core Azure DE skills and coding skills (pyspark, python and SQL).

Out of the 7 open demands, 5 of The Azure Data Engineers should have minimum 5 years of relevant Data Engineering experience.

We are looking for a Senior Data Engineer with strong expertise in GCP, Databricks, and Airflow to design and implement a GCP Cloud Native Data Processing Framework. The ideal candidate will work on building scalable data pipelines and help migrate existing workloads to a modern framework.

• Shift: 2 PM 11 PM
• Work Mode: Hybrid (3 days a week) across Xebia locations
• Notice Period: Immediate joiners or those with a notice period of up to 30 days

Key Responsibilities:

• Design and implement a GCP Native Data Processing Framework leveraging Spark and GCP Cloud Services.
• Develop and maintain data pipelines using Databricks and Airflow for transforming Raw → Silver → Gold data layers.
• Ensure data integrity, consistency, and availability across all systems.
• Collaborate with data engineers, analysts, and stakeholders to optimize performance.
• Document standards and best practices for data engineering workflows.

Required Experience:

• 7-8 years of experience in data engineering, architecture, and pipeline development.
• Strong knowledge of GCP, Databricks, PySpark, and BigQuery.
• Experience with Orchestration tools like Airflow, Dagster, or GCP equivalents.
• Understanding of Data Lake table formats (Delta, Iceberg, etc.).
• Proficiency in Python for scripting and automation.
• Strong problem-solving skills and collaborative mindset.

⚠️ Please apply only if you have not applied recently or are not currently in the interview process for any open roles at Xebia.

Looking forward to your response!

Best regards,

Vijay S

Assistant Manager - TAG

https://www.linkedin.com/in/vijay-selvarajan/

Here is the Job Description - 

Location -- Viman Nagar, Pune

Mode - 5 Days Working

Required Tech Skills:

 ● Strong at PySpark, Python

 ● Good understanding of Data Structure 

 ● Good at SQL query/optimization 

 ● Strong fundamentals of OOPs programming 

 ● Good understanding of AWS Cloud, Big Data. 

 ● Data Lake, AWS Glue, Athena, S3, Kinesis, SQL/NoSQL DB  

The Sr AWS/Azure/GCP Databricks Data Engineer at Koantek will use comprehensive

modern data engineering techniques and methods with Advanced Analytics to support

business decisions for our clients. Your goal is to support the use of data-driven insights

to help our clients achieve business outcomes and objectives. You can collect, aggregate, and analyze structured/unstructured data from multiple internal and external sources and

patterns, insights, and trends to decision-makers. You will help design and build data

pipelines, data streams, reporting tools, information dashboards, data service APIs, data

generators, and other end-user information portals and insight tools. You will be a critical

part of the data supply chain, ensuring that stakeholders can access and manipulate data

for routine and ad hoc analysis to drive business outcomes using Advanced Analytics. You are expected to function as a productive member of a team, working and

communicating proactively with engineering peers, technical lead, project managers, product owners, and resource managers. Requirements:

 Strong experience as an AWS/Azure/GCP Data Engineer and must have

AWS/Azure/GCP Databricks experience.  Expert proficiency in Spark Scala, Python, and spark

 Must have data migration experience from on-prem to cloud

 Hands-on experience in Kinesis to process & analyze Stream Data, Event/IoT Hubs, and Cosmos

 In depth understanding of Azure/AWS/GCP cloud and Data lake and Analytics

solutions on Azure.  Expert level hands-on development Design and Develop applications on Databricks.  Extensive hands-on experience implementing data migration and data processing

using AWS/Azure/GCP services

 In depth understanding of Spark Architecture including Spark Streaming, Spark Core, Spark SQL, Data Frames, RDD caching, Spark MLib

 Hands-on experience with the Technology stack available in the industry for data

management, data ingestion, capture, processing, and curation: Kafka, StreamSets, Attunity, GoldenGate, Map Reduce, Hadoop, Hive, Hbase, Cassandra, Spark, Flume, Hive, Impala, etc

 Hands-on knowledge of data frameworks, data lakes and open-source projects such

asApache Spark, MLflow, and Delta Lake

 Good working knowledge of code versioning tools [such as Git, Bitbucket or SVN]

 Hands-on experience in using Spark SQL with various data sources like JSON, Parquet and Key Value Pair

 Experience preparing data for Data Science and Machine Learning with exposure to- model selection, model lifecycle, hyperparameter tuning, model serving, deep

learning, etc

 Demonstrated experience preparing data, automating and building data pipelines for

AI Use Cases (text, voice, image, IoT data etc. ).  Good to have programming language experience with. NET or Spark/Scala

 Experience in creating tables, partitioning, bucketing, loading and aggregating data

using Spark Scala, Spark SQL/PySpark

 Knowledge of AWS/Azure/GCP DevOps processes like CI/CD as well as Agile tools

and processes including Git, Jenkins, Jira, and Confluence

 Working experience with Visual Studio, PowerShell Scripting, and ARM templates.  Able to build ingestion to ADLS and enable BI layer for Analytics

 Strong understanding of Data Modeling and defining conceptual logical and physical

data models.  Big Data/analytics/information analysis/database management in the cloud

 IoT/event-driven/microservices in the cloud- Experience with private and public cloud

architectures, pros/cons, and migration considerations.  Ability to remain up to date with industry standards and technological advancements

that will enhance data quality and reliability to advance strategic initiatives

 Working knowledge of RESTful APIs, OAuth2 authorization framework and security

best practices for API Gateways

 Guide customers in transforming big data projects, including development and

deployment of big data and AI applications

 Guide customers on Data engineering best practices, provide proof of concept, architect solutions and collaborate when needed

 2+ years of hands-on experience designing and implementing multi-tenant solutions

using AWS/Azure/GCP Databricks for data governance, data pipelines for near real-

time data warehouse, and machine learning solutions.  Over all 5+ years' experience in a software development, data engineering, or data

analytics field using Python, PySpark, Scala, Spark, Java, or equivalent technologies.  hands-on expertise in Apache SparkTM (Scala or Python)

 3+ years of experience working in query tuning, performance tuning, troubleshooting, and debugging Spark and other big data solutions.  Bachelor's or Master's degree in Big Data, Computer Science, Engineering, Mathematics, or similar area of study or equivalent work experience

 Ability to manage competing priorities in a fast-paced environment

 Ability to resolve issues

 Basic experience with or knowledge of agile methodologies

 AWS Certified: Solutions Architect Professional

 Databricks Certified Associate Developer for Apache Spark

 Microsoft Certified: Azure Data Engineer Associate

 GCP Certified: Professional Google Cloud Certified

Key Responsibilities:

Design, develop, and optimize scalable data pipelines and ETL processes.

Work with large datasets using GCP services like BigQuery, Dataflow, and Cloud Storage.

Implement real-time data streaming and processing solutions using Pub/Sub and Dataproc.

Collaborate with cross-functional teams to ensure data quality and governance.

Technical Requirements:

4+ years of experience in Data Engineering.

Strong expertise in GCP services like Workflow,tensorflow, Dataproc, and Cloud Storage.

Proficiency in SQL and programming languages such as Python or Java

.Experience in designing and implementing data pipelines

and working with real-time data processing.

Familiarity with CI/CD pipelines and cloud security best practices.

Job Description :

Job Title : Data Engineer

Location : Pune (Hybrid Work Model)

Experience Required : 4 to 8 Years

Role Overview :

We are seeking talented and driven Data Engineers to join our team in Pune. The ideal candidate will have a strong background in data engineering with expertise in Python, PySpark, and SQL. You will be responsible for designing, building, and maintaining scalable data pipelines and systems that empower our business intelligence and analytics initiatives.

Key Responsibilities:

• Develop, optimize, and maintain ETL pipelines and data workflows.
• Design and implement scalable data solutions using Python, PySpark, and SQL.
• Collaborate with cross-functional teams to gather and analyze data requirements.
• Ensure data quality, integrity, and security throughout the data lifecycle.
• Monitor and troubleshoot data pipelines to ensure reliability and performance.
• Work on hybrid data environments involving on-premise and cloud-based systems.
• Assist in the deployment and maintenance of big data solutions.

Required Skills and Qualifications:

• Bachelor’s degree in Computer Science, Information Technology, or related field.
• 4 to 8 Years of experience in Data Engineering or related roles.
• Proficiency in Python and PySpark for data processing and analysis.
• Strong SQL skills with experience in writing complex queries and optimizing performance.
• Familiarity with data pipeline tools and frameworks.
• Knowledge of cloud platforms such as AWS, Azure, or GCP is a plus.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.

Preferred Qualifications:

• Experience with big data technologies like Hadoop, Hive, or Spark.
• Familiarity with data visualization tools and techniques.
• Knowledge of CI/CD pipelines and DevOps practices in a data engineering context.

Work Model:

• This position follows a hybrid work model, with candidates expected to work from the Pune office as per business needs.

Why Join Us?

• Opportunity to work with cutting-edge technologies.
• Collaborative and innovative work environment.
• Competitive compensation and benefits.
• Clear career progression and growth opportunities.

Job Description

Job Title: Data Engineer

Location: Hyderabad, India

Job Type: Full Time

Experience: 5 – 8 Years

Working Model: On-Site (No remote or work-from-home options available)

Work Schedule: Mountain Time Zone (3:00 PM to 11:00 PM IST)

Role Overview

The Data Engineer will be responsible for designing and implementing scalable backend systems, leveraging Python and PySpark to build high-performance solutions. The role requires a proactive and detail-orientated individual who can solve complex data engineering challenges while collaborating with cross-functional teams to deliver quality results.

Key Responsibilities

• Develop and maintain backend systems using Python and PySpark.
• Optimise and enhance system performance for large-scale data processing.
• Collaborate with cross-functional teams to define requirements and deliver solutions.
• Debug, troubleshoot, and resolve system issues and bottlenecks.
• Follow coding best practices to ensure code quality and maintainability.
• Utilise tools like Palantir Foundry for data management workflows (good to have).

Qualifications

• Strong proficiency in Python backend development.
• Hands-on experience with PySpark for data engineering.
• Excellent problem-solving skills and attention to detail.
• Good communication skills for effective team collaboration.
• Experience with Palantir Foundry or similar platforms is a plus.

Preferred Skills

• Experience with large-scale data processing and pipeline development.
• Familiarity with agile methodologies and development tools.
• Ability to optimise and streamline backend processes effectively.
• 
  

Description

Come Join Us

Experience.com - We make every experience matter more

Position: Senior GCP Data Engineer

Job Location: Chennai (Base Location) / Remote

Employment Type: Full Time

Summary of Position

A Senior Data Engineer is a professional who specializes in preparing big data infrastructure for analytical or operational uses. He/She is responsible for develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity. They collaborate with data scientists and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organisation.

Responsibilities:

• Collaborate with cross-functional teams to define, prioritize, and execute data engineering initiatives aligned with business objectives.
• Design and implement scalable, reliable, and secure data solutions by industry best practices and compliance requirements.
• Drive the adoption of cloud-native technologies and architectural patterns to optimize the performance, cost, and reliability of data pipelines and analytics solutions.
• Mentor and lead a team of Data Engineers.
• Demonstrate a drive to learn and master new technologies and techniques.
• Apply strong problem-solving skills with an emphasis on building data-driven or AI-enhanced products.
• Coordinate with ML/AI and engineering teams to understand data requirements.

Experience & Skills:

• 8+ years of Strong experience in ETL and ELT data from various sources in Data Warehouses
• 8+ years of experience in Python, Pandas, Numpy, and SciPy.
• 5+ years of Experience in GCP 
• 5+ years of Experience in BigQuery, PySpark, and Pub/Sub
• 5+ years of Experience working with and creating data architectures.
• Certified in Google Cloud Professional Data Engineer.
• Advanced proficiency in Google Cloud services such as Dataflow, Dataproc, Dataprep, Data Studio, and Cloud Composer.
• Proficient in writing complex Spark (PySpark) User Defined Functions (UDFs), Spark SQL, and HiveQL.
• Good understanding of Elastic search.
• Experience in assessing and ensuring data quality, data testing, and addressing data quality issues.
• Excellent understanding of Spark architecture and underlying frameworks including storage management.
• Solid background in database design and development, database administration, and software engineering across full life cycles.
• Experience with NoSQL data stores like MongoDB, DocumentDB, and DynamoDB.
• Knowledge of data governance principles and practices, including data lineage, metadata management, and access control mechanisms.
• Experience in implementing and optimizing data security controls, encryption, and compliance measures in GCP environments.
• Ability to troubleshoot complex issues, perform root cause analysis, and implement effective solutions in a timely manner.
• Proficiency in data visualization tools such as Tableau, Looker, or Data Studio to create insightful dashboards and reports for business users.
• Strong communication and interpersonal skills to effectively collaborate with technical and non-technical stakeholders, articulate complex concepts, and drive consensus.
• Experience with agile methodologies and project management tools like Jira or Asana for sprint planning, backlog grooming, and task tracking.

Data Engineer + Integration engineer + Support specialistExp – 5-8 years

Necessary Skills:· SQL & Python / PySpark

· AWS Services: Glue, Appflow, Redshift

· Data warehousing

· Data modelling

Job Description:· Experience of implementing and delivering data solutions and pipelines on AWS Cloud Platform. Design/ implement, and maintain the data architecture for all AWS data services

· A strong understanding of data modelling, data structures, databases (Redshift), and ETL processes

· Work with stakeholders to identify business needs and requirements for data-related projects

Strong SQL and/or Python or PySpark knowledge

· Creating data models that can be used to extract information from various sources & store it in a usable format

· Optimize data models for performance and efficiency

· Write SQL queries to support data analysis and reporting

· Monitor and troubleshoot data pipelines

· Collaborate with software engineers to design and implement data-driven features

· Perform root cause analysis on data issues

· Maintain documentation of the data architecture and ETL processes

· Identifying opportunities to improve performance by improving database structure or indexing methods

· Maintaining existing applications by updating existing code or adding new features to meet new requirements

· Designing and implementing security measures to protect data from unauthorized access or misuse

· Recommending infrastructure changes to improve capacity or performance

Experience in Process industry

Data Engineer + Integration engineer + Support specialistExp – 3-5 years

Necessary Skills:· SQL & Python / PySpark

· AWS Services: Glue, Appflow, Redshift

· Data warehousing basics

· Data modelling basics

Job Description:· Experience of implementing and delivering data solutions and pipelines on AWS Cloud Platform.

· A strong understanding of data modelling, data structures, databases (Redshift)

Strong SQL and/or Python or PySpark knowledge

· Design and implement ETL processes to load data into the data warehouse

· Creating data models that can be used to extract information from various sources & store it in a usable format

· Optimize data models for performance and efficiency

· Write SQL queries to support data analysis and reporting

· Collaborate with team to design and implement data-driven features

· Monitor and troubleshoot data pipelines

· Perform root cause analysis on data issues

· Maintain documentation of the data architecture and ETL processes

· Maintaining existing applications by updating existing code or adding new features to meet new requirements

· Designing and implementing security measures to protect data from unauthorized access or misuse

· Identifying opportunities to improve performance by improving database structure or indexing methods

· Designing and implementing security measures to protect data from unauthorized access or misuse

· Recommending infrastructure changes to improve capacity or performance

Skills / Tools

• SQL & Python / PySpark
• AWS Services: Glue, Appflow, Redshift - Mandatory
• Data warehousing basics
• Data modelling basics
• 
  

Job Description

• Experience of implementing and delivering data solutions and pipelines on AWS Cloud Platform. Design/ implement, and maintain the data architecture for all AWS data services
• A strong understanding of data modelling, data structures, databases (Redshift), and ETL processes
• Work with stakeholders to identify business needs and requirements for data-related project
• Strong SQL and/or Python or PySpark knowledge
• Creating data models that can be used to extract information from various sources & store it in a usable format
• Optimize data models for performance and efficiency
• Write SQL queries to support data analysis and reporting
• Monitor and troubleshoot data pipelines
• Collaborate with software engineers to design and implement data-driven features 
• Perform root cause analysis on data issues 
• Maintain documentation of the data architecture and ETL processes 
• Identifying opportunities to improve performance by improving database structure or indexing methods 
• Maintaining existing applications by updating existing code or adding new features to meet new requirements 
• Designing and implementing security measures to protect data from unauthorized access or misuse 
• Recommending infrastructure changes to improve capacity or performance 
• Experience in Process industry

ob Description: 

We are seeking an experienced Azure Data Engineer with expertise in Azure Data Factory, Azure Databricks, and Azure Data Fabric to lead the migration of our existing data pipeline and processing infrastructure. The ideal candidate will have a strong background in Azure cloud data services, big data analytics, and data engineering, with specific experience in Azure Data Fabric. We are looking for someone who has at least 6 months of hands-on experience with Azure Data Fabric or has successfully completed at least one migration to Azure Data Fabric.

Key Responsibilities:

• Assess the current data architecture using Azure Data Factory and Databricks and develop a detailed migration plan to Azure Data Fabric.
• Design and implement end-to-end data pipelines within Azure Data Fabric, including data ingestion, transformation, storage, and analytics.
• Optimize data workflows to leverage Azure Data Fabric's unified platform for data integration, big data processing, and real-time analytics.
• Ensure seamless integration of data from SharePoint and other sources into Azure Data Fabric, maintaining data quality and integrity.
• Collaborate with business analysts and business stakeholders to align data strategies and optimize the data environment for machine learning and AI workloads.
• Implement security best practices, including data governance, access control, and monitoring within Azure Data Fabric.
• Conduct performance tuning and optimization for data storage and processing within Azure Data Fabric to ensure high availability and cost efficiency.

Key Requirements:

• Proven experience (5+ years) in Azure data engineering with a strong focus on Azure Data Factory and Azure Databricks.
• At least 6 months of hands-on experience with Azure Data Fabric or completion of one migration to Azure Data Fabric.
• Hands-on experience in designing, building, and managing data pipelines, data lakes, and data warehouses on Azure.
• Expertise in Spark, SQL, and data transformation techniques within Azure environments.
• Strong understanding of data governance, security, and compliance in cloud environments.
• Experience with migrating data architectures and optimizing workflows on cloud platforms.
• Ability to work collaboratively with cross-functional teams and communicate technical concepts effectively to non-technical stakeholders.
• Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert) are a plus.

Key requirements:

• The person should have at least 6 months of work experience in Data Fabric.  Make sure the experience is not less than 6 months 
• Solid technical skills: data bricks, data fabric and data factory
• Polished, good communication and interpersonal skills
• The Person should have at least 6 years of experience in Databricks, Datafactory. 

TVARIT GmbH develops and delivers solutions in the field of artificial intelligence (AI) for the Manufacturing, automotive, and process industries. With its software products, TVARIT makes it possible for its customers to make intelligent and well-founded decisions, e.g., in forward-looking Maintenance, increasing the OEE and predictive quality. We have renowned reference customers, competent technology, a good research team from renowned Universities, and the award of a renowned AI prize (e.g., EU Horizon 2020) which makes Tvarit one of the most innovative AI companies in Germany and Europe. 

We are looking for a self-motivated person with a positive "can-do" attitude and excellent oral and written communication skills in English. 

We are seeking a skilled and motivated Data Engineer from the manufacturing Industry with over two years of experience to join our team. As a data engineer, you will be responsible for designing, building, and maintaining the infrastructure required for the collection, storage, processing, and analysis of large and complex data sets. The ideal candidate will have a strong foundation in ETL pipelines and Python, with additional experience in Azure and Terraform being a plus. This role requires a proactive individual who can contribute to our data infrastructure and support our analytics and data science initiatives. 

Skills Required 

• Experience in the manufacturing industry (metal industry is a plus)  
• 2+ years of experience as a Data Engineer 
• Experience in data cleaning & structuring and data manipulation 
• ETL Pipelines: Proven experience in designing, building, and maintaining ETL pipelines. 
• Python: Strong proficiency in Python programming for data manipulation, transformation, and automation. 
• Experience in SQL and data structures  
• Knowledge in big data technologies such as Spark, Flink, Hadoop, Apache and NoSQL databases. 
• Knowledge of cloud technologies (at least one) such as AWS, Azure, and Google Cloud Platform. 
• Proficient in data management and data governance  
• Strong analytical and problem-solving skills. 
• Excellent communication and teamwork abilities. 

Nice To Have 

• Azure: Experience with Azure data services (e.g., Azure Data Factory, Azure Databricks, Azure SQL Database). 
• Terraform: Knowledge of Terraform for infrastructure as code (IaC) to manage cloud. 

Greetings , Wissen Technology is Hiring for the position of Data Engineer

Please find the Job Description for your Reference:

JD

• Design, develop, and maintain data pipelines on AWS EMR (Elastic MapReduce) to support data processing and analytics.
• Implement data ingestion processes from various sources including APIs, databases, and flat files.
• Optimize and tune big data workflows for performance and scalability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Manage and monitor EMR clusters, ensuring high availability and reliability.
• Develop ETL (Extract, Transform, Load) processes to cleanse, transform, and store data in data lakes and data warehouses.
• Implement data security best practices to ensure data is protected and compliant with relevant regulations.
• Create and maintain technical documentation related to data pipelines, workflows, and infrastructure.
• Troubleshoot and resolve issues related to data processing and EMR cluster performance.

Qualifications:

• Bachelor’s degree in Computer Science, Information Technology, or a related field.
• 5+ years of experience in data engineering, with a focus on big data technologies.
• Strong experience with AWS services, particularly EMR, S3, Redshift, Lambda, and Glue.
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with big data frameworks and tools such as Hadoop, Spark, Hive, and Pig.
• Solid understanding of data modeling, ETL processes, and data warehousing concepts.
• Experience with SQL and NoSQL databases.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and the ability to work independently and collaboratively in a team environment