6+ PySpark Jobs in Kolkata | PySpark Job openings in Kolkata

Roles & Responsibilities

• Design, develop, and deliver scalable end-to-end data pipelines using Azure Data Factory, ensuring robust integration

of enterprise-wide data from diverse sources

• Build and optimize data engineering workflows using Databricks and PySpark

• Write efficient, high-performance SQL for data transformation and analysis

• Work with the Azure Cloud platform and associated services, applying strong understanding of data warehousing,

data models, and pipelines

• Provide technical leadership to a team of developers, including code reviews and enforcing best practices across the

development lifecycle

• Oversee CI/CD implementation using Azure DevOps, managing deployments across development, QA, and production

environments with proper change control processes

• Collaborate with cross-functional teams to translate business requirements into scalable data solutions

• Ensure data quality, reliability, and performance across all pipelines and platforms

Ideal Candidate

1Strong Azure Databricks Engineer / Senior Data Engineer Profile

2Mandatory (Experience 1) – Must have minimum 8+ years of overall experience in Data Engineering, Data Development, or related data technology roles, with strong hands-on experience in enterprise data pipeline development.

3Mandatory (Experience 2) – Must have strong hands-on experience with Azure Databricks, including development and optimization of scalable data engineering workflows using Databricks and PySpark.

4Mandatory (Experience 3) – Must have strong hands-on proficiency in PySpark/Python and SQL, with proven experience developing complex data transformations, processing workflows, and performance-optimized queries.

5Mandatory (Experience 4) – Must have hands-on experience with Azure Data Factory (ADF) for designing, developing, and orchestrating end-to-end data pipelines and integrating data from multiple sources.

6Mandatory (Experience 5) – Must have strong experience working on the Azure Cloud platform and associated data services, with solid understanding of data warehousing, data modeling, pipeline architecture, and enterprise data solutions.

7Mandatory (Experience 6) – Must have hands-on experience implementing CI/CD using Azure DevOps, including deployment and release management across development, QA, and production environments.

8Mandatory (Experience 7) – Must have proven technical leadership experience, including code reviews, enforcing development best practices, mentoring developers, and providing technical guidance to a data engineering team.

9Mandatory (Notice Period) – Immediate joiners or candidates who can join within 15 days.

10Mandatory (Note) - The position is open across all Cognizant offices pan India. Candidates must be willing to attend the F2F interview at the nearest Cognizant office location.

Hiring: GCP Data Engineer (FTE)

📍 Location: Bangalore | Chennai | Pune | Gurgaon | Kolkata

💼 Employment Type: Full-Time

Notice Period - Immediate Joiner ( Serving Notice Period )

Work Mode - Hybrid

We are looking for experienced GCP Data Engineers with strong expertise in building scalable cloud data solutions.

Required Skills:

✔ GCP Data Engineering Experience

✔ BigQuery

✔ SQL & Python

✔ PySpark / Apache Spark

✔ Apache Beam / Dataflow

✔ ETL / ELT Pipeline Development

✔ Airflow / Cloud Composer

Responsibilities:

• Design and develop scalable ETL/ELT pipelines on GCP
• Build and optimize BigQuery solutions
• Process large-scale structured/unstructured data using Spark
• Develop automated workflows and cloud-native data pipelines
• Work with GCP services like Dataflow, Dataproc, Cloud Storage, etc.
• 
  

Good to Have:

• Google Cloud Certifications (Professional Data Engineer / Solution Architect)
• Experience with BigTable, Cloud SQL, Spanner, NoSQL databases

🎓 Qualification: Bachelor’s / Master’s in CS, Engineering, or related field

Job Title: PySpark/Scala Developer

Functional Skills: Experience in Credit Risk/Regulatory risk domain

Technical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scripting

Good to Have Skills: Exposure to Machine Learning Techniques

Job Description:

5+ Years of experience with Developing/Fine tuning and implementing programs/applications

Using Python/PySpark/Scala on Big Data/Hadoop Platform.

Roles and Responsibilities:

a)     Work with a Leading Bank’s Risk Management team on specific projects/requirements pertaining to risk Models in

 consumer and wholesale banking

b)     Enhance Machine Learning Models using PySpark or Scala

c)     Work with Data Scientists to Build ML Models based on Business Requirements and Follow ML Cycle to Deploy them all

the way to Production Environment

d)     Participate Feature Engineering, Training Models, Scoring and retraining

e)     Architect Data Pipeline and Automate Data Ingestion and Model Jobs

Skills and competencies:

Required:

·       Strong analytical skills in conducting sophisticated statistical analysis using bureau/vendor data, customer performance

Data and macro-economic data to solve business problems.

·       Working experience in languages PySpark & Scala to develop code to validate and implement models and codes in

Credit Risk/Banking

·       Experience with distributed systems such as Hadoop/MapReduce, Spark, streaming data processing, cloud architecture.

• Familiarity with machine learning frameworks and libraries (like scikit-learn, SparkML, tensorflow, pytorch etc.
• Experience in systems integration, web services, batch processing
• Experience in migrating codes to PySpark/Scala is big Plus
• The ability to act as liaison conveying information needs of the business to IT and data constraints to the business

applies equal conveyance regarding business strategy and IT strategy, business processes and work flow

·       Flexibility in approach and thought process

·       Attitude to learn and comprehend the periodical changes in the regulatory requirement as per FED

Skills and competencies:

Required:

·        Strong analytical skills in conducting sophisticated statistical analysis using bureau/vendor data, customer performance

Data and macro-economic data to solve business problems.

·        Working experience in languages PySpark & Scala to develop code to validate and implement models and codes in

Credit Risk/Banking

·        Experience with distributed systems such as Hadoop/MapReduce, Spark, streaming data processing, cloud architecture.

• Familiarity with machine learning frameworks and libraries (like scikit-learn, SparkML, tensorflow, pytorch etc.
• Experience in systems integration, web services, batch processing
• Experience in migrating codes to PySpark/Scala is big Plus
• The ability to act as liaison conveying information needs of the business to IT and data constraints to the business

applies equal conveyance regarding business strategy and IT strategy, business processes and work flow

·        Flexibility in approach and thought process

·        Attitude to learn and comprehend the periodical changes in the regulatory requirement as per FED

We are looking for a skilled and passionate Data Engineers with a strong foundation in Python programming and hands-on experience working with APIs, AWS cloud, and modern development practices. The ideal candidate will have a keen interest in building scalable backend systems and working with big data tools like PySpark.

Key Responsibilities:

• Write clean, scalable, and efficient Python code.
• Work with Python frameworks such as PySpark for data processing.
• Design, develop, update, and maintain APIs (RESTful).
• Deploy and manage code using GitHub CI/CD pipelines.
• Collaborate with cross-functional teams to define, design, and ship new features.
• Work on AWS cloud services for application deployment and infrastructure.
• Basic database design and interaction with MySQL or DynamoDB.
• Debugging and troubleshooting application issues and performance bottlenecks.

Required Skills & Qualifications:

• 4+ years of hands-on experience with Python development.
• Proficient in Python basics with a strong problem-solving approach.
• Experience with AWS Cloud services (EC2, Lambda, S3, etc.).
• Good understanding of API development and integration.
• Knowledge of GitHub and CI/CD workflows.
• Experience in working with PySpark or similar big data frameworks.
• Basic knowledge of MySQL or DynamoDB.
• Excellent communication skills and a team-oriented mindset.

Nice to Have:

• Experience in containerization (Docker/Kubernetes).
• Familiarity with Agile/Scrum methodologies.

Skills and requirements

• Experience analyzing complex and varied data in a commercial or academic setting.
• Desire to solve new and complex problems every day.
• Excellent ability to communicate scientific results to both technical and non-technical team members.

Desirable

• A degree in a numerically focused discipline such as, Maths, Physics, Chemistry, Engineering or Biological Sciences..
• Hands on experience on Python, Pyspark, SQL
• Hands on experience on building End to End Data Pipelines.
• Hands on Experience on Azure Data Factory, Azure Data Bricks, Data Lake - added advantage
• Hands on Experience in building data pipelines.
• Experience with Bigdata Tools, Hadoop, Hive, Sqoop, Spark, SparkSQL
• Experience with SQL or NoSQL databases for the purposes of data retrieval and management.
• Experience in data warehousing and business intelligence tools, techniques and technology, as well as experience in diving deep on data analysis or technical issues to come up with effective solutions.
• BS degree in math, statistics, computer science or equivalent technical field.
• Experience in data mining structured and unstructured data (SQL, ETL, data warehouse, Machine Learning etc.) in a business environment with large-scale, complex data sets.
• Proven ability to look at solutions in unconventional ways. Sees opportunities to innovate and can lead the way.
• Willing to learn and work on Data Science, ML, AI.