Big Data Spark Lead

Job Title : Senior Data Engineer – Databricks

Experience : 14 to 20 Years

Location : HSR Layout, Bangalore

Work Mode : Hybrid – 3 Days WFO

Shift : 11:30 AM – 07:30 PM IST

Positions : 2

Notice Period : Immediate Joiners Only

Interview : 1 Technical Round + 2 Client Rounds

Role Overview :

We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.

The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.

Must-Have Skills :

• 14 to 20 years of Data Engineering experience
• Databricks & Apache Spark / PySpark
• Python & SQL
• AWS Cloud
• Lakehouse Architecture
• ETL / ELT & Distributed Data Processing
• Batch & Streaming Pipelines
• Data Pipeline Optimization & Data Modeling
• CDC & Incremental Processing
• Git, CI/CD & Testing
• Data Quality, Monitoring & Observability
• Technical Leadership & Stakeholder Management

Key Responsibilities :

• Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
• Own data products from design through production.
• Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
• Optimize pipelines for performance, scalability, reliability, and cost.
• Design scalable data architectures and data models.
• Implement data quality, monitoring, lineage, and CI/CD practices.
• Lead technical discussions and mentor engineering teams.
• Collaborate with business stakeholders, architects, product owners, and engineering teams.
• Remain hands-on while providing technical leadership.

Ideal Candidate :

A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.

🔴 Super Urgent : Only Bangalore-based immediate joiners.

Job Title : Tech Lead – Data Lake Platform

Number of Positions : 2

Experience : 7+ Years

Role Type : Technical Lead / Data Platform Lead

Domain : Data Engineering / Data Platform / AWS

About the Role :

We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.

The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.

As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.

Core Tech Stack :

AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka

Key Responsibilities :

• Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
• Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
• Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
• Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
• Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
• Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
• Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
• Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
• Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
• Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
• Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
• Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.

Must-Have Skills :

• 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
• Strong hands-on experience with AWS Data Services, including:
• Amazon S3
• AWS EMR
• AWS Glue
• Amazon Athena
• Amazon Redshift
• AWS DMS
• AWS Lambda
• Amazon RDS
• AWS IAM
• Strong production experience with Apache Airflow.
• Strong hands-on experience with Apache Spark / PySpark.
• Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
• Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
• Strong understanding of data lake architecture and enterprise data platforms.
• Proven experience leading engineers and driving technical decisions.
• Hands-on experience with production support, troubleshooting, incident management, and RCA.
• Strong understanding of data platform performance, scalability, reliability, and SLA management.

Good-to-Have Skills :

• Experience with DBT and modern data transformation practices.
• Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
• Strong knowledge of Amazon Redshift workload optimization, including :
• Distribution keys
• Sort keys
• Spectrum
• External tables
• Experience with Kafka or other streaming/data ingestion technologies.
• Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
• Experience with Lambda and API Gateway for API-based data serving.
• Experience with enterprise data governance, data quality, security, and compliance.
• Experience in BFSI / Banking / Financial Services / Insurance domain.

Ideal Candidate :

The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.

Job Title: Java + Apache Spark Developer

Experience: 5–10 Years

Location: Mumbai

Work Mode: Hybrid (3 Days WFO)

Job Description:

Looking for hands-on Java Developers with strong experience in Apache Spark and distributed data processing. Candidates should have experience building and optimizing data pipelines, working with Spark SQL, and using Snowflake or Databricks for enterprise data engineering solutions.

Mandatory Skills:

Java, Apache Spark, Spark SQL, Snowflake / Databricks, SQL, ETL.

Technical Stack

- PySpark

- Databricks

- Microsoft Azure

- SQL

- Kubernetes

Key Responsibilities & Expectations

- Must be self-driven, proactive, and capable of independently delivering high-quality solutions.

- Demonstrate strong ownership and accountability for assigned tasks and deliverables.

- Ability to work with minimal supervision while consistently meeting project objectives and quality standards.

Company: Wissen Technology

Position: Databricks Engineer

Experience: 6-10 Years

Location: Bengaluru/Mumbai

Employment Type: Full-time

About Wissen Technology

Wissen Technology is a global technology services company focused on delivering innovative software engineering, data, and technology solutions to leading enterprises. The company works with clients across industries to build scalable, high-performance technology platforms and solve complex business and technology challenges.

With a strong focus on engineering excellence, innovation, and collaboration, Wissen Technology brings together skilled technology professionals across areas such as software engineering, data engineering, cloud, analytics, and digital transformation.

At Wissen Technology, employees have the opportunity to work on challenging technology projects, collaborate with experienced engineering teams, and contribute to solutions that create measurable business impact.

Key Responsibilities

• Design, develop, and maintain scalable data processing applications using Python, PySpark, and Spark.
• Develop and optimize data pipelines and workflows on Databricks.
• Collaborate with data engineers, data scientists, business stakeholders, and other technical teams to understand requirements and deliver high-quality solutions.
• Ensure data integrity, quality, performance, and reliability across data processing pipelines.
• Write clean, maintainable, scalable, and efficient code following established coding standards and best practices.
• Perform data analysis and implement appropriate data validation and quality checks.
• Monitor, troubleshoot, and optimize performance issues across data workflows and pipelines.
• Work with relational databases and develop efficient SQL queries for data extraction and transformation.
• Participate in code reviews, testing, deployment, and continuous improvement of data engineering solutions.
• Use Git/version control and follow established software development and deployment practices.

Required Skills & Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field.
• Proven experience as a Databricks Developer, Data Engineer, or similar role.
• Strong hands-on expertise in Apache Spark and PySpark.
• Strong programming skills in Python; experience with Scala is an advantage.
• Strong proficiency in SQL and hands-on experience with relational databases.
• Practical experience developing and optimizing data pipelines and data processing applications.
• Familiarity with Git and version control systems.
• Strong understanding of data engineering concepts, data transformation, and data validation.
• Excellent analytical and problem-solving skills.
• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

Tech Lead – Data Lake Platform

Recruitment & Payroll Company: Neogencode Technologies

Job Location: Sector 43, Gurugram, Haryana

Work Mode: On-site (Work from Office)

Working Days: 5 days a week

Experience Required: 5+ years

Employment Type: Full-time

About the Opportunity

We are hiring a Tech Lead – Data Lake Platform. This opportunity involves working on an enterprise-grade AWS data lake platform supporting data engineering, analytics, reporting, and business applications.

The candidate will be employed on the payroll of Neogencode Technologies and will work at the client location in Sector 43, Gurugram.

Key Responsibilities

• Own and design end-to-end data lake architecture, including data ingestion, staging, curated layers, and data consumption.
• Lead data engineering pipelines using AWS services, Apache Airflow, Spark/PySpark, and dbt.
• Manage AWS data services such as S3, EMR, Glue, Athena, Redshift, Lambda, RDS, IAM, and DMS.
• Design and support API/data-serving solutions using PostgreSQL, Hasura GraphQL, or similar technologies.
• Improve pipeline performance, platform reliability, monitoring, and service-level agreements (SLAs).
• Ensure data security, access control, personally identifiable information (PII) protection, and governance.
• Mentor data engineers, conduct code reviews, and handle production incidents and root-cause analysis.
• Collaborate with business and application teams to deliver data and reporting solutions.

Required Skills

• 5+ years of experience in data engineering or platform engineering, including technical leadership.
• Strong hands-on experience with AWS data services and enterprise data lake architecture.
• Production experience with Apache Airflow, Spark/PySpark, and SQL.
• Strong understanding of data modeling and layered data architecture.
• Experience with PostgreSQL and Hasura GraphQL or a similar API/data-serving layer.
• Proven experience in technical leadership, production support, and troubleshooting.

Good to Have

• Experience with dbt and lakehouse table formats.
• Knowledge of Kafka or streaming data ingestion.
• Experience with Redshift optimization and workload design.

Job Description:

We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.

Key Responsibilities

·        5-8 years of professional work experience in a relevant field

·        Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.

·        Good Experience on Microsoft Dynamics 365

·        Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.

·        Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.

·        Good knowledge of SQL and Python for data manipulation, transformation, and analysis

·        Understand business requirements to set functional specifications for reporting applications

• Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.
• Data Integration: Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.
• Power BI Integration: Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.
• Performance Optimization: Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.
• Data Modeling: Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.
• Cloud Integration: Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.
• Security & Compliance: Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.
• Technical Skills:
• Proficiency in Databricks, including Delta Lake, Spark SQL.
• Strong programming skills in Python, Scala, or Java.
• Experience with Apache Spark for big data processing.
• Knowledge of SQL for querying and transforming data.
• Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.

Preferred Qualifications

• Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).

Experience with real-time data processing and streaming

Lead Data Engineer – Databricks & Azure

• We are looking for a Lead Data Engineer with 5–7 years of experience in Azure, Databricks, and Microsoft Fabric to lead and mentor the data engineering team. The role involves implementing scalable data solutions using Databricks as the core platform, building modern data pipelines, and enabling seamless integration with Microsoft Fabric and Power BI.

Responsibilities & Requirements:

• Lead and mentor data engineers while driving hands-on technical delivery.
• Implement approved Databricks and Microsoft Fabric architectures and solutions.
• Develop metadata-driven bronze, silver, and gold data pipelines.
• Build reusable batch, incremental, CDC, and streaming ingestion pipelines.
• Strong hands-on experience with Azure Databricks, PySpark, SQL, and Delta Lake.
• Experience with Microsoft Fabric, OneLake, Direct Lake, and Power BI integration.
• Implement data quality checks, reconciliation, source-to-target mapping, and metadata management.
• Configure Unity Catalog for data governance, access control, auditing, and lineage.
• Implement CI/CD, automated deployments, and Datadog monitoring.
• Drive testing, documentation, operational readiness, and knowledge transfer.

Good to have:

• Experience with MDM and master data integration.
• Exposure to Azure DevOps or GitHub Actions.
• Experience with automated data testing and reconciliation frameworks.
• Experience working with onshore stakeholders and distributed engineering teams.
• Strong communication, problem-solving, and team leadership skills.

Job Title : Senior Data Engineer – GCP & Apache Spark

Job Details :

• Number of Positions : 3
• Employment Type : Contract-to-Hire (C2H)
• Experience Required : 5 to 8 Years
• Location : Manyata Tech Park, Bengaluru
• Work Mode : Hybrid – 3 Days in Office per Week
• Preferred Notice Period : Immediate to 15–20 Days
• Project Duration : Initially 6–8 months, with a strong possibility of extension for long-term projects
• Relocation : Candidates from outside Bengaluru can apply if they are willing to relocate.

Role Overview :

We are looking for experienced Senior Data Engineers with strong expertise in Google Cloud Platform (GCP), Apache Spark, BigQuery, and SQL to design, develop, and maintain scalable data pipelines and distributed data processing solutions.

The ideal candidate should have hands-on experience with cloud-based data engineering, end-to-end pipeline architecture, system design, performance optimization, debugging, and CI/CD practices.

Mandatory skills :

Apache Spark (PySpark / Scala), SQL, GCP, BigQuery, GCP Dataproc, Data Pipelines, System Design, CI/CD, Debugging & Problem-Solving.

Key Responsibilities :

• Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and/or Scala.
• Build and optimize batch and distributed data processing workflows on GCP.
• Work extensively with BigQuery for data transformation, validation, analytics, and query optimization.
• Develop and maintain data workflows using GCP services, including Dataproc, BigQuery, and Cloud Storage.
• Design end-to-end data pipeline architectures with a focus on scalability, reliability, and performance.
• Write and optimize complex SQL queries for large-scale datasets.
• Troubleshoot and resolve data processing issues, pipeline failures, and performance bottlenecks.
• Implement CI/CD practices for data engineering workflows, testing, and deployments.
• Monitor pipeline performance and identify opportunities for continuous improvement.
• Collaborate with cross-functional teams to understand requirements and deliver reliable data solutions.
• Participate in code reviews, testing, production deployments, and production support.

Mandatory Skills :

• Strong hands-on experience with Apache Spark, including PySpark and/or Scala.
• Strong SQL skills and experience working with large-scale datasets.
• Hands-on experience with Google Cloud Platform (GCP).
• Practical experience with BigQuery and data warehousing concepts.
• Working knowledge of GCP Dataproc.
• Strong understanding of data pipelines, end-to-end workflows, and system design.
• Practical experience with CI/CD and deployment processes.
• Excellent debugging, troubleshooting, and problem-solving skills.

Preferred Skills :

• Experience with Apache Airflow, Google Cloud Composer, or similar orchestration tools.
• Knowledge of Kafka or other streaming and messaging technologies.
• Proficiency in Python for data engineering and automation.
• Experience with Git-based development workflows.
• Knowledge of Linux/Unix environments and shell scripting.
• Familiarity with monitoring, logging, data quality, and governance practices.

Required Qualifications :

• 5 to 8 years of experience in Data Engineering, Big Data Engineering, or a related field.
• Proven experience developing and maintaining scalable data pipelines.
• Strong understanding of distributed data processing and cloud-based data architecture.
• Ability to troubleshoot complex technical issues and optimize pipeline performance.
• Strong communication, collaboration, and ownership skills.

Interview Process :

Round 1 : AI-Based Interview

• Candidates must complete the AI interview as soon as possible after receiving the invitation.

Round 2 : Virtual Interview Drive

• Interviews are planned for the upcoming drive; exact dates will be communicated separately.

Round 3 : Online Client Interview

• Shortlisted candidates will attend the online interview.
• Candidates must be available throughout the day on the scheduled interview date.

🚨 WE ARE HIRING – SENIOR DATA ENGINEER | BANGALORE 🚨

Looking for experienced Senior Data Engineers with strong expertise in PySpark, Oracle SQL/PLSQL, and Data Modeling!

🔹 Role: Senior Data Engineer

🔹 Experience: 7+ Years

🔹 Location: Bangalore

🔹 CTC: Up to 26 LPA

🔹 Notice Period: Immediate to 10 Days Preferred

💻 Mandatory Skills

✅ PySpark

✅ Oracle SQL / PL/SQL

✅ Data Modeling & Design / Modernization

✅ Python

✅ ETL

✅ Data Pipelines

⚙️ Good to Have / Ecosystem Skills

✅ Kafka

✅ Hadoop

✅ AWS / Azure / GCP

✅ Git

✅ JIRA

📌 Key Responsibilities

• Develop and maintain scalable data pipelines using PySpark

• Work extensively with Oracle SQL/PLSQL for data processing and transformation

• Design and implement data models and data architecture

• Work on data design and modernization initiatives

• Develop and optimize ETL processes and data pipelines

• Handle large volumes of data using PySpark and Python

• Work with technologies such as Kafka, Hadoop, and Cloud platforms

• Collaborate with cross-functional teams to deliver scalable data solutions

• Use Git and JIRA for version control and project tracking

📩 Interested candidates can share their updated CV along with:

Total Experience:

Relevant PySpark Experience:

Relevant Oracle SQL/PLSQL Experience:

Data Modeling Experience:

Current Location:

Notice Period:

Current CTC:

Expected CTC:

#Hiring #DataEngineer #SeniorDataEngineer #PySpark #Oracle #PLSQL #DataModeling #Python #ETL #DataPipelines #Kafka #Hadoop #AWS #Azure #GCP #BangaloreJobs #ITJobs #TechJobs #ImmediateJoiners