Senior Big Data Developer at HL · Bengaluru (Bangalore) · 6 - 15 years · ₹1L - ₹15L / yr · Posted 30 Nov 2022

• Strong experience working with Big Data technologies like Spark (Scala/Java),
• Apache Solr, HIVE, HBase, ElasticSearch, MongoDB, Airflow, Oozie, etc.
• Experience working with Relational databases like MySQL, SQLServer, Oracle etc.
• Good understanding of large system architecture and design
• Experience working in AWS/Azure cloud environment is a plus
• Experience using Version Control tools such as Bitbucket/GIT code repository
• Experience using tools like Maven/Jenkins, JIRA
• Experience working in an Agile software delivery environment, with exposure to
continuous integration and continuous delivery tools
• Passionate about technology and delivering solutions to solve complex business
problems
• Great collaboration and interpersonal skills
• Ability to work with team members and lead by example in code, feature
development, and knowledge sharing

Similar jobs (10)

🚀 WE ARE HIRING | JAVA + APACHE SPARK DEVELOPER
📍 Location: Mumbai
💼 Experience: 5+ Years
🏢 Work Mode: Hybrid (3 Days Work From Office)
Are you an experienced Java + Apache Spark Developer looking for your next career opportunity? We are looking for talented professionals with strong expertise in Java, Big Data technologies, and data engineering.
🔥 Required Technical Skills:
- Strong experience in Java Development
- Hands-on experience with Apache Spark
- Good knowledge of Spark SQL
- Experience with Databricks
- Strong SQL skills
- Experience in ETL processes and data pipelines
- Understanding of distributed data processing
- Good problem-solving and analytical skills
📩 Interested candidates can share their updated resumes:
📧 Email: revathi.r @vysystems.com
📱 WhatsApp: 6 3 8 5 0 7 6 1 4 2

Job Title : Senior Data Engineer – Databricks
Experience : 14 to 20 Years
Location : HSR Layout, Bangalore
Work Mode : Hybrid – 3 Days WFO
Shift : 11:30 AM – 07:30 PM IST
Positions : 2
Notice Period : Immediate Joiners Only
Interview : 1 Technical Round + 2 Client Rounds
Role Overview :
We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.
The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.
Must-Have Skills :
- 14 to 20 years of Data Engineering experience
- Databricks & Apache Spark / PySpark
- Python & SQL
- AWS Cloud
- Lakehouse Architecture
- ETL / ELT & Distributed Data Processing
- Batch & Streaming Pipelines
- Data Pipeline Optimization & Data Modeling
- CDC & Incremental Processing
- Git, CI/CD & Testing
- Data Quality, Monitoring & Observability
- Technical Leadership & Stakeholder Management
Key Responsibilities :
- Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
- Own data products from design through production.
- Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
- Optimize pipelines for performance, scalability, reliability, and cost.
- Design scalable data architectures and data models.
- Implement data quality, monitoring, lineage, and CI/CD practices.
- Lead technical discussions and mentor engineering teams.
- Collaborate with business stakeholders, architects, product owners, and engineering teams.
- Remain hands-on while providing technical leadership.
Ideal Candidate :
A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.
🔴 Super Urgent : Only Bangalore-based immediate joiners.

Job Title: Java + Apache Spark Developer
Experience: 5–10 Years
Location: Mumbai
Work Mode: Hybrid (3 Days WFO)
Job Description:
Looking for hands-on Java Developers with strong experience in Apache Spark and distributed data processing. Candidates should have experience building and optimizing data pipelines, working with Spark SQL, and using Snowflake or Databricks for enterprise data engineering solutions.
Mandatory Skills:
Java, Apache Spark, Spark SQL, Snowflake / Databricks, SQL, ETL.

🚀 Job Title : Data Engineer – AWS, PySpark & Airflow
Experience : 5+ Years
Industry : Health Insurance / BFSI
Location : Sector 43, Gurugram, Haryana
Work Mode : Work From Office (5 Days a Week)
Employment Type : Full-Time
Role Overview :
We are looking for an experienced Data Engineer with strong hands-on expertise in AWS Data Engineering, Data Lake Architecture, Python, SQL, PySpark, and Apache Airflow.
The candidate will be responsible for designing, developing, and maintaining scalable, reliable, and high-performance data pipelines using AWS cloud technologies. The role involves building end-to-end ETL / ELT workflows, implementing data transformation and processing solutions, optimizing pipeline performance, and ensuring the reliability of production data systems.
Mandatory Skills :
AWS Data Engineering | Data Lake Architecture | Amazon S3 | AWS Glue | Amazon EMR | Athena | Redshift | Python | SQL | PySpark | Apache Spark | Apache Airflow | ETL/ELT | PostgreSQL | Data Modelling | Pipeline Optimization | Production Support | Root Cause Analysis
Key Responsibilities :
- Design, develop, and maintain end-to-end data pipelines for data ingestion, staging, transformation, curation, and consumption.
- Build and manage AWS-based data lake solutions using Amazon S3, AWS Glue, Amazon EMR, Athena, and Redshift.
- Develop, maintain, and optimize ETL / ELT pipelines using Python, SQL, Apache Spark, and PySpark.
- Implement and manage workflow orchestration using Apache Airflow, including scheduling, dependency management, retries, monitoring, and failure handling.
- Integrate data from multiple structured and semi-structured sources.
- Design and maintain data models, dimensional models, and analytical data marts.
- Work with relational databases, including PostgreSQL and Amazon RDS.
- Optimize Spark jobs, SQL queries, data transformations, and cloud resource utilization.
- Implement data quality checks, validation rules, and error-handling mechanisms.
- Monitor production data pipelines, troubleshoot failures, and perform root cause analysis (RCA).
- Ensure pipeline reliability, scalability, performance, and compliance with service-level agreements (SLAs).
- Follow data security, access management, governance, and sensitive-data protection best practices.
- Collaborate with business, analytics, and application teams to deliver effective data solutions.
- Maintain technical documentation and follow established data engineering standards.
Required Technical Skills :
- AWS Data Engineering : Strong hands-on experience with AWS-based data engineering and data lake architecture.
- AWS Services : Amazon S3, AWS Glue, Glue Data Catalog, Amazon EMR, Athena, Redshift, AWS Lambda, and IAM.
- Programming : Strong proficiency in Python and SQL.
- Big Data Processing : Hands-on experience with Apache Spark and PySpark.
- Workflow Orchestration : Experience developing and managing workflows using Apache Airflow.
- ETL/ELT Development : Experience building scalable data ingestion, transformation, and processing pipelines.
- Database Technologies : Good understanding of relational databases and PostgreSQL concepts.
- Data Modelling : Experience with data modelling, data transformation, and analytical data structures.
- Production Support : Ability to monitor pipelines, troubleshoot failures, perform RCA, and resolve production issues.
- Performance Optimization : Experience optimizing SQL queries, Spark jobs, and data processing workflows.
Good-to-Have Skills :
- Experience with AWS Database Migration Service (DMS) and Amazon RDS.
- Familiarity with dbt for data transformation and modelling.
- Knowledge of Delta Lake, Apache Iceberg, or Apache Hudi.
- Exposure to Apache Kafka and real-time data streaming.
- Familiarity with GraphQL, Hasura, or similar data-serving technologies.
- Understanding of data governance and data quality frameworks.
- Experience in the BFSI, healthcare, or health insurance domain.
Candidate Requirements :
- 5+ years of relevant experience in Data Engineering.
- Proven hands-on experience building and maintaining production-grade data pipelines on AWS.
- Strong understanding of data lake architecture and large-scale data processing.
- Practical experience with Apache Airflow and Spark / PySpark-based transformations.
- Strong programming, analytical, debugging, and problem-solving skills.
- Experience handling production incidents, pipeline failures, and performance bottlenecks.
- Ability to take ownership of deliverables and work effectively with cross-functional teams.
- Good communication, collaboration, and technical documentation skills.
- Willingness to work from the Gurugram office five days a week.
Ideal Candidate :
The ideal candidate is a hands-on Data Engineer who can independently build, optimize, and maintain AWS-based data pipelines, work with large datasets using PySpark, orchestrate workflows through Airflow, and resolve production issues in a business-critical environment.

Job Title : Tech Lead – Data Lake Platform
Number of Positions : 2
Experience : 7+ Years
Role Type : Technical Lead / Data Platform Lead
Domain : Data Engineering / Data Platform / AWS
About the Role :
We are looking for an experienced Tech Lead – Data Lake Platform to lead the design, development, and operations of an enterprise-scale AWS-based Data Lake Platform.
The platform will ingest data from multiple business systems, process it through structured data layers, and serve data for analytics, reporting, APIs, and operational applications.
As a Tech Lead, you will be responsible for setting the technical direction, leading data engineering teams, driving platform reliability and performance, and owning the platform's delivery, governance, and production support end to end.
Core Tech Stack :
AWS | S3 | EMR | Glue | Athena | Redshift | DMS | Lambda | RDS | IAM | Airflow | Spark/PySpark | SQL | Data Modeling | Hasura | GraphQL | DBT | PostgreSQL/Aurora | Kafka
Key Responsibilities :
- Own the overall Data Lake Platform architecture, covering data ingestion, staging, curated layers, consumption, analytics, and API/data serving.
- Lead the design and development of production-grade data pipelines using AWS Glue, EMR/Spark, Airflow, DBT, Athena, and Redshift.
- Design scalable batch and analytics pipelines with a focus on reliability, performance, data quality, and maintainability.
- Own the API/data-serving architecture from consumption data → RDS/PostgreSQL → Hasura GraphQL → Lambda/API Gateway.
- Drive improvements in platform stability, including orchestration failures, cluster sizing, pipeline SLAs, query performance, and production reliability.
- Design and implement appropriate AWS security, access control, IAM, PII handling, and data governance practices.
- Lead technical discussions, architecture decisions, code reviews, and engineering best practices.
- Mentor and guide data engineers while ensuring high-quality and scalable engineering delivery.
- Own production support, incident management, troubleshooting, and root-cause analysis (RCA) for critical data platform issues.
- Develop and maintain runbooks, operational procedures, monitoring, and incident response practices.
- Collaborate with business, product, application, and analytics teams to onboard new datasets and support reporting, API, and data consumption requirements.
- Ensure the platform meets defined availability, performance, security, data quality, and compliance requirements.
Must-Have Skills :
- 7+ years of experience in Data Engineering, Data Platform Engineering, or related areas, including experience in technical leadership or architecture.
- Strong hands-on experience with AWS Data Services, including:
- Amazon S3
- AWS EMR
- AWS Glue
- Amazon Athena
- Amazon Redshift
- AWS DMS
- AWS Lambda
- Amazon RDS
- AWS IAM
- Strong production experience with Apache Airflow.
- Strong hands-on experience with Apache Spark / PySpark.
- Strong SQL skills and experience with data modeling, including layered data architecture, data marts, and enterprise data models.
- Experience with Hasura or a similar GraphQL layer for PostgreSQL-based data/API serving.
- Strong understanding of data lake architecture and enterprise data platforms.
- Proven experience leading engineers and driving technical decisions.
- Hands-on experience with production support, troubleshooting, incident management, and RCA.
- Strong understanding of data platform performance, scalability, reliability, and SLA management.
Good-to-Have Skills :
- Experience with DBT and modern data transformation practices.
- Experience with lakehouse table formats on Amazon S3, such as Apache Iceberg or similar technologies.
- Strong knowledge of Amazon Redshift workload optimization, including :
- Distribution keys
- Sort keys
- Spectrum
- External tables
- Experience with Kafka or other streaming/data ingestion technologies.
- Experience with PostgreSQL / Amazon Aurora as a data-serving layer.
- Experience with Lambda and API Gateway for API-based data serving.
- Experience with enterprise data governance, data quality, security, and compliance.
- Experience in BFSI / Banking / Financial Services / Insurance domain.
Ideal Candidate :
The ideal candidate is a hands-on Data Platform / Data Engineering Lead who can operate at both the architecture and implementation level. You should be comfortable designing an AWS Data Lake from end to end, leading engineers, troubleshooting production issues, and working closely with business and application teams to deliver reliable data products.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Join our product development team at Planview as a Senior Software Engineer I and become a pivotal force on the Viz Core Team. This role offers the unique opportunity to shape and lead the development of data-processing pipelines and APIs that are at the heart of our software solutions. These solutions are designed to streamline and enhance the efficiency of software delivery, resonating deeply with software engineers who strive to build better and faster.
At Planview Viz, which is powered by the innovative “Flow Framework,” you will tackle complex data challenges and develop scalable solutions within an AWS cloud environment. Your efforts will be crucial in revolutionizing how businesses harness and interpret vast amounts of workflow data, transforming it into actionable insights that propel organizational efficiency and effectiveness.
Responsibilities (What you'll do)
- Create and refine powerful data-processing architectures that integrate seamlessly with a diverse array of external tools, enhancing the way software is delivered across industries.
- Drive operational excellence by critically analyzing problems, defining requirements, and devising robust solutions that push the boundaries of technology.
- Lead and inspire a team of talented engineers, promoting a culture of ownership, meticulous attention to quality, and proactive problem-solving.
- Stay at the cutting edge of technology by updating and expanding your team’s knowledge of cloud architectures, data processing, and advanced analytics, ensuring that you and your team remain leaders in technological innovation.
- Make a direct impact on the efficiency and effectiveness of software delivery worldwide through innovation and leadership.
Qualifications (What you'll bring)
Who We’re Looking For
The ideal candidate is a seasoned professional in cloud software development with a strong foundation in data processing and scalable software systems. You thrive in collaborative environments and are passionate about advancing cloud technology and data architecture to new heights. You are a leader who enjoys mentoring, guiding, and inspiring others.
Preferred Qualifications
Skills, Knowledge, and Expertise
- A degree in Computer Science, Engineering, or a related field.
- 6+ years of experience with a modern programming language, with a focus on back-end systems and cloud-based technologies.
- Strong capability in architecting and designing robust, scalable software systems.
- Proficiency in AWS or another popular cloud platform.
Additional Qualifications
- Experience with big-data technologies such as Spark and Apache Kafka.
- Experience with Java (version 17), Scala, or other JVM programming languages.
- Experience with tools such as Amazon Redshift and MongoDB.
- Experience with continuous integration and continuous deployment tools such as GitHub, Jenkins, Travis CI, or similar platforms.
- Proficiency in test-driven development.
- Experience with containers and orchestration tools such as Kubernetes.
- Experience working on remote or distributed teams and projects.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Dear Candidate,
Thanks for showing interest in the opportunity!!!
As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.
Job Description:
We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.
Primary & Mandatory Skills:
- ETL Development
- Hadoop / HDFS
- PySpark / Apache Spark
- Data Extraction, Transformation & Loading
- SQL and Data Processing
Good to Have:
- Hive / Spark SQL
- Data Pipeline Development
- Big Data Processing
- Python Programming
Company: Mphasis
Role: ETL Developer – Hadoop & PySpark
Experience: As per requirement
Location: Bangalore
Employment Type: Permanent
Technical Stack
- PySpark
- Databricks
- Microsoft Azure
- SQL
- Kubernetes
Key Responsibilities & Expectations
- Must be self-driven, proactive, and capable of independently delivering high-quality solutions.
- Demonstrate strong ownership and accountability for assigned tasks and deliverables.
- Ability to work with minimal supervision while consistently meeting project objectives and quality standards.
We are hiring experienced Data Engineers for a permanent opportunity in Bangalore.
Job Description
Role: Data Engineer
Location: Bangalore
Experience: 6+ Years
Key Responsibilities:
- Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
- Develop data engineering solutions using Python and Scala.
- Process and transform large volumes of structured and unstructured data.
- Build and optimize distributed data processing workflows for performance and scalability.
- Work with big data technologies and cloud-based data processing environments.
- Perform data integration, validation, transformation, and quality checks.
- Troubleshoot data pipeline issues and optimize processing performance.
- Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.
Required Skills:
- Strong hands-on experience with Amazon EMR.
- Proficiency in Python and Scala.
- Experience in big data processing and data pipeline development.
- Good understanding of distributed computing and data transformation techniques.
- Experience in performance tuning and troubleshooting data processing jobs.
- Strong analytical, problem-solving, and communication skills.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
1st virtual , 2nd round F2F
Python pyspark, SQL, data engineer
5+yrs
9+yrs
Bangalore/Hyderabad
immediate to 15days.







