Hadoop Admin Lead at Smartavya · Mumbai · 10 - 12 years · ₹20L - ₹30L / yr · Posted 2 Jan 2025

Key Responsibilities:
• Install, configure, and maintain Hadoop clusters.
• Monitor cluster performance and ensure high availability.
• Manage Hadoop ecosystem components (HDFS, YARN, Ozone, Spark, Kudu, Hive).
• Perform routine cluster maintenance and troubleshooting.
• Implement and manage security and data governance.
• Monitor systems health and optimize performance.
• Collaborate with cross-functional teams to support big data applications.
• Perform Linux administration tasks and manage system configurations.
• Ensure data integrity and backup procedures.

Similar jobs (10)
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Dear Candidate,
Thanks for showing interest in the opportunity!!!
As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.
Job Description:
We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.
Primary & Mandatory Skills:
- ETL Development
- Hadoop / HDFS
- PySpark / Apache Spark
- Data Extraction, Transformation & Loading
- SQL and Data Processing
Good to Have:
- Hive / Spark SQL
- Data Pipeline Development
- Big Data Processing
- Python Programming
Company: Mphasis
Role: ETL Developer – Hadoop & PySpark
Experience: As per requirement
Location: Bangalore
Employment Type: Permanent
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
We are hiring experienced Data Engineers for a permanent opportunity in Bangalore.
Job Description
Role: Data Engineer
Location: Bangalore
Experience: 6+ Years
Key Responsibilities:
- Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
- Develop data engineering solutions using Python and Scala.
- Process and transform large volumes of structured and unstructured data.
- Build and optimize distributed data processing workflows for performance and scalability.
- Work with big data technologies and cloud-based data processing environments.
- Perform data integration, validation, transformation, and quality checks.
- Troubleshoot data pipeline issues and optimize processing performance.
- Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.
Required Skills:
- Strong hands-on experience with Amazon EMR.
- Proficiency in Python and Scala.
- Experience in big data processing and data pipeline development.
- Good understanding of distributed computing and data transformation techniques.
- Experience in performance tuning and troubleshooting data processing jobs.
- Strong analytical, problem-solving, and communication skills.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Position: Backup Data ETL Developer
Experience: 6–9 Years
Location:Mumbai
Employment Type: Full-time
Job Description:
We are looking for a Backup Data ETL Developer with 6–9 years of experience in designing, developing, and supporting data extraction, transformation, and loading (ETL) processes.
The candidate should have strong hands-on experience with Hadoop, Spark, Informatica, and Snowflake, along with expertise in large-scale data processing and data integration technologies.
Key Skills:
Hadoop
Spark
Informatica
Snowflake
ETL Development
Data Integration
Data Transformation
Large-Scale Data Processing
ETL Workflows and Data Pipelines
Troubleshooting and Problem-Solving
Strong analytical and collaboration skills
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
🚨 HIRING | Data Engineer – Amazon EMR 🔥
We are looking for experienced Data Engineers with strong hands-on expertise in Amazon EMR, Python, and Scala.
📌 JOB DETAILS:
💼 Experience: 6+ Years
💰 CTC: Up to 30 LPA
📍 Location: Bangalore
🔑 MANDATORY SKILLS:
• Amazon EMR
• Python
• Scala
• Data Engineering
📋 KEY REQUIREMENTS:
• 6+ years of relevant experience in Data Engineering.
• Strong hands-on experience with Amazon EMR.
• Strong programming skills in Python and Scala.
• Experience with large-scale data processing and data engineering solutions.
• Strong problem-solving and communication skills.
📩 Interested candidates can share their updated resumes:
📧 pavithra.e @vysystems. com
📞 8 9 2 5 8 0 2 4 5 7
🔄 Know someone who would be a good fit? Tag them or share this post!
#Hiring #DataEngineer #DataEngineering #AmazonEMR #Python #Scala #AWSJobs #BangaloreJobs #ITJobs #TechHiring #HiringNow

Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Snowflake Administrator
What You’ll Do
- Observability: Monitor Snowflake queries, virtual warehouses, tasks, data loads, and overall account health. Identify and resolve operational issues.
- FinOps & Cost Optimization: Monitor credit consumption, storage, and warehouse costs. Optimize warehouse sizing, auto-suspend settings, and resource monitors.
- User Management: Manage Snowflake users, roles, onboarding, offboarding, and Role-Based Access Control (RBAC).
- Privilege Access Management: Review grants, privileged roles, access policies, and audit logs to enforce the principle of least privilege.
- Platform Maintenance: Maintain Snowflake account settings, integrations, and routine operational health checks.
- Service Desk Management: Triage incidents and service requests, investigating query failures, access issues, and platform-related problems.
- Enhancements & Automation: Improve Snowflake configurations, automate operational tasks, and streamline support processes.
- Business Continuity & Recovery: Support backup and retention policies, Time Travel, Fail-safe, and data recovery procedures.
- Deployment & CI/CD: Support controlled releases of Snowflake objects, roles, scripts, and configurations through CI/CD pipelines.
- Data Load Troubleshooting: Resolve issues related to Snowpipe, COPY commands, stages, file formats, connectivity, and scheduled data loads.
- Transformation & Data Model Support: Perform first-line troubleshooting of failed tasks, SQL errors, and schema changes.
- Functional Data Support: Investigate initial data availability and quality issues, coordinating with data owners for resolution.
- Project Support: Assist engineering teams with Snowflake environments, integrations, and migrations.
- Documentation: Maintain runbooks, access management procedures, operational guides, and knowledge base articles.
What We’re Looking For
- 5+ years of experience in data platform administration, operations, or production support, including hands-on Snowflake experience.
- Strong knowledge of Snowflake administration, including RBAC, virtual warehouses, query history, performance tuning, and cost optimization.
- Hands-on experience with Snowflake data loading, stages, tasks, Snowpipe, and cloud platforms such as AWS or Azure.
- Proficiency in SQL and familiarity with Python for scripting and automation.
- Experience with CI/CD pipelines, Jira, ITSM processes, and monitoring or automation tools.
- Strong troubleshooting, documentation, communication, and stakeholder management skills.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Description
We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.
Key Responsibilities
- Design, develop, and maintain ETL processes and data pipelines.
- Work with Hadoop and PySpark to process and transform large volumes of data.
- Develop and optimize Python scripts for data processing and automation.
- Write complex SQL queries for data extraction, transformation, and validation.
- Perform data cleansing, integration, transformation, and validation.
- Work with data warehousing concepts and large-scale data processing systems.
- Optimize ETL workflows for performance, scalability, and reliability.
- Troubleshoot data pipeline failures and resolve data quality issues.
- Collaborate with data engineers, analysts, and other technical teams.
- Follow data governance, quality, and development best practices.
Required Skills
- Strong experience in ETL Development – Mandatory
- Hands-on experience with Hadoop – Mandatory
- Strong experience in PySpark – Mandatory
- Good programming skills in Python – Mandatory
- Strong knowledge of SQL – Mandatory
- Experience in Data Processing and Transformation – Mandatory
- Knowledge of Data Warehousing – Mandatory
- Experience with Big Data Technologies – Mandatory
Opportunity Details
Role: ETL Developer – Hadoop & PySpark
Experience: 6–10 Years
Location: Bangalore
Mandatory Skills
ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Title : Senior Data Engineer – GCP & Apache Spark
Job Details :
- Number of Positions : 3
- Employment Type : Contract-to-Hire (C2H)
- Experience Required : 5 to 8 Years
- Location : Manyata Tech Park, Bengaluru
- Work Mode : Hybrid – 3 Days in Office per Week
- Preferred Notice Period : Immediate to 15–20 Days
- Project Duration : Initially 6–8 months, with a strong possibility of extension for long-term projects
- Relocation : Candidates from outside Bengaluru can apply if they are willing to relocate.
Role Overview :
We are looking for experienced Senior Data Engineers with strong expertise in Google Cloud Platform (GCP), Apache Spark, BigQuery, and SQL to design, develop, and maintain scalable data pipelines and distributed data processing solutions.
The ideal candidate should have hands-on experience with cloud-based data engineering, end-to-end pipeline architecture, system design, performance optimization, debugging, and CI/CD practices.
Mandatory skills :
Apache Spark (PySpark / Scala), SQL, GCP, BigQuery, GCP Dataproc, Data Pipelines, System Design, CI/CD, Debugging & Problem-Solving.
Key Responsibilities :
- Design, develop, and maintain scalable data pipelines using Apache Spark, PySpark, and/or Scala.
- Build and optimize batch and distributed data processing workflows on GCP.
- Work extensively with BigQuery for data transformation, validation, analytics, and query optimization.
- Develop and maintain data workflows using GCP services, including Dataproc, BigQuery, and Cloud Storage.
- Design end-to-end data pipeline architectures with a focus on scalability, reliability, and performance.
- Write and optimize complex SQL queries for large-scale datasets.
- Troubleshoot and resolve data processing issues, pipeline failures, and performance bottlenecks.
- Implement CI/CD practices for data engineering workflows, testing, and deployments.
- Monitor pipeline performance and identify opportunities for continuous improvement.
- Collaborate with cross-functional teams to understand requirements and deliver reliable data solutions.
- Participate in code reviews, testing, production deployments, and production support.
Mandatory Skills :
- Strong hands-on experience with Apache Spark, including PySpark and/or Scala.
- Strong SQL skills and experience working with large-scale datasets.
- Hands-on experience with Google Cloud Platform (GCP).
- Practical experience with BigQuery and data warehousing concepts.
- Working knowledge of GCP Dataproc.
- Strong understanding of data pipelines, end-to-end workflows, and system design.
- Practical experience with CI/CD and deployment processes.
- Excellent debugging, troubleshooting, and problem-solving skills.
Preferred Skills :
- Experience with Apache Airflow, Google Cloud Composer, or similar orchestration tools.
- Knowledge of Kafka or other streaming and messaging technologies.
- Proficiency in Python for data engineering and automation.
- Experience with Git-based development workflows.
- Knowledge of Linux/Unix environments and shell scripting.
- Familiarity with monitoring, logging, data quality, and governance practices.
Required Qualifications :
- 5 to 8 years of experience in Data Engineering, Big Data Engineering, or a related field.
- Proven experience developing and maintaining scalable data pipelines.
- Strong understanding of distributed data processing and cloud-based data architecture.
- Ability to troubleshoot complex technical issues and optimize pipeline performance.
- Strong communication, collaboration, and ownership skills.
Interview Process :
Round 1 : AI-Based Interview
- Candidates must complete the AI interview as soon as possible after receiving the invitation.
Round 2 : Virtual Interview Drive
- Interviews are planned for the upcoming drive; exact dates will be communicated separately.
Round 3 : Online Client Interview
- Shortlisted candidates will attend the online interview.
- Candidates must be available throughout the day on the scheduled interview date.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Python, Hadoop, and Apache Spark.
- Develop efficient data processing and transformation workflows for large datasets.
- Write, optimize, and maintain complex SQL queries and PL/SQL procedures.
- Perform data extraction, transformation, and loading (ETL) from multiple data sources.
- Work with structured and unstructured data to support analytics and reporting requirements.
- Optimize data pipelines and queries for performance, scalability, and reliability.
- Ensure data quality, consistency, security, and integrity across data platforms.
- Collaborate with cross-functional teams to understand business requirements and deliver data engineering solutions.
- Use Git, automation, and CI/CD practices to support development and deployment.
Required Skills
- 4.5–10 years of experience in software development or data engineering.
- Strong proficiency in AWS
- Hands-on experience with Hadoop and Apache Spark.
- Good knowledge of SQL and PL/SQL, including query optimization and stored procedures.
- Experience in ETL development, data processing, and data pipeline design.
- Understanding of data warehousing concepts and distributed computing.
- Strong problem-solving, debugging, and performance-tuning skills.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
1st virtual , 2nd round F2F
Python pyspark, SQL, data engineer
5+yrs
9+yrs
Bangalore/Hyderabad
immediate to 15days.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Hiring: ETL Developer – Hadoop & PySpark
Location: Bangalore
Experience: 6–10 Years
Budget: 25–30 LPA
Work Mode: Weekdays
Job Description
We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
- Develop data transformation workflows using PySpark and Python.
- Work with Hadoop and other big data technologies to process and manage large datasets.
- Write complex SQL queries for data extraction, transformation, validation, and analysis.
- Implement data cleansing, data quality checks, and data validation processes.
- Develop and optimize data integration workflows for data warehousing solutions.
- Troubleshoot ETL failures, performance issues, and data discrepancies.
- Optimize PySpark jobs and SQL queries for performance and scalability.
- Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
- Ensure data accuracy, consistency, and adherence to established data governance standards.
Mandatory Skills
- Strong hands-on experience in ETL development.
- Good experience with Hadoop and big data technologies.
- Proficiency in PySpark and Python programming.
- Strong SQL skills, including complex queries and data manipulation.
- Experience in data processing, data transformation, and data integration.
- Understanding of data warehousing concepts and architecture.
- Experience in developing, debugging, and optimizing ETL pipelines.
Required Experience
- 6–10 years of relevant experience in ETL development and data engineering.
- Practical experience handling large datasets using Hadoop and PySpark.
- Strong analytical, problem-solving, and troubleshooting skills.
- Ability to work independently and collaborate effectively within a team.
Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.









