Cutshort logo
For Employers
Molecular Connections logo
Sr. Spark/AWS Developer
Sr. Spark/AWS Developer

Sr. Spark/AWS Developer at Molecular Connections · Bengaluru (Bangalore) · 8 - 10 years · ₹15L - ₹20L / yr · Profitable · Posted 2 Aug 2023

Molecular Connections's logo

Sr. Spark/AWS Developer

Molecular Connections's profile picture
Posted by Molecular Connections
8 - 10 yrs
₹15L - ₹20L / yr
Bengaluru (Bangalore)
Skills
Spark
Hadoop
Big Data
Data engineering
PySpark
Apache Kafka
skill iconScala
skill iconPython
Data modeling
  1. Big data developer with 8+ years of professional IT experience with expertise in Hadoop ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems spanning Big Data.
  2. A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
  3. Expertise in Apache Hadoop ecosystem components like Spark, Hadoop Distributed File Systems(HDFS), HiveMapReduce, Hive, Sqoop, HBase, Zookeeper, YARN, Flume, Pig, Nifi, Scala and Oozie.
  4. Hands on experience in creating real - time data streaming solutions using Apache Spark core, Spark SQL & DataFrames, Kafka, Spark streaming and Apache Storm.
  5. Excellent knowledge of Hadoop architecture and daemons of Hadoop clusters, which include Name node,Data node, Resource manager, Node Manager and Job history server.
  6. Worked on both Cloudera and Horton works in Hadoop Distributions. Experience in managing Hadoop clustersusing Cloudera Manager tool.
  7. Well versed in installation, Configuration, Managing of Big Data and underlying infrastructure of Hadoop Cluster.
  8. Hands on experience in coding MapReduce/Yarn Programs using Java, Scala and Python for analyzing Big Data.
  9. Exposure to Cloudera development environment and management using Cloudera Manager.
  10. Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle .
  11. Implemented Spark using PYTHON and utilizing Data frames and Spark SQL API for faster processing of data and handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
  12. Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
  13. Hands on experience in MLlib from Spark which are used for predictive intelligence, customer segmentation and for smooth maintenance in Spark streaming.
  14. Experience in using Flume to load log files into HDFS and Oozie for workflow design and scheduling.
  15. Experience in optimizing MapReduce jobs to use HDFS efficiently by using various compression mechanisms.
  16. Working on creating data pipeline for different events of ingestion, aggregation, and load consumer response data into Hive external tables in HDFS location to serve as feed for tableau dashboards.
  17. Hands on experience in using Sqoop to import data into HDFS from RDBMS and vice-versa.
  18. In-depth Understanding of Oozie to schedule all Hive/Sqoop/HBase jobs.
  19. Hands on expertise in real time analytics with Apache Spark.
  20. Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
  21. Extensive experience in working with different ETL tool environments like SSIS, Informatica and reporting tool environments like SQL Server Reporting Services (SSRS).
  22. Experience in Microsoft cloud and setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud.
  23. Extensively worked on Spark using Python on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
  24. Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.
  25. Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
  26. Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions.
  27. Experience in creating Data frames using PySpark and performing operation on the Data frames using Python.
  28. In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS and MapReduce Programming Paradigm, High Availability and YARN architecture.
  29. Establishing multiple connections to different Redshift clusters (Bank Prod, Card Prod, SBBDA Cluster) and provide the access for pulling the information we need for analysis. 
  30. Generated various kinds of knowledge reports using Power BI based on Business specification. 
  31. Developed interactive Tableau dashboards to provide a clear understanding of industry specific KPIs using quick filters and parameters to handle them more efficiently.
  32. Well Experience in projects using JIRA, Testing, Maven and Jenkins build tools.
  33. Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3,), focusing on high-availability, fault tolerance, and auto-scaling.
  34. Good experience with use-case development, with Software methodologies like Agile and Waterfall.
  35. Working knowledge of Amazon's Elastic Cloud Compute( EC2 ) infrastructure for computational tasks and Simple Storage Service ( S3 ) as Storage mechanism.
  36. Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark .
  37. Extensive experience in Text Analytics, developing different Statistical Machine Learning solutions to various business problems and generating data visualizations using Python and R.
  38. Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
  39. Hands on experience in Hadoop Big data technology working on MapReduce, Pig, Hive as Analysis tool, Sqoop and Flume data import/export tools.
Read more
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos

About Molecular Connections

Founded :
2001
Type :
Products & Services
Size :
1000-5000
Stage :
Profitable

About

With over two decades of experience in Big Data and Data Science Solutions, Molecular Connections (MC) has been using AI-powered proprietary models to help customers achieve digital transformation. We have been able to successfully build a data-driven decision-making strategy for our customer's digital transformational journey. MC leverages AI, ML and the Linked Data Store to build efficiencies in various verticals and generate new revenue streams for its customers. MC's decades of industry presence and a strong focus on innovation have led us to work with the world's leading pharma and STEM industries to offer end-to-end software development and data insights powered by proprietary workflows and platforms, enabling content engineering across multiple domains. With over 70% of its workforce being women, MC is ranked among the top 15 best companies for women to work for in India. Subsidiary Companies: 1. Molecular Connections Analytics Pvt. Ltd. | URL: https://mcanalytics.co.in/ 2. Molecular Connections Research Pvt. Ltd. | URL: https://mcresearch.co.in/
Read more

Connect with the team

Profile picture
Molecular Connections
Profile picture
Chendil Kumar
Profile picture
Gurminder kaur
Profile picture
Lokanath Khamari

Company social profiles

linkedin

Similar jobs (10)

VY SYSTEMS PRIVATE LIMITED
Bengaluru (Bangalore)
6 - 10 yrs
₹5L - ₹20L / yr
Hadoop
PySpark

Dear Candidate,

Thanks for showing interest in the opportunity!!!

As discussed, we have an opening for ETL Developer – Hadoop & PySpark with Mphasis for the Bangalore location – Permanent.


Job Description:

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, and PySpark. The candidate should have hands-on experience in data processing, building and maintaining ETL pipelines, data transformation, and handling large datasets using Big Data technologies.

Primary & Mandatory Skills:

  • ETL Development
  • Hadoop / HDFS
  • PySpark / Apache Spark
  • Data Extraction, Transformation & Loading
  • SQL and Data Processing

Good to Have:

  • Hive / Spark SQL
  • Data Pipeline Development
  • Big Data Processing
  • Python Programming

Company: Mphasis

Role: ETL Developer – Hadoop & PySpark

Experience: As per requirement

Location: Bangalore

Employment Type: Permanent

Read more
VY SYSTEMS PRIVATE LIMITED
Actively reviewing applications
Dharshini A
Posted by Dharshini A
Bengaluru (Bangalore)
6 - 12 yrs
₹5L - ₹20L / yr
Amazon EMR
skill iconPython
skill iconScala

We are hiring experienced Data Engineers for a permanent opportunity in Bangalore.

Job Description

Role: Data Engineer

Location: Bangalore

Experience: 6+ Years

Key Responsibilities:

  • Design, develop, and maintain scalable data processing pipelines using Amazon EMR.
  • Develop data engineering solutions using Python and Scala.
  • Process and transform large volumes of structured and unstructured data.
  • Build and optimize distributed data processing workflows for performance and scalability.
  • Work with big data technologies and cloud-based data processing environments.
  • Perform data integration, validation, transformation, and quality checks.
  • Troubleshoot data pipeline issues and optimize processing performance.
  • Collaborate with data architects, analysts, and development teams to deliver reliable data solutions.

Required Skills:

  • Strong hands-on experience with Amazon EMR.
  • Proficiency in Python and Scala.
  • Experience in big data processing and data pipeline development.
  • Good understanding of distributed computing and data transformation techniques.
  • Experience in performance tuning and troubleshooting data processing jobs.
  • Strong analytical, problem-solving, and communication skills.


Read more
Trantor
Actively reviewing applications
at Trantor
1 recruiter
Agency job
via Kirubin Technologies by Bala Murugan
Remote only
7 - 12 yrs
Best in industry
Big Data
Apache HBase
EMR
Hadoop
Spark
+1 more

Shift Timing: 3:00 PM to 12:00 AM


Summary of the Role

We are looking for a “Senior Data Platform Engineer” with strong hands-on experience in HBase, Amazon EMR, Hadoop, and Spark to support a production data platform modernisation and migration programme.

The role will focus on migrating and modernising distributed data platforms while maintaining application compatibility, data integrity, performance, and operational stability. The ideal candidate must have real production experience operating and migrating HBase/EMR environments, not only experience with Spark, Databricks, or ETL development.

The engineer will participate in migration planning, cluster-level troubleshooting, data validation, snapshot/restore activities, cutover, rollback, and post-migration verification.


Responsibilities

Design and execute migration and modernization activities for HBase, EMR, Hadoop, and Spark environments.

Support production migration of HBase/EMR workloads across environments.

Perform cluster-level troubleshooting and resolve performance, availability, and compatibility issues.

Plan and execute HBase snapshots, restore, and migration activities.

Validate migrated data using row/region counts, checksums, reconciliation, application behavior, and other appropriate validation methods.

Support HBase data movement using relevant migration mechanisms such as Snapshot/ExportSnapshot/HFile.

Analyze dependencies between HBase, Spark, Hadoop, applications, and supporting platform components.

Support EMR version upgrades and platform modernization.

Participate in production cutovers, maintenance windows, rollback planning, and post-cutover validation.

Troubleshoot migration issues involving data integrity, cluster configuration, application connectivity, performance, and compatibility.

Work with Backend, Cloud, Architect, DevSecOps, Security, and QA teams to resolve cross-platform dependencies.

Support data validation and reconciliation between legacy and target environments.

Document migration procedures, validation results, issues, resolutions, and operational runbooks.

Contribute to automation and repeatable processes for migration and validation activities.


Must-Have Skills

5+ years of experience in Data Engineering / Big Data Engineering.

Strong hands-on production experience with HBase.

Strong experience with Amazon EMR.

Strong experience with Apache Spark.

Strong understanding of Hadoop ecosystem.

Experience with Scala and/or Spark programming.

Experience with S3 and AWS-based data platforms.

Hands-on experience with HBase/EMR migration or modernization.

Experience with production data migration and cutover activities.

Experience validating migrated data through row/region counts, reconciliation, checksums, or equivalent validation techniques.

Experience troubleshooting distributed data platforms at the cluster level.

Strong understanding of dependencies between HBase, Spark, Hadoop, and application workloads.

Experience with production support and migration-related troubleshooting.


Good-to-Have Skills

Experience with HBase 1.x and HBase 2.x.

Experience with EMR 5.x and EMR 7.x.

Experience with HBase Snapshot and ExportSnapshot.

Experience working with HFiles and Coprocessors.

Experience with Apache Phoenix.

Experience with Hive, Presto, and/or Iceberg.

Experience with Kerberos in Hadoop/HBase/EMR environments.

Experience with KMS and encrypted data platforms.

Experience with data integrity validation using checksums.

Experience with rollback and snapshot/restore strategies.

Experience supporting large-scale enterprise data platform modernization.

Read more
Bengaluru (Bangalore)
6 - 10 yrs
₹2L - ₹22L / yr
ETL Development
Hadoop
PySpark
skill iconPython
SQL
+3 more

Job Description

We are looking for an experienced ETL Developer with strong expertise in ETL development, Hadoop, PySpark, Python, and SQL. The candidate should have hands-on experience in developing data pipelines, processing and transforming large datasets, and working with data warehousing and big data technologies.

Key Responsibilities

  • Design, develop, and maintain ETL processes and data pipelines.
  • Work with Hadoop and PySpark to process and transform large volumes of data.
  • Develop and optimize Python scripts for data processing and automation.
  • Write complex SQL queries for data extraction, transformation, and validation.
  • Perform data cleansing, integration, transformation, and validation.
  • Work with data warehousing concepts and large-scale data processing systems.
  • Optimize ETL workflows for performance, scalability, and reliability.
  • Troubleshoot data pipeline failures and resolve data quality issues.
  • Collaborate with data engineers, analysts, and other technical teams.
  • Follow data governance, quality, and development best practices.

Required Skills

  • Strong experience in ETL Development – Mandatory
  • Hands-on experience with Hadoop – Mandatory
  • Strong experience in PySpark – Mandatory
  • Good programming skills in Python – Mandatory
  • Strong knowledge of SQL – Mandatory
  • Experience in Data Processing and Transformation – Mandatory
  • Knowledge of Data Warehousing – Mandatory
  • Experience with Big Data Technologies – Mandatory

Opportunity Details

Role: ETL Developer – Hadoop & PySpark

Experience: 6–10 Years

Location: Bangalore

Mandatory Skills

ETL Development | Hadoop | PySpark | Python | SQL | Data Processing & Transformation | Data Warehousing | Big Data Technologies

Read more
VY SYSTEMS PRIVATE LIMITED
Actively reviewing applications
Revathi R
Posted by Revathi R
Mumbai
5 - 10 yrs
₹5L - ₹25L / yr
Apache Spark
skill iconJava
SQL
databricks
ETL

🚀 WE ARE HIRING | JAVA + APACHE SPARK DEVELOPER

📍 Location: Mumbai

💼 Experience: 5+ Years

🏢 Work Mode: Hybrid (3 Days Work From Office)


Are you an experienced Java + Apache Spark Developer looking for your next career opportunity? We are looking for talented professionals with strong expertise in Java, Big Data technologies, and data engineering.

🔥 Required Technical Skills:

  • Strong experience in Java Development
  • Hands-on experience with Apache Spark
  • Good knowledge of Spark SQL
  • Experience with Databricks
  • Strong SQL skills
  • Experience in ETL processes and data pipelines
  • Understanding of distributed data processing
  • Good problem-solving and analytical skills


📩 Interested candidates can share their updated resumes:

📧 Email: revathi.r @vysystems.com

📱 WhatsApp: 6 3 8 5 0 7 6 1 4 2


Read more
VY SYSTEMS PRIVATE LIMITED
Bengaluru (Bangalore)
6 - 10 yrs
₹5L - ₹30L / yr
ETL
ETL Development
skill iconPython
PySpark

Hiring: ETL Developer – Hadoop & PySpark

Location: Bangalore

Experience: 6–10 Years

Budget: 25–30 LPA

Work Mode: Weekdays

Job Description

We are looking for an experienced ETL Developer with strong expertise in Hadoop, PySpark, Python, and SQL to design, develop, and optimize data processing and transformation solutions. The ideal candidate should have hands-on experience working with big data technologies, building scalable ETL pipelines, and managing data for data warehousing and analytics.

Key Responsibilities

  • Design, develop, and maintain scalable ETL pipelines for large-scale data processing.
  • Develop data transformation workflows using PySpark and Python.
  • Work with Hadoop and other big data technologies to process and manage large datasets.
  • Write complex SQL queries for data extraction, transformation, validation, and analysis.
  • Implement data cleansing, data quality checks, and data validation processes.
  • Develop and optimize data integration workflows for data warehousing solutions.
  • Troubleshoot ETL failures, performance issues, and data discrepancies.
  • Optimize PySpark jobs and SQL queries for performance and scalability.
  • Collaborate with data engineers, architects, analysts, and other stakeholders to deliver reliable data solutions.
  • Ensure data accuracy, consistency, and adherence to established data governance standards.

Mandatory Skills

  • Strong hands-on experience in ETL development.
  • Good experience with Hadoop and big data technologies.
  • Proficiency in PySpark and Python programming.
  • Strong SQL skills, including complex queries and data manipulation.
  • Experience in data processing, data transformation, and data integration.
  • Understanding of data warehousing concepts and architecture.
  • Experience in developing, debugging, and optimizing ETL pipelines.

Required Experience

  • 6–10 years of relevant experience in ETL development and data engineering.
  • Practical experience handling large datasets using Hadoop and PySpark.
  • Strong analytical, problem-solving, and troubleshooting skills.
  • Ability to work independently and collaborate effectively within a team.


Interested candidates can share their updated resumes along with current CTC, expected CTC, notice period, and total relevant experience.

Read more
MNC
MNC
Agency job
via VY SYSTEMS PRIVATE LIMITED by Akilandeswari Panneerselvam
Bengaluru (Bangalore)
6 - 9 yrs
₹8L - ₹15L / yr
ETL
Informatica
Hadoop
Apache Spark
Snow flake schema

Position: Backup Data ETL Developer

Experience: 6–9 Years

Location:Mumbai

Employment Type: Full-time

Job Description:

We are looking for a Backup Data ETL Developer with 6–9 years of experience in designing, developing, and supporting data extraction, transformation, and loading (ETL) processes.

The candidate should have strong hands-on experience with Hadoop, Spark, Informatica, and Snowflake, along with expertise in large-scale data processing and data integration technologies.

Key Skills:

Hadoop

Spark

Informatica

Snowflake

ETL Development

Data Integration

Data Transformation

Large-Scale Data Processing

ETL Workflows and Data Pipelines

Troubleshooting and Problem-Solving

Strong analytical and collaboration skills

Read more
VY SYSTEMS PRIVATE LIMITED
Actively reviewing applications
Revathi R
Posted by Revathi R
Mumbai, Pune, Bengaluru (Bangalore)
6 - 12 yrs
₹5L - ₹25L / yr
skill iconJava
Apache Spark
Snowflake
SQL
Snow flake schema
+2 more

🚀 WE ARE HIRING | JAVA + APACHE SPARK DEVELOPER 🚀


📌 Job Details:

  • Position: Java + Apache Spark Developer
  • Experience: 6+ Years
  • Locations: Mumbai / Bangalore
  • Work Mode: Hybrid (3 days onsite per week)


💻 Required Technical Skills:

  • Java
  • Apache Spark
  • Snowflake
  • Spark SQL
  • Databricks
  • SQL
  • ETL


Interested candidates can share their updated resumes:

Email: revathi.r@ vysystems.com

WhatsApp: 6 3 8 5 0 7 6 1 4 2

Read more
VY SYSTEMS PRIVATE LIMITED
Actively reviewing applications
Pavithra E
Posted by Pavithra E
Bengaluru (Bangalore)
6 - 13 yrs
₹5L - ₹30L / yr
Amazon EMR
skill iconPython
skill iconScala

🚨 HIRING | Data Engineer – Amazon EMR 🔥


We are looking for experienced Data Engineers with strong hands-on expertise in Amazon EMR, Python, and Scala.


📌 JOB DETAILS:


💼 Experience: 6+ Years

💰 CTC: Up to 30 LPA

📍 Location: Bangalore


🔑 MANDATORY SKILLS:


• Amazon EMR

• Python

• Scala

• Data Engineering


📋 KEY REQUIREMENTS:


• 6+ years of relevant experience in Data Engineering.

• Strong hands-on experience with Amazon EMR.

• Strong programming skills in Python and Scala.

• Experience with large-scale data processing and data engineering solutions.

• Strong problem-solving and communication skills.


📩 Interested candidates can share their updated resumes:

📧 pavithra.e @vysystems. com

📞 8 9 2 5 8 0 2 4 5 7


🔄 Know someone who would be a good fit? Tag them or share this post!


#Hiring #DataEngineer #DataEngineering #AmazonEMR #Python #Scala #AWSJobs #BangaloreJobs #ITJobs #TechHiring #HiringNow

Read more
Top MNC
Actively reviewing applications
Top MNC
Agency job
via VY SYSTEMS PRIVATE LIMITED by sandhiya lakshmi
Remote only
9 - 15 yrs
₹22L - ₹28L / yr
Data Engineer
Genie
Azure
skill iconAmazon Web Services (AWS)
SQL
+1 more

Job Summary

We are looking for an experienced Databricks Lead to lead and manage data engineering initiatives. The ideal candidate should have strong hands-on experience with Databricks, data engineering, and cloud-based data platforms, along with the ability to guide a team, drive technical decisions, and ensure successful project delivery.

Key Responsibilities

  • Lead and manage Databricks-based data engineering projects.
  • Design, develop, and implement scalable data pipelines using Databricks.
  • Provide technical guidance and mentorship to team members.
  • Work on data ingestion, transformation, processing, and integration.
  • Develop and optimize ETL/ELT workflows using PySpark and Spark SQL.
  • Handle performance tuning and optimization of Databricks workloads.
  • Collaborate with business stakeholders, architects, and cross-functional teams.
  • Identify and resolve technical issues and ensure smooth project delivery.
  • Follow best practices for data quality, security, and governance.
  • Participate in technical discussions, code reviews, and solution design.

Mandatory Skills

  • Strong hands-on experience with Databricks.
  • Good knowledge of Apache Spark, PySpark, and Spark SQL.
  • Experience in data engineering and ETL/ELT pipelines.
  • Strong programming knowledge in Python and SQL.
  • Experience with cloud platforms such as AWS, Azure, or GCP.
  • Understanding of data lake and lakehouse architecture.
  • Experience in team leadership and technical coordination.
  • Strong problem-solving and communication skills.


Read more
Why apply to jobs via Cutshort
people_solving_puzzle
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
people_verifying_people
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly.
ai_chip
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
companies logo
companies logo
companies logo
companies logo
companies logo
Get to hear about interesting companies hiring right now
Company logo
Company logo
Company logo
Company logo
Company logo
Linkedin iconFollow Cutshort
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos