8+ Big data Jobs in Ahmedabad | Big data Job openings in Ahmedabad

🚀 We Are Hiring: Data Engineer | 4+ Years Experience 🚀

Job description

🔍 Job Title: Data Engineer

📍 Location: Ahmedabad

🚀 Work Mode: On-Site Opportunity

📅 Experience: 4+ Years

🕒 Employment Type: Full-Time

⏱️ Availability : Immediate Joiner Preferred

Join Our Team as a Data Engineer

We are seeking a passionate and experienced Data Engineer to be a part of our dynamic and forward-thinking team in Ahmedabad. This is an exciting opportunity for someone who thrives on transforming raw data into powerful insights and building scalable, high-performance data infrastructure.

As a Data Engineer, you will work closely with data scientists, analysts, and cross-functional teams to design robust data pipelines, optimize data systems, and enable data-driven decision-making across the organization.

Your Key Responsibilities

Architect, build, and maintain scalable and reliable data pipelines from diverse data sources.

Design effective data storage, retrieval mechanisms, and data models to support analytics and business needs.

Implement data validation, transformation, and quality monitoring processes.

Collaborate with cross-functional teams to deliver impactful, data-driven solutions.

Proactively identify bottlenecks and optimize existing workflows and processes.

Provide guidance and mentorship to junior engineers in the team.

Skills & Expertise We’re Looking For

3+ years of hands-on experience in Data Engineering or related roles.

Strong expertise in Python and data pipeline design.

Experience working with Big Data tools like Hadoop, Spark, Hive.

Proficiency with SQL, NoSQL databases, and data warehousing solutions.

Solid experience in cloud platforms - Azure

Familiar with distributed computing, data modeling, and performance tuning.

Understanding of DevOps, Power Automate, and Microsoft Fabric is a plus.

Strong analytical thinking, collaboration skills, Excellent Communication Skill and the ability to work independently or as part of a team.

Qualifications

Bachelor’s degree in Computer Science, Data Science, or a related field.

1. Data Engineer

 Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON

Mandatory Requirements  

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

 QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus:  

• AWS certification
• Spark Streaming 
• Kafka Streaming / Kafka Connect 
• ELK Stack 
• Cassandra / MongoDB 
• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Job Description

Mandatory Requirements 

• Experience in AWS Glue

• Experience in Apache Parquet 

• Proficient in AWS S3 and data lake 

• Knowledge of Snowflake

• Understanding of file-based ingestion best practices.

• Scripting language - Python & pyspark

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 

• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 

• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 

• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.

• Define process improvement opportunities to optimize data collection, insights and displays.

• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 

• Identify and interpret trends and patterns from complex data sets 

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 

• Key participant in regular Scrum ceremonies with the agile teams  

• Proficient at developing queries, writing reports and presenting findings 

• Mentor junior members and bring best industry practices.

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 

• Strong background in math, statistics, computer science, data science or related discipline

• Advanced knowledge one of language: Java, Scala, Python, C# 

• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  

• Proficient with

• Data mining/programming tools (e.g. SAS, SQL, R, Python)

• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

• Data visualization (e.g. Tableau, Looker, MicroStrategy)

• Comfortable learning about and deploying new technologies and tools. 

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 

• Good written and oral communication skills and ability to present results to non-technical audiences 

• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus: 

• AWS certification

• Spark Streaming 

• Kafka Streaming / Kafka Connect 

• ELK Stack 

• Cassandra / MongoDB 

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Rules & Responsibilities:

Technical Skills:

1. .Net– Net, C#, .Net core, MVC, Framework, Web API, Web Services, Micro Service and SQL
2. Azure – Azure Cloud, SaaS, PaaS, IaaS, Azure Relational, and No-SQL Database, Big Data Services

Responsibilities

• Good understanding of and experience in working on Microsoft Azure (IAAS/PAAS/SAAS)
• Ability to architect, design, and implement cloud-based solutions
• Proven track record of designing and implement the IoT-based solutions/Big Data solutions/applications to the Azure cloud platform.
• Experience in building .Net-based enterprise distributed solutions in Windows and Linux.
• Experience in using CI and CD tools. Jenkins/ Azure pipeline and Terraform. Experience in using another tooling such as Ansible, CloudFormation, etc.
• Good understanding of HA/DR Setups in Cloud
• Experience and working knowledge of Virtualization, Networking, Data Center, and Security
• Deep hands-on experience in the design, development, and deployment of business software at scale.
• Strong hands-on experience in Azure Cloud Platform
• Experience in Kubernetes, Docker, and other cloud deployment, container technologies
• Experience / knowledge of other cloud offerings (e.g. AWS, GCP) will be added advantage
• Experience with monitoring tools like Prometheus, Grafana, Datadog, etc.