32+ Spark Jobs in Chennai | Spark Job openings in Chennai

Role**

Ab Initio Developer

Required Technical Skill Set**

               Ab Initio, GDE, EME Conduct IT and Hadoop, Spark

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

(Ideally should not be more than 3-5)

• Should have very good knowledge and implementation experience about ETL/ELT methodology, Data Quality Strategy and Data Principles
• Experience in Project Code Management Tools(Repository/Maven/Version Control/Servers)
• Good knowledge Database Concepts (SQL)
• Good Exposure of Project management tools (JIRA, Confluence)
• Working experience in Agile methodology (Scrum/Kanban)

What you’ll do

• Tame data → pull, clean, and shape structured & unstructured data.
• Orchestrate pipelines → Airflow / Step Functions / ADF… your call.
• Ship models → build, tune, and push to prod on SageMaker, Azure ML, or Vertex AI.
• Scale → Spark / Databricks for the heavy lifting.
• Automate everything → Docker, Kubernetes, CI/CD, MLFlow, Seldon, Kubeflow.
• Pair up → work with engineers, architects, and business folks to solve real problems, fast.

What you bring

• 3+ yrs hands-on MLOps (4-5 yrs total software experience).
• Proven chops on one hyperscaler (AWS, Azure, or GCP).
• Confidence with Databricks / Spark, Python, SQL, TensorFlow / PyTorch / Scikit-learn.
• You debug Kubernetes in your sleep and treat Dockerfiles like breathing.
• You prototype with open-source first, choose the right tool, then make it scale.
• Sharp mind, low ego, bias for action.

Nice-to-haves

• Sagemaker, Azure ML, or Vertex AI in production.
• Love for clean code, clear docs, and crisp PRs.

We are looking for a Senior Data Engineer with strong expertise in GCP, Databricks, and Airflow to design and implement a GCP Cloud Native Data Processing Framework. The ideal candidate will work on building scalable data pipelines and help migrate existing workloads to a modern framework.

• Shift: 2 PM 11 PM
• Work Mode: Hybrid (3 days a week) across Xebia locations
• Notice Period: Immediate joiners or those with a notice period of up to 30 days

Key Responsibilities:

• Design and implement a GCP Native Data Processing Framework leveraging Spark and GCP Cloud Services.
• Develop and maintain data pipelines using Databricks and Airflow for transforming Raw → Silver → Gold data layers.
• Ensure data integrity, consistency, and availability across all systems.
• Collaborate with data engineers, analysts, and stakeholders to optimize performance.
• Document standards and best practices for data engineering workflows.

Required Experience:

• 7-8 years of experience in data engineering, architecture, and pipeline development.
• Strong knowledge of GCP, Databricks, PySpark, and BigQuery.
• Experience with Orchestration tools like Airflow, Dagster, or GCP equivalents.
• Understanding of Data Lake table formats (Delta, Iceberg, etc.).
• Proficiency in Python for scripting and automation.
• Strong problem-solving skills and collaborative mindset.

⚠️ Please apply only if you have not applied recently or are not currently in the interview process for any open roles at Xebia.

Looking forward to your response!

Best regards,

Vijay S

Assistant Manager - TAG

https://www.linkedin.com/in/vijay-selvarajan/

Level of skills and experience:

5 years of hands-on experience in using Python, Spark,Sql.

Experienced in AWS Cloud usage and management.

Experience with Databricks (Lakehouse, ML, Unity Catalog, MLflow).

Experience using various ML models and frameworks such as XGBoost, Lightgbm, Torch.

Experience with orchestrators such as Airflow and Kubeflow.

Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes).

Fundamental understanding of Parquet, Delta Lake and other data file formats.

Proficiency on an IaC tool such as Terraform, CDK or CloudFormation.

Strong written and verbal English communication skill and proficient in communication with non-technical stakeholderst

We are seeking a highly skilled and experienced Offshore Data Engineer . The role involves designing, implementing, and testing data pipelines and products.

Qualifications & Experience:

bachelor's or master's degree in computer science, Information Systems, or a related field.

5+ years of experience in data engineering, with expertise in data architecture and pipeline development.

☁️ Proven experience with GCP, Big Query, Databricks, Airflow, Spark, DBT, and GCP Services.

️ Hands-on experience with ETL processes, SQL, PostgreSQL, MySQL, MongoDB, Cassandra.

Strong proficiency in Python and data modelling.

Experience in testing and validation of data pipelines.

Preferred: Experience with eCommerce systems, data visualization tools (Tableau, Looker), and cloud certifications.

If you meet the above criteria and are interested, please share your updated CV along with the following details:

Total Experience:

Current CTC:

Expected CTC:

Current Location:

Preferred Location:

Notice Period / Last Working Day (if serving notice):

⚠️ Kindly share your details only if you have not applied recently or are not currently in the interview process for any open roles at Xebia.

Looking forward to your response!

We are looking for skilled Data Engineer to design, build, and maintain robust data pipelines and infrastructure. You will play a pivotal role in optimizing data flow, ensuring scalability, and enabling seamless access to structured/unstructured data across the organization. This role requires technical expertise in Python, SQL, ETL/ELT frameworks, and cloud data warehouses, along with strong collaboration skills to partner with cross-functional teams.

Company: BigThinkCode Technologies

URL:

Location: Chennai (Work from office / Hybrid)

Experience: 4 - 6 years

Key Responsibilities:

• Design, develop, and maintain scalable ETL/ELT pipelines to process structured and unstructured data.
• Optimize and manage SQL queries for performance and efficiency in large-scale datasets.
• Experience working with data warehouse solutions (e.g., Redshift, BigQuery, Snowflake) for analytics and reporting.
• Collaborate with data scientists, analysts, and business stakeholders to translate requirements into technical solutions.
• Experience in Implementing solutions for streaming data (e.g., Apache Kafka, AWS Kinesis) is preferred but not mandatory.
• Ensure data quality, governance, and security across pipelines and storage systems.
• Document architectures, processes, and workflows for clarity and reproducibility.

Required Technical Skills:

• Proficiency in Python for scripting, automation, and pipeline development.
• Expertise in SQL (complex queries, optimization, and database design).
• Hands-on experience with ETL/ELT tools (e.g., Apache Airflow, dbt, AWS Glue).
• Experience working with structured data (RDBMS) and unstructured data (JSON, Parquet, Avro).
• Familiarity with cloud-based data warehouses (Redshift, BigQuery, Snowflake).
• Knowledge of version control systems (e.g., Git) and CI/CD practices.

Preferred Qualifications:

• Experience with streaming data technologies (e.g., Kafka, Kinesis, Spark Streaming).
• Exposure to cloud platforms (AWS, GCP, Azure) and their data services.
• Understanding of data modelling (dimensional, star schema) and optimization techniques.

Soft Skills:

• Team player with a collaborative mindset and ability to mentor junior engineers.
• Strong stakeholder management skills to align technical solutions with business goals.
• Excellent communication skills to explain technical concepts to non-technical audiences.
• Proactive problem-solving and adaptability in fast-paced environments.

If interested, apply / reply by sharing your updated profile to connect and discuss.

Regards

Position Overview: We are seeking a talented Data Engineer with expertise in Power BI to join our team. The ideal candidate will be responsible for designing and implementing data pipelines, as well as developing insightful visualizations and reports using Power BI. Additionally, the candidate should have strong skills in Python, data analytics, PySpark, and Databricks. This role requires a blend of technical expertise, analytical thinking, and effective communication skills.

Key Responsibilities:

1. Design, develop, and maintain data pipelines and architectures using PySpark and Databricks.
2. Implement ETL processes to extract, transform, and load data from various sources into data warehouses or data lakes.
3. Collaborate with data analysts and business stakeholders to understand data requirements and translate them into actionable insights.
4. Develop interactive dashboards, reports, and visualizations using Power BI to communicate key metrics and trends.
5. Optimize and tune data pipelines for performance, scalability, and reliability.
6. Monitor and troubleshoot data infrastructure to ensure data quality, integrity, and availability.
7. Implement security measures and best practices to protect sensitive data.
8. Stay updated with emerging technologies and best practices in data engineering and data visualization.
9. Document processes, workflows, and configurations to maintain a comprehensive knowledge base.

Requirements:

1. Bachelor’s degree in Computer Science, Engineering, or related field. (Master’s degree preferred)
2. Proven experience as a Data Engineer with expertise in Power BI, Python, PySpark, and Databricks.
3. Strong proficiency in Power BI, including data modeling, DAX calculations, and creating interactive reports and dashboards.
4. Solid understanding of data analytics concepts and techniques.
5. Experience working with Big Data technologies such as Hadoop, Spark, or Kafka.
6. Proficiency in programming languages such as Python and SQL.
7. Hands-on experience with cloud platforms like AWS, Azure, or Google Cloud.
8. Excellent analytical and problem-solving skills with attention to detail.
9. Strong communication and collaboration skills to work effectively with cross-functional teams.
10. Ability to work independently and manage multiple tasks simultaneously in a fast-paced environment.

Preferred Qualifications:

• Advanced degree in Computer Science, Engineering, or related field.
• Certifications in Power BI or related technologies.
• Experience with data visualization tools other than Power BI (e.g., Tableau, QlikView).
• Knowledge of machine learning concepts and frameworks.

Analytics Job Description

We are hiring an Analytics Engineer to help drive our Business Intelligence efforts. You will

partner closely with leaders across the organization, working together to understand the how

and why of people, team and company challenges, workflows and culture. The team is

responsible for delivering data and insights that drive decision-making, execution, and

investments for our product initiatives.

You will work cross-functionally with product, marketing, sales, engineering, finance, and our

customer-facing teams enabling them with data and narratives about the customer journey.

You’ll also work closely with other data teams, such as data engineering and product analytics,

to ensure we are creating a strong data culture at Blend that enables our cross-functional partners

to be more data-informed.

Role : DataEngineer 

Please find below the JD for the DataEngineer Role..

  Location: Guindy,Chennai

How you’ll contribute:

• Develop objectives and metrics, ensure priorities are data-driven, and balance short-

term and long-term goals

• Develop deep analytical insights to inform and influence product roadmaps and

business decisions and help improve the consumer experience

• Work closely with GTM and supporting operations teams to author and develop core

data sets that empower analyses

• Deeply understand the business and proactively spot risks and opportunities

• Develop dashboards and define metrics that drive key business decisions

• Build and maintain scalable ETL pipelines via solutions such as Fivetran, Hightouch,

and Workato

• Design our Analytics and Business Intelligence architecture, assessing and

implementing new technologies that fitting

• Work with our engineering teams to continually make our data pipelines and tooling

more resilient

Who you are:

• Bachelor’s degree or equivalent required from an accredited institution with a

quantitative focus such as Economics, Operations Research, Statistics, Computer Science OR 1-3 Years of Experience as a Data Analyst, Data Engineer, Data Scientist

• Must have strong SQL and data modeling skills, with experience applying skills to

thoughtfully create data models in a warehouse environment.

• A proven track record of using analysis to drive key decisions and influence change

• Strong storyteller and ability to communicate effectively with managers and

executives

• Demonstrated ability to define metrics for product areas, understand the right

questions to ask and push back on stakeholders in the face of ambiguous, complex

problems, and work with diverse teams with different goals

• A passion for documentation.

• A solution-oriented growth mindset. You’ll need to be a self-starter and thrive in a

dynamic environment.

• A bias towards communication and collaboration with business and technical

stakeholders.

• Quantitative rigor and systems thinking.

• Prior startup experience is preferred, but not required.

• Interest or experience in machine learning techniques (such as clustering, decision

tree, and segmentation)

• Familiarity with a scientific computing language, such as Python, for data wrangling

and statistical analysis

• Experience with a SQL focused data transformation framework such as dbt

• Experience with a Business Intelligence Tool such as Mode/Tableau

Mandatory Skillset:

-Very Strong in SQL

-Spark OR pyspark OR Python

-Shell Scripting

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

Bigdata with cloud:

Experience : 5-10 years

Location : Hyderabad/Chennai

Notice period : 15-20 days Max

1.  Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight

2.  Experience in developing lambda functions with AWS Lambda

3.  Expertise with Spark/PySpark – Candidate should be hands on with PySpark code and should be able to do transformations with Spark

4.  Should be able to code in Python and Scala.

5.  Snowflake experience will be a plus

• 5+ years of experience in a Data Engineering role on cloud environment

• Must have good experience in Scala/PySpark (preferably on data-bricks environment)

• Extensive experience with Transact-SQL.
• Experience in Data-bricks/Spark.

• Strong experience in Dataware house projects
• Expertise in database development projects with ETL processes.
• Manage and maintain data engineering pipelines

• Develop batch processing, streaming and integration solutions
• Experienced in building and operationalizing large-scale enterprise data solutions and applications

• Using one or more of Azure data and analytics services in combination with custom solutions
• Azure Data Lake, Azure SQL DW (Synapse), and SQL Database products or equivalent products from other cloud services providers

• In-depth understanding of data management (e. g. permissions, security, and monitoring).
• Cloud repositories for e.g. Azure GitHub, Git
• Experience in an agile environment (Prefer Azure DevOps).

Good to have

• Manage source data access security
• Automate Azure Data Factory pipelines
• Continuous Integration/Continuous deployment (CICD) pipelines, Source Repositories
• Experience in implementing and maintaining CICD pipelines
• Power BI understanding, Delta Lake house architecture
• Knowledge of software development best practices.
• Excellent analytical and organization skills.
• Effective working in a team as well as working independently.
• Strong written and verbal communication skills.
• Expertise in database development projects and ETL processes.

Job Title: Engineering Manager

Job Location: Chennai, Bangalore
Job Summary
The Engineering Org is looking for a proficient Engineering Manager to join a team that is building exciting
and futuristic Data Products at Condé Nast to enable both internal and external marketers to target
audiences in real time. As an Engineering Manager, you will drive the day-to-day execution of technical
and architectural decisions. EM will own engineering deliverables inclusive of solving dependencies
such as architecture, solutions, sequencing, and working with other engineering delivery teams.This role
is also responsible for driving innovation, prototyping, and recommending solutions. Above all, you will
influence how users interact with Conde Nast’s industry-leading journalism.
● Primary Responsibilities
● Manage a high performing team of Software and Data Engineers within the Data & ML
Engineering team part of Engineering Data Organization.
● Provide leadership and guidance to the team in Data Discovery, Data Ingestion, Transformation
and Storage
● Utilizing product mindset to build, scale and deploy holistic data products after successful
prototyping and drive their engineering implementation
● Provide technical coaching and lead direct reports and other members of adjacent support teams
to the highest level of performance..
● Evaluate performance of direct reports and offer career development guidance.
● Meeting hiring and retention targets of the team & building a high-performance culture
● Handle escalations from internal stakeholders and manage critical issues to resolution.
● Collaborate with Architects, Product Manager, Project Manager and other teams to deliver high
quality products.
● Identify recurring system and application issues and enable engineers to work with release teams,
infra teams, product development, vendors and other stakeholders in investigating and resolving
the cause.
● Required Skills
● 4+ years of managing Software Development teams, preferably in ML and Data
Engineering teams.
● 4+ years of Agile Software development practices
● 12+ years of Software Development experience.
● Excellent Problem Solving and System Design skill
● Hands on: Writing and Reviewing code primarily in Spark, Python and/or Java
● Hand on: Architect & Design end to end Data Pipeline (noSQL databases, Job Schedulers, Big
Data Development preferably on Databricks / Cloud)
● Experience with SOA & Microservice architecture
● Knowledge of Software Engineering best practices with experience on implementing CI/CD,
Log aggregation/Monitoring/alerting for production system
● Working Knowledge of cloud and devops skills (AWS will be preferred)
● Strong verbal and written communication skills.
● Experience in evaluating team member performance and offering career development
guidance.
● Experience in providing technical coaching to direct reports.
● Experience in architecting highly scalable products.
● Experience in collaborating with global stakeholder teams.
● Experience in working on highly available production systems.
● Strong knowledge of software release process and release pipeline.
About Condé Nast
CONDÉ NAST INDIA (DATA)
Over the years, Condé Nast successfully expanded and diversified into digital, TV, and social
platforms - in other words, a staggering amount of user data. Condé Nast made the right move to
invest heavily in understanding this data and formed a whole new Data team entirely dedicated to
data processing, engineering, analytics, and visualization. This team helps drive engagement, fuel
process innovation, further content enrichment, and increase market revenue. The Data team
aimed to create a company culture where data was the common language and facilitate an
environment where insights shared in real-time could improve performance.
The Global Data team operates out of Los Angeles, New York, Chennai, and London. The team at
Condé Nast Chennai works extensively with data to amplify its brands' digital capabilities and boost
online revenue. We are broadly divided into four groups, Data Intelligence, Data Engineering, Data
Science, and Operations (including Product and Marketing Ops, Client Services) along with Data
Strategy and monetization. The teams built capabilities and products to create data-driven solutions
for better audience engagement.
What we look forward to:
We want to welcome bright, new minds into our midst and work together to create diverse forms of
self-expression. At Condé Nast, we encourage the imaginative and celebrate the extraordinary. We
are a media company for the future, with a remarkable past. We are Condé Nast, and It Starts Here.

Function : Sr. DB Developer

Location : India/Gurgaon/Tamilnadu

>> THE INDIVIDUAL

• Have a strong background in data platform creation and management.
• Possess in-depth knowledge of Data Management, Data Modelling, Ingestion  - Able to develop data models and ingestion frameworks based on client requirements and advise on system optimization.
• Hands-on experience in SQL database (PostgreSQL) and No-SQL database (MongoDB) 
• Hands-on experience in performance tuning of DB
• Good to have knowledge of database setup in cluster node
• Should be well versed with data security aspects and data governance framework
• Hands-on experience in Spark, Airflow, ELK.
• Good to have knowledge on any data cleansing tool like apache Griffin
• Preferably getting involved during project implementation so have a background on business knowledge and technical requirement as well.
• Strong analytical and problem-solving skills. Have exposure to data analytics skills and knowledge of advanced data analytical tools will be an advantage.
• Strong written and verbal communication skills (presentation skills).
• Certifications in the above technologies is preferred.

>> Qualification

1. Tech /B.E. / MCA /M. Tech from a reputed institute.

Experience of Data Management, Data Modelling, Ingestion for more than 4 years. Total experience of 8-10 Years

Should have Passion to learn and adapt new technologies, understanding,

solving/troubleshooting issues and risks, able to make informed decisions and ability to

lead the projects.

Your Qualifications

• 2-5 Years’ Experience with functional programming
• Experience with functional programming using Scala with Spark framework.
• Strong understanding of Object-oriented programming, data structures and algorithms
• Good experience in any of the cloud platforms (Azure, AWS, GCP) etc.,
• Experience with distributed (multi-tiered) systems, relational databases and NoSql storage solutions
• Desire to learn new technologies and languages
• Participation in software design, development, and code reviews
• High level of proficiency with Computer Science/Software Engineering knowledge and contribution to the technical skills growth of other team members

Your Responsibility

• Design, build and configure applications to meet business process and application requirements
• Proactively identify and communicate potential issues and concerns and recommend/implement alternative solutions as appropriate.
• Troubleshooting & Optimization of existing solution

Job Sector: IT, Software

Job Type: Permanent

Location: Chennai

Experience: 10 - 20 Years

Salary: 12 – 40 LPA

Education: Any Graduate

Notice Period: Immediate

Key Skills: Python, Spark, AWS, SQL, PySpark

Contact at triple eight two zero nine four two double seven

Job Description:

Requirements

• Minimum 12 years experience
• In depth understanding and knowledge on distributed computing with spark.
• Deep understanding of Spark Architecture and internals
• Proven experience in data ingestion, data integration and data analytics with spark, preferably PySpark.
• Expertise in ETL processes, data warehousing and data lakes.
• Hands on with python for Big data and analytics.
• Hands on in agile scrum model is an added advantage.
• Knowledge on CI/CD and orchestration tools is desirable.
• AWS S3, Redshift, Lambda knowledge is preferred

We are looking for an outstanding Big Data Engineer with experience setting up and maintaining Data Warehouse and Data Lakes for an Organization. This role would closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Roles and Responsibilities:

• Develop and maintain scalable data pipelines and build out new integrations and processes required for optimal extraction, transformation, and loading of data from a wide variety of data sources using 'Big Data' technologies.
• Develop programs in Scala and Python as part of data cleaning and processing.
• Assemble large, complex data sets that meet functional / non-functional business requirements and fostering data-driven decision making across the organization.  
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
• Implement processes and systems to validate data, monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Provide high operational excellence guaranteeing high availability and platform stability.
• Closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Skills:

• Experience with Big Data pipeline, Big Data analytics, Data warehousing.
• Experience with SQL/No-SQL, schema design and dimensional data modeling.
• Strong understanding of Hadoop Architecture, HDFS ecosystem and eexperience with Big Data technology stack such as HBase, Hadoop, Hive, MapReduce.
• Experience in designing systems that process structured as well as unstructured data at large scale.
• Experience in AWS/Spark/Java/Scala/Python development.
• Should have Strong skills in PySpark (Python & SPARK). Ability to create, manage and manipulate Spark Dataframes. Expertise in Spark query tuning and performance optimization.
• Experience in developing efficient software code/frameworks for multiple use cases leveraging Python and big data technologies.
• Prior exposure to streaming data sources such as Kafka.
• Should have knowledge on Shell Scripting and Python scripting.
• High proficiency in database skills (e.g., Complex SQL), for data preparation, cleaning, and data wrangling/munging, with the ability to write advanced queries and create stored procedures.
• Experience with NoSQL databases such as Cassandra / MongoDB.
• Solid experience in all phases of Software Development Lifecycle - plan, design, develop, test, release, maintain and support, decommission.
• Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development).
• Experience building and deploying applications on on-premise and cloud-based infrastructure.
• Having a good understanding of machine learning landscape and concepts. 

Qualifications and Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Big Data Engineer or a similar role for 3-5 years.

Certifications:

Good to have at least one of the Certifications listed here:

    AZ 900 - Azure Fundamentals

    DP 200, DP 201, DP 203, AZ 204 - Data Engineering

    AZ 400 - Devops Certification

Company Description

At Bungee Tech, we help retailers and brands meet customers everywhere and, on every occasion, they are in. We believe that accurate, high-quality data matched with compelling market insights empowers retailers and brands to keep their customers at the center of all innovation and value they are delivering. 

We provide a clear and complete omnichannel picture of their competitive landscape to retailers and brands. We collect billions of data points every day and multiple times in a day from publicly available sources. Using high-quality extraction, we uncover detailed information on products or services, which we automatically match, and then proactively track for price, promotion, and availability. Plus, anything we do not match helps to identify a new assortment opportunity.

Empowered with this unrivalled intelligence, we unlock compelling analytics and insights that once blended with verified partner data from trusted sources such as Nielsen, paints a complete, consolidated picture of the competitive landscape.

We are looking for a Big Data Engineer who will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

You will also be responsible for integrating them with the architecture used in the company.

We're working on the future. If you are seeking an environment where you can drive innovation, If you want to apply state-of-the-art software technologies to solve real world problems, If you want the satisfaction of providing visible benefit to end-users in an iterative fast paced environment, this is your opportunity.

Responsibilities

As an experienced member of the team, in this role, you will:

• Contribute to evolving the technical direction of analytical Systems and play a critical role their design and development

• You will research, design and code, troubleshoot and support. What you create is also what you own.

• Develop the next generation of automation tools for monitoring and measuring data quality, with associated user interfaces.

• Be able to broaden your technical skills and work in an environment that thrives on creativity, efficient execution, and product innovation.

BASIC QUALIFICATIONS

• Bachelor’s degree or higher in an analytical area such as Computer Science, Physics, Mathematics, Statistics, Engineering or similar.
• 5+ years relevant professional experience in Data Engineering and Business Intelligence
• 5+ years in with Advanced SQL (analytical functions), ETL, Data Warehousing.
• Strong knowledge of data warehousing concepts, including data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools and environments, data structures, data modeling and performance tuning.
• Ability to effectively communicate with both business and technical teams.
• Excellent coding skills in Java, Python, C++, or equivalent object-oriented programming language
• Understanding of relational and non-relational databases and basic SQL
• Proficiency with at least one of these scripting languages: Perl / Python / Ruby / shell script

PREFERRED QUALIFICATIONS

• Experience with building data pipelines from application databases.
• Experience with AWS services - S3, Redshift, Spectrum, EMR, Glue, Athena, ELK etc.
• Experience working with Data Lakes.
• Experience providing technical leadership and mentor other engineers for the best practices on the data engineering space
• Sharp problem solving skills and ability to resolve ambiguous requirements
• Experience on working with Big Data
• Knowledge and experience on working with Hive and the Hadoop ecosystem
• Knowledge of Spark
• Experience working with Data Science teams

Role Summary/Purpose:

We are looking for a Developer/Senior Developers to be a part of building advanced analytical platform leveraging Big Data technologies and transform the legacy systems. This role is an exciting, fast-paced, constantly changing and challenging work environment, and will play an important role in resolving and influencing high-level decisions.

Requirements:

• The candidate must be a self-starter, who can work under general guidelines in a fast-spaced environment.
• Overall minimum of 4 to 8 year of software development experience and 2 years in Data Warehousing domain knowledge
• Must have 3 years of hands-on working knowledge on Big Data technologies such as Hadoop, Hive, Hbase, Spark, Kafka, Spark Streaming, SCALA etc…
• Excellent knowledge in SQL & Linux Shell scripting
• Bachelors/Master’s/Engineering Degree from a well-reputed university.
• Strong communication, Interpersonal, Learning and organizing skills matched with the ability to manage stress, Time, and People effectively
• Proven experience in co-ordination of many dependencies and multiple demanding stakeholders in a complex, large-scale deployment environment
• Ability to manage a diverse and challenging stakeholder community
• Diverse knowledge and experience of working on Agile Deliveries and Scrum teams.

Responsibilities

• Should works as a senior developer/individual contributor based on situations
• Should be part of SCRUM discussions and to take requirements
• Adhere to SCRUM timeline and deliver accordingly
• Participate in a team environment for the design, development and implementation
• Should take L3 activities on need basis
• Prepare Unit/SIT/UAT testcase and log the results
• Co-ordinate SIT and UAT Testing. Take feedbacks and provide necessary remediation/recommendation in time.
• Quality delivery and automation should be a top priority
• Co-ordinate change and deployment in time
• Should create healthy harmony within the team
• Owns interaction points with members of core team (e.g.BA team, Testing and business team) and any other relevant stakeholders