50+ Spark Jobs in Bangalore (Bengaluru) | Spark Job openings in Bangalore (Bengaluru)

Key Responsibilities

• Design, develop, and optimize data pipelines using Apache Spark to process large volumes of structured and unstructured data.
• Write efficient and maintainable code in Scala and Python for data extraction, transformation, and loading (ETL) operations.
• Collaborate with cross-functional teams to define data engineering solutions to support analytics and machine learning initiatives.
• Implement and maintain data lake and warehouse solutions using cloud platforms (e.g., AWS, GCP, Azure).
• Ensure data workflows and distributed systems' performance, scalability, and reliability.
• Perform data quality assessments, implement monitoring, and improve data governance practices.
• Assist in migrating and refactoring legacy data systems into modern distributed data processing platforms.
• Provide technical leadership and mentorship to junior engineers and contribute to best practices in coding, testing, and deployment.
• 
  

Qualifications

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• 6+ years of hands-on experience in data engineering, with strong skills in Apache Spark, Scala, and Python.
• Experience with distributed data processing frameworks and real-time data processing.
• Strong experience with big data technologies such as Hadoop, Hive, and Kafka.
• Proficient with relational databases (SQL, PostgreSQL, MySQL) and NoSQL databases (Cassandra, HBase, MongoDB). 
• Knowledge of CI/CD pipelines and DevOps practices for deploying data workflows.
• Strong problem-solving skills and experience with optimizing large-scale data systems.
• Excellent communication and collaboration skills.
• Experience with orchestration tools like Airflow
• Experience with containerization and orchestration (e.g., Docker, Kubernetes)

Key Responsibilities

• Design, develop, and optimize data pipelines using Apache Spark to process large volumes of structured and unstructured data.
• Write efficient and maintainable code in Scala and Python for data extraction, transformation, and loading (ETL) operations.
• Collaborate with cross-functional teams to define data engineering solutions to support analytics and machine learning initiatives.
• Implement and maintain data lake and warehouse solutions using cloud platforms (e.g., AWS, GCP, Azure).
• Ensure data workflows and distributed systems' performance, scalability, and reliability.
• Perform data quality assessments, implement monitoring, and improve data governance practices.
• Assist in migrating and refactoring legacy data systems into modern distributed data processing platforms.
• Provide technical leadership and mentorship to junior engineers and contribute to best practices in coding, testing, and deployment.

Qualifications

• Bachelor's or Master’s degree in Computer Science, Engineering, or a related field.
• 8+ years of hands-on experience in data engineering, with strong skills in Apache Spark, Scala, and Python.
• Experience with distributed data processing frameworks and real-time data processing.
• Strong experience with big data technologies such as Hadoop, Hive, and Kafka.
• Proficient with relational databases (SQL, PostgreSQL, MySQL) and NoSQL databases (Cassandra, HBase, MongoDB). 
• Knowledge of CI/CD pipelines and DevOps practices for deploying data workflows.
• Strong problem-solving skills and experience with optimizing large-scale data systems.
• Excellent communication and collaboration skills.
• Experience with orchestration tools like Airflow
• Experience with containerization and orchestration (e.g., Docker, Kubernetes)

software development and automated testing Proficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and program modifications from supplied specifications using agreed standards and tools, to achieve a well engineered result Proficient and Hands-on Data Warehousing,

Experience with Agile development, Continuous Integration, and Continuous Delivery Ability to effectively interpret technical and business objectives and provide solutions Strong communication skills, with ability to articulate technical solutions effectively across diverse group of stakeholders

Need to be a fast learner willing to adapt to evolving needs of the developer community.

Thanks & Regards

snehalata verma

IT Recruiter --HrBizHub

• Responsible for designing, storing, processing, and maintaining of large-scale data and related infrastructure.
• Can drive multiple projects both from operational and technical standpoint.
• Ideate and build PoV or PoC for new product that can help drive more business.
• Responsible for defining, designing, and implementing data engineering best practices, strategies, and solutions.
• Is an Architect who can guide the customers, team, and overall organization on tools, technologies, and best practices around data engineering.
• Lead architecture discussions, align with business needs, security, and best practices.
• Has strong conceptual understanding of Data Warehousing and ETL, Data Governance and Security, Cloud Computing, and Batch & Real Time data processing
• Has strong execution knowledge of Data Modeling, Databases in general (SQL and NoSQL), software development lifecycle and practices, unit testing, functional programming, etc.
• Understanding of Medallion architecture pattern
• Has worked on at least one cloud platform.
• Has worked as data architect and executed multiple end-end data engineering project.
• Has extensive knowledge of different data architecture designs and data modelling concepts.
• Manages conversation with the client stakeholders to understand the requirement and translate it into technical outcomes.

Required Tech Stack

• Strong proficiency in SQL
• Experience working on any of the three major cloud platforms i.e., AWS/Azure/GCP
• Working knowledge of an ETL and/or orchestration tools like IICS, Talend, Matillion, Airflow, Azure Data Factory, AWS Glue, GCP Composer, etc.
• Working knowledge of one or more OLTP databases (Postgres, MySQL, SQL Server, etc.)
• Working knowledge of one or more Data Warehouse like Snowflake, Redshift, Azure Synapse, Hive, Big Query, etc.
• Proficient in at least one programming language used in data engineering, such as Python (or Scala/Rust/Java)
• Has strong execution knowledge of Data Modeling (star schema, snowflake schema, fact vs dimension tables)
• Proficient in Spark and related applications like Databricks, GCP DataProc, AWS Glue, EMR, etc.
• Has worked on Kafka and real-time streaming.
• Has strong execution knowledge of data architecture design patterns (lambda vs kappa architecture, data harmonization, customer data platforms, etc.)
• Has worked on code and SQL query optimization.
• Strong knowledge of version control systems like Git to manage source code repositories and designing CI/CD pipelines for continuous delivery.
• Has worked on data and networking security (RBAC, secret management, key vaults, vnets, subnets, certificates)

Nielsen, a global company specialising in audience measurement and analytics, is currently seeking a proficient leader in data engineering to join their team in Bangalore, Gurgaon, or Mumbai.

This is a manager of managers role that involves managing multiple scrum teams and overseeing an advanced data platform that analyses audience consumption patterns across various channels like OTT, TV, Radio, and Social Media worldwide. You will be responsible for building and supervising a top-performing data engineering team that delivers data for targeted campaigns. Moreover, you will work with AWS services (S3, Lambda, Kinesis) and other data engineering technologies such as Spark, Scala/Python, Kafka, etc. There may also be opportunities to establish deep integrations with OTT platforms like Netflix, Prime Video, and other.

Responsibilities:

·       Analyze complex data sets to answer specific questions using MMIT’s market access data (MMIT) and Norstella claims data, third-party claims data (IQVIA LAAD, Symphony SHA). Applicant must have experience working with the aforementioned data sets exclusively.

·       Deliver consultative services to clients related to MMIT RWD sets

·       Produce complex analytical reports using data visualization tools such as Power BI or Tableau

·       Define customized technical specifications to surface MMIT RWD in MMIT tools. 

·       Execute work in a timely fashion with high accuracy, while managing various competing priorities; Perform thorough troubleshooting and execute QA; Communicate with internal teams to obtain required data

·       Ensure adherence to documentation requirements, process workflows, timelines, and escalation protocols

·       And other duties as assigned.

Requirements:

·       Bachelor’s Degree or relevant experience required

·       2-5 yrs. of professional experience in RWD analytics using SQL

·       Fundamental understanding of Pharma and Market access space

·       Strong analysis skills and proficiency with tools such as Tableau or PowerBI

·       Excellent written and verbal communication skills.

·       Analytical, critical thinking and creative problem-solving skills.

·       Relationship building skills.

·       Solid organizational skills including attention to detail and multitasking skills.

·       Excellent time management and prioritization skills.

Job Description: Data Engineer

Experience: Over 4 years

Responsibilities:

-       Design, develop, and maintain scalable data pipelines for efficient data extraction, transformation, and loading (ETL) processes.

-       Architect and implement data storage solutions, including data warehouses, data lakes, and data marts, aligned with business needs.

-       Implement robust data quality checks and data cleansing techniques to ensure data accuracy and consistency.

-       Optimize data pipelines for performance, scalability, and cost-effectiveness.

-       Collaborate with data analysts and data scientists to understand data requirements and translate them into technical solutions.

-       Develop and maintain data security measures to ensure data privacy and regulatory compliance.

-       Automate data processing tasks using scripting languages (Python, Bash) and big data frameworks (Spark, Hadoop).

-       Monitor data pipelines and infrastructure for performance and troubleshoot any issues.

-       Stay up to date with the latest trends and technologies in data engineering, including cloud platforms (AWS, Azure, GCP).

-        Document data pipelines, processes, and data models for maintainability and knowledge sharing.

-       Contribute to the overall data governance strategy and best practices.

Qualifications:

-       Strong understanding of data architectures, data modelling principles, and ETL processes.

-       Proficiency in SQL (e.g., MySQL, PostgreSQL) and experience with big data querying languages (e.g., Hive, Spark SQL).

-       Experience with scripting languages (Python, Bash) for data manipulation and automation.

-       Experience with distributed data processing frameworks (Spark, Hadoop) (preferred).

-       Familiarity with cloud platforms (AWS, Azure, GCP) for data storage and processing (a plus).

-       Experience with data quality tools and techniques.

-       Excellent problem-solving, analytical, and critical thinking skills.

-       Strong communication, collaboration, and teamwork abilities.

Must have skills

3 to 6 years

Data Science

SQL, Excel, Big Query - mandate 3+ years

Python/ML, Hadoop, Spark - 2+ years

Requirements

• 3+ years prior experience as a data analyst

• Detail oriented, structural thinking and analytical mindset.

• Proven analytic skills, including data analysis and data validation.

• Technical writing experience in relevant areas, including queries, reports, and presentations.

• Strong SQL and Excel skills with the ability to learn other analytic tools

• Good communication skills (being precise and clear)

• Good to have prior knowledge of python and ML algorithms

Publicis Sapient Overview:

The Senior Associate People Senior Associate L1 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution 

.

Job Summary:

As Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution

The role requires a hands-on technologist who has strong programming background like Java / Scala / Python, should have experience in Data Ingestion, Integration and data Wrangling, Computation, Analytics pipelines and exposure to Hadoop ecosystem components. You are also required to have hands-on knowledge on at least one of AWS, GCP, Azure cloud platforms.

Role & Responsibilities:

Your role is focused on Design, Development and delivery of solutions involving:

• Data Integration, Processing & Governance

• Data Storage and Computation Frameworks, Performance Optimizations

• Analytics & Visualizations

• Infrastructure & Cloud Computing

• Data Management Platforms

• Implement scalable architectural models for data processing and storage

• Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time mode

• Build functionality for data analytics, search and aggregation

Experience Guidelines:

Mandatory Experience and Competencies:

# Competency

1.Overall 5+ years of IT experience with 3+ years in Data related technologies

2.Minimum 2.5 years of experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure / GCP)

3.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline.

4.Strong experience in at least of the programming language Java, Scala, Python. Java preferable

5.Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc

6.Well-versed and working knowledge with data platform related services on at least 1 cloud platform, IAM and data security

Preferred Experience and Knowledge (Good to Have):

# Competency

1.Good knowledge of traditional ETL tools (Informatica, Talend, etc) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands on experience

2.Knowledge on data governance processes (security, lineage, catalog) and tools like Collibra, Alation etc

3.Knowledge on distributed messaging frameworks like ActiveMQ / RabbiMQ / Solace, search & indexing and Micro services architectures

4.Performance tuning and optimization of data pipelines

5.CI/CD – Infra provisioning on cloud, auto build & deployment pipelines, code quality

6.Cloud data specialty and other related Big data technology certifications

Personal Attributes:

• Strong written and verbal communication skills

• Articulation skills

• Good team player

• Self-starter who requires minimal oversight

• Ability to prioritize and manage multiple tasks

• Process orientation and the ability to define and set up processes

• As a data engineer, you will build systems that collect, manage, and convert raw data into usable information for data scientists and business analysts to interpret. You ultimate goal is to make data accessible for organizations to optimize their performance. 
• Work closely with PMs, business analysts to build and improvise data pipelines, identify and model business objects • Write scripts implementing data transformation, data structures, metadata for bringing structure for partially unstructured data and improvise quality of data 
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL
• Own data pipelines - Monitoring, testing, validating and ensuring meaningful data exists in data warehouse with high level of data quality 
• What we look for in the candidate is strong analytical skills with the ability to collect, organise, analyse, and disseminate significant amounts of information with attention to detail and accuracy 
• Create long term and short-term design solutions through collaboration with colleagues
• Proactive to experiment with new tools
• Strong programming skill in python
• Skillset: Python, SQL, ETL frameworks, PySpark and Snowflake 
• Strong communication and interpersonal skills to interact with senior-level management regarding the implementation of changes 
• Willingness to learn and eagerness to contribute to projects 
• Designing datawarehouse and most appropriate DB schema for the data product
• Positive attitude and proactive problem-solving mindset
• Experience in building data pipelines and connectors
• Knowledge on AWS cloud services would be preferred

Publicis Sapient Overview:

The Senior Associate People Senior Associate L1 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution 

.

Job Summary:

As Senior Associate L1 in Data Engineering, you will do technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution

The role requires a hands-on technologist who has strong programming background like Java / Scala / Python, should have experience in Data Ingestion, Integration and data Wrangling, Computation, Analytics pipelines and exposure to Hadoop ecosystem components. Having hands-on knowledge on at least one of AWS, GCP, Azure cloud platforms will be preferable.

Role & Responsibilities:

Job Title: Senior Associate L1 – Data Engineering

Your role is focused on Design, Development and delivery of solutions involving:

• Data Ingestion, Integration and Transformation

• Data Storage and Computation Frameworks, Performance Optimizations

• Analytics & Visualizations

• Infrastructure & Cloud Computing

• Data Management Platforms

• Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time

• Build functionality for data analytics, search and aggregation

Experience Guidelines:

Mandatory Experience and Competencies:

# Competency

1.Overall 3.5+ years of IT experience with 1.5+ years in Data related technologies

2.Minimum 1.5 years of experience in Big Data technologies

3.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline. Working knowledge on real-time data pipelines is added advantage.

4.Strong experience in at least of the programming language Java, Scala, Python. Java preferable

5.Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc

Preferred Experience and Knowledge (Good to Have):

# Competency

1.Good knowledge of traditional ETL tools (Informatica, Talend, etc) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands on experience

2.Knowledge on data governance processes (security, lineage, catalog) and tools like Collibra, Alation etc

3.Knowledge on distributed messaging frameworks like ActiveMQ / RabbiMQ / Solace, search & indexing and Micro services architectures

4.Performance tuning and optimization of data pipelines

5.CI/CD – Infra provisioning on cloud, auto build & deployment pipelines, code quality

6.Working knowledge with data platform related services on at least 1 cloud platform, IAM and data security

7.Cloud data specialty and other related Big data technology certifications

Job Title: Senior Associate L1 – Data Engineering

Personal Attributes:

• Strong written and verbal communication skills

• Articulation skills

• Good team player

• Self-starter who requires minimal oversight

• Ability to prioritize and manage multiple tasks

• Process orientation and the ability to define and set up processes

Title:- Senior Data Engineer 

Experience: 4-6 yrs

Budget: 24-28 lpa

Location: Bangalore 

Work of Mode: Work from office

Primary Skills: Data Bricks, Spark, Pyspark,Sql, Python, AWS

Qualification: Any Engineering degree

Responsibilities:

∙Design and build reusable components, frameworks and libraries at scale to support  analytics products. 

∙Design and implement product features in collaboration with business and Technology

 stakeholders. 

∙Anticipate, identify and solve issues concerning data management to improve data   quality.

∙Clean, prepare and optimize data at scale for ingestion and consumption.

∙Drive the implementation of new data management projects and re-structure of the   current data architecture.

∙Implement complex automated workflows and routines using workflow scheduling tools. 

∙Build continuous integration, test-driven development and production deployment

 frameworks.

∙Drive collaborative reviews of design, code, test plans and dataset implementation  performed by other data engineers in support of maintaining data engineering  standards.

∙Analyze and profile data for the purpose of designing scalable solutions. 

∙Troubleshoot complex data issues and perform root cause analysis to proactively resolve

 product and operational issues.

∙Mentor and develop other data engineers in adopting best practices.

Qualifications:

Primary skillset:

∙Experience working with distributed technology tools for developing Batch and  Streaming pipelines using 

  o SQL, Spark, Python, PySpark [4+ years],

  o Airflow [3+ years],

  o Scala [2+ years].

∙Able to write code which is optimized for performance.

∙Experience in Cloud platform, e.g., AWS, GCP, Azure, etc.

∙Able to quickly pick up new programming languages, technologies, and frameworks.

∙Strong skills building positive relationships across Product and Engineering.

∙Able to influence and communicate effectively, both verbally and written, with team  members and business stakeholders

∙Experience with creating/ configuring Jenkins pipeline for smooth CI/CD process for  Managed Spark jobs, build Docker images, etc.

∙Working knowledge of Data warehousing, Data modelling, Governance and Data   Architecture

Good to have:

∙Experience working with Data platforms, including EMR, Airflow, Databricks (Data

 Engineering & Delta Lake components, and Lakehouse Medallion architecture), etc.

∙Experience working in Agile and Scrum development process.

∙Experience in EMR/ EC2, Databricks etc.

∙Experience working with Data warehousing tools, including SQL database, Presto, and

 Snowflake

∙Experience architecting data product in Streaming, Serverless and Microservices  Architecture and platform.

Job Title Big Data Developer

Job Description

Bachelor's degree in Engineering or Computer Science or equivalent OR Master's in Computer Applications or equivalent.

Solid Experience of software development experience and leading teams of engineers and scrum teams.

4+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and PySpark).

Solid Datawarehousing concepts.

Knowledge of Financial reporting ecosystem will be a plus.

4+ years of experience within Data Engineering/ Data Warehousing using Big Data technologies will be an addon.

Expert on Distributed ecosystem.

Hands-on experience with programming using Core Java or Python/Scala

Expert on Hadoop and Spark Architecture and its working principle

Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes), optimizing joins while processing huge amount of data.

Experience in UNIX shell scripting.

Roles & Responsibilities

Ability to design and develop optimized Data pipelines for batch and real time data processing

Should have experience in analysis, design, development, testing, and implementation of system applications

Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.

Excellent technical and analytical aptitude

Good communication skills.

Excellent Project management skills.

Results driven Approach.

Mandatory SkillsBig Data, PySpark, Hive

Data Engineering : Senior Engineer / Manager

As Senior Engineer/ Manager in Data Engineering, you will translate client requirements into technical design, and implement components for a data engineering solutions. Utilize a deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution.

Must Have skills :

1. GCP

2. Spark streaming : Live data streaming experience is desired.

3. Any 1 coding language: Java/Pyhton /Scala

Skills & Experience :

- Overall experience of MINIMUM 5+ years with Minimum 4 years of relevant experience in Big Data technologies

- Hands-on experience with the Hadoop stack - HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline. Working knowledge on real-time data pipelines is added advantage.

- Strong experience in at least of the programming language Java, Scala, Python. Java preferable

- Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc.

- Well-versed and working knowledge with data platform related services on GCP

- Bachelor's degree and year of work experience of 6 to 12 years or any combination of education, training and/or experience that demonstrates the ability to perform the duties of the position

Your Impact :

- Data Ingestion, Integration and Transformation

- Data Storage and Computation Frameworks, Performance Optimizations

- Analytics & Visualizations

- Infrastructure & Cloud Computing

- Data Management Platforms

- Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time

- Build functionality for data analytics, search and aggregation

You will be responsible for designing, building, and maintaining data pipelines that handle Real-world data at Compile. You will be handling both inbound and outbound data deliveries at Compile for datasets including Claims, Remittances, EHR, SDOH, etc.

You will

• Work on building and maintaining data pipelines (specifically RWD).
• Build, enhance and maintain existing pipelines in pyspark, python and help build analytical insights and datasets.
• Scheduling and maintaining pipeline jobs for RWD.
• Develop, test, and implement data solutions based on the design.
• Design and implement quality checks on existing and new data pipelines.
• Ensure adherence to security and compliance that is required for the products.
• Maintain relationships with various data vendors and track changes and issues across vendors and deliveries.

You have

• Hands-on experience with ETL process (min of 5 years).
• Excellent communication skills and ability to work with multiple vendors.
• High proficiency with Spark, SQL.
• Proficiency in Data modeling, validation, quality check, and data engineering concepts.
• Experience in working with big-data processing technologies using - databricks, dbt, S3, Delta lake, Deequ, Griffin, Snowflake, BigQuery.
• Familiarity with version control technologies, and CI/CD systems.
• Understanding of scheduling tools like Airflow/Prefect.
• Min of 3 years of experience managing data warehouses.
• Familiarity with healthcare datasets is a plus.

Compile embraces diversity and equal opportunity in a serious way. We are committed to building a team of people from many backgrounds, perspectives, and skills. We know the more inclusive we are, the better our work will be.         

1. Big data developer with 8+ years of professional IT experience with expertise in Hadoop ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems spanning Big Data.
2. A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
3. Expertise in Apache Hadoop ecosystem components like Spark, Hadoop Distributed File Systems(HDFS), HiveMapReduce, Hive, Sqoop, HBase, Zookeeper, YARN, Flume, Pig, Nifi, Scala and Oozie.
4. Hands on experience in creating real - time data streaming solutions using Apache Spark core, Spark SQL & DataFrames, Kafka, Spark streaming and Apache Storm.
5. Excellent knowledge of Hadoop architecture and daemons of Hadoop clusters, which include Name node,Data node, Resource manager, Node Manager and Job history server.
6. Worked on both Cloudera and Horton works in Hadoop Distributions. Experience in managing Hadoop clustersusing Cloudera Manager tool.
7. Well versed in installation, Configuration, Managing of Big Data and underlying infrastructure of Hadoop Cluster.
8. Hands on experience in coding MapReduce/Yarn Programs using Java, Scala and Python for analyzing Big Data.
9. Exposure to Cloudera development environment and management using Cloudera Manager.
10. Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle .
11. Implemented Spark using PYTHON and utilizing Data frames and Spark SQL API for faster processing of data and handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
12. Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
13. Hands on experience in MLlib from Spark which are used for predictive intelligence, customer segmentation and for smooth maintenance in Spark streaming.
14. Experience in using Flume to load log files into HDFS and Oozie for workflow design and scheduling.
15. Experience in optimizing MapReduce jobs to use HDFS efficiently by using various compression mechanisms.
16. Working on creating data pipeline for different events of ingestion, aggregation, and load consumer response data into Hive external tables in HDFS location to serve as feed for tableau dashboards.
17. Hands on experience in using Sqoop to import data into HDFS from RDBMS and vice-versa.
18. In-depth Understanding of Oozie to schedule all Hive/Sqoop/HBase jobs.
19. Hands on expertise in real time analytics with Apache Spark.
20. Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
21. Extensive experience in working with different ETL tool environments like SSIS, Informatica and reporting tool environments like SQL Server Reporting Services (SSRS).
22. Experience in Microsoft cloud and setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud.
23. Extensively worked on Spark using Python on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
24. Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.
25. Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
26. Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions.
27. Experience in creating Data frames using PySpark and performing operation on the Data frames using Python.
28. In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS and MapReduce Programming Paradigm, High Availability and YARN architecture.
29. Establishing multiple connections to different Redshift clusters (Bank Prod, Card Prod, SBBDA Cluster) and provide the access for pulling the information we need for analysis. 
30. Generated various kinds of knowledge reports using Power BI based on Business specification. 
31. Developed interactive Tableau dashboards to provide a clear understanding of industry specific KPIs using quick filters and parameters to handle them more efficiently.
32. Well Experience in projects using JIRA, Testing, Maven and Jenkins build tools.
33. Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3,), focusing on high-availability, fault tolerance, and auto-scaling.
34. Good experience with use-case development, with Software methodologies like Agile and Waterfall.
35. Working knowledge of Amazon's Elastic Cloud Compute( EC2 ) infrastructure for computational tasks and Simple Storage Service ( S3 ) as Storage mechanism.
36. Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark .
37. Extensive experience in Text Analytics, developing different Statistical Machine Learning solutions to various business problems and generating data visualizations using Python and R.
38. Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
39. Hands on experience in Hadoop Big data technology working on MapReduce, Pig, Hive as Analysis tool, Sqoop and Flume data import/export tools.

Job Description:

As an Azure Data Engineer, your role will involve designing, developing, and maintaining data solutions on the Azure platform. You will be responsible for building and optimizing data pipelines, ensuring data quality and reliability, and implementing data processing and transformation logic. Your expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), PySpark, and Scala will be essential for performing the following key responsibilities:

Designing and developing data pipelines: You will design and implement scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala. This includes data ingestion, data transformation, and data loading processes.

Data modeling and database design: You will design and implement data models to support efficient data storage, retrieval, and analysis. This may involve working with relational databases, data lakes, or other storage solutions on the Azure platform.

Data integration and orchestration: You will leverage Azure Data Factory (ADF) to orchestrate data integration workflows and manage data movement across various data sources and targets. This includes scheduling and monitoring data pipelines.

Data quality and governance: You will implement data quality checks, validation rules, and data governance processes to ensure data accuracy, consistency, and compliance with relevant regulations and standards.

Performance optimization: You will optimize data pipelines and queries to improve overall system performance and reduce processing time. This may involve tuning SQL queries, optimizing data transformation logic, and leveraging caching techniques.

Monitoring and troubleshooting: You will monitor data pipelines, identify performance bottlenecks, and troubleshoot issues related to data ingestion, processing, and transformation. You will work closely with cross-functional teams to resolve data-related problems.

Documentation and collaboration: You will document data pipelines, data flows, and data transformation processes. You will collaborate with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.

Skills and Qualifications:

Strong experience with Azure Databricks, Python, SQL, ADF, PySpark, and Scala.

Proficiency in designing and developing data pipelines and ETL processes.

Solid understanding of data modeling concepts and database design principles.

Familiarity with data integration and orchestration using Azure Data Factory.

Knowledge of data quality management and data governance practices.

Experience with performance tuning and optimization of data pipelines.

Strong problem-solving and troubleshooting skills related to data engineering.

Excellent collaboration and communication skills to work effectively in cross-functional teams.

Understanding of cloud computing principles and experience with Azure services.

Requirements

Experience

• 5+ years of professional experience in implementing MLOps framework to scale up ML in production.
• Hands-on experience with Kubernetes, Kubeflow, MLflow, Sagemaker, and other ML model experiment management tools including training, inference, and evaluation.
• Experience in ML model serving (TorchServe, TensorFlow Serving, NVIDIA Triton inference server, etc.)
• Proficiency with ML model training frameworks (PyTorch, Pytorch Lightning, Tensorflow, etc.).
• Experience with GPU computing to do data and model training parallelism.
• Solid software engineering skills in developing systems for production.
• Strong expertise in Python.
• Building end-to-end data systems as an ML Engineer, Platform Engineer, or equivalent.
• Experience working with cloud data processing technologies (S3, ECR, Lambda, AWS, Spark, Dask, ElasticSearch, Presto, SQL, etc.).
• Having Geospatial / Remote sensing experience is a plus.

·      Core responsibilities to include analyze business requirements and designs for accuracy and completeness. Develops and maintains relevant product.

·      BlueYonder is seeking a Senior/Principal Architect in the Data Services department (under Luminate Platform ) to act as one of key technology leaders to build and manage BlueYonder’ s technology assets in the Data Platform and Services.

·      This individual will act as a trusted technical advisor and strategic thought leader to the Data Services department. The successful candidate will have the opportunity to lead, participate, guide, and mentor other people in the team on architecture and design in a hands-on manner. You are responsible for technical direction of Data Platform. This position reports to the Global Head, Data Services and will be based in Bangalore, India.

·      Core responsibilities to include Architecting and designing (along with counterparts and distinguished Architects) a ground up cloud native (we use Azure) SaaS product in Order management and micro-fulfillment

·      The team currently comprises of 60+ global associates across US, India (COE) and UK and is expected to grow rapidly. The incumbent will need to have leadership qualities to also mentor junior and mid-level software associates in our team. This person will lead the Data platform architecture – Streaming, Bulk with Snowflake/Elastic Search/other tools

Our current technical environment:

·      Software: Java, Springboot, Gradle, GIT, Hibernate, Rest API, OAuth , Snowflake

·      • Application Architecture: Scalable, Resilient, event driven, secure multi-tenant Microservices architecture

·      • Cloud Architecture: MS Azure (ARM templates, AKS, HD insight, Application gateway, Virtue Networks, Event Hub, Azure AD)

·      Frameworks/Others: Kubernetes, Kafka, Elasticsearch, Spark, NOSQL, RDBMS, Springboot, Gradle GIT, Ignite

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

About Kloud9:

Kloud9 exists with the sole purpose of providing cloud expertise to the retail industry. Our team of cloud architects, engineers and developers help retailers launch a successful cloud initiative so you can quickly realise the benefits of cloud technology. Our standardised, proven cloud adoption methodologies reduce the cloud adoption time and effort so you can directly benefit from lower migration costs.

Kloud9 was founded with the vision of bridging the gap between E-commerce and cloud. The E-commerce of any industry is limiting and poses a huge challenge in terms of the finances spent on physical data structures.

At Kloud9, we know migrating to the cloud is the single most significant technology shift your company faces today. We are your trusted advisors in transformation and are determined to build a deep partnership along the way. Our cloud and retail experts will ease your transition to the cloud.

Our sole focus is to provide cloud expertise to retail industry giving our clients the empowerment that will take their business to the next level. Our team of proficient architects, engineers and developers have been designing, building and implementing solutions for retailers for an average of more than 20 years.

We are a cloud vendor that is both platform and technology independent. Our vendor independence not just provides us with a unique perspective into the cloud market but also ensures that we deliver the cloud solutions available that best meet our clients' requirements.

What we are looking for:

● 3+ years’ experience developing Data & Analytic solutions

● Experience building data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive& Spark

● Experience with relational SQL

● Experience with scripting languages such as Shell, Python

● Experience with source control tools such as GitHub and related dev process

● Experience with workflow scheduling tools such as Airflow

● In-depth knowledge of scalable cloud

● Has a passion for data solutions

● Strong understanding of data structures and algorithms

● Strong understanding of solution and technical design

● Has a strong problem-solving and analytical mindset

● Experience working with Agile Teams.

● Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

● Able to quickly pick up new programming languages, technologies, and frameworks

● Bachelor’s Degree in computer science

Why Explore a Career at Kloud9:

With job opportunities in prime locations of US, London, Poland and Bengaluru, we help build your career paths in cutting edge technologies of AI, Machine Learning and Data Science. Be part of an inclusive and diverse workforce that's changing the face of retail technology with their creativity and innovative solutions. Our vested interest in our employees translates to deliver the best products and solutions to our customers.

About Telstra

Telstra is Australia’s leading telecommunications and technology company, with operations in more than 20 countries, including In India where we’re building a new Innovation and Capability Centre (ICC) in Bangalore.

We’re growing, fast, and for you that means many exciting opportunities to develop your career at Telstra. Join us on this exciting journey, and together, we’ll reimagine the future.

Why Telstra?

• We're an iconic Australian company with a rich heritage that's been built over 100 years. Telstra is Australia's leading Telecommunications and Technology Company. We've been operating internationally for more than 70 years.
• International presence spanning over 20 countries.
• We are one of the 20 largest telecommunications providers globally
• At Telstra, the work is complex and stimulating, but with that comes a great sense of achievement. We are shaping the tomorrow's modes of communication with our innovation driven teams.

Telstra offers an opportunity to make a difference to lives of millions of people by providing the choice of flexibility in work and a rewarding career that you will be proud of!

About the team

Being part of Networks & IT means you'll be part of a team that focuses on extending our network superiority to enable the continued execution of our digital strategy.

With us, you'll be working with world-leading technology and change the way we do IT to ensure business needs drive priorities, accelerating our digitisation programme.

Focus of the role

Any new engineer who comes into data chapter would be mostly into developing reusable data processing and storage frameworks that can be used across data platform.

About you

To be successful in the role, you'll bring skills and experience in:-

Essential 

• Hands-on experience in Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing
• Skilled in the Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
• Azure data factory/Airflow/control-M/Luigi
• PL/SQL
• Exposure to NOSQL(Hbase/Cassandra/GraphDB(Neo4J)/MongoDB)
• File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
• Kafka/Kinesis/Eventhub

Highly Desirable

Experience and knowledgeable on the following:

• Spark Streaming
• Cloud exposure (Azure/AWS/GCP)
• Azure data offerings - ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc.
• Presto/Athena
• Azure DevOps
• Jenkins/ Bamboo/Any similar build tools
• Power BI
• Prior experience in building or working in team building reusable frameworks,
• Data modelling.
• Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
• Exposure to CI/CD
• Code Quality - Static and Dynamic code scans
• Agile SDLC      

If you've got a passion to innovate, succeed as part of a great team, and looking for the next step in your career, we'd welcome you to apply!

___________________________

We’re committed to building a diverse and inclusive workforce in all its forms. We encourage applicants from diverse gender, cultural and linguistic backgrounds and applicants who may be living with a disability. We also offer flexibility in all our roles, to ensure everyone can participate.

To learn more about how we support our people, including accessibility adjustments we can provide you through the recruitment process, visit tel.st/thrive.

About Kloud9:

Kloud9 exists with the sole purpose of providing cloud expertise to the retail industry. Our team of cloud architects, engineers and developers help retailers launch a successful cloud initiative so you can quickly realise the benefits of cloud technology. Our standardised, proven cloud adoption methodologies reduce the cloud adoption time and effort so you can directly benefit from lower migration costs.

Kloud9 was founded with the vision of bridging the gap between E-commerce and cloud. The E-commerce of any industry is limiting and poses a huge challenge in terms of the finances spent on physical data structures.

At Kloud9, we know migrating to the cloud is the single most significant technology shift your company faces today. We are your trusted advisors in transformation and are determined to build a deep partnership along the way. Our cloud and retail experts will ease your transition to the cloud.

Our sole focus is to provide cloud expertise to retail industry giving our clients the empowerment that will take their business to the next level. Our team of proficient architects, engineers and developers have been designing, building and implementing solutions for retailers for an average of more than 20 years.

We are a cloud vendor that is both platform and technology independent. Our vendor independence not just provides us with a unique perspective into the cloud market but also ensures that we deliver the cloud solutions available that best meet our clients' requirements.

What we are looking for:

●       3+ years’ experience developing Big Data & Analytic solutions

●       Experience building data lake solutions leveraging Google Data Products (e.g. Dataproc, AI Building Blocks, Looker, Cloud Data Fusion, Dataprep, etc.), Hive, Spark

●       Experience with relational SQL/No SQL

●       Experience with Spark (Scala/Python/Java) and Kafka

●       Work experience with using Databricks (Data Engineering and Delta Lake components)

●       Experience with source control tools such as GitHub and related dev process

●       Experience with workflow scheduling tools such as Airflow

●       In-depth knowledge of any scalable cloud vendor(GCP preferred)

●       Has a passion for data solutions

●       Strong understanding of data structures and algorithms

●       Strong understanding of solution and technical design

●       Has a strong problem solving and analytical mindset

●       Experience working with Agile Teams.

●       Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

●       Able to quickly pick up new programming languages, technologies, and frameworks

●       Bachelor’s Degree in computer science

Why Explore a Career at Kloud9:

With job opportunities in prime locations of US, London, Poland and Bengaluru, we help build your career paths in cutting edge technologies of AI, Machine Learning and Data Science. Be part of an inclusive and diverse workforce that's changing the face of retail technology with their creativity and innovative solutions. Our vested interest in our employees translates to deliver the best products and solutions to our customers!

• NLP 

• ML programming
• Spark
• Model Deployment
• Experience processing unstructured data and building NLP models
• Experience with big data tools pyspark
• Pipeline orchestration using Airflow and model deployment experience is preferred

Role: Data Engineer  
Company: PayU

Location: Bangalore/ Mumbai

Experience : 2-5 yrs

About Company:

PayU is the payments and fintech business of Prosus, a global consumer internet group and one of the largest technology investors in the world. Operating and investing globally in markets with long-term growth potential, Prosus builds leading consumer internet companies that empower people and enrich communities.

The leading online payment service provider in 36 countries, PayU is dedicated to creating a fast, simple and efficient payment process for merchants and buyers. Focused on empowering people through financial services and creating a world without financial borders where everyone can prosper, PayU is one of the biggest investors in the fintech space globally, with investments totalling $700 million- to date. PayU also specializes in credit products and services for emerging markets across the globe. We are dedicated to removing risks to merchants, allowing consumers to use credit in ways that suit them and enabling a greater number of global citizens to access credit services.

Our local operations in Asia, Central and Eastern Europe, Latin America, the Middle East, Africa and South East Asia enable us to combine the expertise of high growth companies with our own unique local knowledge and technology to ensure that our customers have access to the best financial services.

India is the biggest market for PayU globally and the company has already invested $400 million in this region in last 4 years. PayU in its next phase of growth is developing a full regional fintech ecosystem providing multiple digital financial services in one integrated experience. We are going to do this through 3 mechanisms: build, co-build/partner; select strategic investments. 

PayU supports over 350,000+ merchants and millions of consumers making payments online with over 250 payment methods and 1,800+ payment specialists. The markets in which PayU operates represent a potential consumer base of nearly 2.3 billion people and a huge growth potential for merchants. 

Job responsibilities:

• Design infrastructure for data, especially for but not limited to consumption in machine learning applications 
• Define database architecture needed to combine and link data, and ensure integrity across different sources 
• Ensure performance of data systems for machine learning to customer-facing web and mobile applications using cutting-edge open source frameworks, to highly available RESTful services, to back-end Java based systems 
• Work with large, fast, complex data sets to solve difficult, non-routine analysis problems, applying advanced data handling techniques if needed 
• Build data pipelines, includes implementing, testing, and maintaining infrastructural components related to the data engineering stack.
• Work closely with Data Engineers, ML Engineers and SREs to gather data engineering requirements to prototype, develop, validate and deploy data science and machine learning solutions

Requirements to be successful in this role: 

• Strong knowledge and experience in Python, Pandas, Data wrangling, ETL processes, statistics, data visualisation, Data Modelling and Informatica.
• Strong experience with scalable compute solutions such as in Kafka, Snowflake
• Strong experience with workflow management libraries and tools such as Airflow, AWS Step Functions etc. 
• Strong experience with data engineering practices (i.e. data ingestion pipelines and ETL) 
• A good understanding of machine learning methods, algorithms, pipelines, testing practices and frameworks 
• Preferred) MEng/MSc/PhD degree in computer science, engineering, mathematics, physics, or equivalent (preference: DS/ AI) 
• Experience with designing and implementing tools that support sharing of data, code, practices across organizations at scale 

Job Description

The applicant must have a minimum of 5 years of hands-on IT experience, working on a full software lifecycle in Agile mode.

Good to have experience in data modeling and/or systems architecture.
Responsibilities will include technical analysis, design, development and perform enhancements.

You will participate in all/most of the following activities:
- Working with business analysts and other project leads to understand requirements.
- Modeling and implementing database schemas in DB2 UDB or other relational databases.
- Designing, developing, maintaining and Data processing using Python, DB2, Greenplum, Autosys and other technologies

Skills /Expertise Required :

Work experience in developing large volume database (DB2/Greenplum/Oracle/Sybase).

Good experience in writing stored procedures, integration of database processing, tuning and optimizing database queries.

Strong knowledge of table partitions, high-performance loading and data processing.
Good to have hands-on experience working with Perl or Python.
Hands on development using Spark / KDB / Greenplum platform will be a strong plus.
Designing, developing, maintaining and supporting Data Extract, Transform and Load (ETL) software using Informatica, Shell Scripts, DB2 UDB and Autosys.
Coming up with system architecture/re-design proposals for greater efficiency and ease of maintenance and developing software to turn proposals into implementations.

Need to work with business analysts and other project leads to understand requirements.
Strong collaboration and communication skills

• Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
• Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
• Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
• Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
• Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
• Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
• Analyse large amounts of information to discover trends and patterns
• Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\

• Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
• 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
• Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
• Good understanding of AWS ecosystem for big data analytics
• Hands on experience in creating data pipelines either using tools or by independently writing scripts
• Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
• Strong problem solving skills with an emphasis on product development.
• Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)

We are looking for an exceptionally talented Lead data engineer who has exposure in implementing AWS services to build data pipelines, api integration and designing data warehouse. Candidate with both hands-on and leadership capabilities will be ideal for this position.

Qualification: At least a bachelor’s degree in Science, Engineering, Applied Mathematics. Preferred Masters degree

Job Responsibilities:

• Total 6+ years of experience as a Data Engineer and 2+ years of experience in managing a team

• Have minimum 3 years of AWS Cloud experience.

• Well versed in languages such as Python, PySpark, SQL, NodeJS etc

• Has extensive experience in the real-timeSpark ecosystem and has worked on both real time and batch processing

• Have experience in AWS Glue, EMR, DMS, Lambda, S3, DynamoDB, Step functions, Airflow, RDS, Aurora etc.

• Experience with modern Database systems such as Redshift, Presto, Hive etc.

• Worked on building data lakes in the past on S3 or Apache Hudi

• Solid understanding of Data Warehousing Concepts

• Good to have experience on tools such as Kafka or Kinesis

• Good to have AWS Developer Associate or Solutions Architect Associate Certification

• Have experience in managing a team

• Hands-on experience in any Cloud Platform

• Microsoft Azure Experience

4-5 years of overall experience in software development.
- Experience on Hadoop (Apache/Cloudera/Hortonworks) and/or other Map Reduce Platforms
- Experience on Hive, Pig, Sqoop, Flume and/or Mahout
- Experience on NO-SQL – HBase, Cassandra, MongoDB
- Hands on experience with Spark development,  Knowledge of Storm, Kafka, Scala
- Good knowledge of Java
- Good background of Configuration Management/Ticketing systems like Maven/Ant/JIRA etc.
- Knowledge around any Data Integration and/or EDW tools is plus
- Good to have knowledge of  using Python/Perl/Shell

Please note - Hbase hive and spark are must.

Responsibilities:

* 3+ years of Data Engineering Experience - Design, develop, deliver and maintain data infrastructures.

* SQL Specialist – Strong knowledge and Seasoned experience with SQL Queries

* Languages: Python

* Good communicator, shows initiative, works well with stakeholders.

* Experience working closely with Data Analysts and provide the data they need and guide them on the issues.

* Solid ETL experience and Hadoop/Hive/Pyspark/Presto/ SparkSQL

* Solid communication and articulation skills

* Able to handle stakeholders independently with less interventions of reporting manager.

* Develop strategies to solve problems in logical yet creative ways.

* Create custom reports and presentations accompanied by strong data visualization and storytelling

We would be excited if you have:

* Excellent communication and interpersonal skills

* Ability to meet deadlines and manage project delivery

* Excellent report-writing and presentation skills

* Critical thinking and problem-solving capabilities

• Involvement in the overall application lifecycle
• Design and develop software applications in Scala and Spark
• Understand business requirements and convert them to technical solutions
• Rest API design, implementation, and integration
• Collaborate with Frontend developers and provide mentorship for Junior engineers in the team
• An interest and preferably working experience in agile development methodologies
• A team player, eager to invest in personal and team growth
• Staying up to date with cutting edge technologies and best practices
• Advocate for improvements to product quality, security, and performance

Desired Skills and Experience

• Minimum 1.5+ years of development experience in Scala / Java language
• Strong understanding of the development cycle, programming techniques, and tools.
• Strong problem solving and verbal and written communication skills.
• Experience in working with web development using J2EE or similar frameworks
• Experience in developing REST API’s
• BE in Computer Science
• Experience with Akka or Micro services is a plus
• Experience with Big data technologies like Spark/Hadoop is a plus company offers very competitive compensation packages commensurate with your experience. We offer full benefits, continual career & compensation growth, and many other perks.

• We are looking for a Data Engineer to build the next-generation mobile applications for our world-class fintech product.
• The candidate will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection for cross-functional teams.
• The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimising data systems and building them from the ground up.
• Looking for a person with a strong ability to analyse and provide valuable insights to the product and business team to solve daily business problems.
• You should be able to work in a high-volume environment, have outstanding planning and organisational skills.

Qualifications for Data Engineer

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimising ‘big data’ data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Looking for a candidate with 2-3 years of experience in a Data Engineer role, who is a CS graduate or has an equivalent experience.

What we're looking for?

• Experience with big data tools: Hadoop, Spark, Kafka and other alternate tools.
• Experience with relational SQL and NoSQL databases, including MySql/Postgres and Mongodb.
• Experience with data pipeline and workflow management tools: Luigi, Airflow.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift.
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala.

Job Title: Engineering Manager

Job Location: Chennai, Bangalore
Job Summary
The Engineering Org is looking for a proficient Engineering Manager to join a team that is building exciting
and futuristic Data Products at Condé Nast to enable both internal and external marketers to target
audiences in real time. As an Engineering Manager, you will drive the day-to-day execution of technical
and architectural decisions. EM will own engineering deliverables inclusive of solving dependencies
such as architecture, solutions, sequencing, and working with other engineering delivery teams.This role
is also responsible for driving innovation, prototyping, and recommending solutions. Above all, you will
influence how users interact with Conde Nast’s industry-leading journalism.
● Primary Responsibilities
● Manage a high performing team of Software and Data Engineers within the Data & ML
Engineering team part of Engineering Data Organization.
● Provide leadership and guidance to the team in Data Discovery, Data Ingestion, Transformation
and Storage
● Utilizing product mindset to build, scale and deploy holistic data products after successful
prototyping and drive their engineering implementation
● Provide technical coaching and lead direct reports and other members of adjacent support teams
to the highest level of performance..
● Evaluate performance of direct reports and offer career development guidance.
● Meeting hiring and retention targets of the team & building a high-performance culture
● Handle escalations from internal stakeholders and manage critical issues to resolution.
● Collaborate with Architects, Product Manager, Project Manager and other teams to deliver high
quality products.
● Identify recurring system and application issues and enable engineers to work with release teams,
infra teams, product development, vendors and other stakeholders in investigating and resolving
the cause.
● Required Skills
● 4+ years of managing Software Development teams, preferably in ML and Data
Engineering teams.
● 4+ years of Agile Software development practices
● 12+ years of Software Development experience.
● Excellent Problem Solving and System Design skill
● Hands on: Writing and Reviewing code primarily in Spark, Python and/or Java
● Hand on: Architect & Design end to end Data Pipeline (noSQL databases, Job Schedulers, Big
Data Development preferably on Databricks / Cloud)
● Experience with SOA & Microservice architecture
● Knowledge of Software Engineering best practices with experience on implementing CI/CD,
Log aggregation/Monitoring/alerting for production system
● Working Knowledge of cloud and devops skills (AWS will be preferred)
● Strong verbal and written communication skills.
● Experience in evaluating team member performance and offering career development
guidance.
● Experience in providing technical coaching to direct reports.
● Experience in architecting highly scalable products.
● Experience in collaborating with global stakeholder teams.
● Experience in working on highly available production systems.
● Strong knowledge of software release process and release pipeline.
About Condé Nast
CONDÉ NAST INDIA (DATA)
Over the years, Condé Nast successfully expanded and diversified into digital, TV, and social
platforms - in other words, a staggering amount of user data. Condé Nast made the right move to
invest heavily in understanding this data and formed a whole new Data team entirely dedicated to
data processing, engineering, analytics, and visualization. This team helps drive engagement, fuel
process innovation, further content enrichment, and increase market revenue. The Data team
aimed to create a company culture where data was the common language and facilitate an
environment where insights shared in real-time could improve performance.
The Global Data team operates out of Los Angeles, New York, Chennai, and London. The team at
Condé Nast Chennai works extensively with data to amplify its brands' digital capabilities and boost
online revenue. We are broadly divided into four groups, Data Intelligence, Data Engineering, Data
Science, and Operations (including Product and Marketing Ops, Client Services) along with Data
Strategy and monetization. The teams built capabilities and products to create data-driven solutions
for better audience engagement.
What we look forward to:
We want to welcome bright, new minds into our midst and work together to create diverse forms of
self-expression. At Condé Nast, we encourage the imaginative and celebrate the extraordinary. We
are a media company for the future, with a remarkable past. We are Condé Nast, and It Starts Here.

• 3-6 years of relevant work experience in a Data Engineering role.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimizing data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• A good understanding of Airflow, Spark, NoSQL databases, Kafka is nice to have.
• Premium Institute Candidates only

• We are looking for : Data engineer
• Sprak
• Scala
• Hadoop

• Build campaign generation services which can send app notifications at a speed of 10 million a minute
• Dashboards to show Real time key performance indicators to clients
• Develop complex user segmentation engines which creates segments on Terabytes of data within few seconds
• Building highly available & horizontally scalable platform services for ever growing data
• Use cloud based services like AWS Lambda for blazing fast throughput & auto scalability
• Work on complex analytics on terabytes of data like building Cohorts, Funnels, User path analysis, Recency Frequency & Monetary analysis at blazing speed
• You will build backend services and APIs to create scalable engineering systems.
• As an individual contributor, you will tackle some of our broadest technical challenges that requires deep technical knowledge, hands-on software development and seamless collaboration with all functions.
• You will envision and develop features that are highly reliable and fault tolerant to deliver a superior customer experience.
• Collaborating various highly-functional teams in the company to meet deliverables throughout the software development lifecycle.
• Identify and improvise areas of improvement through data insights and research.

• 2-5 years of experience in backend development and must have worked on Java/shell/Perl/python scripting.
• Solid understanding of engineering best practices, continuous integration, and incremental delivery.
• Strong analytical skills, debugging and troubleshooting skills, product line analysis.
• Follower of agile methodology (Sprint planning, working on JIRA, retrospective etc).
• Proficiency in usage of tools like Docker, Maven, Jenkins and knowledge on frameworks in Java like spring, spring boot, hibernate, JPA.
• Ability to design application modules using various concepts like object oriented, multi-threading, synchronization, caching, fault tolerance, sockets, various IPCs, database interfaces etc.
• Hands on experience on Redis, MySQL and streaming technologies like Kafka producer consumers and NoSQL databases like mongo dB/Cassandra.
• Knowledge about versioning like Git and deployment processes like CICD.

Requirements

• 3+ years work experience with production-grade python. Contribution to open source repos is preferred
• Experience writing concurrent and distributed programs, AWS lambda, Kubernetes, Docker, Spark is preferred.
• Experience with one relational & 1 non-relational DB is preferred
• Prior work in the ML domain will be a big boost

What You’ll Do

• Help realize the product vision: Production-ready machine learning models with monitoring within moments, not months.
• Help companies deploy their machine learning models at scale across a wide range of use-cases and sectors.
• Build integrations with other platforms to make it easy for our customers to use our product without changing their workflow.
• Write maintainable, scalable performant python code
• Building gRPC, rest API servers
• Working with Thrift, Protobufs, etc.

Desired Competencies:

Ø  Expertise in Azure Data Factory V2

Ø  Expertise in other Azure components like Data lake Store, SQL Database, Databricks

Ø  Must have working knowledge of spark programming

Ø  Good exposure to Data Projects dealing with Data Design and Source to Target documentation including defining transformation rules

Ø  Strong knowledge of CICD Process

Ø  Experience in building power BI reports

Ø  Understanding of different components like Pipelines, activities, datasets & linked services

Ø  Exposure to dynamic configuration of pipelines using data sets and linked Services

Ø  Experience in designing, developing and deploying pipelines to higher environments

Ø  Good knowledge on File formats for flexible usage, File location Objects (SFTP, FTP, local, HDFS, ADLS, BLOB, Amazon S3 etc.)

Ø  Strong knowledge in SQL queries

Ø  Must have worked in full life-cycle development from functional design to deployment

Ø  Should have working knowledge of GIT, SVN

Ø  Good experience in establishing connection with heterogeneous sources like Hadoop, Hive, Amazon, Azure, Salesforce, SAP, HANA, API’s, various Databases etc.

Ø  Should have working knowledge of different resources available in Azure like Storage Account, Synapse, Azure SQL Server, Azure Data Bricks, Azure Purview

Ø  Any experience related to metadata management, data modelling, and related tools (Erwin or ER Studio or others) would be preferred

Preferred Qualifications:

Ø  Bachelor's degree in Computer Science or Technology

Ø  Proven success in contributing to a team-oriented environment

Ø  Proven ability to work creatively and analytically in a problem-solving environment

Ø  Excellent communication (written and oral) and interpersonal skills

Qualifications

BE/BTECH

KEY RESPONSIBILITIES :

Location – Bangalore