50+ Spark Jobs in Bangalore (Bengaluru) | Spark Job openings in Bangalore (Bengaluru)

About Us:

At Vahan, we are building India’s first AI powered recruitment marketplace for India’s 300 million strong Blue Collar workforce, opening doors to economic opportunities and brighter futures. Already India’s largest recruitment platform, Vahan is supported by marquee investors like Khosla Ventures, Bharti Airtel, Vijay Shekhar Sharma (CEO, Paytm), and leading executives from Google and Facebook. Our customers include names like Swiggy, Zomato, Rapido, Zepto, and many more. We leverage cutting-edge technology and AI to recruit for the workforces of some of the most recognized companies in the country.

Our vision is ambitious: to become the go-to platform for blue-collar professionals worldwide, empowering them with not just earning opportunities but also the tools, benefits, and support they need to thrive. We aim to impact over a billion lives worldwide, creating a future where everyone has access to economic prosperity. If our vision excites you, Vahan might just be your next adventure. We’re on the hunt for driven individuals who love tackling big challenges. If this sounds like your kind of journey, dive into the details and see where you can make your mark.

What you will be doing:

• Architect and Implement Data Infrastructure: Design, build, and maintain robust and scalable data pipelines and a data warehouse/lake solution using open-source and cloud-based technologies, optimized for both high-frequency small file and large file data ingestion, and real-time data streams. This includes implementing efficient mechanisms for handling high volumes of data arriving at frequent intervals.
• Develop and Optimize Data Processes: Create custom tools, primarily using Python, for data validation, processing, analysis, and automation. Continuously improve ETL/ELT processes for efficiency, reliability, and scalability. This includes building processes to bridge gaps between different databases and data sources, ensuring data consistency and accessibility. This also includes processing and integrating data from streaming sources.
• Lead and Mentor: Collaborate with product, engineering, and business teams to understand data requirements and provide data-driven solutions. Mentor and guide junior data engineers (as the team grows) and foster a culture of data excellence.
• Data Quality and Governance: Proactively identify and address data quality issues. Implement and maintain robust data quality monitoring, alerting, and measurement systems to ensure the accuracy, completeness, and consistency of our data assets. Implement and enforce data governance and security best practices, taking proactive ownership.
• Research: Research and adapt newer technologies to suit the requirements.

You will thrive in this role if you:

• Are a Hands-On Technical Leader: You possess deep technical expertise in data engineering and are comfortable leading by example, diving into code, and setting technical direction.
• Are a Startup-Minded Problem Solver: You thrive in a fast-paced, dynamic environment, are comfortable with ambiguity, and are eager to build from the ground up. You proactively identify and address challenges.
• Are a Collaborative Communicator: You can effectively communicate complex technical concepts to both technical and non-technical audiences and build strong relationships with stakeholders.
• Are a Strategic Thinker: You can think ahead and architect long lasting systems.

At Vahan, you’ll have the opportunity to make a real impact in a sector that touches millions of lives. We’re committed to not only advancing the livelihoods of our workforce but also in taking care of the people who make this mission possible. Here’s what we offer:

• Unlimited PTO: Trust and flexibility to manage your time in the way that works best for you.
• Comprehensive Medical Insurance: We’ve got you covered with plans designed to support you and your loved ones.
• Monthly Wellness Leaves: Regular time off to recharge and focus on what matters most.
• Competitive Pay: Your contributions are recognized and rewarded with a compensation package that reflects your impact.

Join us, and be part of something bigger—where your work drives real, positive change in the world.

🛠️ Key Responsibilities

• Design, build, and maintain scalable data pipelines using Python and Apache Spark (PySpark or Scala APIs)
• Develop and optimize ETL processes for batch and real-time data ingestion
• Collaborate with data scientists, analysts, and DevOps teams to support data-driven solutions
• Ensure data quality, integrity, and governance across all stages of the data lifecycle
• Implement data validation, monitoring, and alerting mechanisms for production pipelines
• Work with cloud platforms (AWS, GCP, or Azure) and tools like Airflow, Kafka, and Delta Lake
• Participate in code reviews, performance tuning, and documentation

🎓 Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• 3–6 years of experience in data engineering with a focus on Python and Spark
• Experience with distributed computing and handling large-scale datasets (10TB+)
• Familiarity with data security, PII handling, and compliance standards is a plus

Senior Data Engineer Job Description

Overview

The Senior Data Engineer will design, develop, and maintain scalable data pipelines and

infrastructure to support data-driven decision-making and advanced analytics. This role requires deep

expertise in data engineering, strong problem-solving skills, and the ability to collaborate with

cross-functional teams to deliver robust data solutions.

Key Responsibilities

Data Pipeline Development: Design, build, and optimize scalable, secure, and reliable data

pipelines to ingest, process, and transform large volumes of structured and unstructured data.

Data Architecture: Architect and maintain data storage solutions, including data lakes, data

warehouses, and databases, ensuring performance, scalability, and cost-efficiency.

Data Integration: Integrate data from diverse sources, including APIs, third-party systems,

and streaming platforms, ensuring data quality and consistency.

Performance Optimization: Monitor and optimize data systems for performance, scalability,

and cost, implementing best practices for partitioning, indexing, and caching.

Collaboration: Work closely with data scientists, analysts, and software engineers to

understand data needs and deliver solutions that enable advanced analytics, machine

learning, and reporting.

Data Governance: Implement data governance policies, ensuring compliance with data

security, privacy regulations (e.g., GDPR, CCPA), and internal standards.

Automation: Develop automated processes for data ingestion, transformation, and validation

to improve efficiency and reduce manual intervention.

Mentorship: Guide and mentor junior data engineers, fostering a culture of technical

excellence and continuous learning.

Troubleshooting: Diagnose and resolve complex data-related issues, ensuring high

availability and reliability of data systems.

Required Qualifications

Education: Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science,

or a related field.

Experience: 5+ years of experience in data engineering or a related role, with a proven track

record of building scalable data pipelines and infrastructure.

Technical Skills:

Proficiency in programming languages such as Python, Java, or Scala.

Expertise in SQL and experience with NoSQL databases (e.g., MongoDB, Cassandra).

Strong experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services

(e.g., Redshift, BigQuery, Snowflake).

Hands-on experience with ETL/ELT tools (e.g., Apache Airflow, Talend, Informatica) and

data integration frameworks.

Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka) and distributed

systems.

Knowledge of containerization and orchestration tools (e.g., Docker, Kubernetes) is a

plus.

Soft Skills:

Excellent problem-solving and analytical skills.

Strong communication and collaboration abilities.

Ability to work in a fast-paced, dynamic environment and manage multiple priorities.

Certifications (optional but preferred): Cloud certifications (e.g., AWS Certified Data Analytics,

Google Professional Data Engineer) or relevant data engineering certifications.

Preferred Qualifica

Experience with real-time data processing and streaming architectures.

Familiarity with machine learning pipelines and MLOps practices.

Knowledge of data visualization tools (e.g., Tableau, Power BI) and their integration with data

pipelines.

Experience in industries with high data complexity, such as finance, healthcare, or

e-commerce.

Work Environment

Location: Hybrid/Remote/On-site (depending on company policy).

Team: Collaborative, cross-functional team environment with data scientists, analysts, and

business stakeholders.

Hours: Full-time, with occasional on-call responsibilities for critical data systems.

Role**

Ab Initio Developer

Required Technical Skill Set**

               Ab Initio, GDE, EME Conduct IT and Hadoop, Spark

Desired Competencies (Technical/Behavioral Competency)

Must-Have**

(Ideally should not be more than 3-5)

• Should have very good knowledge and implementation experience about ETL/ELT methodology, Data Quality Strategy and Data Principles
• Experience in Project Code Management Tools(Repository/Maven/Version Control/Servers)
• Good knowledge Database Concepts (SQL)
• Good Exposure of Project management tools (JIRA, Confluence)
• Working experience in Agile methodology (Scrum/Kanban)

Will need to manage Cassandra/Solr/Spark/Graph clusters.

2. Strong experience in UNIX required, including command line editors, scripting (shell, Python, Perl, etc

3. Expert Understanding of Unix/Linux Operating systems.

4. Monitor Cassandra clusters using OpsCenter,

5. Experience in upgrades of Cassandra/Solr/Spark required

6. Manage nodes of Cassandra/Solr/Spark cluster

7. Cassandra cluster connectivity and security

8. Must-have supported Cassandra cluster in production

9. Using monitoring and management tools (e.g. OpsCenter)

10. Troubleshoot Cassandra/Solr/Spark Cluster

11. Database backup and recovery

12. Disk space management, Automate manual tasks

13. Certification as Datastax Administrator or Developer will be preferred.

14. Additional responsibility to monitor and maintain Tomcat cluster.

15. Should have experience in performance and capacity planning.

16. Should have experience in Datastax Cassandra & Apache Cassandra.

17. Provide 24x7 support for critical production systems.

18. Decommissioning and commissioning the Node on running cluster/existing cluster.

19. Understanding of Cassandra cluster management by using datastax Opscenter, Devcenter and

nodetool utility commands & Other Utilities.

20. Clear understanding of Relational DB / NoSQL Concepts and Architecture.

21. Ability to multi-task and context-switch effectively between different activities and teams

We are looking for a talented MERN Developer with expertise in MongoDB/MySQL, Kubernetes, Python, ETL, Hadoop, and Spark. The ideal candidate will design, develop, and optimize scalable applications while ensuring efficient source code management and implementing Non-Functional Requirements (NFRs).

Key Responsibilities:

• Develop and maintain robust applications using MERN Stack (MongoDB, Express.js, React.js, Node.js).
• Design efficient database architectures (MongoDB/MySQL) for scalable data handling.
• Implement and manage Kubernetes-based deployment strategies for containerized applications.
• Ensure compliance with Non-Functional Requirements (NFRs), including source code management, development tools, and security best practices.
• Develop and integrate Python-based functionalities for data processing and automation.
• Work with ETL pipelines for smooth data transformations.
• Leverage Hadoop and Spark for processing and optimizing large-scale data operations.
• Collaborate with solution architects, DevOps teams, and data engineers to enhance system performance.
• Conduct code reviews, troubleshooting, and performance optimization to ensure seamless application functionality.

Required Skills & Qualifications:

• Proficiency in MERN Stack (MongoDB, Express.js, React.js, Node.js).
• Strong understanding of database technologies (MongoDB/MySQL).
• Experience working with Kubernetes for container orchestration.
• Hands-on knowledge of Non-Functional Requirements (NFRs) in application development.
• Expertise in Python, ETL pipelines, and big data technologies (Hadoop, Spark).
• Strong problem-solving and debugging skills.
• Knowledge of microservices architecture and cloud computing frameworks.

Preferred Qualifications:

• Certifications in cloud computing, Kubernetes, or database management.
• Experience in DevOps, CI/CD automation, and infrastructure management.
• Understanding of security best practices in application development.

Good work experience on Big Data Platforms like Hadoop, Spark, Scala, Hive, Impala, SQL

Experience working on Data Engineering projects 

Good Understanding of SQL

Good understanding of Unix and HDFS commands

Communication

Experience working on Data Analytics and Pyspark

Exposure to Tableau & SAS

We are looking for a Senior Data Engineer with strong expertise in GCP, Databricks, and Airflow to design and implement a GCP Cloud Native Data Processing Framework. The ideal candidate will work on building scalable data pipelines and help migrate existing workloads to a modern framework.

• Shift: 2 PM 11 PM
• Work Mode: Hybrid (3 days a week) across Xebia locations
• Notice Period: Immediate joiners or those with a notice period of up to 30 days

Key Responsibilities:

• Design and implement a GCP Native Data Processing Framework leveraging Spark and GCP Cloud Services.
• Develop and maintain data pipelines using Databricks and Airflow for transforming Raw → Silver → Gold data layers.
• Ensure data integrity, consistency, and availability across all systems.
• Collaborate with data engineers, analysts, and stakeholders to optimize performance.
• Document standards and best practices for data engineering workflows.

Required Experience:

• 7-8 years of experience in data engineering, architecture, and pipeline development.
• Strong knowledge of GCP, Databricks, PySpark, and BigQuery.
• Experience with Orchestration tools like Airflow, Dagster, or GCP equivalents.
• Understanding of Data Lake table formats (Delta, Iceberg, etc.).
• Proficiency in Python for scripting and automation.
• Strong problem-solving skills and collaborative mindset.

⚠️ Please apply only if you have not applied recently or are not currently in the interview process for any open roles at Xebia.

Looking forward to your response!

Best regards,

Vijay S

Assistant Manager - TAG

https://www.linkedin.com/in/vijay-selvarajan/

Level of skills and experience:

5 years of hands-on experience in using Python, Spark,Sql.

Experienced in AWS Cloud usage and management.

Experience with Databricks (Lakehouse, ML, Unity Catalog, MLflow).

Experience using various ML models and frameworks such as XGBoost, Lightgbm, Torch.

Experience with orchestrators such as Airflow and Kubeflow.

Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes).

Fundamental understanding of Parquet, Delta Lake and other data file formats.

Proficiency on an IaC tool such as Terraform, CDK or CloudFormation.

Strong written and verbal English communication skill and proficient in communication with non-technical stakeholderst

We are seeking a highly skilled and experienced Offshore Data Engineer . The role involves designing, implementing, and testing data pipelines and products.

Qualifications & Experience:

bachelor's or master's degree in computer science, Information Systems, or a related field.

5+ years of experience in data engineering, with expertise in data architecture and pipeline development.

☁️ Proven experience with GCP, Big Query, Databricks, Airflow, Spark, DBT, and GCP Services.

️ Hands-on experience with ETL processes, SQL, PostgreSQL, MySQL, MongoDB, Cassandra.

Strong proficiency in Python and data modelling.

Experience in testing and validation of data pipelines.

Preferred: Experience with eCommerce systems, data visualization tools (Tableau, Looker), and cloud certifications.

If you meet the above criteria and are interested, please share your updated CV along with the following details:

Total Experience:

Current CTC:

Expected CTC:

Current Location:

Preferred Location:

Notice Period / Last Working Day (if serving notice):

⚠️ Kindly share your details only if you have not applied recently or are not currently in the interview process for any open roles at Xebia.

Looking forward to your response!

Position : Software Engineer (Java Backend Engineer)

Experience : 4+ Years

📍 Location : Bangalore, India (Hybrid)

Mandatory Skills : Java 8+ (Advanced Features), Spring Boot, Apache Spark (Spark Streaming), SQL & Cosmos DB, Git, Maven, CI/CD (Jenkins, GitHub), Azure Cloud, Agile Scrum.

About the Role :

We are seeking a highly skilled Backend Engineer with expertise in Java, Spark, and microservices architecture to join our dynamic team. The ideal candidate will have a strong background in object-oriented programming, experience with Spark Streaming, and a deep understanding of distributed systems and cloud technologies.

Key Responsibilities :

• Design, develop, and maintain highly scalable microservices and optimized RESTful APIs using Spring Boot and Java 8+.
• Implement and optimize Spark Streaming applications for real-time data processing.
• Utilize advanced Java 8 features, including:
• Functional interfaces & Lambda expressions
• Streams and Parallel Streams
• Completable Futures & Concurrency API improvements
• Enhanced Collections APIs
• Work with relational (SQL) and NoSQL (Cosmos DB) databases, ensuring efficient data modeling and retrieval.
• Develop and manage CI/CD pipelines using Jenkins, GitHub, and related automation tools.
• Collaborate with cross-functional teams, including Product, Business, and Automation, to deliver end-to-end product features.
• Ensure adherence to Agile Scrum practices and participate in code reviews to maintain high-quality standards.
• Deploy and manage applications in Azure Cloud environments.

Minimum Qualifications:

• BS/MS in Computer Science or a related field.
• 4+ Years of experience developing backend applications with Spring Boot and Java 8+.
• 3+ Years of hands-on experience with Git for version control.
• Strong understanding of software design patterns and distributed computing principles.
• Experience with Maven for building and deploying artifacts.
• Proven ability to work in Agile Scrum environments with a collaborative team mindset.
• Prior experience with Azure Cloud Technologies.

Dear Candidate,

We are Urgently hiring QA Automation Engineers and Test leads At Hyderabad and Bangalore

Exp: 6-10 yrs

Locations: Hyderabad ,Bangalore

JD:

we are Hiring Automation Testers with 6-10 years of Automation testing experience using QA automation tools like Java, UFT, Selenium, API Testing, ETL & others

Must Haves:

·        Experience in Financial Domain is a must

·        Extensive Hands-on experience in Design, implement and maintain automation framework using Java, UFT, ETL, Selenium tools and automation concepts.

·        Experience with AWS concept and framework design/ testing.

·        Experience in Data Analysis, Data Validation, Data Cleansing, Data Verification and identifying data mismatch.

·        Experience with Databricks, Python, Spark, Hive, Airflow, etc.

·        Experience in validating and analyzing kubernetics log files.

·        API testing experience

·        Backend testing skills with ability to write SQL queries in Databricks and in Oracle databases

·        Experience in working with globally distributed Agile project teams

·        Ability to work in a fast-paced, globally structured and team-based environment, as well as independently

·        Experience in test management tools like Jira

·        Good written and verbal communication skills

Good To have:

• Business and finance knowledge desirable

Best Regards,

Minakshi Soni

Executive - Talent Acquisition (L2)

Worldwide Locations: USA | HK | IN 

Responsibilities:

·       Analyze complex data sets to answer specific questions using MMIT’s market access data (MMIT) and Norstella claims data, third-party claims data (IQVIA LAAD, Symphony SHA). Applicant must have experience working with the aforementioned data sets exclusively.

·       Deliver consultative services to clients related to MMIT RWD sets

·       Produce complex analytical reports using data visualization tools such as Power BI or Tableau

·       Define customized technical specifications to surface MMIT RWD in MMIT tools. 

·       Execute work in a timely fashion with high accuracy, while managing various competing priorities; Perform thorough troubleshooting and execute QA; Communicate with internal teams to obtain required data

·       Ensure adherence to documentation requirements, process workflows, timelines, and escalation protocols

·       And other duties as assigned.

Requirements:

·       Bachelor’s Degree or relevant experience required

·       2-5 yrs. of professional experience in RWD analytics using SQL

·       Fundamental understanding of Pharma and Market access space

·       Strong analysis skills and proficiency with tools such as Tableau or PowerBI

·       Excellent written and verbal communication skills.

·       Analytical, critical thinking and creative problem-solving skills.

·       Relationship building skills.

·       Solid organizational skills including attention to detail and multitasking skills.

·       Excellent time management and prioritization skills.

software development and automated testing Proficient in Big Data technologies Designs, codes, tests, corrects and documents large and/or complex programs and program modifications from supplied specifications using agreed standards and tools, to achieve a well engineered result Proficient and Hands-on Data Warehousing,

Experience with Agile development, Continuous Integration, and Continuous Delivery Ability to effectively interpret technical and business objectives and provide solutions Strong communication skills, with ability to articulate technical solutions effectively across diverse group of stakeholders

Need to be a fast learner willing to adapt to evolving needs of the developer community.

Thanks & Regards

snehalata verma

IT Recruiter --HrBizHub

• Responsible for designing, storing, processing, and maintaining of large-scale data and related infrastructure.
• Can drive multiple projects both from operational and technical standpoint.
• Ideate and build PoV or PoC for new product that can help drive more business.
• Responsible for defining, designing, and implementing data engineering best practices, strategies, and solutions.
• Is an Architect who can guide the customers, team, and overall organization on tools, technologies, and best practices around data engineering.
• Lead architecture discussions, align with business needs, security, and best practices.
• Has strong conceptual understanding of Data Warehousing and ETL, Data Governance and Security, Cloud Computing, and Batch & Real Time data processing
• Has strong execution knowledge of Data Modeling, Databases in general (SQL and NoSQL), software development lifecycle and practices, unit testing, functional programming, etc.
• Understanding of Medallion architecture pattern
• Has worked on at least one cloud platform.
• Has worked as data architect and executed multiple end-end data engineering project.
• Has extensive knowledge of different data architecture designs and data modelling concepts.
• Manages conversation with the client stakeholders to understand the requirement and translate it into technical outcomes.

Required Tech Stack

• Strong proficiency in SQL
• Experience working on any of the three major cloud platforms i.e., AWS/Azure/GCP
• Working knowledge of an ETL and/or orchestration tools like IICS, Talend, Matillion, Airflow, Azure Data Factory, AWS Glue, GCP Composer, etc.
• Working knowledge of one or more OLTP databases (Postgres, MySQL, SQL Server, etc.)
• Working knowledge of one or more Data Warehouse like Snowflake, Redshift, Azure Synapse, Hive, Big Query, etc.
• Proficient in at least one programming language used in data engineering, such as Python (or Scala/Rust/Java)
• Has strong execution knowledge of Data Modeling (star schema, snowflake schema, fact vs dimension tables)
• Proficient in Spark and related applications like Databricks, GCP DataProc, AWS Glue, EMR, etc.
• Has worked on Kafka and real-time streaming.
• Has strong execution knowledge of data architecture design patterns (lambda vs kappa architecture, data harmonization, customer data platforms, etc.)
• Has worked on code and SQL query optimization.
• Strong knowledge of version control systems like Git to manage source code repositories and designing CI/CD pipelines for continuous delivery.
• Has worked on data and networking security (RBAC, secret management, key vaults, vnets, subnets, certificates)

Nielsen, a global company specialising in audience measurement and analytics, is currently seeking a proficient leader in data engineering to join their team in Bangalore, Gurgaon, or Mumbai.

This is a manager of managers role that involves managing multiple scrum teams and overseeing an advanced data platform that analyses audience consumption patterns across various channels like OTT, TV, Radio, and Social Media worldwide. You will be responsible for building and supervising a top-performing data engineering team that delivers data for targeted campaigns. Moreover, you will work with AWS services (S3, Lambda, Kinesis) and other data engineering technologies such as Spark, Scala/Python, Kafka, etc. There may also be opportunities to establish deep integrations with OTT platforms like Netflix, Prime Video, and other.

Must have skills

3 to 6 years

Data Science

SQL, Excel, Big Query - mandate 3+ years

Python/ML, Hadoop, Spark - 2+ years

Requirements

• 3+ years prior experience as a data analyst

• Detail oriented, structural thinking and analytical mindset.

• Proven analytic skills, including data analysis and data validation.

• Technical writing experience in relevant areas, including queries, reports, and presentations.

• Strong SQL and Excel skills with the ability to learn other analytic tools

• Good communication skills (being precise and clear)

• Good to have prior knowledge of python and ML algorithms

Publicis Sapient Overview:

The Senior Associate People Senior Associate L1 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution 

.

Job Summary:

As Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution

The role requires a hands-on technologist who has strong programming background like Java / Scala / Python, should have experience in Data Ingestion, Integration and data Wrangling, Computation, Analytics pipelines and exposure to Hadoop ecosystem components. You are also required to have hands-on knowledge on at least one of AWS, GCP, Azure cloud platforms.

Role & Responsibilities:

Your role is focused on Design, Development and delivery of solutions involving:

• Data Integration, Processing & Governance

• Data Storage and Computation Frameworks, Performance Optimizations

• Analytics & Visualizations

• Infrastructure & Cloud Computing

• Data Management Platforms

• Implement scalable architectural models for data processing and storage

• Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time mode

• Build functionality for data analytics, search and aggregation

Experience Guidelines:

Mandatory Experience and Competencies:

# Competency

1.Overall 5+ years of IT experience with 3+ years in Data related technologies

2.Minimum 2.5 years of experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure / GCP)

3.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline.

4.Strong experience in at least of the programming language Java, Scala, Python. Java preferable

5.Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc

6.Well-versed and working knowledge with data platform related services on at least 1 cloud platform, IAM and data security

Preferred Experience and Knowledge (Good to Have):

# Competency

1.Good knowledge of traditional ETL tools (Informatica, Talend, etc) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands on experience

2.Knowledge on data governance processes (security, lineage, catalog) and tools like Collibra, Alation etc

3.Knowledge on distributed messaging frameworks like ActiveMQ / RabbiMQ / Solace, search & indexing and Micro services architectures

4.Performance tuning and optimization of data pipelines

5.CI/CD – Infra provisioning on cloud, auto build & deployment pipelines, code quality

6.Cloud data specialty and other related Big data technology certifications

Personal Attributes:

• Strong written and verbal communication skills

• Articulation skills

• Good team player

• Self-starter who requires minimal oversight

• Ability to prioritize and manage multiple tasks

• Process orientation and the ability to define and set up processes

Publicis Sapient Overview:

The Senior Associate People Senior Associate L1 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution 

.

Job Summary:

As Senior Associate L1 in Data Engineering, you will do technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution

The role requires a hands-on technologist who has strong programming background like Java / Scala / Python, should have experience in Data Ingestion, Integration and data Wrangling, Computation, Analytics pipelines and exposure to Hadoop ecosystem components. Having hands-on knowledge on at least one of AWS, GCP, Azure cloud platforms will be preferable.

Role & Responsibilities:

Job Title: Senior Associate L1 – Data Engineering

Your role is focused on Design, Development and delivery of solutions involving:

• Data Ingestion, Integration and Transformation

• Data Storage and Computation Frameworks, Performance Optimizations

• Analytics & Visualizations

• Infrastructure & Cloud Computing

• Data Management Platforms

• Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time

• Build functionality for data analytics, search and aggregation

Experience Guidelines:

Mandatory Experience and Competencies:

# Competency

1.Overall 3.5+ years of IT experience with 1.5+ years in Data related technologies

2.Minimum 1.5 years of experience in Big Data technologies

3.Hands-on experience with the Hadoop stack – HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline. Working knowledge on real-time data pipelines is added advantage.

4.Strong experience in at least of the programming language Java, Scala, Python. Java preferable

5.Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc

Preferred Experience and Knowledge (Good to Have):

# Competency

1.Good knowledge of traditional ETL tools (Informatica, Talend, etc) and database technologies (Oracle, MySQL, SQL Server, Postgres) with hands on experience

2.Knowledge on data governance processes (security, lineage, catalog) and tools like Collibra, Alation etc

3.Knowledge on distributed messaging frameworks like ActiveMQ / RabbiMQ / Solace, search & indexing and Micro services architectures

4.Performance tuning and optimization of data pipelines

5.CI/CD – Infra provisioning on cloud, auto build & deployment pipelines, code quality

6.Working knowledge with data platform related services on at least 1 cloud platform, IAM and data security

7.Cloud data specialty and other related Big data technology certifications

Job Title: Senior Associate L1 – Data Engineering

Personal Attributes:

• Strong written and verbal communication skills

• Articulation skills

• Good team player

• Self-starter who requires minimal oversight

• Ability to prioritize and manage multiple tasks

• Process orientation and the ability to define and set up processes

Job Title Big Data Developer

Job Description

Bachelor's degree in Engineering or Computer Science or equivalent OR Master's in Computer Applications or equivalent.

Solid Experience of software development experience and leading teams of engineers and scrum teams.

4+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and PySpark).

Solid Datawarehousing concepts.

Knowledge of Financial reporting ecosystem will be a plus.

4+ years of experience within Data Engineering/ Data Warehousing using Big Data technologies will be an addon.

Expert on Distributed ecosystem.

Hands-on experience with programming using Core Java or Python/Scala

Expert on Hadoop and Spark Architecture and its working principle

Hands-on experience on writing and understanding complex SQL(Hive/PySpark-dataframes), optimizing joins while processing huge amount of data.

Experience in UNIX shell scripting.

Roles & Responsibilities

Ability to design and develop optimized Data pipelines for batch and real time data processing

Should have experience in analysis, design, development, testing, and implementation of system applications

Demonstrated ability to develop and document technical and functional specifications and analyze software and system processing flows.

Excellent technical and analytical aptitude

Good communication skills.

Excellent Project management skills.

Results driven Approach.

Mandatory SkillsBig Data, PySpark, Hive

Data Engineering : Senior Engineer / Manager

As Senior Engineer/ Manager in Data Engineering, you will translate client requirements into technical design, and implement components for a data engineering solutions. Utilize a deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution.

Must Have skills :

1. GCP

2. Spark streaming : Live data streaming experience is desired.

3. Any 1 coding language: Java/Pyhton /Scala

Skills & Experience :

- Overall experience of MINIMUM 5+ years with Minimum 4 years of relevant experience in Big Data technologies

- Hands-on experience with the Hadoop stack - HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline. Working knowledge on real-time data pipelines is added advantage.

- Strong experience in at least of the programming language Java, Scala, Python. Java preferable

- Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc.

- Well-versed and working knowledge with data platform related services on GCP

- Bachelor's degree and year of work experience of 6 to 12 years or any combination of education, training and/or experience that demonstrates the ability to perform the duties of the position

Your Impact :

- Data Ingestion, Integration and Transformation

- Data Storage and Computation Frameworks, Performance Optimizations

- Analytics & Visualizations

- Infrastructure & Cloud Computing

- Data Management Platforms

- Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time

- Build functionality for data analytics, search and aggregation

You will be responsible for designing, building, and maintaining data pipelines that handle Real-world data at Compile. You will be handling both inbound and outbound data deliveries at Compile for datasets including Claims, Remittances, EHR, SDOH, etc.

You will

• Work on building and maintaining data pipelines (specifically RWD).
• Build, enhance and maintain existing pipelines in pyspark, python and help build analytical insights and datasets.
• Scheduling and maintaining pipeline jobs for RWD.
• Develop, test, and implement data solutions based on the design.
• Design and implement quality checks on existing and new data pipelines.
• Ensure adherence to security and compliance that is required for the products.
• Maintain relationships with various data vendors and track changes and issues across vendors and deliveries.

You have

• Hands-on experience with ETL process (min of 5 years).
• Excellent communication skills and ability to work with multiple vendors.
• High proficiency with Spark, SQL.
• Proficiency in Data modeling, validation, quality check, and data engineering concepts.
• Experience in working with big-data processing technologies using - databricks, dbt, S3, Delta lake, Deequ, Griffin, Snowflake, BigQuery.
• Familiarity with version control technologies, and CI/CD systems.
• Understanding of scheduling tools like Airflow/Prefect.
• Min of 3 years of experience managing data warehouses.
• Familiarity with healthcare datasets is a plus.

Compile embraces diversity and equal opportunity in a serious way. We are committed to building a team of people from many backgrounds, perspectives, and skills. We know the more inclusive we are, the better our work will be.         

1. Big data developer with 8+ years of professional IT experience with expertise in Hadoop ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems spanning Big Data.
2. A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
3. Expertise in Apache Hadoop ecosystem components like Spark, Hadoop Distributed File Systems(HDFS), HiveMapReduce, Hive, Sqoop, HBase, Zookeeper, YARN, Flume, Pig, Nifi, Scala and Oozie.
4. Hands on experience in creating real - time data streaming solutions using Apache Spark core, Spark SQL & DataFrames, Kafka, Spark streaming and Apache Storm.
5. Excellent knowledge of Hadoop architecture and daemons of Hadoop clusters, which include Name node,Data node, Resource manager, Node Manager and Job history server.
6. Worked on both Cloudera and Horton works in Hadoop Distributions. Experience in managing Hadoop clustersusing Cloudera Manager tool.
7. Well versed in installation, Configuration, Managing of Big Data and underlying infrastructure of Hadoop Cluster.
8. Hands on experience in coding MapReduce/Yarn Programs using Java, Scala and Python for analyzing Big Data.
9. Exposure to Cloudera development environment and management using Cloudera Manager.
10. Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle .
11. Implemented Spark using PYTHON and utilizing Data frames and Spark SQL API for faster processing of data and handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
12. Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
13. Hands on experience in MLlib from Spark which are used for predictive intelligence, customer segmentation and for smooth maintenance in Spark streaming.
14. Experience in using Flume to load log files into HDFS and Oozie for workflow design and scheduling.
15. Experience in optimizing MapReduce jobs to use HDFS efficiently by using various compression mechanisms.
16. Working on creating data pipeline for different events of ingestion, aggregation, and load consumer response data into Hive external tables in HDFS location to serve as feed for tableau dashboards.
17. Hands on experience in using Sqoop to import data into HDFS from RDBMS and vice-versa.
18. In-depth Understanding of Oozie to schedule all Hive/Sqoop/HBase jobs.
19. Hands on expertise in real time analytics with Apache Spark.
20. Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
21. Extensive experience in working with different ETL tool environments like SSIS, Informatica and reporting tool environments like SQL Server Reporting Services (SSRS).
22. Experience in Microsoft cloud and setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud.
23. Extensively worked on Spark using Python on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
24. Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.
25. Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
26. Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions.
27. Experience in creating Data frames using PySpark and performing operation on the Data frames using Python.
28. In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS and MapReduce Programming Paradigm, High Availability and YARN architecture.
29. Establishing multiple connections to different Redshift clusters (Bank Prod, Card Prod, SBBDA Cluster) and provide the access for pulling the information we need for analysis. 
30. Generated various kinds of knowledge reports using Power BI based on Business specification. 
31. Developed interactive Tableau dashboards to provide a clear understanding of industry specific KPIs using quick filters and parameters to handle them more efficiently.
32. Well Experience in projects using JIRA, Testing, Maven and Jenkins build tools.
33. Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3,), focusing on high-availability, fault tolerance, and auto-scaling.
34. Good experience with use-case development, with Software methodologies like Agile and Waterfall.
35. Working knowledge of Amazon's Elastic Cloud Compute( EC2 ) infrastructure for computational tasks and Simple Storage Service ( S3 ) as Storage mechanism.
36. Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark .
37. Extensive experience in Text Analytics, developing different Statistical Machine Learning solutions to various business problems and generating data visualizations using Python and R.
38. Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
39. Hands on experience in Hadoop Big data technology working on MapReduce, Pig, Hive as Analysis tool, Sqoop and Flume data import/export tools.

Job Description:

As an Azure Data Engineer, your role will involve designing, developing, and maintaining data solutions on the Azure platform. You will be responsible for building and optimizing data pipelines, ensuring data quality and reliability, and implementing data processing and transformation logic. Your expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), PySpark, and Scala will be essential for performing the following key responsibilities:

Designing and developing data pipelines: You will design and implement scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala. This includes data ingestion, data transformation, and data loading processes.

Data modeling and database design: You will design and implement data models to support efficient data storage, retrieval, and analysis. This may involve working with relational databases, data lakes, or other storage solutions on the Azure platform.

Data integration and orchestration: You will leverage Azure Data Factory (ADF) to orchestrate data integration workflows and manage data movement across various data sources and targets. This includes scheduling and monitoring data pipelines.

Data quality and governance: You will implement data quality checks, validation rules, and data governance processes to ensure data accuracy, consistency, and compliance with relevant regulations and standards.

Performance optimization: You will optimize data pipelines and queries to improve overall system performance and reduce processing time. This may involve tuning SQL queries, optimizing data transformation logic, and leveraging caching techniques.

Monitoring and troubleshooting: You will monitor data pipelines, identify performance bottlenecks, and troubleshoot issues related to data ingestion, processing, and transformation. You will work closely with cross-functional teams to resolve data-related problems.

Documentation and collaboration: You will document data pipelines, data flows, and data transformation processes. You will collaborate with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.

Skills and Qualifications:

Strong experience with Azure Databricks, Python, SQL, ADF, PySpark, and Scala.

Proficiency in designing and developing data pipelines and ETL processes.

Solid understanding of data modeling concepts and database design principles.

Familiarity with data integration and orchestration using Azure Data Factory.

Knowledge of data quality management and data governance practices.

Experience with performance tuning and optimization of data pipelines.

Strong problem-solving and troubleshooting skills related to data engineering.

Excellent collaboration and communication skills to work effectively in cross-functional teams.

Understanding of cloud computing principles and experience with Azure services.

Requirements

Experience

• 5+ years of professional experience in implementing MLOps framework to scale up ML in production.
• Hands-on experience with Kubernetes, Kubeflow, MLflow, Sagemaker, and other ML model experiment management tools including training, inference, and evaluation.
• Experience in ML model serving (TorchServe, TensorFlow Serving, NVIDIA Triton inference server, etc.)
• Proficiency with ML model training frameworks (PyTorch, Pytorch Lightning, Tensorflow, etc.).
• Experience with GPU computing to do data and model training parallelism.
• Solid software engineering skills in developing systems for production.
• Strong expertise in Python.
• Building end-to-end data systems as an ML Engineer, Platform Engineer, or equivalent.
• Experience working with cloud data processing technologies (S3, ECR, Lambda, AWS, Spark, Dask, ElasticSearch, Presto, SQL, etc.).
• Having Geospatial / Remote sensing experience is a plus.

·      Core responsibilities to include analyze business requirements and designs for accuracy and completeness. Develops and maintains relevant product.

·      BlueYonder is seeking a Senior/Principal Architect in the Data Services department (under Luminate Platform ) to act as one of key technology leaders to build and manage BlueYonder’ s technology assets in the Data Platform and Services.

·      This individual will act as a trusted technical advisor and strategic thought leader to the Data Services department. The successful candidate will have the opportunity to lead, participate, guide, and mentor other people in the team on architecture and design in a hands-on manner. You are responsible for technical direction of Data Platform. This position reports to the Global Head, Data Services and will be based in Bangalore, India.

·      Core responsibilities to include Architecting and designing (along with counterparts and distinguished Architects) a ground up cloud native (we use Azure) SaaS product in Order management and micro-fulfillment

·      The team currently comprises of 60+ global associates across US, India (COE) and UK and is expected to grow rapidly. The incumbent will need to have leadership qualities to also mentor junior and mid-level software associates in our team. This person will lead the Data platform architecture – Streaming, Bulk with Snowflake/Elastic Search/other tools

Our current technical environment:

·      Software: Java, Springboot, Gradle, GIT, Hibernate, Rest API, OAuth , Snowflake

·      • Application Architecture: Scalable, Resilient, event driven, secure multi-tenant Microservices architecture

·      • Cloud Architecture: MS Azure (ARM templates, AKS, HD insight, Application gateway, Virtue Networks, Event Hub, Azure AD)

·      Frameworks/Others: Kubernetes, Kafka, Elasticsearch, Spark, NOSQL, RDBMS, Springboot, Gradle GIT, Ignite

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

About Kloud9:

Kloud9 exists with the sole purpose of providing cloud expertise to the retail industry. Our team of cloud architects, engineers and developers help retailers launch a successful cloud initiative so you can quickly realise the benefits of cloud technology. Our standardised, proven cloud adoption methodologies reduce the cloud adoption time and effort so you can directly benefit from lower migration costs.

Kloud9 was founded with the vision of bridging the gap between E-commerce and cloud. The E-commerce of any industry is limiting and poses a huge challenge in terms of the finances spent on physical data structures.

At Kloud9, we know migrating to the cloud is the single most significant technology shift your company faces today. We are your trusted advisors in transformation and are determined to build a deep partnership along the way. Our cloud and retail experts will ease your transition to the cloud.

Our sole focus is to provide cloud expertise to retail industry giving our clients the empowerment that will take their business to the next level. Our team of proficient architects, engineers and developers have been designing, building and implementing solutions for retailers for an average of more than 20 years.

We are a cloud vendor that is both platform and technology independent. Our vendor independence not just provides us with a unique perspective into the cloud market but also ensures that we deliver the cloud solutions available that best meet our clients' requirements.

What we are looking for:

● 3+ years’ experience developing Data & Analytic solutions

● Experience building data lake solutions leveraging one or more of the following AWS, EMR, S3, Hive& Spark

● Experience with relational SQL

● Experience with scripting languages such as Shell, Python

● Experience with source control tools such as GitHub and related dev process

● Experience with workflow scheduling tools such as Airflow

● In-depth knowledge of scalable cloud

● Has a passion for data solutions

● Strong understanding of data structures and algorithms

● Strong understanding of solution and technical design

● Has a strong problem-solving and analytical mindset

● Experience working with Agile Teams.

● Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

● Able to quickly pick up new programming languages, technologies, and frameworks

● Bachelor’s Degree in computer science

Why Explore a Career at Kloud9:

With job opportunities in prime locations of US, London, Poland and Bengaluru, we help build your career paths in cutting edge technologies of AI, Machine Learning and Data Science. Be part of an inclusive and diverse workforce that's changing the face of retail technology with their creativity and innovative solutions. Our vested interest in our employees translates to deliver the best products and solutions to our customers.

About Telstra

Telstra is Australia’s leading telecommunications and technology company, with operations in more than 20 countries, including In India where we’re building a new Innovation and Capability Centre (ICC) in Bangalore.

We’re growing, fast, and for you that means many exciting opportunities to develop your career at Telstra. Join us on this exciting journey, and together, we’ll reimagine the future.

Why Telstra?

• We're an iconic Australian company with a rich heritage that's been built over 100 years. Telstra is Australia's leading Telecommunications and Technology Company. We've been operating internationally for more than 70 years.
• International presence spanning over 20 countries.
• We are one of the 20 largest telecommunications providers globally
• At Telstra, the work is complex and stimulating, but with that comes a great sense of achievement. We are shaping the tomorrow's modes of communication with our innovation driven teams.

Telstra offers an opportunity to make a difference to lives of millions of people by providing the choice of flexibility in work and a rewarding career that you will be proud of!

About the team

Being part of Networks & IT means you'll be part of a team that focuses on extending our network superiority to enable the continued execution of our digital strategy.

With us, you'll be working with world-leading technology and change the way we do IT to ensure business needs drive priorities, accelerating our digitisation programme.

Focus of the role

Any new engineer who comes into data chapter would be mostly into developing reusable data processing and storage frameworks that can be used across data platform.

About you

To be successful in the role, you'll bring skills and experience in:-

Essential 

• Hands-on experience in Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing
• Skilled in the Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
• Azure data factory/Airflow/control-M/Luigi
• PL/SQL
• Exposure to NOSQL(Hbase/Cassandra/GraphDB(Neo4J)/MongoDB)
• File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
• Kafka/Kinesis/Eventhub

Highly Desirable

Experience and knowledgeable on the following:

• Spark Streaming
• Cloud exposure (Azure/AWS/GCP)
• Azure data offerings - ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc.
• Presto/Athena
• Azure DevOps
• Jenkins/ Bamboo/Any similar build tools
• Power BI
• Prior experience in building or working in team building reusable frameworks,
• Data modelling.
• Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
• Exposure to CI/CD
• Code Quality - Static and Dynamic code scans
• Agile SDLC      

If you've got a passion to innovate, succeed as part of a great team, and looking for the next step in your career, we'd welcome you to apply!

___________________________

We’re committed to building a diverse and inclusive workforce in all its forms. We encourage applicants from diverse gender, cultural and linguistic backgrounds and applicants who may be living with a disability. We also offer flexibility in all our roles, to ensure everyone can participate.

To learn more about how we support our people, including accessibility adjustments we can provide you through the recruitment process, visit tel.st/thrive.

About Kloud9:

Kloud9 exists with the sole purpose of providing cloud expertise to the retail industry. Our team of cloud architects, engineers and developers help retailers launch a successful cloud initiative so you can quickly realise the benefits of cloud technology. Our standardised, proven cloud adoption methodologies reduce the cloud adoption time and effort so you can directly benefit from lower migration costs.

Kloud9 was founded with the vision of bridging the gap between E-commerce and cloud. The E-commerce of any industry is limiting and poses a huge challenge in terms of the finances spent on physical data structures.

At Kloud9, we know migrating to the cloud is the single most significant technology shift your company faces today. We are your trusted advisors in transformation and are determined to build a deep partnership along the way. Our cloud and retail experts will ease your transition to the cloud.

Our sole focus is to provide cloud expertise to retail industry giving our clients the empowerment that will take their business to the next level. Our team of proficient architects, engineers and developers have been designing, building and implementing solutions for retailers for an average of more than 20 years.

We are a cloud vendor that is both platform and technology independent. Our vendor independence not just provides us with a unique perspective into the cloud market but also ensures that we deliver the cloud solutions available that best meet our clients' requirements.

What we are looking for:

●       3+ years’ experience developing Big Data & Analytic solutions

●       Experience building data lake solutions leveraging Google Data Products (e.g. Dataproc, AI Building Blocks, Looker, Cloud Data Fusion, Dataprep, etc.), Hive, Spark

●       Experience with relational SQL/No SQL

●       Experience with Spark (Scala/Python/Java) and Kafka

●       Work experience with using Databricks (Data Engineering and Delta Lake components)

●       Experience with source control tools such as GitHub and related dev process

●       Experience with workflow scheduling tools such as Airflow

●       In-depth knowledge of any scalable cloud vendor(GCP preferred)

●       Has a passion for data solutions

●       Strong understanding of data structures and algorithms

●       Strong understanding of solution and technical design

●       Has a strong problem solving and analytical mindset

●       Experience working with Agile Teams.

●       Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders

●       Able to quickly pick up new programming languages, technologies, and frameworks

●       Bachelor’s Degree in computer science

Why Explore a Career at Kloud9:

With job opportunities in prime locations of US, London, Poland and Bengaluru, we help build your career paths in cutting edge technologies of AI, Machine Learning and Data Science. Be part of an inclusive and diverse workforce that's changing the face of retail technology with their creativity and innovative solutions. Our vested interest in our employees translates to deliver the best products and solutions to our customers!

• NLP 

• ML programming
• Spark
• Model Deployment
• Experience processing unstructured data and building NLP models
• Experience with big data tools pyspark
• Pipeline orchestration using Airflow and model deployment experience is preferred

Role: Data Engineer  
Company: PayU

Location: Bangalore/ Mumbai

Experience : 2-5 yrs

About Company:

PayU is the payments and fintech business of Prosus, a global consumer internet group and one of the largest technology investors in the world. Operating and investing globally in markets with long-term growth potential, Prosus builds leading consumer internet companies that empower people and enrich communities.

The leading online payment service provider in 36 countries, PayU is dedicated to creating a fast, simple and efficient payment process for merchants and buyers. Focused on empowering people through financial services and creating a world without financial borders where everyone can prosper, PayU is one of the biggest investors in the fintech space globally, with investments totalling $700 million- to date. PayU also specializes in credit products and services for emerging markets across the globe. We are dedicated to removing risks to merchants, allowing consumers to use credit in ways that suit them and enabling a greater number of global citizens to access credit services.

Our local operations in Asia, Central and Eastern Europe, Latin America, the Middle East, Africa and South East Asia enable us to combine the expertise of high growth companies with our own unique local knowledge and technology to ensure that our customers have access to the best financial services.

India is the biggest market for PayU globally and the company has already invested $400 million in this region in last 4 years. PayU in its next phase of growth is developing a full regional fintech ecosystem providing multiple digital financial services in one integrated experience. We are going to do this through 3 mechanisms: build, co-build/partner; select strategic investments. 

PayU supports over 350,000+ merchants and millions of consumers making payments online with over 250 payment methods and 1,800+ payment specialists. The markets in which PayU operates represent a potential consumer base of nearly 2.3 billion people and a huge growth potential for merchants. 

Job responsibilities:

• Design infrastructure for data, especially for but not limited to consumption in machine learning applications 
• Define database architecture needed to combine and link data, and ensure integrity across different sources 
• Ensure performance of data systems for machine learning to customer-facing web and mobile applications using cutting-edge open source frameworks, to highly available RESTful services, to back-end Java based systems 
• Work with large, fast, complex data sets to solve difficult, non-routine analysis problems, applying advanced data handling techniques if needed 
• Build data pipelines, includes implementing, testing, and maintaining infrastructural components related to the data engineering stack.
• Work closely with Data Engineers, ML Engineers and SREs to gather data engineering requirements to prototype, develop, validate and deploy data science and machine learning solutions

Requirements to be successful in this role: 

• Strong knowledge and experience in Python, Pandas, Data wrangling, ETL processes, statistics, data visualisation, Data Modelling and Informatica.
• Strong experience with scalable compute solutions such as in Kafka, Snowflake
• Strong experience with workflow management libraries and tools such as Airflow, AWS Step Functions etc. 
• Strong experience with data engineering practices (i.e. data ingestion pipelines and ETL) 
• A good understanding of machine learning methods, algorithms, pipelines, testing practices and frameworks 
• Preferred) MEng/MSc/PhD degree in computer science, engineering, mathematics, physics, or equivalent (preference: DS/ AI) 
• Experience with designing and implementing tools that support sharing of data, code, practices across organizations at scale 

• Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
• Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
• Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
• Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
• Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
• Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
• Analyse large amounts of information to discover trends and patterns
• Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\

• Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
• 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
• Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
• Good understanding of AWS ecosystem for big data analytics
• Hands on experience in creating data pipelines either using tools or by independently writing scripts
• Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
• Strong problem solving skills with an emphasis on product development.
• Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)

We are looking for an exceptionally talented Lead data engineer who has exposure in implementing AWS services to build data pipelines, api integration and designing data warehouse. Candidate with both hands-on and leadership capabilities will be ideal for this position.

Qualification: At least a bachelor’s degree in Science, Engineering, Applied Mathematics. Preferred Masters degree

Job Responsibilities:

• Total 6+ years of experience as a Data Engineer and 2+ years of experience in managing a team

• Have minimum 3 years of AWS Cloud experience.

• Well versed in languages such as Python, PySpark, SQL, NodeJS etc

• Has extensive experience in the real-timeSpark ecosystem and has worked on both real time and batch processing

• Have experience in AWS Glue, EMR, DMS, Lambda, S3, DynamoDB, Step functions, Airflow, RDS, Aurora etc.

• Experience with modern Database systems such as Redshift, Presto, Hive etc.

• Worked on building data lakes in the past on S3 or Apache Hudi

• Solid understanding of Data Warehousing Concepts

• Good to have experience on tools such as Kafka or Kinesis

• Good to have AWS Developer Associate or Solutions Architect Associate Certification

• Have experience in managing a team

• Hands-on experience in any Cloud Platform

• Microsoft Azure Experience

4-5 years of overall experience in software development.
- Experience on Hadoop (Apache/Cloudera/Hortonworks) and/or other Map Reduce Platforms
- Experience on Hive, Pig, Sqoop, Flume and/or Mahout
- Experience on NO-SQL – HBase, Cassandra, MongoDB
- Hands on experience with Spark development,  Knowledge of Storm, Kafka, Scala
- Good knowledge of Java
- Good background of Configuration Management/Ticketing systems like Maven/Ant/JIRA etc.
- Knowledge around any Data Integration and/or EDW tools is plus
- Good to have knowledge of  using Python/Perl/Shell

Please note - Hbase hive and spark are must.

Responsibilities:

* 3+ years of Data Engineering Experience - Design, develop, deliver and maintain data infrastructures.

* SQL Specialist – Strong knowledge and Seasoned experience with SQL Queries

* Languages: Python

* Good communicator, shows initiative, works well with stakeholders.

* Experience working closely with Data Analysts and provide the data they need and guide them on the issues.

* Solid ETL experience and Hadoop/Hive/Pyspark/Presto/ SparkSQL

* Solid communication and articulation skills

* Able to handle stakeholders independently with less interventions of reporting manager.

* Develop strategies to solve problems in logical yet creative ways.

* Create custom reports and presentations accompanied by strong data visualization and storytelling

We would be excited if you have:

* Excellent communication and interpersonal skills

* Ability to meet deadlines and manage project delivery

* Excellent report-writing and presentation skills

* Critical thinking and problem-solving capabilities

• Involvement in the overall application lifecycle
• Design and develop software applications in Scala and Spark
• Understand business requirements and convert them to technical solutions
• Rest API design, implementation, and integration
• Collaborate with Frontend developers and provide mentorship for Junior engineers in the team
• An interest and preferably working experience in agile development methodologies
• A team player, eager to invest in personal and team growth
• Staying up to date with cutting edge technologies and best practices
• Advocate for improvements to product quality, security, and performance

Desired Skills and Experience

• Minimum 1.5+ years of development experience in Scala / Java language
• Strong understanding of the development cycle, programming techniques, and tools.
• Strong problem solving and verbal and written communication skills.
• Experience in working with web development using J2EE or similar frameworks
• Experience in developing REST API’s
• BE in Computer Science
• Experience with Akka or Micro services is a plus
• Experience with Big data technologies like Spark/Hadoop is a plus company offers very competitive compensation packages commensurate with your experience. We offer full benefits, continual career & compensation growth, and many other perks.

• We are looking for a Data Engineer to build the next-generation mobile applications for our world-class fintech product.
• The candidate will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection for cross-functional teams.
• The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimising data systems and building them from the ground up.
• Looking for a person with a strong ability to analyse and provide valuable insights to the product and business team to solve daily business problems.
• You should be able to work in a high-volume environment, have outstanding planning and organisational skills.

Qualifications for Data Engineer

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimising ‘big data’ data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Looking for a candidate with 2-3 years of experience in a Data Engineer role, who is a CS graduate or has an equivalent experience.

What we're looking for?

• Experience with big data tools: Hadoop, Spark, Kafka and other alternate tools.
• Experience with relational SQL and NoSQL databases, including MySql/Postgres and Mongodb.
• Experience with data pipeline and workflow management tools: Luigi, Airflow.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift.
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala.

Job Title: Engineering Manager

Job Location: Chennai, Bangalore
Job Summary
The Engineering Org is looking for a proficient Engineering Manager to join a team that is building exciting
and futuristic Data Products at Condé Nast to enable both internal and external marketers to target
audiences in real time. As an Engineering Manager, you will drive the day-to-day execution of technical
and architectural decisions. EM will own engineering deliverables inclusive of solving dependencies
such as architecture, solutions, sequencing, and working with other engineering delivery teams.This role
is also responsible for driving innovation, prototyping, and recommending solutions. Above all, you will
influence how users interact with Conde Nast’s industry-leading journalism.
● Primary Responsibilities
● Manage a high performing team of Software and Data Engineers within the Data & ML
Engineering team part of Engineering Data Organization.
● Provide leadership and guidance to the team in Data Discovery, Data Ingestion, Transformation
and Storage
● Utilizing product mindset to build, scale and deploy holistic data products after successful
prototyping and drive their engineering implementation
● Provide technical coaching and lead direct reports and other members of adjacent support teams
to the highest level of performance..
● Evaluate performance of direct reports and offer career development guidance.
● Meeting hiring and retention targets of the team & building a high-performance culture
● Handle escalations from internal stakeholders and manage critical issues to resolution.
● Collaborate with Architects, Product Manager, Project Manager and other teams to deliver high
quality products.
● Identify recurring system and application issues and enable engineers to work with release teams,
infra teams, product development, vendors and other stakeholders in investigating and resolving
the cause.
● Required Skills
● 4+ years of managing Software Development teams, preferably in ML and Data
Engineering teams.
● 4+ years of Agile Software development practices
● 12+ years of Software Development experience.
● Excellent Problem Solving and System Design skill
● Hands on: Writing and Reviewing code primarily in Spark, Python and/or Java
● Hand on: Architect & Design end to end Data Pipeline (noSQL databases, Job Schedulers, Big
Data Development preferably on Databricks / Cloud)
● Experience with SOA & Microservice architecture
● Knowledge of Software Engineering best practices with experience on implementing CI/CD,
Log aggregation/Monitoring/alerting for production system
● Working Knowledge of cloud and devops skills (AWS will be preferred)
● Strong verbal and written communication skills.
● Experience in evaluating team member performance and offering career development
guidance.
● Experience in providing technical coaching to direct reports.
● Experience in architecting highly scalable products.
● Experience in collaborating with global stakeholder teams.
● Experience in working on highly available production systems.
● Strong knowledge of software release process and release pipeline.
About Condé Nast
CONDÉ NAST INDIA (DATA)
Over the years, Condé Nast successfully expanded and diversified into digital, TV, and social
platforms - in other words, a staggering amount of user data. Condé Nast made the right move to
invest heavily in understanding this data and formed a whole new Data team entirely dedicated to
data processing, engineering, analytics, and visualization. This team helps drive engagement, fuel
process innovation, further content enrichment, and increase market revenue. The Data team
aimed to create a company culture where data was the common language and facilitate an
environment where insights shared in real-time could improve performance.
The Global Data team operates out of Los Angeles, New York, Chennai, and London. The team at
Condé Nast Chennai works extensively with data to amplify its brands' digital capabilities and boost
online revenue. We are broadly divided into four groups, Data Intelligence, Data Engineering, Data
Science, and Operations (including Product and Marketing Ops, Client Services) along with Data
Strategy and monetization. The teams built capabilities and products to create data-driven solutions
for better audience engagement.
What we look forward to:
We want to welcome bright, new minds into our midst and work together to create diverse forms of
self-expression. At Condé Nast, we encourage the imaginative and celebrate the extraordinary. We
are a media company for the future, with a remarkable past. We are Condé Nast, and It Starts Here.

• 3-6 years of relevant work experience in a Data Engineering role.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimizing data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• A good understanding of Airflow, Spark, NoSQL databases, Kafka is nice to have.
• Premium Institute Candidates only

• We are looking for : Data engineer
• Sprak
• Scala
• Hadoop