Sr Hadoop Operations Engineer

DATA ENGINEER

Overview

They started with a singular belief - what is beautiful cannot and should not be defined in marketing meetings. It's defined by the regular people like us, our sisters, our next-door neighbours, and the friends we make on the playground and in lecture halls. That's why we stand for people-proving everything we do. From the inception of a product idea to testing the final formulations before launch, our consumers are a part of each and every process. They guide and inspire us by sharing their stories with us. They tell us not only about the product they need and the skincare issues they face but also the tales of their struggles, dreams and triumphs. Skincare goes deeper than skin. It's a form of self-care for many. Wherever someone is on this journey, we want to cheer them on through the products we make, the content we create and the conversations we have. What we wish to build is more than a brand. We want to build a community that grows and glows together - cheering each other on, sharing knowledge, and ensuring people always have access to skincare that really works.

Job Description:

We are seeking a skilled and motivated Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, developing, and maintaining the data infrastructure and systems that enable efficient data collection, storage, processing, and analysis. You will collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to implement data pipelines and ensure the availability, reliability, and scalability of our data platform.

Responsibilities:

Design and implement scalable and robust data pipelines to collect, process, and store data from various sources.

Develop and maintain data warehouse and ETL (Extract, Transform, Load) processes for data integration and transformation.

Optimize and tune the performance of data systems to ensure efficient data processing and analysis.

Collaborate with data scientists and analysts to understand data requirements and implement solutions for data modeling and analysis.

Identify and resolve data quality issues, ensuring data accuracy, consistency, and completeness.

Implement and maintain data governance and security measures to protect sensitive data.

Monitor and troubleshoot data infrastructure, perform root cause analysis, and implement necessary fixes.

Stay up-to-date with emerging technologies and industry trends in data engineering and recommend their adoption when appropriate.

Qualifications:

Bachelor’s or higher degree in Computer Science, Information Systems, or a related field.

Proven experience as a Data Engineer or similar role, working with large-scale data processing and storage systems.

Strong programming skills in languages such as Python, Java, or Scala.

Experience with big data technologies and frameworks like Hadoop, Spark, or Kafka.

Proficiency in SQL and database management systems (e.g., MySQL, PostgreSQL, or Oracle).

Familiarity with cloud platforms like AWS, Azure, or GCP, and their data services (e.g., S3, Redshift, BigQuery).

Solid understanding of data modeling, data warehousing, and ETL principles.

Knowledge of data integration techniques and tools (e.g., Apache Nifi, Talend, or Informatica).

Strong problem-solving and analytical skills, with the ability to handle complex data challenges.

Excellent communication and collaboration skills to work effectively in a team environment.

Preferred Qualifications:

Advanced knowledge of distributed computing and parallel processing.

Experience with real-time data processing and streaming technologies (e.g., Apache Kafka, Apache Flink).

Familiarity with machine learning concepts and frameworks (e.g., TensorFlow, PyTorch).

Knowledge of containerization and orchestration technologies (e.g., Docker, Kubernetes).

Experience with data visualization and reporting tools (e.g., Tableau, Power BI).

Certification in relevant technologies or data engineering disciplines.

We are seeking an experienced Senior Data Platform Engineer to join our team. The ideal candidate should have extensive experience with Pyspark, Airflow, Presto, Hive, Kafka and Debezium, and should be passionate about developing scalable and reliable data platforms.

Responsibilities:

• Design, develop, and maintain our data platform architecture using Pyspark, Airflow, Presto, Hive, Kafka, and Debezium.
• Develop and maintain ETL processes to ingest, transform, and load data from various sources into our data platform.
• Work closely with data analysts, data scientists, and other stakeholders to understand their requirements and design solutions that meet their needs.
• Implement and maintain data governance policies and procedures to ensure data quality, privacy, and security.
• Continuously monitor and optimize the performance of our data platform to ensure scalability, reliability, and cost-effectiveness.
• Keep up-to-date with the latest trends and technologies in the field of data engineering and share knowledge and best practices with the team.

Requirements:

• Bachelor's degree in Computer Science, Information Technology, or related field.
• 5+ years of experience in data engineering or related fields.
• Strong proficiency in Pyspark, Airflow, Presto, Hive, Datalake, and Debezium.
• Experience with data warehousing, data modeling, and data governance.
• Experience working with large-scale distributed systems and cloud platforms (e.g., AWS, GCP, Azure).
• Strong problem-solving skills and ability to work independently and collaboratively.
• Excellent communication and interpersonal skills.

If you are a self-motivated and driven individual with a passion for data engineering and a strong background in Pyspark, Airflow, Presto, Hive, Datalake, and Debezium, we encourage you to apply for this exciting opportunity. We offer competitive compensation, comprehensive benefits, and a collaborative work environment that fosters innovation and growth.

• Collaborate with the business teams to understand the data environment in the organization; develop and lead the Data Scientists team to test and scale new algorithms through pilots and subsequent scaling up of the solutions
• Influence, build and maintain the large-scale data infrastructure required for the AI projects, and integrate with external IT infrastructure/service
• Act as the single point source for all data related queries; strong understanding of internal and external data sources; provide inputs in deciding data-schemas
• Design, develop and maintain the framework for the analytics solutions pipeline
• Provide inputs to the organization’s initiatives on data quality and help implement frameworks and tools for the various related initiatives
• Work in cross-functional teams of software/machine learning engineers, data scientists, product managers, and others to build the AI ecosystem
• Collaborate with the external organizations including vendors, where required, in respect of all data-related queries as well as implementation initiatives

Technical/Core skills

1. Minimum 3 yrs of exp in Informatica Big data Developer(BDM) in Hadoop environment.
2. Have knowledge of informatica Power exchange (PWX).
3. Minimum 3 yrs of exp in big data querying tool like Hive and Impala.
4. Ability to designing/development of complex mappings using informatica Big data Developer.
5. Create and manage Informatica power exchange and CDC real time implementation
6. Strong Unix knowledge skills for writing shell scripts and troubleshoot of existing scripts.
7. Good knowledge of big data platforms and its framework.
8. Good to have an experience in cloudera data platform (CDP)
9. Experience with building stream processing systems using Kafka and spark
10. Excellent SQL knowledge

Soft skills :

1. Ability to work independently 
2. Strong analytical and problem solving skills
3. Attitude of learning new technology
4. Regular interaction with vendors, partners and stakeholders

•  We are looking for a Data Engineer with 3-5 years experience in Python, SQL, AWS (EC2, S3, Elastic Beanstalk, API Gateway), and Java.
• The applicant must be able to perform Data Mapping (data type conversion, schema harmonization) using Python, SQL, and Java.
• The applicant must be familiar with and have programmed ETL interfaces (OAUTH, REST API, ODBC) using the same languages.
• The company is looking for someone who shows an eagerness to learn and who asks concise questions when communicating with teammates.

1. Proven experience with deploying and tuning Open Source components into enterprise ready production tooling Experience with datacentre (Metal as a Service – MAAS) and cloud deployment technologies (AWS or GCP Architect certificates required)
2. Deep understanding of Linux from kernel mechanisms through user space management
3. Experience on CI/CD (Continuous Integrations and Deployment) system solutions (Jenkins).
4. Using Monitoring tools (local and on public cloud platforms) Nagios, Prometheus, Sensu, ELK, Cloud Watch, Splunk, New Relic etc. to trigger instant alerts, reports and dashboards.  Work closely with the development and infrastructure teams to analyze and design solutions with four nines (99.99%) up-time, globally distributed, clustered, production and non-production virtualized infrastructure. 
5. Wide understanding of IP networking as well as data centre infrastructure

1. Expert with software development tools and sourcecode management, understanding, managing issues, code changes and grouping them into deployment releases in a stable and measurable way to maximize production Must be expert at developing and using ansible roles and configuring deployment templates with jinja2.
2. Solid understanding of data collection tools like Flume, Filebeat, Metricbeat, JMX Exporter agents.
3. Extensive experience operating and tuning the kafka streaming data platform, specifically as a message queue for big data processing
4. Strong understanding and must have experience:
5. Apache spark framework, specifically spark core and spark streaming, 
6. Orchestration platforms, mesos and kubernetes, 
7. Data storage platforms, elasticstack, carbon, clickhouse, cassandra, ceph, hdfs
8. Core presentation technologies kibana, and grafana.
9. Excellent scripting and programming skills (bash, python, java, go, rust). Must have previous experience with “rust” in order to support, improve in house developed products

Red Hat Certified Architect certificate or equivalent required CCNA certificate required 3-5 years of experience running open source big data platforms