Data Engineer

Job Description:

As an Azure Data Engineer, your role will involve designing, developing, and maintaining data solutions on the Azure platform. You will be responsible for building and optimizing data pipelines, ensuring data quality and reliability, and implementing data processing and transformation logic. Your expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), PySpark, and Scala will be essential for performing the following key responsibilities:

Designing and developing data pipelines: You will design and implement scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala. This includes data ingestion, data transformation, and data loading processes.

Data modeling and database design: You will design and implement data models to support efficient data storage, retrieval, and analysis. This may involve working with relational databases, data lakes, or other storage solutions on the Azure platform.

Data integration and orchestration: You will leverage Azure Data Factory (ADF) to orchestrate data integration workflows and manage data movement across various data sources and targets. This includes scheduling and monitoring data pipelines.

Data quality and governance: You will implement data quality checks, validation rules, and data governance processes to ensure data accuracy, consistency, and compliance with relevant regulations and standards.

Performance optimization: You will optimize data pipelines and queries to improve overall system performance and reduce processing time. This may involve tuning SQL queries, optimizing data transformation logic, and leveraging caching techniques.

Monitoring and troubleshooting: You will monitor data pipelines, identify performance bottlenecks, and troubleshoot issues related to data ingestion, processing, and transformation. You will work closely with cross-functional teams to resolve data-related problems.

Documentation and collaboration: You will document data pipelines, data flows, and data transformation processes. You will collaborate with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.

Skills and Qualifications:

Strong experience with Azure Databricks, Python, SQL, ADF, PySpark, and Scala.

Proficiency in designing and developing data pipelines and ETL processes.

Solid understanding of data modeling concepts and database design principles.

Familiarity with data integration and orchestration using Azure Data Factory.

Knowledge of data quality management and data governance practices.

Experience with performance tuning and optimization of data pipelines.

Strong problem-solving and troubleshooting skills related to data engineering.

Excellent collaboration and communication skills to work effectively in cross-functional teams.

Understanding of cloud computing principles and experience with Azure services.

Specific Responsibilities

• Minimum of 2 years Experience in Google Big Query and Google Cloud Platform.
• Design and develop the ETL framework using BigQuery
• Expertise in Big Query concepts like Nested Queries, Clustering, Partitioning, etc.
• Working Experience of Clickstream database, Google Analytics/ Adobe Analytics.
• Should be able to automate the data load from Big Query using APIs or scripting language.
• Good experience in Advanced SQL concepts.
• Good experience with Adobe launch Web, Mobile & e-commerce tag implementation.
• Identify complex fuzzy problems, break them down in smaller parts, and implement creative, data-driven solutions
• Responsible for defining, analyzing, and communicating key metrics and business trends to the management teams
• Identify opportunities to improve conversion & user experience through data. Influence product & feature roadmaps.
• Must have a passion for data quality and be constantly looking to improve the system. Drive data-driven decision making through the stakeholders & drive Change Management
• Understand requirements to translate business problems & technical problems into analytics problems.
• Effective storyboarding and presentation of the solution to the client and leadership.
• Client engagement & management
• Ability to interface effectively with multiple levels of management and functional disciplines.
• Assist in developing/coaching individuals technically as well as on soft skills during the project and as part of Client Project’s training program.

• 2 to 3 years of working experience in Google Big Query & Google Cloud Platform
• Relevant experience in Consumer Tech/CPG/Retail industries
• Bachelor’s in engineering, Computer Science, Math, Statistics or related discipline
• Strong problem solving and web analytical skills. Acute attention to detail.
• Experience in analyzing large, complex, multi-dimensional data sets.
• Experience in one or more roles in an online eCommerce or online support environment.

• Expertise in Google Big Query & Google Cloud Platform
• Experience in Advanced SQL, Scripting language (Python/R)
• Hands-on experience in BI tools (Tableau, Power BI)
• Working Experience & understanding of Adobe Analytics or Google Analytics
• Experience in creating and debugging website & app tracking (Omnibus, Dataslayer, GA debugger, etc.)
• Excellent analytical thinking, analysis, and problem-solving skills.
• Knowledge of other GCP services is a plus

DATA ENGINEER

Overview

They started with a singular belief - what is beautiful cannot and should not be defined in marketing meetings. It's defined by the regular people like us, our sisters, our next-door neighbours, and the friends we make on the playground and in lecture halls. That's why we stand for people-proving everything we do. From the inception of a product idea to testing the final formulations before launch, our consumers are a part of each and every process. They guide and inspire us by sharing their stories with us. They tell us not only about the product they need and the skincare issues they face but also the tales of their struggles, dreams and triumphs. Skincare goes deeper than skin. It's a form of self-care for many. Wherever someone is on this journey, we want to cheer them on through the products we make, the content we create and the conversations we have. What we wish to build is more than a brand. We want to build a community that grows and glows together - cheering each other on, sharing knowledge, and ensuring people always have access to skincare that really works.

Job Description:

We are seeking a skilled and motivated Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, developing, and maintaining the data infrastructure and systems that enable efficient data collection, storage, processing, and analysis. You will collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to implement data pipelines and ensure the availability, reliability, and scalability of our data platform.

Responsibilities:

Design and implement scalable and robust data pipelines to collect, process, and store data from various sources.

Develop and maintain data warehouse and ETL (Extract, Transform, Load) processes for data integration and transformation.

Optimize and tune the performance of data systems to ensure efficient data processing and analysis.

Collaborate with data scientists and analysts to understand data requirements and implement solutions for data modeling and analysis.

Identify and resolve data quality issues, ensuring data accuracy, consistency, and completeness.

Implement and maintain data governance and security measures to protect sensitive data.

Monitor and troubleshoot data infrastructure, perform root cause analysis, and implement necessary fixes.

Stay up-to-date with emerging technologies and industry trends in data engineering and recommend their adoption when appropriate.

Qualifications:

Bachelor’s or higher degree in Computer Science, Information Systems, or a related field.

Proven experience as a Data Engineer or similar role, working with large-scale data processing and storage systems.

Strong programming skills in languages such as Python, Java, or Scala.

Experience with big data technologies and frameworks like Hadoop, Spark, or Kafka.

Proficiency in SQL and database management systems (e.g., MySQL, PostgreSQL, or Oracle).

Familiarity with cloud platforms like AWS, Azure, or GCP, and their data services (e.g., S3, Redshift, BigQuery).

Solid understanding of data modeling, data warehousing, and ETL principles.

Knowledge of data integration techniques and tools (e.g., Apache Nifi, Talend, or Informatica).

Strong problem-solving and analytical skills, with the ability to handle complex data challenges.

Excellent communication and collaboration skills to work effectively in a team environment.

Preferred Qualifications:

Advanced knowledge of distributed computing and parallel processing.

Experience with real-time data processing and streaming technologies (e.g., Apache Kafka, Apache Flink).

Familiarity with machine learning concepts and frameworks (e.g., TensorFlow, PyTorch).

Knowledge of containerization and orchestration technologies (e.g., Docker, Kubernetes).

Experience with data visualization and reporting tools (e.g., Tableau, Power BI).

Certification in relevant technologies or data engineering disciplines.

Experience: 1- 5 Years

Job Location: WFH

No. of Position: Multiple

Qualifications: Ph.D. Must have

Work Timings: 1:30 PM IST to 10:30 PM IST

Functional Area: Data Science

NextGen Invent is currently searching for Data Scientist. This role will directly report to the VP, Data Science in Data Science Practice. The person will work on data science use-cases for the enterprise and must have deep expertise in supervised and unsupervised machine learning, modeling and algorithms with a strong focus on delivering use-cases and solutions at speed and scale to solve business problems.

Job Responsibilities:

• Leverage AI/ML modeling and algorithms to deliver on use cases
• Build modeling solutions at speed and scale to solve business problems
• Develop data science solutions that can be tested and deployed in Agile delivery model
• Implement and scale-up high-availability models and algorithms for various business and corporate functions
• Investigate and create experimental prototypes that work on specific domains and verticals
• Analyze large, complex data sets to reveal underlying patterns, and trends
• Support and enhance existing models to ensure better performance
• Set up and conduct large-scale experiments to test hypotheses and delivery of models

Skills, Knowledge, Experience:

• Must have Ph.D. in an analytical or technical field (e.g. applied mathematics, computer science)
• Strong knowledge of statistical and machine learning methods
• Hands on experience on building models at speed and scale
• Ability to work in a collaborative, transparent style with cross-functional stakeholders across the organization to lead and deliver results
• Strong skills in oral and written communication
• Ability to lead a high-functioning team and develop and train people
• Must have programming experience in SQL, Python and R
• Experience conceiving, implementing and continually improving machine learning projects
• Strong familiarity with higher level trends in artificial intelligence and open-source platforms
• Experience working with AWS, Azure, or similar cloud platform
• Familiarity with visualization techniques and software
• Healthcare experience is a plus
• Experience in Kafka, Chatbot and blockchain is a plus.