Senior Data Engineer

The Client is the world’s largest media investment company. Our team of experts support clients in programmatic, social, paid search, analytics, technology, organic search, affiliate marketing, e-commerce and across traditional channel We are currently looking for a Manager Analyst – Analytics to join us. In this role, you will work on

various projects for the in-house team across data management, reporting, and analytics.

Responsibility:

•       Serve as a Subject Matter Expert on data usage – extraction, manipulation, and inputs for analytics

•       Develop data extraction and manipulation code based on business rules

•       Design and construct data store and procedures for their maintenance  Develop and maintain strong relationships with stakeholders  Write high-quality code as per prescribed standards.

•       Participate in internal projects as required

Requirements:

•       2-5 years for strong experience in working with SQL, Python, ETL development.

•       Strong Experience in writing complex SQLs

•       Good Communication skills

•       Good experience of working with any BI tool like Tableau, Power BI.

•       Familiar with various cloud technologies and their offerings within the data specialization and Data Warehousing.

•       Snowflake, AWS are good to have.

Minimum qualifications:

•       B. Tech./MCA or equivalent preferred

Excellent 2 years Hand on experience on Big data, ETL Development, Data Processing.  

Looking for freelance?

We are seeking a freelance Data Engineer with 7+ years of experience

Skills Required: Deep knowledge in any cloud (AWS, Azure , Google cloud), Data bricks, Data lakes, Data Ware housing Python/Scala , SQL, BI, and other analytics systems

What we are looking for

We are seeking an experienced Senior Data Engineer with experience in architecture, design, and development of highly scalable data integration and data engineering processes

• The Senior Consultant must have a strong understanding and experience with data & analytics solution architecture, including data warehousing, data lakes, ETL/ELT workload patterns, and related BI & analytics systems
• Strong in scripting languages like Python, Scala
• 5+ years of hands-on experience with one or more of these data integration/ETL tools.
• Experience building on-prem data warehousing solutions.
• Experience with designing and developing ETLs, Data Marts, Star Schema
• Designing a data warehouse solution using Synapse or Azure SQL DB
• Experience building pipelines using Synapse or Azure Data Factory to ingest data from various sources
• Understanding of integration run times available in Azure.
• Advanced working SQL knowledge and experience working with relational databases, and queries. authoring (SQL) as well as working familiarity with a variety of database

Roles and Responsibilities:

• Design, develop, and maintain the end-to-end MLOps infrastructure from the ground up, leveraging open-source systems across the entire MLOps landscape.
• Creating pipelines for data ingestion, data transformation, building, testing, and deploying machine learning models, as well as monitoring and maintaining the performance of these models in production.
• Managing the MLOps stack, including version control systems, continuous integration and deployment tools, containerization, orchestration, and monitoring systems.
• Ensure that the MLOps stack is scalable, reliable, and secure.

Skills Required:

• 3-6 years of MLOps experience
• Preferably worked in the startup ecosystem

Primary Skills:

• Experience with E2E MLOps systems like ClearML, Kubeflow, MLFlow etc.
• Technical expertise in MLOps: Should have a deep understanding of the MLOps landscape and be able to leverage open-source systems to build scalable, reliable, and secure MLOps infrastructure.
• Programming skills: Proficient in at least one programming language, such as Python, and have experience with data science libraries, such as TensorFlow, PyTorch, or Scikit-learn.
• DevOps experience: Should have experience with DevOps tools and practices, such as Git, Docker, Kubernetes, and Jenkins.

Secondary Skills:

• Version Control Systems (VCS) tools like Git and Subversion
• Containerization technologies like Docker and Kubernetes
• Cloud Platforms like AWS, Azure, and Google Cloud Platform
• Data Preparation and Management tools like Apache Spark, Apache Hadoop, and SQL databases like PostgreSQL and MySQL
• Machine Learning Frameworks like TensorFlow, PyTorch, and Scikit-learn
• Monitoring and Logging tools like Prometheus, Grafana, and Elasticsearch
• Continuous Integration and Continuous Deployment (CI/CD) tools like Jenkins, GitLab CI, and CircleCI
• Explain ability and Interpretability tools like LIME and SHAP

This profile will include the following responsibilities:

- Develop Parsers for XML and JSON Data sources/feeds

- Write Automation Scripts for product development

- Build API Integrations for 3^rd Party product integration

- Perform Data Analysis

- Research on Machine learning algorithms

- Understand AWS cloud architecture and work with 3 party vendors for deployments

What you need to have:

• Power BI Report and Dashboard development.
• Building Analysis Services reporting models.
• Developing visual reports, KPI scorecards, and dashboards using Power BI desktop.
• Connecting data sources, importing data, and transforming data for Business intelligence.
• Analytical thinking for translating data into informative reports and visuals.
• Should have an edge over making DAX queries in Power BI desktop.
• Expert in using advanced-level calculations on the data set.
• SQL Server with SSAS is must.
• Very good communication skills must be able to discuss the requirements effectively with the client teams, and with internal teams.

Role and Responsibilities

• Build a low latency serving layer that powers DataWeave's Dashboards, Reports, and Analytics functionality
• Build robust RESTful APIs that serve data and insights to DataWeave and other products
• Design user interaction workflows on our products and integrating them with data APIs
• Help stabilize and scale our existing systems. Help design the next generation systems.
• Scale our back end data and analytics pipeline to handle increasingly large amounts of data.
• Work closely with the Head of Products and UX designers to understand the product vision and design philosophy
• Lead/be a part of all major tech decisions. Bring in best practices. Mentor younger team members and interns.
• Constantly think scale, think automation. Measure everything. Optimize proactively.
• Be a tech thought leader. Add passion and vibrance to the team. Push the envelope.

Skills and Requirements

• 8- 15 years of experience building and scaling APIs and web applications.
• Experience building and managing large scale data/analytics systems.
• Have a strong grasp of CS fundamentals and excellent problem solving abilities. Have a good understanding of software design principles and architectural best practices.
• Be passionate about writing code and have experience coding in multiple languages, including at least one scripting language, preferably Python.
• Be able to argue convincingly why feature X of language Y rocks/sucks, or why a certain design decision is right/wrong, and so on.
• Be a self-starter—someone who thrives in fast paced environments with minimal ‘management’.
• Have experience working with multiple storage and indexing technologies such as MySQL, Redis, MongoDB, Cassandra, Elastic.
• Good knowledge (including internals) of messaging systems such as Kafka and RabbitMQ.
• Use the command line like a pro. Be proficient in Git and other essential software development tools.
• Working knowledge of large-scale computational models such as MapReduce and Spark is a bonus.
• Exposure to one or more centralized logging, monitoring, and instrumentation tools, such as Kibana, Graylog, StatsD, Datadog etc.
• Working knowledge of building websites and apps. Good understanding of integration complexities and dependencies.
• Working knowledge linux server administration as well as the AWS ecosystem is desirable.
• It's a huge bonus if you have some personal projects (including open source contributions) that you work on during your spare time. Show off some of your projects you have hosted on GitHub.