Data Engineer

Data Engineer 

Mandatory Requirements  

• Experience in AWS Glue 

• Experience in Apache Parquet  
• Proficient in AWS S3 and data lake  
• Knowledge of Snowflake 
• Understanding of file-based ingestion best practices. 
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS  
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies  
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform  

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations  
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data. 
• Define process improvement opportunities to optimize data collection, insights and displays. 
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible  
• Identify and interpret trends and patterns from complex data sets  

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.  
• Key participant in regular Scrum ceremonies with the agile teams   
• Proficient at developing queries, writing reports and presenting findings  
• Mentor junior members and bring best industry practices  

QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)  
• Strong background in math, statistics, computer science, data science or related discipline 
• Advanced knowledge one of language: Java, Scala, Python, C#  
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake   

• Proficient with 
• Data mining/programming tools (e.g. SAS, SQL, R, Python) 
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum) 
• Data visualization (e.g. Tableau, Looker, MicroStrategy) 
• Comfortable learning about and deploying new technologies and tools.  

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.  
• Good written and oral communication skills and ability to present results to non-technical audiences  
• Knowledge of business intelligence and analytical tools, technologies and techniques. 

Familiarity and experience in the following is a plus:  

• AWS certification 
• Spark Streaming  
• Kafka Streaming / Kafka Connect  
• ELK Stack  
• Cassandra / MongoDB  

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools 

THE IDEAL CANDIDATE WILL

• Engage with executive level stakeholders from client's team to translate business problems to high level solution approach
• Partner closely with practice, and technical teams to craft well-structured comprehensive proposals/ RFP responses clearly highlighting Tredence’s competitive strengths relevant to Client's selection criteria
• Actively explore the client’s business and formulate solution ideas that can improve process efficiency and cut cost, or achieve growth/revenue/profitability targets faster
• Work hands-on across various MLOps problems and provide thought leadership
• Grow and manage large teams with diverse skillsets
• Collaborate, coach, and learn with a growing team of experienced Machine Learning Engineers and Data Scientists

ELIGIBILITY CRITERIA

• BE/BTech/MTech (Specialization/courses in ML/DS)
• At-least 7+ years of Consulting services delivery experience
• Very strong problem-solving skills & work ethics
• Possesses strong analytical/logical thinking, storyboarding and executive communication skills
• 5+ years of experience in Python/R, SQL
• 5+ years of experience in NLP algorithms, Regression & Classification Modelling, Time Series Forecasting
• Hands on work experience in DevOps
• Should have good knowledge in different deployment type like PaaS, SaaS, IaaS
• Exposure on cloud technologies like Azure, AWS or GCP
• Knowledge in python and packages for data analysis (scikit-learn, scipy, numpy, pandas, matplotlib).
• Knowledge of Deep Learning frameworks: Keras, Tensorflow, PyTorch, etc
• Experience with one or more Container-ecosystem (Docker, Kubernetes)
• Experience in building orchestration pipeline to convert plain python models into a deployable API/RESTful endpoint.
• Good understanding of OOP & Data Structures concepts

Nice to Have:

• Exposure to deployment strategies like: Blue/Green, Canary, AB Testing, Multi-arm Bandit
• Experience in Helm is a plus
• Strong understanding of data infrastructure, data warehouse, or data engineering

You can expect to –

• Work with world’ biggest retailers and help them solve some of their most critical problems. Tredence is a preferred analytics vendor for some of the largest Retailers across the globe
• Create multi-million Dollar business opportunities by leveraging impact mindset, cutting edge solutions and industry best practices.
• Work in a diverse environment that keeps evolving
• Hone your entrepreneurial skills as you contribute to growth of the organization

• Product Analytics: This is the first and most obvious role of the Product Analyst. At this capacity, the Product Analyst is responsible for the development and delivery of tangible consumer benefits through the product or service of the business.
• In addition, in this capacity, the Product Analyst is also responsible for measuring and monitoring the product or service’s performance as well as presenting product-related consumer, market, and competitive intelligence.
• Product Strategy: As a member of the Product team, the Product Analyst is responsible for the development and proposal of product strategies.
• Product Management Operations: The Product Analyst also has the obligation to respond in a timely manner to all requests and inquiries for product information or changes. He also performs the initial product analysis in order to assess the need for any requested changes as well as their potential impact.
• At this capacity, the Product Analyst also undertakes financial modeling on the products or services of the business as well as of the target markets in order to bring about an understanding of the relations between the product and the target market. This information is presented to the Marketing Manager and other stakeholders, when necessary.
• Additionally, the Product Analyst produces reports and makes recommendations to the Product Manager and Product Marketing Manager to be used as guidance in decision-making pertaining to the business’s new as well as existent products.
• Initiative: In this capacity, the Product Analyst ensures that there is a good flow of communication between the Product team and other teams. The Product Analyst ensures this by actively participating in team meetings and keeping everyone up to date.
• Pricing and Development: The Product Analyst has the responsibility to monitor the market, competitor activities, as well as any price movements and make recommendations that will be used in key decision making. In this function, the Product Analyst will normally liaise with other departments such as the credit/risk in the business in order to enhance and increase the efficiency of effecting price changes in accordance with market shifts.
• Customer/Market Intelligence: The Product Analyst has the obligation to drive consumer intelligence through the development of external and internal data sources that improve the business’s understanding of the product’s market, competitor activities, and consumer activities.
• In the performance of this role, the Product Analyst develops or adopts research tools, sources, and methods that further support and contribute to the business’s product.

1. Data Steward :

Data Steward will collaborate and work closely within the group software engineering and business division. Data Steward has overall accountability for the group's / Divisions overall data and reporting posture by responsibly managing data assets, data lineage, and data access, supporting sound data analysis. This role requires focus on data strategy, execution, and support for projects, programs, application enhancements, and production data fixes. Makes well-thought-out decisions on complex or ambiguous data issues and establishes the data stewardship and information management strategy and direction for the group. Effectively communicates to individuals at various levels of the technical and business communities. This individual will become part of the corporate Data Quality and Data management/entity resolution team supporting various systems across the board.

Primary Responsibilities:

• Responsible for data quality and data accuracy across all group/division delivery initiatives.
• Responsible for data analysis, data profiling, data modeling, and data mapping capabilities.
• Responsible for reviewing and governing data queries and DML.
• Accountable for the assessment, delivery, quality, accuracy, and tracking of any production data fixes.
• Accountable for the performance, quality, and alignment to requirements for all data query design and development.
• Responsible for defining standards and best practices for data analysis, modeling, and queries.
• Responsible for understanding end-to-end data flows and identifying data dependencies in support of delivery, release, and change management.
• Responsible for the development and maintenance of an enterprise data dictionary that is aligned to data assets and the business glossary for the group responsible for the definition and maintenance of the group's data landscape including overlays with the technology landscape, end-to-end data flow/transformations, and data lineage.
• Responsible for rationalizing the group's reporting posture through the definition and maintenance of a reporting strategy and roadmap.
• Partners with the data governance team to ensure data solutions adhere to the organization’s data principles and guidelines.
• Owns group's data assets including reports, data warehouse, etc.
• Understand customer business use cases and be able to translate them to technical specifications and vision on how to implement a solution.
• Accountable for defining the performance tuning needs for all group data assets and managing the implementation of those requirements within the context of group initiatives as well as steady-state production.
• Partners with others in test data management and masking strategies and the creation of a reusable test data repository.
• Responsible for solving data-related issues and communicating resolutions with other solution domains.
• Actively and consistently support all efforts to simplify and enhance the Clinical Trial Predication use cases.
• Apply knowledge in analytic and statistical algorithms to help customers explore methods to improve their business.
• Contribute toward analytical research projects through all stages including concept formulation, determination of appropriate statistical methodology, data manipulation, research evaluation, and final research report.
• Visualize and report data findings creatively in a variety of visual formats that appropriately provide insight to the stakeholders.
• Achieve defined project goals within customer deadlines; proactively communicate status and escalate issues as needed.

Additional Responsibilities:

• Strong understanding of the Software Development Life Cycle (SDLC) with Agile Methodologies
• Knowledge and understanding of industry-standard/best practices requirements gathering methodologies.
• Knowledge and understanding of Information Technology systems and software development.
• Experience with data modeling and test data management tools.
• Experience in the data integration project • Good problem solving & decision-making skills.
• Good communication skills within the team, site, and with the customer

Knowledge, Skills and Abilities

• Technical expertise in data architecture principles and design aspects of various DBMS and reporting concepts.
• Solid understanding of key DBMS platforms like SQL Server, Azure SQL
• Results-oriented, diligent, and works with a sense of urgency. Assertive, responsible for his/her own work (self-directed), have a strong affinity for defining work in deliverables, and be willing to commit to deadlines.
• Experience in MDM tools like MS DQ, SAS DM Studio, Tamr, Profisee, Reltio etc.
• Experience in Report and Dashboard development
• Statistical and Machine Learning models
• Python (sklearn, numpy, pandas, genism)
• Nice to Have:
• 1yr of ETL experience
• Natural Language Processing
• Neural networks and Deep learning
• xperience in keras,tensorflow,spacy, nltk, LightGBM python library

Interaction :  Frequently interacts with subordinate supervisors.

Education : Bachelor’s degree, preferably in Computer Science, B.E or other quantitative field related to the area of assignment. Professional certification related to the area of assignment may be required

Experience :  7 years of Pharmaceutical /Biotech/life sciences experience, 5 years of Clinical Trials experience and knowledge, Excellent Documentation, Communication, and Presentation Skills including PowerPoint

Job Title: Power BI Developer(Onsite)

Location: Park Centra, Sec 30, Gurgaon

CTC:        8 LPA

Time:       1:00 PM - 10:00 PM

Must Have Skills: 

• Power BI Desktop Software
• Dax Queries
• Data modeling
• Row-level security
• Visualizations
• Data Transformations and filtering
• SSAS and SQL

Job description:

We are looking for a PBI Analytics Lead responsible for efficient Data Visualization/ DAX Queries and Data Modeling. The candidate will work on creating complex Power BI reports. He will be involved in creating complex M, Dax Queries and working on data modeling, Row-level security, Visualizations, Data Transformations, and filtering. He will be closely working with the client team to provide solutions and suggestions on Power BI.

Roles and Responsibilities:

• Accurate, intuitive, and aesthetic Visual Display of Quantitative Information: We generate data, information, and insights through our business, product, brand, research, and talent teams. You would assist in transforming this data into visualizations that represent easy-to-consume visual summaries, dashboards and storyboards. Every graph tells a story.
• Understanding Data: You would be performing and documenting data analysis, data validation, and data mapping/design. You would be mining large datasets to determine its characteristics and select appropriate visualizations.
• Project Owner: You would develop, maintain, and manage advanced reporting, analytics, dashboards and other BI solutions, and would be continuously reviewing and improving existing systems and collaborating with teams to integrate new systems. You would also contribute to the overall data analytics strategy by knowledge sharing and mentoring end users.
• Perform ongoing maintenance & production of Management dashboards, data flows, and automated reporting.
• Manage upstream and downstream impact of all changes on automated reporting/dashboards
• Independently apply problem-solving ability to identify meaningful insights to business
• Identify automation opportunities and work with a wide range of stakeholders to implement the same.
• The ability and self-confidence to work independently and increase the scope of the service line

Requirements: 

• 3+ years of work experience as an Analytics Lead / Senior Analyst / Sr. PBI Developer.
• Sound understanding and knowledge of PBI Visualization and Data Modeling with DAX queries
• Experience in leading and mentoring a small team.

Data Platform engineering at Uber is looking for a strong Technical Lead (Level 5a Engineer) who has built high quality platforms and services that can operate at scale. 5a Engineer at Uber exhibits following qualities: 

• Demonstrate tech expertise › Demonstrate technical skills to go very deep or broad in solving classes of problems or creating broadly leverageable solutions. 
• Execute large scale projects › Define, plan and execute complex and impactful projects. You communicate the vision to peers and stakeholders.
• Collaborate across teams › Domain resource to engineers outside your team and help them leverage the right solutions. Facilitate technical discussions and drive to a consensus.
• Coach engineers › Coach and mentor less experienced engineers and deeply invest in their learning and success. You give and solicit feedback, both positive and negative, to others you work with to help improve the entire team.
• Tech leadership › Lead the effort to define the best practices in your immediate team, and help the broader organization establish better technical or business processes.

What You’ll Do

• Build a scalable, reliable, operable and performant data analytics platform for Uber’s engineers, data scientists, products and operations teams.
• Work alongside the pioneers of big data systems such as Hive, Yarn, Spark, Presto, Kafka, Flink to build out a highly reliable, performant, easy to use software system for Uber’s planet scale of data. 
• Become proficient of multi-tenancy, resource isolation, abuse prevention, self-serve debuggability aspects of a high performant, large scale, service while building these capabilities for Uber's engineers and operation folks.

What You’ll Need

• 7+ years experience in building large scale products, data platforms, distributed systems in a high caliber environment.
• Architecture: Identify and solve major architectural problems by going deep in your field or broad across different teams. Extend, improve, or, when needed, build solutions to address architectural gaps or technical debt.
• Software Engineering/Programming: Create frameworks and abstractions that are reliable and reusable. advanced knowledge of at least one programming language, and are happy to learn more. Our core languages are Java, Python, Go, and Scala.
• Data Engineering: Expertise in one of the big data analytics technologies we currently use such as Apache Hadoop (HDFS and YARN), Apache Hive, Impala, Drill, Spark, Tez, Presto, Calcite, Parquet, Arrow etc. Under the hood experience with similar systems such as Vertica, Apache Impala, Drill, Google Borg, Google BigQuery, Amazon EMR, Amazon RedShift, Docker, Kubernetes, Mesos etc.
• Execution & Results: You tackle large technical projects/problems that are not clearly defined. You anticipate roadblocks and have strategies to de-risk timelines. You orchestrate work that spans multiple teams and keep your stakeholders informed.
• A team player: You believe that you can achieve more on a team that the whole is greater than the sum of its parts. You rely on others’ candid feedback for continuous improvement.
• Business acumen: You understand requirements beyond the written word. Whether you’re working on an API used by other developers, an internal tool consumed by our operation teams, or a feature used by millions of customers, your attention to details leads to a delightful user experience.

REQUIREMENT:

•  Previous experience of working in large scale data engineering
•  4+ years of experience working in data engineering and/or backend technologies with cloud experience (any) is mandatory.
•  Previous experience of architecting and designing backend for large scale data processing.
•  Familiarity and experience of working in different technologies related to data engineering – different database technologies, Hadoop, spark, storm, hive etc.
•  Hands-on and have the ability to contribute a key portion of data engineering backend.
•  Self-inspired and motivated to drive for exceptional results.
•  Familiarity and experience working with different stages of data engineering – data acquisition, data refining, large scale data processing, efficient data storage for business analysis.
•  Familiarity and experience working with different DB technologies and how to scale them.

RESPONSIBILITY:

•  End to end responsibility to come up with data engineering architecture, design, development and then implementation of it.
•  Build data engineering workflow for large scale data processing.
•  Discover opportunities in data acquisition.
•  Bring industry best practices for data engineering workflow.
•  Develop data set processes for data modelling, mining and production.
•  Take additional tech responsibilities for driving an initiative to completion
•  Recommend ways to improve data reliability, efficiency and quality
•  Goes out of their way to reduce complexity.
•  Humble and outgoing - engineering cheerleaders.