Senior Data Engineer - Data Bricks (5+ years exp)

As a part of WWS, your expertise in machine learning helps us extract value from our data. You will lead all the processes from data collection, cleaning, and preprocessing, to training models and situating them to production. The ideal candidate will be passionate about artificial intelligence and stay up-to-date with the latest developments in the field.

What We Expect

• A Bachelor's/Master's degree in IT, computer science, or an advanced related field is preferred.

• At least 3+ years of experience working with ML libraries and packages.

• Familiarity with coding and programming languages, including Python, Java, C++, and SAS.

• Strong experience in programming and statistics.

• Well-versed in Data Science and neural schematics in networking and software.

• Flexibility in shifts is appreciated.

A Full Stack Developer’s Ideal Day At WWS

Design and Develop. The primary protocols include implementing machine learning algorithms and running AI systems experiments and tests. The Designing and development of machine learning systems along with performing statistical analyses falls under day to day activities of the developer.

Algorithm Assertion. The engineers act as critical members of the data science team as their tasks involve researching, asserting, and designing the artificial intelligence responsible for machine learning and maintaining and improving existing artificial intelligence systems.

Research and Development. To analyze large, complex datasets and extract insights and decide on the appropriate technique. research and implement best practices to improve the existing machine learning infrastructure. At most providing support to engineers and product managers in implementing machine learning as the product.

What You Can Expect

• Full-time, salaried positions creamed with welfare programs.

• Competitive salary and tailored training in the core space with recognition potential and annual bonus.

• Periodic performance appraisals.

• Attendance Incentives.

• Working with the best and budding talent in the industry.

• A conducive intangible environment with dynamic benefits.

Why Consider Machine Learning Engineer as a career with WWS?

With a very appealing work environment at WWS, our setting made it easier to build relationships with other staff members and clients. You may also have an opportunity to learn other aspects of environmental office work on the job, which can enhance your experience and qualifications.

Many businesses must proactively react to changing factors — like patterns of customer behavior or prices. Tracking model performance and retraining it once fresher data is available is key to success. This falls under the MLE range of responsibilities for which the requirement has been crucial for many organizations.

Please attach your resume and let us know through email your current address, phone number, and the best time to contact you by phone.

                                          Apply To this Job

Responsibilities

Researches, develops and maintains machine learning and statistical models for

business requirements

Work across the spectrum of statistical modelling including supervised,

unsupervised, & deep learning techniques to apply the right level of solution to

the right problem Coordinate with different functional teams to monitor outcomes and refine/

improve the machine learning models Implements models to uncover patterns and predictions creating business value and innovation

Identify unexplored data opportunities for the business to unlock and maximize

the potential of digital data within the organization

Develop NLP concepts and algorithms to classify and summarize structured/unstructured text data

Qualifications

3+ years of experience solving complex business problems using machine

learning.

Fluency in programming languages such as Python, NLP and Bert, is a must

Strong analytical and critical thinking skills

Experience in building production quality models using state-of-the-art technologies 

Familiarity with databases .

desirable Ability to collaborate on projects and work independently when required.

Previous experience in Fintech/payments domain is a bonus

You should have Bachelor’s or Master’s degree in Computer Science, Statistics

or Mathematics or another quantitative field from a top tier Institute

Job Description

Mandatory Requirements 

• Experience in AWS Glue

• Experience in Apache Parquet 

• Proficient in AWS S3 and data lake 

• Knowledge of Snowflake

• Understanding of file-based ingestion best practices.

• Scripting language - Python & pyspark

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 

• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 

• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 

• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.

• Define process improvement opportunities to optimize data collection, insights and displays.

• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 

• Identify and interpret trends and patterns from complex data sets 

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 

• Key participant in regular Scrum ceremonies with the agile teams  

• Proficient at developing queries, writing reports and presenting findings 

• Mentor junior members and bring best industry practices.

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 

• Strong background in math, statistics, computer science, data science or related discipline

• Advanced knowledge one of language: Java, Scala, Python, C# 

• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  

• Proficient with

• Data mining/programming tools (e.g. SAS, SQL, R, Python)

• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)

• Data visualization (e.g. Tableau, Looker, MicroStrategy)

• Comfortable learning about and deploying new technologies and tools. 

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 

• Good written and oral communication skills and ability to present results to non-technical audiences 

• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus: 

• AWS certification

• Spark Streaming 

• Kafka Streaming / Kafka Connect 

• ELK Stack 

• Cassandra / MongoDB 

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

1. Data Engineer

 Required skill set: AWS GLUE, AWS LAMBDA, AWS SNS/SQS, AWS ATHENA, SPARK, SNOWFLAKE, PYTHON

Mandatory Requirements  

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

 QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Familiarity and experience in the following is a plus:  

• AWS certification
• Spark Streaming 
• Kafka Streaming / Kafka Connect 
• ELK Stack 
• Cassandra / MongoDB 
• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools

Roles and Responsibilities:

• Design, develop, and maintain the end-to-end MLOps infrastructure from the ground up, leveraging open-source systems across the entire MLOps landscape.
• Creating pipelines for data ingestion, data transformation, building, testing, and deploying machine learning models, as well as monitoring and maintaining the performance of these models in production.
• Managing the MLOps stack, including version control systems, continuous integration and deployment tools, containerization, orchestration, and monitoring systems.
• Ensure that the MLOps stack is scalable, reliable, and secure.

Skills Required:

• 3-6 years of MLOps experience
• Preferably worked in the startup ecosystem

Primary Skills:

• Experience with E2E MLOps systems like ClearML, Kubeflow, MLFlow etc.
• Technical expertise in MLOps: Should have a deep understanding of the MLOps landscape and be able to leverage open-source systems to build scalable, reliable, and secure MLOps infrastructure.
• Programming skills: Proficient in at least one programming language, such as Python, and have experience with data science libraries, such as TensorFlow, PyTorch, or Scikit-learn.
• DevOps experience: Should have experience with DevOps tools and practices, such as Git, Docker, Kubernetes, and Jenkins.

Secondary Skills:

• Version Control Systems (VCS) tools like Git and Subversion
• Containerization technologies like Docker and Kubernetes
• Cloud Platforms like AWS, Azure, and Google Cloud Platform
• Data Preparation and Management tools like Apache Spark, Apache Hadoop, and SQL databases like PostgreSQL and MySQL
• Machine Learning Frameworks like TensorFlow, PyTorch, and Scikit-learn
• Monitoring and Logging tools like Prometheus, Grafana, and Elasticsearch
• Continuous Integration and Continuous Deployment (CI/CD) tools like Jenkins, GitLab CI, and CircleCI
• Explain ability and Interpretability tools like LIME and SHAP

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

• Drive and participate in requirements gathering workshops, estimation discussions, design meetings and status review meetings
• Participate and contribute in Solution Design and Solution Architecture for implementing Big Data Projects on-premise and on cloud
• Technical Hands on experience in design, coding, development and managing Large Hadoop implementation
• Proficient in SQL, Hive, PIG, Spark SQL, Shell Scripting, Kafka, Flume, Scoop with large Big Data and Data Warehousing projects with either Java, Python or Scala based Hadoop programming background
• Proficient with various development methodologies like waterfall, agile/scrum and iterative
• Good Interpersonal skills and excellent communication skills for US and UK based clients