AWS Data Engineer

Job Description: 

Machine Learning / AI Engineer (with 3+ years of experience)

We are seeking a highly skilled and passionate Machine Learning / AI Engineer to join our newly established data science practice area. In this role, you will primarily focus on working with Large Language Models (LLMs) and contribute to building generative AI applications. This position offers an exciting opportunity to shape the future of AI technology while charting an interesting career path within our organization.

Responsibilities:

1. Develop and implement machine learning models: Utilize your expertise in machine learning and artificial intelligence to design, develop, and deploy cutting-edge models, with a particular emphasis on Large Language Models (LLMs). Apply your knowledge to solve complex problems and optimize performance.

2. Building generative AI applications: Collaborate with cross-functional teams to conceptualize, design, and build innovative generative AI applications. Work on projects that push the boundaries of AI technology and deliver impactful solutions to real-world problems.

3. Data preprocessing and analysis: Collect, clean, and preprocess large volumes of data for training and evaluation purposes. Conduct exploratory data analysis to gain insights and identify patterns that can enhance the performance of AI models.

4. Model training and evaluation: Develop robust training pipelines for machine learning models, incorporating best practices in model selection, feature engineering, and hyperparameter tuning. Evaluate model performance using appropriate metrics and iterate on the models to improve accuracy and efficiency.

5. Research and stay up to date: Keep abreast of the latest advancements in machine learning, natural language processing, and generative AI. Stay informed about industry trends, emerging techniques, and open-source libraries, and apply relevant findings to enhance the team's capabilities.

6. Collaborate and communicate effectively: Work closely with a multidisciplinary team of data scientists, software engineers, and domain experts to drive AI initiatives. Clearly communicate complex technical concepts and findings to both technical and non-technical stakeholders.

7. Experimentation and prototyping: Explore novel ideas, experiment with new algorithms, and prototype innovative solutions. Foster a culture of innovation and contribute to the continuous improvement of AI methodologies and practices within the organization.

Requirements:

1. Education: Bachelor's or Master's degree in Computer Science, Data Science, or a related field. Relevant certifications in machine learning, deep learning, or AI are a plus.

2. Experience: A minimum of 3+ years of professional experience as a Machine Learning / AI Engineer, with a proven track record of developing and deploying machine learning models in real-world applications.

3. Strong programming skills: Proficiency in Python and experience with machine learning frameworks (e.g., TensorFlow, PyTorch) and libraries (e.g., scikit-learn, pandas). Experience with cloud platforms (e.g., AWS, Azure, GCP) for model deployment is preferred.

4. Deep-learning expertise: Strong understanding of deep learning architectures (e.g., convolutional neural networks, recurrent neural networks, transformers) and familiarity with Large Language Models (LLMs) such as GPT-3, GPT-4, or equivalent.

5. Natural Language Processing (NLP) knowledge: Familiarity with NLP techniques, including tokenization, word embeddings, named entity recognition, sentiment analysis, text classification, and language generation.

6. Data manipulation and preprocessing skills: Proficiency in data manipulation using SQL and experience with data preprocessing techniques (e.g., cleaning, normalization, feature engineering). Familiarity with big data tools (e.g., Spark) is a plus.

7. Problem-solving and analytical thinking: Strong analytical and problem-solving abilities, with a keen eye for detail. Demonstrated experience in translating complex business requirements into practical machine learning solutions.

8. Communication and collaboration: Excellent verbal and written communication skills, with the ability to explain complex technical concepts to diverse stakeholders

Data Engineer 

Mandatory Requirements  

• Experience in AWS Glue 

• Experience in Apache Parquet  
• Proficient in AWS S3 and data lake  
• Knowledge of Snowflake 
• Understanding of file-based ingestion best practices. 
• Scripting language - Python & pyspark 

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS  
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies  
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform  

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations  
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data. 
• Define process improvement opportunities to optimize data collection, insights and displays. 
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible  
• Identify and interpret trends and patterns from complex data sets  

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.  
• Key participant in regular Scrum ceremonies with the agile teams   
• Proficient at developing queries, writing reports and presenting findings  
• Mentor junior members and bring best industry practices  

QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)  
• Strong background in math, statistics, computer science, data science or related discipline 
• Advanced knowledge one of language: Java, Scala, Python, C#  
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake   

• Proficient with 
• Data mining/programming tools (e.g. SAS, SQL, R, Python) 
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum) 
• Data visualization (e.g. Tableau, Looker, MicroStrategy) 
• Comfortable learning about and deploying new technologies and tools.  

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.  
• Good written and oral communication skills and ability to present results to non-technical audiences  
• Knowledge of business intelligence and analytical tools, technologies and techniques. 

Familiarity and experience in the following is a plus:  

• AWS certification 
• Spark Streaming  
• Kafka Streaming / Kafka Connect  
• ELK Stack  
• Cassandra / MongoDB  

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools 

Role Description:

• You will be part of the data delivery team and will have the opportunity to develop a deep understanding of the domain/function.
• You will design and drive the work plan for the optimization/automation and standardization of the processes incorporating best practices to achieve efficiency gains.
• You will run data engineering pipelines, link raw client data with data model, conduct data assessment, perform data quality checks, and transform data using ETL tools.
• You will perform data transformations, modeling, and validation activities, as well as configure applications to the client context. You will also develop scripts to validate, transform, and load raw data using programming languages such as Python and / or PySpark.
• In this role, you will determine database structural requirements by analyzing client operations, applications, and programming.
• You will develop cross-site relationships to enhance idea generation, and manage stakeholders.
• Lastly, you will collaborate with the team to support ongoing business processes by delivering high-quality end products on-time and perform quality checks wherever required.

• Research and develop statistical learning models for data analysis
• Collaborate with product management and engineering departments to understand company needs and devise possible solutions
• Keep up-to-date with latest technology trends
• Communicate results and ideas to key decision makers
• Implement new statistical or other mathematical methodologies as needed for specific models or analysis
• Optimize joint development efforts through appropriate database use and project design

Qualifications/Requirements:

• Masters or PhD in Computer Science, Electrical Engineering, Statistics, Applied Math or equivalent fields with strong mathematical background
• Excellent understanding of machine learning techniques and algorithms, including clustering, anomaly detection, optimization, neural network etc
• 3+ years experiences building data science-driven solutions including data collection, feature selection, model training, post-deployment validation
• Strong hands-on coding skills (preferably in Python) processing large-scale data set and developing machine learning models
• Familiar with one or more machine learning or statistical modeling tools such as Numpy, ScikitLearn, MLlib, Tensorflow
• Good team worker with excellent communication skills written, verbal and presentation

Desired Experience:

• Experience with AWS, S3, Flink, Spark, Kafka, Elastic Search
• Knowledge and experience with NLP technology
• Previous work in a start-up environment

Position: Big Data Engineer

What You'll Do

Punchh is seeking to hire Big Data Engineer at either a senior or tech lead level. Reporting to the Director of Big Data, he/she will play a critical role in leading Punchh’s big data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners.

This role requires close collaborations with data, engineering, and product organizations. His/her job functions include

• Work with large data sets and implement sophisticated data pipelines with both structured and structured data.
• Collaborate with stakeholders to design scalable solutions.
• Manage and optimize our internal data pipeline that supports marketing, customer success and data science to name a few.
• A technical leader of Punchh’s big data platform that supports AI and BI products.
• Work with infra and operations team to monitor and optimize existing infrastructure 
• Occasional business travels are required.

What You'll Need

• 5+ years of experience as a Big Data engineering professional, developing scalable big data solutions.
• Advanced degree in computer science, engineering or other related fields.
• Demonstrated strength in data modeling, data warehousing and SQL.
• Extensive knowledge with cloud technologies, e.g. AWS and Azure.
• Excellent software engineering background. High familiarity with software development life cycle. Familiarity with GitHub/Airflow.
• Advanced knowledge of big data technologies, such as programming language (Python, Java), relational (Postgres, mysql), NoSQL (Mongodb), Hadoop (EMR) and streaming (Kafka, Spark).
• Strong problem solving skills with demonstrated rigor in building and maintaining a complex data pipeline.
• Exceptional communication skills and ability to articulate a complex concept with thoughtful, actionable recommendations.

• Design AWS data ingestion frameworks and pipelines based on the specific needs driven by the Product Owners and user stories…
• Experience building Data Lake using AWS and Hands-on experience in S3, EKS, ECS, AWS Glue, AWS KMS, AWS Firehose, EMR
• Experience Apache Spark Programming with Databricks
• Experience working on NoSQL Databases such as Cassandra, HBase, and Elastic Search
• Hands on experience with leveraging CI/CD to rapidly build & test application code
• Expertise in Data governance and Data Quality
• Experience working with PCI Data and working with data scientists is a plus
• At least 4+ years of experience in the following Big Data frameworks: File Format (Parquet, AVRO, ORC), Resource Management, Distributed Processing and RDBMS
• 5+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using AWS technologies

JD for IOT DE:

The role requires experience in Azure core technologies – IoT Hub/ Event Hub, Stream Analytics, IoT Central, Azure Data Lake Storage, Azure Cosmos, Azure Data Factory, Azure SQL Database, Azure HDInsight / Databricks, SQL data warehouse.

You Have:

• Minimum 2 years of software development experience
• Minimum 2 years of experience in IoT/streaming data pipelines solution development
• Bachelor's and/or Master’s degree in computer science
• Strong Consulting skills in data management including data governance, data quality, security, data integration, processing, and provisioning
• Delivered data management projects with real-time/near real-time data insights delivery on Azure Cloud
• Translated complex analytical requirements into the technical design including data models, ETLs, and Dashboards / Reports
• Experience deploying dashboards and self-service analytics solutions on both relational and non-relational databases
• Experience with different computing paradigms in databases such as In-Memory, Distributed, Massively Parallel Processing
• Successfully delivered large scale IOT data management initiatives covering Plan, Design, Build and Deploy phases leveraging different delivery methodologies including Agile
• Experience in handling telemetry data with Spark Streaming, Kafka, Flink, Scala, Pyspark, Spark SQL.
• Hands-on experience on containers and Dockers
• Exposure to streaming protocols like MQTT and AMQP
• Knowledge of OT network protocols like OPC UA, CAN Bus, and similar protocols
• Strong knowledge of continuous integration, static code analysis, and test-driven development
• Experience in delivering projects in a highly collaborative delivery model with teams at onsite and offshore
• Must have excellent analytical and problem-solving skills
• Delivered change management initiatives focused on driving data platforms adoption across the enterprise
• Strong verbal and written communications skills are a must, as well as the ability to work effectively across internal and external organizations
   

Roles & Responsibilities
 

You Will:

• Translate functional requirements into technical design
• Interact with clients and internal stakeholders to understand the data and platform requirements in detail and determine core Azure services needed to fulfill the technical design
• Design, Develop and Deliver data integration interfaces in ADF and Azure Databricks
• Design, Develop and Deliver data provisioning interfaces to fulfill consumption needs
• Deliver data models on Azure platform, it could be on Azure Cosmos, SQL DW / Synapse, or SQL
• Advise clients on ML Engineering and deploying ML Ops at Scale on AKS
• Automate core activities to minimize the delivery lead times and improve the overall quality
• Optimize platform cost by selecting the right platform services and architecting the solution in a cost-effective manner
• Deploy Azure DevOps and CI CD processes
• Deploy logging and monitoring across the different integration points for critical alerts