GCP Data Engineer

RESPONSIBILITIES:

 Requirement understanding and elicitation, analyze, data/workflows, contribute to product

project and Proof of concept (POC)

 Contribute to prepare design documents and effort estimations.

 Develop AI/ML Models using best in-class ML models.

 Building, testing, and deploying AI/ML solutions.

 Work with Business Analysts and Product Managers to assist with defining functional user

stories.

 Ensure deliverables across teams are of high quality and clearly documented. 

 Recommend best ML practices/Industry standards for any ML use case.

 Proactively take up R and D and recommend solution options for any ML use case.

REQUIREMENTS:

Required Skills

 Overall experience of 4 to 7 Years working on AI/ML framework development

 Good programming knowledge in Python is must.

 Good Knowledge of R and SAS is desired.

 Good hands on and working knowledge SQL, Data Model, CRISP-DM.

 Proficiency with Uni/multivariate statistics, algorithm design, and predictive AI/ML modelling.

 Strong knowledge of machine learning algorithms, linear regression, logistic regression, KNN,

Random Forest, Support Vector Machines and Natural Language Processing.

 Experience with NLP and deep neural networks using synthetic and artificial data.

 Involved in different phases of SDLC and have good working exposure on different SLDC’s like

Agile Methodologies.

• Your responsibilities:
• Build, improve and extend NLP capabilities
• Research and evaluate different approaches to NLP problems
• Must be able to write code that is well designed, produce deliverable results
• Write code that scales and can be deployed to production

• Fundamentals of statistical methods is a must
• Experience in named entity recognition, POS Tagging, Lemmatization, vector representations of textual data and neural networks - RNN, LSTM
• A solid foundation in Python, data structures, algorithms, and general software development skills.
• Ability to apply machine learning to problems that deal with language
• Engineering ability to build robustly scalable pipelines
• Ability to work in a multi-disciplinary team with a strong product focus

Title: Platform Engineer Location: Chennai Work Mode: Hybrid (Remote and Chennai Office) Experience: 4+ years Budget: 16 - 18 LPA

Responsibilities:

• Parse data using Python, create dashboards in Tableau.
• Utilize Jenkins for Airflow pipeline creation and CI/CD maintenance.
• Migrate Datastage jobs to Snowflake, optimize performance.
• Work with HDFS, Hive, Kafka, and basic Spark.
• Develop Python scripts for data parsing, quality checks, and visualization.
• Conduct unit testing and web application testing.
• Implement Apache Airflow and handle production migration.
• Apply data warehousing techniques for data cleansing and dimension modeling.

Requirements:

• 4+ years of experience as a Platform Engineer.
• Strong Python skills, knowledge of Tableau.
• Experience with Jenkins, Snowflake, HDFS, Hive, and Kafka.
• Proficient in Unix Shell Scripting and SQL.
• Familiarity with ETL tools like DataStage and DMExpress.
• Understanding of Apache Airflow.
• Strong problem-solving and communication skills.

Note: Only candidates willing to work in Chennai and available for immediate joining will be considered. Budget for this position is 16 - 18 LPA.

A proficient, independent contributor that assists in technical design, development, implementation, and support of data pipelines; beginning to invest in less-experienced engineers.

Responsibilities:

- Design, Create and maintain on premise and cloud based data integration pipelines. 
- Assemble large, complex data sets that meet functional/non functional business requirements.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources.
- Build analytics tools that utilize the data pipeline to provide actionable insights into key business performance metrics.
- Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
- Create data pipelines to enable BI, Analytics and Data Science teams that assist them in building and optimizing their systems
- Assists in the onboarding, training and development of team members.
- Reviews code changes and pull requests for standardization and best practices
- Evolve existing development to be automated, scalable, resilient, self-serve platforms
- Assist the team in the design and requirements gathering for technical and non technical work to drive the direction of projects

Technical & Business Expertise:

-Hands on integration experience in SSIS/Mulesoft
- Hands on experience Azure Synapse
- Proven advanced level of writing database experience in SQL Server
- Proven advanced level of understanding about Data Lake
- Proven intermediate level of writing Python or similar programming language
- Intermediate understanding of Cloud Platforms (GCP) 
- Intermediate understanding of Data Warehousing
- Advanced Understanding of Source Control (Github)

We are establishing infrastructure for internal and external reporting using Tableau and are looking for someone with experience building visualizations and dashboards in Tableau and using Tableau Server to deliver them to internal and external users. 

Required Experience 

• Implementation of interactive visualizations using Tableau Desktop  
• Integration with Tableau Server and support of production dashboards and embedded reports with it 

• Writing and optimization of SQL queries  
• Proficient in Python including the use of Pandas and numpy libraries to perform data exploration and analysis 
• 3  years of experience working as a Software Engineer / Senior Software Engineer 
• Bachelors in Engineering – can be Electronic and comm , Computer , IT  
• Well versed with Basic Data Structures Algorithms and system design 

• Should be capable of working well in a team – and should possess very good communication skills 
• Self-motivated and fun to work with and organized 
• Productive and efficient working remotely 
• Test driven mindset with a knack for finding issues and problems at earlier stages of development 
• Interest in learning and picking up a wide range of cutting edge technologies 

• Should be curious and interested in learning some Data science related concepts and domain knowledge 
• Work alongside other engineers on the team to elevate technology and consistently apply best practices 

Highly Desirable 

• Data Analytics 

• Experience in AWS cloud or any cloud technologies 
• Experience in BigData technologies and streaming like – pyspark, kafka is a big plus 
• Shell scripting  
• Preferred tech stack – Python, Rest API, Microservices, Flask/Fast API, pandas, numpy, linux, shell scripting, Airflow, pyspark 
• Has a strong backend experience – and worked with Microservices and Rest API’s - Flask, FastAPI, Databases Relational and Non-relational 

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

Must Have Skills

• Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
• Leading in the identification, isolation, resolution and communication of problems within the production environment.
• Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
• Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
• Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
• Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
• Works on multiple platforms and multiple projects concurrently.
• Performs code and unit testing for complex scope modules, and projects
• Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)

• Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
• Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
• Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
• Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
• Working knowledge on Kafka Rest proxy.
• Ensure optimum performance, high availability and stability of solutions.
• Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
• Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
• Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
• Ability to perform data related benchmarking, performance analysis and tuning.
• Strong skills in In-memory applications, Database Design, Data Integration.

• Create and maintain optimal data pipeline architecture,
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field. They should also have experience using the following software/tools:

• Experience with big data tools: Hadoop, Spark, Kafka, etc.
• Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
• Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift
• Experience with stream-processing systems: Storm, Spark-Streaming, etc.
• Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.

•  We are looking for a Data Engineer with 3-5 years experience in Python, SQL, AWS (EC2, S3, Elastic Beanstalk, API Gateway), and Java.
• The applicant must be able to perform Data Mapping (data type conversion, schema harmonization) using Python, SQL, and Java.
• The applicant must be familiar with and have programmed ETL interfaces (OAUTH, REST API, ODBC) using the same languages.
• The company is looking for someone who shows an eagerness to learn and who asks concise questions when communicating with teammates.