Data Engineer

• Minimum 1 years of relevant experience, in PySpark (mandatory)
• Hands on experience in development, test, deploy, maintain and improving data integration pipeline in AWS cloud environment is added plus 
• Ability to play lead role and independently manage 3-5 member of Pyspark development team 
• EMR ,Python and PYspark mandate.
• Knowledge and awareness working with AWS Cloud technologies like Apache Spark, , Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS

Lifesight powers real-time business communications and data solutions that help companies worldwide build better applications and customer experiences by our leading customer intelligence and data platforms that helps brands and enterprises leverage identity resolution and data enrichment to power their customer data strategies like never before. Our industry-leading solution enables the transformation of customer data into actionable insights that help drive business decisions, optimize marketing spend, and improve customer experiences.

About The Job

Lifesight is growing rapidly and seeking a strong Data Engineer to be a key member of the Enterprise Data & Business Intelligence organization with the focus on data engineering services based in Bengaluru, India. You will be joining as one of the few first data engineers in our Bengaluru office and as the early engineers in the data platform. You will have an opportunity to help define our technical strategy and data engineering team culture in India. 

You will design and build data platforms and services, while managing our data infrastructure in cloud environments that fuels strategic business decisions across Lifesight products. 

A successful candidate will be a self-starter, who drives excellence, is ready to jump into a variety of big data technologies & frameworks and is able to coordinate and collaborate with other engineers, as well as mentor other engineers in the team.

Why Join us 

• Be part of building a Zero to One SaaS product
• Discover amazing career opportunities being initial core team member driving tech strategy of Lifesight.
• We have a growth plan for everyone
• Rise to rewarding challenges, enjoy attractive incentives, competitive compensation and more ...
• Develop your skills - you will enabled with [email protected] program to help you with limitless resources for your learning
•  

Key Responsibilities

• Design and implement data management services for data trust, data compliance, data access and metadata management in the form of scalable and configurable while clearly articulating technical rationale behind your design and implementation choices
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Implementing ELT processes
• Monitoring performance and advising any necessary infrastructure changes
• Build data platforms and services to handle both real-time and batch processing of large data sets.
• Take end-to-end ownership of the design and implementation of core product functionality and key technical initiatives.

Qualifications

• 2+ years of data engineering experience is a fast paced company that delivers software
• A deep understanding of designing and building highly scalable Data warehouses and Data pipelines.
• Hands on Experience in SQL, Java/Scala programming languages.
• Deep understanding of Apache Spark, Spark tuning , creating RDDs and building data frames. Create Java/ Scala Spark jobs for data transformation and aggregation.
• Experience in building distributed environments using any of Kafka, Spark, Hive, Hadoop etc.
• Good understanding of architecture and functioning of Distributed database systems 
• Experience working with various file formats like Parquet, Avro etc for large volumes of data
• Experience with one or more NoSQL databases
• Strong understanding of engineering best practices and design principles

Location

This role will be based in our Bengaluru, India office.

WHAT WE CARE ABOUT

We like to solve problems, take initiative, pitch in when needed, and are always up for trying new things. What and how you can contribute is what’s important to us. Our consideration is not limited by the kind of education you have or the specific technologies you have experience with. Variety of technical challenges is one of the best things about working @Lifesight as an engineer, but we do not expect you to know every technology we use when you start. What we care about is that you can learn quickly, efficiently manage your goals, contribute to product strategy, and help develop your team as they solve complex problems using the best tools for the job.

Our culture runs much deeper than just having fun together (though, we do that well too...) – the people we want on our team are trust-builders, generous givers, scrappy problem solvers, and gritty pursuers of excellence.

Does this sound like you? If so, we welcome your application and the chance to meet you.

We are looking for an exceptional Data Scientist who is passionate about data and motivated to build large scale machine learning solutions. This person will be contributing to the analytics of data for insight discovery and development of machine learning pipeline to support modelling of terabytes of daily data for various use cases

Typical persona: Data Science Manager / Architect

Experience: 8+ years programming/engineering experience (with at least last 4 years in big data, Data science)

Must have:

• Hands-on Python: Pandas, Scikit-Learn
• Working knowledge of Kafka
• Able to carry out own tasks and help the team in resolving problems - logical or technical (25% of job)
• Good on analytical & debugging skills
• Strong communication skills

Desired (in order of priorities):

• Go (Strong advantage)
• Airflow (Strong advantage)
• Familiarity & working experience on more than one type of database: relational, object, columnar, graph and other unstructured databases
• Data structures, Algorithms
• Experience with multi-threaded and thread sync concepts
• AWS Sagemaker
• Keras

• Should have strong experience in Python programming minimum 4 Years

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

Must Have Skills

• Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
• Leading in the identification, isolation, resolution and communication of problems within the production environment.
• Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
• Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
• Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
• Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
• Works on multiple platforms and multiple projects concurrently.
• Performs code and unit testing for complex scope modules, and projects
• Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)

• Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
• Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
• Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
• Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
• Working knowledge on Kafka Rest proxy.
• Ensure optimum performance, high availability and stability of solutions.
• Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
• Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
• Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
• Ability to perform data related benchmarking, performance analysis and tuning.
• Strong skills in In-memory applications, Database Design, Data Integration.

Responsibilities:

• Exploring and visualizing data to gain an understanding of it, then identifying differences in data distribution that could affect performance when deploying the model in the real world.
• Verifying data quality, and/or ensuring it via data cleaning.
• Able to adapt and work fast in producing the output which upgrades the decision making of stakeholders using ML.
• To design and develop Machine Learning systems and schemes. 
• To perform statistical analysis and fine-tune models using test results.
• To train and retrain ML systems and models as and when necessary. 
• To deploy ML models in production and maintain the cost of cloud infrastructure.
• To develop Machine Learning apps according to client and data scientist requirements.
• To analyze the problem-solving capabilities and use-cases of ML algorithms and rank them by how successful they are in meeting the objective.

Technical Knowledge:

• Worked with real time problems, solved them using ML and deep learning models deployed in real time and should have some awesome projects under his belt to showcase. 
• Proficiency in Python and experience with working with Jupyter Framework, Google collab and cloud hosted notebooks such as AWS sagemaker, DataBricks etc.
• Proficiency in working with libraries Sklearn, Tensorflow, Open CV2, Pyspark,  Pandas, Numpy and related libraries.
• Expert in visualising and manipulating complex datasets.
• Proficiency in working with visualisation libraries such as seaborn, plotly, matplotlib etc.
• Proficiency in Linear Algebra, statistics and probability required for Machine Learning.
• Proficiency in ML Based algorithms for example, Gradient boosting, stacked Machine learning, classification algorithms and deep learning algorithms. Need to have experience in hypertuning various models and comparing the results of algorithm performance.
• Big data Technologies such as Hadoop stack and Spark. 
• Basic use of clouds (VM’s example EC2).
• Brownie points for Kubernetes and Task Queues.      
• Strong written and verbal communications.
• Experience working in an Agile environment.