Big Data Spark Lead

Job Description: Data Engineer

We are looking for a curious Data Engineer to join our extremely fast-growing Tech Team at StanPlus

About RED.Health (Formerly Stanplus Technologies)

Get to know the team:

Join our team and help us build the world’s fastest and most reliable emergency response system using cutting-edge technology.

Because every second counts in an emergency, we are building systems and flows with 4 9s of reliability to ensure that our technology is always there when people need it the most. We are looking for distributed systems experts who can help us perfect the architecture behind our key design principles: scalability, reliability, programmability, and resiliency. Our system features a powerful dispatch engine that connects emergency service providers with patients in real-time

.

Key Responsibilities

●     Build Data ETL Pipelines

●     Develop data set processes

●     Strong analytic skills related to working with unstructured datasets

●     Evaluate business needs and objectives

●     Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery

●     Interpret trends and patterns

●     Work with data and analytics experts to strive for greater functionality in our data system

●     Build algorithms and prototypes

●     Explore ways to enhance data quality and reliability

●     Work with the Executive, Product, Data, and D   esign teams, to assist with data-related technical issues and support their data infrastructure needs.

●     Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.

Key Requirements

●     Proven experience as a data engineer, software developer, or similar of at least 3 years.

●     Bachelor's / Master’s degree in data engineering, big data analytics, computer engineering, or related field.

●     Experience with big data tools: Hadoop, Spark, Kafka, etc.

●     Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.

●     Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

●     Experience with Azure, AWS cloud services: EC2, EMR, RDS, Redshift

●     Experience with BigQuery

●     Experience with stream-processing systems: Storm, Spark-Streaming, etc.

●     Experience with languages: Python, Java, C++, Scala, SQL, R, etc.

●     Good hands-on with Hive, Presto.

Data Engineering : Senior Engineer / Manager

As Senior Engineer/ Manager in Data Engineering, you will translate client requirements into technical design, and implement components for a data engineering solutions. Utilize a deep understanding of data integration and big data design principles in creating custom solutions or implementing package solutions. You will independently drive design discussions to insure the necessary health of the overall solution.

Must Have skills :

1. GCP

2. Spark streaming : Live data streaming experience is desired.

3. Any 1 coding language: Java/Pyhton /Scala

Skills & Experience :

- Overall experience of MINIMUM 5+ years with Minimum 4 years of relevant experience in Big Data technologies

- Hands-on experience with the Hadoop stack - HDFS, sqoop, kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink, Storm, hive, oozie, airflow and other components required in building end to end data pipeline. Working knowledge on real-time data pipelines is added advantage.

- Strong experience in at least of the programming language Java, Scala, Python. Java preferable

- Hands-on working knowledge of NoSQL and MPP data platforms like Hbase, MongoDb, Cassandra, AWS Redshift, Azure SQLDW, GCP BigQuery etc.

- Well-versed and working knowledge with data platform related services on GCP

- Bachelor's degree and year of work experience of 6 to 12 years or any combination of education, training and/or experience that demonstrates the ability to perform the duties of the position

Your Impact :

- Data Ingestion, Integration and Transformation

- Data Storage and Computation Frameworks, Performance Optimizations

- Analytics & Visualizations

- Infrastructure & Cloud Computing

- Data Management Platforms

- Build functionality for data ingestion from multiple heterogeneous sources in batch & real-time

- Build functionality for data analytics, search and aggregation

• Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
• Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
• Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
• Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
• Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
• Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
• Analyse large amounts of information to discover trends and patterns
• Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\

• Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
• 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
• Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
• Good understanding of AWS ecosystem for big data analytics
• Hands on experience in creating data pipelines either using tools or by independently writing scripts
• Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
• Strong problem solving skills with an emphasis on product development.
• Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)

We are looking for a skilled Senior/Lead Bigdata Engineer to join our team. The role is part of the research and development team, where you with enthusiasm and knowledge are going to be our technical evangelist for the development of our inspection technology and products.

At Elop we are developing product lines for sustainable infrastructure management using our own patented technology for ultrasound scanners and combine this with other sources to see holistic overview of the concrete structure. At Elop we will provide you with world-class colleagues highly motivated to position the company as an international standard of structural health monitoring. With the right character you will be professionally challenged and developed.

This position requires travel to Norway.

Elop is sister company of Simplifai and co-located together in all geographic locations.

https://elop.no/

https://www.simplifai.ai/en/

Roles and Responsibilities

• Define technical scope and objectives through research and participation in requirements gathering and definition of processes
• Ingest and Process data from data sources (Elop Scanner) in raw format into Big Data ecosystem
• Realtime data feed processing using Big Data ecosystem
• Design, review, implement and optimize data transformation processes in Big Data ecosystem
• Test and prototype new data integration/processing tools, techniques and methodologies
• Conversion of MATLAB code into Python/C/C++.
• Participate in overall test planning for the application integrations, functional areas and projects.
• Work with cross functional teams in an Agile/Scrum environment to ensure a quality product is delivered.

Desired Candidate Profile

• Bachelor's degree in Statistics, Computer or equivalent
• 7+ years of experience in Big Data ecosystem, especially Spark, Kafka, Hadoop, HBase.
• 7+ years of hands-on experience in Python/Scala is a must.
• Experience in architecting the big data application is needed.
• Excellent analytical and problem solving skills
• Strong understanding of data analytics and data visualization, and must be able to help development team with visualization of data.
• Experience with signal processing is plus.
• Experience in working on client server architecture is plus.
• Knowledge about database technologies like RDBMS, Graph DB, Document DB, Apache Cassandra, OpenTSDB
• Good communication skills, written and oral, in English

We can Offer

• An everyday life with exciting and challenging tasks with the development of socially beneficial solutions
• Be a part of companys research and Development team to create unique and innovative products
• Colleagues with world-class expertise, and an organization that has ambitions and is highly motivated to position the company as an international player in maintenance support and monitoring of critical infrastructure!
• Good working environment with skilled and committed colleagues an organization with short decision paths.
• Professional challenges and development

• Hands-on experience in any Cloud Platform

• Microsoft Azure Experience

• Responsibilities
       - Responsible for implementation and ongoing administration of Hadoop
  infrastructure.
       - Aligning with the systems engineering team to propose and deploy new
  hardware and software environments required for Hadoop and to expand existing
  environments.
       - Working with data delivery teams to setup new Hadoop users. This job includes
  setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig
  and MapReduce access for the new users.
       - Cluster maintenance as well as creation and removal of nodes using tools like
  Ganglia, Nagios, Cloudera Manager Enterprise, Dell Open Manage and other tools
       - Performance tuning of Hadoop clusters and Hadoop MapReduce routines
       - Screen Hadoop cluster job performances and capacity planning
       - Monitor Hadoop cluster connectivity and security
       - Manage and review Hadoop log files.
       - File system management and monitoring.
       - Diligently teaming with the infrastructure, network, database, application and
  business intelligence teams to guarantee high data quality and availability
       - Collaboration with application teams to install operating system and Hadoop
  updates, patches, version upgrades when required.
      
  READ MORE OF THE JOB DESCRIPTION 
  Qualifications
  Qualifications
       - Bachelors Degree in Information Technology, Computer Science or other
  relevant fields
       - General operational expertise such as good troubleshooting skills,
  understanding of systems capacity, bottlenecks, basics of memory, CPU, OS,
  storage, and networks.
       - Hadoop skills like HBase, Hive, Pig, Mahout
       - Ability to deploy Hadoop cluster, add and remove nodes, keep track of jobs,
  monitor critical parts of the cluster, configure name node high availability, schedule
  and configure it and take backups.
       - Good knowledge of Linux as Hadoop runs on Linux.
       - Familiarity with open source configuration management and deployment tools
  such as Puppet or Chef and Linux scripting.
       Nice to Have
       - Knowledge of Troubleshooting Core Java Applications is a plus.