Spark Scala Developer

Requirements:

● Understanding our data sets and how to bring them together.

● Working with our engineering team to support custom solutions offered to the product development.

● Filling the gap between development, engineering and data ops.

● Creating, maintaining and documenting scripts to support ongoing custom solutions.

● Excellent organizational skills, including attention to precise details

● Strong multitasking skills and ability to work in a fast-paced environment

● 5+ years experience with Python to develop scripts.

● Know your way around RESTFUL APIs.[Able to integrate not necessary to publish]

● You are familiar with pulling and pushing files from SFTP and AWS S3.

● Experience with any Cloud solutions including GCP / AWS / OCI / Azure.

● Familiarity with SQL programming to query and transform data from relational Databases.

● Familiarity to work with Linux (and Linux work environment).

● Excellent written and verbal communication skills

● Extracting, transforming, and loading data into internal databases and Hadoop

● Optimizing our new and existing data pipelines for speed and reliability

● Deploying product build and product improvements

● Documenting and managing multiple repositories of code

● Experience with SQL and NoSQL databases (Casendra, MySQL)

● Hands-on experience in data pipelining and ETL. (Any of these frameworks/tools: Hadoop, BigQuery,

RedShift, Athena)

● Hands-on experience in AirFlow

● Understanding of best practices, common coding patterns and good practices around

● storing, partitioning, warehousing and indexing of data

● Experience in reading the data from Kafka topic (both live stream and offline)

● Experience in PySpark and Data frames

Responsibilities:

You’ll

● Collaborating across an agile team to continuously design, iterate, and develop big data systems.

● Extracting, transforming, and loading data into internal databases.

● Optimizing our new and existing data pipelines for speed and reliability.

● Deploying new products and product improvements.

● Documenting and managing multiple repositories of code.

About Us:

Cognitio Analytics is an award-winning, niche service provider that offers digital transformation solutions powered by AI and machine learning. We help clients realize the full potential of their data assets and the investments made in related technologies, be it analytics and big data platforms, digital technologies or your people. Our solutions include Health Analytics powered by Cognitio’s Health Data Factory that drives better care outcomes and higher ROI on care spending. We specialize in providing Data Governance solutions that help effectively use data in a consistent and compliant manner. Additionally, our smart intelligence solutions enable a deeper understanding of operations through the use of data science and advanced solutions like process mining technologies. We have offices in New Jersey and Connecticut in the USA and in Gurgaon in India.

What we're looking for:

• Ability in data modelling, design, build and deploy DW/BI systems for Insurance, Health Care, Banking, etc.
• Performance tuning of ETL process and SQL queries and recommend & implement ETL and query tuning techniques.
• Develop and create transformation queries, views, and stored procedures for ETL processes, and process automations.
• Translate business needs into technical specifications, evaluate, and improve existing BI systems.
• Use business intelligence and visualization software (e.g., Tableau, Qlik Sense, Power BI etc.) to empower customers to drive their analytics and reporting
• Develop and update technical documentation for BI systems.

Key Technical Skills:

• Hands on experience in MS SQL Server & MSBI (SSIS/SSRS/SSAS), with understanding of database concepts, star schema, SQL Tuning, OLAP, Databricks, Hadoop, Spark, cloud technologies.
• Experience in designing and building complete ETL / SSIS processes and transforming data for ODS, Staging, and Data Warehouse
• Experience building self-service reporting solutions using business intelligence software (e.g., Tableau, Qlik Sense, Power BI etc.)

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

JD for IOT DE:

The role requires experience in Azure core technologies – IoT Hub/ Event Hub, Stream Analytics, IoT Central, Azure Data Lake Storage, Azure Cosmos, Azure Data Factory, Azure SQL Database, Azure HDInsight / Databricks, SQL data warehouse.

You Have:

• Minimum 2 years of software development experience
• Minimum 2 years of experience in IoT/streaming data pipelines solution development
• Bachelor's and/or Master’s degree in computer science
• Strong Consulting skills in data management including data governance, data quality, security, data integration, processing, and provisioning
• Delivered data management projects with real-time/near real-time data insights delivery on Azure Cloud
• Translated complex analytical requirements into the technical design including data models, ETLs, and Dashboards / Reports
• Experience deploying dashboards and self-service analytics solutions on both relational and non-relational databases
• Experience with different computing paradigms in databases such as In-Memory, Distributed, Massively Parallel Processing
• Successfully delivered large scale IOT data management initiatives covering Plan, Design, Build and Deploy phases leveraging different delivery methodologies including Agile
• Experience in handling telemetry data with Spark Streaming, Kafka, Flink, Scala, Pyspark, Spark SQL.
• Hands-on experience on containers and Dockers
• Exposure to streaming protocols like MQTT and AMQP
• Knowledge of OT network protocols like OPC UA, CAN Bus, and similar protocols
• Strong knowledge of continuous integration, static code analysis, and test-driven development
• Experience in delivering projects in a highly collaborative delivery model with teams at onsite and offshore
• Must have excellent analytical and problem-solving skills
• Delivered change management initiatives focused on driving data platforms adoption across the enterprise
• Strong verbal and written communications skills are a must, as well as the ability to work effectively across internal and external organizations
   

Roles & Responsibilities
 

You Will:

• Translate functional requirements into technical design
• Interact with clients and internal stakeholders to understand the data and platform requirements in detail and determine core Azure services needed to fulfill the technical design
• Design, Develop and Deliver data integration interfaces in ADF and Azure Databricks
• Design, Develop and Deliver data provisioning interfaces to fulfill consumption needs
• Deliver data models on Azure platform, it could be on Azure Cosmos, SQL DW / Synapse, or SQL
• Advise clients on ML Engineering and deploying ML Ops at Scale on AKS
• Automate core activities to minimize the delivery lead times and improve the overall quality
• Optimize platform cost by selecting the right platform services and architecting the solution in a cost-effective manner
• Deploy Azure DevOps and CI CD processes
• Deploy logging and monitoring across the different integration points for critical alerts