Scala Spark Engineer

Intuitive cloud (http://www.intuitive.cloud">www.intuitive.cloud) is one of the fastest growing top-tier Cloud Solutions and SDx Engineering solution and service company supporting 80+ Global Enterprise Customer across Americas, Europe and Middle East.

Intuitive is a recognized professional and manage service partner for core superpowers in cloud(public/ Hybrid), security, GRC, DevSecOps, SRE, Application modernization/ containers/ K8 -as-a- service and cloud application delivery.

• 9+ years’ experience as data engineer.
• Must have 4+ Years in implementing data engineering solutions with Databricks.
• This is hands on role building data pipelines using Databricks. Hands-on technical experience with Apache Spark.
• Must have deep expertise in one of the programming languages for data processes (Python, Scala). Experience with Python, PySpark, Hadoop, Hive and/or Spark to write data pipelines and data processing layers
• Must have worked with relational databases like Snowflake. Good SQL experience for writing complex SQL transformation.
• Performance Tuning of Spark SQL running on S3/Data Lake/Delta Lake/ storage and Strong Knowledge on Databricks and Cluster Configurations.
• Hands on architectural experience
• Nice to have Databricks administration including security and infrastructure features of Databricks.

About Slintel (a 6sense company) :

Slintel, a 6sense company,  the leader in capturing technographics-powered buying intent, helps companies uncover the 3% of active buyers in their target market. Slintel evaluates over 100 billion data points and analyzes factors such as buyer journeys, technology adoption patterns, and other digital footprints to deliver market & sales intelligence.

Slintel's customers have access to the buying patterns and contact information of more than 17 million companies and 250 million decision makers across the world.

Slintel is a fast growing B2B SaaS company in the sales and marketing tech space. We are funded by top tier VCs, and going after a billion dollar opportunity. At Slintel, we are building a sales development automation platform that can significantly improve outcomes for sales teams, while reducing the number of hours spent on research and outreach.

We are a big data company and perform deep analysis on technology buying patterns, buyer pain points to understand where buyers are in their journey. Over 100 billion data points are analyzed every week to derive recommendations on where companies should focus their marketing and sales efforts on. Third party intent signals are then clubbed with first party data from CRMs to derive meaningful recommendations on whom to target on any given day.

6sense is headquartered in San Francisco, CA and has 8 office locations across 4 countries.

6sense, an account engagement platform, secured $200 million in a Series E funding round, bringing its total valuation to $5.2 billion 10 months after its $125 million Series D round. The investment was co-led by Blue Owl and MSD Partners, among other new and existing investors.

Linkedin (Slintel) : https://www.linkedin.com/company/slintel/">https://www.linkedin.com/company/slintel/

Industry : Software Development

Company size : 51-200 employees (189 on LinkedIn)

Headquarters : Mountain View, California

Founded : 2016

Specialties : Technographics, lead intelligence, Sales Intelligence, Company Data, and Lead Data.

Website (Slintel) : https://www.slintel.com/slintel">https://www.slintel.com/slintel

Linkedin (6sense) : https://www.linkedin.com/company/6sense/">https://www.linkedin.com/company/6sense/

Industry : Software Development

Company size : 501-1,000 employees (937 on LinkedIn)

Headquarters : San Francisco, California

Founded : 2013

Specialties : Predictive intelligence, Predictive marketing, B2B marketing, and Predictive sales

Website (6sense) : https://6sense.com/">https://6sense.com/

Acquisition News : 

https://inc42.com/buzz/us-based-based-6sense-acquires-b2b-buyer-intelligence-startup-slintel/ 

Funding Details & News :

Slintel funding : https://www.crunchbase.com/organization/slintel">https://www.crunchbase.com/organization/slintel

6sense funding : https://www.crunchbase.com/organization/6sense">https://www.crunchbase.com/organization/6sense

https://www.nasdaq.com/articles/ai-software-firm-6sense-valued-at-%245.2-bln-after-softbank-joins-funding-round">https://www.nasdaq.com/articles/ai-software-firm-6sense-valued-at-%245.2-bln-after-softbank-joins-funding-round

https://www.bloomberg.com/news/articles/2022-01-20/6sense-reaches-5-2-billion-value-with-softbank-joining-round">https://www.bloomberg.com/news/articles/2022-01-20/6sense-reaches-5-2-billion-value-with-softbank-joining-round

https://xipometer.com/en/company/6sense">https://xipometer.com/en/company/6sense

Slintel & 6sense Customers :

https://www.featuredcustomers.com/vendor/slintel/customers

https://www.featuredcustomers.com/vendor/6sense/customers">https://www.featuredcustomers.com/vendor/6sense/customers

Responsibilities

• Work in collaboration with the application team and integration team to design, create, and maintain optimal data pipeline architecture and data structures for Data Lake/Data Warehouse
• Work with stakeholders including the Sales, Product, and Customer Support teams to assist with data-related technical issues and support their data analytics needs
• Assemble large, complex data sets from third-party vendors to meet business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimising data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, Elastic search, MongoDB, and AWS technology
• Streamline existing and introduce enhanced reporting and analysis solutions that leverage complex data sources derived from multiple internal systems
  
  

Requirements

• 3+ years of experience in a Data Engineer role
• Proficiency in Linux
• Must have SQL knowledge and experience working with relational databases, query authoring (SQL) as well as familiarity with databases including Mysql, Mongo, Cassandra, and Athena
• Must have experience with Python/ Scala
• Must have experience with Big Data technologies like Apache Spark
• Must have experience with Apache Airflow
• Experience with data pipeline and ETL tools like AWS Glue
• Experience working with AWS cloud services: EC2 S3 RDS, Redshift and other Data solutions eg. Databricks, Snowflake

Python, SQL, Scala, Spark, ETL

Roles and Responsibilities:

• Design, develop, and maintain the end-to-end MLOps infrastructure from the ground up, leveraging open-source systems across the entire MLOps landscape.
• Creating pipelines for data ingestion, data transformation, building, testing, and deploying machine learning models, as well as monitoring and maintaining the performance of these models in production.
• Managing the MLOps stack, including version control systems, continuous integration and deployment tools, containerization, orchestration, and monitoring systems.
• Ensure that the MLOps stack is scalable, reliable, and secure.

Skills Required:

• 3-6 years of MLOps experience
• Preferably worked in the startup ecosystem

Primary Skills:

• Experience with E2E MLOps systems like ClearML, Kubeflow, MLFlow etc.
• Technical expertise in MLOps: Should have a deep understanding of the MLOps landscape and be able to leverage open-source systems to build scalable, reliable, and secure MLOps infrastructure.
• Programming skills: Proficient in at least one programming language, such as Python, and have experience with data science libraries, such as TensorFlow, PyTorch, or Scikit-learn.
• DevOps experience: Should have experience with DevOps tools and practices, such as Git, Docker, Kubernetes, and Jenkins.

Secondary Skills:

• Version Control Systems (VCS) tools like Git and Subversion
• Containerization technologies like Docker and Kubernetes
• Cloud Platforms like AWS, Azure, and Google Cloud Platform
• Data Preparation and Management tools like Apache Spark, Apache Hadoop, and SQL databases like PostgreSQL and MySQL
• Machine Learning Frameworks like TensorFlow, PyTorch, and Scikit-learn
• Monitoring and Logging tools like Prometheus, Grafana, and Elasticsearch
• Continuous Integration and Continuous Deployment (CI/CD) tools like Jenkins, GitLab CI, and CircleCI
• Explain ability and Interpretability tools like LIME and SHAP

• We are looking for : Data engineer
• Sprak
• Scala
• Hadoop

• Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
• Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
• Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
• Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
• Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
• Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
• Analyse large amounts of information to discover trends and patterns
• Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\

• Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
• 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
• Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
• Good understanding of AWS ecosystem for big data analytics
• Hands on experience in creating data pipelines either using tools or by independently writing scripts
• Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
• Strong problem solving skills with an emphasis on product development.
• Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)

• We are looking for a Data Engineer to build the next-generation mobile applications for our world-class fintech product.
• The candidate will be responsible for expanding and optimising our data and data pipeline architecture, as well as optimising data flow and collection for cross-functional teams.
• The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimising data systems and building them from the ground up.
• Looking for a person with a strong ability to analyse and provide valuable insights to the product and business team to solve daily business problems.
• You should be able to work in a high-volume environment, have outstanding planning and organisational skills.

Qualifications for Data Engineer

• Working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimising ‘big data’ data pipelines, architectures, and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets. Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• Looking for a candidate with 2-3 years of experience in a Data Engineer role, who is a CS graduate or has an equivalent experience.

What we're looking for?

• Experience with big data tools: Hadoop, Spark, Kafka and other alternate tools.
• Experience with relational SQL and NoSQL databases, including MySql/Postgres and Mongodb.
• Experience with data pipeline and workflow management tools: Luigi, Airflow.
• Experience with AWS cloud services: EC2, EMR, RDS, Redshift.
• Experience with stream-processing systems: Storm, Spark-Streaming.
• Experience with object-oriented/object function scripting languages: Python, Java, Scala.

• Research and properly evaluate sources of information to determine possible limitations in reliability or usability
• Apply sampling techniques to effectively determine and define ideal categories to be questioned
• Compare and analyze provided statistical information to identify patterns, relationships and problems
• Define and utilize statistical methods to solve industry-specific problems in varying fields, such as economics and engineering
• Prepare detailed reports for management and other departments by analyzing and interpreting data
• Train assistants and other members of the team how to properly organize findings and read data collected
• Design computer code using various languages to improve and update software and applications
• Refer to previous instances and findings to determine the ideal method for gathering data