Sr Data Engineer - (Python, Pandas)

Data Scientist – Program Embedded 

Job Description:   

We are seeking a highly skilled and motivated senior data scientist to support a big data program. The successful candidate will play a pivotal role in supporting multiple projects in this program covering traditional tasks from revenue management, demand forecasting, improving customer experience to testing/using new tools/platforms such as Copilot Fabric for different purpose. The expected candidate would have deep expertise in machine learning methodology and applications. And he/she should have completed multiple large scale data science projects (full cycle from ideation to BAU). Beyond technical expertise, problem solving in complex set-up will be key to the success for this role. This is a data science role directly embedded into the program/projects, stake holder management and collaborations with patterner are crucial to the success on this role (on top of the deep expertise). 

What we are looking for: 

1. Highly efficient in Python/Pyspark/R. 
2. Understand MLOps concepts, working experience in product industrialization (from Data Science point of view). Experience in building product for live deployment, and continuous development and continuous integration. 
3. Familiar with cloud platforms such as Azure, GCP, and the data management systems on such platform. Familiar with Databricks and product deployment on Databricks. 
4. Experience in ML projects involving techniques: Regression, Time Series, Clustering, Classification, Dimension Reduction, Anomaly detection with traditional ML approaches and DL approaches. 
5. Solid background in statistics, probability distributions, A/B testing validation, univariate/multivariate analysis, hypothesis test for different purpose, data augmentation etc. 
6. Familiar with designing testing framework for different modelling practice/projects based on business needs. 
7. Exposure to Gen AI tools and enthusiastic about experimenting and have new ideas on what can be done. 
8. If they have improved an internal company process using an AI tool, that would be great (e.g. process simplification, manual task automation, auto emails) 
9. Ideally, 10+ years of experience, and have been on independent business facing roles. 
10. CPG or retail as a data scientist would be nice, but not number one priority, especially for those who have navigated through multiple industries. 
11. Being proactive and collaborative would be essential. 

Some projects examples within the program: 

1. Test new tools/platforms such as Copilo, Fabric for commercial reporting. Testing, validation and build trust. 
2. Building algorithms for predicting trend in category, consumptions to support dashboards. 
3. Revenue Growth Management, create/understand the algorithms behind the tools (can be built by 3rd parties) we need to maintain or choose to improve. Able to prioritize and build product roadmap. Able to design new solutions and articulate/quantify the limitation of the solutions. 
4. Demand forecasting, create localized forecasts to improve in store availability. Proper model monitoring for early detection of potential issues in the forecast focusing particularly on improving the end user experience. 

As a Sr. Cloud IoT Engineer with Intuitive, you will be responsible for the data acquisition from devices and sensors that allows the device to connect seamlessly with other systems. You will be researching, creating, testing, and documenting IoT solutions with integrated systems and devices to help the analytics and data science initiatives across our enterprise customers. 

You will be working closely with SMEs in Data Engineering and Cloud Engineering, to create solutions and extend Intuitive's DataOps Engineering Projects and Initiatives. The Sr. Cloud IoT Engineer will be a central critical role for establishing the DataOps/DataX data logistics and management for building data pipelines, enforcing best practices, ownership for building complex and performant Data Lake Environments, work closely with Cloud Infrastructure Architects and DevSecOps automation teams. The Sr. Cloud IoT Engineer is the main point of contact for all things related to ingestion of telemetry data and saturation into time series or other databases. In this role, we expect our DataOps leaders to be obsessed with telemetry data and providing insights to help our end customers. 

  KEY REQUIREMENTS: 

• 10+ years’ experience as data engineer. 
• Must have 5+ Years in implementing IoT engineering solutions with multiple cloud providers and toolsets. 
• This is hands on role building data pipelines using Cloud Native and Partner Solutions. Hands-on technical experience with Data at Scale. 
• Must have deep understanding of solutions like AWS IoT Greengrass edge runtime and cloud services to build, deploy, and manage device software across the fleet of devices.  

• Hands on experience with AWS IoT stack like Core, SiteWise, Kinesis, Lamda, Timestream 
• Performance Tuning of streaming telemetry data based with tools like Grafana, Amazon Cloudwatch and QuickSight informed by business requirements. 
• Good working experience on Web Service Integration, RESTful APIs, WebSockets and MQTT.
• Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration    

About Us:

We are a VC-funded startup solving one of the biggest transportation problems India faces. Most passengers in India travel long distance by IRCTC trains. At time of booking, approx 1 out of every 2 passengers end up with a Waitlisted or RAC ticket. This creates a lot of anxiety for passengers, as Railway only announces only 4 hour before departure if they have a confirmed seat. We solve this problem through our Waitlist & RAC Protection. Protection can be bought against each IRCTC ticket at time of booking. If train ticket is not confirmed, we fly the passenger to the destination. Our team consists of 3 Founders from IIT, IIM and ISB.

Functional Experience:

• Computer Science or IT Engineering background with solid understanding of basics of Data Structures and Algorithms
• 2+ years of data science experience working with large datasets
• Expertise in Python packages like pandas, numPy, sklearn, matplotlib, seaborn, keras and tensorflow
• Expertise in Big Data technologies like Hadoop, Cassandra and PostgreSQL
• Expertise in Cloud computing on AWS with EC2, AutoML, Lambda and RDS
• Good knowledge of Machine Learning and Statistical time series analysis (optional)
• Unparalleled logical ability making you the go to guy for all things related to data
• You love coding like a hobby and are up for a challenge!

Cultural:

• Assume a strong sense of ownership of analytics : Design, develop & deploy
• Collaborate with senior management, operations & business team
• Ensure Quality & sustainability of the architecture
• Motivation to join an early stage startup should go beyond compensation

As a Data Engineer, your role will encompass: 

• Designing and building production data pipelines from ingestion to consumption within a hybrid big data architecture using Scala, Python, Talend etc.   
• Gather and address technical and design requirements.  
• Refactor existing applications to optimize its performance through setting the appropriate architecture and integrating the best practices and standards. 
• Participate in the entire data life-cycle mainly focusing on coding, debugging, and testing. 

• Troubleshoot and debug ETL Pipelines. 
• Documentation of each process. 

Technical Requirements: - 

• BSc degree in Computer Science/Computer Engineering. (Masters is a plus.) 
• 2+ years of experience as a Data Engineer. 

• In-depth understanding of core ETL concepts, Data Modelling, Data Lineage, Data Governance, Data Catalog, etc. 
• 2+ years of work experience in Scala, Python, Java. 
• Good Knowledge on Big Data Tools such as Spark/HDFS/Hive/Flume, etc. 
• Hands on experience on ETL tools like Talend/Informatica is a plus. 
• Good knowledge in Kafka and spark streaming is a big plus. 

• 2+ years of experience in using Azure cloud and its resources/services (like Azure Data factory, Azure Databricks, SQL Synapse, Azure Devops, Logic Apps, Power Bi, Azure Event Hubs, etc). 
• Strong experience in Relational Databases (MySQL, SQL Server)  
• Exposure on data visualization tools like Power BI / Qlik sense / MicroStrategy 
• 2+ years of experience in developing APIs (REST & SOAP protocols). 
• Strong knowledge in Continuous Integration & Continuous Deployment (CI/CD) utilizing Docker containers, Jenkins, etc. 

• Strong competencies in algorithms and software architecture. 
• Excellent analytical and teamwork skills. 

 Good to have: - 

• Previous on-prem working experience is a plus. 
• In-depth understanding of the entire web development process (design, development, and deployment) 

• Previous experience in automated testing including unit testing & UI testing. 

• Total Experience of 7-10 years and should be interested in teaching and research
• 3+ years’ experience in data engineering which includes data ingestion, preparation, provisioning, automated testing, and quality checks.
• 3+ Hands-on experience in Big Data cloud platforms like AWS and GCP, Data Lakes and Data Warehouses
• 3+ years of Big Data and Analytics Technologies. Experience in SQL, writing code in spark engine using python, scala or java Language. Experience in Spark, Scala
• Experience in designing, building, and maintaining ETL systems
• Experience in data pipeline and workflow management tools like Airflow
• Application Development background along with knowledge of Analytics libraries, opensource Natural Language Processing, statistical and big data computing libraries
• Familiarity with Visualization and Reporting Tools like Tableau, Kibana.
• Should be good at storytelling in Technology