Data Engineer

CORE RESPONSIBILITIES

• Create and manage cloud resources in AWS 
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies 
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform 
• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations 
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data.
• Define process improvement opportunities to optimize data collection, insights and displays.
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible 
• Identify and interpret trends and patterns from complex data sets 
• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders. 
• Key participant in regular Scrum ceremonies with the agile teams  
• Proficient at developing queries, writing reports and presenting findings 
• Mentor junior members and bring best industry practices 

QUALIFICATIONS

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales) 
• Strong background in math, statistics, computer science, data science or related discipline
• Advanced knowledge one of language: Java, Scala, Python, C# 
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake  
• Proficient with
• Data mining/programming tools (e.g. SAS, SQL, R, Python)
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum)
• Data visualization (e.g. Tableau, Looker, MicroStrategy)
• Comfortable learning about and deploying new technologies and tools. 
• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines. 
• Good written and oral communication skills and ability to present results to non-technical audiences 
• Knowledge of business intelligence and analytical tools, technologies and techniques.

Mandatory Requirements 

• Experience in AWS Glue
• Experience in Apache Parquet 
• Proficient in AWS S3 and data lake 
• Knowledge of Snowflake
• Understanding of file-based ingestion best practices.
• Scripting language - Python & pyspark

JD Code: SHI-LDE-01 

Version#: 1.0 

Date of JD Creation: 27-March-2023 

Position Title: Lead Data Engineer 

Reporting to: Technical Director 

Location: Bangalore Urban, India (on-site) 

SmartHub.ai (www.smarthub.ai) is a fast-growing Startup headquartered in Palo Alto, CA, and with offices in Seattle and Bangalore. We operate at the intersection of AI, IoT & Edge Computing. With strategic investments from leaders in infrastructure & data management, SmartHub.ai is redefining the Edge IoT space. Our “Software Defined Edge” products help enterprises rapidly accelerate their Edge Infrastructure Management & Intelligence. We empower enterprises to leverage their Edge environment to increase revenue, efficiency of operations, manage safety and digital risks by using Edge and AI technologies. 

SmartHub is an equal opportunity employer and will always be committed to nurture a workplace culture that supports, inspires and respects all individuals, encourages employees to bring their best selves to work, laugh and share. We seek builders who hail from a variety of backgrounds, perspectives and skills to join our team.  

Summary 

This role requires the candidate to translate business and product requirements to build, maintain, optimize data systems which can be relational or non-relational in nature. The candidate is expected to tune and analyse the data including from a short and long-term trend analysis and reporting, AI/ML uses cases. 

We are looking for a talented technical professional with at least 8 years of proven experience in owning, architecting, designing, operating and optimising databases that are used for large scale analytics and reports. 

Responsibilities 

• Provide technical & architectural leadership for the next generation of product development. 
• Innovate, Research & Evaluate new technologies and tools for a quality output. 
• Architect, Design and Implement ensuring scalability, performance and security. 
• Code and implement new algorithms to solve complex problems. 
• Analyze complex data, develop, optimize and transform large data sets both structured and unstructured. 
• Ability to deploy and administrator the database and continuously tuning for performance especially container orchestration stacks such as Kubernetes  
• Develop analytical models and solutions Mentor Junior members technically in Architecture, Designing and robust Coding. 
• Work in an Agile development environment while continuously evaluating and improvising engineering processes 

Required 

• At least 8 years of experience with significant depth in designing and building scalable distributed database systems for enterprise class products, experience of working in product development companies. 
• Should have been feature/component lead for several complex features involving large datasets. 
• Strong background in relational and non-relational database like Postgres, MongoDB, Hadoop etl. 
• Deep exp database optimization, tuning ertise in SQL, Time Series Databases, Apache Drill, HDFS, Spark are good to have 
• Excellent analytical and problem-solving skill sets. 
• Experience in  for high throughput is highly desirable 
• Exposure to database provisioning in Kubernetes/non-Kubernetes environments, configuration and tuning in a highly available mode. 
• Demonstrated ability to provide technical leadership and mentoring to the team 

Come Join Qualifyde’s Interviewing world and be part of the next generation of technical interviews.

QUALIFYDE Tech nerds are a network of software professionals, including software development managers, software engineers, and freelancers covering the full technology stack.

Technical Interviews with QUALIFYDE occur 24/7 with candidates located throughout the world

Conduct interviews based on your own availability — whether that be five times a week, 10 times a week, mornings, nights, or weekends.

Utilize the QUALIFYDE Interviewing infrastructure and process to conduct fair, equitable technical interviews in a wide variety of roles.

Who can apply; -

• Software developers/engineers with hands-on coding experience and experience conducting technical interviews
• Strong Communication Skills in English (Oral & Written)
• Stable, Consistent internet connection
• Equipped with a working webcam

• Design, create, test, and maintain data pipeline architecture in collaboration with the Data Architect.
• Build the infrastructure required for extraction, transformation, and loading of data from a wide variety of data sources using Java, SQL, and Big Data technologies.
• Support the translation of data needs into technical system requirements. Support in building complex queries required by the product teams.
• Build data pipelines that clean, transform, and aggregate data from disparate sources
• Develop, maintain and optimize ETLs to increase data accuracy, data stability, data availability, and pipeline performance.
• Engage with Product Management and Business to deploy and monitor products/services on cloud platforms.
• Stay up-to-date with advances in data persistence and big data technologies and run pilots to design the data architecture to scale with the increased data sets of consumer experience.
• Handle data integration, consolidation, and reconciliation activities for digital consumer / medical products.

Job Qualifications:

• Bachelor’s or master's degree in Computer Science, Information management, Statistics or related field
• 5+ years of experience in the Consumer or Healthcare industry in an analytical role with a focus on building on data pipelines, querying data, analyzing, and clearly presenting analyses to members of the data science team.
• Technical expertise with data models, data mining.
• Hands-on Knowledge of programming languages in Java, Python, R, and Scala.
• Strong knowledge in Big data tools like the snowflake, AWS Redshift, Hadoop, map-reduce, etc.
• Having knowledge in tools like AWS Glue, S3, AWS EMR, Streaming data pipelines, Kafka/Kinesis is desirable.
• Hands-on knowledge in SQL and No-SQL database design.
• Having knowledge in CI/CD for the building and hosting of the solutions.
• Having AWS certification is an added advantage.
• Having Strong knowledge in visualization tools like Tableau, QlikView is an added advantage
• A team player capable of working and integrating across cross-functional teams for implementing project requirements. Experience in technical requirements gathering and documentation.
• Ability to work effectively and independently in a fast-paced agile environment with tight deadlines
• A flexible, pragmatic, and collaborative team player with the innate ability to engage with data architects, analysts, and scientists

We are looking for a self motivated and passionate individual, with strong desire to learn and ability to lead. This position is for a Flight Test Engineer, with exposure to building and flying sUAS (RC Multirotors and Fixed wings). See the detailed job description below.

Responsibilities

• Plan and execute flight test plans for new software features, electronics, sensors, and payloads.
• Perform hands-on mechanical and electrical integration of new hardware components on the internal fleet of test vehicles for R&D and testing.
• Troubleshoot and debug any components of a drone in the office or in the field. Maintenance of vehicles – keep the fleet ready for flight tests.
• Participate in defining and validating customer workflows and enhancing User experience.
• Coordinate cross-team efforts among FlytBase engineers to resolve issues identified during flight tests.
• Drive collaboration with FlytBase Developer team, Business Development team and Customer Support team to incorporate customer feedback and feature requests into FlytBase’s product development cycle.
• Learn about the domain and competitors to propose new drone applications, as well as, improvements in existing applications

Experience/Skills

• Experience in flight testing and operating/piloting small UAS and/or RC aircraft (both fixed-wing and multirotor systems).
• Experience in using flight-planning and ground control station software.
•Familiarity with UAV platforms, like, Pixhawk, DJI, Ardupilot and PX4.
•Experience in integrating, operating, and tuning autopilots on a variety of unmanned vehicles.
•Basic knowledge of electrical test equipment (multimeter, oscilloscope) and UAS sensors.
•Ability to work hands-on with electro-mechanical systems including assembly, disassembly, testing, troubleshooting.
•Good verbal and written communication skills.

Good to have

• RF communications fundamentals.
• Passionate about aerial robots i.e. drones.
• Programming languages and scripting for engineering use (C++, C, MATLAB, Python).

Compensation:
As per industry standards.

Perks:
+ Fast-paced Startup culture
+ Hacker mode environment
+ Great team
+ Flexible work hours
+ Informal dress code
+ Free snacks