Senior GCP Data Lead

Job Description:

As an Azure Data Engineer, your role will involve designing, developing, and maintaining data solutions on the Azure platform. You will be responsible for building and optimizing data pipelines, ensuring data quality and reliability, and implementing data processing and transformation logic. Your expertise in Azure Databricks, Python, SQL, Azure Data Factory (ADF), PySpark, and Scala will be essential for performing the following key responsibilities:

Designing and developing data pipelines: You will design and implement scalable and efficient data pipelines using Azure Databricks, PySpark, and Scala. This includes data ingestion, data transformation, and data loading processes.

Data modeling and database design: You will design and implement data models to support efficient data storage, retrieval, and analysis. This may involve working with relational databases, data lakes, or other storage solutions on the Azure platform.

Data integration and orchestration: You will leverage Azure Data Factory (ADF) to orchestrate data integration workflows and manage data movement across various data sources and targets. This includes scheduling and monitoring data pipelines.

Data quality and governance: You will implement data quality checks, validation rules, and data governance processes to ensure data accuracy, consistency, and compliance with relevant regulations and standards.

Performance optimization: You will optimize data pipelines and queries to improve overall system performance and reduce processing time. This may involve tuning SQL queries, optimizing data transformation logic, and leveraging caching techniques.

Monitoring and troubleshooting: You will monitor data pipelines, identify performance bottlenecks, and troubleshoot issues related to data ingestion, processing, and transformation. You will work closely with cross-functional teams to resolve data-related problems.

Documentation and collaboration: You will document data pipelines, data flows, and data transformation processes. You will collaborate with data scientists, analysts, and other stakeholders to understand their data requirements and provide data engineering support.

Skills and Qualifications:

Strong experience with Azure Databricks, Python, SQL, ADF, PySpark, and Scala.

Proficiency in designing and developing data pipelines and ETL processes.

Solid understanding of data modeling concepts and database design principles.

Familiarity with data integration and orchestration using Azure Data Factory.

Knowledge of data quality management and data governance practices.

Experience with performance tuning and optimization of data pipelines.

Strong problem-solving and troubleshooting skills related to data engineering.

Excellent collaboration and communication skills to work effectively in cross-functional teams.

Understanding of cloud computing principles and experience with Azure services.

JD Code: SHI-LDE-01 

Version#: 1.0 

Date of JD Creation: 27-March-2023 

Position Title: Lead Data Engineer 

Reporting to: Technical Director 

Location: Bangalore Urban, India (on-site) 

SmartHub.ai (www.smarthub.ai) is a fast-growing Startup headquartered in Palo Alto, CA, and with offices in Seattle and Bangalore. We operate at the intersection of AI, IoT & Edge Computing. With strategic investments from leaders in infrastructure & data management, SmartHub.ai is redefining the Edge IoT space. Our “Software Defined Edge” products help enterprises rapidly accelerate their Edge Infrastructure Management & Intelligence. We empower enterprises to leverage their Edge environment to increase revenue, efficiency of operations, manage safety and digital risks by using Edge and AI technologies. 

SmartHub is an equal opportunity employer and will always be committed to nurture a workplace culture that supports, inspires and respects all individuals, encourages employees to bring their best selves to work, laugh and share. We seek builders who hail from a variety of backgrounds, perspectives and skills to join our team.  

Summary 

This role requires the candidate to translate business and product requirements to build, maintain, optimize data systems which can be relational or non-relational in nature. The candidate is expected to tune and analyse the data including from a short and long-term trend analysis and reporting, AI/ML uses cases. 

We are looking for a talented technical professional with at least 8 years of proven experience in owning, architecting, designing, operating and optimising databases that are used for large scale analytics and reports. 

Responsibilities 

• Provide technical & architectural leadership for the next generation of product development. 
• Innovate, Research & Evaluate new technologies and tools for a quality output. 
• Architect, Design and Implement ensuring scalability, performance and security. 
• Code and implement new algorithms to solve complex problems. 
• Analyze complex data, develop, optimize and transform large data sets both structured and unstructured. 
• Ability to deploy and administrator the database and continuously tuning for performance especially container orchestration stacks such as Kubernetes  
• Develop analytical models and solutions Mentor Junior members technically in Architecture, Designing and robust Coding. 
• Work in an Agile development environment while continuously evaluating and improvising engineering processes 

Required 

• At least 8 years of experience with significant depth in designing and building scalable distributed database systems for enterprise class products, experience of working in product development companies. 
• Should have been feature/component lead for several complex features involving large datasets. 
• Strong background in relational and non-relational database like Postgres, MongoDB, Hadoop etl. 
• Deep exp database optimization, tuning ertise in SQL, Time Series Databases, Apache Drill, HDFS, Spark are good to have 
• Excellent analytical and problem-solving skill sets. 
• Experience in  for high throughput is highly desirable 
• Exposure to database provisioning in Kubernetes/non-Kubernetes environments, configuration and tuning in a highly available mode. 
• Demonstrated ability to provide technical leadership and mentoring to the team 

Responsibilities:

• Be the analytical expert in Kaleidofin, managing ambiguous problems by using data to execute sophisticated quantitative modeling and deliver actionable insights.
• Develop comprehensive skills including project management, business judgment, analytical problem solving and technical depth.
• Become an expert on data and trends, both internal and external to Kaleidofin.
• Communicate key state of the business metrics and develop dashboards to enable teams to understand business metrics independently.
• Collaborate with stakeholders across teams to drive data analysis for key business questions, communicate insights and drive the planning process with company executives.
• Automate scheduling and distribution of reports and support auditing and value realization.
• Partner with enterprise architects to define and ensure proposed.
• Business Intelligence solutions adhere to an enterprise reference architecture.
• Design robust data-centric solutions and architecture that incorporates technology and strong BI solutions to scale up and eliminate repetitive tasks

Requirements:

• Experience leading development efforts through all phases of SDLC.
• 5+ years "hands-on" experience designing Analytics and Business Intelligence solutions.
• Experience with Quicksight, PowerBI, Tableau and Qlik is a plus.
• Hands on experience in SQL, data management, and scripting (preferably Python).
• Strong data visualisation design skills, data modeling and inference skills.
• Hands-on and experience in managing small teams.
• Financial services experience preferred, but not mandatory.
• Strong knowledge of architectural principles, tools, frameworks, and best practices.
• Excellent communication and presentation skills to communicate and collaborate with all levels of the organisation.
• Team handling preferred for 5+yrs experience candidates.
• Notice period less than 30 days.

Job Overview

We are looking for a Data Engineer to join our data team to solve data-driven critical

business problems. The hire will be responsible for expanding and optimizing the existing

end-to-end architecture including the data pipeline architecture. The Data Engineer will

collaborate with software developers, database architects, data analysts, data scientists and platform team on data initiatives and will ensure optimal data delivery architecture is

consistent throughout ongoing projects. The right candidate should have hands on in

developing a hybrid set of data-pipelines depending on the business requirements.

Responsibilities

• Develop, construct, test and maintain existing and new data-driven architectures.
• Align architecture with business requirements and provide solutions which fits best
• to solve the business problems.
• Build the infrastructure required for optimal extraction, transformation, and loading
• of data from a wide variety of data sources using SQL and Azure ‘big data’
• technologies.
• Data acquisition from multiple sources across the organization.
• Use programming language and tools efficiently to collate the data.
• Identify ways to improve data reliability, efficiency and quality
• Use data to discover tasks that can be automated.
• Deliver updates to stakeholders based on analytics.
• Set up practices on data reporting and continuous monitoring
  
  

Required Technical Skills

• Graduate in Computer Science or in similar quantitative area
• 1+ years of relevant work experience as a Data Engineer or in a similar role.
• Advanced SQL knowledge, Data-Modelling and experience working with relational
• databases, query authoring (SQL) as well as working familiarity with a variety of
• databases.
• Experience in developing and optimizing ETL pipelines, big data pipelines, and datadriven
• architectures.
• Must have strong big-data core knowledge & experience in programming using Spark - Python/Scala
• Experience with orchestrating tool like Airflow or similar
• Experience with Azure Data Factory is good to have
• Build processes supporting data transformation, data structures, metadata,
• dependency and workload management.
• Experience supporting and working with cross-functional teams in a dynamic
• environment.
• Good understanding of Git workflow, Test-case driven development and using CICD
• is good to have
• Good to have some understanding of Delta tables It would be advantage if the candidate also have below mentioned experience using
• the following software/tools:
• Experience with big data tools: Hadoop, Spark, Hive, etc.
• Experience with relational SQL and NoSQL databases
• Experience with cloud data services
• Experience with object-oriented/object function scripting languages: Python, Scala, etc.

Responsibilities:

• Design and develop strong analytics system and predictive models
• Managing a team of data scientists, machine learning engineers, and big data specialists
• Identify valuable data sources and automate data collection processes
• Undertake pre-processing of structured and unstructured data
• Analyze large amounts of information to discover trends and patterns
• Build predictive models and machine-learning algorithms
• Combine models through ensemble modeling
• Present information using data visualization techniques
• Propose solutions and strategies to business challenges
• Collaborate with engineering and product development teams

Requirements:

• Proven experience as a seasoned Data Scientist
• Good Experience in data mining processes
• Understanding of machine learning and Knowledge of operations research is a value addition
• Strong understanding and experience in R, SQL, and Python; Knowledge base with Scala, Java, or C++ is an asset
• Experience using business intelligence tools (e. g. Tableau) and data frameworks (e. g. Hadoop)
• Strong math skills (e. g. statistics, algebra)
• Problem-solving aptitude
• Excellent communication and presentation skills
• Experience in Natural Language Processing (NLP)
• Strong competitive coding skills
• BSc/BA in Computer Science, Engineering or relevant field; graduate degree in Data Science or other quantitative field is preferred

About Graphene  

Graphene is a Singapore Head quartered AI company which has been recognized as Singapore’s Best  

Start Up By Switzerland’s Seedstarsworld, and also been awarded as best AI platform for healthcare in Vivatech Paris. Graphene India is also a member of the exclusive NASSCOM Deeptech club. We are developing an AI plaform which is disrupting and replacing traditional Market Research with unbiased insights with a focus on healthcare, consumer goods and financial services.  

Graphene was founded by Corporate leaders from Microsoft and P&G, and works closely with the Singapore Government & Universities in creating cutting edge technology which is gaining traction with many Fortune 500 companies in India, Asia and USA.  

Graphene’s culture is grounded in delivering customer delight by recruiting high potential talent and providing an intense learning and collaborative atmosphere, with many ex-employees now hired by large companies across the world.  

Graphene has a 6-year track record of delivering financially sustainable growth and is one of the rare start-ups which is self-funded and is yet profitable and debt free. We have already created a strong bench strength of Singaporean leaders and are recruiting and grooming more talent with a focus on our US expansion.   

Job title: - Data Analyst 

Job Description  

Data Analyst responsible for storage, data enrichment, data transformation, data gathering based on data requests, testing and maintaining data pipelines. 

Responsibilities and Duties  

• Managing end to end data pipeline from data source to visualization layer 
• Ensure data integrity; Ability to pre-empt data errors 
• Organized managing and storage of data 
• Provide quality assurance of data, working with quality assurance analysts if necessary. 
• Commissioning and decommissioning of data sets. 

• Processing confidential data and information according to guidelines. 
• Helping develop reports and analysis. 
• Troubleshooting the reporting database environment and reports. 
• Managing and designing the reporting environment, including data sources, security, and metadata. 
• Supporting the data warehouse in identifying and revising reporting requirements. 

• Supporting initiatives for data integrity and normalization. 
• Evaluating changes and updates to source production systems. 
• Training end-users on new reports and dashboards. 
• Initiate data gathering based on data requirements 
• Analyse the raw data to check if the requirement is satisfied 

Qualifications and Skills   

• Technologies required: Python, SQL/ No-SQL database(CosmosDB)     
• Experience required 2 – 5 Years. Experience in Data Analysis using Python 

•  Understanding of software development life cycle   

• Plan, coordinate, develop, test and support data pipelines, document, support for reporting dashboards (PowerBI) 
• Automation steps needed to transform and enrich data.   
• Communicate issues, risks, and concerns proactively to management. Document the process thoroughly to allow peers to assist with support as needed.   
• Excellent verbal and written communication skills   

If you are an outstanding ETL Developer with a passion for technology and looking forward to being part of a great development organization, we would love to hear from you. We are offering technology consultancy services to our Fortune 500 customers with a primary focus on digital technologies. Our customers are looking for top-tier talents in the industry and willing to compensate based on your skill and expertise. The nature of our engagement is Contract in most cases. If you are looking for the next big step in your career, we are glad to partner with you. 

Below is the job description for your review.

Extensive hands- on experience in designing and developing ETL packages using SSIS

Extensive experience in performance tuning of SSIS packages

In- depth knowledge of data warehousing concepts and ETL systems, relational databases like SQL Server 2012/ 2014.