Data Engineer

• Bring in industry best practices around creating and maintaining robust data pipelines for complex data projects with/without AI component
  • programmatically ingesting data from several static and real-time sources (incl. web scraping)
  • rendering results through dynamic interfaces incl. web / mobile / dashboard with the ability to log usage and granular user feedbacks
  • performance tuning and optimal implementation of complex Python scripts (using SPARK), SQL (using stored procedures, HIVE), and NoSQL queries in a production environment

• Industrialize ML / DL solutions and deploy and manage production services; proactively handle data issues arising on live apps
• Perform ETL on large and complex datasets for AI applications - work closely with data scientists on performance optimization of large-scale ML/DL model training
• Build data tools to facilitate fast data cleaning and statistical analysis
• Ensure data architecture is secure and compliant
• Resolve issues escalated from Business and Functional areas on data quality, accuracy, and availability
• Work closely with APAC CDO and coordinate with a fully decentralized team across different locations in APAC and global HQ (Paris).

•  Expert in structured and unstructured data in traditional and Big data environments – Oracle / SQLserver, MongoDB, Hive / Pig, BigQuery, and Spark
• Have excellent knowledge of Python programming both in traditional and distributed models (PySpark)
• Expert in shell scripting and writing schedulers
• Hands-on experience with Cloud - deploying complex data solutions in hybrid cloud / on-premise environment both for data extraction/storage and computation
• Hands-on experience in deploying production apps using large volumes of data with state-of-the-art technologies like Dockers, Kubernetes, and Kafka
• Strong knowledge of data security best practices
• 5+ years experience in a data engineering role
• Science / Engineering graduate from a Tier-1 university in the country
• And most importantly, you must be a passionate coder who really cares about building apps that can help people do things better, smarter, and faster even when they sleep

Design, implement, and improve the analytics platform

Implement and simplify self-service data query and analysis capabilities of the BI platform

Develop and improve the current BI architecture, emphasizing data security, data quality

and timeliness, scalability, and extensibility

Deploy and use various big data technologies and run pilots to design low latency

data architectures at scale

Collaborate with business analysts, data scientists, product managers, software development engineers,

and other BI teams to develop, implement, and validate KPIs, statistical analyses, data profiling, prediction,

forecasting, clustering, and machine learning algorithms

Educational

At Ganit we are building an elite team, ergo we are seeking candidates who possess the

following backgrounds:

7+ years relevant experience

Expert level skills writing and optimizing complex SQL

Knowledge of data warehousing concepts

Experience in data mining, profiling, and analysis

Experience with complex data modelling, ETL design, and using large databases

in a business environment

Proficiency with Linux command line and systems administration

Experience with languages like Python/Java/Scala

Experience with Big Data technologies such as Hive/Spark

Proven ability to develop unconventional solutions, sees opportunities to

innovate and leads the way

Good experience of working in cloud platforms like AWS, GCP & Azure. Having worked on

projects involving creation of data lake or data warehouse

Excellent verbal and written communication.

Proven interpersonal skills and ability to convey key insights from complex analyses in

summarized business terms. Ability to effectively communicate with multiple teams

Good to have

AWS/GCP/Azure Data Engineer Certification

Experience: 8+ Years

Work Location: Hyderabad

Mode of work: Work from Office

Senior Data Engineer / Architect

Summary of the Role

The Senior Data Engineer / Architect will be a key role within the data and technology team, responsible for engineering and building data solutions that enable seamless use of data within the organization. 

Core Activities

-         Work closely with the business teams and business analysts to understand and document data usage requirements

-         Develop designs relating to data engineering solutions including data pipelines, ETL, data warehouse, data mart and data lake solutions

-         Develop data designs for reporting and other data use requirements

-         Develop data governance solutions that provide data governance services including data security, data quality, data lineage etc.

-         Lead implementation of data use and data quality solutions

-         Provide operational support for users for the implemented data solutions

-         Support development of solutions that automate reporting and business intelligence requirements

-         Support development of machine learning and AI solution using large scale internal and external datasets

Other activities

-         Work on and manage technology projects as and when required

-         Provide user and technical training on data solutions

Skills and Experience

-         At least 5-8 years of experience in a senior data engineer / architect role

-         Strong experience with AWS based data solutions including AWS Redshift, analytics and data governance solutions

-         Strong experience with industry standard data governance / data quality solutions  

-         Strong experience with managing a Postgres SQL data environment

-         Background as a software developer working in AWS / Python will be beneficial

-         Experience with BI tools like Power BI and Tableau

Strong written and oral communication skills

• Should have good hands-on experience in Informatica MDM Customer 360, Data Integration(ETL) using PowerCenter, Data Quality.
• Must have strong skills in Data Analysis, Data Mapping for ETL processes, and Data Modeling.
• Experience with the SIF framework including real-time integration
• Should have experience in building C360 Insights using Informatica
• Should have good experience in creating performant design using Mapplets, Mappings, Workflows for Data Quality(cleansing), ETL.
• Should have experience in building different data warehouse architecture like Enterprise,
• Federated, and Multi-Tier architecture.
• Should have experience in configuring Informatica Data Director in reference to the Data
• Governance of users, IT Managers, and Data Stewards.
• Should have good knowledge in developing complex PL/SQL queries.
• Should have working experience on UNIX and shell scripting to run the Informatica workflows and to control the ETL flow.
• Should know about Informatica Server installation and knowledge on the Administration console.
• Working experience with Developer with Administration is added knowledge.
• Working experience in Amazon Web Services (AWS) is an added advantage. Particularly on AWS S3, Data pipeline, Lambda, Kinesis, DynamoDB, and EMR.
• Should be responsible for the creation of automated BI solutions, including requirements, design,development, testing, and deployment

Primary responsibilities:

• Architect, Design and Build high performance Search systems for personalization, optimization, and targeting
• Designing systems with Solr, Akka, Cassandra, Kafka
• Algorithmic development with primary focus Machine Learning
• Working with rapid and innovative development methodologies like: Kanban, Continuous Integration and Daily deployments
• Participation in design and code reviews and recommend improvements
• Unit testing with JUnit, Performance testing and tuning
• Coordination with internal and external teams
• Mentoring junior engineers
• Participate in Product roadmap and Prioritization discussions and decisions
• Evangelize the solution with Professional services and Customer Success teams

• Must have 5-8 years of experience in handling data
• Must have the ability to interpret large amounts of data and to multi-task
• Must have strong knowledge of and experience with programming (Python), Linux/Bash scripting, databases(SQL, etc)
• Must have strong analytical and critical thinking to resolve business problems using data and tech
•  Must have domain familiarity and interest of – Cloud technologies (GCP/Azure Microsoft/ AWS Amazon), open-source technologies, Enterprise technologies
• Must have the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
• Must have good communication skills
• Working knowledge/exposure to ElasticSearch, PostgreSQL, Athena, PrestoDB, Jupyter Notebook