Data Engineer

• Creating and managing ETL/ELT pipelines based on requirements
• Build PowerBI dashboards and manage datasets needed.
• Work with stakeholders to identify data structures needed for future and perform any transformations including aggregations.
• Build data cubes for real-time visualisation needs and CXO dashboards.

Required Tech Skills

• Microsoft PowerBI & DAX
• Python, Pandas, PyArrow, Jupyter Noteboks, ApacheSpark
• Azure Synapse, Azure DataBricks, Azure HDInsight, Azure Data Factory

Requirements:

• 2+ years of experience (4+ for Senior Data Engineer) with system/data integration, development or implementation of enterprise and/or cloud software Engineering degree in Computer Science, Engineering or related field.
• Extensive hands-on experience with data integration/EAI technologies (File, API, Queues, Streams), ETL Tools and building custom data pipelines.
• Demonstrated proficiency with Python, JavaScript and/or Java
• Familiarity with version control/SCM is a must (experience with git is a plus).
• Experience with relational and NoSQL databases (any vendor) Solid understanding of cloud computing concepts.
• Strong organisational and troubleshooting skills with attention to detail.
• Strong analytical ability, judgment and problem-solving techniques Interpersonal and communication skills with the ability to work effectively in a cross functional team.

• Design the architecture of our big data platform
• Perform and oversee tasks such as writing scripts, calling APIs, web scraping, and writing SQL queries
• Design and implement data stores that support the scalable processing and storage of our high-frequency data
• Maintain our data pipeline
• Customize and oversee integration tools, warehouses, databases, and analytical systems
• Configure and provide availability for data-access tools used by all data scientists

• Key responsibility is to design, develop & maintain efficient Data models for the organization maintained to ensure optimal query performance by the consumption layer.
• Developing, Deploying & maintaining a repository of UDXs written in Java / Python.
• Develop optimal Data Model design, analyzing complex distributed data deployments, and making recommendations to optimize performance basis data consumption patterns, performance expectations, the query is executed on the tables/databases, etc.
• Periodic Database health check and maintenance
• Designing collections in a no-SQL Database for efficient performance
• Document & maintain data dictionary from various sources to enable data governance
• Coordination with Business teams, IT, and other stakeholders to provide best-in-class data pipeline solutions, exposing data via APIs, loading in down streams, No-SQL Databases, etc
• Data Governance Process Implementation and ensuring data security

Requirements

• Extensive working experience in Designing & Implementing Data models in OLAP Data Warehousing solutions (Redshift, Synapse, Snowflake, Teradata, Vertica, etc).
• Programming experience using Python / Java.
• Working knowledge in developing & deploying User-defined Functions (UDXs) using Java / Python.
• Strong understanding & extensive working experience in OLAP Data Warehousing (Redshift, Synapse, Snowflake, Teradata, Vertica, etc) architecture and cloud-native Data Lake (S3, ADLS, BigQuery, etc) Architecture.
• Strong knowledge in Design, Development & Performance tuning of 3NF/Flat/Hybrid Data Model.
• Extensive technical experience in SQL including code optimization techniques.
• Strung knowledge of database performance and tuning, troubleshooting, and tuning.
• Knowledge of collection design in any No-SQL DB (DynamoDB, MongoDB, CosmosDB, etc), along with implementation of best practices.
• Ability to understand business functionality, processes, and flows.
• Good combination of technical and interpersonal skills with strong written and verbal communication; detail-oriented with the ability to work independently.
• Any OLAP DWH DBA Experience and User Management will be added advantage.
• Knowledge in financial industry-specific Data models such as FSLDM, IBM Financial Data Model, etc will be added advantage.
• Experience in Snowflake will be added advantage.
• Working experience in BFSI/NBFC & data understanding of Loan/Mortgage data will be added advantage.

Functional knowledge

• Data Governance & Quality Assurance
• Modern OLAP Database Architecture & Design
• Linux
• Data structures, algorithm & data modeling techniques
• No-SQL database architecture
• Data Security

Required Python ,R

work in handling large-scale data engineering pipelines.
Excellent verbal and written communication skills.
Proficient in PowerPoint or other presentation tools.
Ability to work quickly and accurately on multiple projects.

• Strong Python Coding skills and OOP skills
• Should have worked on Big Data product Architecture
• Should have worked with any one of the SQL-based databases like MySQL, PostgreSQL and any one of
• NoSQL-based databases such as Cassandra, Elasticsearch etc.
• Hands on experience on frameworks like Spark RDD, DataFrame, Dataset
• Experience on development of ETL for data product
• Candidate should have working knowledge on performance optimization, optimal resource utilization, Parallelism and tuning of spark jobs
• Working knowledge on file formats: CSV, JSON, XML, PARQUET, ORC, AVRO
• Good to have working knowledge with any one of the Analytical Databases like Druid, MongoDB, Apache Hive etc.
• Experience to handle real-time data feeds (good to have working knowledge on Apache Kafka or similar tool)

• Python and Scala (Optional), Spark / PySpark, Parallel programming

• Strong in problem solving, algorithms and data structures
• Proficient in Python
• Hands on experience in technologies and tools related to any of NLP, Deep learning, Machine learning, Conversational AI
• Experience with knowledge Graph or any graph based system is plus
• Able to train and deploy models
• Broad knowledge of machine learning algorithms and principles
• Performance profiling and Tuning
• Communicate and propose solutions to business challenges
• Familiarity with at least one of the cloud computing infrastructure - GCP/AWS
• Keep abreast with the latest technological advances
• Team mentoring and leadership skills.

1. Proven experience with deploying and tuning Open Source components into enterprise ready production tooling Experience with datacentre (Metal as a Service – MAAS) and cloud deployment technologies (AWS or GCP Architect certificates required)
2. Deep understanding of Linux from kernel mechanisms through user space management
3. Experience on CI/CD (Continuous Integrations and Deployment) system solutions (Jenkins).
4. Using Monitoring tools (local and on public cloud platforms) Nagios, Prometheus, Sensu, ELK, Cloud Watch, Splunk, New Relic etc. to trigger instant alerts, reports and dashboards.  Work closely with the development and infrastructure teams to analyze and design solutions with four nines (99.99%) up-time, globally distributed, clustered, production and non-production virtualized infrastructure. 
5. Wide understanding of IP networking as well as data centre infrastructure

1. Expert with software development tools and sourcecode management, understanding, managing issues, code changes and grouping them into deployment releases in a stable and measurable way to maximize production Must be expert at developing and using ansible roles and configuring deployment templates with jinja2.
2. Solid understanding of data collection tools like Flume, Filebeat, Metricbeat, JMX Exporter agents.
3. Extensive experience operating and tuning the kafka streaming data platform, specifically as a message queue for big data processing
4. Strong understanding and must have experience:
5. Apache spark framework, specifically spark core and spark streaming, 
6. Orchestration platforms, mesos and kubernetes, 
7. Data storage platforms, elasticstack, carbon, clickhouse, cassandra, ceph, hdfs
8. Core presentation technologies kibana, and grafana.
9. Excellent scripting and programming skills (bash, python, java, go, rust). Must have previous experience with “rust” in order to support, improve in house developed products

Red Hat Certified Architect certificate or equivalent required CCNA certificate required 3-5 years of experience running open source big data platforms