Bigdata Engineer

Function : Sr. DB Developer

Location : India/Gurgaon/Tamilnadu

 

>> THE INDIVIDUAL

• Have a strong background in data platform creation and management.
• Possess in-depth knowledge of Data Management, Data Modelling, Ingestion  - Able to develop data models and ingestion frameworks based on client requirements and advise on system optimization.
• Hands-on experience in SQL database (PostgreSQL) and No-SQL database (MongoDB) 
• Hands-on experience in performance tuning of DB
• Good to have knowledge of database setup in cluster node
• Should be well versed with data security aspects and data governance framework
• Hands-on experience in Spark, Airflow, ELK.
• Good to have knowledge on any data cleansing tool like apache Griffin
• Preferably getting involved during project implementation so have a background on business knowledge and technical requirement as well.
• Strong analytical and problem-solving skills. Have exposure to data analytics skills and knowledge of advanced data analytical tools will be an advantage.
• Strong written and verbal communication skills (presentation skills).
• Certifications in the above technologies is preferred.

 

>> Qualification

 

1. Tech /B.E. / MCA /M. Tech from a reputed institute.

Experience of Data Management, Data Modelling, Ingestion for more than 4 years. Total experience of 8-10 Years

We are establishing infrastructure for internal and external reporting using Tableau and are looking for someone with experience building visualizations and dashboards in Tableau and using Tableau Server to deliver them to internal and external users. 

 

Required Experience 

• Implementation of interactive visualizations using Tableau Desktop  
• Integration with Tableau Server and support of production dashboards and embedded reports with it 

• Writing and optimization of SQL queries  
• Proficient in Python including the use of Pandas and numpy libraries to perform data exploration and analysis 
• 3  years of experience working as a Software Engineer / Senior Software Engineer 
• Bachelors in Engineering – can be Electronic and comm , Computer , IT  
• Well versed with Basic Data Structures Algorithms and system design 

• Should be capable of working well in a team – and should possess very good communication skills 
• Self-motivated and fun to work with and organized 
• Productive and efficient working remotely 
• Test driven mindset with a knack for finding issues and problems at earlier stages of development 
• Interest in learning and picking up a wide range of cutting edge technologies 

• Should be curious and interested in learning some Data science related concepts and domain knowledge 
• Work alongside other engineers on the team to elevate technology and consistently apply best practices 

 

Highly Desirable 

• Data Analytics 

• Experience in AWS cloud or any cloud technologies 
• Experience in BigData technologies and streaming like – pyspark, kafka is a big plus 
• Shell scripting  
• Preferred tech stack – Python, Rest API, Microservices, Flask/Fast API, pandas, numpy, linux, shell scripting, Airflow, pyspark 
• Has a strong backend experience – and worked with Microservices and Rest API’s - Flask, FastAPI, Databases Relational and Non-relational 

Title: Data Engineer (Azure) (Location: Gurgaon/Hyderabad)

Salary: Competitive as per Industry Standard

We are expanding our Data Engineering Team and hiring passionate professionals with extensive

knowledge and experience in building and managing large enterprise data and analytics platforms. We

are looking for creative individuals with strong programming skills, who can understand complex

business and architectural problems and develop solutions. The individual will work closely with the rest

of our data engineering and data science team in implementing and managing Scalable Smart Data

Lakes, Data Ingestion Platforms, Machine Learning and NLP based Analytics Platforms, Hyper-Scale

Processing Clusters, Data Mining and Search Engines.

What You’ll Need:

• 3+ years of industry experience in creating and managing end-to-end Data Solutions, Optimal

Data Processing Pipelines and Architecture dealing with large volume, big data sets of varied

data types.

• Proficiency in Python, Linux and shell scripting.
• Strong knowledge of working with PySpark dataframes, Pandas dataframes for writing efficient pre-processing and other data manipulation tasks.
  ● Strong experience in developing the infrastructure required for data ingestion, optimal

extraction, transformation, and loading of data from a wide variety of data sources using tools like Azure Data Factory,  Azure Databricks (or Jupyter notebooks/ Google Colab) (or other similiar tools).

• Working knowledge of github or other version control tools.
• Experience with creating Restful web services and API platforms.
• Work with data science and infrastructure team members to implement practical machine

learning solutions and pipelines in production.

• Experience with cloud providers like Azure/AWS/GCP.
• Experience with SQL and NoSQL databases. MySQL/ Azure Cosmosdb / Hbase/MongoDB/ Elasticsearch etc.
• Experience with stream-processing systems: Spark-Streaming, Kafka etc and working experience with event driven architectures.
• Strong analytic skills related to working with unstructured datasets.

 

Good to have (to filter or prioritize candidates)

• Experience with testing libraries such as pytest for writing unit-tests for the developed code.
• Knowledge of Machine Learning algorithms and libraries would be good to have,

implementation experience would be an added advantage.

• Knowledge and experience of Datalake, Dockers and Kubernetes would be good to have.
• Knowledge of Azure functions , Elastic search etc will be good to have.

 

• Having experience with model versioning (mlflow) and data versioning will be beneficial
• Having experience with microservices libraries or with python libraries such as flask for hosting ml services and models would be great.

Roles and Responsibilities

● Working on an awesome AI product for the eCommerce domain.
● Build the next-generation information extraction, computer vision product powered
by state-of-the-art AI and Deep Learning techniques.
● Work with an international top-notch engineering team with full commitment to
Machine Learning development.

Desired Candidate Profile

● Passionate about search & AI technologies. Open to collaborating with colleagues &
external contributors.
● Good understanding of the mainstream deep learning models from multiple domains:
computer vision, NLP, reinforcement learning, model optimization, etc.
● Hands-on experience on deep learning frameworks, e.g. Tensorflow, Pytorch, MXNet,
BERT. Able to implement the latest DL model using existing API, open-source libraries
in a short time.
● Hands-on experience with the Cloud-Native techniques. Good understanding of web
services and modern software technologies.
● Maintained/contributed machine learning projects, familiar with the agile software
development process, CICD workflow, ticket management, code-review, version
control, etc.
● Skilled in the following programming languages: Python 3.
● Good English skills especially for writing and reading documentation

Responsibilities:

 

* 3+ years of Data Engineering Experience - Design, develop, deliver and maintain data infrastructures.

* SQL Specialist – Strong knowledge and Seasoned experience with SQL Queries

* Languages: Python

* Good communicator, shows initiative, works well with stakeholders.

* Experience working closely with Data Analysts and provide the data they need and guide them on the issues.

* Solid ETL experience and Hadoop/Hive/Pyspark/Presto/ SparkSQL

* Solid communication and articulation skills

* Able to handle stakeholders independently with less interventions of reporting manager.

* Develop strategies to solve problems in logical yet creative ways.

* Create custom reports and presentations accompanied by strong data visualization and storytelling

 

We would be excited if you have:

 

* Excellent communication and interpersonal skills

* Ability to meet deadlines and manage project delivery

* Excellent report-writing and presentation skills

* Critical thinking and problem-solving capabilities

Technology Skills:

Roles and Responsibilities:
 Mining large volumes of credit behavior data to generate insights around product holdings and monetization opportunities for cross sell
 Use data science to size opportunity and product potential for launch of any new product/pilots
 Build propensity models using heuristics and campaign performance to maximize efficiency.
 Conduct portfolio analysis and establish key metrics for cross sell partnership

Desired profile/Skills:
 2-5 years of experience with a degree in any quantitative discipline such as Engineering, Computer Science, Economics, Statistics or Mathematics
 Excellent problem solving and comprehensive analytical skills – ability to structure ambiguous problem statements, perform detailed analysis and derive crisp insights.
 Solid experience in using python and SQL
 Prior work experience in a financial services space would be highly valued

Location: Bangalore/ Ahmedabad

Intro

Our data and risk team is the core pillar of our business that harnesses alternative data sources to guide the decisions we make at Rely. The team designs, architects, as well as develop and maintain a scalable data platform the powers our machine learning models. Be part of a team that will help millions of consumers across Asia, to be effortlessly in control of their spending and make better decisions.

What will you do
The data engineer is focused on making data correct and accessible, and building scalable systems to access/process it. Another major responsibility is helping AI/ML Engineers write better code.

• Optimize and automate ingestion processes for a variety of data sources such as: click stream, transactional and many other sources.

• Create and maintain optimal data pipeline architecture and ETL processes
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Develop data pipeline and infrastructure to support real-time decisions
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data' technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
  
  

What will you need
• 2+ hands-on experience building and implementation of large scale production pipeline and Data Warehouse
• Experience dealing with large scale

• Proficiency in writing and debugging complex SQLs
• Experience working with AWS big data tools
  • Ability to lead the project and implement best data practises and technology

Data Pipelining

• Strong command in building & optimizing data pipelines, architectures and data sets
• Strong command on relational SQL & noSQL databases including Postgres
• Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

Big Data: Strong experience in big data tools & applications

• Tools: Hadoop, Spark, HDFS etc
• AWS cloud services: EC2, EMR, RDS, Redshift
• Stream-processing systems: Storm, Spark-Streaming, Flink etc.
• Message queuing: RabbitMQ, Spark etc
  
  

Software Development & Debugging

• Strong experience in object-oriented programming/object function scripting languages: Python, Java, C++, Scala, etc
• Strong hold on data structures & algorithms

What would be a bonus

• Prior experience working in a fast-growth Startup
• Prior experience in the payments, fraud, lending, advertising companies dealing with large scale data