Big Data Engineer

at Propellor.ai

DP
Posted by Kajal Jain
icon
Remote only
icon
1 - 4 yrs
icon
₹5L - ₹15L / yr
icon
Full time
Skills
Python
SQL
PySpark
Data engineering
Big Data
Hadoop
Spark

Big Data Engineer/Data Engineer


What we are solving
Welcome to today’s business data world where:
• Unification of all customer data into one platform is a challenge

• Extraction is expensive
• Business users do not have the time/skill to write queries
• High dependency on tech team for written queries

These facts may look scary but there are solutions with real-time self-serve analytics:
• Fully automated data integration from any kind of a data source into a universal schema
• Analytics database that streamlines data indexing, query and analysis into a single platform.
• Start generating value from Day 1 through deep dives, root cause analysis and micro segmentation

At Propellor.ai, this is what we do.
• We help our clients reduce effort and increase effectiveness quickly
• By clearly defining the scope of Projects
• Using Dependable, scalable, future proof technology solution like Big Data Solutions and Cloud Platforms
• Engaging with Data Scientists and Data Engineers to provide End to End Solutions leading to industrialisation of Data Science Model Development and Deployment

What we have achieved so far
Since we started in 2016,
• We have worked across 9 countries with 25+ global brands and 75+ projects
• We have 50+ clients, 100+ Data Sources and 20TB+ data processed daily

Work culture at Propellor.ai
We are a small, remote team that believes in
• Working with a few, but only with highest quality team members who want to become the very best in their fields.
• With each member's belief and faith in what we are solving, we collectively see the Big Picture
• No hierarchy leads us to believe in reaching the decision maker without any hesitation so that our actions can have fruitful and aligned outcomes.
• Each one is a CEO of their domain.So, the criteria while making a choice is so our employees and clients can succeed together!

To read more about us click here:
https://bit.ly/3idXzs0" target="_blank">https://bit.ly/3idXzs0

About the role
We are building an exceptional team of Data engineers who are passionate developers and wants to push the boundaries to solve complex business problems using the latest tech stack. As a Big Data Engineer, you will work with various Technology and Business teams to deliver our Data Engineering offerings to our clients across the globe.

Role Description

• The role would involve big data pre-processing & reporting workflows including collecting, parsing, managing, analysing, and visualizing large sets of data to turn information into business insights
• Develop the software and systems needed for end-to-end execution on large projects
• Work across all phases of SDLC, and use Software Engineering principles to build scalable solutions
• Build the knowledge base required to deliver increasingly complex technology projects
• The role would also involve testing various machine learning models on Big Data and deploying learned models for ongoing scoring and prediction.

Education & Experience
• B.Tech. or Equivalent degree in CS/CE/IT/ECE/EEE 3+ years of experience designing technological solutions to complex data problems, developing & testing modular, reusable, efficient and scalable code to implement those solutions.

Must have (hands-on) experience
• Python and SQL expertise
• Distributed computing frameworks (Hadoop Ecosystem & Spark components)
• Must be proficient in any Cloud computing platforms (AWS/Azure/GCP)  • Experience in in any cloud platform would be preferred - GCP (Big Query/Bigtable, Pub sub, Data Flow, App engine )/ AWS/ Azure

• Linux environment, SQL and Shell scripting Desirable
• Statistical or machine learning DSL like R
• Distributed and low latency (streaming) application architecture
• Row store distributed DBMSs such as Cassandra, CouchDB, MongoDB, etc
. • Familiarity with API design

Hiring Process:
1. One phone screening round to gauge your interest and knowledge of fundamentals
2. An assignment to test your skills and ability to come up with solutions in a certain time
3. Interview 1 with our Data Engineer lead
4. Final Interview with our Data Engineer Lead and the Business Teams

Preferred Immediate Joiners

Read more

About Propellor.ai


Who we are


At Propellor, we are passionate about solving Data Unification challenges faced by our clients. We build solutions using the latest tech stack. We believe all solutions lie in the congruence of Business, Technology, and Data Science. Combining the 3, our team of young Data professionals solves some real-world problems. Here's what we live by:


Skin in the game

We believe that Individual and Collective success orientations both propel us ahead.

 

Cross Fertility

Borrowing from and building on one another’s varied perspectives means we are always viewing business problems with a fresh lens.

 

Sub 25's

A bunch of young turks, who keep our explorer mindset alive and kicking.

 

Future-proofing

Keeping an eye ahead, we are upskilling constantly, staying relevant at any given point in time.

 

Tech Agile

Tech changes quickly. Whatever your stack, we adapt speedily and easily.


If you are evaluating us to be your next employer, we urge you to read more about our team and culture here: https://bit.ly/3ExSNA2. We assure you, it's worth a read!

Read more
Founded
2016
Type
Products & Services
Size
20-100 employees
Stage
Raised funding
View full company details
Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Senior Data Engineer - Big Data

at CodeCraft Technologies Private Limited

Founded 2011  •  Services  •  100-1000 employees  •  Profitable
Data engineering
SQL
Spark
Apache
HiveQL
Big Data
icon
Bengaluru (Bangalore), Mangalore
icon
4 - 8 yrs
icon
Best in industry
Roles and Responsibilities:
• Responsible to Ingest data from files, streams and databases. Process the data with Apache Kafka, Spark, Google
Fire Store, Google BigQuery
• Drive Data Foundation initiatives, like Modelling, Data Quality Management, Data Governance, Data Maturity
Assessments and Data Strategy in support of the key business stakeholders.
• Implementing ETL process using Google BigQuery
• Monitoring performance and advising any necessary infrastructure changes
• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies
Pyspark, Kafka, Google BigQuery, etc.
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing
systems
• Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various
use cases built on the platform
• Provide high operational excellence guaranteeing high availability and platform stability

Desired Profile:
• Deep understanding of the ecosystem, including ingestion (e.g. Kafka, Kinesis, Apache Airflow), processing
frameworks (e.g. Spark, Flink) and storage engines (e.g. Google FIreStore, Google BigQuery).
• Should have indepth understanding of Bigquery architecture, table partitioning, clustering, best practices, type of
tables, etc.
• Should know how to reduce BigQuery costs by reducing the amount of data processed by your queries
• Practical knowledge of Kafka to build real-time streaming data pipelines and applications that adapt to the data
streams.
• Should be able to speed up queries by using denormalized data structures, with or without nested repeated fields
• Implementing ETL jobs using Bigquery
• Understanding of Bigquery ML
• Knowledge on latest database technologies like MongoDB, Cassandra, Data Bricks etc.
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience in GCP and Managed services of GCP
Read more
Job posted by
Priyanka Praveen

Data Engineer

at Product based company

Agency job
via Zyvka Global Services
Spark
Hadoop
Big Data
Data engineering
PySpark
Python
Scala
Amazon Web Services (AWS)
ETL
CleverTap
Linux/Unix
icon
Bengaluru (Bangalore)
icon
3 - 12 yrs
icon
₹5L - ₹30L / yr

Responsibilities:

  • Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
  • Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
  • Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
  • Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
  • Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
  • Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
  • Analyse large amounts of information to discover trends and patterns
  • Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\

Requirements

  • Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
  • 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
  • Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
  • Good understanding of AWS ecosystem for big data analytics
  • Hands on experience in creating data pipelines either using tools or by independently writing scripts
  • Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
  • Strong problem solving skills with an emphasis on product development.
  • Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)
Read more
Job posted by
Ridhima Sharma

DevOps Engineer

at Panamax InfoTech Ltd.

Founded 2004  •  Products & Services  •  100-1000 employees  •  Profitable
Data Warehouse (DWH)
Informatica
ETL
DevOps
Python
Perl
Java
.NET
Shell Scripting
Bash
Terraform
SVN
Maven
Git
Docker
Kubernetes
Chef
Ansible
Puppet
Splunk
Gradle
Software deployment
helm
icon
Remote only
icon
3 - 4 yrs
icon
₹5L - ₹10L / yr
1. Excellent understanding of at least any one of the programming language .NET ,Python, Perl, and Java
2. Good understanding and hands on experience in Shell/Bash scripting, sonarqube, Terraform,
3. Experience with Continuous Integration and Continuous Deployment Pipelines
4. Experience in SVN, Maven, Git and Git workflows
5. Should be able to develop overall strategy for Build & Release management
6. Experience in working with container orchestration tools such as Docker and Kubernetes
7. Good knowledge in Devops Automation Tools like Chef, Ansible, Puppet, helm, splunk, maven, gradle & XL Deploy.etc
8. Managing stakeholders and external interfaces and Setting up tools and required infrastructure
9. Encouraging and building automated processes wherever possible
10. Awareness of critical concepts in DevOps and Agile principles
11. Experience in Cloud infrastructure like AWS, GCP or Azure. In AWS understanding on EC2, S3 & cloud
12. Strong knowledge and hands on experience in unix OS
13.Experience in network, server, application status monitoring and troubleshooting, Security.
14.Design, develop automation suite and integrate with continuous integration process through Jenkins
15. Possess good problem solving and debugging skills. Troubleshoot issues and coordinate with development team to streamline code deployment to generate build
Read more
Job posted by
Bhavani P

Data Scientist

at CarWale

Founded  •   •  employees  • 
Data Science
Data Scientist
R Programming
Python
Machine Learning (ML)
Amazon Web Services (AWS)
icon
Navi Mumbai, Mumbai
icon
3 - 5 yrs
icon
₹10L - ₹15L / yr

About CarWale: CarWale's mission is to bring delight in car buying, we offer a bouquet of reliable tools and services to help car consumers decide on buying the right car, at the right price and from the right partner. CarWale has always strived to serve car buyers and owners in the most comprehensive and convenient way possible. We provide a platform where car buyers and owners can research, buy, sell and come together to discuss and talk about their cars.We aim to empower Indian consumers to make informed car buying and ownership decisions with exhaustive and un-biased information on cars through our expert reviews, owner reviews, detailed specifications and comparisons. We understand that a car is by and large the second-most expensive asset a consumer associates his lifestyle with! Together with CarTrade & BikeWale, we are the market leaders in the personal mobility media space.About the Team:We are a bunch of enthusiastic analysts assisting all business functions with their data needs. We deal with huge but diverse datasets to find relationships, patterns and meaningful insights. Our goal is to help drive growth across the organization by creating a data-driven culture.

We are looking for an experienced Data Scientist who likes to explore opportunities and know their way around data to build world class solutions making a real impact on the business. 

 

Skills / Requirements –

  • 3-5 years of experience working on Data Science projects
  • Experience doing statistical modelling of big data sets
  • Expert in Python, R language with deep knowledge of ML packages
  • Expert in fetching data from SQL
  • Ability to present and explain data to management
  • Knowledge of AWS would be beneficial
  • Demonstrate Structural and Analytical thinking
  • Ability to structure and execute data science project end to end

 

Education –

Bachelor’s degree in a quantitative field (Maths, Statistics, Computer Science). Masters will be preferred.

 

Read more
Job posted by
Vanita Acharya

Data Engineer

at Reddoorz

Founded 2015  •  Product  •  1000-5000 employees  •  Profitable
Python
Amazon Web Services (AWS)
Amazon Redshift
pandas
Airflow
icon
Noida
icon
2 - 6 yrs
icon
₹8L - ₹15L / yr
Should have hands-on: Python, AWS, redshift/snowflake/bigquery/ any MPP database. Should have knowledge of data pipeline and orchestration.
Read more
Job posted by
Atif Imam

Data Engineer

at SenecaGlobal

Founded 2007  •  Products & Services  •  100-1000 employees  •  Profitable
Python
PySpark
Spark
Scala
Microsoft Azure Data factory
icon
Remote, Hyderabad
icon
4 - 6 yrs
icon
₹15L - ₹20L / yr
Should have good experience with Python or Scala/PySpark/Spark/
• Experience with Advanced SQL
• Experience with Azure data factory, data bricks,
• Experience with Azure IOT, Cosmos DB, BLOB Storage
• API management, FHIR API development,
• Proficient with Git and CI/CD best practices
• Experience working with Snowflake is a plus
Read more
Job posted by
Shiva V

Data Engineer

at Top Management Consulting Company

Python
SQL
Amazon Web Services (AWS)
Microsoft Windows Azure
Google Cloud Platform (GCP)
icon
Gurugram, Bengaluru (Bangalore)
icon
2 - 9 yrs
icon
Best in industry
Greetings!!

We are looking out for a technically driven  "Full-Stack Engineer" for one of our premium client

COMPANY DESCRIPTION:
This Company is a global management consulting firm. We are the trusted advisor to the world's leading businesses, governments, and institutions. We work with leading organizations across the private, public and social sectors. 

Qualifications
• Bachelor's degree in computer science or related field; Master's degree is a plus
• 3+ years of relevant work experience
• Meaningful experience with at least two of the following technologies: Python, Scala, Java
• Strong proven experience on distributed processing frameworks (Spark, Hadoop, EMR) and SQL is very
much expected
• Commercial client-facing project experience is helpful, including working in close-knit teams
• Ability to work across structured, semi-structured, and unstructured data, extracting information and
identifying linkages across disparate data sets
• Confirmed ability in clearly communicating complex solutions
• Understandings on Information Security principles to ensure compliant handling and management of
client data
• Experience and interest in Cloud platforms such as: AWS, Azure, Google Platform or Databricks
• Extraordinary attention to detail
Read more
Job posted by
Naveed Mohd
Big Data
Spark
Hadoop
Apache Kafka
Data engineering
Databases
OLTP and OLAP
icon
Remote, Bangalore
icon
3 - 6 yrs
icon
₹6L - ₹13L / yr
  • Work with the application development team to implement database schema, data strategies, build data flows
  • Should have expertise across the database. Responsible for gathering and analyzing data requirements and data flows to other systems
  • Perform database modeling and SQL scripting
  • 8+ years of experience in Database modeling and SQL scripting
  • Experience in OLTP and OLAP database modeling
  • Experience in Data modeling for Relational (PostgreSQL) and Non-relational Databases
  • Strong scripting experience P-SQL (Views, Store procedures), Python, Spark.
  • Develop notebooks & jobs using Python language
  • Big Data stack: Spark, Hadoop, Sqoop, Pig, Hive, Hbase, Flume, Kafka, Storm
  • Should have good understanding on Azure Cosmos DB, Azure DW

Read more
Job posted by
DAMIAN FRANCIS

ETL specialist (for a startup hedge fund)

at Prediction Machine

Founded 2021  •  Products & Services  •  20-100 employees  •  Raised funding
ETL
PySpark
Data engineering
Data engineer
athena
Amazon S3
Machine Learning (ML)
Data Science
Python
Apache Kafka
Apache Spark
Data modeling
Predictive analytics
AWS Glue
icon
Remote only
icon
3 - 8 yrs
icon
$24K - $60K / yr
We are a nascent quant hedge fund; we need to stage financial data and make it easy to run and re-run various preprocessing and ML jobs on the data.
- We are looking for an experienced data engineer to join our team.
- The preprocessing involves ETL tasks, using pyspark, AWS Glue, staging data in parquet formats on S3, and Athena

To succeed in this data engineering position, you should care about well-documented, testable code and data integrity. We have devops who can help with AWS permissions.
We would like to build up a consistent data lake with staged, ready-to-use data, and to build up various scripts that will serve as blueprints for various additional data ingestion and transforms.

If you enjoy setting up something which many others will rely on, and have the relevant ETL expertise, we’d like to work with you.

Responsibilities
- Analyze and organize raw data
- Build data pipelines
- Prepare data for predictive modeling
- Explore ways to enhance data quality and reliability
- Potentially, collaborate with data scientists to support various experiments

Requirements
- Previous experience as a data engineer with the above technologies
Read more
Job posted by
Nataliia Mediana

Data Platform Engineer (SDE 1/2/3)

at Urbancompany (formerly known as Urbanclap)

Founded 2014  •  Services  •  100-1000 employees  •  Raised funding
Apache Kafka
Spark
NodeJS (Node.js)
Python
Hadoop
Apache Hadoop
PySpark
Data Science
Data Visualization
Big Data
icon
Remote, NCR (Delhi | Gurgaon | Noida), Bengaluru (Bangalore)
icon
3 - 8 yrs
icon
Best in industry

Why are we building Urban Company?

 

Organized service commerce is a large yet young industry in India. While India is a very large market for home and local services (~USD 50 Billion in retail spends) and expected to double in the next 5 years, there is no billion-dollar company in this segment today.

 

The industry is bare ~20 years old, with a sub-optimal market architecture typical of an unorganized market - fragmented supply side operated by middlemen. As a result, experiences are broken for both customers and service professionals, each largely relying upon word of mouth to discover the other. The industry can easily be 1.5-2x larger than it is today if the frictions in user and professional journeys are removed - and the experiences made more meaningful and joyful.

 

The Urban Company team is young and passionate, and we see a massive disruption opportunity in his industry. By leveraging technology, and a set of simple yet powerful processes, we wish to build a platform that can organize the world of services - and bring them to your finger-tips. We believe there is the immense value (akin to serendipity) in bringing together customers and professionals looking for each other. In the process, we hope to impact the lives of millions of service entrepreneurs, and transform service commerce the way Amazon transformed product commerce.

 

Urbancompany has grown 3x YOY and so as our tech stack. We have evolved in data-driven approach solving for the product over the last few years. We deal with around 10TB in data analytics with around 50Mn/day.  We adopted the platform thinking pretty early stage of UC. We started building central platform teams who are dedicated to solving core engineering problems around 2-3 years ago and now it has evolved to a full-fledged vertical. Out platform vertical majorly includes Data Engineering, Service and Core Platform, Infrastructure, and Security. We are looking for Data Engineers, a person who loves solving standardization, has strong platform thinking, opinions, and has solved for Data Engineering, Data Science and analytics platforms.

 

Job Responsibilities

  • Platform first approach to engineering problems.
  • Creating highly autonomous systems with minimal manual intervention.
  • Frameworks which can be extended to larger audiences through open source.
  • Extending and modifying the open source projects to adopt as per Urban Company use case.
  • Developer productivity.
  • Highly abstracted and standardized frameworks like micro services, event-driven architecture, etc.

 

Job Requirements/Potential Backgrounds

  • Bachelors/master’s in computer science form top-tier Engineering School.
  • Experience with Data pipeline and workflow management tools like Luigi, Airflow etc.
  • Proven ability to work in a fast paced environment.
  • History and Familiarity of server-side development of APIs, databases, dev-ops and systems.
  • Fanatic about building scalable, reliable data products.
  • Experience with Big data tools: Hadoop, Kafka/Kinesis, Flume, etc. is an added advantage.
  • Experience with Relational SQL and NO SQL databases like HBase, Cassandra etc.
  • Experience with stream processing engines like Spark, Link, Storm, etc. is an added advantage.

 

What UC has in store for you

 

  • A phenomenal work environment, with massive ownership and growth opportunities.
  • A high performance, high-velocity environment at the cutting edge of growth.
  • Strong ownership expectation and freedom to fail.
  • Quick iterations and deployments – fail-fast attitude.
  • Opportunity to work on cutting edge technologies.
  • The massive, and direct impact of the work you do on the lives of people.
Read more
Job posted by
Mohit Agrawal
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at Propellor.ai?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort