Big Data Engineer/Data Engineer
What we are solving
Welcome to today’s business data world where:
• Unification of all customer data into one platform is a challenge
• Extraction is expensive
• Business users do not have the time/skill to write queries
• High dependency on tech team for written queries
These facts may look scary but there are solutions with real-time self-serve analytics:
• Fully automated data integration from any kind of a data source into a universal schema
• Analytics database that streamlines data indexing, query and analysis into a single platform.
• Start generating value from Day 1 through deep dives, root cause analysis and micro segmentation
At Propellor.ai, this is what we do.
• We help our clients reduce effort and increase effectiveness quickly
• By clearly defining the scope of Projects
• Using Dependable, scalable, future proof technology solution like Big Data Solutions and Cloud Platforms
• Engaging with Data Scientists and Data Engineers to provide End to End Solutions leading to industrialisation of Data Science Model Development and Deployment
What we have achieved so far
Since we started in 2016,
• We have worked across 9 countries with 25+ global brands and 75+ projects
• We have 50+ clients, 100+ Data Sources and 20TB+ data processed daily
Work culture at Propellor.ai
We are a small, remote team that believes in
• Working with a few, but only with highest quality team members who want to become the very best in their fields.
• With each member's belief and faith in what we are solving, we collectively see the Big Picture
• No hierarchy leads us to believe in reaching the decision maker without any hesitation so that our actions can have fruitful and aligned outcomes.
• Each one is a CEO of their domain.So, the criteria while making a choice is so our employees and clients can succeed together!
To read more about us click here:
https://bit.ly/3idXzs0" target="_blank">https://bit.ly/3idXzs0
About the role
We are building an exceptional team of Data engineers who are passionate developers and wants to push the boundaries to solve complex business problems using the latest tech stack. As a Big Data Engineer, you will work with various Technology and Business teams to deliver our Data Engineering offerings to our clients across the globe.
Role Description
• The role would involve big data pre-processing & reporting workflows including collecting, parsing, managing, analysing, and visualizing large sets of data to turn information into business insights
• Develop the software and systems needed for end-to-end execution on large projects
• Work across all phases of SDLC, and use Software Engineering principles to build scalable solutions
• Build the knowledge base required to deliver increasingly complex technology projects
• The role would also involve testing various machine learning models on Big Data and deploying learned models for ongoing scoring and prediction.
Education & Experience
• B.Tech. or Equivalent degree in CS/CE/IT/ECE/EEE 3+ years of experience designing technological solutions to complex data problems, developing & testing modular, reusable, efficient and scalable code to implement those solutions.
Must have (hands-on) experience
• Python and SQL expertise
• Distributed computing frameworks (Hadoop Ecosystem & Spark components)
• Must be proficient in any Cloud computing platforms (AWS/Azure/GCP) • Experience in in any cloud platform would be preferred - GCP (Big Query/Bigtable, Pub sub, Data Flow, App engine )/ AWS/ Azure
• Linux environment, SQL and Shell scripting Desirable
• Statistical or machine learning DSL like R
• Distributed and low latency (streaming) application architecture
• Row store distributed DBMSs such as Cassandra, CouchDB, MongoDB, etc
. • Familiarity with API design
Hiring Process:
1. One phone screening round to gauge your interest and knowledge of fundamentals
2. An assignment to test your skills and ability to come up with solutions in a certain time
3. Interview 1 with our Data Engineer lead
4. Final Interview with our Data Engineer Lead and the Business Teams
Preferred Immediate Joiners
About Propellor.ai
Who we are
At Propellor, we are passionate about solving Data Unification challenges faced by our clients. We build solutions using the latest tech stack. We believe all solutions lie in the congruence of Business, Technology, and Data Science. Combining the 3, our team of young Data professionals solves some real-world problems. Here's what we live by:
Skin in the game
We believe that Individual and Collective success orientations both propel us ahead.
Cross Fertility
Borrowing from and building on one another’s varied perspectives means we are always viewing business problems with a fresh lens.
Sub 25's
A bunch of young turks, who keep our explorer mindset alive and kicking.
Future-proofing
Keeping an eye ahead, we are upskilling constantly, staying relevant at any given point in time.
Tech Agile
Tech changes quickly. Whatever your stack, we adapt speedily and easily.
If you are evaluating us to be your next employer, we urge you to read more about our team and culture here: https://bit.ly/3ExSNA2. We assure you, it's worth a read!
Similar jobs
Senior Data Engineer - Big Data
at CodeCraft Technologies Private Limited
• Responsible to Ingest data from files, streams and databases. Process the data with Apache Kafka, Spark, Google
Fire Store, Google BigQuery
• Drive Data Foundation initiatives, like Modelling, Data Quality Management, Data Governance, Data Maturity
Assessments and Data Strategy in support of the key business stakeholders.
• Implementing ETL process using Google BigQuery
• Monitoring performance and advising any necessary infrastructure changes
• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies
Pyspark, Kafka, Google BigQuery, etc.
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing
systems
• Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various
use cases built on the platform
• Provide high operational excellence guaranteeing high availability and platform stability
Desired Profile:
• Deep understanding of the ecosystem, including ingestion (e.g. Kafka, Kinesis, Apache Airflow), processing
frameworks (e.g. Spark, Flink) and storage engines (e.g. Google FIreStore, Google BigQuery).
• Should have indepth understanding of Bigquery architecture, table partitioning, clustering, best practices, type of
tables, etc.
• Should know how to reduce BigQuery costs by reducing the amount of data processed by your queries
• Practical knowledge of Kafka to build real-time streaming data pipelines and applications that adapt to the data
streams.
• Should be able to speed up queries by using denormalized data structures, with or without nested repeated fields
• Implementing ETL jobs using Bigquery
• Understanding of Bigquery ML
• Knowledge on latest database technologies like MongoDB, Cassandra, Data Bricks etc.
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience in GCP and Managed services of GCP
Responsibilities:
- Should act as a technical resource for the Data Science team and be involved in creating and implementing current and future Analytics projects like data lake design, data warehouse design, etc.
- Analysis and design of ETL solutions to store/fetch data from multiple systems like Google Analytics, CleverTap, CRM systems etc.
- Developing and maintaining data pipelines for real time analytics as well as batch analytics use cases.
- Collaborate with data scientists and actively work in the feature engineering and data preparation phase of model building
- Collaborate with product development and dev ops teams in implementing the data collection and aggregation solutions
- Ensure quality and consistency of the data in Data warehouse and follow best data governance practices
- Analyse large amounts of information to discover trends and patterns
- Mine and analyse data from company databases to drive optimization and improvement of product development, marketing techniques and business strategies.\
Requirements
- Bachelor’s or Masters in a highly numerate discipline such as Engineering, Science and Economics
- 2-6 years of proven experience working as a Data Engineer preferably in ecommerce/web based or consumer technologies company
- Hands on experience of working with different big data tools like Hadoop, Spark , Flink, Kafka and so on
- Good understanding of AWS ecosystem for big data analytics
- Hands on experience in creating data pipelines either using tools or by independently writing scripts
- Hands on experience in scripting languages like Python, Scala, Unix Shell scripting and so on
- Strong problem solving skills with an emphasis on product development.
- Experience using business intelligence tools e.g. Tableau, Power BI would be an added advantage (not mandatory)
DevOps Engineer
2. Good understanding and hands on experience in Shell/Bash scripting, sonarqube, Terraform,
3. Experience with Continuous Integration and Continuous Deployment Pipelines
4. Experience in SVN, Maven, Git and Git workflows
5. Should be able to develop overall strategy for Build & Release management
6. Experience in working with container orchestration tools such as Docker and Kubernetes
7. Good knowledge in Devops Automation Tools like Chef, Ansible, Puppet, helm, splunk, maven, gradle & XL Deploy.etc
8. Managing stakeholders and external interfaces and Setting up tools and required infrastructure
9. Encouraging and building automated processes wherever possible
10. Awareness of critical concepts in DevOps and Agile principles
11. Experience in Cloud infrastructure like AWS, GCP or Azure. In AWS understanding on EC2, S3 & cloud
12. Strong knowledge and hands on experience in unix OS
13.Experience in network, server, application status monitoring and troubleshooting, Security.
14.Design, develop automation suite and integrate with continuous integration process through Jenkins
15. Possess good problem solving and debugging skills. Troubleshoot issues and coordinate with development team to streamline code deployment to generate build
About CarWale: CarWale's mission is to bring delight in car buying, we offer a bouquet of reliable tools and services to help car consumers decide on buying the right car, at the right price and from the right partner. CarWale has always strived to serve car buyers and owners in the most comprehensive and convenient way possible. We provide a platform where car buyers and owners can research, buy, sell and come together to discuss and talk about their cars.We aim to empower Indian consumers to make informed car buying and ownership decisions with exhaustive and un-biased information on cars through our expert reviews, owner reviews, detailed specifications and comparisons. We understand that a car is by and large the second-most expensive asset a consumer associates his lifestyle with! Together with CarTrade & BikeWale, we are the market leaders in the personal mobility media space.About the Team:We are a bunch of enthusiastic analysts assisting all business functions with their data needs. We deal with huge but diverse datasets to find relationships, patterns and meaningful insights. Our goal is to help drive growth across the organization by creating a data-driven culture.
We are looking for an experienced Data Scientist who likes to explore opportunities and know their way around data to build world class solutions making a real impact on the business.
Skills / Requirements –
- 3-5 years of experience working on Data Science projects
- Experience doing statistical modelling of big data sets
- Expert in Python, R language with deep knowledge of ML packages
- Expert in fetching data from SQL
- Ability to present and explain data to management
- Knowledge of AWS would be beneficial
- Demonstrate Structural and Analytical thinking
- Ability to structure and execute data science project end to end
Education –
Bachelor’s degree in a quantitative field (Maths, Statistics, Computer Science). Masters will be preferred.
• Experience with Advanced SQL
• Experience with Azure data factory, data bricks,
• Experience with Azure IOT, Cosmos DB, BLOB Storage
• API management, FHIR API development,
• Proficient with Git and CI/CD best practices
• Experience working with Snowflake is a plus
We are looking out for a technically driven "Full-Stack Engineer" for one of our premium client
COMPANY DESCRIPTION:
Qualifications
• Bachelor's degree in computer science or related field; Master's degree is a plus
• 3+ years of relevant work experience
• Meaningful experience with at least two of the following technologies: Python, Scala, Java
• Strong proven experience on distributed processing frameworks (Spark, Hadoop, EMR) and SQL is very
much expected
• Commercial client-facing project experience is helpful, including working in close-knit teams
• Ability to work across structured, semi-structured, and unstructured data, extracting information and
identifying linkages across disparate data sets
• Confirmed ability in clearly communicating complex solutions
• Understandings on Information Security principles to ensure compliant handling and management of
client data
• Experience and interest in Cloud platforms such as: AWS, Azure, Google Platform or Databricks
• Extraordinary attention to detail
- Work with the application development team to implement database schema, data strategies, build data flows
- Should have expertise across the database. Responsible for gathering and analyzing data requirements and data flows to other systems
- Perform database modeling and SQL scripting
- 8+ years of experience in Database modeling and SQL scripting
- Experience in OLTP and OLAP database modeling
- Experience in Data modeling for Relational (PostgreSQL) and Non-relational Databases
- Strong scripting experience P-SQL (Views, Store procedures), Python, Spark.
- Develop notebooks & jobs using Python language
- Big Data stack: Spark, Hadoop, Sqoop, Pig, Hive, Hbase, Flume, Kafka, Storm
- Should have good understanding on Azure Cosmos DB, Azure DW
ETL specialist (for a startup hedge fund)
- We are looking for an experienced data engineer to join our team.
- The preprocessing involves ETL tasks, using pyspark, AWS Glue, staging data in parquet formats on S3, and Athena
To succeed in this data engineering position, you should care about well-documented, testable code and data integrity. We have devops who can help with AWS permissions.
We would like to build up a consistent data lake with staged, ready-to-use data, and to build up various scripts that will serve as blueprints for various additional data ingestion and transforms.
If you enjoy setting up something which many others will rely on, and have the relevant ETL expertise, we’d like to work with you.
Responsibilities
- Analyze and organize raw data
- Build data pipelines
- Prepare data for predictive modeling
- Explore ways to enhance data quality and reliability
- Potentially, collaborate with data scientists to support various experiments
Requirements
- Previous experience as a data engineer with the above technologies
Data Platform Engineer (SDE 1/2/3)
at Urbancompany (formerly known as Urbanclap)
Why are we building Urban Company?
Organized service commerce is a large yet young industry in India. While India is a very large market for home and local services (~USD 50 Billion in retail spends) and expected to double in the next 5 years, there is no billion-dollar company in this segment today.
The industry is bare ~20 years old, with a sub-optimal market architecture typical of an unorganized market - fragmented supply side operated by middlemen. As a result, experiences are broken for both customers and service professionals, each largely relying upon word of mouth to discover the other. The industry can easily be 1.5-2x larger than it is today if the frictions in user and professional journeys are removed - and the experiences made more meaningful and joyful.
The Urban Company team is young and passionate, and we see a massive disruption opportunity in his industry. By leveraging technology, and a set of simple yet powerful processes, we wish to build a platform that can organize the world of services - and bring them to your finger-tips. We believe there is the immense value (akin to serendipity) in bringing together customers and professionals looking for each other. In the process, we hope to impact the lives of millions of service entrepreneurs, and transform service commerce the way Amazon transformed product commerce.
Urbancompany has grown 3x YOY and so as our tech stack. We have evolved in data-driven approach solving for the product over the last few years. We deal with around 10TB in data analytics with around 50Mn/day. We adopted the platform thinking pretty early stage of UC. We started building central platform teams who are dedicated to solving core engineering problems around 2-3 years ago and now it has evolved to a full-fledged vertical. Out platform vertical majorly includes Data Engineering, Service and Core Platform, Infrastructure, and Security. We are looking for Data Engineers, a person who loves solving standardization, has strong platform thinking, opinions, and has solved for Data Engineering, Data Science and analytics platforms.
Job Responsibilities
- Platform first approach to engineering problems.
- Creating highly autonomous systems with minimal manual intervention.
- Frameworks which can be extended to larger audiences through open source.
- Extending and modifying the open source projects to adopt as per Urban Company use case.
- Developer productivity.
- Highly abstracted and standardized frameworks like micro services, event-driven architecture, etc.
Job Requirements/Potential Backgrounds
- Bachelors/master’s in computer science form top-tier Engineering School.
- Experience with Data pipeline and workflow management tools like Luigi, Airflow etc.
- Proven ability to work in a fast paced environment.
- History and Familiarity of server-side development of APIs, databases, dev-ops and systems.
- Fanatic about building scalable, reliable data products.
- Experience with Big data tools: Hadoop, Kafka/Kinesis, Flume, etc. is an added advantage.
- Experience with Relational SQL and NO SQL databases like HBase, Cassandra etc.
- Experience with stream processing engines like Spark, Link, Storm, etc. is an added advantage.
What UC has in store for you
- A phenomenal work environment, with massive ownership and growth opportunities.
- A high performance, high-velocity environment at the cutting edge of growth.
- Strong ownership expectation and freedom to fail.
- Quick iterations and deployments – fail-fast attitude.
- Opportunity to work on cutting edge technologies.
- The massive, and direct impact of the work you do on the lives of people.
