Data Engineer

at Service based company

icon
Pune
icon
6 - 12 yrs
icon
₹6L - ₹28L / yr
icon
Full time
Skills
Big Data
Apache Kafka
Data engineering
Cassandra
Java
Scala

Primary responsibilities:

  • Architect, Design and Build high performance Search systems for personalization, optimization, and targeting
  • Designing systems with Solr, Akka, Cassandra, Kafka
  • Algorithmic development with primary focus Machine Learning
  • Working with rapid and innovative development methodologies like: Kanban, Continuous Integration and Daily deployments
  • Participation in design and code reviews and recommend improvements
  • Unit testing with JUnit, Performance testing and tuning
  • Coordination with internal and external teams
  • Mentoring junior engineers
  • Participate in Product roadmap and Prioritization discussions and decisions
  • Evangelize the solution with Professional services and Customer Success teams

 

Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Data Engineer

at Fintech Company

Agency job
via Jobdost
Python
SQL
Data Warehouse (DWH)
Hadoop
Amazon Web Services (AWS)
DevOps
Git
Selenium
Informatica
ETL
Big Data
Postman
icon
Bengaluru (Bangalore)
icon
2 - 4 yrs
icon
₹7L - ₹12L / yr

Purpose of Job:

Responsible for drawing insights from many sources of data to answer important business
questions and help the organization make better use of data in their daily activities.


Job Responsibilities:

We are looking for a smart and experienced Data Engineer 1 who can work with a senior
manager to
⮚ Build DevOps solutions and CICD pipelines for code deployment
⮚ Build unit test cases for APIs and Code in Python
⮚ Manage AWS resources including EC2, RDS, Cloud Watch, Amazon Aurora etc.
⮚ Build and deliver high quality data architecture and pipelines to support business
and reporting needs
⮚ Deliver on data architecture projects and implementation of next generation BI
solutions
⮚ Interface with other teams to extract, transform, and load data from a wide variety
of data sources
Qualifications:
Education: MS/MTech/Btech graduates or equivalent with focus on data science and
quantitative fields (CS, Eng, Math, Eco)
Work Experience: Proven 1+ years of experience in data mining (SQL, ETL, data
warehouse, etc.) and using SQL databases

 

Skills
Technical Skills
⮚ Proficient in Python and SQL. Familiarity with statistics or analytical techniques
⮚ Data Warehousing Experience with Big Data Technologies (Hadoop, Hive,
Hbase, Pig, Spark, etc.)
⮚ Working knowledge of tools and utilities - AWS, DevOps with Git, Selenium,
Postman, Airflow, PySpark
Soft Skills
⮚ Deep Curiosity and Humility
⮚ Excellent storyteller and communicator
⮚ Design Thinking

Job posted by
Shalaka ZawarRathi
PySpark
Python
Amazon Web Services (AWS)
Apache Spark
Glue semantics
Apache Kafka
Amazon Redshift
AWS Lambda
icon
Chennai, Bengaluru (Bangalore), Pune, Mumbai, Hyderabad
icon
3 - 10 yrs
icon
₹10L - ₹24L / yr
  • Minimum 1 years of relevant experience, in PySpark (mandatory)
  • Hands on experience in development, test, deploy, maintain and improving data integration pipeline in AWS cloud environment is added plus 
  • Ability to play lead role and independently manage 3-5 member of Pyspark development team 
  • EMR ,Python and PYspark mandate.
  • Knowledge and awareness working with AWS Cloud technologies like Apache Spark, , Glue, Kafka, Kinesis, and Lambda in S3, Redshift, RDS
Job posted by
Swagatika Sahoo

Data Engineer

at Lifesight Technologies Pvt. Ltd.

Founded 2017  •  Product  •  20-100 employees  •  Profitable
Data engineering
Apache Spark
Spark
Big Data
Hadoop
PySpark
Python
Java
Scala
icon
Bengaluru (Bangalore)
icon
2 - 5 yrs
icon
₹10L - ₹25L / yr

Lifesight powers real-time business communications and data solutions that help companies worldwide build better applications and customer experiences by our leading customer intelligence and data platforms that helps brands and enterprises leverage identity resolution and data enrichment to power their customer data strategies like never before. Our industry-leading solution enables the transformation of customer data into actionable insights that help drive business decisions, optimize marketing spend, and improve customer experiences.

 

About The Job

 

Lifesight is growing rapidly and seeking a strong Data Engineer to be a key member of the Enterprise Data & Business Intelligence organization with the focus on data engineering services based in Bengaluru, India. You will be joining as one of the few first data engineers in our Bengaluru office and as the early engineers in the data platform. You will have an opportunity to help define our technical strategy and data engineering team culture in India. 

 

You will design and build data platforms and services, while managing our data infrastructure in cloud environments that fuels strategic business decisions across Lifesight products. 

 

A successful candidate will be a self-starter, who drives excellence, is ready to jump into a variety of big data technologies & frameworks and is able to coordinate and collaborate with other engineers, as well as mentor other engineers in the team.


Why Join us 

  • Be part of building a Zero to One SaaS product
  • Discover amazing career opportunities being initial core team member driving tech strategy of Lifesight.
  • We have a growth plan for everyone
  • Rise to rewarding challenges, enjoy attractive incentives, competitive compensation and more ...
  • Develop your skills - you will enabled with [email protected] program to help you with limitless resources for your learning
  •  

Key Responsibilities

  • Design and implement data management services for data trust, data compliance, data access and metadata management in the form of scalable and configurable while clearly articulating technical rationale behind your design and implementation choices
  • Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
  • Implementing ELT processes
  • Monitoring performance and advising any necessary infrastructure changes
  • Build data platforms and services to handle both real-time and batch processing of large data sets.
  • Take end-to-end ownership of the design and implementation of core product functionality and key technical initiatives.

 

Qualifications

  • 2+ years of data engineering experience is a fast paced company that delivers software
  • A deep understanding of designing and building highly scalable Data warehouses and Data pipelines.
  • Hands on Experience in SQL, Java/Scala programming languages.
  • Deep understanding of Apache Spark, Spark tuning , creating RDDs and building data frames. Create Java/ Scala Spark jobs for data transformation and aggregation.
  • Experience in building distributed environments using any of Kafka, Spark, Hive, Hadoop etc.
  • Good understanding of architecture and functioning of Distributed database systems 
  • Experience working with various file formats like Parquet, Avro etc for large volumes of data
  • Experience with one or more NoSQL databases
  • Strong understanding of engineering best practices and design principles

 

Location

This role will be based in our Bengaluru, India office.

 

WHAT WE CARE ABOUT

 

We like to solve problems, take initiative, pitch in when needed, and are always up for trying new things. What and how you can contribute is what’s important to us. Our consideration is not limited by the kind of education you have or the specific technologies you have experience with. Variety of technical challenges is one of the best things about working @Lifesight as an engineer, but we do not expect you to know every technology we use when you start. What we care about is that you can learn quickly, efficiently manage your goals, contribute to product strategy, and help develop your team as they solve complex problems using the best tools for the job.

 

Our culture runs much deeper than just having fun together (though, we do that well too...) – the people we want on our team are trust-builders, generous givers, scrappy problem solvers, and gritty pursuers of excellence.

 

Does this sound like you? If so, we welcome your application and the chance to meet you.

Job posted by
Anil Singh

Chief Architect

at NA

Agency job
via Talent folks
Java
Python
Big Data
icon
Bengaluru (Bangalore)
icon
15 - 25 yrs
icon
₹10L - ₹15L / yr

Job Description

  • Design, development and deployment of highly-available and fault-tolerant enterprise business software at scale.

  • Demonstrate tech expertise to go very deep or broad in solving classes of problems or creating broadly leverage-able solutions.

  • Execute large-scale projects - Provide technical leadership in architecting and building product solutions.

  • Collaborate across teams to deliver a result, from hardworking team members within your group, through smart technologists across lines of business.

  • Be a role model on acting with good judgment and responsibility, helping teams to commit and move forward.

  • Be a humble mentor and trusted advisor for both our talented team members and passionate leaders alike. Deal with differences in opinion in a mature and fair way.

  • Raise the bar by improving standard methodologies, producing best-in-class efficient solutions, code, documentation, testing, and monitoring.

Qualifications

      • 15+ years of relevant engineering experience.

  • Proven record of building and productionizing highly reliable products at scale.

  • Experience with Java and Python

  • Experience with the Big Data technologie is a plus.

  • Ability to assess new technologies and make pragmatic choices that help guide us towards a long-term vision

  • Can collaborate well with several other engineering orgs to articulate requirements and system design

Additional Information

Professional Attributes:
• Team player!

• Great interpersonal skills, deep technical ability, and a portfolio of successful execution.

• Excellent written and verbal communication skills, including the ability to write detailed technical documents.

• Passionate about helping teams grow by inspiring and mentoring engineers.



Job posted by
Rijooshri Saikia

Data Scientist

at a software product company working on petabyte scale data

Agency job
via RS Consultants
Data Science
Data Scientist
Python
Java
Apache Kafka
pandas
NumPy
Scikit-Learn
Amazon Web Services (AWS)
Go Programming (Golang)
airflow
icon
Pune
icon
7 - 15 yrs
icon
₹30L - ₹50L / yr

We are looking for an exceptional Data Scientist who is passionate about data and motivated to build large scale machine learning solutions. This person will be contributing to the analytics of data for insight discovery and development of machine learning pipeline to support modelling of terabytes of daily data for various use cases

 

Typical persona: Data Science Manager / Architect

 

Experience: 8+ years programming/engineering experience (with at least last 4 years in big data, Data science)

 

Must have:

  • Hands-on Python: Pandas, Scikit-Learn
  • Working knowledge of Kafka
  • Able to carry out own tasks and help the team in resolving problems - logical or technical (25% of job)
  • Good on analytical & debugging skills
  • Strong communication skills

Desired (in order of priorities):

  • Go (Strong advantage)
  • Airflow (Strong advantage)
  • Familiarity & working experience on more than one type of database: relational, object, columnar, graph and other unstructured databases
  • Data structures, Algorithms
  • Experience with multi-threaded and thread sync concepts
  • AWS Sagemaker
  • Keras
  • Should have strong experience in Python programming minimum 4 Years
Job posted by
Rahul Inamdar

Kafka Developer

at Datametica Solutions Private Limited

Founded 2013  •  Products & Services  •  100-1000 employees  •  Profitable
Apache Kafka
Big Data
Hadoop
Apache Hive
Java
Google Cloud Platform (GCP)
icon
Pune, Hyderabad
icon
3 - 12 yrs
icon
₹5L - ₹25L / yr

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

 

Must Have Skills

  • Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
  • Leading in the identification, isolation, resolution and communication of problems within the production environment.
  • Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
  • Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
  • Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
  • Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
  • Works on multiple platforms and multiple projects concurrently.
  • Performs code and unit testing for complex scope modules, and projects
  • Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)
  • Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
  • Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
  • Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
  • Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
  • Working knowledge on Kafka Rest proxy.
  • Ensure optimum performance, high availability and stability of solutions.
  • Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
  • Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
  • Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
  • Ability to perform data related benchmarking, performance analysis and tuning.
  • Strong skills in In-memory applications, Database Design, Data Integration.
Job posted by
Nikita Aher

Data Engineer

at Mobile Programming LLC

Founded 1998  •  Services  •  100-1000 employees  •  Profitable
Data Warehouse (DWH)
Big Data
Spark
Apache Kafka
Data engineering
Python
SQL
Linux/Unix
Shell Scripting
DevOps
CI/CD
Docker
Kubernetes
Java
Scala
Data integration
Google Cloud Platform (GCP)
Kafka
Pentaho
icon
Mohali, Gurugram, Bengaluru (Bangalore), Chennai, Hyderabad, Pune
icon
3 - 8 yrs
icon
₹3L - ₹9L / yr
Day-to-day Activities
Develop complex queries, pipelines and software programs to solve analytics and data mining problems
Interact with other data scientists, product managers, and engineers to understand business problems, technical requirements to deliver predictive and smart data solutions
Prototype new applications or data systems
Lead data investigations to troubleshoot data issues that arise along the data pipelines
Collaborate with different product owners to incorporate data science solutions
Maintain and improve data science platform
Must Have
BS/MS/PhD in Computer Science, Electrical Engineering or related disciplines
Strong fundamentals: data structures, algorithms, database
5+ years of software industry experience with 2+ years in analytics, data mining, and/or data warehouse
Fluency with Python
Experience developing web services using REST approaches.
Proficiency with SQL/Unix/Shell
Experience in DevOps (CI/CD, Docker, Kubernetes)
Self-driven, challenge-loving, detail oriented, teamwork spirit, excellent communication skills, ability to multi-task and manage expectations
Preferred
Industry experience with big data processing technologies such as Spark and Kafka
Experience with machine learning algorithms and/or R a plus 
Experience in Java/Scala a plus
Experience with any MPP analytics engines like Vertica
Experience with data integration tools like Pentaho/SAP Analytics Cloud
Job posted by
Apurva kalsotra

Senior Systems Engineer – Big Data

at Couture.ai

Founded 2017  •  Product  •  20-100 employees  •  Profitable
Big Data
Hadoop
DevOps
Apache Spark
Spark
Shell Scripting
Docker
Kubernetes
Chef
Amberi
icon
Bengaluru (Bangalore)
icon
2 - 5 yrs
icon
₹5L - ₹10L / yr
Skills Requirements
 Knowledge of Hadoop ecosystem installation, initial-configuration and performance tuning.
 Expert with Apache Ambari, Spark, Unix Shell scripting, Kubernetes and Docker
 Knowledge on python would be desirable.
 Experience with HDP Manager/clients and various dashboards.
 Understanding on Hadoop Security (Kerberos, Ranger and Knox) and encryption and Data masking.
 Experience with automation/configuration management using Chef, Ansible or an equivalent.
 Strong experience with any Linux distribution.
 Basic understanding of network technologies, CPU, memory and storage.
 Database administration a plus.
Qualifications and Education Requirements
 2 to 4 years of experience with and detailed knowledge of Core Hadoop Components solutions and
dashboards running on Big Data technologies such as Hadoop/Spark.
 Bachelor degree or equivalent in Computer Science or Information Technology or related fields.
Job posted by
Rajesh Kumar

Machine Learning Engineer

at SmartJoules

Founded 2015  •  Product  •  100-500 employees  •  Profitable
Machine Learning (ML)
Python
Big Data
Apache Spark
Deep Learning
icon
Remote, NCR (Delhi | Gurgaon | Noida)
icon
3 - 5 yrs
icon
₹8L - ₹12L / yr

Responsibilities:

  • Exploring and visualizing data to gain an understanding of it, then identifying differences in data distribution that could affect performance when deploying the model in the real world.
  • Verifying data quality, and/or ensuring it via data cleaning.
  • Able to adapt and work fast in producing the output which upgrades the decision making of stakeholders using ML.
  • To design and develop Machine Learning systems and schemes. 
  • To perform statistical analysis and fine-tune models using test results.
  • To train and retrain ML systems and models as and when necessary. 
  • To deploy ML models in production and maintain the cost of cloud infrastructure.
  • To develop Machine Learning apps according to client and data scientist requirements.
  • To analyze the problem-solving capabilities and use-cases of ML algorithms and rank them by how successful they are in meeting the objective.


Technical Knowledge:


  • Worked with real time problems, solved them using ML and deep learning models deployed in real time and should have some awesome projects under his belt to showcase. 
  • Proficiency in Python and experience with working with Jupyter Framework, Google collab and cloud hosted notebooks such as AWS sagemaker, DataBricks etc.
  • Proficiency in working with libraries Sklearn, Tensorflow, Open CV2, Pyspark,  Pandas, Numpy and related libraries.
  • Expert in visualising and manipulating complex datasets.
  • Proficiency in working with visualisation libraries such as seaborn, plotly, matplotlib etc.
  • Proficiency in Linear Algebra, statistics and probability required for Machine Learning.
  • Proficiency in ML Based algorithms for example, Gradient boosting, stacked Machine learning, classification algorithms and deep learning algorithms. Need to have experience in hypertuning various models and comparing the results of algorithm performance.
  • Big data Technologies such as Hadoop stack and Spark. 
  • Basic use of clouds (VM’s example EC2).
  • Brownie points for Kubernetes and Task Queues.      
  • Strong written and verbal communications.
  • Experience working in an Agile environment.
Job posted by
Saksham Dutta

Data Scientist

at Public Vibe

Founded 2016  •  Product  •  20-100 employees  •  Profitable
Java
Data Science
Python
Natural Language Processing (NLP)
Scala
Hadoop
Spark
kafka
icon
Hyderabad
icon
1 - 3 yrs
icon
₹1L - ₹3L / yr
Hi Candidates, Greetings From Publicvibe !!! We are Hiring NLP Engineers/ Data scientists in between 0.6 to 2.5 Years of Experience for our Hyderabad location, if anyone looking out for opportunities or Job change, reach out to us. Regards, Dhaneesha Dominic.
Job posted by
Dhaneesha Dominic
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at Service based company?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort