Big Data Spark Lead

at Datametica Solutions Private Limited

DP
Posted by Sumangali Desai
icon
Pune, Hyderabad
icon
7 - 12 yrs
icon
₹7L - ₹20L / yr
icon
Full time
Skills
Apache Spark
Big Data
Spark
Scala
Hadoop
MapReduce
Java
Apache Hive
We at Datametica Solutions Private Limited are looking for Big Data Spark Lead who have a passion for cloud with knowledge of different on-premise and cloud Data implementation in the field of Big Data and Analytics including and not limiting to Teradata, Netezza, Exadata, Oracle, Cloudera, Hortonworks and alike.
Ideal candidates should have technical experience in migrations and the ability to help customers get value from Datametica's tools and accelerators.

Job Description
Experience : 7+ years
Location : Pune / Hyderabad
Skills :
  • Drive and participate in requirements gathering workshops, estimation discussions, design meetings and status review meetings
  • Participate and contribute in Solution Design and Solution Architecture for implementing Big Data Projects on-premise and on cloud
  • Technical Hands on experience in design, coding, development and managing Large Hadoop implementation
  • Proficient in SQL, Hive, PIG, Spark SQL, Shell Scripting, Kafka, Flume, Scoop with large Big Data and Data Warehousing projects with either Java, Python or Scala based Hadoop programming background
  • Proficient with various development methodologies like waterfall, agile/scrum and iterative
  • Good Interpersonal skills and excellent communication skills for US and UK based clients

About Us!
A global Leader in the Data Warehouse Migration and Modernization to the Cloud, we empower businesses by migrating their Data/Workload/ETL/Analytics to the Cloud by leveraging Automation.

We have expertise in transforming legacy Teradata, Oracle, Hadoop, Netezza, Vertica, Greenplum along with ETLs like Informatica, Datastage, AbInitio & others, to cloud-based data warehousing with other capabilities in data engineering, advanced analytics solutions, data management, data lake and cloud optimization.

Datametica is a key partner of the major cloud service providers - Google, Microsoft, Amazon, Snowflake.


We have our own products!
Eagle –
Data warehouse Assessment & Migration Planning Product
Raven –
Automated Workload Conversion Product
Pelican -
Automated Data Validation Product, which helps automate and accelerate data migration to the cloud.

Why join us!
Datametica is a place to innovate, bring new ideas to live and learn new things. We believe in building a culture of innovation, growth and belonging. Our people and their dedication over these years are the key factors in achieving our success.

Benefits we Provide!
Working with Highly Technical and Passionate, mission-driven people
Subsidized Meals & Snacks
Flexible Schedule
Approachable leadership
Access to various learning tools and programs
Pet Friendly
Certification Reimbursement Policy

Check out more about us on our website below!
www.datametica.com
Read more

About Datametica Solutions Private Limited

A global Leader in the Data Warehouse Migration and Modernization to the Cloud, we empower businesses by migrating their Data/Workload/ETL/Analytics to the Cloud by leveraging Automation.


We have expertise in transforming legacy Teradata, Oracle, Hadoop, Netezza, Vertica, Greenplum along with ETLs like Informatica, Datastage, AbInitio & others, to cloud-based data warehousing with other capabilities in data engineering, advanced analytics solutions, data management, data lake and cloud optimization.


Datametica is a key partner of the major cloud service providers - Google, Microsoft, Amazon, Snowflake.


We have our own products!

Eagle – Data warehouse Assessment & Migration Planning Product

Raven – Automated Workload Conversion Product

Pelican - Automated Data Validation Product, which helps automate and accelerate data migration to the cloud.


Why join us!

Datametica is a place to innovate, bring new ideas to live and learn new things. We believe in building a culture of innovation, growth and belonging. Our people and their dedication over these years are the key factors in achieving our success.


Benefits we Provide!

Working with Highly Technical and Passionate, mission-driven people

Subsidized Meals & Snacks

Flexible Schedule

Approachable leadership

Access to various learning tools and programs

Pet Friendly

Certification Reimbursement Policy


Check out more about us on our website below!

www.datametica.com

Read more
Founded
2013
Type
Products & Services
Size
100-1000 employees
Stage
Profitable
View full company details
Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Data Engineer

at SaaS Company strive to make selling fun with our SaaS incen

Agency job
via Jobdost
Relational Database (RDBMS)
PostgreSQL
MySQL
Python
Spark
Apache Kafka
Celery
OLTP
OLAP
Amazon Web Services (AWS)
Windows Azure
icon
Bengaluru (Bangalore)
icon
4 - 8 yrs
icon
₹20L - ₹25L / yr

What is the role?

You will be responsible for developing and designing front-end web architecture, ensuring the responsiveness of applications, and working alongside graphic designers for web design features, among other duties. You will be responsible for the functional/technical track of the project

Key Responsibilities

  • Develop and automate large-scale, high-performance data processing systems (batch and/or streaming).
  • Build high-quality software engineering practices towards building data infrastructure and pipelines at scale.
  • Lead data engineering projects to ensure pipelines are reliable, efficient, testable, & maintainable
  • Optimize performance to meet high throughput and scale

What are we looking for?

  • 4+ years of relevant industry experience.
  • Working with data at the terabyte scale.
  • Experience designing, building and operating robust distributed systems.
  • Experience designing and deploying high throughput and low latency systems with reliable monitoring and logging practices.
  • Building and leading teams.
  • Working knowledge of relational databases like Postgresql/MySQL.
  • Experience with Python / Spark / Kafka / Celery
  • Experience working with OLTP and OLAP systems
  • Excellent communication skills, both written and verbal.
  • Experience working in cloud e.g., AWS, Azure or GCP

Whom will you work with?

You will work with a top-notch tech team, working closely with the architect and engineering head.

What can you look for?

A wholesome opportunity in a fast-paced environment that will enable you to juggle between concepts, yet maintain the quality of content, interact and share your ideas and have loads of learning while at work. Work with a team of highly talented young professionals and enjoy the benefits of being at this company

We are

We  strive to make selling fun with our SaaS incentive gamification product.  Company  is the #1 gamification software that automates and digitizes Sales Contests and Commission Programs. With game-like elements, rewards, recognitions, and complete access to relevant information, Company turbocharges an entire salesforce. Company also empowers Sales Managers with easy-to-publish game templates, leaderboards, and analytics to help accelerate performances and sustain growth.

We are a fun and high-energy team, with people from diverse backgrounds - united under the passion of getting things done. Rest assured that you shall get complete autonomy in your tasks and ample opportunities to develop your strengths.

Way forward

If you find this role exciting and want to join us in Bangalore, India, then apply by clicking below. Provide your details and upload your resume. All received resumes will be screened, shortlisted candidates will be requested to join for a discussion and on mutual alignment and agreement, we will proceed with hiring.

 
Read more
Job posted by
Mamatha A
Spark
Hadoop
Big Data
Data engineering
PySpark
Amazon Web Services (AWS)
Scala
Python
icon
Pune, Bengaluru (Bangalore), Hyderabad, Nagpur
icon
4 - 9 yrs
icon
₹4L - ₹15L / yr
Greetings..

We have an urgent requirements of Big Data Developer profiles in our reputed MNC company.

Location: Pune/Bangalore/Hyderabad/Nagpur
Experience: 4-9yrs

Skills: Pyspark,AWS
or Spark,Scala,AWS
or Python Aws
Read more
Job posted by
Haina khan

Data Engineer

at Fintech Company

Agency job
via Jobdost
Python
SQL
Data Warehouse (DWH)
Hadoop
Amazon Web Services (AWS)
DevOps
Git
Selenium
Informatica
ETL
Big Data
Postman
icon
Bengaluru (Bangalore)
icon
2 - 4 yrs
icon
₹7L - ₹12L / yr

Purpose of Job:

Responsible for drawing insights from many sources of data to answer important business
questions and help the organization make better use of data in their daily activities.


Job Responsibilities:

We are looking for a smart and experienced Data Engineer 1 who can work with a senior
manager to
⮚ Build DevOps solutions and CICD pipelines for code deployment
⮚ Build unit test cases for APIs and Code in Python
⮚ Manage AWS resources including EC2, RDS, Cloud Watch, Amazon Aurora etc.
⮚ Build and deliver high quality data architecture and pipelines to support business
and reporting needs
⮚ Deliver on data architecture projects and implementation of next generation BI
solutions
⮚ Interface with other teams to extract, transform, and load data from a wide variety
of data sources
Qualifications:
Education: MS/MTech/Btech graduates or equivalent with focus on data science and
quantitative fields (CS, Eng, Math, Eco)
Work Experience: Proven 1+ years of experience in data mining (SQL, ETL, data
warehouse, etc.) and using SQL databases

 

Skills
Technical Skills
⮚ Proficient in Python and SQL. Familiarity with statistics or analytical techniques
⮚ Data Warehousing Experience with Big Data Technologies (Hadoop, Hive,
Hbase, Pig, Spark, etc.)
⮚ Working knowledge of tools and utilities - AWS, DevOps with Git, Selenium,
Postman, Airflow, PySpark
Soft Skills
⮚ Deep Curiosity and Humility
⮚ Excellent storyteller and communicator
⮚ Design Thinking

Read more
Job posted by
Sathish Kumar

Senior Data Engineer - Big Data

at CodeCraft Technologies Private Limited

Founded 2011  •  Services  •  100-1000 employees  •  Profitable
Data engineering
SQL
Spark
Apache
HiveQL
Big Data
icon
Bengaluru (Bangalore), Mangalore
icon
4 - 8 yrs
icon
Best in industry
Roles and Responsibilities:
• Responsible to Ingest data from files, streams and databases. Process the data with Apache Kafka, Spark, Google
Fire Store, Google BigQuery
• Drive Data Foundation initiatives, like Modelling, Data Quality Management, Data Governance, Data Maturity
Assessments and Data Strategy in support of the key business stakeholders.
• Implementing ETL process using Google BigQuery
• Monitoring performance and advising any necessary infrastructure changes
• Implement scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies
Pyspark, Kafka, Google BigQuery, etc.
• Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing
systems
• Develop efficient software code for multiple use cases leveraging Python and Big Data technologies for various
use cases built on the platform
• Provide high operational excellence guaranteeing high availability and platform stability

Desired Profile:
• Deep understanding of the ecosystem, including ingestion (e.g. Kafka, Kinesis, Apache Airflow), processing
frameworks (e.g. Spark, Flink) and storage engines (e.g. Google FIreStore, Google BigQuery).
• Should have indepth understanding of Bigquery architecture, table partitioning, clustering, best practices, type of
tables, etc.
• Should know how to reduce BigQuery costs by reducing the amount of data processed by your queries
• Practical knowledge of Kafka to build real-time streaming data pipelines and applications that adapt to the data
streams.
• Should be able to speed up queries by using denormalized data structures, with or without nested repeated fields
• Implementing ETL jobs using Bigquery
• Understanding of Bigquery ML
• Knowledge on latest database technologies like MongoDB, Cassandra, Data Bricks etc.
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Experience in GCP and Managed services of GCP
Read more
Job posted by
Priyanka Praveen

Big Data Engineer

at YourHRfolks

Founded 2022  •  Services  •  employees  •  Bootstrapped
Big Data
Hadoop
Spark
Apache Kafka
Amazon Web Services (AWS)
MongoDB
PL/SQL
icon
Remote, Jaipur, NCR (Delhi | Gurgaon | Noida), Chennai, Bangarmau
icon
5 - 10 yrs
icon
₹15L - ₹30L / yr

Position: Big Data Engineer

What You'll Do

Punchh is seeking to hire Big Data Engineer at either a senior or tech lead level. Reporting to the Director of Big Data, he/she will play a critical role in leading Punchh’s big data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners.

This role requires close collaborations with data, engineering, and product organizations. His/her job functions include

  • Work with large data sets and implement sophisticated data pipelines with both structured and structured data.
  • Collaborate with stakeholders to design scalable solutions.
  • Manage and optimize our internal data pipeline that supports marketing, customer success and data science to name a few.
  • A technical leader of Punchh’s big data platform that supports AI and BI products.
  • Work with infra and operations team to monitor and optimize existing infrastructure 
  • Occasional business travels are required.

What You'll Need

  • 5+ years of experience as a Big Data engineering professional, developing scalable big data solutions.
  • Advanced degree in computer science, engineering or other related fields.
  • Demonstrated strength in data modeling, data warehousing and SQL.
  • Extensive knowledge with cloud technologies, e.g. AWS and Azure.
  • Excellent software engineering background. High familiarity with software development life cycle. Familiarity with GitHub/Airflow.
  • Advanced knowledge of big data technologies, such as programming language (Python, Java), relational (Postgres, mysql), NoSQL (Mongodb), Hadoop (EMR) and streaming (Kafka, Spark).
  • Strong problem solving skills with demonstrated rigor in building and maintaining a complex data pipeline.
  • Exceptional communication skills and ability to articulate a complex concept with thoughtful, actionable recommendations.
Read more
Job posted by
Bharat Saxena

Kafka Developer

at Datametica Solutions Private Limited

Founded 2013  •  Products & Services  •  100-1000 employees  •  Profitable
Apache Kafka
Big Data
Hadoop
Apache Hive
Java
Google Cloud Platform (GCP)
icon
Pune, Hyderabad
icon
3 - 12 yrs
icon
₹5L - ₹25L / yr

Summary
Our Kafka developer has a combination of technical skills, communication skills and business knowledge. The developer should be able to work on multiple medium to large projects. The successful candidate will have excellent technical skills of Apache/Confluent Kafka, Enterprise Data WareHouse preferable GCP BigQuery or any equivalent Cloud EDW and also will be able to take oral and written business requirements and develop efficient code to meet set deliverables.

 

Must Have Skills

  • Participate in the development, enhancement and maintenance of data applications both as an individual contributor and as a lead.
  • Leading in the identification, isolation, resolution and communication of problems within the production environment.
  • Leading developer and applying technical skills Apache/Confluent Kafka (Preferred) AWS Kinesis (Optional), Cloud Enterprise Data Warehouse Google BigQuery (Preferred) or AWS RedShift or SnowFlakes (Optional)
  • Design recommending best approach suited for data movement from different sources to Cloud EDW using Apache/Confluent Kafka
  • Performs independent functional and technical analysis for major projects supporting several corporate initiatives.
  • Communicate and Work with IT partners and user community with various levels from Sr Management to detailed developer to business SME for project definition .
  • Works on multiple platforms and multiple projects concurrently.
  • Performs code and unit testing for complex scope modules, and projects
  • Provide expertise and hands on experience working on Kafka connect using schema registry in a very high volume environment (~900 Million messages)
  • Provide expertise in Kafka brokers, zookeepers, KSQL, KStream and Kafka Control center.
  • Provide expertise and hands on experience working on AvroConverters, JsonConverters, and StringConverters.
  • Provide expertise and hands on experience working on Kafka connectors such as MQ connectors, Elastic Search connectors, JDBC connectors, File stream connector,  JMS source connectors, Tasks, Workers, converters, Transforms.
  • Provide expertise and hands on experience on custom connectors using the Kafka core concepts and API.
  • Working knowledge on Kafka Rest proxy.
  • Ensure optimum performance, high availability and stability of solutions.
  • Create topics, setup redundancy cluster, deploy monitoring tools, alerts and has good knowledge of best practices.
  • Create stubs for producers, consumers and consumer groups for helping onboard applications from different languages/platforms.  Leverage Hadoop ecosystem knowledge to design, and develop capabilities to deliver our solutions using Spark, Scala, Python, Hive, Kafka and other things in the Hadoop ecosystem. 
  • Use automation tools like provisioning using Jenkins, Udeploy or relevant technologies
  • Ability to perform data related benchmarking, performance analysis and tuning.
  • Strong skills in In-memory applications, Database Design, Data Integration.
Read more
Job posted by
Nikita Aher

Data Engineer

at Prescience Decision Solutions

Founded 2017  •  Products & Services  •  20-100 employees  •  Profitable
Big Data
ETL
Spark
Apache Kafka
Apache Spark
Python
SQL
Java
Databricks
icon
Bengaluru (Bangalore)
icon
3 - 7 yrs
icon
₹10L - ₹20L / yr

The Data Engineer would be responsible for selecting and integrating Big Data tools and frameworks required. Would implement Data Ingestion & ETL/ELT processes

Required Experience, Skills and Qualifications:

  • Hands on experience on Big Data tools/technologies like Spark,  Databricks, Map Reduce, Hive, HDFS.
  • Expertise and excellent understanding of big data toolset such as Sqoop, Spark-streaming, Kafka, NiFi
  • Proficiency in any of the programming language: Python/ Scala/  Java with 4+ years’ experience
  • Experience in Cloud infrastructures like MS Azure, Data lake etc
  • Good working knowledge in NoSQL DB (Mongo, HBase, Casandra)
Read more
Job posted by
Shivakumar K

Data Engineer

at Our client company is into Analytics. (RF1)

Agency job
via Multi Recruit
Data Engineer
Big Data
Python
Amazon Web Services (AWS)
SQL
Java
ETL
icon
Bengaluru (Bangalore)
icon
3 - 5 yrs
icon
₹12L - ₹14L / yr
  •  We are looking for a Data Engineer with 3-5 years experience in Python, SQL, AWS (EC2, S3, Elastic Beanstalk, API Gateway), and Java.
  • The applicant must be able to perform Data Mapping (data type conversion, schema harmonization) using Python, SQL, and Java.
  • The applicant must be familiar with and have programmed ETL interfaces (OAUTH, REST API, ODBC) using the same languages.
  • The company is looking for someone who shows an eagerness to learn and who asks concise questions when communicating with teammates.
Read more
Job posted by
Ragul Ragul

Hadoop Developer

at Object Technology Solutions Inc. (OTSI)

Founded 1999  •  Services  •  100-1000 employees  •  Profitable
Scala
Big Data
Spark
Hadoop
icon
NCR (Delhi | Gurgaon | Noida)
icon
3 - 5 yrs
icon
₹2L - ₹8L / yr

Dear Candidate,,        

 

Greetings of the day!

As discussed, Please find the below job description.

 

Job Title           :            Hadoop developer

Experience      :            3+ years

Job Location   :            New Delhi

Job type           :            Permanent

 

Knowledge and Skills Required:

 

Brief Skills:

Hadoop, Spark, Scala and Spark SQL

 

Main Skills:

  1. Strong experience in Hadoop development
  2. Experience in Spark
  3. Experience in Scala
  4. Experience in Spark SQL


Why OTSi!

Working with OTSi gives you the assurance of a successful, fast-paced career.

Exposure to infinite opportunities to learn and grow, familiarization with cutting-edge technologies, cross-domain experience and a harmonious environment are some of the prime attractions for a career-driven workforce.

Join us today, as we assure you 2000+ friends and a great career; Happiness begins at a great workplace..!

Feel free to refer this opportunity to your friends and associates.

                                                                                                                                                                           

About OTSI: (CMMI Level 3): Founded in 1999 and headquartered in Overland Park, Kansas, OTSI offers global reach and local delivery to companies of all sizes, from start-ups to Fortune 500s. Through offices across the US and around the world, we provide universal access to exceptional talent and innovative solutions in a variety of delivery models to reduce overall risk while optimizing outcomes & enabling our customers to thrive in a global economy.
OTSI's global presence, scalable and sustainable world-class infrastructure, business continuity processes, ISO 9001:2000, CMMI 3 certifications makes us a preferred service provider for our clients. OTSI has the expertise in different technologies enhanced by our partnerships and alliances with industry giants like HP, Microsoft, IBM, Oracle, and SAP and others. Highly repetitive local company with a proven success of serving the UAE Government IT needs is seeking to attract, employ and develop people with exceptional skills who want to make a difference in a challenging environment.Object Technology Solutions India Pvt Ltd is a leading Global Information Technology (IT) Services and Solutions company offering a wide array of Solutions for a range of key Verticals. The company is headquartered in Overland Park, Kansas, and has a strong presence in US, Europe and Asia-Pacific with a Global Delivery Center based in India. OTSI offers a broad range of IT application solutions and services including; e-Business solutions, Enterprise Resource Planning (ERP) implementation and Post Implementation Support, Application development, Application Maintenance, Software customizations services.

 

OTSI Partners & Practices

  • SAP Partner
  • Microsoft Silver Partner
  • Oracle Gold Partner
  • Microsoft CoE
  • DevOps Consulting
  • Cloud
  • Mobile & IoT
  • Digital Transformation
  • Big data & Analytics
  • Testing Solutions

 

OTSI Honor’s & Awards:

  • #91 in Inc.5000 .
  • Fastest growing IT Companies in Inc.5000…
Read more
Job posted by
Uma Nageswara Rao Guggilam

Data engineer - Machine learning

at It is India’s biggest vernacular e-sports gaming platform.

Machine Learning (ML)
Data Structures
Data engineering
Big Data
Neural networks
icon
NCR (Delhi | Gurgaon | Noida)
icon
3 - 7 yrs
icon
₹12L - ₹34L / yr
• Experience with Big Data, Neural network (deep learning), and reinforcement learning • Ability to design machine learning systems • Research and implement appropriate ML algorithms and tools • Develop machine learning applications according to requirements • Select appropriate datasets and data representation methods • Run machine learning tests and experiments • Perform statistical analysis and fine-tuning using test results • Extend existing ML libraries and frameworks • Keep abreast of developments in the field • Understanding of data structures, data modeling and software architecture • Deep knowledge of math, probability, statistics and algorithms • Ability to write robust code in Python, Java and R Familiarity with machine learning frameworks (like Keras or PyTorch) and libraries (like scikit-learn)
Read more
Job posted by
Silky Malik
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at Datametica Solutions Private Limited?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort