i

Big Data Engineer

i
Posted by Vijay Hemnath
Apply to this job
i
Chennai
i
2 - 5 yrs
i
₹6L - ₹25L / yr (ESOP available)
Skills
Big Data
Datawarehousing
Scala
Machine Learning (ML)
Deep Learning
SQL
Data modeling
Hadoop
Spark
Apache Hive
PySpark
Python
Amazon Web Services (AWS)
Java
Cassandra
DevOps
HDFS
Job description

We are looking for an outstanding Big Data Engineer with experience setting up and maintaining Data Warehouse and Data Lakes for an Organization. This role would closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Roles and Responsibilities:

  • Develop and maintain scalable data pipelines and build out new integrations and processes required for optimal extraction, transformation, and loading of data from a wide variety of data sources using 'Big Data' technologies.
  • Develop programs in Scala and Python as part of data cleaning and processing.
  • Assemble large, complex data sets that meet functional / non-functional business requirements and fostering data-driven decision making across the organization.  
  • Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
  • Implement processes and systems to validate data, monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
  • Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Provide high operational excellence guaranteeing high availability and platform stability.
  • Closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Skills:

  • Experience with Big Data pipeline, Big Data analytics, Data warehousing.
  • Experience with SQL/No-SQL, schema design and dimensional data modeling.
  • Strong understanding of Hadoop Architecture, HDFS ecosystem and eexperience with Big Data technology stack such as HBase, Hadoop, Hive, MapReduce.
  • Experience in designing systems that process structured as well as unstructured data at large scale.
  • Experience in AWS/Spark/Java/Scala/Python development.
  • Should have Strong skills in PySpark (Python & SPARK). Ability to create, manage and manipulate Spark Dataframes. Expertise in Spark query tuning and performance optimization.
  • Experience in developing efficient software code/frameworks for multiple use cases leveraging Python and big data technologies.
  • Prior exposure to streaming data sources such as Kafka.
  • Should have knowledge on Shell Scripting and Python scripting.
  • High proficiency in database skills (e.g., Complex SQL), for data preparation, cleaning, and data wrangling/munging, with the ability to write advanced queries and create stored procedures.
  • Experience with NoSQL databases such as Cassandra / MongoDB.
  • Solid experience in all phases of Software Development Lifecycle - plan, design, develop, test, release, maintain and support, decommission.
  • Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development).
  • Experience building and deploying applications on on-premise and cloud-based infrastructure.
  • Having a good understanding of machine learning landscape and concepts. 

 

Qualifications and Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Big Data Engineer or a similar role for 3-5 years.

Certifications:

Good to have at least one of the Certifications listed here:

    AZ 900 - Azure Fundamentals

    DP 200, DP 201, DP 203, AZ 204 - Data Engineering

    AZ 400 - Devops Certification

About Netmeds.com
undefined
Founded
2015
Type
Product
Size
500-1000 employees
Stage
Raised funding
Why apply to jobs via CutShort
i
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
i
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
i
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
6212
Companies hiring
Similar jobs
i
at Client is a Machine Learning company based in New Delhi.
Agency job
Data Science
R Programming
Python
Machine Learning (ML)
Entity Framework
Natural Language Processing (NLP)
Computer Vision
i
NCR (Delhi | Gurgaon | Noida)
i
2 - 6 yrs
i
₹10L - ₹25L / yr

Job Responsibilities

  • Design machine learning systems
  • Research and implement appropriate ML algorithms and tools
  • Develop machine learning applications according to requirements
  • Select appropriate datasets and data representation methods
  • Run machine learning tests and experiments
  • Perform statistical analysis and fine-tuning using test results
  • Train and retrain systems when necessary

 

Requirements for the Job

 

  1. Bachelor’s/Master's/PhD in Computer Science, Mathematics, Statistics or equivalent field andmust have a minimum of 2 years of overall experience in tier one colleges 
  1. Minimum 1 year of experience working as a Data Scientist in deploying ML at scale in production
  2. Experience in machine learning techniques (e.g. NLP, Computer Vision, BERT, LSTM etc..) andframeworks (e.g. TensorFlow, PyTorch, Scikit-learn, etc.)
  1. Working knowledge in deployment of Python systems (using Flask, Tensorflow Serving)
  2. Previous experience in following areas will be preferred: Natural Language Processing(NLP) - Using LSTM and BERT; chatbots or dialogue systems, machine translation, comprehension of text, text summarization.
  3. Computer Vision - Deep Neural Networks/CNNs for object detection and image classification, transfer learning pipeline and object detection/instance segmentation (Mask R-CNN, Yolo, SSD).
Read more
Job posted by
i
Sathish Kumar
Apply for job
i
Founded 2016  •  Products & Services  •  20-100 employees  •  Raised funding
C
CUDA
C++
Python
OpenCV
TensorFlow
PyTorch
i
Hyderabad
i
3 - 6 yrs
i
₹6L - ₹25L / yr

We are seeking Machine Learning Engineers to join our engineering team, with C/C++ background to work on TensorRT, CuDNN, PyTorch C++ API, and good knowledge of handling databases. The position will involve taking these skills and applying them to real-world problems.


Responsibility: 

  • Build computer vision algorithms on resource constrained devices
  • Work on cutting edge problems in Deep Learning for Internal AI Accelerator SW Stack
  • Develop & integrate functional and performance models of accelerators
  • Analyzing the accuracy of Neural Network on Functional models and correlate with HW implementation
  • Be part of discussions in defining the next gen HW accelerators
  • Research on various numerics related to Machine Learning, optimizing it based on design and performance constraints 

Essential

  • Experience in embedded Computer Vision (Open CV), SIMD, and parallel computing, with a deep understanding of CV algorithms and multimedia image formats
  • Fluent in working with Python
  • Fluent in C and C++ as well as experience in CUDA
  • Efficient in SW development in Linux, with a deep understanding of operating system e.g. Linux, 
  • Good knowledge in SoCs e.g. Tegra, with efficient use of Software development tools like debuggers
  • Excellent written and verbal interpersonal skills and an eye for detail 
  • Good organization skills, with a logical approach to problem-solving, time management and ability to prioritize 

Ways to stand out from the crowd:

  • Experience with visual geometry and deep learning in a shipping product context
  • Worked on real-time Image Processing and/or computer vision systems
  • Software development on embedded platforms or large scale cloud services
  • Experience with GPGPU programming (CUDA and OpenGL)
  • Worked on atleast one mainstream deep learning frameworks, including TensorFlow, Caffe(2), MXNet, PyTorch
Read more
Job posted by
i
Aditi Pandya
Apply for job
i
Founded 2018  •  Services  •  employees  •  Bootstrapped
MicroStrategy
SQL
Business Intelligence (BI)
i
Remote, Bengaluru (Bangalore)
i
5 - 10 yrs
i
₹5L - ₹25L / yr
Microstrategy. Having knowledge on Database like SQL. And also knowledge on BI concepts.Strong communication skills.
Read more
Job posted by
i
Swagatika Sahoo
Apply for job
i
Founded 2017  •  Product  •  100-500 employees  •  Raised funding
Data Science
Python
Image Processing
CNN
Convolution neural network
OpenCV
Deep Learning
i
Nagpur, Hyderabad
i
- yrs
i
₹5L - ₹25L / yr

Responsibilities: 
 

  • The Machine & Deep Machine Learning Software Engineer (Expertise in Computer Vision) will be an early member of a growing team with responsibilities for designing and developing highly scalable machine learning solutions that impact many areas of our business. 
  • The individual in this role will help in the design and development of Neural Network (especially Convolution Neural Networks) & ML solutions based on our reference architecture which is underpinned by big data & cloud technology, micro-service architecture and high performing compute infrastructure. 
  • Typical daily activities include contributing to all phases of algorithm development including ideation, prototyping, design, and development production implementation. 


Required Skills: 
 

  • An ideal candidate will have a background in software engineering and data science with expertise in machine learning algorithms, statistical analysis tools, and distributed systems. 
  • Experience in building machine learning applications, and broad knowledge of machine learning APIs, tools, and open-source libraries 
  • Strong coding skills and fundamentals in data structures, predictive modeling, and big data concepts 
  • Experience in designing full stack ML solutions in a distributed computing environment 
  • Experience working with Python, Tensor Flow, Kera’s, Sci-kit, pandas, NumPy, AZURE, AWS GPU
  • Excellent communication skills with multiple levels of the organization 
  • Image CNN, Image processing, MRCNN, FRCNN experience is a must.
Read more
Job posted by
i
KalyaniMuley
Apply for job
i
at installation and deployment of business automation systems
Agency job
Python
ETL
SQL
Amazon Web Services (AWS)
Airflow
i
Pune, Mumbai, Bengaluru (Bangalore)
i
5 - 10 yrs
i
₹8L - ₹18L / yr
Job Title: DBA MongoDB

Work Location: Pune/Mumbai/Bangalore

Experience:5+yrs

Package:Open

Notice Period:Immediate Joiners

Its a Full Time Opportunity with Our Client

Mandatory Skills:Airflow/Composer,Python,ETL Developer,SQL & GCP/AWS

Job Description:
--3+ years of Airflow/Composer experience
--Strong proficiency in Python with an emphasis in building data pipelines
--Ability to write complex SQL to perform common types of analysis and aggregations
--Strong ELT/ ETL designer/developer with 5+ years’ experience
--Strong SQL & Python Expert level with 5+ years’ experience
--Performance tuning with SQL
--Structured & unstructured data expertise
--Cloud environment development & operations experience (GCP, AWS)
Read more
Job posted by
i
Venkat B
Apply for job
i
at PayU
Founded 2002  •  Product  •  500-1000 employees  •  Profitable
PySpark
SQL
SQL Azure
Amazon Web Services (AWS)
Python
i
Remote, Bengaluru (Bangalore)
i
5 - 10 yrs
i
₹10L - ₹40L / yr

 

 

We are looking to add a Lead BigData Engineer to our team, Would love to connect and tell you more about what were building, and to learn more about what interests you. 

 

Company Name : PayU - Kindly visit our company site for more details .

 

Location : Bangalore, Mumbai & Gurgaon

 

Role and Background Information:

As a Lead/Manager - Data Engineering, you will execute the strategy and roadmap for data engineering at PayU, you will define, evolve and mature PayU's existing data platform and processing framework that handles terabytes of data from our various properties.

You will work with a team of data engineers, analytics and data scientist teams to identify the needs and contribute towards building the next generation data ecosystem.

You will engage with analysts and leaders to research and develop new data engineering capabilities.

Responsible to ingest data from files, streams and databases. Process the data with Python and Pyspark and its storage into time-series database.

Develop programs in Python as part of data extraction, data cleaning, transformation and processing. Develop and maintain scalable data pipelines.  

Rest API’s development.

 

What you’d need to bring to the table:  

Overall 5-10 years of experience as Data Engineer.

Advanced working SQL knowledge to create complex queries.

Experience in working with Time-series Database, relational databases as well as working familiarity with a variety of databases (structured & Unstructured).

Hands On experience on visualization tools like Grafana & Power BI.

Experience in designing and implementing scalable architecture.

Good experience in doing object-oriented programming in python.

Very strong in Object-Oriented Analysis and Design (OOAD).

Strong knowledge on REST APIs.

 Experience working on Azure Cloud services IaaS PaaS.

Hands on Experience in working on Microsoft Azure Services like ADLS/Blob Storage solutions, Event Hubs, Service Bus, scale sets, Load Balancers, Azure Functions, Databricks. - Hands on Experience in working on Kafka.

Knowledge on continuous integration/continuous deployment.

Experience of data migration and deployment from On-Prem to Cloud environment and vice-versa.

 Individual Contributor.

Education-

Bachelor of Engineering (IIT/NIT/Bits is Preferred)

Read more
Job posted by
i
Vishakha Sonde
Apply for job
i
Founded 2011  •  Product  •  100-500 employees  •  Profitable
Machine Learning (ML)
Deep Learning
OpenCV
i
Chennai
i
8 - 12 yrs
i
₹20L - ₹35L / yr

About Toyota Connected

If you want to change the way the world works, transform the automotive industry and positively impact others on a global scale, then Toyota Connected is the right place for you! Within our collaborative, fast-paced environment we focus on continual improvement and work in a highly iterative way to deliver exceptional value in the form of connected products and services that wow and delight our customers and the world around us.

 

About the Team

Toyota Connected India is hiring talented engineers at Chennai to use Deep Learning, Computer vision, Big data, high performance cloud-based services and other cutting-edge technologies to transform the customer experience with their vehicle. Come help us re-imagine what mobility can be today and for years to come!


 

Job Description

The Toyota Connected team is looking for a Senior ML Engineer (Computer Vision) to be a part of a highly talented engineering team to help create new products and services from the ground up for the next generation connected vehicles. We are looking for team members that are required to be creative in solving problems, excited to work in new technology areas and be ready to wear multiple hats to get things done in a highly energized, fast-paced, innovative and collaborative startup environment.


 

What you will do

• Develop solutions using Machine Learning/Deep Learning and other advanced technologies to solve a variety of problems

• Develop image analysis algorithms and deep learning architectures to solve Computer vision related problems.

• Implement cutting edge machine learning techniques in image classification, object detection, semantic segmentation, sequence modeling, etc. using frameworks such as OpenCV, TensorFlow and Pytorch.

• Translate user stories and business requirements to technical solutions by building quick prototypes or proof of concepts with several business and technical stakeholder groups in both internal and external organizations

• Partner with leaders in the area and have insights to select off the shelf components vs building from the scratch

• Convert the proof of concepts to production-grade solutions that can scale for hundreds of thousands of users

• Be hands-on where required and lead from the front in following best practices in development and CI/CD methods

• Own delivery of features from top to bottom, from concept to code to production

• Develop tools and libraries that will enable rapid and scalable development in the future

You are a successful candidate if

• You are smart and can demonstrate it

• You have 8+ years of experience as software engineer with minimum 3 years hands-on experience delivering products or solutions that utilized Computer Vision

• Strong experience in deploying solutions to production, with hands-on experience in any public cloud environment (AWS, GCP or Azure)

• Excellent proficiency in Open CV or related computer vision frameworks and libraries

• Mathematical understanding of a variety of statistical learning algorithms (Reinforcement Learning, Supervised/Unsupervised, Graphical Models)

• Expertise in a variety of Deep Learning architectures including Residual Networks, RNN/CNN, Transformer, and Transfer Learning. And experience in delivering value using these in real production environments for real customers

• You have deep proficiency in Python and at least one other major programming language (C++, Java, Golang)

• You are very fluent in one or more ML tools/libraries like Tensorflow, Pytorch, Caffe, and/or Theano and have solved several real-life problems using these

• We think the knowledge acquired earning a degree in Computer Science or Math would be of great value in this position, but if you're smart and have the experience that backs up your abilities, for us, talent trumps degree every time


 

What is in it for you? 

• Top of the line compensation!

• You'll be treated like the professional we know you are and left to manage your own time and work load.

• Yearly gym membership reimbursement. & Free catered lunches. 

• No dress code! We trust you are responsible enough to choose what’s appropriate to wear for the day.

• Opportunity to build products that improves the safety and convenience of millions of customers. 
Our Core Values

  • Empathetic: We begin making decisions by looking at the world from the perspective of our customers, teammates, and partners.
  • Passionate: We are here to build something great, not just for the money. We are always looking to improve the experience of our millions of customers
  • Innovative: We experiment with ideas to get to the best solution. Any constraint is a challenge, and we love looking for creative ways to solve them.
  • Collaborative: When it comes to people, we think the whole is greater than its parts and that everyone has a role to play in the success!
  •  
Read more
Job posted by
i
Suganya R
Apply for job
i
Founded 2002  •  Services  •  100-1000 employees  •  Profitable
Microsoft SSIS
ETL
SQL Server Integration Services (SSIS)
SQL
SQL server
i
Mumbai
i
2 - 4 yrs
i
₹4L - ₹8L / yr
Should be able to connect with multiple data sources like excel, csv, SQL server oracle etc.
 Should be able to use the transformations components to transform the data
 Should possess knowledge on incremental load, full load etc.
 Should Design, build and deploy effective packages
 Should be able to schedule these packages through task schedulers
 Implement stored procedures and effectively query a database
 Translate requirements from the business and analyst into technical code
 Identify and test for bugs and bottlenecks in the ETL solution
 Ensure the best possible performance and quality in the packages
 Provide support and fix issues in the packages
 Writes advanced SQL including some query tuning
 Experience in the identification of data quality
 Some database design experience is helpful
 Experience designing and building complete ETL/SSIS processes moving and transforming data for
ODS, Staging, and Data Warehousing
Read more
Job posted by
i
Sweety Patyal
Apply for job
i
Founded 2008  •  Products & Services  •  20-100 employees  •  Profitable
Machine Learning (ML)
Python
Data Science
i
Pune
i
- yrs
i
₹7L - ₹15L / yr
We are looking for a Machine Learning Engineer with 3+ years of experience with a background in Statistics and hands-on experience in the Python ecosystem, using sound  Software Engineering practices. Skills & Knowledge: - Formal knowledge of fundamentals of probability & statistics along with the ability to apply basic statistical analysis methods like hypothesis testing, t-tests, ANOVA etc. - Hands-on knowledge of data formats, data extraction, loading, wrangling, transformation, pre-processing and analysis. - Thorough understanding of data-modeling and machine-learning concepts - Complete understanding and ability to apply, implement and adapt standard implementations of machine learning algorithms - Good understanding and ability to apply and adapt Neural Networks and Deep Learning, including common high-level Deep Learning architectures like CNNs and RNNs - Fundamentals of computer science & programming, especially Data structures (like multi-dimensional arrays, trees, and graphs) and Algorithms (like searching, sorting, and dynamic programming) - Fundamentals of software engineering and system design, such as requirements analysis, REST APIs, database queries, system and library calls, version control, etc. Languages and Libraries: - Hands-on experience with Python and Python Libraries for data analysis and machine learning, especially Scikit-learn, Tensorflow, Pandas, Numpy, Statsmodels, and Scipy. - Experience with R and its ecosystem is a plus - Knowledge of other open source machine learning and data modeling frameworks like Spark MLlib, H2O, etc. is a plus
Read more
Job posted by
i
Juzar Malubhoy
Apply for job
i
at A Fintech startup in Dubai
Agency job
via Jobbie
Data Science
R Programming
Python
i
Remote, Dubai, Bengaluru (Bangalore), Mumbai
i
- yrs
i
₹14L - ₹38L / yr
RESPONSIBILITIES AND QUALIFICATIONS The mission of the Marcus Surveillance Analytics team is to deliver a platform which detects security incidents which have a tangible business impact and actionable response. You will work alongside industry leading technologists from who have recently joined Goldman from across consumer security, technology, fintech, finance and quant firms. The role has a broad scope which will involve interacting with senior leaders of Goldman and the Consumer business on a regular basis. The position is hands-on and requires a driven and “take ownership” oriented individual who is intently focused on execution. You will work directly with developers, business leaders, vendors and partners in order to deliver security assets to the consumer business. Develop a team, vision and platform which identifies/prioritizes actionable security & fraud risks which have tangible businesses impact across Goldman's consumer and commercial banking businesses. Develop response and recovery technology and programs to ensure resilience from fraud and abuse events. Manage, develop and operationalize analytics which discover security & fraud events and identifies risks for all of Goldman's consumer businesses. Partner with fraud / abuse operations and leadership to ensure consumer fraud rates are within industry norms and own outcomes related to fraud improvements. Skills And Experience We Are Looking For BA/BS degree in Computer Science, Cybersecurity, or other relevant Computer/Data/Engineering degrees 2+ years of experience as a security professional or data analyst/scientist/engineer Python, PySpark, R, Bash, SQL, Splunk (search, ES, UBA) Experience with cloud infrastructure/big data tool sets Visualization tools such as Tableau or D3 Research and development to create innovative predictive detections for security and fraud Build a 24/7 real-time monitoring system with long term vision for scaling to new lines of consumer businesses Strong focus on customer experience and product usability Ability to work closely with the business, fraud, and security incident response teams on creating actionable detections
Read more
Job posted by
i
Sourav Nandi
Apply for job
Did not find a job you were looking for?
i
Search
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on CutShort.
iiiii
Want to apply for this role at Netmeds.com?
i
Apply for this job
Why apply via CutShort?
Connect with actual hiring teams and get their fast response. No spam.