Data Engineer - Azure

at Global consulting company

Agency job
icon
Pune
icon
2 - 5 yrs
icon
₹10L - ₹30L / yr
icon
Full time
Skills
SQL Azure
Data migration
Windows Azure
Big Data
PySpark
Relational Database (RDBMS)
ETL
Amazon Web Services (AWS)

2-4 years of experience in developing ETL activities for Azure – Big data, relational databases, and data warehouse solutions.

 

Extensive hands-on experience implementing data migration and data processing using Azure services: ADLS, Azure Data Factory, Azure Functions, Synapse/DW, Azure SQL DB, Azure Analysis Service, Azure Databricks, Azure Data Catalog, ML Studio, AI/ML, Snowflake, etc.

 

Well versed in DevOps and CI/CD deployments

 

Cloud migration methodologies and processes including tools like Azure Data Factory, Data Migration Service, SSIS, etc.

 

Minimum of 2 years of RDBMS experience

 

Experience with private and public cloud architectures, pros/cons, and migration considerations.

 

Nice-to-Have Skills/Qualifications:

 

- DevOps on an Azure platform

- Experience developing and deploying ETL solutions on Azure

- IoT, event-driven, microservices, Containers/Kubernetes in the cloud

- Familiarity with the technology stack available in the industry for metadata management: Data Governance, Data Quality, MDM, Lineage, Data Catalog etc.

- Multi-cloud experience a plus - Azure, AWS, Google

   

Professional Skill Requirements

Proven ability to build, manage and foster a team-oriented environment

Proven ability to work creatively and analytically in a problem-solving environment

Desire to work in an information systems environment

Excellent communication (written and oral) and interpersonal skills

Excellent leadership and management skills

Excellent organizational, multi-tasking, and time-management skills

Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Junior Data Scientist

at Fintech Company

Agency job
via Jobdost
Python
SQL
Machine Learning (ML)
Data Science
Natural Language Processing (NLP)
Computer Vision
Amazon Web Services (AWS)
Git
icon
Bengaluru (Bangalore)
icon
1 - 3 yrs
icon
₹9L - ₹12L / yr
Job Responsibilities:

1+ years of proven experience in ML/AI with Python

Work with the manager through the entire analytical and machine learning model life cycle:

⮚ Define the problem statement

⮚ Build and clean datasets

⮚ Exploratory data analysis

⮚ Feature engineering

⮚ Apply ML algorithms and assess the performance

⮚ Codify for deployment

⮚ Test and troubleshoot the code

⮚ Communicate analysis to stakeholders

Technical Skills

⮚ Proven experience in usage of Python and SQL

⮚ Excellent in programming and statistics

⮚ Working knowledge of tools and utilities - AWS, DevOps with Git, Selenium, Postman, Airflow, PySpark

Job posted by
Shalaka ZawarRathi

Machine Learning Engineer

at Top Management Consulting Company

Data Science
Machine Learning (ML)
Natural Language Processing (NLP)
Computer Vision
PySpark
Python
icon
Gurugram, Bengaluru (Bangalore)
icon
2 - 9 yrs
icon
₹10L - ₹38L / yr
Greetings!!

We are looking for a Machine Learning engineer for on of our premium client.
Experience: 2-9 years
Location: Gurgaon/Bangalore
Tech Stack:

Python, PySpark, the Python Scientific Stack; MLFlow, Grafana, Prometheus for machine learning pipeline management and monitoring; SQL, Airflow, Databricks, our own open-source data pipelining framework called Kedro, Dask/RAPIDS; Django, GraphQL and ReactJS for horizontal product development; container technologies such as Docker and Kubernetes, CircleCI/Jenkins for CI/CD, cloud solutions such as AWS, GCP, and Azure as well as Terraform and Cloudformation for deployment
Job posted by
Naveed Mohd

Database Developer

at Mobile Programming LLC

Founded 1998  •  Services  •  100-1000 employees  •  Profitable
PySpark
Data engineering
Big Data
Hadoop
Spark
Databases
Google Cloud Platform (GCP)
icon
Remote only
icon
4 - 7 yrs
icon
₹10L - ₹14L / yr

Job ID: RP100

Work Location: Remote

Required Experience: 4 to 7 years

Job Description

  • Must have Google Cloud Big Query experience
  • Strong experience with data analysis, data modeling and governance, with excellent analytical and problem-solving abilities
  • Good knowledge of Data Warehouses, data flow ETL pipelines
  • Design, configuration/administration of database software in Cloud platform.
  • Monitoring, Troubleshooting, and Performance tuning the DB objects.
  • Experience on Table Partition, Clustered Table, Materialized View, External Tables etc.

Anyone RDBMS technologies

  • Good experience in DB design with knowledge of ER Diagram, PK/FK, Stored procedure, Function, Triggers, and Indexes.
  • Understanding the requirement of the App team and creating the necessary DB objects by following the best practices.
  • Managing logins and database users, as well as database roles, application roles, and other security principles within the database.
  • Deep knowledge about Indexes, Performance tuning, and Complex SQL Query patterns.
  • Monitoring, Tuning, and Troubleshooting the database-related issues.

About Us:

Mobile programming LLC is a US-based digital transformation company. We help enterprises transform ideas into innovative and intelligent solutions, governing the Internet of Things, Digital Commerce, Business Intelligence Analytics, and Cloud Programming. Bring your challenges to us, we will give you the smartest solutions. From conceptualizing and engineering to advanced manufacturing, we help customers build and scale products fit for the global marketplace.

Mobile programming LLC has offices located in Los Angeles, San Jose, Glendale, San Diego, Phoenix, Plano, New York, Fort Lauderdale, and Boston. Mobile programming is SAP Preferred Vendor, Apple Adjunct Partner, Google Empaneled Mobile Vendor, and Microsoft Gold Certified Partner.

Job posted by
Garima Walia

Data Engineer (Azure)

at Scry Analytics

Founded 2015  •  Product  •  100-500 employees  •  Profitable
PySpark
Data engineering
Big Data
Hadoop
Spark
Windows Azure
Amazon Web Services (AWS)
Google Cloud Platform (GCP)
SQL
NOSQL Databases
Apache Kafka
icon
Remote only
icon
3 - 8 yrs
icon
₹15L - ₹20L / yr

Title: Data Engineer (Azure) (Location: Gurgaon/Hyderabad)

Salary: Competitive as per Industry Standard

We are expanding our Data Engineering Team and hiring passionate professionals with extensive

knowledge and experience in building and managing large enterprise data and analytics platforms. We

are looking for creative individuals with strong programming skills, who can understand complex

business and architectural problems and develop solutions. The individual will work closely with the rest

of our data engineering and data science team in implementing and managing Scalable Smart Data

Lakes, Data Ingestion Platforms, Machine Learning and NLP based Analytics Platforms, Hyper-Scale

Processing Clusters, Data Mining and Search Engines.

What You’ll Need:

  • 3+ years of industry experience in creating and managing end-to-end Data Solutions, Optimal

Data Processing Pipelines and Architecture dealing with large volume, big data sets of varied

data types.

  • Proficiency in Python, Linux and shell scripting.
  • Strong knowledge of working with PySpark dataframes, Pandas dataframes for writing efficient pre-processing and other data manipulation tasks.
    ● Strong experience in developing the infrastructure required for data ingestion, optimal

extraction, transformation, and loading of data from a wide variety of data sources using tools like Azure Data Factory,  Azure Databricks (or Jupyter notebooks/ Google Colab) (or other similiar tools).

  • Working knowledge of github or other version control tools.
  • Experience with creating Restful web services and API platforms.
  • Work with data science and infrastructure team members to implement practical machine

learning solutions and pipelines in production.

  • Experience with cloud providers like Azure/AWS/GCP.
  • Experience with SQL and NoSQL databases. MySQL/ Azure Cosmosdb / Hbase/MongoDB/ Elasticsearch etc.
  • Experience with stream-processing systems: Spark-Streaming, Kafka etc and working experience with event driven architectures.
  • Strong analytic skills related to working with unstructured datasets.

 

Good to have (to filter or prioritize candidates)

  • Experience with testing libraries such as pytest for writing unit-tests for the developed code.
  • Knowledge of Machine Learning algorithms and libraries would be good to have,

implementation experience would be an added advantage.

  • Knowledge and experience of Datalake, Dockers and Kubernetes would be good to have.
  • Knowledge of Azure functions , Elastic search etc will be good to have.

 

  • Having experience with model versioning (mlflow) and data versioning will be beneficial
  • Having experience with microservices libraries or with python libraries such as flask for hosting ml services and models would be great.
Job posted by
Siddarth Thakur

SQL Engineers

at Datametica Solutions Private Limited

Founded 2013  •  Products & Services  •  100-1000 employees  •  Profitable
ETL
SQL
Data engineering
Analytics
PL/SQL
Shell Scripting
Linux/Unix
Datawarehousing
icon
Pune, Hyderabad
icon
4 - 10 yrs
icon
₹5L - ₹20L / yr

We at Datametica Solutions Private Limited are looking for SQL Engineers who have a passion for cloud with knowledge of different on-premise and cloud Data implementation in the field of Big Data and Analytics including and not limiting to Teradata, Netezza, Exadata, Oracle, Cloudera, Hortonworks and alike. 

Ideal candidates should have technical experience in migrations and the ability to help customers get value from Datametica's tools and accelerators.

Job Description

Experience : 4-10 years

Location : Pune

 


Mandatory Skills - 

  • Strong in ETL/SQL development
  • Strong Data Warehousing skills
  • Hands-on experience working with Unix/Linux
  • Development experience in Enterprise Data warehouse projects
  • Good to have experience working with Python, shell scripting
  •  

Opportunities -

  • Selected candidates will be provided training opportunities on one or more of the following: Google Cloud, AWS, DevOps Tools, Big Data technologies like Hadoop, Pig, Hive, Spark, Sqoop, Flume and Kafka
  • Would get chance to be part of the enterprise-grade implementation of Cloud and Big Data systems
  • Will play an active role in setting up the Modern data platform based on Cloud and Big Data
  • Would be part of teams with rich experience in various aspects of distributed systems and computing


 

About Us!

A global Leader in the Data Warehouse Migration and Modernization to the Cloud, we empower businesses by migrating their Data/Workload/ETL/Analytics to the Cloud by leveraging Automation.

 

We have expertise in transforming legacy Teradata, Oracle, Hadoop, Netezza, Vertica, Greenplum along with ETLs like Informatica, Datastage, AbInitio & others, to cloud-based data warehousing with other capabilities in data engineering, advanced analytics solutions, data management, data lake and cloud optimization.

 

Datametica is a key partner of the major cloud service providers - Google, Microsoft, Amazon, Snowflake.

 

We have our own products!

Eagle – Data warehouse Assessment & Migration Planning Product

Raven – Automated Workload Conversion Product

Pelican - Automated Data Validation Product, which helps automate and accelerate data migration to the cloud.

 

Why join us!

Datametica is a place to innovate, bring new ideas to live and learn new things. We believe in building a culture of innovation, growth and belonging. Our people and their dedication over these years are the key factors in achieving our success.

 

 

Benefits we Provide!

Working with Highly Technical and Passionate, mission-driven people

Subsidized Meals & Snacks

Flexible Schedule

Approachable leadership

Access to various learning tools and programs

Pet Friendly

Certification Reimbursement Policy

 

Check out more about us on our website below!

www.datametica.com

Job posted by
Sayali Kachi

AI Engineer

at StatusNeo

Founded 2020  •  Products & Services  •  100-1000 employees  •  Profitable
Artificial Intelligence (AI)
Amazon Web Services (AWS)
Windows Azure
Hadoop
Scala
Python
Google Cloud Platform (GCP)
postgres
icon
Gurugram, Hyderabad, Bengaluru (Bangalore)
icon
1 - 3 yrs
icon
₹3L - ₹12L / yr


·       Build data products and processes alongside the core engineering and technology team.

·       Collaborate with senior data scientists to curate, wrangle, and prepare data for use in their advanced analytical models

·       Integrate data from a variety of sources, assuring that they adhere to data quality and accessibility standards

·       Modify and improve data engineering processes to handle ever larger, more complex, and more types of data sources and pipelines

·       Use Hadoop architecture and HDFS commands to design and optimize data queries at scale

·       Evaluate and experiment with novel data engineering tools and advises information technology leads and partners about new capabilities to determine optimal solutions for particular technical problems or designated use cases .
Job posted by
Alex P
PySpark
Data engineering
Big Data
Hadoop
Spark
Shell Scripting
Python
Apache Sqoop
Cloudera
Relational Database (RDBMS)
MS-Excel
icon
Navi Mumbai
icon
3 - 5 yrs
icon
₹7L - ₹18L / yr
  • Proficiency in shell scripting
  • Proficiency in automation of tasks
  • Proficiency in Pyspark/Python
  • Proficiency in writing and understanding of sqoop
  • Understanding of CloudEra manager
  • Good understanding of RDBMS
  • Good understanding of Excel

 

Job posted by
Shivani more

Data Engineer

at Healthifyme

Founded 2012  •  Products & Services  •  100-1000 employees  •  Raised funding
Python
SQL
Data engineering
Big Data
Data Warehouse (DWH)
ETL
Apache Spark
flink
icon
Bengaluru (Bangalore)
icon
3 - 4 yrs
icon
₹18L - ₹35L / yr

Responsibilities:

  • Design, construct, install, test and maintain data pipeline and data management systems.
  • Ensure that all systems meet the business/company requirements as well as industry practices.
  • Integrate up-and-coming data management and software engineering technologies into existing data structures.
  • Processes for data mining, data modeling, and data production.
  • Create custom software components and analytics applications.
  • Collaborate with members of your team (eg, Data Architects, the Software team, Data Scientists) on the project's goals.
  • Recommend different ways to constantly improve data reliability and quality.

 

Requirements:

  • Experience in a related field with real-world skills and testimonials from former employees.
  • Familiar with data warehouses like Redshift, Bigquery and Athena.
  • Familiar with data processing systems like flink, spark and storm. Develop set
  • Proficiency in Python and SQL. Possible work experience and proof of technical expertise.
  • You may also consider a Master's degree in computer engineering or science in order to fine-tune your skills while on the job. (Although a Master's isn't required, it is always appreciated).
  • Intellectual curiosity to find new and unusual ways of how to solve data management issues.
  • Ability to approach data organization challenges while keeping an eye on what's important.
  • Minimal data science knowledge is a Must, should understand a bit of analytics.
Job posted by
Jaya Harjai

Lead Data Engineer

at Discite Analytics Private Limited

Founded 2020  •  Products & Services  •  20-100 employees  •  Raised funding
Hadoop
Big Data
Data engineering
Spark
Apache Beam
Apache Kafka
Data Warehouse (DWH)
ETL
Python
Java
Scala
C++
MySQL
PostgreSQL
MongoDB
Cassandra
Windows Azure
Amazon Web Services (AWS)
icon
Ahmedabad
icon
4 - 7 yrs
icon
₹12L - ₹20L / yr
Responsibilities:
1. Communicate with the clients and understand their business requirements.
2. Build, train, and manage your own team of junior data engineers.
3. Assemble large, complex data sets that meet the client’s business requirements.
4. Identify, design and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
5. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources, including the cloud.
6. Assist clients with data-related technical issues and support their data infrastructure requirements.
7. Work with data scientists and analytics experts to strive for greater functionality.

Skills required: (experience with at least most of these)
1. Experience with Big Data tools-Hadoop, Spark, Apache Beam, Kafka etc.
2. Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
3. Experience in ETL and Data Warehousing.
4. Experience and firm understanding of relational and non-relational databases like MySQL, MS SQL Server, Postgres, MongoDB, Cassandra etc.
5. Experience with cloud platforms like AWS, GCP and Azure.
6. Experience with workflow management using tools like Apache Airflow.
Job posted by
Uma Sravya B

Big Data Engineer

at Netmeds.com

Founded 2015  •  Product  •  500-1000 employees  •  Raised funding
Big Data
Hadoop
Apache Hive
Scala
Spark
Datawarehousing
Machine Learning (ML)
Deep Learning
SQL
Data modeling
PySpark
Python
Amazon Web Services (AWS)
Java
Cassandra
DevOps
HDFS
icon
Chennai
icon
2 - 5 yrs
icon
₹6L - ₹25L / yr

We are looking for an outstanding Big Data Engineer with experience setting up and maintaining Data Warehouse and Data Lakes for an Organization. This role would closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Roles and Responsibilities:

  • Develop and maintain scalable data pipelines and build out new integrations and processes required for optimal extraction, transformation, and loading of data from a wide variety of data sources using 'Big Data' technologies.
  • Develop programs in Scala and Python as part of data cleaning and processing.
  • Assemble large, complex data sets that meet functional / non-functional business requirements and fostering data-driven decision making across the organization.  
  • Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
  • Implement processes and systems to validate data, monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
  • Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
  • Provide high operational excellence guaranteeing high availability and platform stability.
  • Closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Skills:

  • Experience with Big Data pipeline, Big Data analytics, Data warehousing.
  • Experience with SQL/No-SQL, schema design and dimensional data modeling.
  • Strong understanding of Hadoop Architecture, HDFS ecosystem and eexperience with Big Data technology stack such as HBase, Hadoop, Hive, MapReduce.
  • Experience in designing systems that process structured as well as unstructured data at large scale.
  • Experience in AWS/Spark/Java/Scala/Python development.
  • Should have Strong skills in PySpark (Python & SPARK). Ability to create, manage and manipulate Spark Dataframes. Expertise in Spark query tuning and performance optimization.
  • Experience in developing efficient software code/frameworks for multiple use cases leveraging Python and big data technologies.
  • Prior exposure to streaming data sources such as Kafka.
  • Should have knowledge on Shell Scripting and Python scripting.
  • High proficiency in database skills (e.g., Complex SQL), for data preparation, cleaning, and data wrangling/munging, with the ability to write advanced queries and create stored procedures.
  • Experience with NoSQL databases such as Cassandra / MongoDB.
  • Solid experience in all phases of Software Development Lifecycle - plan, design, develop, test, release, maintain and support, decommission.
  • Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development).
  • Experience building and deploying applications on on-premise and cloud-based infrastructure.
  • Having a good understanding of machine learning landscape and concepts. 

 

Qualifications and Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Big Data Engineer or a similar role for 3-5 years.

Certifications:

Good to have at least one of the Certifications listed here:

    AZ 900 - Azure Fundamentals

    DP 200, DP 201, DP 203, AZ 204 - Data Engineering

    AZ 400 - Devops Certification

Job posted by
Vijay Hemnath
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at Global consulting company?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort