Data Engineer- Pyspark+SQL (Onsite opportunity- UAE Abu Dhabi/Dubai)
at UAE Client
6 - 12 yrs
₹18L - ₹25L / yr
Abu Dhabi, Dubai
Skills
PySpark
Big Data
Spark
Data Warehouse (DWH)
SQL
Data engineering
Apache Synapse
Must-Have Skills:
• Good experience in Pyspark - Including Dataframe core functions and Spark SQL
• Good experience in SQL DBs - Be able to write queries including fair complexity.
• Should have excellent experience in Big Data programming for data transformation and aggregations
• Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
• Good customer communication.
• Good Analytical skill
- Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
- Experience in migrating on-premise data warehouses to data platforms on AZURE cloud.
- Designing and implementing data engineering, ingestion, and transformation functions
-
Azure Synapse or Azure SQL data warehouse
-
Spark on Azure is available in HD insights and data bricks
Read more
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Subodh Popalwar
Software Engineer, MemorresFor 2 years, I had trouble finding a company with good work culture and a role that will help me grow in my career. Soon after I started using Cutshort, I had access to information about the work culture, compensation and what each company was clearly offering.
Companies hiring on Cutshort
About UAE Client
Founded
Type
Size
Stage
About
N/A
Company social profiles
N/A
Similar jobs
Delhi
2 - 5 yrs
₹10L - ₹12L / yr
Spotfire
Qlikview
Tableau
PowerBI
Data Visualization
+5 more
Job Title:
Data Analyst
Job Brief:
The successful candidate will turn data into information, information into insight and insight into business decisions.
Data Analyst Job Duties
Data analyst responsibilities include conducting full lifecycle analysis to include requirements, activities and design. Data analysts will develop analysis and reporting capabilities. They will also monitor performance
and quality control plans to identify improvements.
About Us
We began in 2015 with an entrepreneurial vision to bring a digital change in the manufacturing landscape of India. With a team of 300+ we are working towards the digital transformation of business in the manufacturing industry across domains like Footwear, Apparel, Textile, Accessories etc. We are backed by investors such as
Info Edge (Naukri.com), Matrix Partners, Sequoia, Water Bridge Ventures and select Industry leaders.
Today, we have enabled 2000+ Manufacturers to digitize their distribution channel.
Responsibilities
● Interpret data, analyze results using statistical techniques and provide ongoing reports.
● Develop and implement databases, data collection systems, data analytics and other strategies that
optimize statistical efficiency and quality.
● Acquire data from primary or secondary data sources and maintain databases/data systems.
● Identify, analyze, and interpret trends or patterns in complex data sets.
● Filter and “clean” data by reviewing computer reports, printouts, and performance indicators to locate and
correct code problems.
● Work with management to prioritize business and information needs.
● Locate and define new process improvement opportunities.
Requirements
● Proven working experience as a Data Analyst or Business Data Analyst.
● Technical expertise regarding data models, database design development, data mining and segmentation
techniques.
● Strong knowledge of and experience with reporting packages (Business Objects etc), databases (SQL etc),
programming (XML, Javascript, or ETL frameworks).
● Knowledge of statistics and experience using statistical packages for analyzing datasets (Excel, SPSS, SAS
etc).
● Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of
information with attention to detail and accuracy.
● Adept at queries, report writing and presenting findings.
Job Location
South Delhi, New Delhi
Data Analyst
Job Brief:
The successful candidate will turn data into information, information into insight and insight into business decisions.
Data Analyst Job Duties
Data analyst responsibilities include conducting full lifecycle analysis to include requirements, activities and design. Data analysts will develop analysis and reporting capabilities. They will also monitor performance
and quality control plans to identify improvements.
About Us
We began in 2015 with an entrepreneurial vision to bring a digital change in the manufacturing landscape of India. With a team of 300+ we are working towards the digital transformation of business in the manufacturing industry across domains like Footwear, Apparel, Textile, Accessories etc. We are backed by investors such as
Info Edge (Naukri.com), Matrix Partners, Sequoia, Water Bridge Ventures and select Industry leaders.
Today, we have enabled 2000+ Manufacturers to digitize their distribution channel.
Responsibilities
● Interpret data, analyze results using statistical techniques and provide ongoing reports.
● Develop and implement databases, data collection systems, data analytics and other strategies that
optimize statistical efficiency and quality.
● Acquire data from primary or secondary data sources and maintain databases/data systems.
● Identify, analyze, and interpret trends or patterns in complex data sets.
● Filter and “clean” data by reviewing computer reports, printouts, and performance indicators to locate and
correct code problems.
● Work with management to prioritize business and information needs.
● Locate and define new process improvement opportunities.
Requirements
● Proven working experience as a Data Analyst or Business Data Analyst.
● Technical expertise regarding data models, database design development, data mining and segmentation
techniques.
● Strong knowledge of and experience with reporting packages (Business Objects etc), databases (SQL etc),
programming (XML, Javascript, or ETL frameworks).
● Knowledge of statistics and experience using statistical packages for analyzing datasets (Excel, SPSS, SAS
etc).
● Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of
information with attention to detail and accuracy.
● Adept at queries, report writing and presenting findings.
Job Location
South Delhi, New Delhi
Read more
Bengaluru (Bangalore), Hyderabad, Pune
3 - 7 yrs
Best in industry
Spark
Hadoop
NOSQL Databases
Apache Kafka
About Telstra
Telstra is Australia’s leading telecommunications and technology company, with operations in more than 20 countries, including In India where we’re building a new Innovation and Capability Centre (ICC) in Bangalore.
We’re growing, fast, and for you that means many exciting opportunities to develop your career at Telstra. Join us on this exciting journey, and together, we’ll reimagine the future.
Why Telstra?
- We're an iconic Australian company with a rich heritage that's been built over 100 years. Telstra is Australia's leading Telecommunications and Technology Company. We've been operating internationally for more than 70 years.
- International presence spanning over 20 countries.
- We are one of the 20 largest telecommunications providers globally
- At Telstra, the work is complex and stimulating, but with that comes a great sense of achievement. We are shaping the tomorrow's modes of communication with our innovation driven teams.
Telstra offers an opportunity to make a difference to lives of millions of people by providing the choice of flexibility in work and a rewarding career that you will be proud of!
About the team
Being part of Networks & IT means you'll be part of a team that focuses on extending our network superiority to enable the continued execution of our digital strategy.
With us, you'll be working with world-leading technology and change the way we do IT to ensure business needs drive priorities, accelerating our digitisation programme.
Focus of the role
Any new engineer who comes into data chapter would be mostly into developing reusable data processing and storage frameworks that can be used across data platform.
About you
To be successful in the role, you'll bring skills and experience in:-
Essential
- Hands-on experience in Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing
- Skilled in the Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
- Azure data factory/Airflow/control-M/Luigi
- PL/SQL
- Exposure to NOSQL(Hbase/Cassandra/GraphDB(Neo4J)/MongoDB)
- File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
- Kafka/Kinesis/Eventhub
Highly Desirable
Experience and knowledgeable on the following:
- Spark Streaming
- Cloud exposure (Azure/AWS/GCP)
- Azure data offerings - ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc.
- Presto/Athena
- Azure DevOps
- Jenkins/ Bamboo/Any similar build tools
- Power BI
- Prior experience in building or working in team building reusable frameworks,
- Data modelling.
- Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
- Exposure to CI/CD
- Code Quality - Static and Dynamic code scans
- Agile SDLC
If you've got a passion to innovate, succeed as part of a great team, and looking for the next step in your career, we'd welcome you to apply!
___________________________
We’re committed to building a diverse and inclusive workforce in all its forms. We encourage applicants from diverse gender, cultural and linguistic backgrounds and applicants who may be living with a disability. We also offer flexibility in all our roles, to ensure everyone can participate.
To learn more about how we support our people, including accessibility adjustments we can provide you through the recruitment process, visit tel.st/thrive.
Read more
Bengaluru (Bangalore)
3 - 8 yrs
₹15L - ₹18L / yr
Data AnalyticsSQL server
SQL
Data Analyst
1. Ability to work independently and to set priorities while managing several projects simultaneously; strong attention to detail is essential.
2.Collaborates with Business Systems Analysts and/or directly with key business users to ensure business requirements and report specifications are documented accurately and completely.
3.Develop data field mapping documentation.
4. Document data sources and processing flow.
5. Ability to design, refine and enhance existing reports from source systems or data warehouse.
6.Ability to analyze and optimize data including data deduplication required for reports.
7. Analysis and rationalization of reports.
8. Support QA and UAT teams in defining test scenarios and clarifying requirements.
9. Effectively communicate results of the data analysis to internal and external customers to support decision making.
10.Follows established SDLC, change control, release management and incident management processes.
11.Perform source data analysis and assessment.
12. Perform data profiling to capture business and technical rules.
13. Track and help to remediate issues and defects due to data quality exceptions.
Read more
Mumbai
2 - 4 yrs
₹4L - ₹8L / yr
ETL
SQL Server Integration Services (SSIS)
Microsoft SSIS
SQL
SQL server
Should be able to connect with multiple data sources like excel, csv, SQL server oracle etc.
Should be able to use the transformations components to transform the data
Should possess knowledge on incremental load, full load etc.
Should Design, build and deploy effective packages
Should be able to schedule these packages through task schedulers
Implement stored procedures and effectively query a database
Translate requirements from the business and analyst into technical code
Identify and test for bugs and bottlenecks in the ETL solution
Ensure the best possible performance and quality in the packages
Provide support and fix issues in the packages
Writes advanced SQL including some query tuning
Experience in the identification of data quality
Some database design experience is helpful
Experience designing and building complete ETL/SSIS processes moving and transforming data for
ODS, Staging, and Data Warehousing
Should be able to use the transformations components to transform the data
Should possess knowledge on incremental load, full load etc.
Should Design, build and deploy effective packages
Should be able to schedule these packages through task schedulers
Implement stored procedures and effectively query a database
Translate requirements from the business and analyst into technical code
Identify and test for bugs and bottlenecks in the ETL solution
Ensure the best possible performance and quality in the packages
Provide support and fix issues in the packages
Writes advanced SQL including some query tuning
Experience in the identification of data quality
Some database design experience is helpful
Experience designing and building complete ETL/SSIS processes moving and transforming data for
ODS, Staging, and Data Warehousing
Read more
Ahmedabad
4 - 7 yrs
₹12L - ₹20L / yr
Hadoop
Big Data
Data engineering
Spark
Apache Beam
+13 more
Responsibilities:
1. Communicate with the clients and understand their business requirements.
2. Build, train, and manage your own team of junior data engineers.
3. Assemble large, complex data sets that meet the client’s business requirements.
4. Identify, design and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
5. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources, including the cloud.
6. Assist clients with data-related technical issues and support their data infrastructure requirements.
7. Work with data scientists and analytics experts to strive for greater functionality.
Skills required: (experience with at least most of these)
1. Experience with Big Data tools-Hadoop, Spark, Apache Beam, Kafka etc.
2. Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
3. Experience in ETL and Data Warehousing.
4. Experience and firm understanding of relational and non-relational databases like MySQL, MS SQL Server, Postgres, MongoDB, Cassandra etc.
5. Experience with cloud platforms like AWS, GCP and Azure.
6. Experience with workflow management using tools like Apache Airflow.
1. Communicate with the clients and understand their business requirements.
2. Build, train, and manage your own team of junior data engineers.
3. Assemble large, complex data sets that meet the client’s business requirements.
4. Identify, design and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
5. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources, including the cloud.
6. Assist clients with data-related technical issues and support their data infrastructure requirements.
7. Work with data scientists and analytics experts to strive for greater functionality.
Skills required: (experience with at least most of these)
1. Experience with Big Data tools-Hadoop, Spark, Apache Beam, Kafka etc.
2. Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
3. Experience in ETL and Data Warehousing.
4. Experience and firm understanding of relational and non-relational databases like MySQL, MS SQL Server, Postgres, MongoDB, Cassandra etc.
5. Experience with cloud platforms like AWS, GCP and Azure.
6. Experience with workflow management using tools like Apache Airflow.
Read more
Remote only
2 - 3 yrs
₹5L - ₹8L / yr
Data Analyticsdata analyst
Apache Synapse
SQL
SAP MDG ( Master Data Governance)
+1 more
The Data Analyst in the CoE will provide end to end solution development, working in conjunction with the Domain Leads and Technology Partners. He is responsible for the delivery of solutions and solution changes which are driven by the business requirements as well as providing technical and development capabilities. Knowledge
Read more
Pune
5 - 8 yrs
₹10L - ₹17L / yr
PythonBig Data
Amazon Web Services (AWS)Windows Azure
Google Cloud Platform (GCP)
+3 more
- Must have 5-8 years of experience in handling data
- Must have the ability to interpret large amounts of data and to multi-task
- Must have strong knowledge of and experience with programming (Python), Linux/Bash scripting, databases(SQL, etc)
- Must have strong analytical and critical thinking to resolve business problems using data and tech
- Must have domain familiarity and interest of – Cloud technologies (GCP/Azure Microsoft/ AWS Amazon), open-source technologies, Enterprise technologies
- Must have the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
- Must have good communication skills
- Working knowledge/exposure to ElasticSearch, PostgreSQL, Athena, PrestoDB, Jupyter Notebook
Read more
Pune
2 - 5 yrs
₹5L - ₹10L / yr
Pandas
Numpy
Bash
Structured Query Language
Python +2 more
We are an early stage startup working in the space of analytics, big data, machine learning, data visualization on multiple platforms and SaaS.
We have our offices in Palo Alto and WTC, Kharadi, Pune and got some marque names as our customers.
We are looking for really good Python programmer who MUST have scientific programming experience (Python, etc.)
Hands-on with numpy and the Python scientific stack is a must.
Demonstrated ability to track and work with 100s-1000s of files and GB-TB of data.
Exposure to ML and Data mining algorithms.
Need to be comfortable working in a Unix environment and SQL.
You will be required to do following:
Using command line tools to perform data conversion and analysis
Supporting other team members in retrieving and archiving experimental results
Quickly writing scripts to automate routine analysis tasks
Creating insightful, simple graphics to represent complex trends
Explore/design/invent new tools and design patterns to solve complex big data problems
Experience working on a long-term, lab-based project (academic experience acceptable)
Read more
Bengaluru (Bangalore)
2 - 10 yrs
₹8L - ₹35L / yr
PythonHadoop
Spark
Amazon Web Services (AWS)Big Data
+2 more
Intro
Our data and risk team is the core pillar of our business that harnesses alternative data sources to guide the decisions we make at Rely. The team designs, architects, as well as develop and maintain a scalable data platform the powers our machine learning models. Be part of a team that will help millions of consumers across Asia, to be effortlessly in control of their spending and make better decisions.
What will you do
The data engineer is focused on making data correct and accessible, and building scalable systems to access/process it. Another major responsibility is helping AI/ML Engineers write better code.
• Optimize and automate ingestion processes for a variety of data sources such as: click stream, transactional and many other sources.
- Create and maintain optimal data pipeline architecture and ETL processes
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Develop data pipeline and infrastructure to support real-time decisions
- Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data' technologies.
- Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
- Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
What will you need
• 2+ hands-on experience building and implementation of large scale production pipeline and Data Warehouse
• Experience dealing with large scale
- Proficiency in writing and debugging complex SQLs
- Experience working with AWS big data tools
• Ability to lead the project and implement best data practises and technology
Data Pipelining
- Strong command in building & optimizing data pipelines, architectures and data sets
- Strong command on relational SQL & noSQL databases including Postgres
- Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
Big Data: Strong experience in big data tools & applications
- Tools: Hadoop, Spark, HDFS etc
- AWS cloud services: EC2, EMR, RDS, Redshift
- Stream-processing systems: Storm, Spark-Streaming, Flink etc.
- Message queuing: RabbitMQ, Spark etc
Software Development & Debugging
- Strong experience in object-oriented programming/object function scripting languages: Python, Java, C++, Scala, etc
- Strong hold on data structures & algorithms
What would be a bonus
- Prior experience working in a fast-growth Startup
- Prior experience in the payments, fraud, lending, advertising companies dealing with large scale data
Read more
Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
21,01,133
Matches delivered
37,12,187
Network size
15,000
Companies hiring
Did not find a job you were looking for?
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Follow CutshortUsers love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Subodh Popalwar
Software Engineer, MemorresFor 2 years, I had trouble finding a company with good work culture and a role that will help me grow in my career. Soon after I started using Cutshort, I had access to information about the work culture, compensation and what each company was clearly offering.
Companies hiring on Cutshort