Data Engineer and Data Bricks

at A Global IT Service company

Agency job
icon
Bengaluru (Bangalore)
icon
5 - 8 yrs
icon
₹20L - ₹30L / yr
icon
Full time
Skills
Data engineering
Data Bricks
data engineer
PySpark
ETL
Azure Data Bricks
SSIS
Azure Data Factory
  • Insurance P&C and Specialty domain experience a plus
  • Experience in a cloud-based architecture preferred, such as Databricks, Azure Data Lake, Azure Data Factory, etc.
  • Strong understanding of ETL fundamentals and solutions. Should be proficient in writing advanced / complex SQL, expertise in performance tuning and optimization of SQL queries required.
  • Strong experience in Python/PySpark and Spark SQL
  • Experience in troubleshooting data issues, analyzing end to end data pipelines, and working with various teams in resolving issues and solving complex problems.
  • Strong experience developing Spark applications using PySpark and SQL for data extraction, transformation, and aggregation from multiple formats for analyzing & transforming the data to uncover insights and actionable intelligence for internal and external use
Read more
Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Data Scientist

at EnterpriseMinds

Founded 2017  •  Products & Services  •  100-1000 employees  •  Profitable
Machine Learning (ML)
Natural Language Processing (NLP)
Python
Data Science
PySpark
icon
Bengaluru (Bangalore)
icon
3 - 6 yrs
icon
₹7L - ₹30L / yr

Exp: 3-6 Yrs
Location: Bangalore
Notice: Immediate to 15 days

Responsibilities:

  • Develop advanced algorithms that solve problems of large dimensionality in a computationally efficient and statistically effective manner;
  • Execute statistical and data mining techniques (e.g. hypothesis testing, machine learning and retrieval processes) on large data sets to identify trends, figures and other relevant information;
  • Evaluate emerging datasets and technologies that may contribute to our analytical platform;
  • Participate in development of select assets/accelerators that create scale;
  • Contribute to thought leadership through research and publication support;
  • Guide and mentor Associates on teams.

Qualifications:

 
  • 3-6 years of relevant post-collegiate work experience;
  • Knowledge of big data/advanced analytics concepts and algorithms (e.g. text mining, social listening, recommender systems, predictive modeling, etc.);
  • Should have experience on NLP, Pyspark
  • Exposure to tools/platforms (e.g. Hadoop eco system and database systems);
  • Agile project planning and project management skills;
  • Relevant domain knowledge preferred; (healthcare/transportation/hi-tech/insurance);
  • Excellent oral and written communication skills;
  • Strong attention to detail, with a research-focused mindset;
  • Excellent critical thinking and problem solving skills;
  • High motivation, good work ethic and maturity.
Read more
Job posted by
Komal Samudrala

Python Developer (Data Engineer)

at Milestone Hr Consultancy

Founded 2016  •  Services  •  employees  •  Raised funding
Python
Django
Data engineering
Apache Hive
Apache Spark
icon
Remote, Hyderabad
icon
3 - 8 yrs
icon
₹6L - ₹16L / yr
We are currently looking for passionate Data Engineers to join our team and mission. In this role, you will help doctors from across the world improve care and save lives by helping extract insights and predict risk. Our Data Engineers ensure that data are ingested and prepared, ready for insights and intelligence to be derived from them. We’re looking for smart individuals to join our incredibly talented team, that is on a mission to transform healthcare.As a Data Engineer you will be engaged in some or all of the following activities:• Implement, test and deploy distributed data ingestion, data processing and feature engineering systems computing on large volumes of Healthcare data using a variety of open source and proprietary technologies.• Design data architectures and schemas optimized for analytics and machine learning.• Implement telemetry to monitor the performance and operations of data pipelines.• Develop tools and libraries to implement and manage data processing pipelines, including ingestion, cleaning, transformation, and feature computation.• Work with large data sets, and integrate diverse data sources, data types and data structures.• Work with Data Scientists, Machine Learning Engineers and Visualization Engineers to understand data requirements, and translate them into production-ready data pipelines.• Write and automate unit, functional, integration and performance tests in a Continuous Integration environment.• Take initiative to find solutions to technical challenges for healthcare data.You are a great match if you have some or all of the following skills and qualifications.• Strong understanding of database design and feature engineering to support Machine Learning and analytics.• At least 3 years of industry experience building, testing and deploying large-scale, distributed data processing systems.• Proficiency in working with multiple data processing tools and query languages (Python, Spark, SQL, etc.).• Excellent understanding of distributed computing concepts and Big Data technologies (Spark, Hive, etc.).• Proficiency in performance tuning and optimization of data processing pipelines.• Attention to detail and focus on software quality, with experience in software testing.• Strong cross discipline communication skills and teamwork.• Demonstrated clear and thorough logical and analytical thinking, as well as problem solving skills.• Bachelor or Masters in Computer Science or related field. Skill - Apache Spark-Python-Hive Skill Description - Skill1– SparkSkill2- PythonSkill3 – Hive, SQL Responsibility - Sr. data engineer"
Read more
Job posted by
Jyoti Sharma

Senior Software Engineer/Technical Lead - Data Fabric

at IDfy

Founded 2011  •  Products & Services  •  100-1000 employees  •  Raised funding
Data Warehouse (DWH)
Informatica
ETL
ETL architecture
Responsive Design
Apache Beam
InfluxDB
SQL
OLAP
icon
Mumbai
icon
3 - 10 yrs
icon
₹15L - ₹45L / yr

Who is IDfy?

 

IDfy is the Fintech ScaleUp of the Year 2021. We build technology products that identify people accurately. This helps businesses prevent fraud and engage with the genuine with the least amount of friction. If you have opened an account with HDFC Bank or ordered from Amazon and Zomato or transacted through Paytm and BharatPe or played on Dream11 and MPL, you might have already experienced IDfy. Without even knowing it. Well…that’s just how we roll. Global credit rating giant TransUnion is an investor in IDfy. So are international venture capitalists like MegaDelta Capital, BEENEXT, and Dream Incubator. Blume Ventures is an early investor and continues to place its faith in us. We have kept our 500 clients safe from fraud while helping the honest get the opportunities they deserve. Our 350-people strong family works and plays out of our offices in suburban Mumbai. IDfy has run verifications on 100 million people. In the next 2 years, we want to touch a billion users. If you wish to be part of this journey filled with lots of action and learning, we welcome you to be part of the team!

 

What are we looking for?

 

As a senior software engineer in Data Fabric POD, you would be responsible for producing and implementing functional software solutions. You will work with upper management to define software requirements and take the lead on operational and technical projects. You would be working with a data management and science platform which provides Data as a service (DAAS) and Insight as a service (IAAS) to internal employees and external stakeholders.

 

You are eager to learn technology-agnostic who loves working with data and drawing insights from it. You have excellent organization and problem-solving skills and are looking to build the tools of the future. You have exceptional communication skills and leadership skills and the ability to make quick decisions.

 

YOE: 3 - 10 yrs

Position: Sr. Software Engineer/Module Lead/Technical Lead

 

Responsibilities:

  • Work break-down and orchestrating the development of components for each sprint.
  • Identifying risks and forming contingency plans to mitigate them.
  • Liaising with team members, management, and clients to ensure projects are completed to standard.
  • Inventing new approaches to detecting existing fraud. You will also stay ahead of the game by predicting future fraud techniques and building solutions to prevent them.
  • Developing Zero Defect Software that is secured, instrumented, and resilient.
  • Creating design artifacts before implementation.
  • Developing Test Cases before or in parallel with implementation.
  • Ensuring software developed passes static code analysis, performance, and load test.
  • Developing various kinds of components (such as UI Components, APIs, Business Components, image Processing, etc. ) that define the IDfy Platforms which drive cutting-edge Fraud Detection and Analytics.
  • Developing software using Agile Methodology and tools that support the same.

 

Requirements:

  • Apache BEAM, Clickhouse, Grafana, InfluxDB, Elixir, BigQuery, Logstash.
  • An understanding of Product Development Methodologies.
  • Strong understanding of relational databases especially SQL and hands-on experience with OLAP.
  • Experience in the creation of data ingestion pipelines and ETL pipeline (Good to have Apache Beam or Apache Airflow experience).
  • Strong design skills in defining API Data Contracts / OOAD / Microservices / Data Models.

 

Good to have:

  • Experience with TimeSeries DBs (we use InfluxDB) and Alerting / Anomaly Detection Frameworks.
  • Visualization Layers: Metabase, PowerBI, Tableau.
  • Experience in developing software in the Cloud such as GCP / AWS.
  • A passion to explore new technologies and express yourself through technical blogs.
Read more
Job posted by
Stuti Srivastava

AGM Data Engineering

at ACT FIBERNET

Founded 2008  •  Services  •  100-1000 employees  •  Profitable
Data engineering
Data Engineer
Hadoop
Informatica
Qlikview
Datapipeline
icon
Bengaluru (Bangalore)
icon
9 - 14 yrs
icon
₹20L - ₹36L / yr

Key  Responsibilities :

  • Development of proprietary processes and procedures designed to process various data streams around critical databases in the org
  • Manage technical resources around data technologies, including relational databases, NO SQL DBs, business intelligence databases, scripting languages, big data tools and technologies, visualization tools.
  • Creation of a project plan including timelines and critical milestones to success in support of the project
  • Identification of the vital skill sets/staff required to complete the project
  • Identification of crucial sources of the data needed to achieve the objective.

 

Skill Requirement :

  • Experience with data pipeline processes and tools
  • Well versed in the Data domains (Data Warehousing, Data Governance, MDM, Data Quality, Data Catalog, Analytics, BI, Operational Data Store, Metadata, Unstructured Data, ETL, ESB)
  • Experience with an existing ETL tool e.g Informatica and Ab initio etc
  • Deep understanding of big data systems like Hadoop, Spark, YARN, Hive, Ranger, Ambari
  • Deep knowledge of Qlik ecosystems like  Qlikview, Qliksense, and Nprinting
  • Python, or a similar programming language
  • Exposure to data science and machine learning
  • Comfort working in a fast-paced environment

Soft attributes :

  • Independence: Must have the ability to work on his/her own without constant direction or supervision. He/she must be self-motivated and possess a strong work ethic to strive to put forth extra effort continually
  • Creativity: Must be able to generate imaginative, innovative solutions that meet the needs of the organization. You must be a strategic thinker/solution seller and should be able to think of integrated solutions (with field force apps, customer apps, CCT solutions etc.). Hence, it would be best to approach each unique situation/challenge in different ways using the same tools.
  • Resilience: Must remain effective in high-pressure situations, using both positive and negative outcomes as an incentive to move forward toward fulfilling commitments to achieving personal and team goals.
Read more
Job posted by
Sumit Sindhwani

Tableau Developer

at Digitop

Agency job
via Nu-Pie
Tableau
Analytical Skills
Dashboard
Data extraction
ETL
Data loading
Software Testing (QA)
icon
Remote, Bengaluru (Bangalore)
icon
2 - 4 yrs
icon
₹4L - ₹9L / yr
  • Hands-on development/maintenance experience in Tableau: Developing, maintaining, and managing advanced reporting, analytics, dashboards and other BI solutions using Tableau
  • Reviewing and improving existing Tableau dashboards and data models/ systems and collaborating with teams to integrate new systems
  • Provide support and expertise to the business community to assist with better utilization of Tableau
  • Understand business requirements, conduct analysis and recommend solution options for intelligent dashboards in Tableau
  • Experience with Data Extraction, Transformation and Load (ETL) – knowledge of how to extract, transform and load data
  • Execute SQL data queries across multiple data sources in support of business intelligence reporting needs. Format query results / reports in various ways
  • Participates in QA testing, liaising with other project team members and being responsive to client's needs, all with an eye on details in a fast-paced environment
  • Performing and documenting data analysis, data validation, and data mapping/design

 

 

Key Performance Indicators (Indicate how performance will be measured: indicators, activities…)

KPIs will be outlined in detail in the goal sheet

 

 

Ideal Background (State the minimum and desirable education and experience level)

 

Education

Minimum:  Graduation, preferably in Science

Experience requirement:                                      

·        Minimum: 2-3 years’ relevant work experience in the field of reporting and data analytics using Tableau.

·        Tableau certifications would be preferred

·        Work experience in the regulated medical device / Pharmaceutical industry would be an added advantage, but not mandatory

Languages:

Minimum: English (written and spoken)

 

 

 

Specific Professional Competencies: Indicate any other soft/technical/professional knowledge and skills requirements

 

  • Extensive experience in developing, maintaining and managing Tableau driven dashboards & analytics and working knowledge of Tableau administration /architecture.
  • A solid understanding of SQL, rational databases, and normalization
  • Proficiency in use of query and reporting analysis tools
  • Competency in Excel (macros, pivot tables, etc.)
  • Degree in Mathematics, Computer Science, Information Systems, or related field.

 

 

Read more
Job posted by
Sanjay Biswakarma

Data Engineer- SQL+PySpark

at Fragma Data Systems

Founded 2015  •  Products & Services  •  employees  •  Profitable
Spark
PySpark
Big Data
Python
SQL
Windows Azure
icon
Remote, Bengaluru (Bangalore)
icon
1 - 5 yrs
icon
₹5L - ₹15L / yr
Must-Have Skills:
• Good experience in Pyspark - Including Dataframe core functions and Spark SQL
• Good experience in SQL DBs - Be able to write queries including fair complexity.
• Should have excellent experience in Big Data programming for data transformation and aggregations
• Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
• Good customer communication.
• Good Analytical skill
 
 
Technology Skills (Good to Have):
  • Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
  • Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. 
  • Designing and implementing data engineering, ingestion, and transformation functions
  • Azure Synapse or Azure SQL data warehouse
  • Spark on Azure is available in HD insights and data bricks
Read more
Job posted by
Evelyn Charles
PySpark
SQL
Data engineering
Big Data
Hadoop
Spark
icon
Remote only
icon
4.5 - 12 yrs
icon
₹20L - ₹30L / yr

Must Have Skills:

 

  • Good experience in Pyspark - Including Dataframe core functions and Spark SQL
  • Good experience in SQL DBs - Be able to write queries including fair complexity.
  • Should have excellent experience in Big Data programming for data transformation and aggregations
  • Good at ELT architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
  • Good customer communication.
  • Good Analytical skills

 

Technology Skills (Good to Have):

 

  • Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
  • Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. 
  • Designing and implementing data engineering, ingestion, and transformation functions
  • Azure Synapse or Azure SQL data warehouse
  • Spark on Azure is available in HD insights and data bricks

 

Read more
Job posted by
Evelyn Charles
PySpark
Python
Spark
icon
Bengaluru (Bangalore)
icon
3 - 7 yrs
icon
₹8L - ₹16L / yr
Roles and Responsibilities:

• Responsible for developing and maintaining applications with PySpark 
• Contribute to the overall design and architecture of the application developed and deployed.
• Performance Tuning wrt to executor sizing and other environmental parameters, code optimization, partitions tuning, etc.
• Interact with business users to understand requirements and troubleshoot issues.
• Implement Projects based on functional specifications.

Must-Have Skills:

• Good experience in Pyspark - Including Dataframe core functions and Spark SQL
• Good experience in SQL DBs - Be able to write queries including fair complexity.
• Should have excellent experience in Big Data programming for data transformation and aggregations
• Good at ETL architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.
• Good customer communication.
• Good Analytical skills
Read more
Job posted by
Priyanka U
Data engineering
Python
SQL
Spark
PySpark
Cassandra
Groovy
Amazon Web Services (AWS)
Amazon S3
Windows Azure
Foundry
Good Clinical Practice
E2
R
palantir
icon
Bengaluru (Bangalore), Pune, Noida, NCR (Delhi | Gurgaon | Noida)
icon
7 - 10 yrs
icon
₹20L - ₹25L / yr
  1. Sr. Data Engineer:

 Core Skills – Data Engineering, Big Data, Pyspark, Spark SQL and Python

Candidate with prior Palantir Cloud Foundry OR Clinical Trial Data Model background is preferred

Major accountabilities:

  • Responsible for Data Engineering, Foundry Data Pipeline Creation, Foundry Analysis & Reporting, Slate Application development, re-usable code development & management and Integrating Internal or External System with Foundry for data ingestion with high quality.
  • Have good understanding on Foundry Platform landscape and it’s capabilities
  • Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
  • Defines company data assets (data models), Pyspark, spark SQL, jobs to populate data models.
  • Designs data integrations and data quality framework.
  • Design & Implement integration with Internal, External Systems, F1 AWS platform using Foundry Data Connector or Magritte Agent
  • Collaboration with data scientists, data analyst and technology teams to document and leverage their understanding of the Foundry integration with different data sources - Actively participate in agile work practices
  • Coordinating with Quality Engineer to ensure the all quality controls, naming convention & best practices have been followed

Desired Candidate Profile :

  • Strong data engineering background
  • Experience with Clinical Data Model is preferred
  • Experience in
    • SQL Server ,Postgres, Cassandra, Hadoop, and Spark for distributed data storage and parallel computing
    • Java and Groovy for our back-end applications and data integration tools
    • Python for data processing and analysis
    • Cloud infrastructure based on AWS EC2 and S3
  • 7+ years IT experience, 2+ years’ experience in Palantir Foundry Platform, 4+ years’ experience in Big Data platform
  • 5+ years of Python and Pyspark development experience
  • Strong troubleshooting and problem solving skills
  • BTech or master's degree in computer science or a related technical field
  • Experience designing, building, and maintaining big data pipelines systems
  • Hands-on experience on Palantir Foundry Platform and Foundry custom Apps development
  • Able to design and implement data integration between Palantir Foundry and external Apps based on Foundry data connector framework
  • Hands-on in programming languages primarily Python, R, Java, Unix shell scripts
  • Hand-on experience in AWS / Azure cloud platform and stack
  • Strong in API based architecture and concept, able to do quick PoC using API integration and development
  • Knowledge of machine learning and AI
  • Skill and comfort working in a rapidly changing environment with dynamic objectives and iteration with users.

 Demonstrated ability to continuously learn, work independently, and make decisions with minimal supervision

Read more
Job posted by
RAHUL BATTA

ETL Talend developer

at Rivet Systems Pvt Ltd.

Founded 2011  •  Products & Services  •  20-100 employees  •  Profitable
ETL
Hadoop
Big Data
Pig
Spark
Apache Hive
Talend
icon
Bengaluru (Bangalore)
icon
5 - 19 yrs
icon
₹10L - ₹30L / yr
Strong exposure in ETL / Big Data / Talend / Hadoop / Spark / Hive / Pig

To be considered as a candidate for a Senior Data Engineer position, a person must have a proven track record of architecting data solutions on current and advanced technical platforms. They must have leadership abilities to lead a team providing data centric solutions with best practices and modern technologies in mind. They look to build collaborative relationships across all levels of the business and the IT organization. They possess analytic and problem-solving skills and have the ability to research and provide appropriate guidance for synthesizing complex information and extract business value. Have the intellectual curiosity and ability to deliver solutions with creativity and quality. Effectively work with business and customers to obtain business value for the requested work. Able to communicate technical results to both technical and non-technical users using effective story telling techniques and visualizations. Demonstrated ability to perform high quality work with innovation both independently and collaboratively.

Read more
Job posted by
Shobha B K
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at A Global IT Service company?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort