11+ ETL management Jobs in Bangalore (Bengaluru) | ETL management Job openings in Bangalore (Bengaluru)

Your key responsibilities

• Create and maintain optimal data pipeline architecture. Should have experience in building batch/real-time ETL Data Pipelines. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources
• The individual will be responsible for solution design, integration, data sourcing, transformation, database design and implementation of complex data warehousing solutions.
• Responsible for development, support, maintenance, and implementation of a complex project module
• Provide expertise in area and advanced knowledge of applications programming and ensure application design adheres to the overall architecture blueprint
• Utilize advanced knowledge of system flow and develop standards for coding, testing, debugging, and implementation
• Resolve variety of high impact problems/projects through in-depth evaluation of complex business processes, system processes, and industry standards
• Continually improve ongoing reporting and analysis processes, automating or simplifying self-service support.
• complete reporting solutions.
• Preparation of HLD about architecture of the application and high level design.
• Preparation of LLD about job design, job description and in detail information of the jobs.
• Preparation of Unit Test cases and execution of the same.
• Provide technical guidance and mentoring to application development teams throughout all the phases of the software development life cycle

Skills and attributes for success

• Strong experience in SQL. Proficient in writing performant SQL working with large data volumes. Proficiency in writing and debugging complex SQLs.
• Strong experience in database system Microsoft Azure. Experienced in Azure Data Factory.
• Strong in Data Warehousing concepts. Experience with large-scale data warehousing architecture and data modelling.

• Should have enough experience to work on Power Shell Scripting

• Able to guide the team through the development, testing and implementation stages and review the completed work effectively
• Able to make quick decisions and solve technical problems to provide an efficient environment for project implementation
• Primary owner of delivery, timelines. Review code was written by other engineers.
• Maintain highest levels of development practices including technical design, solution development, systems configuration, test documentation/execution, issue identification and resolution, writing clean, modular and self-sustaining code, with repeatable quality and predictability
• Must have understanding of business intelligence development in the IT industry
• Outstanding written and verbal communication skills
• Should be adept in SDLC process - requirement analysis, time estimation, design, development, testing and maintenance
• Hands-on experience in installing, configuring, operating, and monitoring CI/CD pipeline tools
• Should be able to orchestrate and automate pipeline
• Good to have : Knowledge of distributed systems such as Hadoop, Hive, Spark

To qualify for the role, you must have

• Bachelor's Degree in Computer Science, Economics, Engineering, IT, Mathematics, or related field preferred
• More than 6 years of experience in ETL development projects
• Proven experience in delivering effective technical ETL strategies
• Microsoft Azure project experience
• Technologies: ETL- ADF, SQL, Azure components (must-have), Python (nice to have)

Ideally, you’ll also have

Job Responsibilities

• Design, build & test ETL processes using Python & SQL for the corporate data warehouse
• Inform, influence, support, and execute our product decisions
• Maintain advertising data integrity by working closely with R&D to organize and store data in a format that provides accurate data and allows the business to quickly identify issues.
• Evaluate and prototype new technologies in the area of data processing
• Think quickly, communicate clearly and work collaboratively with product, data, engineering, QA and operations teams
• High energy level, strong team player and good work ethic
• Data analysis, understanding of business requirements and translation into logical pipelines & processes
• Identification, analysis & resolution of production & development bugs
• Support the release process including completing & reviewing documentation
• Configure data mappings & transformations to orchestrate data integration & validation
• Provide subject matter expertise
• Document solutions, tools & processes
• Create & support test plans with hands-on testing
• Peer reviews of work developed by other data engineers within the team
• Establish good working relationships & communication channels with relevant departments

Skills and Qualifications we look for

• University degree 2.1 or higher (or equivalent) in a relevant subject. Master’s degree in any data subject will be a strong advantage.
• 4 - 6 years experience with data engineering.
• Strong coding ability and software development experience in Python.
• Strong hands-on experience with SQL and Data Processing.
• Google cloud platform (Cloud composer, Dataflow, Cloud function, Bigquery, Cloud storage, dataproc)
• Good working experience in any one of the ETL tools (Airflow would be preferable).
• Should possess strong analytical and problem solving skills.
• Good to have skills - Apache pyspark, CircleCI, Terraform
• Motivated, self-directed, able to work with ambiguity and interested in emerging technologies, agile and collaborative processes.
• Understanding & experience of agile / scrum delivery methodology

You will be responsible for designing, building, and maintaining data pipelines that handle Real-world data at Compile. You will be handling both inbound and outbound data deliveries at Compile for datasets including Claims, Remittances, EHR, SDOH, etc.

You will

• Work on building and maintaining data pipelines (specifically RWD).
• Build, enhance and maintain existing pipelines in pyspark, python and help build analytical insights and datasets.
• Scheduling and maintaining pipeline jobs for RWD.
• Develop, test, and implement data solutions based on the design.
• Design and implement quality checks on existing and new data pipelines.
• Ensure adherence to security and compliance that is required for the products.
• Maintain relationships with various data vendors and track changes and issues across vendors and deliveries.

You have

• Hands-on experience with ETL process (min of 5 years).
• Excellent communication skills and ability to work with multiple vendors.
• High proficiency with Spark, SQL.
• Proficiency in Data modeling, validation, quality check, and data engineering concepts.
• Experience in working with big-data processing technologies using - databricks, dbt, S3, Delta lake, Deequ, Griffin, Snowflake, BigQuery.
• Familiarity with version control technologies, and CI/CD systems.
• Understanding of scheduling tools like Airflow/Prefect.
• Min of 3 years of experience managing data warehouses.
• Familiarity with healthcare datasets is a plus.

Compile embraces diversity and equal opportunity in a serious way. We are committed to building a team of people from many backgrounds, perspectives, and skills. We know the more inclusive we are, the better our work will be.         

1.      Core Responsibilities

·        Leading solutions for data engineering

·        Maintain the integrity of both the design and the data that is held within the architecture

·        Champion and educate people in the development and use of data engineering best practises

·        Support the Head of Data Engineering and lead by example

·        Contribute to the development of database management services and associated processes relating to the delivery of data solutions

·        Provide requirements analysis, documentation, development, delivery and maintenance of data platforms.

·        Develop database requirements in a structured and logical manner ensuring delivery is aligned with business prioritisation and best practise

·        Design and deliver performance enhancements, application migration processes and version upgrades across a pipeline of BI environments.

·        Provide support for the scoping and delivery of BI capability to internal users.

·        Identify risks and issues and escalate to Line / Project manager. 

·        Work with clients, existing asset owners & their service providers and non BI development staff to clarify and deliver work stream objectives in timescales that deliver to the overall project expectations.

·        Develop and maintain documentation in support of all BI processes.

·        Proactively identify cost-justifiable improvements to data manipulation processes.

·        Research and promote relevant BI tools and processes that contribute to increased efficiency and capability in support of corporate objectives.

·        Promote a culture that embraces change, continuous improvement and a ‘can do’ attitude.

·        Demonstrate enthusiasm and self-motivation at all times.

·        Establish effective working relationships with other internal teams to drive improved efficiency and effective processes.

·        Be a champion for high quality data and use of strategic data repositories, associated relational model, and Data Warehouse for optimising the delivery of accurate, consistent and reliable business intelligence

·        Ensure that you fully understand and comply with the organisation’s Risk Management Policies as they relate to your area of responsibility and demonstrate in your day to day work that you put customers at the heart of everything you do.

·        Ensure that you fully understand and comply with the organisation’s Data Governance Policies as they relate to your area of responsibility and demonstrate in your day to day work that you treat data as an important corporate asset which must be protected and managed.

·        Maintain the company’s compliance standards and ensure timely completion of all mandatory on-line training modules and attestations.

2.      Experience Requirements

·        5 years Data Engineering / ETL development experience is essential

·        5 years data design experience in an MI / BI / Analytics environment (Kimball, lake house, data lake) is essential

·        5 years experience of working in a structured Change Management project lifecycle is essential

·        Experience of working in a financial services environment is desirable

·        Experience of dealing with senior management within a large organisation is desirable

·        5 years experience of developing in conjunction with large complex projects and programmes is desirable

·        Experience mentoring other members of the team on best practise and internal standards is essential

·        Experience with cloud data platforms desirable (Microsoft Azure) is desirable

3.      Knowledge Requirements

·        A strong knowledge of business intelligence solutions and an ability to translate this into data solutions for the broader business is essential

·        Strong demonstrable knowledge of data warehouse methodologies

·        Robust understanding of high level business processes is essential

·        Understanding of data migration, including reconciliation, data cleanse and cutover is desirable

Role: Data Engineer  
Company: PayU

Location: Bangalore/ Mumbai

Experience : 2-5 yrs

About Company:

PayU is the payments and fintech business of Prosus, a global consumer internet group and one of the largest technology investors in the world. Operating and investing globally in markets with long-term growth potential, Prosus builds leading consumer internet companies that empower people and enrich communities.

The leading online payment service provider in 36 countries, PayU is dedicated to creating a fast, simple and efficient payment process for merchants and buyers. Focused on empowering people through financial services and creating a world without financial borders where everyone can prosper, PayU is one of the biggest investors in the fintech space globally, with investments totalling $700 million- to date. PayU also specializes in credit products and services for emerging markets across the globe. We are dedicated to removing risks to merchants, allowing consumers to use credit in ways that suit them and enabling a greater number of global citizens to access credit services.

Our local operations in Asia, Central and Eastern Europe, Latin America, the Middle East, Africa and South East Asia enable us to combine the expertise of high growth companies with our own unique local knowledge and technology to ensure that our customers have access to the best financial services.

India is the biggest market for PayU globally and the company has already invested $400 million in this region in last 4 years. PayU in its next phase of growth is developing a full regional fintech ecosystem providing multiple digital financial services in one integrated experience. We are going to do this through 3 mechanisms: build, co-build/partner; select strategic investments. 

PayU supports over 350,000+ merchants and millions of consumers making payments online with over 250 payment methods and 1,800+ payment specialists. The markets in which PayU operates represent a potential consumer base of nearly 2.3 billion people and a huge growth potential for merchants. 

Job responsibilities:

• Design infrastructure for data, especially for but not limited to consumption in machine learning applications 
• Define database architecture needed to combine and link data, and ensure integrity across different sources 
• Ensure performance of data systems for machine learning to customer-facing web and mobile applications using cutting-edge open source frameworks, to highly available RESTful services, to back-end Java based systems 
• Work with large, fast, complex data sets to solve difficult, non-routine analysis problems, applying advanced data handling techniques if needed 
• Build data pipelines, includes implementing, testing, and maintaining infrastructural components related to the data engineering stack.
• Work closely with Data Engineers, ML Engineers and SREs to gather data engineering requirements to prototype, develop, validate and deploy data science and machine learning solutions

Requirements to be successful in this role: 

• Strong knowledge and experience in Python, Pandas, Data wrangling, ETL processes, statistics, data visualisation, Data Modelling and Informatica.
• Strong experience with scalable compute solutions such as in Kafka, Snowflake
• Strong experience with workflow management libraries and tools such as Airflow, AWS Step Functions etc. 
• Strong experience with data engineering practices (i.e. data ingestion pipelines and ETL) 
• A good understanding of machine learning methods, algorithms, pipelines, testing practices and frameworks 
• Preferred) MEng/MSc/PhD degree in computer science, engineering, mathematics, physics, or equivalent (preference: DS/ AI) 
• Experience with designing and implementing tools that support sharing of data, code, practices across organizations at scale 

Job Description

The applicant must have a minimum of 5 years of hands-on IT experience, working on a full software lifecycle in Agile mode.

Good to have experience in data modeling and/or systems architecture.
Responsibilities will include technical analysis, design, development and perform enhancements.

You will participate in all/most of the following activities:
- Working with business analysts and other project leads to understand requirements.
- Modeling and implementing database schemas in DB2 UDB or other relational databases.
- Designing, developing, maintaining and Data processing using Python, DB2, Greenplum, Autosys and other technologies

Skills /Expertise Required :

Work experience in developing large volume database (DB2/Greenplum/Oracle/Sybase).

Good experience in writing stored procedures, integration of database processing, tuning and optimizing database queries.

Strong knowledge of table partitions, high-performance loading and data processing.
Good to have hands-on experience working with Perl or Python.
Hands on development using Spark / KDB / Greenplum platform will be a strong plus.
Designing, developing, maintaining and supporting Data Extract, Transform and Load (ETL) software using Informatica, Shell Scripts, DB2 UDB and Autosys.
Coming up with system architecture/re-design proposals for greater efficiency and ease of maintenance and developing software to turn proposals into implementations.

Need to work with business analysts and other project leads to understand requirements.
Strong collaboration and communication skills

• Hands-on experience in any Cloud Platform

• Microsoft Azure Experience

• Ensure and own Data integrity across distributed systems.
• Extract, Transform and Load data from multiple systems for reporting into BI platform.
• Create Data Sets and Data models to build intelligence upon.
• Develop and own various integration tools and data points.
• Hands-on development and/or design within the project in order to maintain timelines.
• Work closely with the Project manager to deliver on business requirements OTIF (on time in full)
• Understand the cross-functional business data points thoroughly and be SPOC for all data-related queries.
• Work with both Web Analytics and Backend Data analytics.
• Support the rest of the BI team in generating reports and analysis
• Quickly learn and use Bespoke & third party SaaS reporting tools with little documentation.
• Assist in presenting demos and preparing materials for Leadership.

• Strong experience in Datawarehouse modeling techniques and SQL queries

• A good understanding of designing, developing, deploying, and maintaining Power BI report solutions
• Ability to create KPIs, visualizations, reports, and dashboards based on business requirement
• Knowledge and experience in prototyping, designing, and requirement analysis
• Be able to implement row-level security on data and understand application security layer models in Power BI
• Proficiency in making DAX queries in Power BI desktop.
• Expertise in using advanced level calculations on data sets
• Experience in the Fintech domain and stakeholder management.

The Data Engineer would be responsible for selecting and integrating Big Data tools and frameworks required. Would implement Data Ingestion & ETL/ELT processes

Required Experience, Skills and Qualifications:

• Hands on experience on Big Data tools/technologies like Spark,  Databricks, Map Reduce, Hive, HDFS.
• Expertise and excellent understanding of big data toolset such as Sqoop, Spark-streaming, Kafka, NiFi
• Proficiency in any of the programming language: Python/ Scala/  Java with 4+ years’ experience
• Experience in Cloud infrastructures like MS Azure, Data lake etc
• Good working knowledge in NoSQL DB (Mongo, HBase, Casandra)