Azure Data Engineer

Job Description – Data Engineer

Role: Data Engineer

Experience: 9+ Years

Location: Bangalore / Hyderabad

Notice Period: Immediate to 15 Days

Interview Process: 1st Round – Virtual | 2nd Round – F2F

Mandatory Skills

• Python
• PySpark
• SQL
• Azure Data Engineering

Job Description

We are looking for an experienced Azure Data Engineer with 9+ years of experience and strong hands-on expertise in Python, PySpark, SQL, and Azure Data Engineering.

Key Responsibilities

• Develop and maintain scalable data engineering solutions using Azure.
• Build and optimize data processing pipelines using PySpark and Python.
• Write complex SQL queries for data extraction and transformation.
• Work with Azure data services and cloud-based data platforms.
• Perform data processing, transformation, and integration.
• Troubleshoot data pipeline and production issues.
• Collaborate with technical and business teams to deliver data solutions.

Preferred: Immediate to 15 Days joiners.

8-12 years experience on Data Engineering role working with Databricks & Azure Cloud technologies.

• Bachelor’s degree in computer science, Information Technology, or related field.

• Strong proficiency in PySpark, Python, SQL.

• Strong experience in data modeling, ETL/ELT pipeline development, and automation • Hands-on experience with performance tuning of data pipelines and workflow

• Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.

• Experience with data modeling, ETL processes, Delta Lake and data warehousing.

• Experience on Delta Live Tables, Autoloader & Unity Catalog.

Preferred - Knowledge of the insurance industry and its data requirements.

• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.

• Excellent communication and problem-solving skills to work effectively with diverse teams

• Excellent problem-solving skills and ability to work under tight deadlines. 

Solution Architect – AZURE Data Engineering

Job Overview

We are looking for an experienced Solution Architect – Data Engineering with strong expertise in designing data solutions and hands-on experience with Azure, Synapse, PySpark, Data Warehousing, and Data Lakes. The ideal candidate should have strong architectural and data engineering knowledge.

Key Responsibilities

• Design and implement scalable data architecture and solutions.
• Develop and manage Data Warehouse and Data Lake architectures.
• Design data platforms using Medallion Architecture.
• Lead Data Engineering and ETL activities.
• Work with Azure Synapse Analytics for data processing and analytics.
• Develop data solutions using PySpark / Apache Spark.
• Define and implement data validation and data quality processes.
• Collaborate with business, data, and technology teams to deliver effective data solutions.

Required Skills

• Strong experience in Solution Architecture / Data Architecture.
• Strong knowledge of Data Warehouse and Data Lake architecture.
• Good understanding of Medallion Architecture.
• Strong experience in Data Engineering and ETL.
• Hands-on experience with Microsoft Azure and Azure Synapse Analytics.
• Strong knowledge of PySpark / Apache Spark.
• Experience with Data Validation and Data Quality.
• Good communication and stakeholder management skills.
• 
  

Experience

8+ Years

role: data engineer

Python pyspark, SQL, data engineer

5+yrs

Bangalore/Hyderabad

immediate to 15days.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

9+yrs

Bangalore/Hyderabad

immediate to 15days.

Responsibilities and JD

Job Description: We are looking for a Senior Developer with strong expertise in PySpark, Databricks, and Snowflake to build scalable data engineering solutions and enterprise data platforms.

Key Responsibilities:

1. Design, develop, and maintain ETL/ELT pipelines using PySpark, Databricks, and Snowflake.
2. Develop batch and real-time data processing solutions for structured and semi-structured data.
3. Build and optimize Databricks notebooks, workflows, and Delta Lake solutions.
4. Design and implement Snowflake databases, schemas, views, stored procedures, tasks, and streams.
5. Develop scalable data models, data marts, and data warehouse solutions.
6. Optimize PySpark jobs, Databricks workloads, and Snowflake queries for performance and cost efficiency.
7. Implement data quality, validation, governance, and security controls.
8. Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.
9. Manage source control and CI/CD deployments using Git and Azure DevOps.
10. Troubleshoot production issues, perform root cause analysis, and ensure pipeline reliability.
11. Mentor junior team members and participate in code reviews and technical design discussions.

Required Skills: PySpark, Databricks, Snowflake, Python, SQL.

Experience: 5+ years of Data Engineering experience with strong hands-on expertise in PySpark, Databricks, and Snowflake.

Job Description:

We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.

Key Responsibilities

·        5-8 years of professional work experience in a relevant field

·        Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.

·        Good Experience on Microsoft Dynamics 365

·        Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.

·        Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.

·        Good knowledge of SQL and Python for data manipulation, transformation, and analysis

·        Understand business requirements to set functional specifications for reporting applications

• Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.
• Data Integration: Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.
• Power BI Integration: Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.
• Performance Optimization: Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.
• Data Modeling: Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.
• Cloud Integration: Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.
• Security & Compliance: Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.
• Technical Skills:
• Proficiency in Databricks, including Delta Lake, Spark SQL.
• Strong programming skills in Python, Scala, or Java.
• Experience with Apache Spark for big data processing.
• Knowledge of SQL for querying and transforming data.
• Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.

Preferred Qualifications

• Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).

Experience with real-time data processing and streaming

About Us:

The QX Impact was launched with a mission to make A.I accessible and affordable and deliver AI Products/Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation. We believe without insights; businesses will continue to face challenges to better understand their customers and even lose them. Secondly, without insights businesses won't’ be able to deliver differentiated products/services; and finally, without insights, businesses can’t achieve a new level of “Operational Excellence” is crucial to remain competitive, meeting rising customer expectations, expanding markets, and digitalization.

Job Summary:

We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions.

As a Senior Data Engineer, you will work with internal and external data, collaborate with data scientists, and contribute to the design, development, and deployment of innovative solutions.

Key Responsibilities:

• Design, develop, test, and maintain optimal data pipeline and ETL architectures.
• Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
• Prepare and optimize data for predictive and prescriptive modeling.
• Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
• Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
• Utilize big data tools and frameworks to optimize data acquisition and preparation.
• Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
• Develop and curate data models for analytics, dashboards, and reports.
• Conduct code reviews, maintain production-level code, and implement testing approaches.
• Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
• Drive innovation and implement efficient new approaches to data engineering tasks.

Must-Have Skills:

• Bachelor’s degree in Computer Science, Mathematics, Engineering, or a related field.
• 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
• 3–5 years of experience designing and implementing data warehouse solutions.
• Expertise in SQL, including developing stored procedures (SP) and applying advanced data design concepts.
• Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
• Experience with AWS or Azure services (e.g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
• Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
• Hands-on experience with orchestration tools (e.g., Apache Airflow, Prefect).
• Knowledge of CI/CD processes, version control (e.g., Git, Jenkins), and deployment automation.
• Strong problem-solving, communication, and collaboration skills.

Good-to-Have Skills:

• Experience in integrating ERP data into data lakes.
• Experience with traditional ETL tools (e.g., Talend, Pentaho).

Competencies:

• Tech Savvy - Anticipating and adopting innovations in business-building digital and technology applications.
• Self-Development - Actively seeking new ways to grow and be challenged using both formal and informal development channels.
• Action Oriented - Taking on new opportunities and tough challenges with a sense of urgency, high energy, and enthusiasm.
• Customer Focus - Building strong customer relationships and delivering customer-centric solutions.
• Optimize Work Processes - Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.

Why Join Us?

• Be part of a collaborative and agile team driving cutting-edge AI and data engineering solutions.
• Work on impactful projects that make a difference across industries.
• Opportunities for professional growth and continuous learning.
• Competitive salary and benefits package.

Application Details

Ready to make an impact? Apply today and become part of the QX Impact team!

1st virtual , 2nd round F2F -

Python pyspark, SQL, data engineer

5+yrs

Bang/hyderabad

immediate to 15days.

Location: Bangalore Experience: 5 to 7 years Employment type: Full-time, permanent Work Hours: General Shift (10.00 AM to 7.00 PM) website: www.amazech.com Qualifications: Minimum B.E./B.Tech, or higher in Computer Science, Information Technology, Data Engineering, or a related field, with a good academic background. Key Responsibilities: • Design, develop, and maintain end-to-end data pipelines. • Build and optimize ETL/ELT processes for large-scale data processing. • Implement Azure-based data solutions ensuring scalability and performance. • Collaborate with stakeholders to understand business data requirements. • Perform performance tuning and optimization across data platforms. • Support production environments, conduct root cause analysis, and resolve data-related issues. • Maintain comprehensive technical documentation and structured knowledge transfer documents. • Work effectively within Agile/Scrum frameworks and contribute to sprint planning and delivery. • Ensure secure and compliant data handling using Azure best practices. Required Skills & Experience • End-to-end ETL/ELT pipeline development, integration, and performance optimization • Azure Synapse, Azure Logic Apps, Azure SQL, and Azure Databricks, with a strong focus on performance tuning • Microsoft Azure services, including Storage Accounts, Key Vault, and Cognitive Services • Advanced proficiency in Python development and modern productivity tools such as GitHub Copilot and Cursor • Strong documentation discipline, including technical design documentation, and structured knowledge transfer documents • Experience operating within Agile/Scrum frameworks, including production support, root cause analysis, and effective stakeholder collaboration