Databricks Engineer

About the Role

We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.

Key Responsibilities

• Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
• Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
• Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
• Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
• Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
• Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.

Required Skills

• Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
• Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
• Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
• Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
• Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
• Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
• Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
• Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
• Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
• Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.

Nice to Have

• Hands-on exposure to Microsoft Fabric for data integration and analytics.
• Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
• Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
• Domain expertise in financial services.

About the Role

We are looking for a Forward Deployed Engineer with strong hands-on experience in Databricks and AI-assisted software development using Cursor. The role is suited for an engineer who can work directly with clients and internal teams to understand business problems, rapidly build solutions, and take them from prototype to production.

The ideal candidate should have strong expertise in Python, SQL, Databricks, PySpark, data engineering, APIs, and modern AI-assisted development workflows, along with excellent problem-solving and client-facing skills.

Key Responsibilities

Forward Deployed Engineering

• Work directly with clients and stakeholders to understand business and technical requirements.
• Translate business problems into scalable technical and data solutions.
• Rapidly prototype, test, iterate, and productionize solutions.
• Collaborate with engineering, data, AI, and delivery teams to implement customer solutions.
• Troubleshoot production issues and continuously improve deployed solutions.
• Act as a technical bridge between clients and internal engineering teams.

Databricks & Data Engineering

• Design and develop scalable data solutions using Databricks, PySpark, Python, and SQL.
• Build and optimize ETL/ELT and data processing pipelines.
• Work with Databricks Lakehouse, Delta Lake, Unity Catalog, and Databricks Workflows.
• Develop data ingestion and transformation pipelines for structured and semi-structured data.
• Optimize Databricks workloads for performance, scalability, reliability, and cost.
• Integrate Databricks with databases, APIs, cloud platforms, and enterprise applications.

Cursor & AI-Assisted Development

• Use Cursor and AI-assisted development workflows to accelerate software development, debugging, refactoring, and documentation.
• Effectively use AI coding assistants to understand existing codebases and develop new features.
• Apply appropriate engineering judgment to review, validate, test, and secure AI-generated code.
• Use AI-assisted development for rapid prototyping and proof-of-concept development.
• Work with modern AI/LLM APIs and tools where required for customer solutions.
• Stay current with emerging AI-assisted software engineering practices.

Production & Deployment

• Develop production-ready applications, APIs, and data pipelines.
• Work with Git, CI/CD, APIs, containers, and cloud environments.
• Monitor application and pipeline performance and resolve production issues.
• Ensure solutions meet requirements for scalability, security, reliability, and maintainability.
• Collaborate with Data Scientists and ML Engineers to integrate AI/ML capabilities into production systems.

Required Skills & Experience

• 4+ years of experience in Software Engineering, Data Engineering, AI Engineering, or a related field.
• Strong hands-on experience with Databricks.
• Strong proficiency in Python, PySpark, and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Experience building production-grade data pipelines.
• Hands-on experience with Cursor or similar AI-powered coding assistants.
• Strong understanding of REST APIs and system integrations.
• Experience working with cloud platforms such as AWS, Azure, or GCP.
• Strong debugging, problem-solving, and analytical skills.
• Excellent communication and client-facing abilities.

Preferred Skills

• Experience with Databricks Unity Catalog, Workflows, and MLflow.
• Experience with Generative AI / LLM applications.
• Knowledge of Claude, OpenAI, Azure OpenAI, or other LLM platforms.
• Experience with RAG, vector databases, embeddings, or AI agents.
• Experience with Docker, Kubernetes, and CI/CD.
• Experience in a consulting, customer-facing engineering, or professional services environment.
• Exposure to Agile/Scrum methodologies.

Key Competencies

• Strong problem-solving and ownership mindset
• Ability to work in ambiguous and fast-paced environments.
• Strong client/stakeholder management skills.
• Ability to understand business requirements and convert them into technical solutions.
• Strong communication and presentation skills.
• Ability to rapidly learn new technologies and tools.
• Comfortable working with AI-assisted development while maintaining high engineering standards.

Education

• Bachelor's or master’s degree in computer science, Information Technology, Engineering, Data Science, or a related discipline.

We are looking for a skilled Python & PySpark Developer with strong expertise in Big Data technologies, Spark, SQL/PL-SQL, and REST API development using Flask or Django. The ideal candidate should have experience building scalable data pipelines, processing large datasets, developing APIs, and working with distributed computing frameworks.

Key Responsibilities

• Develop, optimize, and maintain scalable data pipelines using PySpark and Apache Spark.
• Design, develop, and optimize complex SQL and PL/SQL queries, stored procedures, functions, and database objects.
• Build and maintain RESTful APIs using Flask or Django.
• Develop robust Python applications for data engineering and backend services.
• Process and analyze large-scale datasets using Big Data technologies.
• Optimize Spark jobs for performance, scalability, and reliability.
• Integrate APIs with internal and external systems.
• Collaborate with cross-functional teams including Data Engineers, Data Scientists, and Application Developers.
• Troubleshoot production issues and implement performance improvements.
• Follow coding standards, version control, and CI/CD best practices.

Mandatory Skills

• Strong proficiency in Python programming.
• Hands-on experience with PySpark and Apache Spark.
• Strong SQL coding skills.
• Experience with PL/SQL development.
• Experience in Big Data ecosystem.
• REST API development using Flask or Django.
• Experience in developing and consuming Python APIs.
• Knowledge of data processing, ETL, and distributed computing.
• Experience with Git/version control.

Preferred Skills

• Experience with Hadoop ecosystem (Hive, HDFS, YARN).
• Exposure to cloud platforms such as AWS, Azure, or GCP.
• Knowledge of Airflow or other workflow orchestration tools.
• Experience with Docker and Kubernetes.
• Familiarity with Kafka or other streaming technologies.
• Understanding of CI/CD pipelines.

Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• 4–8+ years of experience in Python and Big Data development (can be adjusted based on the role).

Required Experience

• Strong hands-on experience in Python, PySpark, and Apache Spark.
• Extensive experience writing optimized SQL and PL/SQL code.
• Experience developing REST APIs using Flask or Django.
• Experience working with large-scale data processing and ETL pipelines.
• Strong analytical, debugging, and problem-solving skills.

Mandatory Skills: Python, PySpark, SQL Coding, Apache Spark, Big Data, Flask/Django (REST API), PL/SQL, Python APIs.

Responsibilities and JD

Job Description: We are looking for a Senior Developer with strong expertise in PySpark, Databricks, and Snowflake to build scalable data engineering solutions and enterprise data platforms.

Key Responsibilities:

1. Design, develop, and maintain ETL/ELT pipelines using PySpark, Databricks, and Snowflake.
2. Develop batch and real-time data processing solutions for structured and semi-structured data.
3. Build and optimize Databricks notebooks, workflows, and Delta Lake solutions.
4. Design and implement Snowflake databases, schemas, views, stored procedures, tasks, and streams.
5. Develop scalable data models, data marts, and data warehouse solutions.
6. Optimize PySpark jobs, Databricks workloads, and Snowflake queries for performance and cost efficiency.
7. Implement data quality, validation, governance, and security controls.
8. Collaborate with business stakeholders, architects, and cross-functional teams to deliver data solutions.
9. Manage source control and CI/CD deployments using Git and Azure DevOps.
10. Troubleshoot production issues, perform root cause analysis, and ensure pipeline reliability.
11. Mentor junior team members and participate in code reviews and technical design discussions.

Required Skills: PySpark, Databricks, Snowflake, Python, SQL.

Experience: 5+ years of Data Engineering experience with strong hands-on expertise in PySpark, Databricks, and Snowflake.

1st virtual , 2nd round F2F

Python pyspark, SQL, data engineer

5+yrs

9+yrs

Bangalore/Hyderabad

immediate to 15days.

Data Engineer Hiring Post

🚨 Hiring: Data Engineer | PySpark + Python + SQL

We are looking for experienced Data Engineers to join our team!

🔹 Experience: 5 to 9 Years

🔹 Locations: Bangalore / Hyderabad

🔹 Interview Process:

• 1st Round – Virtual

• 2nd Round – Face-to-Face (Karat Test)

🔑 Key Skills:

✅ PySpark

✅ SQL

✅ Python

✅ ETL

📩 Interested candidates can share their updated resume.

#Hiring #DataEngineer #PySpark #Python #SQL #ETL #BangaloreJobs #HyderabadJobs #TechHiring #ImmediateHiring

Job Summary

We are looking for a skilled and experienced Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Python, PySpark, Data Modeling, and Power BI, with hands-on experience in designing, developing, and optimizing scalable data solutions. The role requires working closely with business stakeholders, data architects, and analytics teams to build robust data pipelines and semantic models that enable data-driven decision-making.

Technical Skills

• Strong hands-on experience in Python and PySpark development.
• Expertise in building and optimizing Data Engineering solutions and ETL pipelines.
• Strong understanding of Data Modeling concepts (Star Schema, Snowflake Schema, Dimensional Modeling).
• Experience with Power BI Data Modeling and Semantic Layer development.
• Proficiency in DAX (Data Analysis Expressions).
• Experience designing and managing Semantic Models in Power BI.
• Strong SQL skills and experience working with large datasets.
• Knowledge of data warehousing concepts and best practices.

Preferred Skills

• Experience with cloud platforms such as Azure, AWS, or GCP.
• Exposure to modern data platforms like Databricks.
• Understanding of data governance and data quality frameworks.

Senior Data Engineer – PySpark & Oracle

Experience: 7+ Years

Location: Bangalore

Notice Period: Immediate to 10 Days

Key Skills:

• Strong expertise in Data Modeling, Data Design & Modernization
• Primary skills: PySpark, Oracle SQL/PLSQL
• Secondary skills: Python, ETL & Data Pipelines
• Experience with Kafka and Hadoop
• Exposure to AWS / Azure / GCP
• Good knowledge of Git and JIRA

Roles & Responsibilities:

• Design, develop, and modernize scalable data models and data architecture.
• Develop and optimize data processing solutions using PySpark and Oracle SQL/PLSQL.
• Build and maintain robust ETL workflows and data pipelines.
• Work with Kafka, Hadoop, and cloud platforms for data processing and integration.
• Perform data transformation, optimization, and performance tuning.
• Collaborate with technical teams on data design, development, testing, and deployment.

Job Description – Azure Data Engineer

Role: Azure Data Engineer

Experience: 9+ Years

Location: Bangalore / Hyderabad

Notice Period: Immediate to 15 Days

Interview Process: 1st Round – Virtual | 2nd Round – F2F

Mandatory Skills

• Python
• PySpark
• SQL
• Azure Data Engineering

Job Description

We are looking for an experienced Azure Data Engineer with 9+ years of experience and strong hands-on expertise in Python, PySpark, SQL, and Azure Data Engineering.

Key Responsibilities

• Develop and maintain scalable data engineering solutions using Azure.
• Build and optimize data processing pipelines using PySpark and Python.
• Write complex SQL queries for data extraction and transformation.
• Work with Azure data services and cloud-based data platforms.
• Perform data processing, transformation, and integration.
• Troubleshoot data pipeline and production issues.
• Collaborate with technical and business teams to deliver data solutions.

Preferred: Immediate to 15 Days joiners.

🚨 WE ARE HIRING – SENIOR DATA ENGINEER | BANGALORE 🚨

Looking for experienced Senior Data Engineers with strong expertise in PySpark, Oracle SQL/PLSQL, and Data Modeling!

🔹 Role: Senior Data Engineer

🔹 Experience: 7+ Years

🔹 Location: Bangalore

🔹 CTC: Up to 26 LPA

🔹 Notice Period: Immediate to 10 Days Preferred

💻 Mandatory Skills

✅ PySpark

✅ Oracle SQL / PL/SQL

✅ Data Modeling & Design / Modernization

✅ Python

✅ ETL

✅ Data Pipelines

⚙️ Good to Have / Ecosystem Skills

✅ Kafka

✅ Hadoop

✅ AWS / Azure / GCP

✅ Git

✅ JIRA

📌 Key Responsibilities

• Develop and maintain scalable data pipelines using PySpark

• Work extensively with Oracle SQL/PLSQL for data processing and transformation

• Design and implement data models and data architecture

• Work on data design and modernization initiatives

• Develop and optimize ETL processes and data pipelines

• Handle large volumes of data using PySpark and Python

• Work with technologies such as Kafka, Hadoop, and Cloud platforms

• Collaborate with cross-functional teams to deliver scalable data solutions

• Use Git and JIRA for version control and project tracking

📩 Interested candidates can share their updated CV along with:

Total Experience:

Relevant PySpark Experience:

Relevant Oracle SQL/PLSQL Experience:

Data Modeling Experience:

Current Location:

Notice Period:

Current CTC:

Expected CTC:

#Hiring #DataEngineer #SeniorDataEngineer #PySpark #Oracle #PLSQL #DataModeling #Python #ETL #DataPipelines #Kafka #Hadoop #AWS #Azure #GCP #BangaloreJobs #ITJobs #TechJobs #ImmediateJoiners