Data Engineer

Job Summary

We are seeking an experienced Databricks Developer with strong skills in PySpark, SQL, Python, and hands-on experience deploying data solutions on AWS (preferred), Azure. The role involves designing, developing, and optimizing scalable data pipelines and analytics workflows on the Databricks platform.

Key Responsibilities

- Develop and optimize ETL/ELT pipelines using Databricks and PySpark.

- Build scalable data workflows on AWS (EC2, S3, Glue, Lambda, IAM) or Azure (ADF, ADLS, Synapse).

- Implement and manage Delta Lake (ACID, schema evolution, time travel).

- Write efficient, complex SQL for transformation and analytics.

- Build and support batch and streaming ingestion (Kafka, Kinesis, EventHub).

- Optimize Databricks clusters, jobs, notebooks, and PySpark performance.

- Collaborate with cross-functional teams to deliver reliable data solutions.

- Ensure data governance, security, and compliance.

- Troubleshoot pipelines and support CI/CD deployments.

Required Skills & Experience

- 4–8 years in Data Engineering / Big Data development.

- Strong hands-on experience with Databricks (clusters, jobs, workflows).

- Advanced PySpark and strong Python skills.

- Expert-level SQL (complex queries, window functions).

- Practical experience with AWS (preferred) or Azure cloud services.

- Experience with Delta Lake, Parquet, and data lake architectures.

- Familiarity with CI/CD tools (GitHub Actions, Azure DevOps, Jenkins).

- Good understanding of data modeling, optimization, and distributed systems.