About the Role:
We are seeking an experienced Data Engineer to lead and execute the migration of existing Databricks-based pipelines to Snowflake. The role requires strong expertise in PySpark/Spark, Snowflake, DBT, and Airflow with additional exposure to DevOps and CI/CD practices. The candidate will be responsible for re-architecting data
pipelines, ensuring data consistency, scalability, and performance in Snowflake, and enabling robust automation and monitoring across environments.
Key Responsibilities
Databricks to Snowflake Migration
· Analyze and understand existing pipelines and frameworks in Databricks (PySpark/Spark).
· Re-architect pipelines for execution in Snowflake using efficient SQL-based processing.
· Translate Databricks notebooks/jobs into Snowflake/DBT equivalents.
· Ensure a smooth transition with data consistency, performance, and scalability.
Snowflake
· Hands-on experience with storage integrations, staging (internal/external), Snowpipe, tables/views, COPY INTO, CREATE OR ALTER, and file formats.
· Implement RBAC (role-based access control), data governance, and performance tuning.
· Design and optimize SQL queries for large-scale data processing.
DBT (with Snowflake)
· Implement and manage models, macros, materializations, and SQL execution within DBT.
· Use DBT for modular development, version control, and multi-environment deployments.
Airflow (Orchestration)
· Design and manage DAGs to automate workflows and ensure reliability.
· Handle task dependencies, error recovery, monitoring, and integrations (Cosmos, Astronomer, Docker).
DevOps & CI/CD
· Develop and manage CI/CD pipelines for Snowflake and DBT using GitHub Actions, Azure DevOps, or equivalent.
· Manage version-controlled environments and ensure smooth promotion of changes across dev, test, and prod.
Monitoring & Observability
· Implement monitoring, alerting, and logging for data pipelines.
· Build self-healing or alert-driven mechanisms for critical/severe issue detection.
· Ensure system reliability and proactive issue resolution.
Required Skills & Qualifications
· 5+ years of experience in data engineering with focus on cloud data platforms.
· Strong expertise in:
· Databricks (PySpark/Spark) – analysis, transformations, dependencies.
· Snowflake – architecture, SQL, performance tuning, security (RBAC).
· DBT – modular model development, macros, deployments.
· Airflow – DAG design, orchestration, and error handling.
· Experience in CI/CD pipeline development (GitHub Actions, Azure DevOps).
· Solid understanding of data modeling, ETL/ELT processes, and best practices.
· Excellent problem-solving, communication, and stakeholder collaboration skills.
Good to Have
· Exposure to Docker/Kubernetes for orchestration.
· Knowledge of Azure Data Services (ADF, ADLS) or similar cloud tools.
· Experience with data governance, lineage, and metadata management.
Education
· Bachelor’s / Master’s degree in Computer Science, Engineering, or related field.