Lead Data Engineer at Searce Inc · Mumbai, Pune, Bengaluru (Bangalore) · 7 - 10 years · Profitable · Posted 10 Jul 2026
Amazon Web Services (AWS)
Python
ScalaLead Data Engineer
What are we looking for
real solver?
Solver? Absolutely. But not the usual kind. We're searching for the architects of the audacious & the pioneers of the possible. If you're the type to dismantle assumptions, re-engineer ‘best practices,’ and build solutions that make the future possible NOW, then you're speaking our language.
Your Responsibilities
What you will wake up to solve.
- Lead Technical Design & Data Architecture: Architect and lead the end-to-end development of scalable, cloud-native data platforms. You’ll guide the squad on critical architectural decisions—choosing between Batch vs. Streaming or ETL vs. ELT—while remaining 100% hands-on, contributing high-quality, production-grade code.
- Build High-Velocity Data Pipelines: Drive the implementation of robust data transports and ingestion frameworks using Python, SQL, and Spark. You will build integration layers that connect heterogeneous sources (SaaS, RDBMS, NoSQL) into unified, high-availability environments like BigQuery, Snowflake, or Redshift.
- Mentor & Elevate the Squad: Foster a culture of technical excellence by mentoring and inspiring a team of data analysts and engineers. Lead deep-dive code reviews, promote best-practice data modeling (Star/Snowflake schema), and ensure the squad adopts modern engineering standards like CI/CD for data.
- Drive AI-Ready Data Strategy: Be the expert in designing data foundations optimized for AI and Machine Learning. You will champion the use of GCP (Dataflow, Pub/Sub, BigQuery) and AWS (Lambda, Glue, EMR) to create "clean room" environments that fuel advanced analytics and generative AI models.
- Partner with Clients as a Technical DRI: Act as the Directly Responsible Individual for client success. Translate ambiguous business questions into elegant data services, manage project deliverables using Agile methodologies, and ensure that the data provided is accurate, consistent, and mission-critical.
- Troubleshoot & Optimize for Scale: Own the reliability of the reporting layer. You will proactively monitor pipelines, troubleshoot complex transformation bottlenecks, and propose ways to improve platform performance and cost-efficiency.
- Innovate and Build Reusable IP: Spearhead the creation of reusable data frameworks, custom operators, and transformation libraries that accelerate future projects and establish Searce’s unique technical advantage in the market.
Welcome to Searce
The AI-Native tech consultancy that's rewriting the rules.
Searce is an AI-native, engineering-led, modern tech consultancy that empowers clients to futurify their business by delivering intelligent, impactful, real business outcomes. Searce solvers co-innovate with clients as their trusted transformational partners ensuring sustained competitive advantage. Searce clients realize smarter, faster, better business outcomes delivered by AI-native Searce solver squads.
Functional Skills
the solver personas.
- The Data Architect: This persona deconstructs ambiguous business goals into scalable, elegant data blueprints. They don't just move data; they design the foundation—from schema design to partitioning strategies—that allows data scientists and analysts to thrive, foreseeing technical bottlenecks and making pragmatic trade-offs.
- The Player-Coach: As a hands-on leader, this persona leads from the front by writing exemplary, production-grade SQL and Python while simultaneously mentoring and elevating the skills of the squad. Their success is measured by the team's ability to deliver high-quality, maintainable code and their growth as engineers.
- The Pragmatic Innovator: This individual balances a passion for modern data tech (like Generative AI and Real-time Streaming) with a sharp focus on business outcomes. They champion new tools where they add real value but are disciplined enough to choose stable, cost-effective solutions to meet deadlines and deliver robust products.
- The Client-Facing Technologist: This persona acts as the crucial technical bridge between the data squad and the client. They build trust by listening actively, explaining complex data concepts (like data latency or idempotency) in simple terms, and demonstrating how engineering decisions align with the client’s strategic goals.
- The Quality Craftsman: This individual possesses an unwavering commitment to data integrity and treats data engineering as a craft. They are the guardian of the reporting layer, advocating for robust testing, data validation frameworks, and clean, modular code to ensure the long-term reliability of the data platform.
Experience & Relevance
- Engineering Depth: 7-10 years of professional experience in end-to-end data product development. You have a portfolio that proves your ability to build complex, high-velocity pipelines for both Batch and Streaming workloads.
- Cloud-Native Fluency: Deep, hands-on experience designing and deploying scalable data solutions on at least one major cloud platform (AWS, GCP, or Azure). You are comfortable navigating the nuances of EMR, BigQuery, or Synapse at scale.
- AI-Native Workflow: You don’t just build for AI; you build with AI. You must be proficient in using AI coding assistants (e.g., GitHub Copilot) to accelerate your delivery and have a track record of building the data foundations required for Generative AI.
- Architectural Portfolio: Evidence of leading 2-3 large-scale transformations—including platform migrations, data lakehouse builds, or real-time analytics architectures.
- Client-Facing Acumen: You have direct experience in a consultative, client-facing role. You can confidently translate a CEO’s business vision into a Lead Engineer’s technical specification without losing anything in translation.
Join the ‘real solvers’
ready to futurify?
If you are excited by the possibilities of what an AI-native engineering-led, modern tech consultancy can do to futurify businesses, apply here and experience the ‘Art of the possible’. Don’t Just Send a Resume. Send a Statement.
Shubham Vishwakarma
Full Stack Developer - Averlon
About Searce Inc
About
What is ‘searce’
Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’. It signifies our way of working: To improve to the finest degree of excellence, ‘solving for better’ every time. Searcians are passionate improvers & solvers who love to question the status quo.
The primary purpose of all of us, at Searce, is driving intelligent, impactful & futuristic business outcomes using new-age technology. This purpose is driven passionately by HAPPIER people who aim to become better, everyday.
What we do
Searce is a modern tech consulting firm that empowers clients to futurify their businesses, leveraging Cloud, AI & Analytics.
- We are a category defining niche’ cloud-native technology consulting company, specializing in modernizing (improve, automate & transform) the full-scope of infra, app, process & work
- We partner with clients in their ‘beyond x’ journey to drive intelligent, impactful & futuristic business outcomes
- We are the most preferred tech partner of choice when it comes to ‘solving for better’ for the new-age tech startups & digital enterprises, leading disruption in their industries
- Our Service Offerings: We offer Advanced Cloud, Data & App Modernization, Cloud Consulting, Management & Improvement (DevOps, SysOps & Cloud Managed Services), Applied AI & Analytics services
- As one of the top 5 niche’ full scope global partners for Google Cloud & a preferred partner for AWS, we are the most preferred ‘engineering-led’ tech company of choice when it comes to solving complex business problems.
Who we are
We are passionate improvers, solvers & futurists. Driven by our engineering excellence mindset, we care most about delivering intelligent, impactful & futuristic business outcomes. Searcians are motivated by continuous improvement & solving for better in everything we do.
At the core, a Searcian is self-driven to become better, everyday. In passionate pursuit of the finest degree of excellence we drive exceptional outcomes in everything we do.
We believe that trust is the most important value. We also believe that we need to ‘earn the trust’, everytime one engages with us. And earning trust for us is far more important than anything else. We aim to be the *most trusted* tech consulting partner for our clients.
We are HAPPIER at heart. Humble, Adaptable, Positive, Passionate, Innovative, Excellence focused, & Responsible. We live the HAPPIER Culture Code.
Being HAPPIER.
How we work
- Customers. Partners. Our aim is to build relationships with customers for life. And meaningfully improve the life of every customer.
- We do what we say. We say what we do. We are uncomfortably honest and transparent. Being genuine wins trust & makes people happier.
- Mistakes are encouraged. We make mistakes. Tons of those. Everyday. And we don’t mind apologizing to our juniors, peers or superiors. We are no ego-doers.
- Underpromise. Overdeliver. We work with a deep desire to go above and beyond in everything we do. Everytime.
So, If you are passionate about tech, future & what you read above (we really are!), apply here to experience the ‘Art of Possible’
Company social profiles
Similar jobs (10)
GCP Data engineer
Exp 9+yrs
Bangalore/Hyderabad
Lead exp+ Python + pyspark + sql or plsql
immediate to 15days joiners
Call or WhatsApp: 6 3 8 5 0 7 6 1 4 2
Job Title : Senior Data Engineer – Databricks
Experience : 14 to 20 Years
Location : HSR Layout, Bangalore
Work Mode : Hybrid – 3 Days WFO
Shift : 11:30 AM – 07:30 PM IST
Positions : 2
Notice Period : Immediate Joiners Only
Interview : 1 Technical Round + 2 Client Rounds
Role Overview :
We are looking for a Senior Data Engineer to build and lead enterprise-scale data platforms for a Switzerland-based commodity client.
The role requires a strong hands-on Data Engineering professional with expertise in Databricks, PySpark, Python, SQL, and AWS, along with technical leadership and stakeholder management experience.
Must-Have Skills :
- 14 to 20 years of Data Engineering experience
- Databricks & Apache Spark / PySpark
- Python & SQL
- AWS Cloud
- Lakehouse Architecture
- ETL / ELT & Distributed Data Processing
- Batch & Streaming Pipelines
- Data Pipeline Optimization & Data Modeling
- CDC & Incremental Processing
- Git, CI/CD & Testing
- Data Quality, Monitoring & Observability
- Technical Leadership & Stakeholder Management
Key Responsibilities :
- Design and build scalable data pipelines using Databricks, PySpark, Python, SQL, and AWS.
- Own data products from design through production.
- Develop batch / streaming pipelines and reusable ETL / ELT frameworks.
- Optimize pipelines for performance, scalability, reliability, and cost.
- Design scalable data architectures and data models.
- Implement data quality, monitoring, lineage, and CI/CD practices.
- Lead technical discussions and mentor engineering teams.
- Collaborate with business stakeholders, architects, product owners, and engineering teams.
- Remain hands-on while providing technical leadership.
Ideal Candidate :
A 14 to 20 years experienced, hands-on Data Engineering leader with strong Databricks + PySpark + AWS expertise, excellent communication, stakeholder management, and experience delivering enterprise-scale data platforms.
🔴 Super Urgent : Only Bangalore-based immediate joiners.
🚀 Hiring: Senior GCP Data Engineer
We are looking for experienced Senior Data Engineers with strong hands-on expertise in Google Cloud Platform (GCP), Python, PySpark, and SQL. Candidates with team lead experience will be preferred.
📌 Job Details:
- Role: Senior GCP Data Engineer
- Experience: 8+ Years
- Budget: Up to ₹30 LPA
- Location: Bangalore / Hyderabad
- Relocation: Candidates from Chennai and Pune willing to relocate are welcome.
- Notice Period: Immediate to 15 days
- Interview Process: L1 – Virtual | L2 / DM – Face-to-Face (F2F)
💻 Required Technical Skills:
- Strong hands-on experience with GCP data engineering.
- Excellent proficiency in Python, PySpark, and SQL.
- Experience in designing, developing, and optimizing data pipelines.
- Good understanding of ETL/ELT processes and large-scale data processing.
- Experience working with cloud-based data platforms and data warehousing.
- Strong analytical, problem-solving, and communication skills.
⭐ Preferred Skills:
- Previous experience as a Technical Lead or leading a data engineering team.
- Knowledge of GCP services such as BigQuery, Dataproc, Dataflow, and Cloud Storage.
1. Company Description:
Tatvic, is a marketing analytics company focusing on generating insights from data using long association with Google & its infrastructure. We breed,recognize and reward performance. As a company we are growing very fast & we are under transformation. To enable this transformation we need future leaders with eyesight which balances execution & strategic understanding.
Website: www.tatvic.com
2. Role Details
- Role: Lead Data Engineer
- Band: L4
- Department: Data Engineering
- Reporting to: L5 / Head of Delivery
- Location: Ahmedabad
- Job Type: Full Time
- Experience: 6-8 years of relevant experience
- Education: Bachelor’s or Master’s degree in Computer Science, Data Science, or a related technical field.
3. Mission & Role Overview
- Mission: To create high-level organizational impact by leading a team to design and deliver scalable, high-performance cloud-native data solutions. The Lead Data Engineer (L4) is responsible for ensuring technical excellence—specifically in BigQuery optimization and secure GCP/AWS architectures—while driving the transition to Agentic AI workflows and creating value through the development of L1-L3 engineers.
- Role Overview: This is a pivotal leadership role on the Management Track, responsible for end-to-end data outcomes for the function. The Lead Engineer ensures the quality of engineering execution, manages team utilization and resource allocation, and serves as the primary technical representative in cross-functional forums with Marketing, Strategy, and Product teams.
4. Key Responsibilities
- Lead the design, development, and deployment of scalable and efficient cloud-based data pipelines and architectures.
- Build reusable data solutions and software components that serve multiple customers and use cases.
- Architect cloud environments using GCP, ensuring best practices in IAM, Security, and Networking.
- Conduct Exploratory Data Analysis (EDA) on clickstream and behavioral datasets to derive insights.
- Manage end-to-end ETL processes, including data extraction, transformation, and loading using Python, SQL, and Airflow.
- Design and build production-grade software (APIs, services, internal tools) with sound engineering practices — version control, testing, CI/CD, and code review.
- Extend data platforms with AI capabilities — integrating LLMs, RAG pipelines, and agentic workflows where they add clear business value.
- Perform Root Cause Analysis (RCA) and resolve data quality, pipeline, or application issues.
- Collaborate cross-functionally with Marketing, Insights, Product, and Engineering teams to align data initiatives with business goals.
- Document architectures, design decisions, and operations clearly and consistently.
- Coach and mentor team members, ensuring technical excellence and continuous learning.
- Participate in recruitment and knowledge-sharing through interviews, blogs, and webinars.
5. Must-Have & Good-to-Have Skills
Must-Have Skills:
- Strong understanding of clickstream data and data pipeline architecture
- Hands-on experience with cloud platforms (GCP)
- Expertise in SQL, Python, and Airflow
- Solid software engineering foundation — API design, testing, version control, and deployment practices
- Proven ability to perform exploratory data analysis and data validation
- Knowledge of cloud security, IAM, and networking best practices
- Working knowledge of building or integrating AI/LLM-based components into existing systems
- Excellent problem-solving, stakeholder management, and communication skills
Good-to-Have Skills:
- Certifications such as: Google Cloud Professional Data Engineer / Cloud Architect
- AWS Certified Data Analytics or Azure Data Engineer Associate
- Google Analytics 4 (GAIQ)
- Experience in marketing attribution, commerce data, or analytics ecosystems
- Exposure to cost optimization and cloud resource management
- Experience with agent frameworks (LangChain, ADK, MCP) or LLM APIs
- Experience building automation frameworks or internal data tools
6. Not-a-Fit Criteria
- Individual Contributor Focus: Operates as the team's best individual contributor instead of building team capability or empowering L3s.
- Information Hoarding: Keeps architectural patterns or tribal knowledge trapped in their head rather than creating reusable playbooks and documentation.
- Avoidance of Conflict: Avoids difficult performance conversations with underperforming team members, allowing capability gaps to persist.
- Activity Over Outcomes: Measures success by "tasks completed" rather than pipeline stability, data accuracy, or business impact.
- Functional Siloing: Fails to build partnerships with Marketing or Product teams, leading to misaligned data delivery.
Our Culture & Perks - Where Passion Meets Purpose:
- Competitive CTC: Based on experience and skillset
- Medical Insurance for you and your family
- Return-to-Work Program for New Moms — because career breaks shouldn’t break momentum
- 1 Work From Home Day/Month — flexibility when life calls
- 2 Birthday Leaves/Year — one for you, one for someone you love
- Monthly Engagement Activities — from game nights to learning sessions
- Inclusive Culture & Equal Opportunity Employer — diversity drives our innovation
- We Think Like Engineers & Athletes: we build, we iterate, we scale, we win — together
- 5-day workweek — because balance drives brilliance
Job Description – Senior GCP Data Engineer
Position: Senior GCP Data Engineer
Experience: 8+ Years
Budget: Up to 30 LPA
Location: Bangalore / Hyderabad
Relocation: Candidates from Chennai / Pune willing to relocate are also welcome.
Notice Period: Immediate to 15 Days
Interview Process:
- Round 1: Virtual Interview
- Round 2 / DM Round: Face-to-Face Interview
Required Skills
- Strong hands-on experience in GCP Data Engineering.
- Good programming experience in Python.
- Strong experience with PySpark / Apache Spark.
- Proficiency in SQL and database concepts.
- Experience in developing and maintaining data pipelines.
- Good understanding of data processing, transformation, and integration.
- Experience in handling large-scale data processing.
- Strong problem-solving and debugging skills.
- Good understanding of cloud-based data engineering concepts.
- Experience in optimizing data pipelines for performance and scalability.
Preferred Skills
- Previous experience in a Lead Data Engineer role.
- Experience in technical leadership, mentoring, and guiding team members.
- Ability to collaborate with cross-functional teams and deliver data engineering solutions.
Roles & Responsibilities
- Design, develop, and maintain scalable data pipelines using GCP.
- Develop data processing and transformation workflows using Python and PySpark.
- Write and optimize SQL queries for efficient data processing.
- Build and maintain reliable data engineering solutions.
- Troubleshoot data pipeline issues and optimize performance.
- Ensure data quality, reliability, and scalability.
- Collaborate with stakeholders and team members to deliver project requirements.
- Lead and guide team members where applicable.
- Important: Candidates must have strong Senior Data Engineer experience. Lead experience is preferred but not mandatory.
1st virtual , 2nd round F2F
Python pyspark, SQL, data engineer
5+yrs
9+yrs
Bangalore/Hyderabad
immediate to 15days.
Job Summary
Role Overview
We are looking for an experienced Data Engineer with strong expertise in Python, ETL, Advanced SQL, CI/CD, DevOps, and Data Analytics. The ideal candidate should have hands-on experience designing and developing scalable data pipelines, transforming large datasets, and supporting data-driven applications.
Experience with Google Cloud Platform (GCP) will be an added advantage.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines using Python and SQL.
- Develop complex and optimized SQL queries, stored procedures, and data transformations.
- Build and maintain reliable data integration workflows across multiple data sources.
- Perform data cleansing, validation, transformation, and quality checks.
- Analyze data and provide insights to support business and technical requirements.
- Implement and maintain CI/CD pipelines for data engineering applications.
- Work with DevOps practices and tools to automate deployments, monitoring, and infrastructure processes.
- Troubleshoot data pipeline failures, performance issues, and production incidents.
- Optimize data processing workflows for performance, scalability, and reliability.
- Collaborate with Data Analysts, Data Scientists, Developers, and other stakeholders.
- Follow best practices for version control, testing, documentation, and deployment.
- Contribute to cloud-based data engineering initiatives, preferably on GCP.
Required Skills
- 5–7 years of hands-on experience in Data Engineering.
- Strong programming skills in Python.
- Strong expertise in Advanced SQL and database concepts.
- Hands-on experience with ETL/ELT processes and data pipelines.
- Good understanding of Data Warehousing and Data Modeling concepts.
- Experience with CI/CD practices and tools.
- Strong understanding of DevOps principles, automation, and deployment processes.
- Strong data analytics and problem-solving skills.
- Experience working with large datasets and performance optimization.
- Good understanding of Git/version control and software development best practices.
Good to Have
- Hands-on experience with Google Cloud Platform (GCP).
- Exposure to GCP data services such as BigQuery, Cloud Storage, Dataflow, Composer, or Pub/Sub.
- Experience with containerization/orchestration technologies such as Docker/Kubernetes.
- Experience with workflow orchestration tools such as Airflow.
- Knowledge of cloud-based data architecture and distributed data processing.
Preferred Candidate Profile
- Strong analytical and problem-solving abilities.
- Good communication and stakeholder management skills.
- Ability to work independently as well as in a collaborative team environment.
- Strong ownership of data pipelines and production systems.
- Candidates who can join at short notice are preferred.
Mandatory Skills
Data Engineer, Python , ETL, GCP, Advanced SQL, Strong Data Analytics skills, CICD, Devops
role: data engineer
Python pyspark, SQL, data engineer
5+yrs
Bangalore/Hyderabad
immediate to 15days.
About Solutionara
Solutionara is an AI-led IT Consulting and Technology Services organization delivering enterprise transformation through Cloud, Integration, Data Engineering, Digital Experience, AI, and Enterprise Applications.
Headquartered in the United States with a Global Delivery Center (GDC) in Noida, India, Solutionara partners with enterprise organizations across retail, manufacturing, distribution, consumer goods, and other industries to design, implement, and support scalable technology solutions.
As a rapidly growing consulting organization, we foster a culture of ownership, innovation, collaboration, and continuous improvement, empowering our teams to contribute beyond their immediate responsibilities while delivering exceptional value to our global clients.
Every team member is expected to think like a consultant and business problem solver—understanding client objectives, challenging assumptions when appropriate, leveraging AI responsibly, and continuously identifying opportunities to improve quality, productivity, and customer outcomes.
AI is embedded into how we work, but it is an enabler—not a replacement for expertise, critical thinking, or accountability.
Role Overview
Solutionara is seeking a Senior Data Engineer with strong data engineering and database expertise to design, build, optimize, and support enterprise-scale data solutions.
This is a hands-on engineering position requiring strong SQL, Python, ETL/ELT, database, cloud, and troubleshooting capabilities. The engineer will work closely with technical leads and client stakeholders while taking ownership of major engineering components.
Location: Noida, India – Onsite | Employment Type: Full-Time | Experience: 6–9 Years
Key Responsibilities
Data Engineering
- Design, develop, test, deploy, and support scalable ETL/ELT pipelines.
- Build ingestion and transformation solutions across databases, APIs, files, and enterprise applications.
- Develop high-quality SQL and Python-based data processing solutions.
- Implement batch and incremental data-processing patterns.
- Develop reusable engineering components and frameworks.
- Support complex data migration and transformation activities.
- Perform source-to-target mapping and technical data analysis.
- Implement pipeline logging, monitoring, exception handling, and recovery mechanisms.
Database & Platform Engineering
- Design and optimize relational database schemas and data structures.
- Analyze and improve SQL/query performance.
- Support indexing, partitioning, storage optimization, and database performance.
- Implement appropriate database security and access controls.
- Support database monitoring, availability, backup/recovery, and operational requirements in collaboration with infrastructure/platform teams.
- Troubleshoot database and pipeline performance issues.
- Contribute to cloud data-platform configuration and optimization.
Data Quality
- Implement automated validation and data-quality controls.
- Develop reconciliation processes between source and target systems.
- Identify duplicate, incomplete, inconsistent, and invalid data.
- Implement cleansing, standardization, validation, and exception-management logic.
- Work with technical and business stakeholders to understand and implement business data rules.
Technical Delivery
- Convert technical designs into reliable production solutions.
- Participate in technical design and code reviews.
- Mentor junior/mid-level engineers.
- Support estimation and technical planning.
- Troubleshoot complex production and integration issues.
- Collaborate with technical leads, application teams, DevOps, QA, BI, and client teams.
- Document technical designs, mappings, configurations, operational procedures, and engineering decisions.
Consulting & AI-Led Working
- Understand the business purpose behind assigned engineering requirements.
- Raise risks, assumptions, dependencies, and potential improvements proactively.
- Communicate effectively with client and internal stakeholders.
- Use AI-assisted tools to improve coding, testing, analysis, documentation, and troubleshooting productivity.
- Independently verify AI-generated code and recommendations before implementation.
- Identify opportunities for engineering automation and process improvement.
Required Qualifications
- 6–9 years of experience in data engineering, database engineering, data integration, or related areas.
- Advanced SQL skills.
- Strong hands-on Python experience.
- Strong ETL/ELT development experience.
- Strong relational database knowledge, such as SQL Server, PostgreSQL, Oracle, or equivalent.
- Experience with data modeling and database design.
- Experience with AWS, Azure, or another major cloud platform.
- Experience with modern data technologies such as Databricks, Snowflake, Redshift, Synapse, Glue, ADF, Airflow, dbt, or equivalent.
- Experience with performance tuning and optimization.
- Experience with data validation and reconciliation.
- Strong troubleshooting and analytical skills.
- Ability to take ownership of engineering deliverables with limited supervision.
Preferred Qualifications
- Experience working with enterprise ERP, CRM, PIM, PLM, MES, or related systems.
- Experience with manufacturing, retail, distribution, or consumer-goods data.
- Experience with CI/CD and infrastructure/platform engineering practices.
- Experience with API and event-driven integrations.
- Previous consulting or technology-services experience.
What Success Looks Like
- Pipelines and databases perform reliably at enterprise scale.
- Engineering solutions are maintainable, reusable, and well documented.
- Data issues are detected and addressed systematically.
- Junior engineers receive effective technical guidance.
- Problems are owned through resolution rather than simply escalated.
- Engineering decisions reflect both technical requirements and client business objectives.
About the Role
We are looking for a Senior Data Engineer with strong hands-on expertise in Databricks, Python, PySpark, and SQL to build scalable, high-performance data engineering solutions. You’ll architect and develop large scale, high-performance data pipelines capable of handling massive real-time and batch data volumes across multiple business systems. Databricks is the core enterprise data and processing platform for this role. You will also use Apache Airflow for workflow orchestration and dbt for ELT transformations, and will contribute to designing reliable, secure, and governed data platforms that enable analytics, reporting, and AI-driven use cases.
Key Responsibilities
- Design and implement large-scale data pipelines using Python/PySpark, Databricks, and Microsoft Fabric.
- Develop and optimize data processing workloads in Databricks using PySpark and Spark SQL, with a strong focus on scalability, reliability, performance, and maintainability.
- Develop and maintain dbt models including layered architecture, incremental models, snapshots, macros, testing, and documentation.
- Design, develop, and maintain Apache Airflow DAGs for orchestrating reliable, scalable, and observable data pipelines.
- Design and implement data quality, observability, and governance frameworks, including automated testing, monitoring, lineage, access control, and data privacy standards.
- Partner with analytics, product, and business stakeholders to turn requirements into trustworthy datasets, and raise the engineering bar through design discussions, code reviews, and mentoring junior engineers.
Required Skills
- Strong expertise in Python for developing scalable, modular, and production-ready data engineering applications.
- Strong expertise in PySpark, including DataFrame API, Spark SQL, Structured Streaming, partitioning strategies, joins, caching, handling data skew, and Spark performance optimization.
- Strong hands-on experience with Databricks for data ingestion, transformation, processing, and optimization, including Delta Lake, Unity Catalog, Databricks Workflows, notebooks, jobs, and Databricks-native data engineering capabilities.
- Strong experience in Databricks/Spark performance tuning, including query and job optimization, partitioning, file sizing, caching, join optimization, handling data skew, and efficient use of compute resources.
- Hands-on experience with Delta Lake, including transactional data processing, schema management, incremental data processing, and reliable batch and streaming data pipelines.
- Hands-on experience in developing dbt projects using layered architecture, incremental models, snapshots, macros/Jinja, testing, documentation, and deployment best practices.
- Expertise in advanced SQL and data modelling — dimensional modeling, slowly changing dimensions, schema evolution, and query optimization.
- Hands-on experience in developing and managing Apache Airflow DAGs, scheduling workflows, dependency management, retries, backfills, and operational monitoring.
- Hands-on experience with at least one major cloud platform (AWS, Azure or GCP).
- Strong problem-solving skills and the ability to work independently with business and analytics stakeholders.
Nice to Have
- Hands-on exposure to Microsoft Fabric for data integration and analytics.
- Experience using AI coding assistants (e.g. Claude Code, GitHub Copilot) as part of a development workflow.
- Familiarity with modern DevOps practices, including CI/CD pipelines, Infrastructure as Code (IaC), and containerization (Docker/Kubernetes).
- Domain expertise in financial services.
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon