Data Architect (Dremio Lakehouse) at AI company · Bengaluru (Bangalore), Mumbai, Hyderabad, Gurugram · 5 - 17 years · ₹30L - ₹45L / yr · Posted 24 Dec 2025

Review Criteria
- Strong Dremio / Lakehouse Data Architect profile
- 5+ years of experience in Data Architecture / Data Engineering, with minimum 3+ years hands-on in Dremio
- Strong expertise in SQL optimization, data modeling, query performance tuning, and designing analytical schemas for large-scale systems
- Deep experience with cloud object storage (S3 / ADLS / GCS) and file formats such as Parquet, Delta, Iceberg along with distributed query planning concepts
- Hands-on experience integrating data via APIs, JDBC, Delta/Parquet, object storage, and coordinating with data engineering pipelines (Airflow, DBT, Kafka, Spark, etc.)
- Proven experience designing and implementing lakehouse architecture including ingestion, curation, semantic modeling, reflections/caching optimization, and enabling governed analytics
- Strong understanding of data governance, lineage, RBAC-based access control, and enterprise security best practices
- Excellent communication skills with ability to work closely with BI, data science, and engineering teams; strong documentation discipline
- Candidates must come from enterprise data modernization, cloud-native, or analytics-driven companies
Preferred
- Preferred (Nice-to-have) – Experience integrating Dremio with BI tools (Tableau, Power BI, Looker) or data catalogs (Collibra, Alation, Purview); familiarity with Snowflake, Databricks, or BigQuery environments
Job Specific Criteria
- CV Attachment is mandatory
- How many years of experience you have with Dremio?
- Which is your preferred job location (Mumbai / Bengaluru / Hyderabad / Gurgaon)?
- Are you okay with 3 Days WFO?
- Virtual Interview requires video to be on, are you okay with it?
Role & Responsibilities
You will be responsible for architecting, implementing, and optimizing Dremio-based data lakehouse environments integrated with cloud storage, BI, and data engineering ecosystems. The role requires a strong balance of architecture design, data modeling, query optimization, and governance enablement in large-scale analytical environments.
- Design and implement Dremio lakehouse architecture on cloud (AWS/Azure/Snowflake/Databricks ecosystem).
- Define data ingestion, curation, and semantic modeling strategies to support analytics and AI workloads.
- Optimize Dremio reflections, caching, and query performance for diverse data consumption patterns.
- Collaborate with data engineering teams to integrate data sources via APIs, JDBC, Delta/Parquet, and object storage layers (S3/ADLS).
- Establish best practices for data security, lineage, and access control aligned with enterprise governance policies.
- Support self-service analytics by enabling governed data products and semantic layers.
- Develop reusable design patterns, documentation, and standards for Dremio deployment, monitoring, and scaling.
- Work closely with BI and data science teams to ensure fast, reliable, and well-modeled access to enterprise data.
Ideal Candidate
- Bachelor’s or master’s in computer science, Information Systems, or related field.
- 5+ years in data architecture and engineering, with 3+ years in Dremio or modern lakehouse platforms.
- Strong expertise in SQL optimization, data modeling, and performance tuning within Dremio or similar query engines (Presto, Trino, Athena).
- Hands-on experience with cloud storage (S3, ADLS, GCS), Parquet/Delta/Iceberg formats, and distributed query planning.
- Knowledge of data integration tools and pipelines (Airflow, DBT, Kafka, Spark, etc.).
- Familiarity with enterprise data governance, metadata management, and role-based access control (RBAC).
- Excellent problem-solving, documentation, and stakeholder communication skills.

Similar jobs (10)
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Position: Technical Architect – Data Engineering
Job Summary:
- We are looking for an experienced Technical Architect to lead the design and implementation of modern cloud-based data platforms.
- The ideal candidate should have strong expertise in Azure and/or AWS, Databricks, Snowflake, modern data architecture, and large-scale data engineering.
- The candidate will work closely with business stakeholders, architects, and engineering teams to deliver scalable, secure, and high-performance data solutions.
Key Responsibilities
• Design enterprise-scale data lakehouse and data warehouse architectures.
• Define data ingestion, transformation, and serving architecture.
• Lead architecture discussions and technical governance.
• Design scalable ETL/ELT frameworks using Databricks and Snowflake.
• Define best practices for security, performance, CI/CD, and DevOps.
• Guide engineering teams on implementation.
• Collaborate with business, product, and data governance teams.
Must Have Skills :
• Azure or AWS
• Databricks
• Snowflake
• Data Lakehouse Architecture
• PySpark
• SQL
• Data Modelling
• ETL/ELT Architecture
• Performance Optimization
• CI/CD
• Terraform or Infrastructure as Code (preferred)
Preferred
• Insurance domain
• dbt
• Unity Catalog
• Delta Lake
• Iceberg
• Azure Data Factory / AWS Glue
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Data Architect – Databricks & AWS
- Strong experience in Data Architecture, Data Engineering, Databricks, Apache Spark/PySpark, Python, and Advanced SQL.
- Design and implement scalable ETL/ELT pipelines, data platforms, and Lakehouse architectures using Medallion Architecture.
- Experience with Databricks, Databricks Workflows, Unity Catalog OR Databricks Jobs
- Strong knowledge of Delta Lake, Databricks Workflows, Delta Live Tables (DLT), and dimensional data modeling.
- Hands-on experience with AWS services such as S3, Glue, IAM, Lambda, and CloudWatch.
- Experience with Apache Airflow, Data Warehouse concepts, Git/CI-CD, performance optimization, and data quality.
- Good to have exposure to Kafka/Structured Streaming, Unity Catalog, and modern data governance.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Title: Senior Data Architect
Experience: 7+ Years
Location: India (Ahmedabad / Hybrid/Remote)
Job Summary
We are looking for an experienced Senior Data Architect to support the modernization and enhancement of Enterprise Analytic Data Warehouse (ADW). The candidate will be responsible for evaluating existing data architecture, designing scalable data models, assessing data integration across clinical, operational and financial domains, and establishing data governance and data quality standards.
Key Responsibilities
- Assess and review the existing Enterprise Analytic Data Warehouse architecture, database structures and data integration processes.
- Design and improve enterprise data models covering clinical, operational, financial and revenue-cycle information.
- Define data architecture standards, database structures, data relationships and integration strategies.
- Evaluate data flows between source systems and the data warehouse.
- Design and review source-to-target mappings and data lineage.
- Identify architectural gaps affecting data completeness, consistency, integrity and reporting accuracy.
- Establish standardized data definitions, data governance practices and metadata management.
- Ensure consistent patient, facility, payer and other critical identifiers across different datasets.
- Support integration of billing, claims, payment, denial and accounts receivable (AR) data.
- Work with SQL and Python-based data processing solutions to investigate and validate data structures.
- Recommend improvements to warehouse scalability, performance, reliability and maintainability.
- Develop a prioritized architecture and modernization roadmap for future implementation.
- Coordinate with data engineers, analytics engineers and business stakeholders.
Required Technical Skills
- Strong experience in Data Architecture and Enterprise Data Warehousing.
- Advanced SQL and relational database concepts.
- Experience with data modelling (conceptual, logical and physical).
- Knowledge of ETL/ELT pipelines and data integration.
- Experience with data profiling, data lineage and metadata management.
- Understanding of data governance, data quality and data validation frameworks.
- Experience working with large and complex datasets.
- Python knowledge for data analysis and validation.
- Experience with cloud-based data platforms or modern data warehouse technologies.
Preferred Skills
- Experience with Databricks, Azure, AWS or similar cloud data platforms.
- Experience with healthcare data warehouses.
- Knowledge of clinical, financial, billing and claims datasets.
- Understanding of HIPAA, PHI and healthcare data security principles.
- Experience with healthcare analytics, reimbursement and revenue-cycle data.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Description: Associate Director-Data Engineering (Databricks Lakehouse Architect + Cloud Data Engineering Leader)
About our Client -
Client is a leading SAP consulting and digital transformation partner committed to delivering innovative, high-impact solutions across enterprise analytics, cloud, and intelligent business platforms.
As enterprises modernize their data landscapes, the role of traditional data warehousing is rapidly evolving into scalable cloud-native Data Lakehouse architectures. Applexus is expanding its modern data engineering capabilities to help customers integrate enterprise data from SAP and non-SAP ecosystems, build advanced analytics platforms, and enable future-ready AI-driven solutions.
We are looking for an accomplished Databricks Lakehouse Architect / Cloud Data Engineering Leader who can play a key role in this transformation journey — designing modern data platforms, leading cloud data engineering initiatives, and bridging enterprise analytics with next-generation technologies.
The ideal candidate will combine strong hands-on expertise in Databricks, Spark/PySpark, Delta Lake, and Azure/AWS cloud platforms with the ability to engage customers, architect solutions, guide engineering teams, and drive successful delivery.
Role Overview:
We are looking for a hands-on Data Engineering leader who can architect modern cloud data platforms using Databricks Lakehouse and lead solution design, delivery, and engineering teams. This role requires strong technical ownership rather than a pure people management background.
Key Responsibilities:
• Architect scalable Data Lakehouse platforms
• Own solution design, estimation, architecture reviews and client discussions
• Build and optimize enterprise data pipelines
• Provide technical guidance to data engineering teams
• Establish best practices around performance, governance and architecture
Must Have - Databricks Architecture:
• Databricks Lakehouse design
• Delta Lake implementation
• Apache Spark / PySpark engineering
• Medallion architecture (Bronze/Silver/Gold layers)
• Data pipeline optimization
• Unity Catalog and data governance exposure
Cloud Platform Expertise (depth in one required):
Microsoft Azure:
• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS)
• Azure Synapse Analytics
OR
Amazon AWS:
• Databricks on AWS
• Amazon S3
• AWS Glue
• Redshift
• Athena
Good to Have:
• Snowflake
• ML Engineering exposure
• Cloud/Data Engineering certifications
Ideal Candidate:
A hands-on architect who can talk to customers, design solutions, estimate projects, guide engineers and review architecture while helping build a modern Data Engineering practice.
We are looking for an accomplished Databricks Lakehouse Architect / Cloud Data Engineering Leader who can play a key role in this transformation journey — designing modern data platforms, leading cloud data engineering initiatives, and bridging enterprise analytics with next-generation technologies.
The ideal candidate will combine strong hands-on expertise in Databricks, Spark/PySpark, Delta Lake, and Azure/AWS cloud platforms with the ability to engage customers, architect solutions, guide engineering teams, and drive successful delivery.
If you are a hands-on Data Architect who enjoys building modern data platforms and wants to move beyond execution into solution ownership and leadership, I would love to connect.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Summary
We are looking for an experienced Data Engineer to build and optimize scalable data pipelines and AWS-based data platforms.
Key Responsibilities
- Develop and maintain data pipelines using Python and PySpark.
- Design and implement data lakehouse solutions on AWS.
- Build and manage data workflows using AWS Glue, EMR, ECS, and Step Functions.
- Work with DynamoDB and other data storage technologies.
- Monitor data pipelines and ensure data quality, performance, and reliability.
- Collaborate with stakeholders and engineering teams to deliver data solutions.
Mandatory Skills
- 5+ years of overall professional experience.
- 3+ years of experience in data platforms and big data technologies.
- Strong proficiency in Python and PySpark.
- Hands-on experience with AWS EMR, ECS, Glue, Step Functions, and DynamoDB.
- Experience designing and implementing AWS data lakehouse architectures.
Good to Have
- Apache Spark and Apache Iceberg.
- AWS Lambda, EventBridge, Kinesis, and Redshift.
- Experience with real-time data pipelines and event-driven architectures.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
About Us:
The QX Impact was launched with a mission to make A.I accessible and affordable and deliver AI Products/Solutions at scale for the enterprises by bringing the power of Data, AI, and Engineering to drive digital transformation. We believe without insights; businesses will continue to face challenges to better understand their customers and even lose them. Secondly, without insights businesses won't’ be able to deliver differentiated products/services; and finally, without insights, businesses can’t achieve a new level of “Operational Excellence” is crucial to remain competitive, meeting rising customer expectations, expanding markets, and digitalization.
Job Summary:
We are looking for a Senior Data Engineer who is creative, collaborative, and adaptable to join our agile team of data scientists, engineers, and UX developers. The role focuses on building and maintaining robust data pipelines to support advanced analytics, data science, and BI solutions.
As a Senior Data Engineer, you will work with internal and external data, collaborate with data scientists, and contribute to the design, development, and deployment of innovative solutions.
Key Responsibilities:
- Design, develop, test, and maintain optimal data pipeline and ETL architectures.
- Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
- Prepare and optimize data for predictive and prescriptive modeling.
- Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
- Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
- Utilize big data tools and frameworks to optimize data acquisition and preparation.
- Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
- Develop and curate data models for analytics, dashboards, and reports.
- Conduct code reviews, maintain production-level code, and implement testing approaches.
- Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
- Drive innovation and implement efficient new approaches to data engineering tasks.
Must-Have Skills:
- Bachelor’s degree in Computer Science, Mathematics, Engineering, or a related field.
- 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
- 3–5 years of experience designing and implementing data warehouse solutions.
- Expertise in SQL, including developing stored procedures (SP) and applying advanced data design concepts.
- Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
- Experience with AWS or Azure services (e.g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
- Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
- Hands-on experience with orchestration tools (e.g., Apache Airflow, Prefect).
- Knowledge of CI/CD processes, version control (e.g., Git, Jenkins), and deployment automation.
- Strong problem-solving, communication, and collaboration skills.
Good-to-Have Skills:
- Experience in integrating ERP data into data lakes.
- Experience with traditional ETL tools (e.g., Talend, Pentaho).
Competencies:
- Tech Savvy - Anticipating and adopting innovations in business-building digital and technology applications.
- Self-Development - Actively seeking new ways to grow and be challenged using both formal and informal development channels.
- Action Oriented - Taking on new opportunities and tough challenges with a sense of urgency, high energy, and enthusiasm.
- Customer Focus - Building strong customer relationships and delivering customer-centric solutions.
- Optimize Work Processes - Knowing the most effective and efficient processes to get things done, with a focus on continuous improvement.
Why Join Us?
- Be part of a collaborative and agile team driving cutting-edge AI and data engineering solutions.
- Work on impactful projects that make a difference across industries.
- Opportunities for professional growth and continuous learning.
- Competitive salary and benefits package.
Application Details
Ready to make an impact? Apply today and become part of the QX Impact team!
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Key Skills:
Python & System Design: Strong expertise in Python and system design, with proven experience architecting scalable, production-grade systems.
LLM & Agentic AI: Proven experience building LLM-based and agentic systems from the ground up and evaluating them based on reproducibility, calibration, and cost.
Cloud & Data Architecture: Deep understanding of data and cloud architecture across AWS and Azure, including multi-store data design, distributed systems, streaming, security, and cost optimization.
Azure Databricks: Hands-on production experience with Azure Databricks architecture, including Unity Catalog, Delta Lake, cluster policy design, job orchestration, and Terraform provisioning.
Supply Chain & Inventory Management: Strong domain knowledge of supply chain and inventory operations, including demand planning, MRP logic, inventory parameters, and translating business planning decisions into data requirements.
System Architecture & Technical Decision-Making: Ability to make sound architectural decisions based on first principles, evaluate technical trade-offs, and avoid unnecessary complexity and abstraction.
LLM & Agentic AI: Proven experience building LLM-based and agentic systems from the ground up and evaluating them based on reproducibility, calibration, and cost.
Technical Leadership: Ability to influence teams without direct authority, establish technical standards, prepare technical specifications and documentation, and mentor team members.
Experience in pharmaceutical or regulated industries is highly valued.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
IMMEDIATE JOINERS ONLY
Key Responsibilities:
1 Data Flow Mapping & Lineage
2 Design and produce end-to-end data flow maps for critical data items – from market source/core systems, through Data Lake staging, zone 2 and consumption zone 3, to downstream reporting, dashboards and analytics use cases.
3 Document data lineage and transformations for critical data elements identified in the Master Data Requirements exercise, ensuring traceability that is audit-defensible and supports regulatory, governance and AI-readiness needs.
4 Map data availability by country/Data Lake against the Product Hierarchy (to profit-centre level), building the consolidated Commercial Data Landscape view of what exists, where it sits (source only, staging, or consumption) and what is not captured at all.
5 Document cross-system dependencies and cross-border data flows, flagging data residency, privacy and regulatory considerations to the appropriate governance forums.
6 Data Inventory, Data Model Alignment & Gap Analysis
• Own and mature the data asset inventory, catalogue and data dictionary aligned to the consolidated Master Data Requirements Template – resolving “Not Specified”, “TBD” and “validation required” availability statuses through structured validation with Market IT and Data Engineering.
• Align business data requirements to the conceptual and logical data model of the target Data Lakehouse, ensuring the dependency between data model design and lineage tracking of critical data items is explicitly documented.
• Perform gap and hydration analysis by market and product, quantifying the scale of data that must be brought into the Data Lakes and providing sizing input to the Data Foundation Lead, DAE architects and Data Engineering delivery plans.
• Contribute to the Data Consolidation Plan towards a single source of truth, including recommended sequencing based on business priority (Top 200 fields, Horizon 1–3 priorities).
• Stakeholder Engagement & Workshops
• Work hand-in-hand with the Data Foundation Lead, Data & Analytics Programme Manager, Data Engineers, DAE/infrastructure architects, Local Market IT and Business stakeholders (Regional and Market) as the connective tissue between business data needs and technical implementation.
• Plan and facilitate structured interviews and cross-functional validation workshops with business, technology and data stakeholders to corroborate actual operating data flows against documented processes – establishing a fact-based, evidence-backed view rather than relying on documentation alone.
• Translate ambiguous business asks into clear data definitions, mapping rules and acceptance criteria that Data Engineering can build against, and play back findings to senior stakeholders in clear, visual, decision-oriented formats.
• Manage competing priorities across multiple markets with tact, keeping momentum in an environment of constrained capacity and differing local system landscapes.
• Governance, Ownership & Documentation
• Produce and maintain the Data Ownership & Accountability Matrix – mapping data owners, stewards and custodians per data asset and highlighting ownership gaps for remediation.
• Classify data assets by sensitivity and business function, aligning to Data Protection Policy, PDPA and applicable regulatory expectations (e.g. MAS guidelines) in partnership with Risk & Compliance.
• Ensure all data flow maps, lineage records and inventory artefacts are version-controlled, audit-ready and handed over cleanly into BAU data governance processes.
• Produce source to Datalake mapping document and data catalogue for missing data elements to complete Datalake hydration
Key Skills
• Data flow mapping and lineage documentation – proven ability to draw clear, accurate source-to-consumption data flow maps across complex multi-system, multi-market landscapes.
• Data architecture and modelling – strong grasp of conceptual/logical data models, data lake/lakehouse zone architectures (staging, conformed, consumption), data marts and MDM concepts.
• Data inventory, catalogue, dictionary and metadata management; familiarity with lineage/catalogue tooling (e.g. Purview, Collibra) and diagramming tools (e.g. Visio, ErWin, Lucidchart).
• Hands-on data profiling and validation skills (SQL essential) to verify what actually exists in source systems and lake zones.
• Strong stakeholder management – this is critical: able to build trust and drive outcomes across regional leadership, market business teams, Market IT and engineering, and to facilitate effective cross-functional workshops.
• Insurance domain knowledge – ideally commercial and specialty lines, with an understanding of policy, claims, reinsurance and underwriting data.
• Data quality, governance and privacy concepts, including cross-border data considerations.
• Clear written and visual communication – able to make complex data landscapes understandable to executives.
Experience & Qualification Requirements:
• 8+ years’ experience in data architecture, data business analysis or information architecture roles, with at least:
• 3+ years documenting data flows, lineage or data landscapes across multi-system / multi-market environments.
• Experience in Insurance or Financial Services, ideally within data foundation, data lake or data governance programmes.
• Demonstrated delivery of data inventory / data mapping / current-state data landscape assessments involving both business and IT stakeholders.
• Relevant certification (e.g. DAMA CDMP, TOGAF, cloud data platform certifications) preferred.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
SENIOR DATA ENGINEER – DATABRICKS & SNOWFLAKE
Experience: 7+ Years
Location: Remote
Employment Type: Full-Time
Budget: Up to ₹30 LPA
Time Zone: US Eastern Time (ET)
Shift: US ET Night Shift
Job Description
We are looking for an experienced Senior Data Engineer with strong expertise in Databricks, Snowflake, Apache Spark, and AWS to design, develop, and maintain scalable data engineering solutions.
The candidate will work on integrating data from multiple sources, building modern data pipelines, developing enterprise data models, and preparing data platforms for AI/ML applications.
Experience in healthcare, scientific data, or the histopathology domain will be an added advantage.
Key Responsibilities
- Design, build, and maintain scalable data pipelines to acquire, integrate, and manage data from diverse sources such as laboratory systems, MES, clinical supply, quality systems, and external partners.
- Develop and optimize data workflows for structured and unstructured data using Python, PySpark, R, SQL, Databricks, and Snowflake.
- Design and maintain enterprise-level data models and data repositories.
- Perform ETL development, data warehousing, database design, and data integration.
- Ensure data quality, versioning, traceability, and consistency to support AI/ML readiness.
- Collaborate with stakeholders and technical teams to translate business requirements into technical solutions.
- Manage multiple projects while ensuring scalability, performance, and reliability.
Required Skills
- 7+ years of experience in Data Engineering.
- Strong hands-on experience with Databricks and Snowflake.
- Excellent knowledge of Apache Spark / PySpark.
- Strong experience with AWS cloud services and AWS RDS.
- Experience with Oracle and Microsoft SQL Server (MSSQL).
- Proficiency in Python, R, and SQL.
- Strong understanding of ETL, Data Warehousing (DWH), data modeling, and database design.
- Experience with dbt and cloud-based data architectures.
- Knowledge of NoSQL and graph databases.
- Strong analytical, problem-solving, and stakeholder-management skills.
- Good English communication skills (B2 level or above).
- Willingness to work in the US Eastern Time (ET) night shift.
Preferred / Good to Have
- Experience in histopathology, healthcare, life sciences, or scientific data environments.
- Knowledge of CDISC, HL7, FHIR, OMOP, DICOM, or other regulated data standards.
- Experience handling high-dimensional data, including imaging and sensor data.
- Familiarity with MLOps and machine learning model deployment workflows.
- Knowledge of Manufacturing Execution Systems (MES) and Laboratory Information Management Systems (LIMS).
- Exposure to knowledge graphs and ontology-driven architectures.
Education
Bachelor's degree in Engineering, Data Science, Life Sciences, Computer Science, or a related field. A master's degree is preferred.
Interested candidates can apply with their updated resumes.
Applications on this job are not being reviewed actively. You may apply but please follow up with the company via other channels too.
Job Description:
We are seeking a skilled Senior Data Engineer with expertise in Databricks to join our dynamic data team. The ideal candidate will design, build, and maintain scalable data pipelines and architectures to support our organization's data-driven initiatives and leverage Databricks to process large-scale datasets, optimize data workflows, enable advanced analytics and machine learning, and integrate Power BI for data visualization and reporting.
Key Responsibilities
· 5-8 years of professional work experience in a relevant field
· Proficient in Microsoft Fabric platform, Azure Databricks, ADF, Delta Lake, SQL Data Warehouse, Unity Catalog.
· Good Experience on Microsoft Dynamics 365
· Experience/ prior knowledge on semi structure data and Structured Streaming, Azure synapse, data lake, data warehouse.
· Proficient in creating Azure Data Factory pipelines for ETL/ELT processing; copy activity, custom Azure development etc.
· Good knowledge of SQL and Python for data manipulation, transformation, and analysis
· Understand business requirements to set functional specifications for reporting applications
- Data Pipeline Development: Design, develop, and maintain robust, scalable data pipelines using Databricks, Apache Spark, and other cloud-based technologies.
- Data Integration: Ingest, transform, and integrate data from diverse sources, including APIs, databases, streaming platforms, and third-party systems, into Databricks for analytics and reporting.
- Power BI Integration: Develop and optimize data models and datasets in Databricks for use in Power BI, ensuring efficient data connections and high-quality visualizations.
- Performance Optimization: Optimize data workflows, API calls, and queries on Databricks and Power BI for performance, cost-efficiency, and scalability.
- Data Modeling: Build and maintain data models to support business requirements, ensuring data quality, consistency, and accessibility for analytics and reporting.
- Cloud Integration: Implement data solutions on cloud platforms (e.g., AWS, Azure, GCP) integrated with Databricks, Power BI, and API ecosystems.
- Security & Compliance: Implement data governance, security, and compliance best practices within Databricks, Power BI, and API environments.
- Technical Skills:
- Proficiency in Databricks, including Delta Lake, Spark SQL.
- Strong programming skills in Python, Scala, or Java.
- Experience with Apache Spark for big data processing.
- Knowledge of SQL for querying and transforming data.
- Proficiency in Power BI for creating data models, DAX queries, and interactive dashboards.
Preferred Qualifications
- Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional).
Experience with real-time data processing and streaming














