20+ Data engineering Jobs in Chennai | Data engineering Job openings in Chennai

We are looking for an experienced Data Engineer with strong expertise in AWS, DBT, Databricks, and Apache Airflow to join our growing data engineering team.

Immediate joiners preferred

Role Overview 

The ideal candidate will design, develop, and maintain scalable data pipelines and data platforms to support analytics and business intelligence initiatives.

Key Responsibilities

1. Design and build scalable data pipelines using AWS, Databricks, DBT, and Airflow.
2. Develop and optimize ETL/ELT workflows for large-scale data processing.
3. Implement data transformation models using DBT.
4. Orchestrate workflows using Apache Airflow.
5. Work with Databricks for big data processing and analytics.
6. Ensure data quality, reliability, and performance optimization.
7. Collaborate with data analysts, engineers, and business teams.

Required Skills

1. Strong experience with AWS data services
2. Hands-on experience with Databricks
3. Experience in DBT (Data Build Tool)
4. Workflow orchestration using Apache Airflow
5. Strong SQL and Python skills
6. Experience in data warehousing and ETL pipelines

At BigThinkCode, our technology solves complex problems. We are looking for talented Data engineer to join our Data team at Chennai.

Please find below our job description, if interested apply / reply sharing your profile to connect and discuss.

Company: BigThinkCode Technologies

URL: https://www.bigthinkcode.com/

Work location: Chennai (work from office)

Experience required: 3-5 years

Work location: Chennai

Joining time: Immediate – 4 weeks

Work Mode: Work from office (Hybrid)

About the role: We are looking for a skilled Data Engineer with hands-on expertise in Dagster orchestration, modern data pipeline development, and Medallion architecture implementation. The ideal candidate will design, build, and optimize scalable data pipelines with strong SQL proficiency, data modelling expertise.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Dagster.
• Build and manage Dagster components such as:

o  Ops / Assets

o  Schedules

o  Sensors

o  Jobs

o  Resource definitions

• Implement and maintain Medallion Architecture (Bronze, Silver, Gold layers).
• Write optimized and production-grade SQL scripts for transformations and data validation.

Must Have

• 3+ years of experience in Data Engineering.
•  Strong hands-on experience with Dagster and workflow orchestration.
• Solid understanding of data pipeline design patterns.
• Experience implementing Medallion Architecture.
• Advanced SQL skills (complex joins, CTEs, performance tuning).
• Experience working with GCP cloud data platform.

Why Join Us:

• Collaborative work environment.
• Exposure to modern tools and scalable application architectures.
• Medical cover for employee and eligible dependents.
• Tax beneficial salary structure.
• Comprehensive leave policy
• Competency development training programs.

Hi,

We are seeking a senior data leader with deep functional expertise in Salesforce Sales and Service domains to own the enterprise data model, metrics, and analytical outcomes supporting Sales, Service, and Customer Operations.

This role is business‑first and data‑centric. The successful candidate understands how Salesforce Sales Cloud and Service Cloud data is generated, evolves over time, and is consumed by business teams, and ensures analytics accurately reflect operational reality.

Snowflake serves as the enterprise analytics platform, but Salesforce domain mastery and functional data expertise are the primary requirements for success in this role.

Core Responsibilities

Salesforce Sales & Service Data Ownership

·                 Act as the data owner and architect for Salesforce Sales and Service domains.

• Own Sales data including leads, accounts, opportunities, pipeline, bookings, revenue, forecasting, and CPQ (if applicable).
• Own Service data including cases, case lifecycle, SLAs, backlog, escalations, and service performance metrics.
• Define and govern enterprise‑wide KPI and metric definitions across Sales and Service.
• Ensure alignment between Salesforce operational definitions and analytics/reporting outputs.
• Own cross‑functional metrics spanning Sales, Service, and the customer lifecycle (e.g., customer health, renewals, churn).

Business‑Driven Data Modeling

·                 Design Salesforce‑centric analytical data models that accurately reflect Sales and Service processes.

• Model sales stage progression, pipeline history, and forecast changes over time.
• Model service case lifecycle, SLA compliance, backlog aging, and resolution metrics.
• Handle Salesforce‑specific complexities such as slowly changing dimensions (ownership, territory, account hierarchies).
• Ensure data models support operational dashboards, executive reporting, and advanced analytics.

Analytics Enablement & Business Partnership

·                 Partner closely with Sales Operations, Service Operations, Revenue Operations, Finance, and Analytics teams.

• Translate business questions into trusted, reusable analytical datasets.
• Identify data quality issues or Salesforce process gaps impacting reporting and drive remediation.
• Enable self‑service analytics through well‑documented, certified data products.

Technical Responsibilities (Enabling Focus)

·                 Architect and govern Salesforce data ingestion and modeling on Snowflake.

• Guide ELT/ETL strategies for Salesforce objects such as Opportunities, Accounts, Activities, Cases, and Entitlements.
• Ensure reconciliation and auditability between Salesforce, Finance, and analytics layers.
• Define data access, security, and governance aligned with Salesforce usage patterns.
• Partner with data engineering teams on scalability, performance, and cost efficiency.

Required Experience & Skills

Salesforce Sales & Service Domain Expertise (Must‑Have)

·                 Extensive hands‑on experience working with Salesforce Sales Cloud and Service Cloud data.

• Strong understanding of sales pipeline management, forecasting, and revenue reporting.
• Strong understanding of service case workflows, SLAs, backlog management, and service performance measurement.
• Experience working directly with Sales Operations and Service Operations teams.
• Ability to identify when Salesforce configuration or process issues cause reporting inconsistencies.

Data & Analytics Expertise

·                 10+ years working with business‑critical analytical data.

• Proven experience defining KPIs, metrics, and semantic models for Sales and Service domains.
• Strong SQL and analytical skills to validate business logic and data outcomes.
• Experience supporting BI and analytics platforms such as Tableau, Power BI, or MicroStrategy.

Platform Experience 

·                 Experience using Snowflake as an enterprise analytics platform.

• Understanding of modern ELT/ETL and cloud data architecture concepts.
• Familiarity with data governance, lineage, and access control best practices.

Leadership & Collaboration

·                 Acts as a bridge between business stakeholders and technical teams.

• Comfortable challenging requirements using business and data context.
• Mentors engineers and analysts on Salesforce data nuances and business meaning.
• Strong communicator able to explain complex Salesforce data behavior to non‑technical leaders.

Thanks,

Ampera Talent Team

Role Summary

Provide hands-on leadership across data and analytics initiatives, helping drive execution, coordination, and delivery in a fast-moving, evolving environment.

Key Responsibilities

• Partner closely with Data & Analytics leadership to translate priorities into executable plans.

• Drive delivery across cross-functional teams (data engineering, analytics, platform).

• Manage dependencies, risks, and timelines in a complex enterprise setting.

• Act as a trusted execution lead during periods of rapid change.

Key Skills

• Strong experience in data and analytics programs in large enterprises.

• Comfortable operating with ambiguity and evolving strategy.

• Excellent stakeholder communication and coordination skills.

• Prior healthcare or regulated-industry experience preferred.

Required Skills & Qualifications

Technical Skills

• Strong hands-on experience with Databricks and Apache Spark.
• Proficiency in Python and SQL.
• Proven experience in data mapping, transformation, and data modeling.
• Experience integrating data from APIs, databases, and cloud storage.
• Solid understanding of ETL/ELT concepts and data warehousing principles.

Key Responsibilities

Data Source Identification & Quality Assessment

Data Mapping & Integration

• Define and maintain comprehensive data mapping between source systems and Databricks tables.
• Design and implement scalable ETL/ELT pipelines using Databricks and Apache Spark.

Databricks & Data Modeling

• Develop and optimize Databricks workloads using Spark and Delta Lake.
• Design efficient data models optimized for performance, analytics, and API consumption.

Job Description – Senior Technical Business Analyst

Location: Trivandrum (Preferred) | Open to any location in India

Shift Timings - 8 hours window between the 7:30 PM IST - 4:30 AM IST

About the Role

We are seeking highly motivated and analytically strong Senior Technical Business Analysts who can work seamlessly with business and technology stakeholders to convert a one-line problem statement into a well-defined project or opportunity. This role is ideal for fresh graduates who have a strong foundation in data analytics, data engineering, data visualization, and data science, along with a strong drive to learn, collaborate, and grow in a dynamic, fast-paced environment.

As a Technical Business Analyst, you will be responsible for translating complex business challenges into actionable user stories, analytical models, and executable tasks in Jira. You will work across the entire data lifecycle—from understanding business context to delivering insights, solutions, and measurable outcomes.

Key Responsibilities

Business & Analytical Responsibilities

• Partner with business teams to understand one-line problem statements and translate them into detailed business requirements, opportunities, and project scope.
• Conduct exploratory data analysis (EDA) to uncover trends, patterns, and business insights.
• Create documentation including Business Requirement Documents (BRDs), user stories, process flows, and analytical models.
• Break down business needs into concise, actionable, and development-ready user stories in Jira.

Data & Technical Responsibilities

• Collaborate with data engineering teams to design, review, and validate data pipelines, data models, and ETL/ELT workflows.
• Build dashboards, reports, and data visualizations using leading BI tools to communicate insights effectively.
• Apply foundational data science concepts such as statistical analysis, predictive modeling, and machine learning fundamentals.
• Validate and ensure data quality, consistency, and accuracy across datasets and systems.

Collaboration & Execution

• Work closely with product, engineering, BI, and operations teams to support the end-to-end delivery of analytical solutions.
• Assist in development, testing, and rollout of data-driven solutions.
• Present findings, insights, and recommendations clearly and confidently to both technical and non-technical stakeholders.

Required Skillsets

Core Technical Skills

• 6+ years of Technical Business Analyst experience within an overall professional experience of 8+ years
• Data Analytics: SQL, descriptive analytics, business problem framing.
• Data Engineering (Foundational): Understanding of data warehousing, ETL/ELT processes, cloud data platforms (AWS/GCP/Azure preferred).
• Data Visualization: Experience with Power BI, Tableau, or equivalent tools.
• Data Science (Basic/Intermediate): Python/R, statistical methods, fundamentals of ML algorithms.

Soft Skills

• Strong analytical thinking and structured problem-solving capability.
• Ability to convert business problems into clear technical requirements.
• Excellent communication, documentation, and presentation skills.
• High curiosity, adaptability, and eagerness to learn new tools and techniques.

Educational Qualifications

• BE/B.Tech or equivalent in:
• Computer Science / IT
• Data Science

What We Look For

• Demonstrated passion for data and analytics through projects and certifications.
• Strong commitment to continuous learning and innovation.
• Ability to work both independently and in collaborative team environments.
• Passion for solving business problems using data-driven approaches.
• Proven ability (or aptitude) to convert a one-line business problem into a structured project or opportunity.

Why Join Us?

• Exposure to modern data platforms, analytics tools, and AI technologies.
• A culture that promotes innovation, ownership, and continuous learning.
• Supportive environment to build a strong career in data and analytics.

Skills: Data Analytics, Business Analysis, Sql

Must-Haves

Technical Business Analyst (6+ years), SQL, Data Visualization (Power BI, Tableau), Data Engineering (ETL/ELT, cloud platforms), Python/R

Google Data Engineer - SSE

Position Description

Google Cloud Data Engineer

Notice Period: Immediate to 30 days serving

Job Description:

We are seeking a highly skilled Data Engineer with extensive experience in Google Cloud Platform (GCP) data services and big data technologies. The ideal candidate will be responsible for designing, implementing, and optimizing scalable data solutions while ensuring high performance, reliability, and security.

Key Responsibilities:

• Design, develop, and maintain scalable data pipelines and architectures using GCP data services.

• Implement and optimize solutions using BigQuery, Dataproc, Composer, Pub/Sub, Dataflow, GCS, and BigTable.

• Work with GCP databases such as Bigtable, Spanner, CloudSQL, AlloyDB, ensuring performance, security, and availability.

• Develop and manage data processing workflows using Apache Spark, Hadoop, Hive, Kafka, and other Big Data technologies.

• Ensure data governance and security using Dataplex, Data Catalog, and other GCP governance tooling.

• Collaborate with DevOps teams to build CI/CD pipelines for data workloads using Cloud Build, Artifact Registry, and Terraform.

• Optimize query performance and data storage across structured and unstructured datasets.

• Design and implement streaming data solutions using Pub/Sub, Kafka, or equivalent technologies.

Required Skills & Qualifications:

• 8-15 years of experience

• Strong expertise in GCP Dataflow, Pub/Sub, Cloud Composer, Cloud Workflow, BigQuery, Cloud Run, Cloud Build.

• Proficiency in Python and Java, with hands-on experience in data processing and ETL pipelines.

• In-depth knowledge of relational databases (SQL, MySQL, PostgreSQL, Oracle) and NoSQL databases (MongoDB, Scylla, Cassandra, DynamoDB).

• Experience with Big Data platforms such as Cloudera, Hortonworks, MapR, Azure HDInsight, IBM Open Platform.

• Strong understanding of AWS Data services such as Redshift, RDS, Athena, SQS/Kinesis.

• Familiarity with data formats such as Avro, ORC, Parquet.

• Experience handling large-scale data migrations and implementing data lake architectures.

• Expertise in data modeling, data warehousing, and distributed data processing frameworks.

• Deep understanding of data formats such as Avro, ORC, Parquet.

• Certification in GCP Data Engineering Certification or equivalent.

Good to Have:

• Experience in BigQuery, Presto, or equivalent.

• Exposure to Hadoop, Spark, Oozie, HBase.

• Understanding of cloud database migration strategies.

• Knowledge of GCP data governance and security best practices.

Technical Architect (Databricks)

• 10+ Years Data Engineering Experience with expertise in Databricks
• 3+ years of consulting experience
• Completed Data Engineering Professional certification & required classes
• Minimum 2-3 projects delivered with hands-on experience in Databricks
• Completed Apache Spark Programming with Databricks, Data Engineering with Databricks, Optimizing Apache Spark™ on Databricks
• Experience in Spark and/or Hadoop, Flink, Presto, other popular big data engines
• Familiarity with Databricks multi-hop pipeline architecture

Sr. Data Engineer (Databricks)

• 5+ Years Data Engineering Experience with expertise in Databricks
• Completed Data Engineering Associate certification & required classes
• Minimum 1 project delivered with hands-on experience in development on Databricks
• Completed Apache Spark Programming with Databricks, Data Engineering with Databricks, Optimizing Apache Spark™ on Databricks
• SQL delivery experience, and familiarity with Bigquery, Synapse or Redshift
• Proficient in Python, knowledge of additional databricks programming languages (Scala)

Work Mode: Hybrid

Need B.Tech, BE, M.Tech, ME candidates - Mandatory

Must-Have Skills:

● Educational Qualification :- B.Tech, BE, M.Tech, ME in any field.

● Minimum of 3 years of proven experience as a Data Engineer.

● Strong proficiency in Python programming language and SQL.

● Experience in DataBricks and setting up and managing data pipelines, data warehouses/lakes.

● Good comprehension and critical thinking skills.

● Kindly note Salary bracket will vary according to the exp. of the candidate - 

- Experience from 4 yrs to 6 yrs - Salary upto 22 LPA

- Experience from 5 yrs to 8 yrs - Salary upto 30 LPA

- Experience more than 8 yrs - Salary upto 40 LPA

We are seeking a skilled Cloud Data Engineer who has experience with cloud data platforms like AWS or Azure and especially Snowflake and dbt to join our dynamic team. As a consultant, you will be responsible for developing new data platforms and create the data processes. You will collaborate with cross-functional teams to design, develop, and deploy high-quality frontend solutions. 

Responsibilities:

Customer consulting: You develop data-driven products in the Snowflake Cloud and connect data & analytics with specialist departments. You develop ELT processes using dbt (data build tool) 

Specifying requirements: You develop concrete requirements for future-proof cloud data architectures.

Develop data routes: You design scalable and powerful data management processes.

Analyze data: You derive sound findings from data sets and present them in an understandable way.

Requirements:

Requirements management and project experience: You successfully implement cloud-based data & analytics projects.

Data architectures: You are proficient in DWH/data lake concepts and modeling with Data Vault 2.0.

Cloud expertise: You have extensive knowledge of Snowflake, dbt and other cloud technologies (e.g. MS Azure, AWS, GCP).

SQL know-how: You have a sound and solid knowledge of SQL.

Data management: You are familiar with topics such as master data management and data quality.

Bachelor's degree in computer science, or a related field.

Strong communication and collaboration abilities to work effectively in a team environment.

Skills & Requirements

Cloud Data Engineering, AWS, Azure, Snowflake, dbt, ELT processes, Data-driven consulting, Cloud data architectures, Scalable data management, Data analysis, Requirements management, Data warehousing, Data lake, Data Vault 2.0, SQL, Master data management, Data quality, GCP, Strong communication, Collaboration.

Job Summary:

Seeking an experienced Senior Data Engineer to lead data ingestion, transformation, and optimization initiatives using the modern Apache and Azure data stack. The role involves working on scalable pipelines, large-scale distributed systems, and data lake management.

Core Responsibilities:

· Build and manage high-volume data pipelines using Spark/Databricks.

· Implement ELT frameworks using Azure Data Factory/Synapse Pipelines.

· Optimize large-scale datasets in Delta/Iceberg formats.

· Implement robust data quality, monitoring, and governance layers.

· Collaborate with Data Scientists, Analysts, and Business stakeholders.

Technical Stack:

· Big Data: Apache Spark, Kafka, Hive, Airflow, Hudi/Iceberg

· Cloud: Azure (Synapse, ADF, ADLS Gen2), Databricks, AWS (Glue/S3)

· Languages: Python, Scala, SQL

· Storage Formats: Delta Lake, Iceberg, Parquet, ORC

· CI/CD: Azure DevOps, Terraform (infra as code), Git

Senior Data Engineer (Apache Stack + Databricks/Synapse)

Share cv to

Thirega@ vysystems dot com - WhatsApp - 91Five0033Five2Three

JOB DESCRIPTION:. THE IDEAL CANDIDATE WILL:

• Ensure new features and subject areas are modelled to integrate with existing structures and provide a consistent view. Develop and maintain documentation of the data architecture, data flow and data models of the data warehouse appropriate for various audiences. Provide direction on adoption of Cloud technologies (Snowflake) and industry best practices in the field of data warehouse architecture and modelling.

• Providing technical leadership to large enterprise scale projects. You will also be responsible for preparing estimates and defining technical solutions to proposals (RFPs). This role requires a broad range of skills and the ability to step into different roles depending on the size and scope of the project Roles & Responsibilities.

ELIGIBILITY CRITERIA: Desired Experience/Skills:
• Must have total 5+ yrs. in IT and 2+ years' experience working as a snowflake Data Architect and 4+ years in Data warehouse, ETL, BI projects.
• Must have experience at least two end to end implementation of Snowflake cloud data warehouse and 3 end to end data warehouse implementations on-premise preferably on Oracle.

• Expertise in Snowflake – data modelling, ELT using Snowflake SQL, implementing complex stored Procedures and standard DWH and ETL concepts
• Expertise in Snowflake advanced concepts like setting up resource monitors, RBAC controls, virtual warehouse sizing, query performance tuning, Zero copy clone, time travel and understand how to use these features
• Expertise in deploying Snowflake features such as data sharing, events and lake-house patterns
• Hands-on experience with Snowflake utilities, SnowSQL, SnowPipe, Big Data model techniques using Python
• Experience in Data Migration from RDBMS to Snowflake cloud data warehouse
• Deep understanding of relational as well as NoSQL data stores, methods and approaches (star and snowflake, dimensional modelling)
• Experience with data security and data access controls and design
• Experience with AWS or Azure data storage and management technologies such as S3 and ADLS
• Build processes supporting data transformation, data structures, metadata, dependency and workload management
• Proficiency in RDBMS, complex SQL, PL/SQL, Unix Shell Scripting, performance tuning and troubleshoot
• Provide resolution to an extensive range of complicated data pipeline related problems, proactively and as issues surface
• Must have expertise in AWS or Azure Platform as a Service (PAAS)
• Certified Snowflake cloud data warehouse Architect (Desirable)
• Should be able to troubleshoot problems across infrastructure, platform and application domains.
• Must have experience of Agile development methodologies
• Strong written communication skills. Is effective and persuasive in both written and oral communication

Nice to have Skills/Qualifications:Bachelor's and/or master’s degree in computer science or equivalent experience.
• Strong communication, analytical and problem-solving skills with a high attention to detail.

About you:
• You are self-motivated, collaborative, eager to learn, and hands on
• You love trying out new apps, and find yourself coming up with ideas to improve them
• You stay ahead with all the latest trends and technologies
• You are particular about following industry best practices and have high standards regarding quality

• Create and maintain optimal data pipeline architecture,
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
• Keep our data separated and secure across national boundaries through multiple data centers and AWS regions.
• Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Work with data and analytics experts to strive for greater functionality in our data systems.

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong analytic skills related to working with unstructured datasets.
• Build processes supporting data transformation, data structures, metadata, dependency and workload management.
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores.
• Strong project management and organizational skills.
• Experience supporting and working with cross-functional teams in a dynamic environment.
• We are looking for a candidate with 5+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field. They should also have experience using the following software/tools: Experience with big
  • data tools: Hadoop, Spark, Kafka, etc.
  • Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
  • Experience with data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
  • Experience with AWS cloud services: EC2, EMR, RDS, Redshift
  • Experience with stream-processing systems: Storm, Spark-Streaming, etc.
  • Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.

Looking Data Enginner for our OWn organization-

Notice Period- 15-30 days
CTC- upto 15 lpa

Preferred Technical Expertise 

1. Expertise in Python programming.
2. Proficient in Pandas/Numpy Libraries. 
3. Experience with Django framework and API Development.
4. Proficient in writing complex queries using SQL
5. Hands on experience with Apache Airflow.
6. Experience with source code versioning tools such as GIT, Bitbucket etc.

 Good to have Skills:

1. Create and maintain Optimal Data Pipeline Architecture
2. Experienced in handling large structured data.
3. Demonstrated ability in solutions covering data ingestion, data cleansing, ETL, Data mart creation and exposing data for consumers.
4. Experience with any cloud platform (GCP is a plus)
5. Experience with JQuery, HTML, Javascript, CSS is a plus.

We are looking for passionate, talented and super-smart engineers to join our product development team. If you are someone who innovates, loves solving hard problems, and enjoys end-to-end product development, then this job is for you! You will be working with some of the best developers in the industry in a self-organising, agile environment where talent is valued over job title or years of experience.

Responsibilities:

• You will be involved in end-to-end development of VIMANA technology, adhering to our development practices and expected quality standards.
• You will be part of a highly collaborative Agile team which passionately follows SAFe Agile practices, including pair-programming, PR reviews, TDD, and Continuous Integration/Delivery (CI/CD).
• You will be working with cutting-edge technologies and tools for stream processing using Java, NodeJS and Python, using frameworks like Spring, RxJS etc.
• You will be leveraging big data technologies like Kafka, Elasticsearch and Spark, processing more than 10 Billion events per day to build a maintainable system at scale.
• You will be building Domain Driven APIs as part of a micro-service architecture.
• You will be part of a DevOps culture where you will get to work with production systems, including operations, deployment, and maintenance.
• You will have an opportunity to continuously grow and build your capabilities, learning new technologies, languages, and platforms.

Requirements:

• Undergraduate degree in Computer Science or a related field, or equivalent practical experience.
• 2 to 5 years of product development experience.
• Experience building applications using Java, NodeJS, or Python.
• Deep knowledge in Object-Oriented Design Principles, Data Structures, Dependency Management, and Algorithms.
• Working knowledge of message queuing, stream processing, and highly scalable Big Data technologies.

• Experience in working with Agile software methodologies (XP, Scrum, Kanban), TDD and Continuous Integration (CI/CD).
• Experience using no-SQL databases like MongoDB or Elasticsearch.
• Prior experience with container orchestrators like Kubernetes is a plus.

• We recognize that our dedicated team members make us successful and we offer competitive salaries.
• We are a workplace that values work-life balance, provides flexible working hours, and full time remote work options.
• You will be part of a team that is highly motivated to learn and work on cutting edge technologies, tools, and development practices.
• Bon Appetit! Enjoy catered breakfasts, lunches and free snacks!

Job Sector: IT, Software

Job Type: Permanent

Location: Chennai

Experience: 10 - 20 Years

Salary: 12 – 40 LPA

Education: Any Graduate

Notice Period: Immediate

Key Skills: Python, Spark, AWS, SQL, PySpark

Contact at triple eight two zero nine four two double seven

Job Description:

Requirements

• Minimum 12 years experience
• In depth understanding and knowledge on distributed computing with spark.
• Deep understanding of Spark Architecture and internals
• Proven experience in data ingestion, data integration and data analytics with spark, preferably PySpark.
• Expertise in ETL processes, data warehousing and data lakes.
• Hands on with python for Big data and analytics.
• Hands on in agile scrum model is an added advantage.
• Knowledge on CI/CD and orchestration tools is desirable.
• AWS S3, Redshift, Lambda knowledge is preferred

• Expertise in designing and implementing enterprise scale database (OLTP) and Data warehouse solutions.
• Hands on experience in implementing Azure SQL Database, Azure SQL Date warehouse (Azure Synapse Analytics) and big data processing using Azure Databricks and Azure HD Insight.
• Expert in writing T-SQL programming for complex stored procedures, functions, views and query optimization.
• Should be aware of Database development for both on-premise and SAAS Applications using SQL Server and PostgreSQL.
• Experience in ETL and ELT implementations using Azure Data Factory V2 and SSIS.
• Experience and expertise in building machine learning models using Logistic and linear regression, Decision tree  and Random forest Algorithms.
• PolyBase queries for exporting and importing data into Azure Data Lake.
• Building data models both tabular and multidimensional using SQL Server data tools.
• Writing data preparation, cleaning and processing steps using Python, SCALA, and R.
• Programming experience using python libraries NumPy, Pandas and Matplotlib.
• Implementing NOSQL databases and writing queries using cypher.
• Designing end user visualizations using Power BI, QlikView and Tableau.
• Experience working with all versions of SQL Server 2005/2008/2008R2/2012/2014/2016/2017/2019
• Experience using the expression languages MDX and DAX.
• Experience in migrating on-premise SQL server database to Microsoft Azure.
• Hands on experience in using Azure blob storage, Azure Data Lake Storage Gen1 and Azure Data Lake Storage Gen2.
• Performance tuning complex SQL queries, hands on experience using SQL Extended events.
• Data modeling using Power BI for Adhoc reporting.
• Raw data load automation using T-SQL and SSIS
• Expert in migrating existing on-premise database to SQL Azure.
• Experience in using U-SQL for Azure Data Lake Analytics.
• Hands on experience in generating SSRS reports using MDX.
• Experience in designing predictive models using Python and SQL Server.
• Developing machine learning models using Azure Databricks and SQL Server

Company Description

At Bungee Tech, we help retailers and brands meet customers everywhere and, on every occasion, they are in. We believe that accurate, high-quality data matched with compelling market insights empowers retailers and brands to keep their customers at the center of all innovation and value they are delivering. 

We provide a clear and complete omnichannel picture of their competitive landscape to retailers and brands. We collect billions of data points every day and multiple times in a day from publicly available sources. Using high-quality extraction, we uncover detailed information on products or services, which we automatically match, and then proactively track for price, promotion, and availability. Plus, anything we do not match helps to identify a new assortment opportunity.

Empowered with this unrivalled intelligence, we unlock compelling analytics and insights that once blended with verified partner data from trusted sources such as Nielsen, paints a complete, consolidated picture of the competitive landscape.

We are looking for a Big Data Engineer who will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

You will also be responsible for integrating them with the architecture used in the company.

We're working on the future. If you are seeking an environment where you can drive innovation, If you want to apply state-of-the-art software technologies to solve real world problems, If you want the satisfaction of providing visible benefit to end-users in an iterative fast paced environment, this is your opportunity.

Responsibilities

As an experienced member of the team, in this role, you will:

• Contribute to evolving the technical direction of analytical Systems and play a critical role their design and development

• You will research, design and code, troubleshoot and support. What you create is also what you own.

• Develop the next generation of automation tools for monitoring and measuring data quality, with associated user interfaces.

• Be able to broaden your technical skills and work in an environment that thrives on creativity, efficient execution, and product innovation.

BASIC QUALIFICATIONS

• Bachelor’s degree or higher in an analytical area such as Computer Science, Physics, Mathematics, Statistics, Engineering or similar.
• 5+ years relevant professional experience in Data Engineering and Business Intelligence
• 5+ years in with Advanced SQL (analytical functions), ETL, Data Warehousing.
• Strong knowledge of data warehousing concepts, including data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools and environments, data structures, data modeling and performance tuning.
• Ability to effectively communicate with both business and technical teams.
• Excellent coding skills in Java, Python, C++, or equivalent object-oriented programming language
• Understanding of relational and non-relational databases and basic SQL
• Proficiency with at least one of these scripting languages: Perl / Python / Ruby / shell script

PREFERRED QUALIFICATIONS

• Experience with building data pipelines from application databases.
• Experience with AWS services - S3, Redshift, Spectrum, EMR, Glue, Athena, ELK etc.
• Experience working with Data Lakes.
• Experience providing technical leadership and mentor other engineers for the best practices on the data engineering space
• Sharp problem solving skills and ability to resolve ambiguous requirements
• Experience on working with Big Data
• Knowledge and experience on working with Hive and the Hadoop ecosystem
• Knowledge of Spark
• Experience working with Data Science teams

• Must have the experience of leading teams and drive customer interactions
• Must have multiple successful deployments user stories
• Extensive hands on experience in Apache Spark along with HiveQL
• Sound knowledge in Amazon Web Services or any other Cloud environment.
• Experienced in data flow orchestration using Apache Airflow
• JSON, XML, CSV, Parquet file formats with snappy compression.
• File movements between HDFS and AWS S3
• Experience in shell scripting and scripting to automate report generation and migration of reports to AWS S3
• Worked in building a data pipeline using Pandas and Flask FrameworkGood Familiarity with Anaconda and Jupyternotebook