30+ Big data Jobs in Chennai | Big data Job openings in Chennai

Company Introduction

About Hexr Factory :

We are always exploring the possibilities to bridge the physical and digital worlds. We design and build Metaverse & Digital twin technologies for the future of industry and entertainment.

Job type & Location

Project Role: C++ Developer

Project Role Description :

The primary focus will be the development of all core engines, designing back-end components, integrating data storage, and ensuring high performance and responsiveness to requests from the front end. You will also be responsible for integrating the front-end elements built by your co-workers/Third-Party into the application.

Work Experience: 2 - 8 years

Work location: Chennai

Must-Have Skills: C++, OpenCV, Ruby, Boost C++ libraries, MySQL, MQTT. 

Key Responsibilities:

• Extensive knowledge of C++ frameworks and libraries that utilize openCCTV, FFMPEG, video processing, and analytics.
• Multi-threading programming, Distributed and Parallel computing, Big Data technologies, SQL Server programming with T-SQL, and Microsoft data access technologies.
• Familiar with libraries like OpenCV, FFmpeg, GStreamer, and Directshow.
• Extensive knowledge of RTSP, RTMP, and HLS video streaming protocols.
• Candidate should know about release activities, source control, merging, and branching concepts.
• Ability to analyze and visualize BIG data effectively.
• Machine learning:- KNN, SVM, Text Search.
• Familiar with QGIS.
• A good understanding of Computer Vision and Image Processing concepts and algorithms.
• Thorough knowledge of the standard library, STL containers, and algorithms.
• Strong background in object-oriented design, prioritizing test ability and re-usability.
• Familiarity with embedded systems design and low-level hardware interactions.
• Proven track record of identifying bottlenecks and bugs, and devising solutions to these problems.
• Hands-on Algorithm development and implementation.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Skills Required : 

• Experience with existing computer vision toolkits such as Open-CV.
• Current trends within Computer Vision and Image Processing in academia and community.
• Deep learning using convolution neural networks for object classification, recognition, or sequence modeling.
• Experience with any of the following: Object detection and target tracking, simultaneous localization and mapping, 3D reconstruction, camera calibration, behavior analysis, automated video surveillance, virtual makeup, and related fields.
• Proficient understanding of code versioning tools, such as Git.
• Passionate about new technology and innovation.
• Understanding the nature of asynchronous programming and its quirks and workarounds.
• Excellent verbal and written communication skills.

Title: Platform Engineer Location: Chennai Work Mode: Hybrid (Remote and Chennai Office) Experience: 4+ years Budget: 16 - 18 LPA

Responsibilities:

• Parse data using Python, create dashboards in Tableau.
• Utilize Jenkins for Airflow pipeline creation and CI/CD maintenance.
• Migrate Datastage jobs to Snowflake, optimize performance.
• Work with HDFS, Hive, Kafka, and basic Spark.
• Develop Python scripts for data parsing, quality checks, and visualization.
• Conduct unit testing and web application testing.
• Implement Apache Airflow and handle production migration.
• Apply data warehousing techniques for data cleansing and dimension modeling.

Requirements:

• 4+ years of experience as a Platform Engineer.
• Strong Python skills, knowledge of Tableau.
• Experience with Jenkins, Snowflake, HDFS, Hive, and Kafka.
• Proficient in Unix Shell Scripting and SQL.
• Familiarity with ETL tools like DataStage and DMExpress.
• Understanding of Apache Airflow.
• Strong problem-solving and communication skills.

Note: Only candidates willing to work in Chennai and available for immediate joining will be considered. Budget for this position is 16 - 18 LPA.

• KSQL
• Data Engineering spectrum (Java/Spark)
• Spark Scala / Kafka Streaming
• Confluent Kafka components
• Basic understanding of Hadoop

We are looking for a Big Data Engineer with java for Chennai Location

Location : Chennai 

Exp : 11 to 15 Years 

Job description

Required Skill:

1. Candidate should have minimum 7 years of experience as total

2. Candidate should have minimum 4 years of experience in Big Data design and development

3. Candidate should have experience in Java, Spark, Hive & Hadoop, Python 

4. Candidate should have experience in any RDBMS.

Roles & Responsibility:

1. To create work plans, monitor and track the work schedule for on time delivery as per the defined quality standards.

2. To develop and guide the team members in enhancing their technical capabilities and increasing productivity.

3. To ensure process improvement and compliance in the assigned module, and participate in technical discussions or review.

4. To prepare and submit status reports for minimizing exposure and risks on the project or closure of escalation

Regards,

Priyanka S

7P8R9I9Y4A0N8K8A7S7

Data Engineer- Senior

Cubera is a data company revolutionizing big data analytics and Adtech through data share value principles wherein the users entrust their data to us. We refine the art of understanding, processing, extracting, and evaluating the data that is entrusted to us. We are a gateway for brands to increase their lead efficiency as the world moves towards web3.

What are you going to do?

Design & Develop high performance and scalable solutions that meet the needs of our customers.

Closely work with the Product Management, Architects and cross functional teams.

Build and deploy large-scale systems in Java/Python.

Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

Create data tools for analytics and data scientist team members that assist them in building and optimizing their algorithms.

Follow best practices that can be adopted in Bigdata stack.

Use your engineering experience and technical skills to drive the features and mentor the engineers.

What are we looking for ( Competencies) :

Bachelor’s degree in computer science, computer engineering, or related technical discipline.

Overall 5 to 8 years of programming experience in Java, Python including object-oriented design.

Data handling frameworks: Should have a working knowledge of one or more data handling frameworks like- Hive, Spark, Storm, Flink, Beam, Airflow, Nifi etc.

Data Infrastructure: Should have experience in building, deploying and maintaining applications on popular cloud infrastructure like AWS, GCP etc.

Data Store: Must have expertise in one of general-purpose No-SQL data stores like Elasticsearch, MongoDB, Redis, RedShift, etc.

Strong sense of ownership, focus on quality, responsiveness, efficiency, and innovation.

Ability to work with distributed teams in a collaborative and productive manner.

Benefits:

Competitive Salary Packages and benefits.

Collaborative, lively and an upbeat work environment with young professionals.

Job Category: Development

Job Type: Full Time

Job Location: Bangalore

Bigdata with cloud:

Experience : 5-10 years

Location : Hyderabad/Chennai

Notice period : 15-20 days Max

1.  Expertise in building AWS Data Engineering pipelines with AWS Glue -> Athena -> Quick sight

2.  Experience in developing lambda functions with AWS Lambda

3.  Expertise with Spark/PySpark – Candidate should be hands on with PySpark code and should be able to do transformations with Spark

4.  Should be able to code in Python and Scala.

5.  Snowflake experience will be a plus

GCP  Data Analyst profile must have below skills sets :

• Knowledge of programming languages like https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.simplilearn.com%2Ftutorials%2Fsql-tutorial%2Fhow-to-become-sql-developer&;data=05%7C01%7Ca_anjali%40hcl.com%7C4ae720b3f3cc45c3e04608da3346b335%7C189de737c93a4f5a8b686f4ca9941912%7C0%7C0%7C637878675987971859%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EImfaJAD1KHOyrBQ7FkbaPl1STtfnf4QdQlbjw72%2BmE%3D&reserved=0" target="_blank">SQL, Oracle, R, MATLAB, Java and https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.simplilearn.com%2Fwhy-learn-python-a-guide-to-unlock-your-python-career-article&;data=05%7C01%7Ca_anjali%40hcl.com%7C4ae720b3f3cc45c3e04608da3346b335%7C189de737c93a4f5a8b686f4ca9941912%7C0%7C0%7C637878675987971859%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Z2n1Xy%2F3YN6nQqSweU5T7EfUTa1kPAAjbCMTWxDCh%2FY%3D&reserved=0" target="_blank">Python
• Data cleansing, data visualization, data wrangling
• Data modeling , data warehouse concepts
• Adapt to Big data platform like Hadoop, Spark for stream & batch processing
• GCP (Cloud Dataproc, Cloud Dataflow, Cloud Datalab, Cloud Dataprep, BigQuery, Cloud Datastore, Cloud Datafusion, Auto ML etc)

Lead QA: more than 5 years experience , led the team of more than 5 people in big data platform, should have experience in Test Automation framework, should have experience of Test process documentation

Job Title: Engineering Manager

Job Location: Chennai, Bangalore
Job Summary
The Engineering Org is looking for a proficient Engineering Manager to join a team that is building exciting
and futuristic Data Products at Condé Nast to enable both internal and external marketers to target
audiences in real time. As an Engineering Manager, you will drive the day-to-day execution of technical
and architectural decisions. EM will own engineering deliverables inclusive of solving dependencies
such as architecture, solutions, sequencing, and working with other engineering delivery teams.This role
is also responsible for driving innovation, prototyping, and recommending solutions. Above all, you will
influence how users interact with Conde Nast’s industry-leading journalism.
● Primary Responsibilities
● Manage a high performing team of Software and Data Engineers within the Data & ML
Engineering team part of Engineering Data Organization.
● Provide leadership and guidance to the team in Data Discovery, Data Ingestion, Transformation
and Storage
● Utilizing product mindset to build, scale and deploy holistic data products after successful
prototyping and drive their engineering implementation
● Provide technical coaching and lead direct reports and other members of adjacent support teams
to the highest level of performance..
● Evaluate performance of direct reports and offer career development guidance.
● Meeting hiring and retention targets of the team & building a high-performance culture
● Handle escalations from internal stakeholders and manage critical issues to resolution.
● Collaborate with Architects, Product Manager, Project Manager and other teams to deliver high
quality products.
● Identify recurring system and application issues and enable engineers to work with release teams,
infra teams, product development, vendors and other stakeholders in investigating and resolving
the cause.
● Required Skills
● 4+ years of managing Software Development teams, preferably in ML and Data
Engineering teams.
● 4+ years of Agile Software development practices
● 12+ years of Software Development experience.
● Excellent Problem Solving and System Design skill
● Hands on: Writing and Reviewing code primarily in Spark, Python and/or Java
● Hand on: Architect & Design end to end Data Pipeline (noSQL databases, Job Schedulers, Big
Data Development preferably on Databricks / Cloud)
● Experience with SOA & Microservice architecture
● Knowledge of Software Engineering best practices with experience on implementing CI/CD,
Log aggregation/Monitoring/alerting for production system
● Working Knowledge of cloud and devops skills (AWS will be preferred)
● Strong verbal and written communication skills.
● Experience in evaluating team member performance and offering career development
guidance.
● Experience in providing technical coaching to direct reports.
● Experience in architecting highly scalable products.
● Experience in collaborating with global stakeholder teams.
● Experience in working on highly available production systems.
● Strong knowledge of software release process and release pipeline.
About Condé Nast
CONDÉ NAST INDIA (DATA)
Over the years, Condé Nast successfully expanded and diversified into digital, TV, and social
platforms - in other words, a staggering amount of user data. Condé Nast made the right move to
invest heavily in understanding this data and formed a whole new Data team entirely dedicated to
data processing, engineering, analytics, and visualization. This team helps drive engagement, fuel
process innovation, further content enrichment, and increase market revenue. The Data team
aimed to create a company culture where data was the common language and facilitate an
environment where insights shared in real-time could improve performance.
The Global Data team operates out of Los Angeles, New York, Chennai, and London. The team at
Condé Nast Chennai works extensively with data to amplify its brands' digital capabilities and boost
online revenue. We are broadly divided into four groups, Data Intelligence, Data Engineering, Data
Science, and Operations (including Product and Marketing Ops, Client Services) along with Data
Strategy and monetization. The teams built capabilities and products to create data-driven solutions
for better audience engagement.
What we look forward to:
We want to welcome bright, new minds into our midst and work together to create diverse forms of
self-expression. At Condé Nast, we encourage the imaginative and celebrate the extraordinary. We
are a media company for the future, with a remarkable past. We are Condé Nast, and It Starts Here.

Responsibilities:

• Must be able to write quality code and build secure, highly available systems.
• Assemble large, complex datasets that meet functional / non-functional business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing datadelivery, re-designing infrastructure for greater scalability, etc with the guidance.
• Create datatools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader.
• Monitoring performance and advising any necessary infrastructure changes.
• Defining dataretention policies.
• Implementing the ETL process and optimal data pipeline architecture
• Build analytics tools that utilize the datapipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
• Create design documents that describe the functionality, capacity, architecture, and process.
• Develop, test, and implement datasolutions based on finalized design documents.
• Work with dataand analytics experts to strive for greater functionality in our data
• Proactively identify potential production issues and recommend and implement solutions

Skillsets:

• Good understanding of optimal extraction, transformation, and loading of datafrom a wide variety of data sources using SQL and AWS ‘big data’ technologies.
• Proficient understanding of distributed computing principles
• Experience in working with batch processing/ real-time systems using various open-source technologies like NoSQL, Spark, Pig, Hive, Apache Airflow.
• Implemented complex projects dealing with the considerable datasize (PB).
• Optimization techniques (performance, scalability, monitoring, etc.)
• Experience with integration of datafrom multiple data sources
• Experience with NoSQL databases, such as HBase, Cassandra, MongoDB, etc.,
• Knowledge of various ETL techniques and frameworks, such as Flume
• Experience with various messaging systems, such as Kafka or RabbitMQ
• Good understanding of Lambda Architecture, along with its advantages and drawbacks
• Creation of DAGs for dataengineering
• Expert at Python /Scala programming, especially for dataengineering/ ETL purposes

Job Sector: IT, Software

Job Type: Permanent

Location: Chennai

Experience: 10 - 20 Years

Salary: 12 – 40 LPA

Education: Any Graduate

Notice Period: Immediate

Key Skills: Python, Spark, AWS, SQL, PySpark

Contact at triple eight two zero nine four two double seven

Job Description:

Requirements

• Minimum 12 years experience
• In depth understanding and knowledge on distributed computing with spark.
• Deep understanding of Spark Architecture and internals
• Proven experience in data ingestion, data integration and data analytics with spark, preferably PySpark.
• Expertise in ETL processes, data warehousing and data lakes.
• Hands on with python for Big data and analytics.
• Hands on in agile scrum model is an added advantage.
• Knowledge on CI/CD and orchestration tools is desirable.
• AWS S3, Redshift, Lambda knowledge is preferred

Client  An IT Services Major, hiring for a leading insurance player.

Position: SENIOR CONSULTANT

Job Description:

• Azure admin- senior consultant with HD Insights(Big data)

Skills and Experience

• Microsoft Azure Administrator certification
• Bigdata project experience in Azure HDInsight Stack. big data processing frameworks such as Spark, Hadoop, Hive, Kafka or Hbase.
• Preferred: Insurance or BFSI domain experience
• 5 to 5 years of experience is required.

We are looking for an outstanding Big Data Engineer with experience setting up and maintaining Data Warehouse and Data Lakes for an Organization. This role would closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Roles and Responsibilities:

• Develop and maintain scalable data pipelines and build out new integrations and processes required for optimal extraction, transformation, and loading of data from a wide variety of data sources using 'Big Data' technologies.
• Develop programs in Scala and Python as part of data cleaning and processing.
• Assemble large, complex data sets that meet functional / non-functional business requirements and fostering data-driven decision making across the organization.  
• Responsible to design and develop distributed, high volume, high velocity multi-threaded event processing systems.
• Implement processes and systems to validate data, monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
• Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Provide high operational excellence guaranteeing high availability and platform stability.
• Closely collaborate with the Data Science team and assist the team build and deploy machine learning and deep learning models on big data analytics platforms.

Skills:

• Experience with Big Data pipeline, Big Data analytics, Data warehousing.
• Experience with SQL/No-SQL, schema design and dimensional data modeling.
• Strong understanding of Hadoop Architecture, HDFS ecosystem and eexperience with Big Data technology stack such as HBase, Hadoop, Hive, MapReduce.
• Experience in designing systems that process structured as well as unstructured data at large scale.
• Experience in AWS/Spark/Java/Scala/Python development.
• Should have Strong skills in PySpark (Python & SPARK). Ability to create, manage and manipulate Spark Dataframes. Expertise in Spark query tuning and performance optimization.
• Experience in developing efficient software code/frameworks for multiple use cases leveraging Python and big data technologies.
• Prior exposure to streaming data sources such as Kafka.
• Should have knowledge on Shell Scripting and Python scripting.
• High proficiency in database skills (e.g., Complex SQL), for data preparation, cleaning, and data wrangling/munging, with the ability to write advanced queries and create stored procedures.
• Experience with NoSQL databases such as Cassandra / MongoDB.
• Solid experience in all phases of Software Development Lifecycle - plan, design, develop, test, release, maintain and support, decommission.
• Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development).
• Experience building and deploying applications on on-premise and cloud-based infrastructure.
• Having a good understanding of machine learning landscape and concepts. 

Qualifications and Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Big Data Engineer or a similar role for 3-5 years.

Certifications:

Good to have at least one of the Certifications listed here:

    AZ 900 - Azure Fundamentals

    DP 200, DP 201, DP 203, AZ 204 - Data Engineering

    AZ 400 - Devops Certification

We are looking for an outstanding ML Architect (Deployments) with expertise in deploying Machine Learning solutions/models into production and scaling them to serve millions of customers. A candidate with an adaptable and productive working style which fits in a fast-moving environment.

Skills:

- 5+ years deploying Machine Learning pipelines in large enterprise production systems.

- Experience developing end to end ML solutions from business hypothesis to deployment / understanding the entirety of the ML development life cycle.
- Expert in modern software development practices; solid experience using source control management (CI/CD).
- Proficient in designing relevant architecture / microservices to fulfil application integration, model monitoring, training / re-training, model management, model deployment, model experimentation/development, alert mechanisms.
- Experience with public cloud platforms (Azure, AWS, GCP).
- Serverless services like lambda, azure functions, and/or cloud functions.
- Orchestration services like data factory, data pipeline, and/or data flow.
- Data science workbench/managed services like azure machine learning, sagemaker, and/or AI platform.
- Data warehouse services like snowflake, redshift, bigquery, azure sql dw, AWS Redshift.
- Distributed computing services like Pyspark, EMR, Databricks.
- Data storage services like cloud storage, S3, blob, S3 Glacier.
- Data visualization tools like Power BI, Tableau, Quicksight, and/or Qlik.
- Proven experience serving up predictive algorithms and analytics through batch and real-time APIs.
- Solid working experience with software engineers, data scientists, product owners, business analysts, project managers, and business stakeholders to design the holistic solution.
- Strong technical acumen around automated testing.
- Extensive background in statistical analysis and modeling (distributions, hypothesis testing, probability theory, etc.)
- Strong hands-on experience with statistical packages and ML libraries (e.g., Python scikit learn, Spark MLlib, etc.)
- Experience in effective data exploration and visualization (e.g., Excel, Power BI, Tableau, Qlik, etc.)
- Experience in developing and debugging in one or more of the languages Java, Python.
- Ability to work in cross functional teams.
- Apply Machine Learning techniques in production including, but not limited to, neuralnets, regression, decision trees, random forests, ensembles, SVM, Bayesian models, K-Means, etc.

Roles and Responsibilities:

Deploying ML models into production, and scaling them to serve millions of customers.

Technical solutioning skills with deep understanding of technical API integrations, AI / Data Science, BigData and public cloud architectures / deployments in a SaaS environment.

Strong stakeholder relationship management skills - able to influence and manage the expectations of senior executives.
Strong networking skills with the ability to build and maintain strong relationships with both business, operations and technology teams internally and externally.

Provide software design and programming support to projects.

 Qualifications & Experience:

Engineering and post graduate candidates, preferably in Computer Science, from premier institutions with proven work experience as a Machine Learning Architect (Deployments) or a similar role for 5-7 years.

• Partnering with internal business owners (product, marketing, edit, etc.) to understand needs and develop custom analysis to optimize for user engagement and retention
• Good understanding of the underlying business and workings of cross functional teams for successful execution
• Design and develop analyses based on business requirement needs and challenges.
• Leveraging statistical analysis on consumer research and data mining projects, including segmentation, clustering, factor analysis, multivariate regression, predictive modeling, etc.
• Providing statistical analysis on custom research projects and consult on A/B testing and other statistical analysis as needed. Other reports and custom analysis as required.
• Identify and use appropriate investigative and analytical technologies to interpret and verify results.
• Apply and learn a wide variety of tools and languages to achieve results
• Use best practices to develop statistical and/ or machine learning techniques to build models that address business needs.

• 2 - 4 years  of relevant experience in Data science.
• Preferred education: Bachelor's degree in a technical field or equivalent experience.
• Experience in advanced analytics, model building, statistical modeling, optimization, and machine learning algorithms.
• Machine Learning Algorithms: Crystal clear understanding, coding, implementation, error analysis, model tuning knowledge on Linear Regression, Logistic Regression, SVM, shallow Neural Networks, clustering, Decision Trees, Random forest, XGBoost, Recommender Systems, ARIMA and Anomaly Detection. Feature selection, hyper parameters tuning, model selection and error analysis, boosting and ensemble methods.
• Strong with programming languages like Python and data processing using SQL or equivalent and ability to experiment with newer open source tools.
• Experience in normalizing data to ensure it is homogeneous and consistently formatted to enable sorting, query and analysis.
• Experience designing, developing, implementing and maintaining a database and programs to manage data analysis efforts.
• Experience with big data and cloud computing viz. Spark, Hadoop (MapReduce, PIG, HIVE).
• Experience in risk and credit score domains preferred.

Company Description

At Bungee Tech, we help retailers and brands meet customers everywhere and, on every occasion, they are in. We believe that accurate, high-quality data matched with compelling market insights empowers retailers and brands to keep their customers at the center of all innovation and value they are delivering. 

We provide a clear and complete omnichannel picture of their competitive landscape to retailers and brands. We collect billions of data points every day and multiple times in a day from publicly available sources. Using high-quality extraction, we uncover detailed information on products or services, which we automatically match, and then proactively track for price, promotion, and availability. Plus, anything we do not match helps to identify a new assortment opportunity.

Empowered with this unrivalled intelligence, we unlock compelling analytics and insights that once blended with verified partner data from trusted sources such as Nielsen, paints a complete, consolidated picture of the competitive landscape.

We are looking for a Big Data Engineer who will work on the collecting, storing, processing, and analyzing of huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them.

You will also be responsible for integrating them with the architecture used in the company.

We're working on the future. If you are seeking an environment where you can drive innovation, If you want to apply state-of-the-art software technologies to solve real world problems, If you want the satisfaction of providing visible benefit to end-users in an iterative fast paced environment, this is your opportunity.

Responsibilities

As an experienced member of the team, in this role, you will:

• Contribute to evolving the technical direction of analytical Systems and play a critical role their design and development

• You will research, design and code, troubleshoot and support. What you create is also what you own.

• Develop the next generation of automation tools for monitoring and measuring data quality, with associated user interfaces.

• Be able to broaden your technical skills and work in an environment that thrives on creativity, efficient execution, and product innovation.

BASIC QUALIFICATIONS

• Bachelor’s degree or higher in an analytical area such as Computer Science, Physics, Mathematics, Statistics, Engineering or similar.
• 5+ years relevant professional experience in Data Engineering and Business Intelligence
• 5+ years in with Advanced SQL (analytical functions), ETL, Data Warehousing.
• Strong knowledge of data warehousing concepts, including data warehouse technical architectures, infrastructure components, ETL/ ELT and reporting/analytic tools and environments, data structures, data modeling and performance tuning.
• Ability to effectively communicate with both business and technical teams.
• Excellent coding skills in Java, Python, C++, or equivalent object-oriented programming language
• Understanding of relational and non-relational databases and basic SQL
• Proficiency with at least one of these scripting languages: Perl / Python / Ruby / shell script

PREFERRED QUALIFICATIONS

• Experience with building data pipelines from application databases.
• Experience with AWS services - S3, Redshift, Spectrum, EMR, Glue, Athena, ELK etc.
• Experience working with Data Lakes.
• Experience providing technical leadership and mentor other engineers for the best practices on the data engineering space
• Sharp problem solving skills and ability to resolve ambiguous requirements
• Experience on working with Big Data
• Knowledge and experience on working with Hive and the Hadoop ecosystem
• Knowledge of Spark
• Experience working with Data Science teams

Position: Big Data Engineer

What You'll Do

Punchh is seeking to hire Big Data Engineer at either a senior or tech lead level. Reporting to the Director of Big Data, he/she will play a critical role in leading Punchh’s big data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners.

This role requires close collaborations with data, engineering, and product organizations. His/her job functions include

• Work with large data sets and implement sophisticated data pipelines with both structured and structured data.
• Collaborate with stakeholders to design scalable solutions.
• Manage and optimize our internal data pipeline that supports marketing, customer success and data science to name a few.
• A technical leader of Punchh’s big data platform that supports AI and BI products.
• Work with infra and operations team to monitor and optimize existing infrastructure 
• Occasional business travels are required.

What You'll Need

• 5+ years of experience as a Big Data engineering professional, developing scalable big data solutions.
• Advanced degree in computer science, engineering or other related fields.
• Demonstrated strength in data modeling, data warehousing and SQL.
• Extensive knowledge with cloud technologies, e.g. AWS and Azure.
• Excellent software engineering background. High familiarity with software development life cycle. Familiarity with GitHub/Airflow.
• Advanced knowledge of big data technologies, such as programming language (Python, Java), relational (Postgres, mysql), NoSQL (Mongodb), Hadoop (EMR) and streaming (Kafka, Spark).
• Strong problem solving skills with demonstrated rigor in building and maintaining a complex data pipeline.
• Exceptional communication skills and ability to articulate a complex concept with thoughtful, actionable recommendations.

Role Summary/Purpose:

We are looking for a Developer/Senior Developers to be a part of building advanced analytical platform leveraging Big Data technologies and transform the legacy systems. This role is an exciting, fast-paced, constantly changing and challenging work environment, and will play an important role in resolving and influencing high-level decisions.

Requirements:

• The candidate must be a self-starter, who can work under general guidelines in a fast-spaced environment.
• Overall minimum of 4 to 8 year of software development experience and 2 years in Data Warehousing domain knowledge
• Must have 3 years of hands-on working knowledge on Big Data technologies such as Hadoop, Hive, Hbase, Spark, Kafka, Spark Streaming, SCALA etc…
• Excellent knowledge in SQL & Linux Shell scripting
• Bachelors/Master’s/Engineering Degree from a well-reputed university.
• Strong communication, Interpersonal, Learning and organizing skills matched with the ability to manage stress, Time, and People effectively
• Proven experience in co-ordination of many dependencies and multiple demanding stakeholders in a complex, large-scale deployment environment
• Ability to manage a diverse and challenging stakeholder community
• Diverse knowledge and experience of working on Agile Deliveries and Scrum teams.

Responsibilities

• Should works as a senior developer/individual contributor based on situations
• Should be part of SCRUM discussions and to take requirements
• Adhere to SCRUM timeline and deliver accordingly
• Participate in a team environment for the design, development and implementation
• Should take L3 activities on need basis
• Prepare Unit/SIT/UAT testcase and log the results
• Co-ordinate SIT and UAT Testing. Take feedbacks and provide necessary remediation/recommendation in time.
• Quality delivery and automation should be a top priority
• Co-ordinate change and deployment in time
• Should create healthy harmony within the team
• Owns interaction points with members of core team (e.g.BA team, Testing and business team) and any other relevant stakeholders

Requires a bachelor's degree in area of specialty and experience in the field or in a related area. Familiar with standard concepts, practices, and procedures within a particular field. Relies on experience and judgment to plan and accomplish goals. Performs a variety of tasks. A degree of creativity and latitude is required. Typically reports to a supervisor or manager.

Designs, develops, and implements web-based Java applications to support business requirements. Follows approved life cycle methodologies, creates design documents, and performs program coding and testing. Resolves technical issues through debugging, research, and investigation.

Additional Job Details:

Strong in Java, Spring, Spring Boot, REST and developing MicroServices.

Knowledge or experience , Cassandra preferred

Knowledge or experience on Kafka

Good to have but not must

Good to know:

Reporting tools like Splunk/Grafana

Protobuf

Python/Ruby