Big Data Engineer

Position Overview: We are seeking a talented Data Engineer with expertise in Power BI to join our team. The ideal candidate will be responsible for designing and implementing data pipelines, as well as developing insightful visualizations and reports using Power BI. Additionally, the candidate should have strong skills in Python, data analytics, PySpark, and Databricks. This role requires a blend of technical expertise, analytical thinking, and effective communication skills.

Key Responsibilities:

1. Design, develop, and maintain data pipelines and architectures using PySpark and Databricks.
2. Implement ETL processes to extract, transform, and load data from various sources into data warehouses or data lakes.
3. Collaborate with data analysts and business stakeholders to understand data requirements and translate them into actionable insights.
4. Develop interactive dashboards, reports, and visualizations using Power BI to communicate key metrics and trends.
5. Optimize and tune data pipelines for performance, scalability, and reliability.
6. Monitor and troubleshoot data infrastructure to ensure data quality, integrity, and availability.
7. Implement security measures and best practices to protect sensitive data.
8. Stay updated with emerging technologies and best practices in data engineering and data visualization.
9. Document processes, workflows, and configurations to maintain a comprehensive knowledge base.

Requirements:

1. Bachelor’s degree in Computer Science, Engineering, or related field. (Master’s degree preferred)
2. Proven experience as a Data Engineer with expertise in Power BI, Python, PySpark, and Databricks.
3. Strong proficiency in Power BI, including data modeling, DAX calculations, and creating interactive reports and dashboards.
4. Solid understanding of data analytics concepts and techniques.
5. Experience working with Big Data technologies such as Hadoop, Spark, or Kafka.
6. Proficiency in programming languages such as Python and SQL.
7. Hands-on experience with cloud platforms like AWS, Azure, or Google Cloud.
8. Excellent analytical and problem-solving skills with attention to detail.
9. Strong communication and collaboration skills to work effectively with cross-functional teams.
10. Ability to work independently and manage multiple tasks simultaneously in a fast-paced environment.

Preferred Qualifications:

• Advanced degree in Computer Science, Engineering, or related field.
• Certifications in Power BI or related technologies.
• Experience with data visualization tools other than Power BI (e.g., Tableau, QlikView).
• Knowledge of machine learning concepts and frameworks.

Job Responsibilities

• Design machine learning systems
• Research and implement appropriate ML algorithms and tools
• Develop machine learning applications according to requirements
• Select appropriate datasets and data representation methods
• Run machine learning tests and experiments
• Perform statistical analysis and fine-tuning using test results
• Train and retrain systems when necessary

Requirements for the Job

1. Bachelor’s/Master's/PhD in Computer Science, Mathematics, Statistics or equivalent field andmust have a minimum of 2 years of overall experience in tier one colleges 

1. Minimum 1 year of experience working as a Data Scientist in deploying ML at scale in production
2. Experience in machine learning techniques (e.g. NLP, Computer Vision, BERT, LSTM etc..) andframeworks (e.g. TensorFlow, PyTorch, Scikit-learn, etc.)

1. Working knowledge in deployment of Python systems (using Flask, Tensorflow Serving)
2. Previous experience in following areas will be preferred: Natural Language Processing(NLP) - Using LSTM and BERT; chatbots or dialogue systems, machine translation, comprehension of text, text summarization.
3. Computer Vision - Deep Neural Networks/CNNs for object detection and image classification, transfer learning pipeline and object detection/instance segmentation (Mask R-CNN, Yolo, SSD).

• 5+ years of experience in a Data Engineering role on cloud environment

• Must have good experience in Scala/PySpark (preferably on data-bricks environment)

• Extensive experience with Transact-SQL.
• Experience in Data-bricks/Spark.

• Strong experience in Dataware house projects
• Expertise in database development projects with ETL processes.
• Manage and maintain data engineering pipelines

• Develop batch processing, streaming and integration solutions
• Experienced in building and operationalizing large-scale enterprise data solutions and applications

• Using one or more of Azure data and analytics services in combination with custom solutions
• Azure Data Lake, Azure SQL DW (Synapse), and SQL Database products or equivalent products from other cloud services providers

• In-depth understanding of data management (e. g. permissions, security, and monitoring).
• Cloud repositories for e.g. Azure GitHub, Git
• Experience in an agile environment (Prefer Azure DevOps).

Good to have

• Manage source data access security
• Automate Azure Data Factory pipelines
• Continuous Integration/Continuous deployment (CICD) pipelines, Source Repositories
• Experience in implementing and maintaining CICD pipelines
• Power BI understanding, Delta Lake house architecture
• Knowledge of software development best practices.
• Excellent analytical and organization skills.
• Effective working in a team as well as working independently.
• Strong written and verbal communication skills.
• Expertise in database development projects and ETL processes.

JD for IOT DE:

The role requires experience in Azure core technologies – IoT Hub/ Event Hub, Stream Analytics, IoT Central, Azure Data Lake Storage, Azure Cosmos, Azure Data Factory, Azure SQL Database, Azure HDInsight / Databricks, SQL data warehouse.

You Have:

• Minimum 2 years of software development experience
• Minimum 2 years of experience in IoT/streaming data pipelines solution development
• Bachelor's and/or Master’s degree in computer science
• Strong Consulting skills in data management including data governance, data quality, security, data integration, processing, and provisioning
• Delivered data management projects with real-time/near real-time data insights delivery on Azure Cloud
• Translated complex analytical requirements into the technical design including data models, ETLs, and Dashboards / Reports
• Experience deploying dashboards and self-service analytics solutions on both relational and non-relational databases
• Experience with different computing paradigms in databases such as In-Memory, Distributed, Massively Parallel Processing
• Successfully delivered large scale IOT data management initiatives covering Plan, Design, Build and Deploy phases leveraging different delivery methodologies including Agile
• Experience in handling telemetry data with Spark Streaming, Kafka, Flink, Scala, Pyspark, Spark SQL.
• Hands-on experience on containers and Dockers
• Exposure to streaming protocols like MQTT and AMQP
• Knowledge of OT network protocols like OPC UA, CAN Bus, and similar protocols
• Strong knowledge of continuous integration, static code analysis, and test-driven development
• Experience in delivering projects in a highly collaborative delivery model with teams at onsite and offshore
• Must have excellent analytical and problem-solving skills
• Delivered change management initiatives focused on driving data platforms adoption across the enterprise
• Strong verbal and written communications skills are a must, as well as the ability to work effectively across internal and external organizations
   

Roles & Responsibilities
 

You Will:

• Translate functional requirements into technical design
• Interact with clients and internal stakeholders to understand the data and platform requirements in detail and determine core Azure services needed to fulfill the technical design
• Design, Develop and Deliver data integration interfaces in ADF and Azure Databricks
• Design, Develop and Deliver data provisioning interfaces to fulfill consumption needs
• Deliver data models on Azure platform, it could be on Azure Cosmos, SQL DW / Synapse, or SQL
• Advise clients on ML Engineering and deploying ML Ops at Scale on AKS
• Automate core activities to minimize the delivery lead times and improve the overall quality
• Optimize platform cost by selecting the right platform services and architecting the solution in a cost-effective manner
• Deploy Azure DevOps and CI CD processes
• Deploy logging and monitoring across the different integration points for critical alerts

• Building and operationalizing large scale enterprise data solutions and applications using one or more of AZURE data and analytics services in combination with custom solutions - Azure Synapse/Azure SQL DWH, Azure Data Lake, Azure Blob Storage, Spark, HDInsights, Databricks, CosmosDB, EventHub/IOTHub.
• Experience in migrating on-premise data warehouses to data platforms on AZURE cloud. 
• Designing and implementing data engineering, ingestion, and transformation functions

• Experience with Azure Analysis Services
• Experience in Power BI
• Experience with third-party solutions like Attunity/Stream sets, Informatica
• Experience with PreSales activities (Responding to RFPs, Executing Quick POCs)
• Capacity Planning and Performance Tuning on Azure Stack and Spark.

Role Summary/Purpose:

We are looking for a Developer/Senior Developers to be a part of building advanced analytical platform leveraging Big Data technologies and transform the legacy systems. This role is an exciting, fast-paced, constantly changing and challenging work environment, and will play an important role in resolving and influencing high-level decisions.

Requirements:

• The candidate must be a self-starter, who can work under general guidelines in a fast-spaced environment.
• Overall minimum of 4 to 8 year of software development experience and 2 years in Data Warehousing domain knowledge
• Must have 3 years of hands-on working knowledge on Big Data technologies such as Hadoop, Hive, Hbase, Spark, Kafka, Spark Streaming, SCALA etc…
• Excellent knowledge in SQL & Linux Shell scripting
• Bachelors/Master’s/Engineering Degree from a well-reputed university.
• Strong communication, Interpersonal, Learning and organizing skills matched with the ability to manage stress, Time, and People effectively
• Proven experience in co-ordination of many dependencies and multiple demanding stakeholders in a complex, large-scale deployment environment
• Ability to manage a diverse and challenging stakeholder community
• Diverse knowledge and experience of working on Agile Deliveries and Scrum teams.

Responsibilities

• Should works as a senior developer/individual contributor based on situations
• Should be part of SCRUM discussions and to take requirements
• Adhere to SCRUM timeline and deliver accordingly
• Participate in a team environment for the design, development and implementation
• Should take L3 activities on need basis
• Prepare Unit/SIT/UAT testcase and log the results
• Co-ordinate SIT and UAT Testing. Take feedbacks and provide necessary remediation/recommendation in time.
• Quality delivery and automation should be a top priority
• Co-ordinate change and deployment in time
• Should create healthy harmony within the team
• Owns interaction points with members of core team (e.g.BA team, Testing and business team) and any other relevant stakeholders

About the Role: 

Punchh Tech India Pvt. is looking for a Senior Data Analyst – Business Insights to join our team. If you're excited to be part of a winning team, Punchh is a great place to grow your career.

This position is responsible for discovering the important trends among the complex data generated on Punchh platform, that have high business impact (influencing product features and roadmap). Creating hypotheses around these trends, validate them with statistical significance and make recommendations

Reporting to: Director, Analytics

Job Location: Jaipur

Experience Required: 4-6 years

What You’ll Do

• Take ownership of custom data analysis projects/requests and work closely with end users (both internal and external clients) to deliver the results
• Identify successful implementation/utilization of product features and contribute to the best-practices playbook for client facing teams (Customer Success)
• Strive towards building mini business intelligence products that add value to the client base
• Represent the company’s expertise in advanced analytics in a variety of media outlets such as client interactions, conferences, blogs, and interviews.

What You’ll Need

• Masters in business/behavioral economics/statistics with a strong interest in marketing technology
• Proven track record of at least 5 years uncovering business insights, especially related to Behavioral Economics and adding value to businesses
• Proficient in using the proper statistical and econometric approaches to establish the presence and strength of trends in data. Strong statistical knowledge is mandatory.
• Extensive prior exposure in causal inference studies, based on both longitudinal and latitudinal data.
• Excellent experience using Python (or R) to analyze data from extremely large or complex data sets
• Exceptional data querying skills (Snowflake/Redshift, Spark, Presto/Athena, to name a few)
• Ability to effectively articulate complex ideas in simple and effective presentations to diverse groups of stakeholders.
• Experience working with a visualization tool (preferably, but not restricted to Tableau)
• Domain expertise: extensive exposure to retail business, restaurant business or worked on loyalty programs and promotion/campaign effectiveness
• Should be self-organized and be able to proactively identify problems and propose solutions
• Gels well within and across teams, work with stakeholders from various functions such as Product, Customer Success, Implementations among others
• As the stakeholders on business side are based out of US, should be flexible to schedule meetings convenient to the West Coast timings
• Effective in working autonomously to get things done and taking the initiatives to anticipate needs of executive leadership
• Able and willing to relocate to Jaipur post pandemic.

Benefits:

• Medical Coverage, to keep you and your family healthy.
• Compensation that stacks up with other tech companies in your area.
• Paid vacation days and holidays to rest and relax.
• Healthy lunch provided daily to fuel you through your work.
• Opportunities for career growth and training support, including fun team building events.
• Flexibility and a comfortable work environment for you to feel your best.