Big Data Developer

Responsibilities:

• Build customer facing solution for Data Observability product to monitor Data Pipelines

• Work on POCs to build new data pipeline monitoring capabilities.

• Building next-generation scalable, reliable, flexible, high-performance data pipeline capabilities for ingestion of data from multiple sources containing complex dataset.

•Continuously improve services you own, making them more performant, and utilising resources in the most optimised way.

• Collaborate closely with engineering, data science team and product team to propose an optimal solution for a given problem statement

• Working closely with DevOps team on performance monitoring and MLOps

Required Skills:

• 3+ Years of Data related technology experience.

• Good understanding of distributed computing principles

• Experience in Apache Spark

•  Hands on programming with Python

• Knowledge of Hadoop v2, Map Reduce, HDFS

• Experience with building stream-processing systems, using technologies such as Apache Storm, Spark-Streaming or Flink

• Experience with messaging systems, such as Kafka or RabbitMQ

• Good understanding of Big Data querying tools, such as Hive

• Experience with integration of data from multiple data sources

• Good understanding of SQL queries, joins, stored procedures, relational schemas

• Experience with NoSQL databases, such as HBase, Cassandra/Scylla, MongoDB

• Knowledge of ETL techniques and frameworks

• Performance tuning of Spark Jobs

• General understanding of Data Quality is a plus point

• Experience on Databricks,snowflake and BigQuery or similar lake houses would be a big plus

• Nice to have some knowledge in DevOps

 Job DescriptionPosition: Sr Data Engineer – Databricks & AWS

Experience: 4 - 5 Years

Company Profile:

Exponentia.ai is an AI tech organization with a presence across India, Singapore, the Middle East, and the UK. We are an innovative and disruptive organization, working on cutting-edge technology to help our clients transform into the enterprises of the future. We provide artificial intelligence-based products/platforms capable of automated cognitive decision-making to improve productivity, quality, and economics of the underlying business processes. Currently, we are transforming ourselves and rapidly expanding our business.

Exponentia.ai has developed long-term relationships with world-class clients such as PayPal, PayU, SBI Group, HDFC Life, Kotak Securities, Wockhardt and Adani Group amongst others.

One of the top partners of Cloudera (leading analytics player) and Qlik (leader in BI technologies), Exponentia.ai has recently been awarded the ‘Innovation Partner Award’ by Qlik in 2017.

Get to know more about us on our website: http://www.exponentia.ai/ and Life @Exponentia.

​Role Overview: 

·         A Data Engineer understands the client requirements and develops and delivers the data engineering solutions as per the scope.

·         The role requires good skills in the development of solutions using various services required for data architecture on Databricks Delta Lake, streaming, AWS, ETL Development, and data modeling.

Job Responsibilities

•         Design of data solutions on Databricks including delta lake, data warehouse, data marts and other data solutions to support the analytics needs of the organization.

•         Apply best practices during design in data modeling (logical, physical) and ETL pipelines (streaming and batch) using cloud-based services.

•         Design, develop and manage the pipelining (collection, storage, access), data engineering (data quality, ETL, Data Modelling) and understanding (documentation, exploration) of the data.

•         Interact with stakeholders regarding data landscape understanding, conducting discovery exercises, developing proof of concepts and demonstrating it to stakeholders.

Technical Skills 

•         Has more than 2 Years of experience in developing data lakes, and datamarts on the Databricks platform.

•         Proven skill sets in AWS Data Lake services such as - AWS Glue, S3, Lambda, SNS, IAM, and skills in Spark, Python, and SQL.

•         Experience in Pentaho

•         Good understanding of developing a data warehouse, data marts etc.

•         Has a good understanding of system architectures, and design patterns and should be able to design and develop applications using these principles.

Personality Traits

•         Good collaboration and communication skills

•         Excellent problem-solving skills to be able to structure the right analytical solutions.

•         Strong sense of teamwork, ownership, and accountability

•         Analytical and conceptual thinking 

•         Ability to work in a fast-paced environment with tight schedules.

•         Good presentation skills with the ability to convey complex ideas to peers and management.

Education:

BE / ME / MS/MCA.

Key  Responsibilities :

• Development of proprietary processes and procedures designed to process various data streams around critical databases in the org
• Manage technical resources around data technologies, including relational databases, NO SQL DBs, business intelligence databases, scripting languages, big data tools and technologies, visualization tools.
• Creation of a project plan including timelines and critical milestones to success in support of the project
• Identification of the vital skill sets/staff required to complete the project
• Identification of crucial sources of the data needed to achieve the objective.

Skill Requirement :

• Experience with data pipeline processes and tools
• Well versed in the Data domains (Data Warehousing, Data Governance, MDM, Data Quality, Data Catalog, Analytics, BI, Operational Data Store, Metadata, Unstructured Data, ETL, ESB)
• Experience with an existing ETL tool e.g Informatica and Ab initio etc
• Deep understanding of big data systems like Hadoop, Spark, YARN, Hive, Ranger, Ambari
• Deep knowledge of Qlik ecosystems like  Qlikview, Qliksense, and Nprinting
• Python, or a similar programming language
• Exposure to data science and machine learning
• Comfort working in a fast-paced environment

Soft attributes :

• Independence: Must have the ability to work on his/her own without constant direction or supervision. He/she must be self-motivated and possess a strong work ethic to strive to put forth extra effort continually
• Creativity: Must be able to generate imaginative, innovative solutions that meet the needs of the organization. You must be a strategic thinker/solution seller and should be able to think of integrated solutions (with field force apps, customer apps, CCT solutions etc.). Hence, it would be best to approach each unique situation/challenge in different ways using the same tools.
• Resilience: Must remain effective in high-pressure situations, using both positive and negative outcomes as an incentive to move forward toward fulfilling commitments to achieving personal and team goals.

As AWS Data Engineer, you are a full-stack data engineer that loves solving business problems. You work with business leads, analysts, and data scientists to understand the business domain and engage with fellow engineers to build data products that empower better decision-making. You are passionate about the data quality of our business metrics and the flexibility of your solution that scales to respond to broader business questions. 

If you love to solve problems using your skills, then come join the Team Mactores. We have a casual and fun office environment that actively steers clear of rigid "corporate" culture, focuses on productivity and creativity, and allows you to be part of a world-class team while still being yourself.

What you will do?

• Write efficient code in - PySpark, Amazon Glue
• Write SQL Queries in - Amazon Athena, Amazon Redshift
• Explore new technologies and learn new techniques to solve business problems creatively
• Collaborate with many teams - engineering and business, to build better data products and services 
• Deliver the projects along with the team collaboratively and manage updates to customers on time

What are we looking for?

• 1 to 3 years of experience in Apache Spark, PySpark, Amazon Glue
• 2+ years of experience in writing ETL jobs using pySpark, and SparkSQL
• 2+ years of experience in SQL queries and stored procedures
• Have a deep understanding of all the Dataframe API with all the transformation functions supported by Spark 2.7+

You will be preferred if you have

• Prior experience in working on AWS EMR, Apache Airflow
• Certifications AWS Certified Big Data – Specialty OR Cloudera Certified Big Data Engineer OR Hortonworks Certified Big Data Engineer
• Understanding of DataOps Engineering

Life at Mactores

We care about creating a culture that makes a real difference in the lives of every Mactorian. Our 10 Core Leadership Principles that honor Decision-making, Leadership, Collaboration, and Curiosity drive how we work.

1. Be one step ahead

2. Deliver the best

3. Be bold

4. Pay attention to the detail

5. Enjoy the challenge

6. Be curious and take action

7. Take leadership

8. Own it

9. Deliver value

10. Be collaborative

We would like you to read more details about the work culture on https://mactores.com/careers 

The Path to Joining the Mactores Team

At Mactores, our recruitment process is structured around three distinct stages:

Pre-Employment Assessment: 

You will be invited to participate in a series of pre-employment evaluations to assess your technical proficiency and suitability for the role.

Managerial Interview: The hiring manager will engage with you in multiple discussions, lasting anywhere from 30 minutes to an hour, to assess your technical skills, hands-on experience, leadership potential, and communication abilities.

HR Discussion: During this 30-minute session, you'll have the opportunity to discuss the offer and next steps with a member of the HR team.

At Mactores, we are committed to providing equal opportunities in all of our employment practices, and we do not discriminate based on race, religion, gender, national origin, age, disability, marital status, military status, genetic information, or any other category protected by federal, state, and local laws. This policy extends to all aspects of the employment relationship, including recruitment, compensation, promotions, transfers, disciplinary action, layoff, training, and social and recreational programs. All employment decisions will be made in compliance with these principles.

Deep-Rooted.Co is on a mission to get Fresh, Clean, Community (Local farmer) produce from harvest to reach your home with a promise of quality first! Our values are rooted in trust, convenience, and dependability, with a bunch of learning & fun thrown in.

Founded out of Bangalore by Arvind, Avinash, Guru and Santosh, with the support of our Investors Accel, Omnivore & Mayfield, we raised $7.5 million in Seed, Series A and Debt funding till date from investors include ACCEL, Omnivore, Mayfield among others. Our brand Deep-Rooted.Co which was launched in August 2020 was the first of its kind as India’s Fruits & Vegetables (F&V) which is present in Bangalore & Hyderabad and on a journey of expansion to newer cities which will be managed seamlessly through Tech platform that has been designed and built to transform the Agri-Tech sector.

Deep-Rooted.Co is committed to building a diverse and inclusive workplace and is an equal-opportunity employer.  

How is this possible? It’s because we work with smart people. We are looking for Engineers in Bangalore to work with thehttps://www.linkedin.com/in/gururajsrao/"> Product Leader (Founder) andhttps://www.linkedin.com/in/sriki77/"> CTO and this is a meaningful project for us and we are sure you will love the project as it touches everyday life and is fun. This will be a virtual consultation.

We want to start the conversation about the project we have for you, but before that, we want to connect with you to know what’s on your mind. Do drop a note sharing your mobile number and letting us know when we can catch up.

Purpose of the role:

* As a startup we have data distributed all across various sources like Excel, Google Sheets, Databases etc. We need swift decision making based a on a lot of data that exists as we grow. You help us bring together all this data and put it in a data model that can be used in business decision making.
* Handle nuances of Excel and Google Sheets API.
* Pull data in and manage it growth, freshness and correctness.
* Transform data in a format that aids easy decision-making for Product, Marketing and Business Heads.
* Understand the business problem, solve the same using the technology and take it to production - no hand offs - full path to production is yours.

Technical expertise:

* Good Knowledge And Experience with Programming languages - Java, SQL,Python.
* Good Knowledge of Data Warehousing, Data Architecture.
* Experience with Data Transformations and ETL; 
* Experience with API tools and more closed systems like Excel, Google Sheets etc.
* Experience AWS Cloud Platform and Lambda
* Experience with distributed data processing tools.
* Experiences with container-based deployments on cloud.

 Skills:

Java, SQL, Python, Data Build Tool, Lambda, HTTP, Rest API, Extract Transform Load.

Dear Candidate,,        

Greetings of the day!

As discussed, Please find the below job description.

Job Title           :            Hadoop developer

Experience      :            3+ years

Job Location   :            New Delhi

Job type           :            Permanent

Knowledge and Skills Required:

Brief Skills:

Hadoop, Spark, Scala and Spark SQL

Main Skills:

1. Strong experience in Hadoop development
2. Experience in Spark
3. Experience in Scala
4. Experience in Spark SQL

Why OTSi!

Working with OTSi gives you the assurance of a successful, fast-paced career.

Exposure to infinite opportunities to learn and grow, familiarization with cutting-edge technologies, cross-domain experience and a harmonious environment are some of the prime attractions for a career-driven workforce.

Join us today, as we assure you 2000+ friends and a great career; Happiness begins at a great workplace..!

Feel free to refer this opportunity to your friends and associates.

About OTSI: (CMMI Level 3): Founded in 1999 and headquartered in Overland Park, Kansas, OTSI offers global reach and local delivery to companies of all sizes, from start-ups to Fortune 500s. Through offices across the US and around the world, we provide universal access to exceptional talent and innovative solutions in a variety of delivery models to reduce overall risk while optimizing outcomes & enabling our customers to thrive in a global economy.http://otsi-usa.com/?page_id=2806">
OTSI's global presence, scalable and sustainable world-class infrastructure, business continuity processes, ISO 9001:2000, CMMI 3 certifications makes us a preferred service provider for our clients. OTSI has the expertise in different technologies enhanced by our http://otsi-usa.com/?page_id=2933">partnerships and alliances with industry giants like HP, Microsoft, IBM, Oracle, and SAP and others. Highly repetitive local company with a proven success of serving the UAE Government IT needs is seeking to attract, employ and develop people with exceptional skills who want to make a difference in a challenging environment.Object Technology Solutions India Pvt Ltd is a leading Global Information Technology (IT) Services and Solutions company offering a wide array of Solutions for a range of key Verticals. The company is headquartered in Overland Park, Kansas, and has a strong presence in US, Europe and Asia-Pacific with a Global Delivery Center based in India. OTSI offers a broad range of IT application solutions and services including; e-Business solutions, Enterprise Resource Planning (ERP) implementation and Post Implementation Support, Application development, Application Maintenance, Software customizations services.

OTSI Partners & Practices

• SAP Partner
• Microsoft Silver Partner
• Oracle Gold Partner
• Microsoft CoE
• DevOps Consulting
• Cloud
• Mobile & IoT
• Digital Transformation
• Big data & Analytics
• Testing Solutions

OTSI Honor’s & Awards:

• #91 in Inc.5000 .
• Fastest growing IT Companies in Inc.5000…

Intro

Our data and risk team is the core pillar of our business that harnesses alternative data sources to guide the decisions we make at Rely. The team designs, architects, as well as develop and maintain a scalable data platform the powers our machine learning models. Be part of a team that will help millions of consumers across Asia, to be effortlessly in control of their spending and make better decisions.

What will you do
The data engineer is focused on making data correct and accessible, and building scalable systems to access/process it. Another major responsibility is helping AI/ML Engineers write better code.

• Optimize and automate ingestion processes for a variety of data sources such as: click stream, transactional and many other sources.

• Create and maintain optimal data pipeline architecture and ETL processes
• Assemble large, complex data sets that meet functional / non-functional business requirements.
• Develop data pipeline and infrastructure to support real-time decisions
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS big data' technologies.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs.
  
  

What will you need
• 2+ hands-on experience building and implementation of large scale production pipeline and Data Warehouse
• Experience dealing with large scale

• Proficiency in writing and debugging complex SQLs
• Experience working with AWS big data tools
  • Ability to lead the project and implement best data practises and technology

Data Pipelining

• Strong command in building & optimizing data pipelines, architectures and data sets
• Strong command on relational SQL & noSQL databases including Postgres
• Data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.

Big Data: Strong experience in big data tools & applications

• Tools: Hadoop, Spark, HDFS etc
• AWS cloud services: EC2, EMR, RDS, Redshift
• Stream-processing systems: Storm, Spark-Streaming, Flink etc.
• Message queuing: RabbitMQ, Spark etc
  
  

Software Development & Debugging

• Strong experience in object-oriented programming/object function scripting languages: Python, Java, C++, Scala, etc
• Strong hold on data structures & algorithms

What would be a bonus

• Prior experience working in a fast-growth Startup
• Prior experience in the payments, fraud, lending, advertising companies dealing with large scale data