Data Engineer

at building a cutting-edge data science department to serve the older adult community and marketplace.

Agency job
via HyrHub
icon
Chandigarh
icon
5 - 8 yrs
icon
₹8L - ₹15L / yr
icon
Full time
Skills
Big Data
Hadoop
Apache Hive
Data Warehouse (DWH)
PySpark
Cloud Computing

We are currently seeking talented and highly motivated Data Engineers to lead in the development of our discovery and support platform. The successful candidate will join a small, global team of data focused associates that have successfully built, and maintained a best of class traditional, Kimball based, SQL server founded, data warehouse.  The successful candidate will lead the conversion of the existing data structure into an AWS focused, big data framework and assist in identifying and pipelining existing and augmented data sets into this environment. The successful candidate must be able to lead and assist in architecting and constructing the AWS foundation and initial data ports.

 

Specific responsibilities will be to:

  • Lead and assist in design, deploy, and maintain robust methods for data management and analysis, primarily using the AWS cloud
  • Develop computational methods for integrating multiple data sources to facilitate target and algorithmic
  • Provide computational tools to ensure trustworthy data sources and facilitate reproducible
  • Provide leadership around architecting, designing, and building target AWS data environment (like data lake and data warehouse).
  • Work with on staff subject-matter experts to evaluate existing data sources, DW, ETL ports, existing stove type data sources and available augmentation data sets.
  • Implement methods for execution of high-throughput assays and subsequent acquisition, management, and analysis of the
  • Assist in the communications of complex scientific, software and data concepts and
  • Assist in the identification and hiring of additional data engineer associates.

Job Requirements:

  • Master’s Degree (or equivalent experience) in computer science, data science or a scientific field that has relevance to healthcare in the United States
  • Extensive experience in the use of a high-level programming language (i.e., Python or Scala) and relevant AWS services.
  • Experience in AWS cloud services like S3, Glue, Lake Formation, Athena, and others.
  • Experience in creating and managing Data Lakes and Data Warehouses.
  • Experience with big data tools like Hadoop, Hive, Talend, Apache Spark, Kafka.
  • Advance SQL scripting.
  • Database Management Systems (for example, Oracle, MySQL or MS SQL Server)
  • Hands on experience in data transformation tools, data processing and data modeling on a big data environment.
  • Understanding the basics of distributed systems.
  • Experience working and communicating with subject matter expert
  • The ability to work independently as well as to collaborate on multidisciplinary, global teams in a startup fashion with traditional data warehouse skilled data associates and business teams unfamiliar with data science techniques
  • Strong communication, data presentation and visualization
Read more
Why apply to jobs via Cutshort
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly. No 3rd party agencies here.
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
2101133
Matches delivered
3712187
Network size
15000
Companies hiring

Similar jobs

Senior Data Engineer

at Klubworks

Founded 2019  •  Product  •  20-100 employees  •  Raised funding
Spark
Hadoop
Big Data
Data engineering
PySpark
Python
C++
icon
Bengaluru (Bangalore)
icon
4 - 8 yrs
icon
Best in industry

We are searching for an accountable, multitalented data engineer to facilitate the operations of our data scientists. The data engineer will be responsible for employing machine learning techniques to create and sustain structures that allow for the analysis of data while remaining familiar with dominant programming and deployment strategies in the field. During various aspects of this process, you should collaborate with coworkers to ensure that your approach meets the needs of each project.

To ensure success as a data engineer, you should demonstrate flexibility, creativity, and the capacity to receive and utilize constructive criticism. A formidable data engineer will demonstrate unsatiated curiosity and outstanding interpersonal skills.

Responsibilities:

  • Liaising with coworkers and clients to elucidate the requirements for each task.
  • Conceptualizing and generating infrastructure that allows big data to be accessed and analyzed.
  • Reformulating existing frameworks to optimize their functioning.
  • Testing such structures to ensure that they are fit for use.
  • Preparing raw data for manipulation by data scientists.
  • Detecting and correcting errors in your work.
  • Ensuring that your work remains backed up and readily accessible to relevant coworkers.
  • Remaining up-to-date with industry standards and technological advancements that will improve the quality of your outputs.


Requirements:

  • Bachelor's degree in data engineering, big data analytics, computer engineering, or related field.
  • Master's degree in a relevant field is advantageous.
  • Proven experience as a data engineer, software developer, or similar.
  • Expert proficiency in Python, C++, Java, R, and SQL.
  • Familiarity with Hadoop or suitable equivalent.
  • Excellent analytical and problem-solving skills.
  • A knack for independence and group work.
  • Scrupulous approach to duties.
  • Capacity to successfully manage a pipeline of duties with minimal supervision.
Read more
Job posted by
Anupam Arya

Data Engineer

at Shyftlabs

Founded 2019  •  Products & Services  •  0-20 employees  •  Bootstrapped
SQL
Snow flake schema
ETL
Python
Data-flow analysis
Big Data
icon
Hyderabad
icon
3 - 4 yrs
icon
₹8L - ₹20L / yr

We celebrate diversity, embrace a data-driven culture, and deeply encourage professional development through classes, certifications, and conferences. The reciprocity of sharing knowledge and growth with each other, our clients, and partners is a foundation we live by. Employees at Shyftlabs enjoy unlimited paid time off, 11 paid holidays, comprehensive health, vision, and dental benefits, and profit-sharing.

 

 

Key Responsibilities

 

  • Design, implement and operate stable, scalable, low-cost solutions to flow data from production systems into the data lake and into end-user-facing applications.
  • Design automated processes for in-depth analysis databases.
  • Design automated data control processes.
  • Collaborate with the software development team to build and test the designed solutions.
  • Learn, publish, analyze and improve management information dashboards, operational business metrics decks, and key performance indicators.
  • Improve tools, and processes, scale existing solutions, and create new solutions as required based on stakeholder needs.
  • Provide in-depth analysis to management with the support of accounting, finance, and transportation teams.
  • Perform monthly variance analysis and identify risks & opportunities.

 

Basic Qualifications

 

  • 3+ years of experience as a Data Engineer or in a similar role
  • Experience with data modeling, data warehousing, and building ETL pipelines
  • Experience in SQL

 

Preferred Qualifications

 

  • Degree in Computer Science, Engineering, Mathematics, or a related field and 4+ years of industry experience
  • Graduate degree in Computer Science, Engineering or related technical field
  • Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations
  • Proficiency with at least one Object Oriented language (e.g. Java, Python, Ruby)
  • Strong customer focus, ownership, urgency, and drive.
  • Excellent communication skills and the ability to work well in a team.
  • Effective analytical, troubleshooting, and problem-solving skills.
  • Experience building data products incrementally and integrating and managing datasets from multiple sources
  • Experience with AWS Tools and Technologies (Redshift, S3, EC2, Glue)
  • Expertise with Data modeling skills, Advanced SQL with Oracle, MySQL, and Columnar Databases
  • Experience with Snowflake
 
See less
 
Read more
Job posted by
rajan bharti

AGM Data Engineering

at ACT FIBERNET

Founded 2008  •  Services  •  100-1000 employees  •  Profitable
Data engineering
Data Engineer
Hadoop
Informatica
Qlikview
Datapipeline
icon
Bengaluru (Bangalore)
icon
9 - 14 yrs
icon
₹20L - ₹36L / yr

Key  Responsibilities :

  • Development of proprietary processes and procedures designed to process various data streams around critical databases in the org
  • Manage technical resources around data technologies, including relational databases, NO SQL DBs, business intelligence databases, scripting languages, big data tools and technologies, visualization tools.
  • Creation of a project plan including timelines and critical milestones to success in support of the project
  • Identification of the vital skill sets/staff required to complete the project
  • Identification of crucial sources of the data needed to achieve the objective.

 

Skill Requirement :

  • Experience with data pipeline processes and tools
  • Well versed in the Data domains (Data Warehousing, Data Governance, MDM, Data Quality, Data Catalog, Analytics, BI, Operational Data Store, Metadata, Unstructured Data, ETL, ESB)
  • Experience with an existing ETL tool e.g Informatica and Ab initio etc
  • Deep understanding of big data systems like Hadoop, Spark, YARN, Hive, Ranger, Ambari
  • Deep knowledge of Qlik ecosystems like  Qlikview, Qliksense, and Nprinting
  • Python, or a similar programming language
  • Exposure to data science and machine learning
  • Comfort working in a fast-paced environment

Soft attributes :

  • Independence: Must have the ability to work on his/her own without constant direction or supervision. He/she must be self-motivated and possess a strong work ethic to strive to put forth extra effort continually
  • Creativity: Must be able to generate imaginative, innovative solutions that meet the needs of the organization. You must be a strategic thinker/solution seller and should be able to think of integrated solutions (with field force apps, customer apps, CCT solutions etc.). Hence, it would be best to approach each unique situation/challenge in different ways using the same tools.
  • Resilience: Must remain effective in high-pressure situations, using both positive and negative outcomes as an incentive to move forward toward fulfilling commitments to achieving personal and team goals.
Read more
Job posted by
Sumit Sindhwani
ETL
Data Warehouse (DWH)
ETL Developer
Relational Database (RDBMS)
Spark
Hadoop
SQL server
SSIS
ADF
Python
Java
talend
Azure Data Factory
icon
Bengaluru (Bangalore)
icon
5 - 8 yrs
icon
₹8L - ₹13L / yr

 Minimum of 4 years’ experience of working on DW/ETL projects and expert hands-on working knowledge of ETL tools.

Experience with Data Management & data warehouse development

Star schemas, Data Vaults, RDBMS, and ODS

Change Data capture

Slowly changing dimensions

Data governance

Data quality

Partitioning and tuning

Data Stewardship

Survivorship

Fuzzy Matching

Concurrency

Vertical and horizontal scaling

ELT, ETL

Spark, Hadoop, MPP, RDBMS

Experience with Dev/OPS architecture, implementation and operation

Hand's on working knowledge of Unix/Linux

Building Complex SQL Queries. Expert SQL and data analysis skills, ability to debug and fix data issue.

Complex ETL program design coding

Experience in Shell Scripting, Batch Scripting.

Good communication (oral & written) and inter-personal skills

Expert SQL and data analysis skill, ability to debug and fix data issue Work closely with business teams to understand their business needs and participate in requirements gathering, while creating artifacts and seek business approval.

Helping business define new requirements, Participating in End user meetings to derive and define the business requirement, propose cost effective solutions for data analytics and familiarize the team with the customer needs, specifications, design targets & techniques to support task performance and delivery.

Propose good design & solutions and adherence to the best Design & Standard practices.

Review & Propose industry best tools & technology for ever changing business rules and data set. Conduct Proof of Concepts (POC) with new tools & technologies to derive convincing benchmarks.

Prepare the plan, design and document the architecture, High-Level Topology Design, Functional Design, and review the same with customer IT managers and provide detailed knowledge to the development team to familiarize them with customer requirements, specifications, design standards and techniques.

Review code developed by other programmers, mentor, guide and monitor their work ensuring adherence to programming and documentation policies.

Work with functional business analysts to ensure that application programs are functioning as defined. 

Capture user-feedback/comments on the delivered systems and document it for the client and project manager’s review. Review all deliverables before final delivery to client for quality adherence.

Technologies (Select based on requirement)

Databases - Oracle, Teradata, Postgres, SQL Server, Big Data, Snowflake, or Redshift

Tools – Talend, Informatica, SSIS, Matillion, Glue, or Azure Data Factory

Utilities for bulk loading and extracting

Languages – SQL, PL-SQL, T-SQL, Python, Java, or Scala

J/ODBC, JSON

Data Virtualization Data services development

Service Delivery - REST, Web Services

Data Virtualization Delivery – Denodo

 

ELT, ETL

Cloud certification Azure

Complex SQL Queries

 

Data Ingestion, Data Modeling (Domain), Consumption(RDMS)
Read more
Job posted by
Jerrin Thomas

AI Engineer

at StatusNeo

Founded 2020  •  Products & Services  •  100-1000 employees  •  Profitable
Artificial Intelligence (AI)
Amazon Web Services (AWS)
Windows Azure
Hadoop
Scala
Python
Google Cloud Platform (GCP)
postgres
icon
Gurugram, Hyderabad, Bengaluru (Bangalore)
icon
1 - 3 yrs
icon
₹3L - ₹12L / yr


·       Build data products and processes alongside the core engineering and technology team.

·       Collaborate with senior data scientists to curate, wrangle, and prepare data for use in their advanced analytical models

·       Integrate data from a variety of sources, assuring that they adhere to data quality and accessibility standards

·       Modify and improve data engineering processes to handle ever larger, more complex, and more types of data sources and pipelines

·       Use Hadoop architecture and HDFS commands to design and optimize data queries at scale

·       Evaluate and experiment with novel data engineering tools and advises information technology leads and partners about new capabilities to determine optimal solutions for particular technical problems or designated use cases .
Read more
Job posted by
Alex P

Data Engineer

at Intergral Add Science

Java
Hadoop
Apache Spark
Scala
Python
SQL
Data architecture
data pipeline
icon
Pune
icon
5 - 8 yrs
icon
₹9L - ₹25L / yr
  • 6+ years of recent hands-on Java development
  • Developing data pipelines in AWS or Google Cloud
  • Java, Python, JavaScript programming languages
  • Great understanding of designing for performance, scalability, and reliability of data intensive application
  • Hadoop MapReduce, Spark, Pig. Understanding of database fundamentals and advanced SQL knowledge.
  • In-depth understanding of object oriented programming concepts and design patterns
  • Ability to communicate clearly to technical and non-technical audiences, verbally and in writing
  • Understanding of full software development life cycle, agile development and continuous integration
  • Experience in Agile methodologies including Scrum and Kanban
Read more
Job posted by
Prashma S R

Data Warehouse Developer

at Business development E-commerce

Agency job
via Skyleaf Consultants
Data Warehouse (DWH)
ETL
Amazon Redshift
icon
Remote, NCR (Delhi | Gurgaon | Noida)
icon
3 - 12 yrs
icon
₹8L - ₹14L / yr

Responsible for planning, connecting, designing, scheduling, and deploying data warehouse systems. Develops, monitors, and maintains ETL processes, reporting applications, and data warehouse design.

Role and Responsibility

·         Plan, create, coordinate, and deploy data warehouses.

·         Design end user interface.

·         Create best practices for data loading and extraction.

·         Develop data architecture, data modeling, and ETFL mapping solutions within structured data warehouse environment.

·         Develop reporting applications and data warehouse consistency.

·         Facilitate requirements gathering using expert listening skills and develop unique simple solutions to meet the immediate and long-term needs of business customers.

·         Supervise design throughout implementation process.

·         Design and build cubes while performing custom scripts.

·         Develop and implement ETL routines according to the DWH design and architecture.

·         Support the development and validation required through the lifecycle of the DWH and Business Intelligence systems, maintain user connectivity, and provide adequate security for data warehouse.

·         Monitor the DWH and BI systems performance and integrity provide corrective and preventative maintenance as required.

·         Manage multiple projects at once.

DESIRABLE SKILL SET

·         Experience with technologies such as MySQL, MongoDB, SQL Server 2008, as well as with newer ones like SSIS and stored procedures

·         Exceptional experience developing codes, testing for quality assurance, administering RDBMS, and monitoring of database

·         High proficiency in dimensional modeling techniques and their applications

·         Strong analytical, consultative, and communication skills; as well as the ability to make good judgment and work with both technical and business personnel

·         Several years working experience with Tableau,  MicroStrategy, Information Builders, and other reporting and analytical tools

·         Working knowledge of SAS and R code used in data processing and modeling tasks

·         Strong experience with Hadoop, Impala, Pig, Hive, YARN, and other “big data” technologies such as AWS Redshift or Google Big Data

 

Read more
Job posted by
harpreet kaur

Technical Architect

at E-Commerce Product Based Company

Agency job
via myApps Solutions
Technical Architecture
Big Data
IT Solutioning
Python
Rest API
icon
Bengaluru (Bangalore)
icon
8 - 15 yrs
icon
₹15L - ₹30L / yr

Role and Responsibilities

  • Build a low latency serving layer that powers DataWeave's Dashboards, Reports, and Analytics functionality
  • Build robust RESTful APIs that serve data and insights to DataWeave and other products
  • Design user interaction workflows on our products and integrating them with data APIs
  • Help stabilize and scale our existing systems. Help design the next generation systems.
  • Scale our back end data and analytics pipeline to handle increasingly large amounts of data.
  • Work closely with the Head of Products and UX designers to understand the product vision and design philosophy
  • Lead/be a part of all major tech decisions. Bring in best practices. Mentor younger team members and interns.
  • Constantly think scale, think automation. Measure everything. Optimize proactively.
  • Be a tech thought leader. Add passion and vibrance to the team. Push the envelope.

 

Skills and Requirements

  • 8- 15 years of experience building and scaling APIs and web applications.
  • Experience building and managing large scale data/analytics systems.
  • Have a strong grasp of CS fundamentals and excellent problem solving abilities. Have a good understanding of software design principles and architectural best practices.
  • Be passionate about writing code and have experience coding in multiple languages, including at least one scripting language, preferably Python.
  • Be able to argue convincingly why feature X of language Y rocks/sucks, or why a certain design decision is right/wrong, and so on.
  • Be a self-starter—someone who thrives in fast paced environments with minimal ‘management’.
  • Have experience working with multiple storage and indexing technologies such as MySQL, Redis, MongoDB, Cassandra, Elastic.
  • Good knowledge (including internals) of messaging systems such as Kafka and RabbitMQ.
  • Use the command line like a pro. Be proficient in Git and other essential software development tools.
  • Working knowledge of large-scale computational models such as MapReduce and Spark is a bonus.
  • Exposure to one or more centralized logging, monitoring, and instrumentation tools, such as Kibana, Graylog, StatsD, Datadog etc.
  • Working knowledge of building websites and apps. Good understanding of integration complexities and dependencies.
  • Working knowledge linux server administration as well as the AWS ecosystem is desirable.
  • It's a huge bonus if you have some personal projects (including open source contributions) that you work on during your spare time. Show off some of your projects you have hosted on GitHub.
Read more
Job posted by
BasavRaj P S

Big Data Developer

at GeakMinds Technologies Pvt Ltd

Founded 2011  •  Services  •  100-1000 employees  •  Profitable
Hadoop
Big Data
HDFS
Apache Sqoop
Apache Flume
Apache HBase
Apache Kafka
icon
Chennai
icon
1 - 5 yrs
icon
₹1L - ₹6L / yr
• Looking for Big Data Engineer with 3+ years of experience. • Hands-on experience with MapReduce-based platforms, like Pig, Spark, Shark. • Hands-on experience with data pipeline tools like Kafka, Storm, Spark Streaming. • Store and query data with Sqoop, Hive, MySQL, HBase, Cassandra, MongoDB, Drill, Phoenix, and Presto. • Hands-on experience in managing Big Data on a cluster with HDFS and MapReduce. • Handle streaming data in real time with Kafka, Flume, Spark Streaming, Flink, and Storm. • Experience with Azure cloud, Cognitive Services, Databricks is preferred.
Read more
Job posted by
John Richardson

Technical Architect

at Nitor Infotech

Founded 2006  •  Products & Services  •  100-1000 employees  •  Profitable
Amazon Web Services (AWS)
Big Data
Business Intelligence (BI)
icon
Pune
icon
9 - 100 yrs
icon
₹13L - ₹25L / yr
The hunt is for a AWS BigData /DWH Architect with the ability to manage effective relationships with a wide range of stakeholders (customers & team members alike). Incumbent will demonstrate personal commitment and accountability to ensure standards are continuously sustained and improved both within the internal teams, and with partner organizations and suppliers. We at Nitor Infotech a Product Engineering Services company are always on hunt for some best talents in the IT industry & keeping with our trend of What next in IT. We are scouting for result oriented resources with passion for product, technology services, and creating great customer experiences. Someone who can take our current expertise & footprint of Nitor Infotech Inc., to an altogether different dimension & level in tune with the emerging market trends and ensure Brilliance @ Work continues to prevail in whatever we do. Nitor Infotech works with global ISVs to help them build and accelerate their product development. Nitor is able to do so because of the fact that product development is its DNA. This DNA is enriched by its 10 years of expertise, best practices and frameworks & Accelerators. Because of this ability Nitor Infotech has been able to build business relationships with product companies having revenues from $50 Million to $1 Billion. • 7-12+ years of relevant experience of working in Database, BI and Analytics space with over 0-2 yrs of architecting and designing data warehouse experience including 2 to 3 yrs in Big Data ecosystem • Experience in data warehouse design in AWS • Strong architecting, programming, design skills and proven track record of architecting and building large scale, distributed big data solutions • Professional and technical advice on Big Data concepts and technologies, in particular highlighting the business potential through real-time analysis • Provides technical leadership in Big Data space (Hadoop Stack like M/R, HDFS, Pig, Hive, HBase, Flume, Sqoop, etc. NoSQL stores like Mongodb, Cassandra, HBase etc.) • Performance tuning of Hadoop clusters and Hadoop MapReduce routines. • Evaluate and recommend Big Data technology stack for the platform • Drive significant technology initiatives end to end and across multiple layers of architecture • Should have breadth of BI knowledge which includes:  MSBI, Database design, new visualization tools like Tableau, Qlik View, Power BI  Understand internals and intricacies of Old and New DB platform which includes:  Strong RDMS DB Fundamentals either of it SQL Server/ MySQL/ Oracle  DB and DWH design  Designing Semantic Model using OLAP and Tabular model using MS and Non MS tools  No SQL DBs including Document, Graph, Search and Columnar DBs • Excellent communication skills and strong ability to build good rapport with prospect and existing customers • Be a Mentor and go to person for Jr. team members in the team Qualification & Experience: · Educational qualification: BE/ME/B.Tech/M.Tech, BCA/MCA/BCS/MCS, any other degree with relevant IT qualification.
Read more
Job posted by
Balakumar Mohan
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
Get to hear about interesting companies hiring right now
iconFollow Cutshort
Want to apply to this role at building a cutting-edge data science department to serve the older adult community and marketplace.?
Why apply via Cutshort?
Connect with actual hiring teams and get their fast response. No spam.
Learn more
Get to hear about interesting companies hiring right now
iconFollow Cutshort