Data Engineer

About the role:

 Hopscotch is looking for a passionate Data Engineer to join our team. You will work closely with other teams like data analytics, marketing, data science and individual product teams to specify, validate, prototype, scale, and deploy data pipelines features and data architecture.

Here’s what will be expected out of you:

➢ Ability to work in a fast-paced startup mindset. Should be able to manage all aspects of data extraction transfer and load activities.

➢ Develop data pipelines that make data available across platforms.

➢ Should be comfortable in executing ETL (Extract, Transform and Load) processes which include data ingestion, data cleaning and curation into a data warehouse, database, or data platform.

➢ Work on various aspects of the AI/ML ecosystem – data modeling, data and ML pipelines.

➢ Work closely with Devops and senior Architect to come up with scalable system and model architectures for enabling real-time and batch services.

What we want:

➢ 5+ years of experience as a data engineer or data scientist with a focus on data engineering and ETL jobs.

➢ Well versed with the concept of Data warehousing, Data Modelling and/or Data Analysis.

➢ Experience using & building pipelines and performing ETL with industry-standard best practices on Redshift (more than 2+ years).

➢ Ability to troubleshoot and solve performance issues with data ingestion, data processing & query execution on Redshift.

➢ Good understanding of orchestration tools like Airflow.

 ➢ Strong Python and SQL coding skills.

➢ Strong Experience in distributed systems like spark.

➢ Experience with AWS Data and ML Technologies (AWS Glue,MWAA, Data Pipeline,EMR,Athena, Redshift,Lambda etc).

➢ Solid hands on with various data extraction techniques like CDC or Time/batch based and the related tools (Debezium, AWS DMS, Kafka Connect, etc) for near real time and batch data extraction.

Note :

Product based companies, Ecommerce companies is added advantage

Toplyne is a global SaaS product built to help revenue teams, at businesses with a self-service motion, and a large user-base, identify which users to spend time on, when and for what outcome. Think self-service or freemium-led companies like Figma, Notion, Freshworks, and Slack. We do this by helping companies recognize signals across their - product engagement, sales, billing, and marketing data.

Founded in June 2021, Toplyne is backed by marquee investors like Sequoia,Together fund and a bunch of well known angels. You can read more about us on - https://bit.ly/ForbesToplyne" target="_blank">https://bit.ly/ForbesToplyne , https://bit.ly/YourstoryToplyne" target="_blank">https://bit.ly/YourstoryToplyne.

• Design, Develop and maintain scalable data pipelines and Data warehouse to support continuing increases in data volume and complexity.

• Develop and implement processes and systems to supervise data quality, data mining and ensuring production data is always accurate and available for key partners and business processes that depend on it.

• Perform data analysis required to solve data related issues and assist in the resolution of data issues.

• Complete ownership - You’ll build highly scalable platforms and services that support rapidly growing data needs in Toplyne. There’s no instruction book, it’s yours to write. You’ll figure it out, ship it, and iterate.

• 3-6 years of relevant work experience in a Data Engineering role.

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.

• Experience building and optimising data pipelines, architectures and data sets.

• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.

• Strong analytic skills related to working with unstructured datasets.

• Good understanding of Airflow, Spark, NoSql databases, Kakfa is nice to have.

We are an early stage start-up, building new fintech products for small businesses. Founders are IIT-IIM alumni, with prior experience across management consulting, venture capital and fintech startups. We are driven by the vision to empower small business owners with technology and dramatically improve their access to financial services. To start with, we are building a simple, yet powerful solution to address a deep pain point for these owners: cash flow management. Over time, we will also add digital banking and 1-click financing to our suite of offerings.

We have developed an MVP which is being tested in the market. We have closed our seed funding from marquee global investors and are now actively building a world class tech team. We are a young, passionate team with a strong grip on this space and are looking to on-board enthusiastic, entrepreneurial individuals to partner with us in this exciting journey. We offer a high degree of autonomy, a collaborative fast-paced work environment and most importantly, a chance to create unparalleled impact using technology.

Reach out if you want to get in on the ground floor of something which can turbocharge SME banking in India!

Technology stack at Velocity comprises a wide variety of cutting edge technologies like, NodeJS, Ruby on Rails, Reactive Programming,, Kubernetes, AWS, NodeJS, Python, ReactJS, Redux (Saga) Redis, Lambda etc. 

Key Responsibilities

• Responsible for building data and analytical engineering pipelines with standard ELT patterns, implementing data compaction pipelines, data modelling and overseeing overall data quality

• Work with the Office of the CTO as an active member of our architecture guild

• Writing pipelines to consume the data from multiple sources

• Writing a data transformation layer using DBT to transform millions of data into data warehouses.

• Implement Data warehouse entities with common re-usable data model designs with automation and data quality capabilities

• Identify downstream implications of data loads/migration (e.g., data quality, regulatory)

What To Bring

• 5+ years of software development experience, a startup experience is a plus.

• Past experience of working with Airflow and DBT is preferred

• 5+ years of experience working in any backend programming language. 

• Strong first-hand experience with data pipelines and relational databases such as Oracle, Postgres, SQL Server or MySQL

• Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development)

• Experienced with the formulation of ideas; building proof-of-concept (POC) and converting them to production-ready projects

• Experience building and deploying applications on on-premise and AWS or Google Cloud cloud-based infrastructure

• Basic understanding of Kubernetes & docker is a must.

• Experience in data processing (ETL, ELT) and/or cloud-based platforms

• Working proficiency and communication skills in verbal and written English.

As a Data Engineer, your role will encompass: 

• Designing and building production data pipelines from ingestion to consumption within a hybrid big data architecture using Scala, Python, Talend etc.   
• Gather and address technical and design requirements.  
• Refactor existing applications to optimize its performance through setting the appropriate architecture and integrating the best practices and standards. 
• Participate in the entire data life-cycle mainly focusing on coding, debugging, and testing. 

• Troubleshoot and debug ETL Pipelines. 
• Documentation of each process. 

Technical Requirements: - 

• BSc degree in Computer Science/Computer Engineering. (Masters is a plus.) 
• 2+ years of experience as a Data Engineer. 

• In-depth understanding of core ETL concepts, Data Modelling, Data Lineage, Data Governance, Data Catalog, etc. 
• 2+ years of work experience in Scala, Python, Java. 
• Good Knowledge on Big Data Tools such as Spark/HDFS/Hive/Flume, etc. 
• Hands on experience on ETL tools like Talend/Informatica is a plus. 
• Good knowledge in Kafka and spark streaming is a big plus. 

• 2+ years of experience in using Azure cloud and its resources/services (like Azure Data factory, Azure Databricks, SQL Synapse, Azure Devops, Logic Apps, Power Bi, Azure Event Hubs, etc). 
• Strong experience in Relational Databases (MySQL, SQL Server)  
• Exposure on data visualization tools like Power BI / Qlik sense / MicroStrategy 
• 2+ years of experience in developing APIs (REST & SOAP protocols). 
• Strong knowledge in Continuous Integration & Continuous Deployment (CI/CD) utilizing Docker containers, Jenkins, etc. 

• Strong competencies in algorithms and software architecture. 
• Excellent analytical and teamwork skills. 

 Good to have: - 

• Previous on-prem working experience is a plus. 
• In-depth understanding of the entire web development process (design, development, and deployment) 

• Previous experience in automated testing including unit testing & UI testing. 

What are we looking for:

1. Strong experience in MySQL and writing advanced queries
2. Strong experience in Bash and Python
3. Familiarity with ElasticSearch, Redis, Java, NodeJS, ClickHouse, S3
4. Exposure to cloud services such as AWS, Azure, or GCP
5. 2+ years of experience in the production support
6. Strong experience in log management and performance monitoring like ELK, Prometheus + Grafana, logging services on various cloud platforms
7. Strong understanding of Linux OSes like Ubuntu, CentOS / Redhat Linux
8. Interest in learning new languages / framework as needed
9. Good written and oral communications skills
10. A growth mindset and passionate about building things from the ground up, and most importantly, you should be fun to work with

As a product solutions engineer, you will:

1. Analyze recorded runtime issues, diagnose and do occasional code fixes of low to medium complexity
2. Work with developers to find and correct more complex issues
3. Address urgent issues quickly, work within and measure against customer SLAs
4. Using shell and python scripts, and use scripting to actively automate manual / repetitive activities
5. Build anomaly detectors wherever applicable
6. Pass articulated feedback from customers to the development and product team
7. Maintain ongoing record of the operation of problem analysis and resolution in a on call monitoring system
8. Offer technical support needed in development

Required Python ,R

work in handling large-scale data engineering pipelines.
Excellent verbal and written communication skills.
Proficient in PowerPoint or other presentation tools.
Ability to work quickly and accurately on multiple projects.

Graphene is an award-winning AI company, developing customized insights and data solutions for corporate clients. With a focus on healthcare, consumer goods and financial services, our proprietary AI platform is disrupting market research with an approach that allows us to get into the mind of customers to a degree unprecedented in traditional market research.

Graphene was founded by corporate leaders from Microsoft and P&G and works closely with the Singapore Government & universities in creating cutting edge technology. We are gaining traction with many Fortune 500 companies globally.

Graphene has a 6-year track record of delivering financially sustainable growth and is one of the few start-ups which are self-funded, yet profitable and debt free.

We already have a strong bench strength of leaders in place. Now, we are looking to groom more talents for our expansion into the US. Join us and take both our growths to the next level!

• Primary Purpose: As part of a highly productive and creative AI (NLP) analytics team, optimize algorithms/models for performance and scalability, engineer & implement machine learning algorithms into services and pipelines to be consumed at web-scale
• Daily Grind: Interface with data scientists, project managers, and the engineering team to achieve sprint goals on the product roadmap, and ensure healthy models, endpoints, CI/CD,
• Career Progression: Senior ML Engineer, ML Architect

• Work in a product-development team capable of independently authoring software products.
• Guide junior programmers, set up the architecture, and follow modular development approaches.
• Design and develop code which is well documented.
• Optimize of the application for maximum speed and scalability
• Adhere to the best Information security and Devops practices.
• Research and develop new approaches to problems.
• Design and implement schemas and databases with respect to the AI application
• Cross-pollinated with other teams.

HARD AND SOFT SKILLS

Must Have

• Problem-solving abilities
• Extremely strong programming background – data structures and algorithm
• Advanced Machine Learning: TensorFlow, Keras
• Python, spaCy, NLTK, Word2Vec, Graph databases, Knowledge-graph, BERT (derived models), Hyperparameter tuning
• Experience with OOPs and design patterns
• Exposure to RDBMS/NoSQL
• Test Driven Development Methodology

Good to Have

• Working in cloud-native environments (preferably Azure)
• Microservices
• Enterprise Design Patterns
• Microservices Architecture
• Distributed Systems

• Total Experience of 7-10 years and should be interested in teaching and research
• 3+ years’ experience in data engineering which includes data ingestion, preparation, provisioning, automated testing, and quality checks.
• 3+ Hands-on experience in Big Data cloud platforms like AWS and GCP, Data Lakes and Data Warehouses
• 3+ years of Big Data and Analytics Technologies. Experience in SQL, writing code in spark engine using python, scala or java Language. Experience in Spark, Scala
• Experience in designing, building, and maintaining ETL systems
• Experience in data pipeline and workflow management tools like Airflow
• Application Development background along with knowledge of Analytics libraries, opensource Natural Language Processing, statistical and big data computing libraries
• Familiarity with Visualization and Reporting Tools like Tableau, Kibana.
• Should be good at storytelling in Technology