Data Engineer

Data Engineer 

  

Mandatory Requirements  

• Experience in AWS Glue 

• Experience in Apache Parquet  
• Proficient in AWS S3 and data lake  
• Knowledge of Snowflake 
• Understanding of file-based ingestion best practices. 
• Scripting language - Python & pyspark 

  

CORE RESPONSIBILITIES 

• Create and manage cloud resources in AWS  
• Data ingestion from different data sources which exposes data using different technologies, such as: RDBMS, REST HTTP API, flat files, Streams, and Time series data based on various proprietary systems. Implement data ingestion and processing with the help of Big Data technologies  
• Data processing/transformation using various technologies such as Spark and Cloud Services. You will need to understand your part of business logic and implement it using the language supported by the base data platform  

• Develop automated data quality check to make sure right data enters the platform and verifying the results of the calculations  
• Develop an infrastructure to collect, transform, combine and publish/distribute customer data. 
• Define process improvement opportunities to optimize data collection, insights and displays. 
• Ensure data and results are accessible, scalable, efficient, accurate, complete and flexible  
• Identify and interpret trends and patterns from complex data sets  

• Construct a framework utilizing data visualization tools and techniques to present consolidated analytical and actionable results to relevant stakeholders.  
• Key participant in regular Scrum ceremonies with the agile teams   
• Proficient at developing queries, writing reports and presenting findings  
• Mentor junior members and bring best industry practices  

  

QUALIFICATIONS 

• 5-7+ years’ experience as data engineer in consumer finance or equivalent industry (consumer loans, collections, servicing, optional product, and insurance sales)  
• Strong background in math, statistics, computer science, data science or related discipline 
• Advanced knowledge one of language: Java, Scala, Python, C#  
• Production experience with: HDFS, YARN, Hive, Spark, Kafka, Oozie / Airflow, Amazon Web Services (AWS), Docker / Kubernetes, Snowflake   

• Proficient with 
• Data mining/programming tools (e.g. SAS, SQL, R, Python) 
• Database technologies (e.g. PostgreSQL, Redshift, Snowflake. and Greenplum) 
• Data visualization (e.g. Tableau, Looker, MicroStrategy) 
• Comfortable learning about and deploying new technologies and tools.  

• Organizational skills and the ability to handle multiple projects and priorities simultaneously and meet established deadlines.  
• Good written and oral communication skills and ability to present results to non-technical audiences  
• Knowledge of business intelligence and analytical tools, technologies and techniques. 

  

Familiarity and experience in the following is a plus:  

• AWS certification 
• Spark Streaming  
• Kafka Streaming / Kafka Connect  
• ELK Stack  
• Cassandra / MongoDB  

• CI/CD: Jenkins, GitLab, Jira, Confluence other related tools 

Key Skills
o Convert machine learning models into APIs for applications accessibility
o Running machine learning tests and experiments
o Implementing appropriate ML algorithms
o Creating machine learning models and retraining systems
o Study and transform data science prototypes
o Design machine learning systems
o Research and implement appropriate ML algorithms and tools
o Train and retrain systems when necessary
o Test and deploy models
o Use AI to empower the company with novel capabilities
o Designing and developing machine learning and deep learning system
o Outstanding analytical and problem-solving skills
• Alexa
o Excellent in Python programming
o Experience with AWS Lamda
o Experience with Alexa skills
o Alexa skill directives
• Google
o Excellent in NodeJS programming
o Experience with GCP - Dialog Flow and Actions on Google
o Using built-in intents and developing custom intents
o API integration and Postman knowledge

We are an early stage start-up, building new fintech products for small businesses. Founders are IIT-IIM alumni, with prior experience across management consulting, venture capital and fintech startups. We are driven by the vision to empower small business owners with technology and dramatically improve their access to financial services. To start with, we are building a simple, yet powerful solution to address a deep pain point for these owners: cash flow management. Over time, we will also add digital banking and 1-click financing to our suite of offerings.

 

We have developed an MVP which is being tested in the market. We have closed our seed funding from marquee global investors and are now actively building a world class tech team. We are a young, passionate team with a strong grip on this space and are looking to on-board enthusiastic, entrepreneurial individuals to partner with us in this exciting journey. We offer a high degree of autonomy, a collaborative fast-paced work environment and most importantly, a chance to create unparalleled impact using technology.

 

Reach out if you want to get in on the ground floor of something which can turbocharge SME banking in India!

 

Technology stack at Velocity comprises a wide variety of cutting edge technologies like, NodeJS, Ruby on Rails, Reactive Programming,, Kubernetes, AWS, NodeJS, Python, ReactJS, Redux (Saga) Redis, Lambda etc. 

 

Key Responsibilities

• Responsible for building data and analytical engineering pipelines with standard ELT patterns, implementing data compaction pipelines, data modelling and overseeing overall data quality

• Work with the Office of the CTO as an active member of our architecture guild

• Writing pipelines to consume the data from multiple sources

• Writing a data transformation layer using DBT to transform millions of data into data warehouses.

• Implement Data warehouse entities with common re-usable data model designs with automation and data quality capabilities

• Identify downstream implications of data loads/migration (e.g., data quality, regulatory)

 

What To Bring

• 5+ years of software development experience, a startup experience is a plus.

• Past experience of working with Airflow and DBT is preferred

• 5+ years of experience working in any backend programming language. 

• Strong first-hand experience with data pipelines and relational databases such as Oracle, Postgres, SQL Server or MySQL

• Experience with DevOps tools (GitHub, Travis CI, and JIRA) and methodologies (Lean, Agile, Scrum, Test Driven Development)

• Experienced with the formulation of ideas; building proof-of-concept (POC) and converting them to production-ready projects

• Experience building and deploying applications on on-premise and AWS or Google Cloud cloud-based infrastructure

• Basic understanding of Kubernetes & docker is a must.

• Experience in data processing (ETL, ELT) and/or cloud-based platforms

• Working proficiency and communication skills in verbal and written English.

 

Dear All,
we are looking for candidates who have good experiance with
BI/DW Experience of 3 - 6 years with Spark, Scala, SQL expertise
and Azure.
Azure background is needed.
     * Spark hands on : Must have
     * Scala hands on : Must have
     * SQL expertise : Expert
     * Azure background : Must have
     * Python hands on : Good to have
     * ADF, Data Bricks: Good to have
     * Should be able to communicate effectively and deliver technology
implementation end to end
Looking for candidates who can join 15 to 30 Days and who will avaailable immeiate.


Regards
Gayatri P
Fragma Data Systems

We are a nascent quant hedge fund; we need to stage financial data and make it easy to run and re-run various preprocessing and ML jobs on the data.
- We are looking for an experienced data engineer to join our team.
- The preprocessing involves ETL tasks, using pyspark, AWS Glue, staging data in parquet formats on S3, and Athena

To succeed in this data engineering position, you should care about well-documented, testable code and data integrity. We have devops who can help with AWS permissions.
We would like to build up a consistent data lake with staged, ready-to-use data, and to build up various scripts that will serve as blueprints for various additional data ingestion and transforms.

If you enjoy setting up something which many others will rely on, and have the relevant ETL expertise, we’d like to work with you.

Responsibilities
- Analyze and organize raw data
- Build data pipelines
- Prepare data for predictive modeling
- Explore ways to enhance data quality and reliability
- Potentially, collaborate with data scientists to support various experiments

Requirements
- Previous experience as a data engineer with the above technologies

About Artivatic :Artivatic is technology startup that uses AI/ML/Deeplearning to build intelligent products & solutions for finance, healthcare & insurance businesses. It is based out of Bangalore with 20+ team focus on technology. Artivatic building is cutting edge solutions to enable 750 Millions plus people to get insurance, financial access and health benefits with alternative data sources to increase their productivity, efficiency, automation power and profitability, hence improving their way of doing business more intelligently & seamlessly. Artivatic offers lending underwriting, credit/insurance underwriting, fraud, prediction, personalization, recommendation, risk profiling, consumer profiling intelligence, KYC Automation & Compliance, automated decisions, monitoring, claims processing, sentiment/psychology behaviour, auto insurance claims, travel insurance, disease prediction for insurance and more. We have raised US $300K earlier and built products successfully and also done few PoCs successfully with some top enterprises in Insurance, Banking & Health sector. Currently, 4 months away from generating continuous revenue.Skills : - We at artivatic are seeking for passionate, talented and research focused computer engineer with strong machine learning and computer vision background to help build industry-leading technology with a focus in document text extraction and parsing using OCR across different languages.Qualifications :- Bachelors or Master degree in Computer Science, Computer vision or related field with specialization in Image Processing or machine learning.- Research experience in Deep Learning models for Image processing or OCR related field is preferred.- Publication record in Deep Learning models for Computer Vision conferences/journals is a plus.Required Skills :- Excellent skills developing in Python in Linux environment. Programming skills with multi-threaded GPU Cuda computing and API Solutions.- Experience applying machine learning and computer vision principles to real-world data and working in Scanned and Documented Images.- Good knowledge of Computer Science, math and statistics fundamentals (algorithms and data structures, meshing, sampling theory, linear algebra, etc.)- Knowledge of data science technologies such as Python, Pandas, Scipy, Numpy, matplotlib, etc.- Broad Computer Vision knowledge - Construction, Feature Detection, Segmentation, Classification; Machine/Deep Learning - Algorithm Evaluation, Preparation, Analysis, Modeling and Execution.- Familiarity with OpenCV, Dlib, Yolo, Capslule Network or similar and Open Source AR platforms and products- Strong problem solving and logical skills.- A go-getter kind of attitude with a willingness to learn new technologies.- Well versed in software design paradigms and good development practices.Responsibilities :- Developing novel algorithms and modeling techniques to advance the state of the art in- Document and Text Extraction.- Image recognition, Object Identification and Visual Recognition - Working closely with R&D and Machine Learning engineers implementing algorithms that power user and developer-facing products.- Be responsible for measuring and optimizing the quality of your algorithmsExperience : 3 Years+ Location : Sony World Signal, Koramangala 4th Block, Bangalore

WyngCommerce is building state of the art AI software for the Global Consumer Brands & Retailers to enable best-in-class customer experiences. Our vision is to democratize machine learning algorithms for our customers and help them realize dramatic improvements in speed, cost and flexibility. Backed by a clutch of prominent angel investors & having some of the category leaders in the retail industry as clients, we are looking to hire for our data science team. The data science team at WyngCommerce is on a mission to challenge the norms and re-imagine how retail business should be run across the world. As a Junior Data Scientist in the team, you will be driving and owning the thought leadership and impact on one of our core data science problems. You will work collaboratively with the founders, clients and engineering team to formulate complex problems, run Exploratory Data Analysis and test hypotheses, implement ML-based solutions and fine tune them with more data. This is a high impact role with goals that directly impact our business. Your Role & Responsibilities: - Implement data-driven solutions based on advanced ML and optimization algorithms to address business problems - Research, experiment, and innovate ML/statistical approaches in various application areas of interest and contribute to IP - Partner with engineering teams to build scalable, efficient, automated ML-based pipelines (training/evaluation/monitoring) - Deploy, maintain, and debug ML/decision models in production environment - Analyze and assess data to ensure high data quality and correctness of downstream processes - Communicate results to stakeholders and present data/insights to participate in and drive decision making Desired Skills & Experiences: - Bachelors or Masters in a quantitative field from a top tier college - 1-2 years experience in a data science / analytics role in a technology / analytics company - Solid mathematical background (especially in linear algebra & probability theory) - Familiarity with theoretical aspects of common ML techniques (generalized linear models, ensembles, SVMs, clustering algos, graphical models, etc.), statistical tests/metrics, experiment design, and evaluation methodologies - Demonstrable track record of dealing with ambiguity, prioritizing needs, bias for iterative learning, and delivering results in a dynamic environment with minimal guidance - Hands-on experience in at least one of the following: (a) Anomaly Detection, (b) Time Series Analysis, (c) Product Clustering, (d) Demand Forecasting, (e) Intertemporal Optimization - Good programming skills (fluent in Java/Python/SQL) with experience of using common ML toolkits (e.g., sklearn, tensor flow, keras, nltk) to build models for real world problems - Computational thinking and familiarity with practical application requirements (e.g., latency, memory, processing time) - Excellent written and verbal communication skills for both technical and non-technical audiences - (Plus Point) Experience of applying ML / other techniques in the domain of supply chain - and particularly in retail - for inventory optimization, demand forecasting, assortment planning, and other such problems - (Nice to have) Research experience and publications in top ML/Data science conferences

WyngCommerce is building a Global Enterprise AI Platform for top tier brands and retailers to drive profitability for our clients. Our vision is to develop a self-learning retail backend that enables our clients to become more agile and responsive to demand and supply volatilities. We are looking for a Business Analyst to join our team. As a BA, you will take end-to-end ownership of on-boarding new clients, running proof-of-concepts and pilots with them on different AI product applications, and ensuring timely product roll-out and customer success for the clients. You will also be expected to drive significant inputs to the sales, engineering, data science and product team to help us build for scale. An eye for detail, ability to process and analyze data quickly, and communicating effectively with different teams (Client, Sales and Engineering) are the qualities we are looking for. There will be opportunities to grow up within the same job family (lead a team of analysts) or to move to other areas of business like customer success, data science or product management. KEY RESPONSIBILITIES: - Understand the client deliverables from the sales team and come back to them with the timelines of solution / product delivery - Coordinate with relevant stakeholders within the client team to configure the WyngCommerce platform to their business, set-up processes for regular data sharing - Drive relevant anomaly detection analyses and work on data pre-processing to prepare the data for the WyngCommerce Analytics engine - Drive rigorous testing of results from the WyngCommerce engine and apply manual overrides, wherever required before pushing the results to the client - Evaluate business outcomes of different engagements with the client and prepare the analysis for the business benefits to the clients KEY REQUIREMENTS: - 0-2 years of experience in analytics role (preferably client facing which required you to interface with multiple stakeholders) - Hands-on experience in data analysis (descriptive), visualization and data pre-processing - Hands-on experience in python, especially in data processing and visualization libraries like pandas, numpy, matplotlib, seaborn - Good understanding of statistical and predictive modeling concepts (you not need be completely hands-on) - Excellent analytical thinking, and problem solving skills - Experience in project management and handling client communications - Excellent communication (written/verbal) skills, including logically structuring and delivering presentations

The Candidate should be have: - good understanding of Statistical concepts - worked on Data Analysis and Model building for 1 year - ability to implement Data warehouse and Visualisation tools (IBM, Amazon or Tableau) - use of ETL tools - understanding of scoring models The candidate will be required: - to build models for approval or rejection of loans - build various reports (standard for monthly reporting) to optimise business - implement datawarehosue The candidate should be self-starter as well as work without supervision. You will be the 1st and only employee for this role for the next 6 months.