BigData Developer (Spark+Python)

Job Title

Data Analyst

Job Brief

The successful candidate will turn data into information, information into insight and insight into business decisions.

Data Analyst Job Duties

Data analyst responsibilities include conducting full lifecycle analysis to include requirements, activities and design. Data analysts will develop analysis and reporting capabilities. They will also monitor performance and quality control plans to identify improvements.

Responsibilities

● Interpret data, analyze results using statistical techniques and provide ongoing reports.

● Develop and implement databases, data collection systems, data analytics and other strategies that optimize statistical efficiency and quality.

● Acquire data fromprimary orsecondary data sources andmaintain databases/data systems.

● Identify, analyze, and interpret trends orpatternsin complex data sets.

● Filter and “clean” data by reviewing computerreports, printouts, and performance indicatorsto locate and correct code problems.

● Work withmanagementto prioritize business and information needs.

● Locate and define new processimprovement opportunities. 

Requirements

● Proven working experienceas aData Analyst or BusinessDataAnalyst.

● Technical expertise regarding data models, database design development, data mining and segmentation techniques.

● Strong knowledge of and experience with reporting packages (Business Objects etc), databases (SQL etc), programming (XML, Javascript, or ETL frameworks).

● Knowledge of statistics and experience using statistical packages for analyzing datasets (Excel, SPSS, SAS etc).

● Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.

● Adept atqueries,reportwriting and presenting findings.

Job Location SouthDelhi, New Delhi 

Carsome’s Data Department is on the lookout for a Data Scientist/Senior Data Scientist who has a strong passion in building data powered products.

Data Science function under the Data Department has a responsibility for standardisation of methods, mentoring team of data science resources/interns, including code libraries and documentation, quality assurance of outputs, modeling techniques and statistics, leveraging a variety of technologies, open-source languages, and cloud computing platform. 

You will get to lead & implement projects such as price optimization/prediction, enabling iconic personalization experiences for our customer, inventory optimization etc.

Job Descriptions

• Identifying and integrating datasets that can be leveraged through our product and work closely with data engineering team to develop data products.
• Execute analytical experiments methodically to help solve various problems and make a true impact across functions such as operations, finance, logistics, marketing. 
• Identify, prioritize, and design testing opportunities that will inform algorithm enhancements. 
• Devise and utilize algorithms and models to mine big data stores, perform data and error analysis to improve models and clean and validate data for uniformity and accuracy.
• Unlock insights by analyzing large amounts of complex website traffic and transactional data. 
• Implement analytical models into production by collaborating with data analytics engineers.

Technical Requirements

• Expertise in model design, training, evaluation, and implementation ML Algorithm expertise K-nearest neighbors, Random Forests, Naive Bayes, Regression Models. PyTorch, TensorFlow, Keras, deep learning expertise, tSNE, gradient boosting expertise, regression implementation expertise, Python, Pyspark, SQL, R, AWS Sagemaker /personalize etc.
• Machine Learning / Data Science Certification

Experience & Education 

• Bachelor’s in Engineering / Master’s in Data Science  / Postgraduate Certificate in Data Science. 

• Creating and managing ETL/ELT pipelines based on requirements
• Build PowerBI dashboards and manage datasets needed.
• Work with stakeholders to identify data structures needed for future and perform any transformations including aggregations.
• Build data cubes for real-time visualisation needs and CXO dashboards.

Required Tech Skills

• Microsoft PowerBI & DAX
• Python, Pandas, PyArrow, Jupyter Noteboks, ApacheSpark
• Azure Synapse, Azure DataBricks, Azure HDInsight, Azure Data Factory

You will be responsible for designing, building, and maintaining data pipelines that handle Real-world data at Compile. You will be handling both inbound and outbound data deliveries at Compile for datasets including Claims, Remittances, EHR, SDOH, etc.

You will

• Work on building and maintaining data pipelines (specifically RWD).
• Build, enhance and maintain existing pipelines in pyspark, python and help build analytical insights and datasets.
• Scheduling and maintaining pipeline jobs for RWD.
• Develop, test, and implement data solutions based on the design.
• Design and implement quality checks on existing and new data pipelines.
• Ensure adherence to security and compliance that is required for the products.
• Maintain relationships with various data vendors and track changes and issues across vendors and deliveries.

You have

• Hands-on experience with ETL process (min of 5 years).
• Excellent communication skills and ability to work with multiple vendors.
• High proficiency with Spark, SQL.
• Proficiency in Data modeling, validation, quality check, and data engineering concepts.
• Experience in working with big-data processing technologies using - databricks, dbt, S3, Delta lake, Deequ, Griffin, Snowflake, BigQuery.
• Familiarity with version control technologies, and CI/CD systems.
• Understanding of scheduling tools like Airflow/Prefect.
• Min of 3 years of experience managing data warehouses.
• Familiarity with healthcare datasets is a plus.

Compile embraces diversity and equal opportunity in a serious way. We are committed to building a team of people from many backgrounds, perspectives, and skills. We know the more inclusive we are, the better our work will be.         

About the role:

 Hopscotch is looking for a passionate Data Engineer to join our team. You will work closely with other teams like data analytics, marketing, data science and individual product teams to specify, validate, prototype, scale, and deploy data pipelines features and data architecture.

Here’s what will be expected out of you:

➢ Ability to work in a fast-paced startup mindset. Should be able to manage all aspects of data extraction transfer and load activities.

➢ Develop data pipelines that make data available across platforms.

➢ Should be comfortable in executing ETL (Extract, Transform and Load) processes which include data ingestion, data cleaning and curation into a data warehouse, database, or data platform.

➢ Work on various aspects of the AI/ML ecosystem – data modeling, data and ML pipelines.

➢ Work closely with Devops and senior Architect to come up with scalable system and model architectures for enabling real-time and batch services.

What we want:

➢ 5+ years of experience as a data engineer or data scientist with a focus on data engineering and ETL jobs.

➢ Well versed with the concept of Data warehousing, Data Modelling and/or Data Analysis.

➢ Experience using & building pipelines and performing ETL with industry-standard best practices on Redshift (more than 2+ years).

➢ Ability to troubleshoot and solve performance issues with data ingestion, data processing & query execution on Redshift.

➢ Good understanding of orchestration tools like Airflow.

 ➢ Strong Python and SQL coding skills.

➢ Strong Experience in distributed systems like spark.

➢ Experience with AWS Data and ML Technologies (AWS Glue,MWAA, Data Pipeline,EMR,Athena, Redshift,Lambda etc).

➢ Solid hands on with various data extraction techniques like CDC or Time/batch based and the related tools (Debezium, AWS DMS, Kafka Connect, etc) for near real time and batch data extraction.

Note :

Product based companies, Ecommerce companies is added advantage

Datametica is looking for talented SQL engineers who would get training & the opportunity to work on Cloud and Big Data Analytics.

Mandatory Skills:

• Strong in SQL development
• Hands-on at least one scripting language - preferably shell scripting
• Development experience in Data warehouse projects

Opportunities:

• Selected candidates will be provided training opportunities on one or more of the following: Google Cloud, AWS, DevOps Tools, Big Data technologies like Hadoop, Pig, Hive, Spark, Sqoop, Flume, and KafkaWould get a chance to be part of the enterprise-grade implementation of Cloud and Big Data systems
• Will play an active role in setting up the Modern data platform based on Cloud and Big Data
• Would be part of teams with rich experience in various aspects of distributed systems and computing

About Turing:

Turing enables U.S. companies to hire the world’s best remote software engineers. 100+ companies including those backed by Sequoia, Andreessen, Google Ventures, Benchmark, Founders Fund, Kleiner, Lightspeed, and Bessemer have hired Turing engineers. For more than 180,000 engineers across 140 countries, we are the preferred platform for finding remote U.S. software engineering roles. We offer a wide range of full-time remote opportunities for full-stack, backend, frontend, DevOps, mobile, and AI/ML engineers.

We are growing fast (our revenue 15x’d in the past 12 months and is accelerating), and we have raised $14M in seed funding (https://tcrn.ch/3lNKbM9">one of the largest in Silicon Valley) from:

• Facebook’s 1st CTO and Quora’s Co-Founder (Adam D’Angelo)
• Executives from Google, Facebook, Square, Amazon, and Twitter
• Foundation Capital (investors in Uber, Netflix, Chegg, Lending Club, etc.)
• Cyan Banister
• Founder of Upwork (Beerud Sheth)

We also raised a much larger round of funding in October 2020 that we will be publicly announcing over the coming month. 

Some articles about Turing:

• https://techcrunch.com/2020/08/25/turing-raises-14m-to-help-source-vet-place-and-manage-remote-developers-in-tech-jobs/">TechCrunch: Turing raises $14M seed to help source, vet, place, and manage remote developers
• https://www.theinformation.com/articles/six-startups-prospering-during-coronavirus">The Information: Six Startups Prospering During Coronavirus
• https://medium.com/@cyanbanister/turing-helps-the-world-level-up-ff44b4e6415d">Cyan Banister: Turing Helps the World Level Up
• https://turing.com/boundarylessblog/2019/10/the-future-of-work-is-remote/the-future-of-work/">Jonathan Siddharth (Turing CEO): The Future of Work is Remote.

Turing is led by successful repeat founders Jonathan Siddharth and Vijay Krishnan, whose last A.I. company leveraged elite remote talent and had a successful acquisition. (https://techcrunch.com/2017/02/23/revcontent-acquires-rover/">Techcrunch story). Turing’s leadership team is composed of ex-Engineering and Sales leadership from Facebook, Google, Uber, and Capgemini.

About the role:

Software developers from all over the world have taken 200,000+ tests and interviews on Turing. Turing has also recommended thousands of developers to its customers and got customer feedback in terms of customer interview pass/fail data and data from the success of the collaboration with a U.S customer. This generates a massive proprietary dataset with a rich feature set comprising resume and test/interview features and labels in the form of actual customer feedback. Continuing rapid growth in our business creates an ever-increasing data advantage for us. 

We are looking for a Machine Learning Scientist who can help solve a whole range of exciting and valuable machine learning problems at Turing. Turing collects a lot of valuable heterogeneous signals about software developers including their resume, GitHub profile and associated code and a lot of fine-grained signals from Turing’s own screening tests and interviews (that span various areas including Computer Science fundamentals, project ownership and collaboration, communication skills, proactivity and tech stack skills), their history of successful collaboration with different companies on Turing, etc. 

A machine learning scientist at Turing will help create deep developer profiles that are a good representation of a developer’s strengths and weaknesses as it relates to their probability of getting successfully matched to one of Turing’s partner companies and having a fruitful long-term collaboration. The ML scientist will build models that are able to rank developers for different jobs based on their probability of success at the job. 

     
 You will also help make Turing’s tests more efficient by assessing their ability to predict the probability of a successful match of a developer with at least one company. The prior probability of a registered developer getting matched with a customer is about 1%. We want our tests to adaptively reduce perplexity as steeply as possible and move this probability estimate rapidly toward either 0% or 100%; maximize expected information-gain per unit time in other words. 

As an ML Scientist on the team, you will have a unique opportunity to make an impact by advancing ML models and systems, as well as uncovering new opportunities to apply machine learning concepts to Turing product(s).

This role will directly report to Turing’s founder and CTO, https://www.linkedin.com/in/vijay0/">Vijay Krishnan. This is his https://scholar.google.com/citations?user=uCRc7DgAAAAJ&;hl=en">Google Scholar profile. 

Responsibilities:

• Enhance our existing machine learning systems using your core coding skills and ML knowledge.
• Take end to end ownership of machine learning systems - from data pipelines, feature engineering, candidate extraction, model training, as well as integration into our production systems.
• Utilize state-of-the-art ML modeling techniques to predict user interactions and the direct impact on the company’s top-line metrics.
• Design features and builds large scale recommendation systems to improve targeting and engagement.
• Identify new opportunities to apply machine learning to different parts of our product(s) to drive value for our customers.

Minimum Requirements:

• BS, MS, or Ph.D. in Computer Science or a relevant technical field (AI/ML preferred).
• Extensive experience building scalable machine learning systems and data-driven products working with cross-functional teams
• Expertise in machine learning fundamentals, applicable to search - Learning to Rank, Deep Learning, Tree-Based Models, Recommendation Systems, Relevance and Data mining, understanding of NLP approaches like W2V or Bert.
• 2+ years of experience applying machine learning methods in settings like recommender systems, search, user modeling, graph representation learning, natural language processing.
• Strong understanding of neural network/deep learning, feature engineering, feature selection, optimization algorithms. Proven ability to dig deep into practical problems and choose the right ML method to solve them.
• Strong programming skills in Python and fluency in data manipulation (SQL, Spark, Pandas) and machine learning (scikit-learn, XGBoost, Keras/Tensorflow) tools.
• Good understanding of mathematical foundations of machine learning algorithms.
• Ability to be available for meetings and communication during Turing's "coordination hours" (Mon - Fri: 8 am to 12 pm PST).

Other Nice-to-have Requirements:

• First author publications in ICML, ICLR, NeurIPS, KDD, SIGIR, and related conferences/journals. 
• Strong performance in Kaggle competitions.
• 5+ years of industry experience or a Ph.D. with 3+ years of industry experience in applied machine learning in similar problems e.g. ranking, recommendation, ads, etc.
• Strong communication skills.
• Experienced in leading large-scale multi-engineering projects.
• Flexible, and a positive team player with outstanding interpersonal skills.