Tech Lead - Web Scraping and Crawling

Our client provides data solutions for fraud prevention

Tech Lead - Web Scraping and Crawling

Tech Lead - Web Scraping and Crawling at Our client provides data solutions for fraud prevention · Remote only · 4 - 10 years · ₹10L - ₹20L / yr · Remote only · Posted 24 Dec 2020

Tech Lead - Web Scraping and Crawling

at Our client provides data solutions for fraud prevention

Agency job

via Purple Squirrel Consulting Services

4 - 10 yrs

₹10L - ₹20L / yr

Remote only

Skills

Web Scraping

Web crawling

Selenium

Selenium Web driver

Beautiful Soup

Data Science

web crawler

Javascript

Java

Python

HTML/CSS

Manage individual projects priorities, deadlines, and deliverables
Gather and process raw data at scale (including writing scripts, web scraping, calling/create

APIs, etc.) from the web / internet
Develop frameworks for automating and maintaining constant flow of data from multiple

sources
Identify, analysis, design, and implement internal process improvements
Design and implement tooling upgrades to increase stability and data quality
Help team to fix issues that occur in test and production environments
Automate software development processes, including build, deploy, and test
Manage and guide the team members

REQUIRED QUALIFICATIONS:

 4+ years of web crawling/ scraping experience is a must
 Strong knowledge of scraping frameworks such as Scrapy, Beautiful Soup, HTQL, Jsoup, Web-

Harvest and others
Excellent verbal, written, and interpersonal communication skills in English
 Good to have Experience of complex crawling (like captcha, Mobile OTP based crawling,

bypassing proxy)
 Sound Knowledge in Bot Management Techniques
 Experience in various data extraction methods (like data extraction from PDF Files, web

pages, etc)
 Good understanding of HTML DOM, CSS, Javascript, and RESTful web services

 Good to have understanding of AWS
 Experience with Linux
 Experience with Java / Python

Manage individual projects priorities, deadlines, and deliverables
Gather and process raw data at scale (including writing scripts, web scraping, calling/create

APIs, etc.) from the web / internet
Develop frameworks for automating and maintaining constant flow of data from multiple

sources
Identify, analysis, design, and implement internal process improvements
Design and implement tooling upgrades to increase stability and data quality
Help team to fix issues that occur in test and production environments
Automate software development processes, including build, deploy, and test
Manage and guide the team members

REQUIRED QUALIFICATIONS:

 4+ years of web crawling/ scraping experience is a must
 Strong knowledge of scraping frameworks such as Scrapy, Beautiful Soup, HTQL, Jsoup, Web-

Harvest and others
Excellent verbal, written, and interpersonal communication skills in English
 Good to have Experience of complex crawling (like captcha, Mobile OTP based crawling,

bypassing proxy)
 Sound Knowledge in Bot Management Techniques
 Experience in various data extraction methods (like data extraction from PDF Files, web

pages, etc)
 Good understanding of HTML DOM, CSS, Javascript, and RESTful web services

 Good to have understanding of AWS
 Experience with Linux
 Experience with Java / Python

Users love Cutshort

Read about what our users have to say about finding their next opportunity on Cutshort.

Shubham Vishwakarma

Full Stack Developer - Averlon

I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.