Web Scraping Data Engineer at TransForm Solution · Surat · 3 - 5 years · ₹3L - ₹7L / yr · Profitable · Posted 7 Jul 2022
Javascript
Python
HTML/CSS- Design, build web crawlers to scrape data and URLs.
- Integrate the data crawled and scraped into our databases
- Create more/better ways to crawl relevant information
- Strong knowledge of web technologies (HTML, CSS, Javascript, XPath, Regex)
- Understanding of data privacy policies (esp. GDPR) and personally identifiable information
- Develop automated and reusable routines for extracting information from various data sources
- Prepare requirement summary and re-confirm with Operation team
- Translate business requirements into specific solutions
- Ability to relay technical information to non-technical users
- Demonstrate Effective problem solving and analytical skill
- Ability to pay attention to detail, pro-active, critical thinking and accuracy is essential
- Ability to work to deadlines and give realistic estimates
Skills & Expertise
- 2+ years of web scraping experience
- Experience with two or more of the following web scraping frameworks and tools: Selenium, Scrapy, Import.io, Webhose.io, ScrapingHub, ParseHub, Phantombuster, Octoparse, Puppeter, etc.
- Basic knowledge of data engineering (database ingestion, ETL, etc.)
- Solution orientation and "can do" attitude - with a desire to tackle complex problems.
Shubham Vishwakarma
Full Stack Developer - Averlon
About TransForm Solution
About
TransForm Solutions is an International BPO company with world-class execution offering global clients customized services and exceptional customer engagement since 2002.
We have evolved to provide a wide range of operations and back-office services since then, for companies all over the world (including Fortune 500s) in industries as diverse as consulting, financial services, eCommerce, education, logistics and real estate.
Company social profiles
Similar jobs (1)
Python
DjangoDjango + Scraper Developer – Job Description
Position: Django + Scraper Developer
Job Summary
We are looking for a skilled Django + Web Scraping Developer with 3+ years of experience in developing scalable web applications using Django and building robust web scrapers. The ideal candidate should have strong Python knowledge, experience with scraping frameworks and APIs, and the ability to work with databases and third-party integrations.
Key Responsibilities
- Develop, maintain, and enhance web applications using Python and Django.
- Design and implement reliable web scraping solutions for extracting structured data from websites.
- Develop scrapers using tools such as Scrapy, Selenium, BeautifulSoup, Requests, or similar technologies.
- Handle dynamic websites, pagination, authentication, proxies, CAPTCHA-related challenges, and anti-bot mechanisms where legally and technically appropriate.
- Clean, validate, transform, and store scraped data in databases.
- Develop and integrate REST APIs and third-party APIs.
- Work with databases such as MySQL, PostgreSQL, or MongoDB.
- Optimize scraper performance, reliability, and data accuracy.
- Troubleshoot and fix issues related to scraping, Django applications, APIs, and databases.
- Write clean, reusable, and maintainable Python code.
- Collaborate with developers, project managers, and other stakeholders to understand project requirements.
- Perform testing, debugging, and performance optimization.
Required Skills
- 3+ years of professional experience in Python/Django development.
- Strong knowledge of Python and Django.
- Hands-on experience with Web Scraping / Data Extraction.
- Experience with Scrapy, Selenium, BeautifulSoup, Requests, or equivalent tools.
- Good understanding of HTML, CSS, JavaScript, DOM, and HTTP/HTTPS.
- Experience working with REST APIs and JSON.
- Good knowledge of SQL and databases such as MySQL/PostgreSQL.
- Understanding of Git/version control.
- Strong debugging and problem-solving skills.
Good to Have
- Experience with Celery, Redis, Docker, and Linux.
- Knowledge of cloud platforms such as AWS.
- Experience handling large-scale scraping projects.
- Understanding of proxy rotation, browser automation, and anti-bot techniques.
- Experience with data pipelines and ETL processes.
What We Offer
- Opportunity to work on challenging and real-world projects.
- Collaborative and growth-oriented work environment.
- Learning and professional development opportunities.
- Competitive salary based on skills and experience.
How to Apply
Interested candidates can share their updated resume along with their current salary, expected salary, notice period, and total experience.
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon