Cutshort logo
For Employers
Agentic AI platform for retail & ecommerce logo
Web Scraping Specialist
Agentic AI platform for retail & ecommerce
Web Scraping Specialist

Web Scraping Specialist at Agentic AI platform for retail & ecommerce · Remote only · 4 - 8 years · Remote only · Posted 19 Nov 2026

Cutshort Lightning's logo

Web Scraping Specialist

at "Agentic AI platform for retail & ecommerce"

Exclusive
job in Cutshort Lightning by Ariba Khan
4 - 8 yrs
Best in industry
Remote only
Skills
skill iconPython
skill iconJavascript
skill iconHTML/CSS
Beautiful Soup
Web Scraping
Selenium

Position Overview:

We are seeking a highly skilled Web Scraping Architect to join our team. The successful candidate will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately. As a Web Scraping Specialist, you will play a crucial role in collecting data for competitor analysis, and other business intelligence purposes.


Responsibilities:

  • Scalability/Performance: Lead and provide expertise in scraping at scale e-commerce marketplaces.
  • Data Source Identification: Identify relevant websites and online sources from which data needs to be scraped. Collaborate with the team to understand data requirements and objectives
  • Web Scraping Design: Develop and implement effective web scraping strategies to extract data from targeted websites. This includes selecting appropriate tools, libraries, or frameworks for the task
  • Data Extraction: Create and maintain web scraping scripts or programs to extract the required data. Ensure the code is optimized, reliable, and can handle changes in the website's structure
  • Data Cleansing and Validation: Cleanse and validate the collected data to eliminate errors, inconsistencies, and duplicates. Ensure data integrity and accuracy throughout the process
  • Monitoring and Maintenance: Continuously monitor and maintain the web scraping processes. Address any issues that arise due to website changes, data format modifications, or anti-scraping mechanisms
  • Scalability and Performance: Optimize web scraping procedures for efficiency and scalability, especially when dealing with a large volume of data or multiple data sources
  • Compliance and Legal Considerations: Stay up-to-date with legal and ethical considerations related to web scraping, including website terms of service, copyright, and privacy regulations
  • Documentation: Maintain detailed documentation of web scraping processes, data sources, and methodologies. Create clear and concise instructions for others to follow
  • Collaboration: Collaborate with other teams such as data analysts, developers, and business stakeholders to understand data requirements and deliver insights effectively
  • Security: Implement security measures to ensure the confidentiality and protection of sensitive data throughout the scraping process


Requirements:

  • Proven experience of 4+ years as a Web Scraping Specialist or similar role, with a track record of successful web scraping projects
  • Expertise in handling dynamic content, user-agent rotation, bypass CAPTCHAs, rate limits, and utilization of proxy services
  • Knowledge on browser fingerprinting
  • Has leadership experience
  • Proficiency in programming languages commonly used for web scraping, such as Python, BeautifulSoup, Scrapy, or Selenium
  • Strong knowledge of HTML, CSS, XPath, and other web technologies relevant to web scraping and Coding
  • Knowledge and experience in best of class data storage and retrieval of large volume of scraped data.
  • Understanding of web scraping best practices, including handling dynamic content, user-agent rotation, and IP address management
  • Attention to detail and the ability to handle and process large volumes of data accurately
  • Familiarity with data cleansing techniques and data validation processes
  • Good communication skills and the ability to collaborate effectively with cross-functional teams
  • Knowledge of web scraping ethics, legal considerations, and compliance with website terms of service
  • Strong problem-solving skills and the ability to adapt to changing web environments


Preferred Qualifications:

  • Bachelor's degree in Computer Science, Data Science, Information Technology, or related fields
  • Experience with cloud-based solutions and distributed web scraping systems
  • Familiarity with APIs and data extraction from non-public sources
  • Knowledge of machine learning techniques for data extraction and natural language processing is desired but not mandatory
  • Prior experience in handling large-scale data projects and working with big data frameworks
  • Understanding of various data formats such as JSON, XML, CSV, etc
  • Experience with version control systems like Git
Read more
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos

Similar jobs (2)

Resources Valley
Actively reviewing applications
at Resources Valley
1 recruiter
Manind Gupta
Posted by Manind Gupta
Jaipur
5 - 15 yrs
₹10L - ₹40L / yr
skill iconPython
skill iconDjango

Responsibilities

  • Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
  • Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session or token-based login flows.
  • Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
  • Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
  • Integrate and manage complex workflows across multiple portals, APIs, and data sources.
  • Contribute to architectural decisions, tooling, and best practices.
  • Responsible for Code reviews and deployments.

 

Qualifications

  • Bachelor’s or master’s degree in computer science, software engineering, or related field, or equivalent professional experience
  • 10+ years of experience in Python development, with a strong focus on automation and data extraction
  • Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests
  • In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session or cookie management
  • Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager
  • Experience with asynchronous programming, headless browsers, and JavaScript-rendered content
  • Solid understanding of web protocols (i.e. HTTP, HTTPS, cookies, headers), and the ability to reverse-engineer network calls and authentication mechanisms
  • Comfortable working with APIs, JSON/XML, and data transformation
  • Experience with CI or CD pipelines, Docker, and infrastructure as code (i.e. CloudFormation, Terraform)
  • Familiar with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions
  • Prior experience scraping from secure or enterprise-level portals (i.e. financial, healthcare, legal)
  • Background in data engineering or ETL workflows is a plus
  • Exposure to Python testing frameworks and writing unit or integration tests


Read more
Commercify360
Viny Deshmukh
Posted by Viny Deshmukh
Gurugram
2 - 5 yrs
₹4.8L - ₹10L / yr
skill iconPython

About Us:

Datum builds market intelligence solutions for the retail industry. We transform public and proprietary data into actionable insights that help retail brands decide where to expand, compete, and grow. We are a small, fast-moving team that works closely with customers and

believes in shipping impactful products quickly.

About the Role

We're looking for a Full Stack Platform Engineer to build and own our end-to-end product ecosystem, including:

● Data acquisition through scalable web scraping and ETL pipelines

● Customer-facing analytics platform

● Geospatial analysis engine powering retail insights

You'll work across frontend, backend, data engineering, cloud infrastructure, and geospatial systems while collaborating directly with the founder and customers.

Key Responsibilities

● Build scalable web scrapers and ETL pipelines

● Develop customer-facing features using React/Next.js

● Design REST & WebSocket APIs

● Work with PostgreSQL/PostGIS and geospatial data

● Own deployment, CI/CD, monitoring, and cloud infrastructure

● Translate customer feedback into product features

Must-Have Skills

● 3–6 years of Full Stack development experience

● Python (FastAPI/Django)

● React or Next.js with JavaScript/TypeScript

● Web scraping using Scrapy, Playwright, Selenium, or BeautifulSoup

● PostgreSQL (PostGIS preferred)

● REST APIs, JWT/OAuth

● Docker, Git, AWS/GCP/Azure

● Understanding of proxy rotation and anti-bot techniques

Good to Have

● Mapbox, Leaflet, or Google Maps Platform

● Airflow, Redis, Kafka, or SQS

● Experience with large-scale scraping (Google Maps, Zomato, Justdial, etc.)

● Geospatial analytics or retail domain experience

● Startup experience with end-to-end ownership

Read more
Why apply to jobs via Cutshort
people_solving_puzzle
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
people_verifying_people
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly.
ai_chip
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
companies logo
companies logo
companies logo
companies logo
companies logo
Get to hear about interesting companies hiring right now
Company logo
Company logo
Company logo
Company logo
Company logo
Linkedin iconFollow Cutshort
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos