Hypersonix Inc

About the Company

Hypersonix.ai is disrupting the e-commerce space with AI, ML, and advanced decision-making capabilities to drive real-time business insights. Built from the ground up using modern technologies, Hypersonix simplifies data consumption for customers across various industry verticals. We are seeking a well-rounded, hands-on product leader to help manage key capabilities and features in our platform.

Position Overview

We are seeking a highly skilled Web Scraping Architect to join our team. The successful candidate will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately. As a Web Scraping Specialist, you will play a crucial role in collecting data for competitor analysis and other business intelligence purposes.

Responsibilities

• Scalability/Performance: Lead and provide expertise in scraping at scale e-commerce marketplaces.
• Data Source Identification: Identify relevant websites and online sources from which data needs to be scraped. Collaborate with the team to understand data requirements and objectives.
• Web Scraping Design: Develop and implement effective web scraping strategies to extract data from targeted websites. This includes selecting appropriate tools, libraries, or frameworks for the task.
• Data Extraction: Create and maintain web scraping scripts or programs to extract the required data. Ensure the code is optimized, reliable, and can handle changes in the website's structure.
• Data Cleansing and Validation: Cleanse and validate the collected data to eliminate errors, inconsistencies, and duplicates. Ensure data integrity and accuracy throughout the process.
• Monitoring and Maintenance: Continuously monitor and maintain the web scraping processes. Address any issues that arise due to website changes, data format modifications, or anti-scraping mechanisms.
• Scalability and Performance: Optimize web scraping procedures for efficiency and scalability, especially when dealing with a large volume of data or multiple data sources.
• Compliance and Legal Considerations: Stay up-to-date with legal and ethical considerations related to web scraping, including website terms of service, copyright, and privacy regulations.
• Documentation: Maintain detailed documentation of web scraping processes, data sources, and methodologies. Create clear and concise instructions for others to follow.
• Collaboration: Collaborate with other teams such as data analysts, developers, and business stakeholders to understand data requirements and deliver insights effectively.
• Security: Implement security measures to ensure the confidentiality and protection of sensitive data throughout the scraping process.

Requirements

• Proven experience of 6+ years as a Web Scraping Specialist or similar role, with a track record of successful web scraping projects
• Expertise in handling dynamic content, user-agent rotation, bypassing CAPTCHAs, rate limits, and use of proxy services
• Knowledge of browser fingerprinting
• Has leadership experience
• Proficiency in programming languages commonly used for web scraping, such as Python, BeautifulSoup, Scrapy, or Selenium
• Strong knowledge of HTML, CSS, XPath, and other web technologies relevant to web scraping and coding
• Knowledge and experience in best-of-class data storage and retrieval for large volumes of scraped data
• Understanding of web scraping best practices, including handling dynamic content, user-agent rotation, and IP address management
• Attention to detail and ability to handle and process large volumes of data accurately
• Familiarity with data cleansing techniques and data validation processes
• Good communication skills and ability to collaborate effectively with cross-functional teams
• Knowledge of web scraping ethics, legal considerations, and compliance with website terms of service
• Strong problem-solving skills and adaptability to changing web environments

Preferred Qualifications

• Bachelor’s degree in Computer Science, Data Science, Information Technology, or related fields
• Experience with cloud-based solutions and distributed web scraping systems
• Familiarity with APIs and data extraction from non-public sources
• Knowledge of machine learning techniques for data extraction and natural language processing is desired but not mandatory
• Prior experience in handling large-scale data projects and working with big data frameworks
• Understanding of various data formats such as JSON, XML, CSV, etc.
• Experience with version control systems like Git

About the Company

Hypersonix.ai is disrupting the e-commerce space with AI, ML and advanced decision capabilities to drive real-time business insights. Hypersonix.ai has been built ground up with new age technology to simplify the consumption of data for our customers in various industry verticals. Hypersonix.ai is seeking a well-rounded, hands-on product leader to help lead product management of key capabilities and features.

About the Role

We are looking for talented and driven Data Engineers at various levels to work with customers to build the data warehouse, analytical dashboards and ML capabilities as per customer needs.

Roles and Responsibilities

• Create and maintain optimal data pipeline architecture
• Assemble large, complex data sets that meet functional / non-functional business requirements; should write complex queries in an optimized way
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and AWS ‘big data’ technologies
• Run ad-hoc analysis utilizing the data pipeline to provide actionable insights
• Work with stakeholders including the Executive, Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs
• Keep our data separated and secure across national boundaries through multiple data centers and AWS regions
• Work with analytics and data scientist team members and assist them in building and optimizing our product into an innovative industry leader

Requirements

• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
• Strong analytic skills related to working with unstructured datasets
• Build processes supporting data transformation, data structures, metadata, dependency and workload management
• A successful history of manipulating, processing and extracting value from large disconnected datasets
• Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores
• Experience supporting and working with cross-functional teams in a dynamic environment
• We are looking for a candidate with 4+ years of experience in a Data Engineer role, who has attained a Graduate degree in Computer Science, Information Technology or completed MCA.