Web Scraping Data Engineer

at TransForm Solution

Web Scraping Data Engineer

TransForm Solution

Company

Home

Web Scraping Data Engineer

at TransForm Solution

Posted by sowmiya Venkat

3 - 5 yrs

₹3L - ₹7L / yr

Surat

Skills

Javascript

Python

Selenium

Web Scraping

HTML/CSS

XPath

Design, build web crawlers to scrape data and URLs.
Integrate the data crawled and scraped into our databases
Create more/better ways to crawl relevant information
Strong knowledge of web technologies (HTML, CSS, Javascript, XPath, Regex)
Understanding of data privacy policies (esp. GDPR) and personally identifiable information
Develop automated and reusable routines for extracting information from various data sources
Prepare requirement summary and re-confirm with Operation team
Translate business requirements into specific solutions
Ability to relay technical information to non-technical users
Demonstrate Effective problem solving and analytical skill
Ability to pay attention to detail, pro-active, critical thinking and accuracy is essential
Ability to work to deadlines and give realistic estimates

Skills & Expertise

2+ years of web scraping experience

Experience with two or more of the following web scraping frameworks and tools: Selenium, Scrapy, Import.io, Webhose.io, ScrapingHub, ParseHub, Phantombuster, Octoparse, Puppeter, etc.
Basic knowledge of data engineering (database ingestion, ETL, etc.)
Solution orientation and "can do" attitude - with a desire to tackle complex problems.

Design, build web crawlers to scrape data and URLs.
Integrate the data crawled and scraped into our databases
Create more/better ways to crawl relevant information
Strong knowledge of web technologies (HTML, CSS, Javascript, XPath, Regex)
Understanding of data privacy policies (esp. GDPR) and personally identifiable information
Develop automated and reusable routines for extracting information from various data sources
Prepare requirement summary and re-confirm with Operation team
Translate business requirements into specific solutions
Ability to relay technical information to non-technical users
Demonstrate Effective problem solving and analytical skill
Ability to pay attention to detail, pro-active, critical thinking and accuracy is essential
Ability to work to deadlines and give realistic estimates

Skills & Expertise

2+ years of web scraping experience

Experience with two or more of the following web scraping frameworks and tools: Selenium, Scrapy, Import.io, Webhose.io, ScrapingHub, ParseHub, Phantombuster, Octoparse, Puppeter, etc.
Basic knowledge of data engineering (database ingestion, ETL, etc.)
Solution orientation and "can do" attitude - with a desire to tackle complex problems.

Users love Cutshort

Read about what our users have to say about finding their next opportunity on Cutshort.

Subodh Popalwar

Software Engineer, Memorres

For 2 years, I had trouble finding a company with good work culture and a role that will help me grow in my career. Soon after I started using Cutshort, I had access to information about the work culture, compensation and what each company was clearly offering.

Companies hiring on Cutshort

About TransForm Solution

Founded :

2002

Type

Size :

20-100

Stage :

Profitable

About

TransForm Solutions is an International BPO company with world-class execution offering global clients customized services and exceptional customer engagement since 2002.

We have evolved to provide a wide range of operations and back-office services since then, for companies all over the world (including Fortune 500s) in industries as diverse as consulting, financial services, eCommerce, education, logistics and real estate.

TransForm Solutions is an International BPO company with world-class execution offering global clients customized services and exceptional customer engagement since 2002.

Connect with the team

sowmiya Venkat

Connect

Seema Shiliwala

Connect

Company social profiles

Similar jobs

Test Automation Engineer

Large automotive manufacturer

Agency job

via Bullhorn Consultants by Hemant G

Bengaluru (Bangalore)

3 - 7 yrs

₹1L - ₹15L / yr

Test Automation (QA)

Manual testing

Python

dspace

test

Automating the test plans, develop the test scripts from requirements, execution of test scripts and report generation for model based algorithms on the HIL bench.; Conducting Software release testing; Reviews and Delivery of final product.

Data Engineer

at Scremer

Posted by Sathish Dhawan

Pune, Mumbai

6 - 11 yrs

₹15L - ₹15L / yr

Amazon Web Services (AWS)

Python

Java

Spark

Primary Skills

DynamoDB, Java, Kafka, Spark, Amazon Redshift, AWS Lake Formation, AWS Glue, Python

Skills:

Good work experience showing growth as a Data Engineer.

Hands On programming experience

Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS Lake Formation.

Excellent knowledge in: Python, Scala/Java, Spark, AWS (Lambda, Step Functions, Dynamodb, EMR), Terraform, UI (Angular), Git, Mavena

Experience of performance optimization in Batch and Real time processing applications

Expertise in Data Governance and Data Security Implementation

Good hands-on design and programming skills building reusable tools and products Experience developing in AWS or similar cloud platforms. Preferred:, ECS, EKS, S3, EMR, DynamoDB, Aurora, Redshift, Quick Sight or similar.

Familiarity with systems with very high volume of transactions, micro service design, or data processing pipelines (Spark).

Knowledge and hands-on experience with server less technologies such as Lambda, MSK, MWAA, Kinesis Analytics a plus.

Expertise in practices like Agile, Peer reviews, Continuous Integration

Roles and responsibilities:

Determining project requirements and developing work schedules for the team.

Delegating tasks and achieving daily, weekly, and monthly goals.

Responsible for designing, building, testing, and deploying the software releases.

Salary: 25LPA-40LPA

Primary Skills

DynamoDB, Java, Kafka, Spark, Amazon Redshift, AWS Lake Formation, AWS Glue, Python

Skills:

Good work experience showing growth as a Data Engineer.

Hands On programming experience

Implementation Experience on Kafka, Kinesis, Spark, AWS Glue, AWS Lake Formation.

Excellent knowledge in: Python, Scala/Java, Spark, AWS (Lambda, Step Functions, Dynamodb, EMR), Terraform, UI (Angular), Git, Mavena

Experience of performance optimization in Batch and Real time processing applications

Expertise in Data Governance and Data Security Implementation

Familiarity with systems with very high volume of transactions, micro service design, or data processing pipelines (Spark).

Knowledge and hands-on experience with server less technologies such as Lambda, MSK, MWAA, Kinesis Analytics a plus.

Expertise in practices like Agile, Peer reviews, Continuous Integration

Roles and responsibilities:

Determining project requirements and developing work schedules for the team.

Delegating tasks and achieving daily, weekly, and monthly goals.

Responsible for designing, building, testing, and deploying the software releases.

Salary: 25LPA-40LPA

Data Engineer

Series 'A' funded Silicon Valley based BI startup

Agency job

via Qrata by Prajakta Kulkarni

Bengaluru (Bangalore)

4 - 6 yrs

₹30L - ₹45L / yr

Data engineering

Data Engineer

Scala

Data Warehouse (DWH)

Big Data

+7 more

It is the leader in capturing technographics-powered buying intent, helps
companies uncover the 3% of active buyers in their target market. It evaluates
over 100 billion data points and analyzes factors such as buyer journeys, technology
adoption patterns, and other digital footprints to deliver market & sales intelligence.
Its customers have access to the buying patterns and contact information of
more than 17 million companies and 70 million decision makers across the world.

Role – Data Engineer

Responsibilities

 Work in collaboration with the application team and integration team to
design, create, and maintain optimal data pipeline architecture and data
structures for Data Lake/Data Warehouse.
 Work with stakeholders including the Sales, Product, and Customer Support
teams to assist with data-related technical issues and support their data
analytics needs.
 Assemble large, complex data sets from third-party vendors to meet business
requirements.
 Identify, design, and implement internal process improvements: automating
manual processes, optimizing data delivery, re-designing infrastructure for
greater scalability, etc.
 Build the infrastructure required for optimal extraction, transformation, and
loading of data from a wide variety of data sources using SQL, Elasticsearch,
MongoDB, and AWS technology.
 Streamline existing and introduce enhanced reporting and analysis solutions
that leverage complex data sources derived from multiple internal systems.

Requirements
 5+ years of experience in a Data Engineer role.
 Proficiency in Linux.
 Must have SQL knowledge and experience working with relational databases,
query authoring (SQL) as well as familiarity with databases including Mysql,
Mongo, Cassandra, and Athena.
 Must have experience with Python/Scala.
 Must have experience with Big Data technologies like Apache Spark.
 Must have experience with Apache Airflow.
 Experience with data pipeline and ETL tools like AWS Glue.
 Experience working with AWS cloud services: EC2, S3, RDS, Redshift.

Solutions Architect

at Simform

1 video

2 recruiters

Posted by Satendra Chauhan

Ahmedabad

10 - 15 yrs

₹10L - ₹15L / yr

NodeJS (Node.js)

Python

Solution architecture

Scalability

We are looking for a passionate Solutions Architect with a diverse backend technology background to lead large engineering teams at Simform. You should have a good understanding of software architecture design, backend technologies, and cloud platforms and have experience leading a team of developers.

Role + Responsibilities:

Designing, modifying, and testing technical architecture.
Helping project teams with solutions architecture, troubleshooting, and technical implementation assistance.
Occasionally working as a Tech Lead if the situation demands
Use your expertise in different tech-stack/languages, tech architecture, etc., to drive improvements in the accuracy, quality, the efficiency of projects.
Advocating for process improvements and helping develop solutions.
Providing technical leadership to a team throughout the project lifecycle.
Oversee project teams' work and provide assistance for any troubleshooting or technical implementation guidance.
Ensure standards and best practices are implemented and practiced by large cross-technical teams.

Person Specification and Qualifications:

Technical background with over 10 years dabbling into multiple technologies and grasping crux of them.
Significant hands-on experience with Node/Python
Experience in building reliable, robust, secure, and high-performance in-production software systems with a large user base.
Strong understanding of Cloud (Preferably AWS) and conceptual understanding of CI/CD, and DevOps.
Strong interpersonal skills including mentoring, coaching, collaborating, and team building.
Creative approach to problem-solving with the ability to focus on details while maintaining the “big picture” view
Exposure and expertise in design patterns and software design patterns in general

Role + Responsibilities:

Designing, modifying, and testing technical architecture.
Helping project teams with solutions architecture, troubleshooting, and technical implementation assistance.
Occasionally working as a Tech Lead if the situation demands
Use your expertise in different tech-stack/languages, tech architecture, etc., to drive improvements in the accuracy, quality, the efficiency of projects.
Advocating for process improvements and helping develop solutions.
Providing technical leadership to a team throughout the project lifecycle.
Oversee project teams' work and provide assistance for any troubleshooting or technical implementation guidance.
Ensure standards and best practices are implemented and practiced by large cross-technical teams.

Person Specification and Qualifications:

Technical background with over 10 years dabbling into multiple technologies and grasping crux of them.
Significant hands-on experience with Node/Python
Experience in building reliable, robust, secure, and high-performance in-production software systems with a large user base.
Strong understanding of Cloud (Preferably AWS) and conceptual understanding of CI/CD, and DevOps.
Strong interpersonal skills including mentoring, coaching, collaborating, and team building.
Creative approach to problem-solving with the ability to focus on details while maintaining the “big picture” view
Exposure and expertise in design patterns and software design patterns in general

Software Developer - EDI

IT Software Services

Agency job

via MNR Solutions by Neeraj Shukla

Remote, Noida, NCR (Delhi | Gurgaon | Noida)

2 - 7 yrs

₹5L - ₹16L / yr

EDI

Javascript

TIBCO

AS2

AS3

+3 more

Job Description :

•Proficiency in at least one EDI tool like Sterling B2B Integrator, TIBCO etc.
•Sound knowledge of EDI standards like ANSI X12, EDIFACT, TRADACOM, XML, Rosetta Net etc.
•Good understanding of XSLT, XSD and XML parsing processes.
•Having WebMethods knowledge in EDI domain will be huge plus.
•Good Knowledge of scripting languages like JavaScript, Perl etc.
•Good knowledge of file sharing protocols like AS2, AS3, FTP/SFTP, HTTP/HTTPS etc.
•Experience in EDI Mapping to/from following platforms/technologies (not limited to)

oOracle
oSAP (IDOC)
oMainframe.
oCSV/XML/JSON/Flat File e.Word/Excel/PDF

•Good understanding of Windows environment.
•Good working knowledge SQL and relation databases (Oracle/MySQL etc.).

Job Description :

oOracle
oSAP (IDOC)
oMainframe.
oCSV/XML/JSON/Flat File e.Word/Excel/PDF

•Good understanding of Windows environment.
•Good working knowledge SQL and relation databases (Oracle/MySQL etc.).

Oracle PL/SQL Developer

US based Product MNC

Agency job

via Gipfel & Schnell Consultings Pvt Ltd by Suma Latha

Remote, Bengaluru (Bangalore)

3 - 10 yrs

₹12L - ₹20L / yr

Oracle

Oracle SQL Developer

PL/SQL

Oracle Application Express (APEX)

Javascript

+1 more

• Minimum of 3 years of work experience.
• Experience working in complex enterprise solutions is a plus.
• Very strong Oracle PL/SQL and SQL are required.
• Strong CSS/JQuery/JavaScript would be needed.
• Experience with Oracle APEX development is required.
• Experience using Bitbucket /GitHub for version controlling

Project Engineer Intern

at Helical IT Solutions

4 recruiters

Posted by Bhavani Thanga

Hyderabad

0 - 0 yrs

₹1.2L - ₹3.5L / yr

PySpark

Data engineering

Big Data

Hadoop

Spark

+4 more

Job description

About Company
Helical Insight an open source Business Intelligence tool from Helical IT Solutions Pvt. Ltd,
based out of Hyderabad, is looking for fresher’s having strong knowledge on SQL. Helical
Insight has more than 50+ clients from various sectors. It has been awarded the most promising
company in the Business Intelligence space. We are looking for rockstar team mate to join
our company.
Job Brief
We are looking for a Business Intelligence (BI) Developer to create and manage BI and analytics
solutions that turn data into knowledge.
In this role, you should have a background in data and business analysis. You should be
analytical and an excellent communicator. If you also have a business acumen and
problemsolving aptitude, we’d like to meet you. Excellent knowledge on SQLQuery is required.
Basic knowledge on HTML CSS and JS is required.
You would be working closely with customers of various domain to understand their data,
understand their business requirement and deliver the required analytics in form of varous
reports dashboards etc. Excellent client interfacing role with opportunity to work across various
sectors and geographies as well as varioud kind of DB including NoSQL, RDBMS, graph db,
Columnar DB etc
Skill set and Qualification required
Responsibilities
 Attending client calls to get requriement, show progress
 Translate business needs to technical specifications
 Design, build and deploy BI solutions (e.g. reporting tools)
 Maintain and support data analytics platforms)
 Conduct unit testing and troubleshooting
 Evaluate and improve existing BI systems
 Collaborate with teams to integrate systems
 Develop and execute database queries and conduct analyses
 Create visualizations and reports for requested projects
 Develop and update technical documentation
Requirements
 Excellent expertise on SQLQueries
 Proven experience as a BI Developer or Data Scientist
 Background in data warehouse design (e.g. dimensional modeling) and data mining
 In-depth understanding of database management systems, online analytical processing
(OLAP) and ETL (Extract, transform, load) framework
 Familiarity with BI technologies
 Proven abilities to take initiative and be innovative
 Analytical mind with a problem-solving aptitude
 BE in Computer Science/IT
Education: BE/ BTech/ MCA/BCA/ MTech/ MS, or equivalent preferred.
Interested candidates call us on +91 7569 765 162

Job description

Data Engineer

at Fragma Data Systems

8 recruiters

Posted by Sudarshini K

Bengaluru (Bangalore)

2 - 6 yrs

₹8L - ₹14L / yr

ETL

Big Data

Hadoop

PySpark

SQL

+4 more

Roles and Responsibilities:

• Responsible for developing and maintaining applications with PySpark

• Contribute to the overall design and architecture of the application developed and deployed.

• Performance Tuning wrt to executor sizing and other environmental parameters, code optimization, partitions tuning, etc.

• Interact with business users to understand requirements and troubleshoot issues.

• Implement Projects based on functional specifications.

Must Have Skills:

• Good experience in Pyspark - Including Dataframe core functions and Spark SQL

• Good experience in SQL DBs - Be able to write queries including fair complexity.

• Should have excellent experience in Big Data programming for data transformation and aggregations

• Good at ETL architecture. Business rules processing and data extraction from Data Lake into data streams for business consumption.

• Good customer communication.

• Good Analytical skills