Staff Engineer

Leverage your expertise in Python to design and implement distributed systems for web scraping, ensuring robust data extraction from diverse web sources.

• Develop and optimize scalable, self-healing scraping frameworks, integrated with AI tools for intelligent automation of the data collection process.

• Implement monitoring, logging, and alerting mechanisms to ensure high availability and performance of distributed web scraping systems.

• Work with large-scale NoSQL databases (e.g., MongoDB) to store and query scraped data efficiently.

• Collaborate with cross-functional teams to research and implement innovative AI-driven solutions for data extraction and automation.

• Ensure data integrity and security while interacting with various web sources.

Required Skills: • Extensive experience with Python and web frameworks like Flask, FastAPI, or Django.

• Experience with AI tools and machine learning libraries to enhance and automate scraping processes

. • Solid understanding of building and maintaining distributed systems, with hands-on experience in parallel programming (multithreading, asynchronous, multiprocessing).

• Working knowledge of asynchronous queue systems like Redis, Celery, RabbitMQ, etc., to handle distributed scraping tasks.

• Proven experience with web mining, scraping tools(e.g., Scrapy, BeautifulSoup, Selenium), and handling dynamic content.

• Proficiency in working with NoSQL data storage systems like MongoDB, including querying and handling large datasets.

• Knowledge of working with variousfront-end technologies and how various websites are built

• Writing efficient, reusable, testable, and scalable code
• Understanding, analyzing, and implementing – Business needs, feature modification requests, conversion into software components
• Integration of user-oriented elements into different applications, data storage solutions
• Developing – Backend components to enhance performance and receptiveness, server-side logic, and platform, statistical learning models, highly responsive web applications
• Designing and implementing – High availability and low latency applications, data protection and security features
• Performance tuning and automation of application
• Working with Python libraries like Pandas, NumPy, etc.
• Creating predictive models for AI and ML-based features
• Keeping abreast with the latest technology and trends
• Fine-tune and develop AI/ML-based algorithms based on results

Good proficiency in,

• Python frameworks like Django, etc.
• Web frameworks and RESTful APIs
• Core Python fundamentals and programming
• Code packaging, release, and deployment
• Database knowledge
• Circles, conditional and control statements
• Object-relational mapping
• Code versioning tools like Git, Bitbucket

Fundamental understanding of,

• Front-end technologies like JS, CSS3 and HTML5
• AI, ML, Deep Learning, Version Control, Neural networking
• Data visualization, statistics, data analytics
• Design principles that are executable for a scalable app
• Creating predictive models
• Libraries like Tensorflow, Scikit-learn, etc
• Multi-process architecture
• Basic knowledge about Object Relational Mapper libraries
• Ability to integrate databases and various data sources into a unified system
• Basic knowledge about Object Relational Mapper libraries
• Ability to integrate databases and various data sources into a unified system

Role- Software Development Engineer-2

As a Software Development Engineer at Amazon, you have industry-leading technical abilities and demonstrate breadth and depth of knowledge. You build software to deliver business impact, making smart technology choices. You work in a team and drive things forward.

Top Skills

You write high quality, maintainable, and robust code, often in Java or C++ or C#

You recognize and adopt best practices in software engineering: design, testing, version control, documentation, build, deployment, and operations.

You have experience building scalable software systems that are high-performance, highly-available, highly transactional, low latency and massively distributed.

Roles & Responsibilities

You solve problems at their root, stepping back to understand the broader context.

You develop pragmatic solutions and build flexible systems that balance engineering complexity and timely delivery, creating business impact.

You understand a broad range of data structures and algorithms and apply them to deliver high-performing applications.

You recognize and use design patterns to solve business problems.

You understand how operating systems work, perform and scale.

You continually align your work with Amazon’s business objectives and seek to deliver business value.

You collaborate to ensure that decisions are based on the merit of the proposal, not the proposer.

You proactively support knowledge-sharing and build good working relationships within the team and with others in Amazon.

You communicate clearly with your team and with other groups and listen effectively.

Skills & Experience

Bachelors or Masters in Computer Science or relevant technical field.

Experience in software development and full product life-cycle.

Excellent programming skills in any object-oriented programming languages - preferably Java, C/C++/C#, Perl, Python, or Ruby.

Strong knowledge of data structures, algorithms, and designing for performance, scalability, and availability.

Proficiency in SQL and data modeling.

• Minimum 1-year experience is mandatory
• Proven Software Development Skills in Laravel Framework and Core PHP
• Good knowledge and experience in OOPS concepts and Core PHP
• Good Database designing skills in MySQL
• Experience in known APIs like Google, Facebook, etc.
• Knowledge of JavaScript, JQuery, JASON, AngularJS will help
• Passion for Software Development and Coding
• Good Knowledge and Experience in WordPress will help.

We are looking for an experienced engineer with superb technical skills. You will primarily be responsible for architecting and building large scale data pipelines that delivers AI and Analytical solutions to our customers. The right candidate will enthusiastically take ownership in developing and managing a continuously improving, robust, scalable software solutions. The successful candidate will be curious, creative, ambitious, self motivated, flexible, and have a bias towards taking action. As part of the early engineering team, you will have a chance to make a measurable impact in future of Thinkdeeply as well as having a significant amount of responsibility.

Although your primary responsibilities will be around back-end work, we prize individuals who are willing to step in and contribute to other areas including automation, tooling, and management applications. Experience with or desire to learn Machine Learning a plus.

Experience

12+ Years

Location

Hyderabad

Skills

Bachelors/Masters/Phd in CS or equivalent industry experience

10+ years of industry experience in java related frameworks such as Spring and/or Typesafe

Experience with scripting languages. Python experience highly desirable. 5+ Industry experience in python

Experience with popular modern web frameworks such as Spring boot, Play framework, or Django

Demonstrated expertise of building and shipping cloud native applications

Experience in administering (including setting up, managing, monitoring) data processing pipelines (both streaming and batch) using frameworks such as Kafka, ELK Stack, Fluentd

Experience in API development using Swagger

Strong expertise with containerization technologies including kubernetes, docker-compose

Experience with cloud platform services such as AWS, Azure or GCP.

Implementing automated testing platforms and unit tests

Proficient understanding of code versioning tools, such as Git

Familiarity with continuous integration, Jenkins

Responsibilities

Architect, Design and Implement Large scale data processing pipelines

Design and Implement APIs

Assist in dev ops operations

Identify performance bottlenecks and bugs, and devise solutions to these problems

Help maintain code quality, organization, and documentation

Communicate with stakeholders regarding various aspects of solution.

Mentor team members on best practices