Leverage your expertise in Python to design and implement distributed systems for web scraping, ensuring robust data extraction from diverse web sources.
• Develop and optimize scalable, self-healing scraping frameworks, integrated with AI tools for intelligent automation of the data collection process.
• Implement monitoring, logging, and alerting mechanisms to ensure high availability and performance of distributed web scraping systems.
• Work with large-scale NoSQL databases (e.g., MongoDB) to store and query scraped data efficiently.
• Collaborate with cross-functional teams to research and implement innovative AI-driven solutions for data extraction and automation.
• Ensure data integrity and security while interacting with various web sources.
Required Skills: • Extensive experience with Python and web frameworks like Flask, FastAPI, or Django.
• Experience with AI tools and machine learning libraries to enhance and automate scraping processes
. • Solid understanding of building and maintaining distributed systems, with hands-on experience in parallel programming (multithreading, asynchronous, multiprocessing).
• Working knowledge of asynchronous queue systems like Redis, Celery, RabbitMQ, etc., to handle distributed scraping tasks.
• Proven experience with web mining, scraping tools(e.g., Scrapy, BeautifulSoup, Selenium), and handling dynamic content.
• Proficiency in working with NoSQL data storage systems like MongoDB, including querying and handling large datasets.
• Knowledge of working with variousfront-end technologies and how various websites are built
About Draup
About
Similar jobs
Python. Django Rest Framework experience would be great, but not essential! We prioritise
quality over quantity.
Postgres with InfluxDB, Oracle, MySQL and Redis experience a plus.
Docker, Kubernetes, Helm, OpenShift and associated tooling.
AWS, especially ECS, Lambda, RDS and DynamoDB. Performance and uptime are super
important to us.
The challenge of scaling a global, distributed API to 10,000+ requests per second.
We have SDKs in a bunch of languages, so the more polyglot you are the better.
If you like writing JS and React that would be awesome too.
Python Developer
6-8 Years
Mumbai
N.p only immediate or who is serving LwD is 1st week of july.
- Python knowledge: object-oriented programming: inheritance, abstract classes, dataclass, dependency injection, design patterns: comand-query, repository, adapter, hexagonal architecture, swagger/Open API, flask, connexion
- Experience on AWS services: lambda, ecs, sqs, s3, dynamodb, auroradb
- Experience with following libraries boto3, behave, pytest, moto, localstack, docker
- Basic knowledge about terraform, gitlab ci
- Experience with SQL DB
Your Responsibilities :
1) Building highly scalable and secured applications/microservices and responsive APIs
2) Assessing the security risks associated with the platform architecture and suggesting/implementing suitable changes for the same.
3) Architect and implement back-end services to support our mobile and web clients.
4) Creating and implementing database schemas that represent and support business processes.
5) Use best software development practices and processes including unit testing, code reviews, and site monitoring post deployment.
6) Continue to research and evaluate new technologies that may improve performance and elegance of our software implementation.
7) Explore and Integrate third-party APIs when required for the project.
8) Work closely with the rest of the team to bring our features into production.
Your skills :
- Hands-on experience in Django Framework.
- Strong Knowledge of Python.
- Strong Knowledge of MVC design pattern and object oriented programming concepts.
- Strong knowledge of RDBMS like Postgresql and Database schema design.
- Experience in developing robust and secure RESTful web API and microservices using Django/Python.
- Experience in Test Driven Development.
- Experience in handling No-SQL MongoDB or similar.
- Hands-on experience in GitHub.
- Experience in CI, CD tools like jenkins.
- Experience in managing cloud-based web applications on Amazon Web Services or Google Cloud.
- You love code and up for the challenge!
· Lead from the front, when it comes to delivering high quality work products. Serve as a mentor to the team members
· Create and define performance metrics. Ideate, innovate and hack through the existing systems to improve performance.
· Perform code reviews, set coding practices and guidelines within the team.
· Make sense of the product specs, be nimble in getting the issues resolved. Be a thought leader. · Solid understanding and knowledge of Java/C/C++. You should understand the concepts of multithreading, I/O, messaging, instrumentation etc and their implementation in Java/Any other Object-Oriented Language.
· Proficiency in the implementation of algorithms and the use of advanced data structures to solve problems in computing. A solid foundation in the principles of computer science is desired.
· Ability to write clean, modular, and loosely coupled code by the use of design patterns.
· He/She should be able to break down and architect complex software systems. For example- Build
Hi All,
We are hiring!!
Company: SpringML India Pvt Ltd.
Role:Lead Data Engineer
Location: Hyderabad
Website: https://springml.com/">https://springml.com/
About Company:
At SpringML, we are all about empowering the 'doers' in companies to make smarter decisions with their data. Our predictive analytics products and solutions apply machine learning to today's most pressing business problems so customers get insights they can trust to drive business growth.
We are a tight-knit, friendly team of passionate and driven people who are dedicated to learning, get excited to solve tough problems and like seeing results, fast. Our core values include placing our customers first, empathy and transparency, and innovation. We are a team with a focus on individual responsibility, rapid personal growth, and execution. If you share similar traits, we want you on our team.
What's the opportunity?
SpringML is looking to hire a top-notch Lead Data Engineer who is passionate about working with data and using the latest distributed framework to process large dataset.
As a Lead Data Engineer, your primary role will be to design and build data pipelines. You will be focused on helping client projects on data integration, data prep and implementing machine learning on datasets.
In this role, you will work on some of the latest technologies, collaborate with partners on early win, consultative approach with clients, interact daily with executive leadership, and help build a great company. Chosen team members will be part of the core team and play a critical role in scaling up our emerging practice.
Responsibilities:
- Ability to work as a member of a team assigned to design and implement data integration solutions.
- Build Data pipelines using standard frameworks in Hadoop, Apache Beam and other open-source solutions.
- Learn quickly – ability to understand and rapidly comprehend new areas – functional and technical – and apply detailed and critical thinking to customer solutions.
- Propose design solutions and recommend best practices for large scale data analysis
Skills:
- B.tech degree in computer science, mathematics or other relevant fields.
- 6+years of experience in ETL, Data Warehouse, Visualization and building data pipelines.
- Strong Programming skills – experience and expertise in one of the following: Java, Python, Scala, C.
- Proficient in big data/distributed computing frameworks such as Apache Spark, Kafka,
- Experience with Agile implementation methodology
Perks
- Best hardware and software to help you get the job done in the best way possible
- Remote office allowance to support you regardless of where you work
- Learning and development allowance to help you grow your career
- Flexible leave policy to give you more control over your time.
- Health insurance for your family and you to keep you well.
Required experience and skills
- Bachelors degree in Computer Science and Engineering. Bonus: Masters degree.9+ years of total full-time work experience, preferably shipping SaaS applications.
- Startup experience strongly desired.
- Played a pivotal role in scaling a company and led teams in an agile environment where iteration has led to perfection.
- Proven track record of on time project deliveries with balanced calls on technical designs and tech debt.
- Expert proficiency in:
- 1. Ruby, Ruby on Rails, Mysql, Linux, Git, AWS, CI/CD, NewRelic.
- 2. broad range of internet technologies and applications.
- 3. interacting, building, and improving the performance of systems end-to-end.
- 4. architecting/designing complex systems
- Strong collaborator and thought partner to Product team.Driven by outcomes more than output.
- Have led projects with small, dynamic, and fast-paced teams (3-5 engineers), with great flexility and adaptability on large projects, delegating working effectively.
- Viewed by the team as their evangelist, role model and bar raiser.
- Past teams and co-workers attest to your prowess & agility in building great products, and driving sound technical decision making.Mentored engineers within and outside own team.
- High on EQ, humility, curiosity and self drive.
Responsibilities
We use a rubric / career ladder as a framework to help our engineers grow in their careers. Please refer to the SDE IV section in the rubric (https://docs.google.com/spreadsheets/d/1z_51E1mZ3qf0MBuAyfHjKE1HN6h_yghAx2WMaB1z-Ms/edit?usp=sharing">click to view) to understand our expectations on your role and responsibilities.
Your responsibilities include:
- Leading completion and delivery of multiple concurrent projects with clear deliverable timelines and milestones
- Providing key input and guidance in helping to solve our most difficult and challenging issues, and providing recommendations for preventing or mitigating the effects of the issues in the future
- Identifying inconsistencies caused by lack of best practices and patterns, and proactively collaborate with the team to gather input to define and rollout new best practices
- Mentoring other engineers on increasing their impact both within and outside of their team, including identifying opportunities for them to do so
- Participate in shaping longer term technical and architectural goals in consideration of product/company strategy, organizational growth, scalability, security, and other factors
About Tophatter
You’ll spend time on the following Developing high-volume, low-latency applications for mission-critical systems and delivering high-availability and performance. Write maintainable/scalable/efficient code. Contributing in all phases of the development lifecycle. Continue to innovate and build solutions to support business growth and launch new features. Object-Oriented Analysis and design using common design patterns. Given a fuzzy product requirement, work with other stakeholders, product to understand & refine the requirements and change them to tech requirements Work in a cross-functional team, collaborating with peers during entire SDLC. Managing the developed applications and live support.
|
We’re Excited About You If You Have Excellent coding skills in Java/J2EE Strong problem-solving skills, data structures and algorithms. Have experience in building and maintaining large scale web applications in service-oriented architecture. Good Understanding of Java 8 with proven hands-on skills Experience in Spring Framework - Spring Boot, Spring Data, Spring REST Experience in Spring Reactive Good understanding of SQL and MYSQL Knowledge of professional software engineering practices & best practices for the full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations Understands version control branching strategies Understands different major language paradigms (object oriented / functional /reactive) Understanding of AWS is a plus Can reason about complexity of algorithms (time and space) Can debug code at ease and ensure quality code is produced Good verbal and written communication skills
|