Data Engineer

Job Type: Full Time

Shift: US Time Zone

About the Role:

We are seeking a talented Senior Python Developer with strong hands-on experience in building agentic systems (multi-agent systems). This role requires expertise in Python, PostgreSQL, and the ability to work on a dynamic platform that supports multi-agent architectures. You will be responsible for developing, enhancing, and maintaining platforms that involve complex agent-based interactions, while also supporting the database systems and overall infrastructure.

Key Responsibilities:

Design, implement, and maintain multi-agent systems to support complex interactions and decision-making processes.

Develop scalable and efficient Python code for integrating and managing agents and their communication layers.

Work extensively with PostgreSQL databases to design schemas, optimize queries, and manage data storage effectively.

Collaborate with cross-functional teams to integrate agentic systems into larger platforms and ensure their smooth operation.

Troubleshoot, debug, and optimize Python-based solutions and database interactions.

Write high-quality, maintainable, and well-documented code.

Provide technical support for the platform, ensuring its scalability and performance over time.

Mentor junior developers and provide technical guidance as needed.

Required Skills and Experience:

5+ years of strong hands-on experience in Python development.

Experience in building and deploying multi-agent systems or agent-based frameworks.

Proficiency in PostgreSQL, including schema design, query optimization, and database management.

Knowledge of agent-based modeling or similar paradigms is highly desirable.

Familiarity with various Python libraries for handling complex algorithms and system integration.

Experience in supporting and maintaining platforms or systems post-deployment.

Strong understanding of software development best practices and version control (Git).

Excellent problem-solving skills and the ability to think algorithmically.

Strong communication skills with the ability to explain complex technical concepts.

Preferred Qualifications:

A degree in Computer Science, Engineering, or a related field.

Experience with cloud platforms (AWS, Azure, etc.) is a plus.

Knowledge of agent communication languages (ACL) or other protocols related to multi-agent systems is beneficial.

Familiarity with Docker, Kubernetes, or other containerization tools.

• Build campaign generation services which can send app notifications at a speed of 10 million a minute
• Dashboards to show Real time key performance indicators to clients
• Develop complex user segmentation engines which creates segments on Terabytes of data within few seconds
• Building highly available & horizontally scalable platform services for ever growing data
• Use cloud based services like AWS Lambda for blazing fast throughput & auto scalability
• Work on complex analytics on terabytes of data like building Cohorts, Funnels, User path analysis, Recency Frequency & Monetary analysis at blazing speed
• You will build backend services and APIs to create scalable engineering systems.
• As an individual contributor, you will tackle some of our broadest technical challenges that requires deep technical knowledge, hands-on software development and seamless collaboration with all functions.
• You will envision and develop features that are highly reliable and fault tolerant to deliver a superior customer experience.
• Collaborating various highly-functional teams in the company to meet deliverables throughout the software development lifecycle.
• Identify and improvise areas of improvement through data insights and research.

• 2-5 years of experience in backend development and must have worked on Java/shell/Perl/python scripting.
• Solid understanding of engineering best practices, continuous integration, and incremental delivery.
• Strong analytical skills, debugging and troubleshooting skills, product line analysis.
• Follower of agile methodology (Sprint planning, working on JIRA, retrospective etc).
• Proficiency in usage of tools like Docker, Maven, Jenkins and knowledge on frameworks in Java like spring, spring boot, hibernate, JPA.
• Ability to design application modules using various concepts like object oriented, multi-threading, synchronization, caching, fault tolerance, sockets, various IPCs, database interfaces etc.
• Hands on experience on Redis, MySQL and streaming technologies like Kafka producer consumers and NoSQL databases like mongo dB/Cassandra.
• Knowledge about versioning like Git and deployment processes like CICD.

• Bachelor's or Master’s degree in Computer Science or equivalent area
• 10 to 20 years of experience in software development
• Hands-on experience designing and building B2B or B2C products
• 3+ years architecting SaaS/Web based customer facing products, leading engineering teams as software/technical architect
• Experiences of engineering practices such as code refactoring, microservices, design and enterprise integration patterns, test and design-driven development, continuous integration, building highly scalable applications, application and infrastructure security
• Strong cloud infrastructure experience with AWS and/or Azure
• Experience building event driven systems and working with message queues/topics
• Broad working experience across multiple programming languages and frameworks with in-depth experience in one or more of the following: .Net, Java, Scala or Go-lang
• Hands-on experience with relational databases like SQL Server, PostgreSQL and document stores like Elasticsearch or MongoDB
• Hands-on experience with Big Data processing technologies like Hadoop/Spark is a plus
• Hands-on experience with container technologies like Docker, Kubernetes
• Knowledge of Agile software development process

Experience:

The candidate should have about 2+ years of experience with design and development in Java/Scala. Experience in algorithm, data-structure, database and distributed System is mandatory.

Required Skills:

Mandatory: -

1. Core Java or Scala
2. Experience in Big Data, Spark
3. Extensive experience in developing spark job. Should possess good Oops knowledge and be aware of enterprise application design patterns.
4. Should have the ability to analyze, design, develop and test complexity of spark job.
5. Working knowledge of Unix/Linux.
6. Hands on experience in Spark, creating RDD, applying operation - transformation-action

Good To have: -

1. Python
2. Spark streaming
3. Py Spark
4. Azure/AWS Cloud Knowledge of Data Storage and Compute side

• Bachelor’s or master’s degree in Computer Engineering, Computer Science, Computer Applications, Mathematics, Statistics, or related technical field. Relevant experience of at least 3 years in lieu of above if from a different stream of education.

• Well-versed in and 3+ hands-on demonstrable experience with: ▪ Stream & Batch Big Data Pipeline Processing using Apache Spark and/or Apache Flink.
  ▪ Distributed Cloud Native Computing including Server less Functions
  ▪ Relational, Object Store, Document, Graph, etc. Database Design & Implementation
  ▪ Micro services Architecture, API Modeling, Design, & Programming

• 3+ years of hands-on development experience in Apache Spark using Scala and/or Java.

• Ability to write executable code for Services using Spark RDD, Spark SQL, Structured Streaming, Spark MLLib, etc. with deep technical understanding of Spark Processing Framework.

• In-depth knowledge of standard programming languages such as Scala and/or Java.

• 3+ years of hands-on development experience in one or more libraries & frameworks such as Apache Kafka, Akka, Apache Storm, Apache Nifi, Zookeeper, Hadoop ecosystem (i.e., HDFS, YARN, MapReduce, Oozie & Hive), etc.; extra points if you can demonstrate your knowledge with working examples.

• 3+ years of hands-on development experience in one or more Relational and NoSQL datastores such as PostgreSQL, Cassandra, HBase, MongoDB, DynamoDB, Elastic Search, Neo4J, etc.

• Practical knowledge of distributed systems involving partitioning, bucketing, CAP theorem, replication, horizontal scaling, etc.

• Passion for distilling large volumes of data, analyze performance, scalability, and capacity performance issues in Big Data Platforms.

• Ability to clearly distinguish system and Spark Job performances and perform spark performance tuning and resource optimization.

• Perform benchmarking/stress tests and document the best practices for different applications.

• Proactively work with tenants on improving the overall performance and ensure the system is resilient, and scalable.

• Good understanding of Virtualization & Containerization; must demonstrate experience in technologies such as Kubernetes, Istio, Docker, OpenShift, Anthos, Oracle VirtualBox, Vagrant, etc.

• Well-versed with demonstrable working experience with API Management, API Gateway, Service Mesh, Identity & Access Management, Data Protection & Encryption.

  Hands-on experience with demonstrable working experience with DevOps tools and platforms viz., Jira, GIT, Jenkins, Code Quality & Security Plugins, Maven, Artifactory, Terraform, Ansible/Chef/Puppet, Spinnaker, etc.

• Well-versed in AWS and/or Azure or and/or Google Cloud; must demonstrate experience in at least FIVE (5) services offered under AWS and/or Azure or and/or Google Cloud in any categories: Compute or Storage, Database, Networking & Content Delivery, Management & Governance, Analytics, Security, Identity, & Compliance (or) equivalent demonstrable Cloud Platform experience.

• Good understanding of Storage, Networks and Storage Networking basics which will enable you to work in a Cloud environment.

• Good understanding of Network, Data, and Application Security basics which will enable you to work in a Cloud as well as Business Applications / API services environment.

Mentor To Go is an android and web based platform to help students improve their work readiness through self-learning sessions, personalised mentorship provided by mentor professionals from a range of careers, and work experience opportunities.

The technology platform implements  functionalities such as: screening surveys of mentors and mentees, training content for mentors, matching through an algorithm, and mentorship through the delivery of structured activities. 

The application technology architecture comprises of the following:

o   a Python Django web application with NGinx and Gunicorn

o   PostgreSQL database

o   A React Android and Web app

o   Rest APIs which interact between the Android app and the Django webserver

Our vision at Mentor To Go is to provide career mentorship to young people anywhere, anytime across the length and breadth of India.

We are looking for a backend software developer who is passionate about using web technologies to solve social problems and creating experiences that are elegant and effective. You will own the back end development of features of the Mentor To Go platform from conceptualisation through design and testing, working closely with other developers, the technical project manager, and the program operations team. You are user-centric, continuously demonstrate strategic & analytical abilities, and are laser focused on executing at speed. You must have the ability to succeed in a fast paced environment, where success is dependent on your ability to collaborate with cross-functional team members in a positive, productive, and transparent way.

Minimum requirements 

• Bachelor’s degree in engineering preferably  CS  or related discipline  
• 3+ years experience working in Unix/Linux environments building web and mobile applications
• Proficiency in developing and deploying cloud based Python/Postgresql applications preferably using frameworks such as Django
• Experience with design and development of moderately complex software projects
• Strong written and oral communication skills
• Familiarity with version control software such as Git

Desired  requirements 

• 5+ years of overall relevant work experience
• Experience building apps using React/React-Native
• Basic understanding of AWS cloud
• Ability to participate in technical discussions and help make technical trade-offs

About Mentor Together

https://mentortogether.org/">Mentor Together is India’s first and largest youth mentoring  non-profit organization with a mission to facilitate empowering mentoring relationships and networks that help young people break the inequalities of opportunity and actualise their potential.

http://bit.ly/mentortogo">Mentor To Go is the world’s first mobile mentoring platform created by Mentor Together with the support of Cisco India, LinkedIn Social Impact, British Telecom and Sterlite Technologies.

• Minimum 7 years of relevant work experience in similar roles.
• Hands-on experience developing and delivering scalable multi-tenant SaaS applications on AWS platform.
• In-depth knowledge of Spring, Spring Boot, Java, REST Web Services, SQL databases, microservices, GRAND stack, SQL and NoSQL databases.
• In-depth knowledge and experience developing and delivering scalable data lakes, data ingestion and processing pipelines, data access microservices.
• In-depth knowledge of AWS platform, tools and services, specifically AWS networking and security, Route53, API Gateway, ECS/Fargate, RDS; Java/Spring development; modern database and data processing technologies; DevOps; microservices architecture; container/Docker technology.
• Outstanding collaboration and communication skills. Ability to effectively collaborate with distributed team.
• Understand and practice agile development methodology.
• Prior experience working in a software product company.
• Prior experience with security product development.

Nice to Have:

• AWS Certified Developer certification is highly desired.
• Prior experience with Apache Spark and Scala.

Spark / Scala experience should be more than 2 years.

Combination with Java & Scala is fine or we are even fine with Big Data Developer with strong Core Java Concepts. - Scala / Spark Developer.

Strong proficiency Scala on Spark (Hadoop) - Scala + Java is also preferred

Complete SDLC process and Agile Methodology (Scrum)

Version control / Git