Test Engineer - AI/ML

Title: AI/ML Test Engineer – GenAI

Location - Mumbai

Technical Skills

• Strong experience in Generative AI, LLMs, and Agentic AI systems

• Hands-on expertise with AI evaluation frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)

• Proficiency in Python and AI/ML development libraries

• Knowledge of Prompt Engineering, prompt testing, and optimization

Ability to define and track evaluation metrics such as accuracy, relevance, groundedness, hallucination rate, latency, and user satisfaction

• Experience in creating automated evaluation pipelines and benchmarking frameworks

• Strong understanding of AI safety, guardrails, bias testing, and responsible AI practices

• Familiarity with REST APIs, JSON, vector databases, and knowledge retrieval systems

• Experience in A/B testing, human-in-the-loop evaluation, and red teaming

• Strong experience in Manual Testing of AI/GenAI applications, including functional, exploratory, UAT, regression, and end-to-end testing

• Expertise in validating Agent Reasoning, Tool Calling, Workflow Execution, and Response Quality

• Hands-on experience in Automation Testing using Python frameworks

Key Responsibilities

• Design, execute, and automate evaluation strategies for Agentic AI applications.
• Develop evaluation datasets, test cases, and benchmark suites.
• Measure and improve agent performance, reasoning quality, tool usage, and workflow effectiveness.
• Analyze model outputs and identify hallucinations, biases, safety risks, and failure patterns.
• Collaborate with AI Engineers, Product Teams, and Domain Experts to improve agent quality and reliability.
• Generate evaluation reports, dashboards, and actionable recommendations.

Required Skills:-

• 4+ years of experience in Software Testing (Manual & Automation).
• Strong experience with Selenium or Playwright.
• Hands-on experience in API Testing (Postman, REST Assured, Swagger, etc.).
• Experience with CI/CD pipelines (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• Strong understanding of Functional, Regression, and Non-functional Testing.
• Excellent analytical and debugging skills.

AI-Specific Skills

• Hands-on experience in GenAI / AI Testing.
• Experience testing LLM-based applications.
• Strong understanding of prompt engineering and prompt validation.
• Knowledge of LLM behavior, model variability, and non-deterministic outputs.
• Experience validating AI outputs for accuracy, hallucinations, bias, and safety.
• Test data management for AI applications.
• Understanding of Responsible AI testing concepts.

Strong QA Engineer Profile with manual + automation testing across web, API, and AI-driven features

2

Mandatory (Experience 1): Must have 2+ years in QA / software testing, covering both manual and automation testing

3

Mandatory (Experience 2): Must have experience designing, writing, and maintaining test plans, test cases, and test data for web, API, and AI-driven features

4

Mandatory (Experience 3): Must have experience performing functional, regression, integration, and exploratory testing across releases

5

Mandatory (Tech skill 1): Must have hands-on automation experience with a framework such as Selenium, Playwright, Cypress, or similar

6

Mandatory (Tech skill 2): Experience testing AI/ML or data products (validating accuracy, consistency, edge cases, bias, graceful failure handling)

7

Mandatory (Tech skill 3): Must have API testing experience with tools like Postman, REST Assured, or equivalent, and experience testing REST APIs and data pipelines

8

Mandatory (Tech skill 4): Must have working knowledge of Python, JavaScript, or Java for test automation

9

Mandatory (Tech skill 5): Must have SQL and data skills — able to write queries to validate data and back-end behaviour

10

Mandatory (Skill 1): Must have a strong grasp of QA methodologies, SDLC/STLC, and Agile/Scrum practices.

11

Mandatory (Skill 2): Must have an analytical mindset with sharp attention to detail and clear defect communication

12

Mandatory (Skill 3): Must have strong communication skills

13

Preferred (CI/CD): Experience integrating automated tests into CI/CD pipelines (Jenkins, GitHub Actions)

14

Preferred (Domain): Cloud/AWS and insurance domain exposure

Strong QA Engineer Profile with manual + automation testing across web, API, and AI-driven features

2

Mandatory (Experience 1): Must have 2+ years in QA / software testing, covering both manual and automation testing

3

Mandatory (Experience 2): Must have experience designing, writing, and maintaining test plans, test cases, and test data for web, API, and AI-driven features

4

Mandatory (Experience 3): Must have experience performing functional, regression, integration, and exploratory testing across releases

5

Mandatory (Tech skill 1): Must have hands-on automation experience with a framework such as Selenium, Playwright, Cypress, or similar

6

Mandatory (Tech skill 2): Experience testing AI/ML or data products (validating accuracy, consistency, edge cases, bias, graceful failure handling)

7

Mandatory (Tech skill 3): Must have API testing experience with tools like Postman, REST Assured, or equivalent, and experience testing REST APIs and data pipelines

8

Mandatory (Tech skill 4): Must have working knowledge of Python, JavaScript, or Java for test automation

9

Mandatory (Tech skill 5): Must have SQL and data skills — able to write queries to validate data and back-end behaviour

10

Mandatory (Skill 1): Must have a strong grasp of QA methodologies, SDLC/STLC, and Agile/Scrum practices.

11

Mandatory (Skill 2): Must have an analytical mindset with sharp attention to detail and clear defect communication

12

Mandatory (Skill 3): Must have strong communication skills

13

Preferred (CI/CD): Experience integrating automated tests into CI/CD pipelines (Jenkins, GitHub Actions)

14

Preferred (Domain): Cloud/AWS and insurance domain exposure

Role Overview:

As an AI Executor/AI Automation Engineer, you will be responsible for designing and integrating AI capabilities into production systems using Python and key ML libraries. This role requires a strong backend development foundation and a proven track record of deploying AI use cases using tools like TensorFlow, Keras, or OpenAI APIs. You'll work cross-functionally to deliver scalable AI-driven solutions.

Key Responsibilities:

• Design and develop backend solutions using Python, with a focus on AI-driven features.
• Implement and integrate AI/ML models using tools like OpenAI, Hugging Face, or Lang Chain.
• Use core Python libraries (NumPy, Pandas, TensorFlow, Keras) to process data, train, or implement models.
• Translate business needs into AI use cases and deliver working solutions.
• Collaborate with product, engineering, and data teams to define integration workflows.
• Develop REST APIs and micro services to deploy AI components within applications.
• Maintain and optimize AI systems for scalability, performance, and reliability.
• Keep pace with advancements in the AI/ML landscape and evaluate tools for continuous improvement.

Required Skills & Qualifications:

• 2+ years of professional experience as an AI/ML Engineer, including strong backend development expertise in Python.
• Proficiency in libraries such as NumPy, Pandas, TensorFlow, and Keras
• Practical exposure to AI platforms/APIs (e.g., OpenAI, LangChain, Hugging Face)
• Solid understanding of REST APIs, micro services, and integration practices
• Ability to work independently in a remote setup with strong communication and ownership
• Excellent problem-solving and debugging capabilities
• Experience with the MERN stack will be an added advantage.

About NonStop io Technologies

NonStop io Technologies is a value-driven company with a strong focus on process-oriented software engineering. We specialize in Product Development and have a decade's worth of experience in building web and mobile applications across various domains. NonStop io Technologies follows core principles that guide its operations and believes in staying invested in a product's vision for the long term. We are a small but proud group of individuals who believe in the 'givers gain' philosophy and strive to provide value in order to seek value. We are committed to and specialize in building cutting-edge technology products and serving as trusted technology partners for startups and enterprises. We pride ourselves on fostering innovation, learning, and community engagement. Join us to work on impactful projects in a collaborative and vibrant environment.

Brief Description:

We're seeking an AI/ML Engineer to join our team. As AI/ML Engineer, you will be responsible for designing, developing, and implementing artificial intelligence (AI) and machine learning (ML) solutions to solve real-world business problems. You will work closely with engineering teams, including software engineers, domain experts, and product managers, to deploy and integrate Applied AI/ML solutions into the products that are being built at NonStop io. Your role will involve researching cutting-edge algorithms and data processing techniques, and implementing scalable solutions to drive innovation and improve the overall user experience.

Responsibilities

● Applied AI/ML engineering; Building engineering solutions on top of the AI/ML tooling available in the industry today. Eg: Engineering APIs around OpenAI

● AI/ML Model Development: Design, develop, and implement machine learning models and algorithms that address specific business challenges, such as natural language processing, computer vision, recommendation systems, anomaly detection, etc.

● Data Preprocessing and Feature Engineering: Cleanse, preprocess, and transform raw data into suitable formats for training and testing AI/ML models. Perform feature engineering to extract relevant features from the data

● Model Training and Evaluation: Train and validate AI/ML models using diverse datasets to achieve optimal performance. Employ appropriate evaluation metrics to assess model accuracy, precision, recall, and other relevant metrics

● Data Visualization: Create clear and insightful data visualizations to aid in understanding data patterns, model behaviour, and performance metrics

● Deployment and Integration: Collaborate with software engineers and DevOps teams to deploy AI/ML models into production environments and integrate them into various applications and systems

● Data Security and Privacy: Ensure compliance with data privacy regulations and implement security measures to protect sensitive information used in AI/ML processes

● Continuous Learning: Stay updated with the latest advancements in AI/ML research, tools, and technologies, and apply them to improve existing models and develop novel solutions

● Documentation: Maintain detailed documentation of the AI/ML development process, including code, models, algorithms, and methodologies for easy understanding and future reference.

Qualifications & Skills

● Bachelor's, Master's, or PhD in Computer Science, Data Science, Machine Learning, or a related field. Advanced degrees or certifications in AI/ML are a plus

● Proven experience as an AI/ML Engineer, Data Scientist, or related role, ideally with a strong portfolio of AI/ML projects

● Proficiency in programming languages commonly used for AI/ML. Preferably Python

● Familiarity with popular AI/ML libraries and frameworks, such as TensorFlow, PyTorch, scikit-learn, etc.

● Familiarity with popular AI/ML Models such as GPT3, GPT4, Llama2, BERT etc.

● Strong understanding of machine learning algorithms, statistics, and data structures

● Experience with data preprocessing, data wrangling, and feature engineering

● Knowledge of deep learning architectures, neural networks, and transfer learning

● Familiarity with cloud platforms and services (e.g., AWS, Azure, Google Cloud) for scalable AI/ML deployment

● Solid understanding of software engineering principles and best practices for writing maintainable and scalable code

● Excellent analytical and problem-solving skills, with the ability to think critically and propose innovative solutions

● Effective communication skills to collaborate with cross-functional teams and present complex technical concepts to non-technical stakeholders

About Us

We’re building the next generation of AI-powered business software, and we’re looking for people who want to shape that future with us. With Lumen, we’re reimagining how users interact with CRM — moving beyond screens, menus and dashboards to an intelligent interface where users can simply ask AI to take actions, retrieve knowledge, generate insights and get work done. With Agent Studio, we’re enabling businesses to build, test and deploy their own AI agents for real-world workflows. And with Invorto, we’re bringing AI to voice, allowing businesses to create intelligent voice agents tailored to their customer and operational use cases.

What makes this especially exciting is the stage and scale of the opportunity. You’ll get to work on genuinely hard problems across LLMs, agents, reasoning, orchestration, voice AI, evaluation, reliability and enterprise security — not as isolated experiments, but as products used in real business workflows. You’ll have the opportunity to build zero-to-one, own meaningful parts of the product end-to-end, work closely with customers, experiment rapidly, and see your work reach production at scale.

Why join now? Because the playbook for enterprise AI is still being written. You won’t just be implementing someone else’s roadmap — you’ll help define the product, architecture and experiences that become that playbook. Expect high ownership, fast iteration, hard technical and product problems, direct customer impact, and the chance to build AI systems that have to work reliably in the real world — not just in a demo.

About the Role

We are looking for a QA Engineer who specializes in testing agentic AI platforms. You will design and automate quality processes for systems that involve LLMs, autonomous agents, tool use and orchestration across Lumen and Agent Studio — ensuring that AI-driven workflows behave reliably, safely and predictably in production, not just in a demo.

What You’ll Do

• Design and build automated test suites and evaluation frameworks for agentic AI workflows, including multi-step and tool-calling behaviors.
• Use AI/LLM-based QA tools and evaluation frameworks to test model outputs, agent decisions and end-to-end task completion at scale.
• Define quality metrics and benchmarks for agent reliability, correctness, latency and safety, and track them over releases.
• Identify edge cases, failure modes and regressions specific to non-deterministic AI systems, and build automated checks to catch them early.
• Integrate automated agent/LLM testing into CI/CD pipelines to support fast, reliable iteration.
• Partner closely with AI/ML and backend engineers to reproduce issues, root-cause failures and validate fixes.
• Work with customers and customer-facing teams to understand real-world usage patterns and translate them into test scenarios.

What We’re Looking For

• 2–4 years of QA/test automation experience, including hands-on work testing agentic AI or LLM-based platforms.
• Practical experience using AI-focused QA/evaluation tools to test agent behavior, prompts and model outputs.
• Strong scripting/automation skills (Python preferred) to build and maintain test frameworks.
• Understanding of how LLM-based agents work — tool calling, orchestration, memory, reasoning chains — well enough to design meaningful test cases.
• Comfort working with non-deterministic systems and designing evaluation approaches beyond traditional pass/fail testing.
• Strong communication skills — this is a customer-facing role, and you will be expected to clearly articulate technical concepts, decisions and trade-offs to both technical and non-technical stakeholders, including customers.

Good to Have

• Experience testing voice AI or real-time conversational systems.
• Familiarity with CRM or enterprise SaaS platforms.
• Exposure to enterprise security or compliance testing for AI systems.

Kody Technolab Limited is seeking an experienced AI/ML Engineer to design, develop, and deploy cutting-edge Artificial Intelligence and Machine Learning solutions. The ideal candidate will have

strong expertise in Machine Learning, Deep Learning, Generative AI, LLMs, MLOps, and cloud-based AI deployments.

Key Responsibilities

• Design, develop, and deploy Machine Learning and Deep Learning models for classification, regression, recommendation systems, NLP, Computer Vision, and Generative AI applications.

• Build and maintain end-to-end ML pipelines including data preprocessing, feature engineering, model training, validation, evaluation, and deployment.

• Develop AI solutions using PyTorch, TensorFlow, Scikit-learn, Hugging Face, and related frameworks.

• Work with Large Language Models (LLMs) and foundation models such as GPT, BERT, Llama, Claude, and Stable Diffusion.

• Collaborate with product, engineering, and business teams to translate requirements into scalable AI solutions.

• Optimize model performance, scalability, and reliability for production environments.

• Implement MLOps best practices using tools such as MLflow, Docker, Kubernetes, and Kubeflow.

• Stay updated with emerging trends and research in AI, ML, Deep Learning, and Generative AI.

Required Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, Artificial Intelligence, Mathematics, or a related field.

• 7+ years of hands-on experience in AI/ML product development.

• Strong proficiency in Python and ML frameworks including Scikit-learn, TensorFlow, PyTorch, and Hugging Face.

• Experience with Generative AI, LLMs, GANs, VAEs, diffusion models, and prompt engineering.

• Strong understanding of the ML lifecycle including model training, tuning, deployment, monitoring, and optimization.

• Experience with MLOps tools such as MLflow, Docker, Kubeflow, and CI/CD pipelines.

• Experience with AWS, Azure, or GCP cloud platforms.

• Strong problem-solving and analytical skills.

Preferred Skills

• Fine-tuning and deployment of Large Language Models.

• Experience with RAG (Retrieval Augmented Generation) architectures.

• Contributions to open-source AI projects or research publications.

• Knowledge of model interpretability, data annotation, and feature engineering.

• C++ experience for high-performance AI applications.

Why Join Kody Technolab Limited?

Opportunity to work on innovative AI products, Generative AI solutions, robotics integrations,

and enterprise-scale applications while collaborating with a highly skilled technology team.

Visit the Website to know more about us.

Company Website - Kody Technolab | Deep Tech Company in Robotics & AI Solution

Kody Robots | Robotics Company in India for Autonomous Robots

AI Engineer

We are seeking an AI Engineering specialist focused on AI evaluation, and continuous quality improvement for Ezra MetLife's employee-facing AI platform. This role will establish and scale the testing strategy for enterprise AI agents, ensuring high response quality, reliability, and production readiness. The engineer will build automated regression testing framework (preferred Playwright ), define AI evaluation methodologies, analyze AI performance metrics, and partner with engineering teams to continuously improve answer quality, grounding accuracy, and customer experience. This position is critical to enabling confidence as Ezra expands its AI agent portfolio and employee-facing capabilities.

Required Skills & Experience

• C# and .NET development experience

• Experience with at least one AI evaluation framework (e.g., prompt evaluation, RAG evaluation, LLM quality assessment)

• Microsoft Agent Framework (preferred) or similar enterprise agent frameworks

• Experience with Azure OpenAI / Azure AI Foundry

• Microsoft 365 Agent SDK

• Azure AI Search, RAG pipelines, and retrieval quality testing

• Infrastructure as Code using Terraform

• Experience building automated testing and AI quality validation processes

• Familiarity with telemetry analysis, AI observability, and performance measurement

• Strong analytical skills with a passion for improving AI response quality and reliability

Skills: AI Agents~Core .NET Technologies~C# 5.0

Experience Required: 10 & Above

Location: Hyderabad :5+ relevant exp in AI + .NET

Strong AI Engineer / Machine Learning Engineer profiles.

2

Mandatory (Experience 1) – Must have minimum 3+ years of hands-on experience in Data Science, Machine Learning, Applied AI, NLP, Deep Learning, or Generative AI solutions.

3

Mandatory (Experience 2) – Must have strong hands-on experience in Python programming, SQL, data analysis, feature engineering, model development, and production-grade ML applications.

4

Mandatory (Experience 3) – Must have experience working with Machine Learning and Deep Learning frameworks such as PyTorch, TensorFlow, Keras, Scikit-learn, or equivalent.

5

Mandatory (Experience 4) – Must have hands-on experience working on NLP, embeddings, semantic search, text classification, document understanding, recommendation systems, or similar AI/ML use cases.

6

Mandatory (Experience 5) – Must have experience working with Large Language Models (LLMs) such as GPT, Llama, Mistral, Claude, Gemini, Phi, or similar foundation models.

7

Mandatory (Experience 6) – Must have hands-on experience building or implementing RAG (Retrieval Augmented Generation) systems, vector search, knowledge retrieval, embeddings, chunking, indexing, or semantic retrieval solutions.

8

Mandatory (Experience 7) – Must have experience working with Git, CI/CD practices, production environments, and scalable AI/ML systems.

9

Mandatory (CTC) – The CTC breakup offered will be 75% fixed + 25% variable, as per company policy.

10

Mandatory (Age) - Candidate's Age should be below 28 Years