AI QA Engineer

About Us

We’re building the next generation of AI-powered business software, and we’re looking for people who want to shape that future with us. With Lumen, we’re reimagining how users interact with CRM — moving beyond screens, menus and dashboards to an intelligent interface where users can simply ask AI to take actions, retrieve knowledge, generate insights and get work done. With Agent Studio, we’re enabling businesses to build, test and deploy their own AI agents for real-world workflows. And with Invorto, we’re bringing AI to voice, allowing businesses to create intelligent voice agents tailored to their customer and operational use cases.

What makes this especially exciting is the stage and scale of the opportunity. You’ll get to work on genuinely hard problems across LLMs, agents, reasoning, orchestration, voice AI, evaluation, reliability and enterprise security — not as isolated experiments, but as products used in real business workflows. You’ll have the opportunity to build zero-to-one, own meaningful parts of the product end-to-end, work closely with customers, experiment rapidly, and see your work reach production at scale.

Why join now? Because the playbook for enterprise AI is still being written. You won’t just be implementing someone else’s roadmap — you’ll help define the product, architecture and experiences that become that playbook. Expect high ownership, fast iteration, hard technical and product problems, direct customer impact, and the chance to build AI systems that have to work reliably in the real world — not just in a demo.

About the Role

We are looking for a QA Engineer who specializes in testing agentic AI platforms. You will design and automate quality processes for systems that involve LLMs, autonomous agents, tool use and orchestration across Lumen and Agent Studio — ensuring that AI-driven workflows behave reliably, safely and predictably in production, not just in a demo.

What You’ll Do

• Design and build automated test suites and evaluation frameworks for agentic AI workflows, including multi-step and tool-calling behaviors.
• Use AI/LLM-based QA tools and evaluation frameworks to test model outputs, agent decisions and end-to-end task completion at scale.
• Define quality metrics and benchmarks for agent reliability, correctness, latency and safety, and track them over releases.
• Identify edge cases, failure modes and regressions specific to non-deterministic AI systems, and build automated checks to catch them early.
• Integrate automated agent/LLM testing into CI/CD pipelines to support fast, reliable iteration.
• Partner closely with AI/ML and backend engineers to reproduce issues, root-cause failures and validate fixes.
• Work with customers and customer-facing teams to understand real-world usage patterns and translate them into test scenarios.

What We’re Looking For

• 2–4 years of QA/test automation experience, including hands-on work testing agentic AI or LLM-based platforms.
• Practical experience using AI-focused QA/evaluation tools to test agent behavior, prompts and model outputs.
• Strong scripting/automation skills (Python preferred) to build and maintain test frameworks.
• Understanding of how LLM-based agents work — tool calling, orchestration, memory, reasoning chains — well enough to design meaningful test cases.
• Comfort working with non-deterministic systems and designing evaluation approaches beyond traditional pass/fail testing.
• Strong communication skills — this is a customer-facing role, and you will be expected to clearly articulate technical concepts, decisions and trade-offs to both technical and non-technical stakeholders, including customers.

Good to Have

• Experience testing voice AI or real-time conversational systems.
• Familiarity with CRM or enterprise SaaS platforms.
• Exposure to enterprise security or compliance testing for AI systems.

Required Skills:-

• 4+ years of experience in Software Testing (Manual & Automation).
• Strong experience with Selenium or Playwright.
• Hands-on experience in API Testing (Postman, REST Assured, Swagger, etc.).
• Experience with CI/CD pipelines (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
• Strong understanding of Functional, Regression, and Non-functional Testing.
• Excellent analytical and debugging skills.

AI-Specific Skills

• Hands-on experience in GenAI / AI Testing.
• Experience testing LLM-based applications.
• Strong understanding of prompt engineering and prompt validation.
• Knowledge of LLM behavior, model variability, and non-deterministic outputs.
• Experience validating AI outputs for accuracy, hallucinations, bias, and safety.
• Test data management for AI applications.
• Understanding of Responsible AI testing concepts.

AI Engineer

We are seeking an AI Engineering specialist focused on AI evaluation, and continuous quality improvement for Ezra MetLife's employee-facing AI platform. This role will establish and scale the testing strategy for enterprise AI agents, ensuring high response quality, reliability, and production readiness. The engineer will build automated regression testing framework (preferred Playwright ), define AI evaluation methodologies, analyze AI performance metrics, and partner with engineering teams to continuously improve answer quality, grounding accuracy, and customer experience. This position is critical to enabling confidence as Ezra expands its AI agent portfolio and employee-facing capabilities.

Required Skills & Experience

• C# and .NET development experience

• Experience with at least one AI evaluation framework (e.g., prompt evaluation, RAG evaluation, LLM quality assessment)

• Microsoft Agent Framework (preferred) or similar enterprise agent frameworks

• Experience with Azure OpenAI / Azure AI Foundry

• Microsoft 365 Agent SDK

• Azure AI Search, RAG pipelines, and retrieval quality testing

• Infrastructure as Code using Terraform

• Experience building automated testing and AI quality validation processes

• Familiarity with telemetry analysis, AI observability, and performance measurement

• Strong analytical skills with a passion for improving AI response quality and reliability

Skills: AI Agents~Core .NET Technologies~C# 5.0

Experience Required: 10 & Above

Location: Hyderabad :5+ relevant exp in AI + .NET

Strong QA Engineer Profile with manual + automation testing across web, API, and AI-driven features

2

Mandatory (Experience 1): Must have 2+ years in QA / software testing, covering both manual and automation testing

3

Mandatory (Experience 2): Must have experience designing, writing, and maintaining test plans, test cases, and test data for web, API, and AI-driven features

4

Mandatory (Experience 3): Must have experience performing functional, regression, integration, and exploratory testing across releases

5

Mandatory (Tech skill 1): Must have hands-on automation experience with a framework such as Selenium, Playwright, Cypress, or similar

6

Mandatory (Tech skill 2): Experience testing AI/ML or data products (validating accuracy, consistency, edge cases, bias, graceful failure handling)

7

Mandatory (Tech skill 3): Must have API testing experience with tools like Postman, REST Assured, or equivalent, and experience testing REST APIs and data pipelines

8

Mandatory (Tech skill 4): Must have working knowledge of Python, JavaScript, or Java for test automation

9

Mandatory (Tech skill 5): Must have SQL and data skills — able to write queries to validate data and back-end behaviour

10

Mandatory (Skill 1): Must have a strong grasp of QA methodologies, SDLC/STLC, and Agile/Scrum practices.

11

Mandatory (Skill 2): Must have an analytical mindset with sharp attention to detail and clear defect communication

12

Mandatory (Skill 3): Must have strong communication skills

13

Preferred (CI/CD): Experience integrating automated tests into CI/CD pipelines (Jenkins, GitHub Actions)

14

Preferred (Domain): Cloud/AWS and insurance domain exposure

Strong QA Engineer Profile with manual + automation testing across web, API, and AI-driven features

2

Mandatory (Experience 1): Must have 2+ years in QA / software testing, covering both manual and automation testing

3

Mandatory (Experience 2): Must have experience designing, writing, and maintaining test plans, test cases, and test data for web, API, and AI-driven features

4

Mandatory (Experience 3): Must have experience performing functional, regression, integration, and exploratory testing across releases

5

Mandatory (Tech skill 1): Must have hands-on automation experience with a framework such as Selenium, Playwright, Cypress, or similar

6

Mandatory (Tech skill 2): Experience testing AI/ML or data products (validating accuracy, consistency, edge cases, bias, graceful failure handling)

7

Mandatory (Tech skill 3): Must have API testing experience with tools like Postman, REST Assured, or equivalent, and experience testing REST APIs and data pipelines

8

Mandatory (Tech skill 4): Must have working knowledge of Python, JavaScript, or Java for test automation

9

Mandatory (Tech skill 5): Must have SQL and data skills — able to write queries to validate data and back-end behaviour

10

Mandatory (Skill 1): Must have a strong grasp of QA methodologies, SDLC/STLC, and Agile/Scrum practices.

11

Mandatory (Skill 2): Must have an analytical mindset with sharp attention to detail and clear defect communication

12

Mandatory (Skill 3): Must have strong communication skills

13

Preferred (CI/CD): Experience integrating automated tests into CI/CD pipelines (Jenkins, GitHub Actions)

14

Preferred (Domain): Cloud/AWS and insurance domain exposure

Total exp: 5-8 Years

Client: MRI Software

Location: Vadodara

Budget: Look for decent with experience

Mode interview: 1 Virtual

Mode of Work: Hybrid

QA Engineer:

Job Summary/Objective 

Quality Assurance Engineers at MRI collaborate closely with Product Management and Product Development teams to deliver high-quality, high-performance products for our clients. In this role, QA Engineers are responsible for creating and contributing to the documentation of user stories and acceptance criteria, working alongside product owners and development teams. They define manual and automation testing strategies, tools, scenarios, and plans for both web and mobile products, ensuring comprehensive functional and performance testing. 

 QA Engineers will execute static, dynamic, and performance testing while providing detailed reporting and feedback throughout the development lifecycle. They are expected to demonstrate strong technical skills, including expertise in back-end development, database queries, and keen attention to detail. Effective written, verbal, and organizational communication is essential. 

 In addition, QA Engineers will leverage AI to optimize and automate repetitive tasks, thereby enhancing the overall efficiency and effectiveness of QA processes, ensuring a faster, more accurate testing cycle.

Tools and technology:

1.     Tools: Jira, Xray, TFVC, Git

2.     API: REST, SOAP, SoapUI, Postman

3.     Automation: C#, JavaScript, TypeScript, Ruby, Playwright, SpecFlow

4.     Database: SQL

About the team

SecurITe’s mission is to build an Agentic‑AI driven security platform that protects critical infrastructure from modern cyber threats. Our focus is on delivering highly performant, resilient, and intelligent network security systems that help defenders stay ahead of adversaries.

About the Role

We’re looking for a seasoned Senior Quality Engineer to provide technical leadership and architectural oversight for our next‑generation cybersecurity AI platform. In this high-impact role, you will define the technical strategy for quality assurance, ensuring our agentic AI transforms cyber defense with unparalleled reliability.

You will be responsible for the end-to-end quality lifecycle, from architectural reviews to the deployment of scalable automation frameworks. Beyond technical execution, you will serve as a mentor to junior team members, fostering a culture of technical excellence and driving the strategy that ensures our solutions meet the rigorous demands of critical infrastructure protection.

What You’ll Do

●    Defining and driving comprehensive QA strategies and roadmaps for the cybersecurity platform.

●    Designing, developing, and executing test plans, test cases, and automated scripts to ensure software quality.

●    Performing functional, regression, performance, scalability and security testing to identify bugs or defects.

●    Collaborating with developers, product managers, and other stakeholders to understand product requirements and testing needs.

●    Identifying, documenting, and tracking software defects, ensuring clear communication of issues and their resolutions.

●    Leading deep-dive root-cause analysis for critical system defects and security vulnerabilities.

●    Conducting thorough reviews of product specifications and software design to identify potential areas of concern before testing.

●    Architecting and designing complex, scalable test automation frameworks to optimize CI/CD velocity.

●    Ensuring the software meets customer and business requirements by validating the functionality and performance.

●    Assisting in continuously improving QA processes, tools, and best practices to enhance software testing efficiency and effectiveness.

●    Supporting user acceptance testing (UAT) and assisting clients with product validation.

●    Mentoring junior and mid-level engineers, providing technical guidance and conducting architectural reviews.

Required Experience

●    A Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or a related field.

●    8-10 years of proven experience in quality engineering, specifically within network cybersecurity, Identity Providers, or AI-integrated platforms.

●    Expertise in manual and automated testing.

●    Deep domain expertise in complex system validation and advanced automation practices at scale.

●    Proficiency in programming languages like Python to build and run automated test scripts.

●    "Strong knowledge of software testing methodologies, performance testing tools (e.g., JMeter, k6), and security traffic generation/simulation tools (e.g., Ixia BreakingPoint, Scapy, or Snort/Suricata traffic generators)."

●    Understanding of continuous integration/continuous deployment (CI/CD) pipelines and version control systems like Git.

●    Strong communication skills for documenting test results and interacting with cross-functional teams.

●    Excellent analytical skills, attention to detail, and problem-solving ability.

●    Ability to work independently as well as collaboratively in a team environment.

●    A curious mindset with a willingness to quickly learn new technologies and testing tools.

Required Skills & Qualifications

●    Familiarity with cloud-based testing environments (GCP, AWS, Azure).

●    Experience with cybersecurity products or cloud services or IDP or Web UI

The Mindset

●    Problem Solver: You thrive on complex, ambiguous challenges and engineer elegant solutions.

●    Ownership‑Driven: You take initiative, move fast, and deliver outcomes without hand‑holding.

●    Continuous Learner: You stay ahead of the curve in AI, ML, and emerging technologies.

●    Startup DNA: You excel in fast‑moving environments where priorities evolve and impact is immediate.

Total exp: 5-8 Years

Client: MRI Software

Location: Vadodara

Budget: Look for decent with experience

Mode interview: 1 Virtual

Mode of Work: Hybrid

QA Engineer:

Job Summary/Objective 

Quality Assurance Engineers at MRI collaborate closely with Product Management and Product Development teams to deliver high-quality, high-performance products for our clients. In this role, QA Engineers are responsible for creating and contributing to the documentation of user stories and acceptance criteria, working alongside product owners and development teams. They define manual and automation testing strategies, tools, scenarios, and plans for both web and mobile products, ensuring comprehensive functional and performance testing. 

 QA Engineers will execute static, dynamic, and performance testing while providing detailed reporting and feedback throughout the development lifecycle. They are expected to demonstrate strong technical skills, including expertise in back-end development, database queries, and keen attention to detail. Effective written, verbal, and organizational communication is essential. 

 In addition, QA Engineers will leverage AI to optimize and automate repetitive tasks, thereby enhancing the overall efficiency and effectiveness of QA processes, ensuring a faster, more accurate testing cycle.

Tools and technology:

1.     Tools: Jira, Xray, TFVC, Git

2.     API: REST, SOAP, SoapUI, Postman

3.     Automation: C#, JavaScript, TypeScript, Ruby, Playwright, SpecFlow

4.     Database: SQL

Job title: Chief Agentic Quality Architect

Type: Full-Time | Contract

Location: Remote

Role Overview

Equity Partners builds profitable growth by acquiring and operating enterprise software companies — refining a proprietary operating model across 40+ acquisitions and two decades of hands-on experience, now supercharged by our patented agentic AI platform. We're hiring a Chief Agentic Quality Architect to lead the transition from traditional scripted testing to an AI-augmented quality ecosystem. In this role, you'll audit and rebuild our quality engineering foundations, deploy agentic automation across critical business flows, and build the guardrails needed to keep AI-generated code production-ready.

Key Skills

• 10+ years in QA automation engineering, SDET, or test architecture roles
• Expert-level proficiency in Playwright, Cypress, or Selenium
• Hands-on experience using LLMs (Claude, GPT-4, etc.) and agentic frameworks to generate code or automate workflows
• Deep understanding of integrating quality gates into AWS-based CI/CD pipelines or similar environments
• Architectural mindset, with the ability to design "Behavioral Snapshots" to safeguard critical business logic during rapid transformation

Responsibilities

• Conduct a comprehensive audit of the existing test estate across unit, integration, API, UI, sanity, and regression layers, producing a Current State & Gap Coverage Report
• Architect and own a phased quality engineering roadmap across two-week, one-month, and three-month delivery horizons
• Deploy agentic test generation — transforming business requirements into executable Playwright/Cypress scripts, generating synthetic test data, and implementing self-healing automation
• Design regression strategies and quality gates specifically tuned to catch hallucinations and logic errors in AI-generated code
• Build, mentor, and upskill a specialist QA team fluent in AI-assisted testing and agentic automation frameworks

If you're ready to rewrite the rules of quality for an AI-native world, we want you leading the charge.

Title: AI/ML Test Engineer – GenAI

Location - Mumbai

Technical Skills

• Strong experience in Generative AI, LLMs, and Agentic AI systems

• Hands-on expertise with AI evaluation frameworks (RAGAS, DeepEval, TruLens, LangSmith, Promptfoo, etc.)

• Proficiency in Python and AI/ML development libraries

• Knowledge of Prompt Engineering, prompt testing, and optimization

Ability to define and track evaluation metrics such as accuracy, relevance, groundedness, hallucination rate, latency, and user satisfaction

• Experience in creating automated evaluation pipelines and benchmarking frameworks

• Strong understanding of AI safety, guardrails, bias testing, and responsible AI practices

• Familiarity with REST APIs, JSON, vector databases, and knowledge retrieval systems

• Experience in A/B testing, human-in-the-loop evaluation, and red teaming

• Strong experience in Manual Testing of AI/GenAI applications, including functional, exploratory, UAT, regression, and end-to-end testing

• Expertise in validating Agent Reasoning, Tool Calling, Workflow Execution, and Response Quality

• Hands-on experience in Automation Testing using Python frameworks

Key Responsibilities

• Design, execute, and automate evaluation strategies for Agentic AI applications.
• Develop evaluation datasets, test cases, and benchmark suites.
• Measure and improve agent performance, reasoning quality, tool usage, and workflow effectiveness.
• Analyze model outputs and identify hallucinations, biases, safety risks, and failure patterns.
• Collaborate with AI Engineers, Product Teams, and Domain Experts to improve agent quality and reliability.
• Generate evaluation reports, dashboards, and actionable recommendations.