Gen AI QA Engineer

Principal Software Engineer

Location: Bangalore India

Department: AI Engineering / Quality Assurance

Experience Level: Mid to Senior

We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites, including:
• Unit, integration, and end-to-end (E2E)
• Functional, regression, performance, and safety testing
• Validate AI system behavior, including:
• Non-deterministic LLM outputs
• Hallucinations and edge cases
• Multi-step agent decision-making
• Design and manage evaluation systems:
• Golden datasets
• Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs, including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

Core Engineering

• Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

AI / ML & Agentic Systems

• Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
• RAG architectures and vector databases (Pinecone, Weaviate)
• Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

• Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge of evaluation methods:
• LLM-as-a-judge
• BLEU, ROUGE, semantic similarity scoring
• Experience with prompt and agent regression testing
• Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

• AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
• Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
• Monitoring: Prometheus, Grafana, OpenTelemetry

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus on quality KPIs
• Detail-oriented with a strong bias toward automation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications
• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readiness

Preferred

• Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to:
• Shift-left testing practices
• Production observability and monitoring
• Chaos or resilience testing

Senior-Level Differentiators

Education

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field

Nice-to-have:

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

What Success Looks Like

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation

TECHJAYS

THE AI REIMAGINATION COMPANY

AI Quality Engineer

Position : AI Quality Engineer

Experience : 3–10 Years

Location : Coimbatore / Hybrid

Joining : Immediate / Within 30 Days

Primary SkillsManual & Functional Testing, AI/ML Application Testing, LLM Testing, Prompt Testing, Test Case Design, API Testing, Regression Testing, Defect Management

Preferred SkillsRAG Testing, Chatbot Testing, AI Response Evaluation, Selenium, Playwright, Cypress, Python, JavaScript, SQL, Postman, JIRA, BDD

About Techjays

At Techjays, we're helping businesses reimagine how they build, operate, and scale with AI at the core. We focus on developing intelligent, scalable, and production-grade AI solutions that solve real-world business challenges.

Our global team brings together experienced technology professionals from organizations such as Google, NetApp, ADP, Cognizant Consulting, and Capgemini. We work with global clients to build AI-powered applications, intelligent automation platforms, and enterprise software solutions.

We're looking for an AI Quality Engineer with strong functional testing expertise and prior hands-on experience testing AI/ML-powered applications.

This role goes beyond traditional software testing. You will be responsible for validating the quality, accuracy, reliability, and behaviour of AI-powered features, ensuring that intelligent applications perform consistently across different scenarios and real-world use cases.

The ideal candidate should understand both traditional QA methodologies and the unique testing challenges associated with AI systems, including non-deterministic outputs, prompt behaviour, hallucinations, response relevance, and edge-case validation.

Prior hands-on experience testing AI/ML-based products or applications is mandatory. Candidates with only traditional manual testing experience and no AI project exposure will not be a fit.

Key Responsibilities

1. AI Application & LLM Testing

• Perform functional and quality testing of AI-powered applications, LLM-based products, chatbots, and intelligent automation systems.
• Design test scenarios to validate AI-generated responses for accuracy, relevance, consistency, completeness, and contextual understanding.
• Test prompts, prompt variations, and conversational flows to identify unexpected outputs and behavioural inconsistencies.
• Evaluate AI responses for hallucinations, incorrect information, irrelevant answers, and context retention issues.
• Validate AI application behaviour across different user inputs, edge cases, and complex scenarios.
• Test RAG-based applications by validating document retrieval, response relevance, contextual accuracy, and source grounding.
• Evaluate AI-powered search, recommendation engines, and other intelligent product features.
• Collaborate with AI engineers and data science teams to define acceptance criteria and quality benchmarks.

2. Functional & Manual Testing

• Analyse business and technical requirements and convert them into comprehensive test scenarios and test cases.
• Execute functional, regression, integration, system, and user acceptance testing across web and mobile applications.
• Validate end-to-end application workflows, business logic, user interactions, and data processing.
• Identify functional gaps, unexpected behaviour, and usability issues.
• Perform exploratory testing to uncover defects that may not be covered by predefined test cases.
• Ensure application functionality remains stable across different releases and environments.
• Review requirements early in the development lifecycle to identify potential quality risks.

3. AI Quality Evaluation & Validation

• Develop structured testing approaches for non-deterministic AI outputs using behaviour-based, range-based, and scenario-based validation.
• Evaluate AI response quality against expected outcomes and predefined acceptance criteria.
• Validate prompt sensitivity, context handling, conversation memory, and multi-turn interactions.
• Identify failure patterns involving hallucinations, prompt injection, incorrect retrieval, and inconsistent responses.
• Test AI system performance across different input formats, languages, and user scenarios where applicable.
• Maintain test datasets and evaluation scenarios for repeatable AI quality assessment.
• Work with engineering teams to identify root causes of AI-related defects and verify fixes.

4. API & Integration Testing

• Perform manual API testing using Postman or similar tools.
• Validate API request and response structures, status codes, authentication, authorization, and error handling.
• Test integrations between AI services, backend systems, databases, and external APIs.
• Validate data flow between application components and identify integration issues.
• Test API behaviour under different input conditions, including invalid inputs and boundary cases.
• Verify API responses and ensure consistency between frontend and backend functionality.

5. Defect Management & Quality Reporting

• Identify, document, track, and manage defects throughout the testing lifecycle.
• Maintain detailed test cases, test execution reports, defect logs, and test documentation.
• Collaborate with developers to reproduce issues, analyse root causes, and validate fixes.
• Track test coverage, defect trends, and application quality metrics.
• Communicate release risks, unresolved defects, and testing status to relevant stakeholders.
• Participate in release readiness reviews and provide quality sign-off for assigned modules.

6. Automation & Continuous Improvement

• Identify repetitive test scenarios that can be automated to improve testing efficiency.
• Support automation initiatives using Selenium, Playwright, or Cypress.
• Work with developers to improve testability and integrate testing practices into CI/CD workflows.
• Explore AI-assisted testing tools and approaches to improve test coverage and efficiency.
• Continuously improve testing methodologies for both conventional software applications and AI-powered systems.

Required Skills & Experience

1. Functional & Manual QA

• 3–5 years of hands-on experience in manual and functional software testing.
• Strong understanding of software testing methodologies, SDLC, STLC, and defect management.
• Experience designing and executing test cases for functional, regression, integration, and end-to-end testing.
• Strong knowledge of Agile/Scrum methodologies and sprint-based QA processes.
• Experience testing web applications, APIs, and mobile applications.
• Ability to analyse requirements and identify functional gaps, edge cases, and potential risks.

2. Mandatory AI/ML Testing Experience

• Prior hands-on experience testing AI/ML-powered applications or products.
• Practical experience testing LLM-based applications, AI chatbots, conversational systems, or intelligent automation features.
• Understanding of prompt testing and evaluating AI-generated responses.
• Ability to validate AI outputs for accuracy, relevance, consistency, and contextual correctness.
• Experience identifying hallucinations, unexpected AI behaviour, and response-quality issues.
• Understanding of AI application workflows, model outputs, and the challenges associated with non-deterministic systems.
• Experience collaborating with AI/ML engineers or data science teams during testing and validation.

3. API & Integration Testing

• Hands-on experience with manual API testing using Postman.
• Understanding of REST APIs, JSON, request/response validation, authentication, and error handling.
• Experience validating integrations between frontend, backend, and external services.
• Basic understanding of database validation and SQL queries.

4. Testing Tools & Technologies

• Experience with JIRA and Confluence for defect tracking and test documentation.
• Familiarity with Postman, SQL, and browser developer tools.
• Understanding of test management and reporting practices.
• Basic knowledge of BDD frameworks such as Cucumber/Gherkin is an advantage.

Preferred Skills

• Experience testing RAG-based applications, including document retrieval and contextual response validation.
• Exposure to vector databases and semantic search testing.
• Familiarity with LangChain, LangGraph, LlamaIndex, or similar AI frameworks.
• Understanding of LLM evaluation frameworks and AI quality assessment methodologies.
• Experience with AI evaluation tools such as Ragas, DeepEval, or similar frameworks.
• Exposure to prompt injection, adversarial testing, and responsible AI validation.
• Working knowledge of Selenium, Playwright, or Cypress.
• Basic scripting knowledge in Python or JavaScript.
• Familiarity with CI/CD pipelines and automated testing practices.
• Exposure to cross-browser testing tools such as BrowserStack or Sauce Labs.

What We're Looking For Beyond Skills

• AI Testing Mindset: Understands that testing AI applications requires more than checking whether a feature works. Can evaluate response quality, accuracy, relevance, and behaviour.
• Analytical Thinking: Ability to identify unusual scenarios, inconsistent outputs, and hidden quality issues.
• Strong QA Fundamentals: Applies structured testing methodologies while adapting them to AI-powered applications.
• Attention to Detail: Documents defects clearly and identifies patterns in application and AI behaviour.
• Ownership: Takes responsibility for testing quality from requirement analysis through release validation.
• Collaboration: Works effectively with developers, AI engineers, product managers, and data science teams.
• Continuous Learning: Interested in exploring emerging AI technologies and evolving testing methodologies.
• Communication: Clearly communicates test results, defect impact, quality risks, and release readiness.

What We Offer

• Competitive compensation aligned with experience and expertise.
• Flexible work environment (Coimbatore office / Hybrid).
• Paid holidays and flexible time off.
• Medical insurance covering self and family, up to ₹4 lakh per person.
• Opportunity to work on real-world AI-powered products and enterprise applications.
• Hands-on exposure to LLMs, RAG systems, agentic AI, and modern AI engineering practices.
• Opportunities to develop expertise in AI quality engineering and automation testing.
• A collaborative, multicultural environment with opportunities for professional growth.
• A culture that values clarity, integrity, ownership, and continuous learning.

Join Techjays

If you're passionate about software quality and have hands-on experience validating AI-powered applications, we'd love to hear from you.

Join us in building reliable, intelligent, and production-ready AI systems.

Job Summary

We are looking for an experienced AI Automation Tester with hands-on experience in testing AI-powered products and applications. The ideal candidate should have strong expertise in Playwright automation, along with practical experience in LLM evaluations (Evals), Golden Datasets (Goldens), and AI Observability.

Key Responsibilities

• Design, develop and execute automated test scripts using Playwright.
• Test AI-powered applications, LLM-based products and Generative AI solutions.
• Work on Evals to assess AI response accuracy, relevance, quality and consistency.
• Create, maintain and validate Golden Datasets (Goldens) for benchmarking AI outputs.
• Implement and work with AI Observability to monitor model performance, response quality, latency and failures.
• Perform functional, regression, API and end-to-end testing.
• Identify, report and track defects to ensure product quality and reliability.

Designation: Test Lead

Location: Hyderabad_Skyview

Job Summary: We are seeking a highly skilled and motivated SDET with strong Java testing expertise and AI engineering exposure to join our dynamic team in Hyderabad. The ideal candidate will have a strong background in software testing with 5+ years of experience, with a focus on automation testing using Selenium and Java. As a Test Lead, you will be responsible for leading testing efforts, ensuring the quality of our software products, and mentoring junior team members. Your expertise in software development and testing methodologies will be crucial in driving our testing strategy and improving our processes.

Responsibilities:

• Lead the software testing efforts for various projects, ensuring high-quality deliverables.
• Design, develop, and execute automated test scripts using Selenium with Java.
• Collaborate with development teams to understand requirements and provide feedback on testability.
• Mentor and guide junior testers in best practices for software testing and automation.
• Develop and maintain test plans, test cases, and test data.
• Identify, document, and track defects using appropriate tools.
• Participate in code reviews and provide input on testing strategies.
• Continuously improve testing processes and methodologies to enhance efficiency.
• Report testing progress and results to stakeholders, ensuring transparency and accountability.
• Stay updated with the latest industry trends and technologies in software testing.

Mandatory Skills:

• Proficiency in Selenium with Java for automated testing.
• Strong understanding of software testing methodologies and best practices.
• Experience in developing and executing test plans and test cases.
• Familiarity with defect tracking tools and test management software.
• Excellent analytical and problem-solving skills.
• Strong communication and interpersonal skills.
• Experience using AI tools for development and testing automation.
• Exposure to AI testing frameworks is preferred.
• Experience in creating test data and performing end to end testing.

Preferred Skills:

• Experience with performance testing tools.
• Knowledge of continuous integration and continuous deployment (CI/CD) processes.
• Familiarity with Agile methodologies.
• Experience in API testing and tools like Postman or SoapUI.
• Understanding of database testing and SQL.

Qualifications:

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Relevant certifications in software testing (e.g., ISTQB) are a plus.
• Proven track record of leading testing efforts in a software development environment.
•  

About the team

SecurITe’s mission is to build an Agentic‑AI driven security platform that protects critical infrastructure from modern cyber threats. Our focus is on delivering highly performant, resilient, and intelligent network security systems that help defenders stay ahead of adversaries.

About the Role

We’re looking for a seasoned Senior Quality Engineer to provide technical leadership and architectural oversight for our next‑generation cybersecurity AI platform. In this high-impact role, you will define the technical strategy for quality assurance, ensuring our agentic AI transforms cyber defense with unparalleled reliability.

You will be responsible for the end-to-end quality lifecycle, from architectural reviews to the deployment of scalable automation frameworks. Beyond technical execution, you will serve as a mentor to junior team members, fostering a culture of technical excellence and driving the strategy that ensures our solutions meet the rigorous demands of critical infrastructure protection.

What You’ll Do

●    Defining and driving comprehensive QA strategies and roadmaps for the cybersecurity platform.

●    Designing, developing, and executing test plans, test cases, and automated scripts to ensure software quality.

●    Performing functional, regression, performance, scalability and security testing to identify bugs or defects.

●    Collaborating with developers, product managers, and other stakeholders to understand product requirements and testing needs.

●    Identifying, documenting, and tracking software defects, ensuring clear communication of issues and their resolutions.

●    Leading deep-dive root-cause analysis for critical system defects and security vulnerabilities.

●    Conducting thorough reviews of product specifications and software design to identify potential areas of concern before testing.

●    Architecting and designing complex, scalable test automation frameworks to optimize CI/CD velocity.

●    Ensuring the software meets customer and business requirements by validating the functionality and performance.

●    Assisting in continuously improving QA processes, tools, and best practices to enhance software testing efficiency and effectiveness.

●    Supporting user acceptance testing (UAT) and assisting clients with product validation.

●    Mentoring junior and mid-level engineers, providing technical guidance and conducting architectural reviews.

Required Experience

●    A Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or a related field.

●    8-10 years of proven experience in quality engineering, specifically within network cybersecurity, Identity Providers, or AI-integrated platforms.

●    Expertise in manual and automated testing.

●    Deep domain expertise in complex system validation and advanced automation practices at scale.

●    Proficiency in programming languages like Python to build and run automated test scripts.

●    "Strong knowledge of software testing methodologies, performance testing tools (e.g., JMeter, k6), and security traffic generation/simulation tools (e.g., Ixia BreakingPoint, Scapy, or Snort/Suricata traffic generators)."

●    Understanding of continuous integration/continuous deployment (CI/CD) pipelines and version control systems like Git.

●    Strong communication skills for documenting test results and interacting with cross-functional teams.

●    Excellent analytical skills, attention to detail, and problem-solving ability.

●    Ability to work independently as well as collaboratively in a team environment.

●    A curious mindset with a willingness to quickly learn new technologies and testing tools.

Required Skills & Qualifications

●    Familiarity with cloud-based testing environments (GCP, AWS, Azure).

●    Experience with cybersecurity products or cloud services or IDP or Web UI

The Mindset

●    Problem Solver: You thrive on complex, ambiguous challenges and engineer elegant solutions.

●    Ownership‑Driven: You take initiative, move fast, and deliver outcomes without hand‑holding.

●    Continuous Learner: You stay ahead of the curve in AI, ML, and emerging technologies.

●    Startup DNA: You excel in fast‑moving environments where priorities evolve and impact is immediate.

Strong Senior QA/SDET Profile with hands-on automation building and QA-function ownership

2

Mandatory (Experience 1): Must have 6+ years in QA/test automation/SDET roles with clear progression in scope (Engineer to Senior to Lead)in a B2B SaaS product company

3

Mandatory (Experience 2): Must have direct experience scaling a QA function at a SaaS company and building automation frameworks or QA processes from scratch

4

Mandatory (Tech skill 1): Must have hands-on UI automation experience with Selenium (Java + TestNG) or Playwright / Cypress (JavaScript / TypeScript) using maintainable design patterns like Page Object Model

5

Mandatory (Tech skill 2): Must have API testing & automation experience with Postman and RestAssured (or equivalent) for functional, integration, and contract testing

6

Mandatory (Tech skill 3): Must have experience with CI/CD integration (Jenkins or GitHub Actions) and Git for version control

7

Mandatory (Skill 1): Must have a strong grasp of Agile / Scrum delivery and JIRA for defect and test management

8

Mandatory (Skill 2): Must have experience leading or mentoring QA engineers and collaborating cross-functionally with developers and product teams

9

Mandatory (Skill 3): Must have solid experience using AI in test automation and manual testing exposure

10

Mandatory (Communication): Must have strong communication skills for cross-functional collaboration

11

Mandatory (Company): Must come from a B2B SaaS

Looking for 3 years exp SDET - Java Selenium, API Automation, rest assured, Jmeter Writing data driven functional testscripts using java selenium Writng scripts for API automation with RestAssured Enterprise grade test frameworks, test automation and CI/CD Immediate joiner, Location: Bangalore Work from office.

Please apply with NoticePeriod:

CTC:

ECTC:

Work from office(Y/N):

Location Bangalore (Y/N):

About the role

We need a senior QA professional to assess our current QA practices, set up the right processes, and guide junior team members on execution. You will then scale the practice with Playwright automation.

Responsibilities

Assess existing QA processes, tools and coverage; identify gaps and drive an improvement roadmap.

Define test strategy, QA processes and quality metrics (coverage, defect leakage, release sign-off).

Guide and mentor junior QA engineers on test design, execution and automation.

Build a scalable Playwright (TypeScript) framework for UI and API tests.

Integrate automation into CI/CD with quality gates and clear reporting.

Drive adoption of AI in testing: agentic AI for test generation, self-healing scripts and failure triage, using Playwright MCP and similar tools.

Define QA approaches for AI-powered features, including LLM output evaluation and agent workflow testing.

Partner with dev and product teams to make quality a shared responsibility.

Must have

8–10 years in QA, 4+ years in test automation.

2+ years hands-on Playwright with TypeScript/JavaScript.

Proven experience assessing and setting up a QA practice from scratch.

Strong API testing, CI/CD (GitHub Actions, Jenkins or similar) and Git skills.

Experience leading and coaching junior QA engineers.

Hands-on use of AI coding assistants and agentic AI tools (e.g. Playwright MCP, Claude, GitHub Copilot) in QA workflows.

Good to have

Selenium/Cypress to Playwright migration experience.

Performance, accessibility or visual testing exposure.

ISTQB or equivalent certification.

Experience building or testing MCP servers, AI agents or LLM-based applications (evals, prompt and guardrail testing).

Need Automation Tester with relevant information asked. Career gap is allowed too. Manual Tester also can apply

Job Role: Senior QA Engineer

Company: Appmaker by StarApps

Location: Kochi, Kerala (On-Site)

Experience: 3–5 Years

Employment Type: Full-time

About Appmaker by StarApps

At StarApps, we build tools like Appmaker that help thousands of global e-commerce merchants turn their web stores into native mobile apps.

About the Role

We're looking for an experienced QA Engineer to take overall ownership of our QA cycle. You'll do hands-on testing, build strong quality control so fewer issues reach QA in the first place, and

guide our current QA team. Your goal: catch issues early, reduce the number of QA cycles per release, and ship high-quality apps faster.

Key Responsibilities

• QA Ownership: Own the end-to-end QA cycle, from requirement review to final releasesign-off.
• Quality Control: Set up quality checks, entry/exit criteria and checklists that stop issues early and reduce repeat QA cycles.
• Team Guidance: Mentor and guide the current QA team, review their test cases and bug reports, and raise the team's testing standards.
• Test Planning: Create test plans and test cases from the requirements and designs shared, covering functional, regression and edge cases.
• Device & Design Parity Testing: Test on real Android and iOS devices, and run design parity checks against approved designs.
• Bug Management: Report, prioritise and track bugs, document regressed bugs, and verify fixes with the dev team.
• Process Improvement: Find the root causes of recurring bugs and improve QA processes, tools and documentation.
• Communication: Keep management updated on release quality, risks and issues found.
• AI-Assisted Testing: Use AI tools to generate test cases and speed up testing, while reviewing the output carefully.

What We're Looking For

• Experience: 3–5 years in software QA, with strong experience testing mobile apps.
• Ownership: Proven experience owning QA for releases and reducing QA cycles or bug
• leakage.
• Leadership: Experience guiding or mentoring junior QA engineers.
• Testing Expertise: Strong knowledge of test planning, functional, regression,
• smoke/sanity testing and the bug life cycle.
• QA Tools: Hands-on experience with QA tools like BrowserStack, Appium and bug
• trackers such as Jira, and the judgment to pick the right tool for the job.
• AI Tools: Experience or strong interest in using AI tools for testing.
• Attention to Detail: You notice when a price, image, spacing or button looks off.
• Communication: Clear written and spoken English for reports, updates and working with developers and management.

Bonus Points

• Experience testing Shopify or e-commerce apps.
• Experience with test automation (Appium, Playwright or similar) or API testing (e.g.
• Postman).
• Experience setting up QA processes from scratch in a fast-moving team.

Why Join Us?

• Real Impact: Own the quality of apps used by thousands of global brands and millions of shoppers.
• Leadership: Shape how QA works at Appmaker and grow the QA team.
• Modern QA: Work with AI-driven testing tools and a modern workflow.
• Growth & Culture: High ownership and a supportive team in our Kochi office.