Data Annotation at Thomasmount Consulting · Kochi (Cochin) · 5 - 9 years · ₹20L - ₹22L / yr · Profitable · Posted 14 Oct 2026

Role Overview
We are looking for an experienced professional to evaluate AI-generated responses for conversational AI agents used in travel-related products. The role focuses on assessing response accuracy, relevance, completeness, and quality to improve AI performance.
Key Responsibilities
- Evaluate AI-generated responses against predefined guidelines and quality standards.
- Perform data annotation, labeling, classification, and quality reviews.
- Identify incorrect, misleading, irrelevant, or incomplete responses and potential AI hallucinations.
- Analyze evaluation data to identify recurring errors, quality gaps, and improvement opportunities.
- Track quality metrics, document findings, and recommend improvements.
- Use Excel/Google Sheets and basic SQL, where applicable, to analyze evaluation data.
- Collaborate with AI and Machine Learning teams to improve conversational AI performance.
Required Skills
- 5–8 years of experience in Data Annotation, AI Evaluation, Content Quality, Search Relevance, or Quality Assurance.
- Strong analytical, problem-solving, and attention-to-detail skills.
- Experience applying evaluation guidelines and quality standards consistently.
- Proficiency in Excel or Google Sheets.
- Good written communication and documentation skills.
- Understanding of quality metrics and data analysis.
Good to Have
- Exposure to Generative AI, LLMs, conversational AI, or AI response evaluation.
- Experience with Human-in-the-Loop (HITL), AI training data validation, or hallucination detection.
- Basic SQL knowledge.
- Experience in travel, hospitality, or search quality evaluation.

About Thomasmount Consulting
About
Similar jobs (10)
The Role
You own AI systems end to end. From the speech-to-text models that turn audio into text, to the diarization that separates and identifies speakers, to the agentic layer that turns conversation into memory and action, to the observability and evaluation that keep all of it honest in production. This is a wide role by design. You will own model selection, serving, and production reliability. If you want to tune one model and ignore the system around it, this is not the role.
What You Will Own
• Speech-to-text. Evaluate, integrate, and optimize STT models across cloud and self-hosted. Drive accuracy and cost trade-offs with ground-truth metrics.
• Speaker diarization and identification. Push accuracy on hard, real-world, multi-speaker audio.
• Agentic AI. Build the memory and retrieval pipeline, LLM orchestration, and the agent workflows that sit on top of captured conversation.
• Model serving and infrastructure. Stand up and optimize self-hosted serving (vLLM, Triton class). Own latency, throughput, and cost per user.
Observability
An always-on wearable means models run in production every second, on messy real-world audio. You own the visibility into that.
• Instrument the full audio-to-memory pipeline: STT, diarization, retrieval, and LLM calls.
• Define and track model-quality SLOs in production: transcription drift, diarization error over time, retrieval relevance, latency, throughput, and cost per user.
• Build dashboards and alerting so model degradation is caught before users feel it.
• Trace failures across a distributed, always-on system using metrics, logs, and traces.
• Close the loop. Production signals feed back into evaluation and model selection.
Evaluation
We do not ship what we cannot measure. You own the systems that prove a model is actually better, not just newer.
• Build and own ground-truth evaluation harnesses for every model in the stack.
• Measure with real metrics: WER for transcription, DER for diarization, Recall and F1 for retrieval and speaker identification.
• Build and maintain labeled benchmark datasets that reflect real, messy, multi-speaker audio.
• Run regression and A/B evaluations on every model swap, prompt change, or pipeline update. Nothing ships on a vibe.
• Reject anecdotal proxies, single confidence scores, and cherry-picked examples as evidence of quality.
What We Are Looking For
• 3 to 5 years as an AI/ML engineer with production systems behind you. Engineering and production experience is non-negotiable.
• Depth across the modern AI stack: LLMs, speech models, vector retrieval, model serving.
• Strong software engineering. You write code that ships and survives contact with real users.
• Fluency in Python and the production ML ecosystem.
• Comfort with cloud infrastructure (GCP a plus) and containerized deployment on Kubernetes.
• A working command of observability and evaluation. You measure first and trust metrics over intuition.
• First-principles reasoning and metric discipline.
Nice to Have
• Research background or publications. A strong signal, not a substitute for production work.
• Audio and speech ML experience (STT, diarization, voice).
• Experience self-hosting and optimizing open models.
• Experience with LLM gateway and agent orchestration patterns.
• Experience building eval harnesses or production model-monitoring systems.
Requirements
Agentic work is must. Audio is good to have
. Self hosting models is a must
Experience with LLM gateway and agent orchestration is a must have
The recruiter has not been active on this job recently. You may apply but please expect a delayed response.
About Us
We’re building the next generation of AI-powered business software, and we’re looking for people who want to shape that future with us. With Lumen, we’re reimagining how users interact with CRM — moving beyond screens, menus and dashboards to an intelligent interface where users can simply ask AI to take actions, retrieve knowledge, generate insights and get work done. With Agent Studio, we’re enabling businesses to build, test and deploy their own AI agents for real-world workflows. And with Invorto, we’re bringing AI to voice, allowing businesses to create intelligent voice agents tailored to their customer and operational use cases.
What makes this especially exciting is the stage and scale of the opportunity. You’ll get to work on genuinely hard problems across LLMs, agents, reasoning, orchestration, voice AI, evaluation, reliability and enterprise security — not as isolated experiments, but as products used in real business workflows. You’ll have the opportunity to build zero-to-one, own meaningful parts of the product end-to-end, work closely with customers, experiment rapidly, and see your work reach production at scale.
Why join now? Because the playbook for enterprise AI is still being written. You won’t just be implementing someone else’s roadmap — you’ll help define the product, architecture and experiences that become that playbook. Expect high ownership, fast iteration, hard technical and product problems, direct customer impact, and the chance to build AI systems that have to work reliably in the real world — not just in a demo.
About the Role
We are looking for a QA Engineer who specializes in testing agentic AI platforms. You will design and automate quality processes for systems that involve LLMs, autonomous agents, tool use and orchestration across Lumen and Agent Studio — ensuring that AI-driven workflows behave reliably, safely and predictably in production, not just in a demo.
What You’ll Do
- Design and build automated test suites and evaluation frameworks for agentic AI workflows, including multi-step and tool-calling behaviors.
- Use AI/LLM-based QA tools and evaluation frameworks to test model outputs, agent decisions and end-to-end task completion at scale.
- Define quality metrics and benchmarks for agent reliability, correctness, latency and safety, and track them over releases.
- Identify edge cases, failure modes and regressions specific to non-deterministic AI systems, and build automated checks to catch them early.
- Integrate automated agent/LLM testing into CI/CD pipelines to support fast, reliable iteration.
- Partner closely with AI/ML and backend engineers to reproduce issues, root-cause failures and validate fixes.
- Work with customers and customer-facing teams to understand real-world usage patterns and translate them into test scenarios.
What We’re Looking For
- 2–4 years of QA/test automation experience, including hands-on work testing agentic AI or LLM-based platforms.
- Practical experience using AI-focused QA/evaluation tools to test agent behavior, prompts and model outputs.
- Strong scripting/automation skills (Python preferred) to build and maintain test frameworks.
- Understanding of how LLM-based agents work — tool calling, orchestration, memory, reasoning chains — well enough to design meaningful test cases.
- Comfort working with non-deterministic systems and designing evaluation approaches beyond traditional pass/fail testing.
- Strong communication skills — this is a customer-facing role, and you will be expected to clearly articulate technical concepts, decisions and trade-offs to both technical and non-technical stakeholders, including customers.
Good to Have
- Experience testing voice AI or real-time conversational systems.
- Familiarity with CRM or enterprise SaaS platforms.
- Exposure to enterprise security or compliance testing for AI systems.
About Us
Invorto is our Voice AI product, bringing intelligent voice agents to real-world customer and operational use cases. Our voice pipeline is built in Python, running an STT → LLM → TTS architecture on top of the Pipecat framework.
This is a chance to work on hard problems in voice AI — latency, accuracy, naturalness, and reliability — building zero-to-one, owning your area end-to-end, and shipping to production at scale.
Note: This is a customer-facing role, and strong communication skills are essential.
About the Role
We're looking for a Voice AI Research Engineer to join the Invorto team and help build and continuously improve the voice AI systems that power our intelligent voice agents. This role is focused on the specialized craft of voice AI — designing evaluation and automation frameworks that ensure our STT, LLM, and TTS pipeline performs reliably in real-world, production conditions.
What You'll Do
- Design and build automated testing and quality frameworks for our STT → LLM → TTS voice pipeline, built on Pipecat
- Evaluate and benchmark STT, LLM, and TTS/ASR components on accuracy, latency, naturalness, and robustness across accents, languages, and real-world audio conditions
- Work hands-on with STT, TTS, and ASR models — fine-tuning, evaluating, and improving them for production use cases
- Identify failure modes and edge cases across the pipeline (background noise, accents, interruptions, turn-taking, latency, pipeline-stage handoffs) and build systems to catch them before production
- Collaborate closely with engineering to integrate quality checks and automation into the voice agent development lifecycle within the Pipecat-based architecture
- Research and stay current with advances in voice AI, and bring in new techniques, models, and tools to improve pipeline performance
- Work directly with customers to understand real-world voice use cases and translate them into evaluation criteria and quality benchmarks
- Partner with product and engineering to define what "production-grade quality" means for voice agents and drive the team toward it
What We're Looking For
- 4–6 years of experience, with a specialization in voice AI systems and automated quality evaluation
- Hands-on experience with STT (Speech-to-Text), TTS (Text-to-Speech), and ASR (Automatic Speech Recognition) models
- Experience designing and building automated testing/evaluation frameworks for voice or speech systems
- Strong understanding of what drives voice AI quality — accuracy, latency, naturalness, and robustness to real-world variability
- Strong programming skills in Python; familiarity with Pipecat or similar voice pipeline/orchestration frameworks is a plus
- Understanding of STT → LLM → TTS pipeline architectures and the trade-offs involved at each stage
- Research mindset — comfortable exploring new models, techniques, and tools and translating them into practical improvements
- Excellent communication skills — this is a customer-facing role, and you'll regularly engage directly with customers to understand needs and validate quality expectations
About the Job :
We are looking for a passionate and driven AI Intern to join our dynamic team. As an intern, you will have the opportunity to work on real-world projects, develop AI models, and collaborate with experienced professionals in the field. This internship is designed to provide hands-on experience in AI and machine learning, offering you the chance to contribute to impactful projects while enhancing your skills.
Job Description:
We are seeking a talented Artificial Intelligence Specialist to join our dynamic team. As an AI Specialist, you will be responsible for developing, implementing, and optimizing AI models and algorithms. You will collaborate closely with cross-functional teams to integrate AI capabilities into our products and services. The ideal candidate should have a strong background in machine learning, deep learning, and natural language processing, with a passion for applying AI to real-world problems.
Responsibilities:
- Design, develop, and deploy AI models and algorithms.
- Conduct data analysis and pre-processing to prepare data for modeling.
- Implement and optimize machine learning algorithms.
- Collaborate with software engineers to integrate AI models into production systems.
- Evaluate and improve the performance of existing AI models.
- Stay updated with the latest advancements in AI research and apply them to enhance our products.
- Provide technical guidance and mentorship to junior team members.
Requirements:
- Any Graduate / Bachelor's degree in Computer Science, Engineering, Mathematics, or a related field; Master's degree preferred.
- Proven experience in developing and implementing machine learning models and algorithms.
- Strong programming skills in languages such as Python, R, or Java.
Benefits :
- Internship Certificate
- Letter of Recommendation
- Performance-Based Stipend
- Part-time work from home (2-3 hours per day)
- 5 days a week, fully flexible shift
AI Engineer
We are seeking an AI Engineering specialist focused on AI evaluation, and continuous quality improvement for Ezra MetLife's employee-facing AI platform. This role will establish and scale the testing strategy for enterprise AI agents, ensuring high response quality, reliability, and production readiness. The engineer will build automated regression testing framework (preferred Playwright ), define AI evaluation methodologies, analyze AI performance metrics, and partner with engineering teams to continuously improve answer quality, grounding accuracy, and customer experience. This position is critical to enabling confidence as Ezra expands its AI agent portfolio and employee-facing capabilities.
Required Skills & Experience
• C# and .NET development experience
• Experience with at least one AI evaluation framework (e.g., prompt evaluation, RAG evaluation, LLM quality assessment)
• Microsoft Agent Framework (preferred) or similar enterprise agent frameworks
• Experience with Azure OpenAI / Azure AI Foundry
• Microsoft 365 Agent SDK
• Azure AI Search, RAG pipelines, and retrieval quality testing
• Infrastructure as Code using Terraform
• Experience building automated testing and AI quality validation processes
• Familiarity with telemetry analysis, AI observability, and performance measurement
• Strong analytical skills with a passion for improving AI response quality and reliability
Skills: AI Agents~Core .NET Technologies~C# 5.0
Experience Required: 10 & Above
Location: Hyderabad :5+ relevant exp in AI + .NET
Required Skills:-
- 4+ years of experience in Software Testing (Manual & Automation).
- Strong experience with Selenium or Playwright.
- Hands-on experience in API Testing (Postman, REST Assured, Swagger, etc.).
- Experience with CI/CD pipelines (Jenkins, GitLab CI, Azure DevOps, GitHub Actions).
- Strong understanding of Functional, Regression, and Non-functional Testing.
- Excellent analytical and debugging skills.
AI-Specific Skills
- Hands-on experience in GenAI / AI Testing.
- Experience testing LLM-based applications.
- Strong understanding of prompt engineering and prompt validation.
- Knowledge of LLM behavior, model variability, and non-deterministic outputs.
- Experience validating AI outputs for accuracy, hallucinations, bias, and safety.
- Test data management for AI applications.
- Understanding of Responsible AI testing concepts.
Role Overview:
We are looking for AI Market Evaluators to join our growing team. In this role, you will review daily market indicators prepared by our AI systems and provide simple directional input ("Up" or "Down"). No trading or financial experience is required—we provide all the training and support you need.
Key Responsibilities:
- Review daily market charts and indicators sent to your inbox
- Provide directional input ("Up" or "Down") to help refine our AI trading models
- Maintain consistent daily participation (5-10 minutes)
- Communicate with our support team if any issues arise

As an intern for the Artificial Intelligence business automation team, you will be focusing on researching and developing AI tools that automate some parts of our business automation. As business automation is important throughout the firm, you will have the opportunity to collaborate with teams across Springer.
🔹 Key Responsibilities
• Design, develop, and deploy production-grade AI/ML and Generative AI solutions
• Work on GEO, AEO, and SGE initiatives to improve visibility across AI-driven search platforms
• Optimize content and digital experiences for conversational queries and LLM-based search
• Develop solutions using LLMs, NLP, embeddings, semantic search, RAG, and vector databases
• Analyze search intent, AI-generated responses, citations, retrieval patterns, and content discoverability
• Build frameworks to measure GEO/AEO strategies and AI-search performance
• Collaborate with Product, Engineering, Content, SEO, Marketing, and Business teams
• Improve solution accuracy, relevance, latency, and user experience
🔹 Mandatory Requirements
✅ 1–4 years of professional experience
✅ Minimum 1 year of hands-on experience in GEO, AEO, or SGE
✅ Experience with prompt engineering, embeddings, vector search, or RAG systems
✅ Understanding of semantic search and entity-based optimization
✅ Exposure to ChatGPT, Google Gemini, or similar LLM platforms
✅ Knowledge of schema, context building, content structuring, and knowledge representation
🎓 Preferred Education
B.Tech, M.Tech, Integrated M.Sc., or MS from a Tier-1 engineering institute such as IIT, NIT, BITS, VIT, DTU, or NSUT.
About E2M:
E2M Solutions works as a trusted white-label partner for digital agencies. We support agencies with consistent and reliable delivery through services such as website design, web development, eCommerce, SEO, AI SEO, PPC, AI automation, and content writing .Founded on strong business ethics, we are an equal opportunity organization powered by 300+ experienced professionals, partnering with 400+ digital agencies across the US, UK, Canada, Europe, and Australia. At E2M, we value ownership, consistency, and people who are committed to doing meaningful work and growing together .If you’re someone who dreams big and has the gumption to make them come true, E2M has a place for you.
Role Overview:
We are seeking a highly skilled and client-centric AI Consultant/AI Adoption Specialist to join our growing team. In this pivotal role, you'll serve as a vital link between our clients' strategic objectives and the transformative power of AI. You'll primarily focus on understanding their needs, scoping opportunities, and architecting actionable AI roadmaps.
Key Responsibilities:
- Collaborate closely with clients to understand their challenges and identify opportunities to apply AI.
- Assess client requirements and prepare solution strategies using AI tools and methodologies.
- Work with internal teams to design, propose, and help execute AI-powered solutions.
- Provide AI-based recommendations that align with the client’s business objectives.
- Communicate technical possibilities in a business-friendly manner to decision-makers.
- Take ownership of the client journey from discovery to implementation and support.
- Stay updated with AI trends, tools, and real-world use cases that can benefit clients.
Required Skills & Qualifications:
- Minimum 2+ Years of hands on experience into Custom AI Development.
- Minimum 3+ years of experience in roles like Project Manager, Customer Success Manager, or Account Manager, preferably in a service-based company or digital agency.
- Strong understanding of AI concepts, trends, and tools (e.g., NLP, ML, Chatbots, Automation, native cloud technologies).
- Some hands-on experience in AI projects – either through execution, coordination, or implementation.
- Ability to manage multiple client engagements and communicate effectively with both technical and non-technical stakeholders.
- Strong problem-solving mind set with the ability to translate business needs into AI opportunities.
- Flexible to work with international clients, especially in the US time zone as needed.










