Voice AI Research Engineer

We are looking for a hands-on Voice AI Engineer to own the technical delivery of an enterprise voice automation program for an industrial equipment and services business.

You will take the solution from discovery and architecture through agent development, integrations, testing, deployment and ongoing optimisation. The initial focus is inbound calls across service, parts and rental, with approximately 300 calls per day.

This role suits someone who can turn a working demonstration into a reliable production service and work directly with customer teams to make it effective.

What you will build

A connected Voice AI solution that answers calls on existing business numbers, identifies customers and equipment, understands requests and routes conversations to the appropriate workflow.

The solution will capture structured call records, integrate with existing CRM and service-management systems, and transfer urgent or complex calls to people. Customer and equipment information will come from a database refreshed through scheduled data synchronisation.

Your responsibilities

• Own technical delivery: Translate business requirements into architecture, implementation plans, integration specifications and acceptance criteria.
• Build voice agents: Develop reception and specialist agents for service, parts and rental, with shared context and reliable hand-offs.
• Improve conversation quality: Configure speech recognition, voice generation and language models. Optimise latency, interruptions, turn-taking, pronunciation and handling of background noise.
• Implement telephony: Configure inbound routing, existing-number connectivity, concurrent calls, live transfers, after-hours handling and fallback routes.
• Connect business systems: Build APIs, webhooks and middleware for CRM, service queues and customer databases. Implement scheduled synchronisation, retries and duplicate prevention.
• Capture usable records: Generate accurate summaries and structured fields covering customer details, equipment, reason for calling, urgency, outcomes and relevant sales interest.
• Build safeguards: Implement call-duration and monthly-usage caps, alerts, escalation rules and checks that requests have been successfully logged.
• Test and launch: Develop realistic test scenarios, run integration and load testing, support customer acceptance testing and manage production deployment.
• Operate and improve: Monitor call quality, integration failures, latency and cost per minute. Investigate incidents and refine agent behaviour using transcripts and customer feedback.
• Document the solution: Maintain architecture diagrams, configuration records, deployment procedures and operational runbooks.

What we are looking for

• Strong development skills in Python and/or JavaScript/TypeScript.
• Experience shipping at least one Voice AI or conversational AI solution into production.
• Hands-on understanding of speech-to-text, text-to-speech, language models and tool/function calling.
• Experience with telephony APIs, SIP, call routing and human transfers.
• Strong API integration skills, including authentication, webhooks and asynchronous processing.
• Working knowledge of SQL databases, data synchronisation and cloud deployment.
• Experience with logging, monitoring, error handling and production troubleshooting.
• Ability to implement secure access, protect personal information and manage recordings and transcripts appropriately.
• Confidence working directly with customers, explaining technical trade-offs and taking ownership of delivery.

Useful additional experience

• Multi-agent orchestration and context management.
• Integration with enterprise CRM, workflow or document-management platforms.
• Retrieval-based knowledge systems and structured information extraction.
• Contact-centre, field-service, equipment rental or industrial service workflows.
• Automated conversation testing and evaluation.

What success looks like

• A stable deployment that meets agreed call-handling and integration targets.
• Clear, natural conversations with appropriate escalation to people.
• Accurate service requests and sales leads reaching the correct systems.
• Reliable controls over usage, cost and failure handling.
• Measurable improvements following testing and customer feedback.

Why join

You will own a live enterprise implementation from the first technical discussion through production operations. You will have direct access to decision-makers and the opportunity to build reusable components for future Voice AI deployments.

How to apply

Share your CV and a brief description of a Voice AI solution you have built. Explain your personal contribution, the integrations involved and one production challenge you solved.

Principal Software Engineer

Location: Bangalore India

Department: AI Engineering / Quality Assurance

Experience Level: Mid to Senior

We are looking for a highly motivated Senior AI Test / Automation Engineer to design and scale automated validation frameworks for AI/ML models, LLM-based applications, and agentic systems. This role is critical to ensure that AI solutions meet enterprise standards for quality, reliability, safety, and compliance before and after production deployment.

Key Responsibilities

• Build and maintain AI test automation frameworks for pre-qualification and continuous validation of models and agent workflows
• Develop comprehensive test suites, including:
• Unit, integration, and end-to-end (E2E)
• Functional, regression, performance, and safety testing
• Validate AI system behavior, including:
• Non-deterministic LLM outputs
• Hallucinations and edge cases
• Multi-step agent decision-making
• Design and manage evaluation systems:
• Golden datasets
• Benchmarking pipelines (accuracy, latency, reliability)
• Automate testing within CI/CD pipelines for model updates, prompt changes, and tool integrations
• Implement observability and telemetry to enable traceability, monitoring, and audit readiness
• Collaborate cross-functionally with ML, MLOps, Product, and Security teams to define quality gates and release criteria
• Track and report quality KPIs, including test coverage, defect leakage, and system reliability
• Drive root-cause analysis and continuous improvement across the AI testing lifecycle

Required Skills

Core Engineering

• Strong programming skills in Python; familiarity with Bash, TypeScript, or Go
• Experience with test automation frameworks such as PyTest, Playwright, Selenium, or Cypress
• Proficiency in CI/CD tools (GitHub Actions, Jenkins, GitLab CI)
• Experience with cloud platforms (AWS, Azure, GCP) and containers (Docker, Kubernetes)

AI / ML & Agentic Systems

• Hands-on experience with LLM ecosystems (OpenAI, Anthropic, Bedrock)
• Familiarity with:
• RAG architectures and vector databases (Pinecone, Weaviate)
• Agent frameworks (LangChain, LlamaIndex, AutoGen)

AI Testing Techniques

• Experience with non-deterministic testing approaches (statistical assertions, tolerance thresholds)
• Knowledge of evaluation methods:
• LLM-as-a-judge
• BLEU, ROUGE, semantic similarity scoring
• Experience with prompt and agent regression testing
• Understanding of AI safety testing, including adversarial testing, bias/fairness validation, and jailbreak detection

Tooling (Preferred)

• AI testing & observability tools: LangSmith, TruLens, Arize, Weights & Biases
• Evaluation tools: DeepEval, Ragas, PromptFoo, Giskard
• Monitoring: Prometheus, Grafana, OpenTelemetry

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and cross-functional collaboration
• Data-driven mindset with focus on quality KPIs
• Detail-oriented with a strong bias toward automation and scalability

Experience Requirements

• 7+ years in QA, SDET, or test automation engineering
• Proven experience building and scaling automation frameworks
• Hands-on experience with AI/ML systems or LLM-based applications
• Experience testing RAG pipelines or agentic workflows
• Owned end-to-end AI test strategy and architecture
• Defined quality metrics and release gates
• Delivered scalable validation pipelines for production AI systems
• Supported audit and compliance readiness

Preferred

• Experience in enterprise or regulated environments (SOC2, ISO 27001, etc.)
• Exposure to:
• Shift-left testing practices
• Production observability and monitoring
• Chaos or resilience testing

Senior-Level Differentiators

Education

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field

Nice-to-have:

• ISTQB certification
• Cloud/ML certifications (AWS, Azure, GCP)
• AI testing certifications

What Success Looks Like

• AI systems that are accurate, reliable, and safe
• Fully automated test pipelines integrated into CI/CD
• Measurable improvements in defect leakage and model quality
• Strong observability and auditability across AI systems
• Scalable validation frameworks supporting rapid AI innovation

About the Company

The client is revolutionising the way businesses operate through cutting-edge technological solutions. Their focus is on developing intelligent agents and agentic workflows that automate processes and eliminate the need for human effort wherever possible. By leveraging

advanced AI and machine learning, they create systems that enhance productivity and drive efficiency.

Their expertise extends to the fintech, healthcare and medical technology sectors, where they develop innovative solutions that improve patient outcomes and streamline medical operations.

From medical devices to healthcare platforms, their work sits at the intersection of technology and medicine, pushing the boundaries of what's possible. The team is dedicated to continuous learning and growth, ensuring the team members are always at the forefront of the tech landscape.

About the Role

This is a senior, hands-on engineering role at the heart of our product team. You will be one of the most technical people in the room — setting the architecture for our real-time voice AI

agents and building the hardest parts of it yourself. From the systems that power live conversations to the interfaces our clients rely on, you will own how the product is engineered end to end.

We are looking for a genuine lead full-stack engineer with the depth to make architecture decisions that hold up as we scale, and the appetite to still be in the code every day. You should be as comfortable designing the backend services behind a live voice agent as you are shaping a clean interface on top of them — and comfortable being the person others turn to when something is hard.

You will work directly with the founder and product leadership on a fast-moving product, with real influence over technical direction. This is a role for someone who wants ownership at the level of "how the whole thing is built," not just individual features — and who raises the bar for

everyone around them.

What You'll Own

Set the technical direction

• Own the architecture of our core systems — the real-time voice agents, backend
• services, data and APIs — making the decisions that keep the product fast, reliable and scalable as it grows.
• Lead the hardest engineering problems and solve them personally.
• Establish engineering standards — code quality, review practices, testing and technical patterns that the team builds to.
• Drive technical strategy with the founder and product leadership — shaping the roadmap, flagging risk early, and turning product ambition into a sound technical plan.

Build the product end to end

• Design, build and ship features across the stack — backend services, APIs and front-ends — owning them from idea to production.
• Build the client-facing surfaces — dashboards, review tools and configuration interfaces that let our clients run and trust the product.
• Design and evolve the data models and APIs that hold up as we scale across clients.

Make it reliable and fast

• Own production quality — put the monitoring and alerting in place so issues are caught before clients feel them, and performance stays within target.
• Care about performance — find and fix bottlenecks across the stack.
• Build for correctness — put the testing and evaluation in place that keeps the product behaving predictably as it changes.

Lead through the team

• Mentor and grow engineers — through code review, pairing, and setting a technical example others learn from.
• Multiply the team's output — unblock others and lift the overall quality of the codebase.
• Take features from ambiguity to done — turn a rough product goal into a shipped, working capability with minimal hand-holding, and help others do the same.

What We're Looking For

• 8+ years of professional software engineering experience, with significant depth across backend and a track record of owning systems, not just features.
• Strong backend engineering, ideally in Python — building and scaling production services and APIs..
• Proven architecture and system-design ability — you have designed systems that scaled, and can reason clearly about trade-offs.
• Solid fundamentals across APIs, databases and cloud infrastructure.
• Experience building real-time and/or AI-powered products — or clear, demonstrable ability to lead in this area.
• A history of technical leadership — setting standards, mentoring engineers, and being trusted with the hardest problems — while remaining hands-on.
• Excellent communication and a genuine ownership mindset — someone who can be handed an ambiguous, high-stakes problem and be trusted to see it through.

Nice to Have

• Experience working with AI / large language models in production.
• Experience with voice or other real-time products.
• Exposure to healthcare, fintech, or other regulated / high-stakes domains.
• Experience as an early or senior engineer in a startup, where you set direction and wore many hats.

Product Engineer — Role Summary

We are looking for a Product Engineer to build user-facing products that combine AI capabilities with practical applications. You will work at the intersection of software engineering and AI, developing autonomous, agent-driven systems that solve complex educational and research problems.

Key Responsibilities:

• Product Development: Design, build, and deploy production-ready applications powered by LLMs and AI agents.
• Data Engineering: Build scalable ETL/ELT pipelines to process structured and unstructured data, including text and audio, for RAG and model fine-tuning.
• Agentic Workflows: Develop multi-step AI agents with tool calling, APIs, databases, search, reasoning, and memory.
• Rapid Prototyping: Turn ideas and research concepts into interactive, production-ready applications.
• AI Integration: Use frameworks such as LangChain, LlamaIndex, AutoGen, or custom orchestrators to integrate AI into scalable systems.
• User Experience: Transform raw AI outputs into reliable, intuitive, and responsive user experiences.
• Collaboration: Work closely with ML researchers and data engineers to integrate custom and fine-tuned models.
• Observability: Monitor agent behavior, manage edge cases, reduce hallucinations, and improve reliability in production.

The ideal candidate combines strong software engineering, AI/LLM expertise, data engineering, and product thinking, with the ability to take an AI concept from prototype to production.

About the team

SecurITe’s mission is to build an Agentic‑AI driven security platform that protects critical infrastructure from modern cyber threats. Our focus is on delivering highly performant, resilient, and intelligent network security systems that help defenders stay ahead of adversaries.

About the Role

We’re looking for a seasoned Senior Quality Engineer to provide technical leadership and architectural oversight for our next‑generation cybersecurity AI platform. In this high-impact role, you will define the technical strategy for quality assurance, ensuring our agentic AI transforms cyber defense with unparalleled reliability.

You will be responsible for the end-to-end quality lifecycle, from architectural reviews to the deployment of scalable automation frameworks. Beyond technical execution, you will serve as a mentor to junior team members, fostering a culture of technical excellence and driving the strategy that ensures our solutions meet the rigorous demands of critical infrastructure protection.

What You’ll Do

●    Defining and driving comprehensive QA strategies and roadmaps for the cybersecurity platform.

●    Designing, developing, and executing test plans, test cases, and automated scripts to ensure software quality.

●    Performing functional, regression, performance, scalability and security testing to identify bugs or defects.

●    Collaborating with developers, product managers, and other stakeholders to understand product requirements and testing needs.

●    Identifying, documenting, and tracking software defects, ensuring clear communication of issues and their resolutions.

●    Leading deep-dive root-cause analysis for critical system defects and security vulnerabilities.

●    Conducting thorough reviews of product specifications and software design to identify potential areas of concern before testing.

●    Architecting and designing complex, scalable test automation frameworks to optimize CI/CD velocity.

●    Ensuring the software meets customer and business requirements by validating the functionality and performance.

●    Assisting in continuously improving QA processes, tools, and best practices to enhance software testing efficiency and effectiveness.

●    Supporting user acceptance testing (UAT) and assisting clients with product validation.

●    Mentoring junior and mid-level engineers, providing technical guidance and conducting architectural reviews.

Required Experience

●    A Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or a related field.

●    8-10 years of proven experience in quality engineering, specifically within network cybersecurity, Identity Providers, or AI-integrated platforms.

●    Expertise in manual and automated testing.

●    Deep domain expertise in complex system validation and advanced automation practices at scale.

●    Proficiency in programming languages like Python to build and run automated test scripts.

●    "Strong knowledge of software testing methodologies, performance testing tools (e.g., JMeter, k6), and security traffic generation/simulation tools (e.g., Ixia BreakingPoint, Scapy, or Snort/Suricata traffic generators)."

●    Understanding of continuous integration/continuous deployment (CI/CD) pipelines and version control systems like Git.

●    Strong communication skills for documenting test results and interacting with cross-functional teams.

●    Excellent analytical skills, attention to detail, and problem-solving ability.

●    Ability to work independently as well as collaboratively in a team environment.

●    A curious mindset with a willingness to quickly learn new technologies and testing tools.

Required Skills & Qualifications

●    Familiarity with cloud-based testing environments (GCP, AWS, Azure).

●    Experience with cybersecurity products or cloud services or IDP or Web UI

The Mindset

●    Problem Solver: You thrive on complex, ambiguous challenges and engineer elegant solutions.

●    Ownership‑Driven: You take initiative, move fast, and deliver outcomes without hand‑holding.

●    Continuous Learner: You stay ahead of the curve in AI, ML, and emerging technologies.

●    Startup DNA: You excel in fast‑moving environments where priorities evolve and impact is immediate.

AI / Backend Engineer — Remote (India / Overseas)

Company: Ink'd

Compensation: $1,500–$2,500 USD/month, depending on experience

Type: Independent contractor, long-term opportunity

Ink'd is building a voice-first AI platform for residential real estate transactions. Our AI assistant allows real estate agents to generate contracts, automate workflows, and complete transaction documentation through natural conversation.

We're looking for a highly capable AI/backend engineer to help scale and maintain our production platform.

Responsibilities

• Build and maintain production AI agent workflows
• Develop integrations with third-party APIs and services
• Work with LLMs, structured data extraction, and document generation
• Troubleshoot production systems and improve reliability
• Build scalable backend services and maintain databases
• Collaborate directly with the founder and engineering team

Technical experience

• Python, FastAPI, Node.js, and REST APIs
• LLM APIs, tool calling, and agent orchestration
• PostgreSQL and cloud infrastructure
• GitHub, CI/CD, debugging, and deployment
• Experience with voice AI, WebSockets, or document processing is a major plus

What we're looking for

Someone who has independently shipped production software, can troubleshoot unfamiliar systems, communicates clearly, and takes ownership of their work.

Please include your GitHub, two examples of production systems you've built, your specific contributions, monthly compensation expectations, and availability.

Senior AI Engineer

Code Generation, Agent Architecture & LLM Systems

📍 Mumbai (On-site) | Full-time | 5+ years

About the Role:

Unico Connect is an AI-first technology partner that builds custom mobile, web, and AI products for clients across multiple geographies.

We are hiring a Senior AI Engineer for a dedicated client engagement focused on building an AI-powered application builder platform - a product where users describe software in plain English and the system generates, previews, and iteratively refines working code.

The mandatory requirement for this role is hands-on production experience shipping LLM-powered systems with agent architectures, with experience in code generation or developer tooling contexts a strong advantage.

The role is product-focused and deeply hands-on. You will own everything between the user's prompt and correct code landing in the project: the agentic loop, code generation pipeline, context management, evaluation suite, and model cost strategy.

You will work alongside the Senior MLOps Engineer who operationalises the infrastructure around your system, and collaborate closely with backend, frontend, and DevOps engineers.

Responsibilities:

Agent Architecture

Design and own the agentic loop for the platform - request interpretation, planning, tool-calling sequence (read file, edit file, run build, search code, install package), and stop conditions.

Make and revisit architectural decisions on single-agent vs. multi-agent designs, including planner/executor splits and dedicated build-repair sub-agents.

Code Generation Pipeline

Own the end-to-end generation flow: task classification, context gathering, planning, targeted edits, verification, and commit.

Implement diff/search-replace-based file editing with fuzzy matching and fallback strategies.

Enforce scope discipline so the agent makes minimal diffs and does not modify code it was not asked to touch.

Self-Repair Loop

Build and tune the automated repair loop that pipes compiler, lint, build, and runtime errors back to the model with retry budgets and model escalation.

This loop is the primary quality lever - the difference between 60-70% and 90%+ build success rates.

Context Management

Build file-relevance retrieval so the agent sees the right files, not the whole codebase: dependency graphs, AST/tree-sitter-based chunking, embeddings, recency signals, and hybrid retrieval.

Implement conversation summarisation and memory for long sessions, and address long-project degradation through codebase summaries and periodic consistency passes.

Own token budgeting and prompt caching strategy.

Prompt Engineering as a Discipline

Own the system prompt and per-task prompt variants (new feature, bug fix, styling change).

Maintain few-shot examples and enforce coding conventions, stack rules, and prohibited behaviours such as no hardcoded secrets and no whole-file rewrites.

Version prompts like code with changelogs and rollback capability.

Evaluation and Quality Measurement

Design and own the evaluation suite: representative test prompts run on every prompt and model change, scored on build success rate, instruction adherence, and output quality including LLM-as-judge and visual/screenshot checks where relevant.

Define regression gates that block quality-degrading changes from shipping.

Treat evals the way engineers treat automated testing: versioned, automated, and tracked over time.

This responsibility is non-negotiable at this level.

Model Strategy and Cost

Design model routing - cheap and fast models for classification and small edits, frontier models for complex generation.

Drive cost optimisation through prompt caching, diff-based edits over full-file rewrites, and tighter context selection.

Track cost per agent run and tokens per task; evaluate new model releases against the eval suite and lead migrations when results justify it.

Safety and Reliability of Agent Behaviour

Defend against prompt injection from user content and fetched web content.

Ensure secrets never appear in generated client code.

Define what the agent's tools may and may not do in collaboration with the platform team.

Contribute to output moderation and abuse-pattern awareness.

Mentorship and Engineering Standards

Run code reviews, define engineering conventions for AI work, and raise the engineering bar across the AI team.

Work closely with the Senior MLOps Engineer on handoff of eval design, prompt configurations, and model routing logic.

Requirements:

Hands-on Production Ownership of LLM-Powered Systems with Agent Architectures (Mandatory)

Must have personally shipped and operated at least one complex production AI system - agentic, multi-step, or code generation - with end-to-end ownership of architecture, evaluation, and cost.

POCs, internal demos, and tutorial-grade work do not qualify.

5+ Years of Professional Software or AI Engineering Experience

With at least 3 years focused on LLM applications, AI engineering, or production AI systems.

Candidates with strong backend backgrounds and a clear, substantive pivot into LLM systems qualify.

Strong Python Proficiency and Service Development

Production-grade Python with FastAPI or equivalent: type hints, async patterns, streaming responses, testing, and packaging.

Not notebook-only.

Depth Across LLM APIs and Agent Systems

Production experience with at least two of OpenAI, Anthropic Claude, Google Gemini, or open-weight models (vLLM, Ollama, Together).

Production experience with at least one agent framework (LangGraph, CrewAI, AutoGen, LlamaIndex Agents) or hand-rolled equivalent.

Hands-on with tool calling, structured outputs, and multi-step reasoning.

Demonstrated, Systematic Evaluation Practice - Non-Negotiable

Must have built evaluation harnesses that gate production releases, not ad-hoc testing.

Hands-on with at least one of LangSmith, Langfuse, Promptfoo, Ragas, or DeepEval.

Candidates with no systematic answer to evaluation should not be considered at senior level regardless of other strengths.

Cost Discipline for Production AI

Track record of measurable cost optimisation on production AI features.

Able to speak in specifics: cost per request, savings achieved through caching or model routing, context reduction decisions.

AWS Working Knowledge

Hands-on with EC2, S3, IAM, and Docker.

Comfort with CI/CD workflows and deploying AI services.

Awareness of LLM Security Failure Modes

Familiar with prompt injection patterns, understands that system prompt rules alone are insufficient, and has experience with output validation and content safety in production.

Nice to Have

• Experience with AST/tree-sitter tooling, diff-based editing systems, or compiler-adjacent work
• MCP server authoring
• Open-source AI contributions
• Published technical writing on LLM systems
• Multi-modal model experience
• Fine-tuning exposure (LoRA, QLoRA, PEFT)

LLM/AI Model Engineer – Training, Evaluation & Benchmarking

Location: New Delhi, India (On-site)

Employment Type: Full-time

Experience: 3–5 years

Industry: Enterprise AI / Generative AI / Machine Learning / NLP

About QubeLabs

QubeLabs is building the next generation of Enterprise AI Systems that transform workforce operations and intelligence for the financial services industry.

Our platform combines Conversational AI, Agentic AI, workflow automation, proprietary language models and enterprise intelligence. Our flagship product, QubeLabs Workmate, serves banks, NBFCs, MFIs, wealth management firms, insurance companies and fintechs across India and Europe.

At the core of our AI platform is the Vectro Series, our proprietary family of language models designed for enterprise and financial-services use cases.

We are looking for an LLM/AI Model Engineer to build the training pipelines, datasets and evaluation infrastructure required to continuously improve the Vectro Series.

Role Overview

You will work closely with the Head of AI, Research Partners and engineering teams to implement and operationalize LLM training approaches and build a reliable model improvement lifecycle:

Data Preparation → Training → Evaluation → Benchmarking → Error Analysis → Model Improvement

The role combines hands-on LLM fine-tuning, dataset engineering, tokenization, evaluation framework development and model quality management.

Key Responsibilities

1. LLM Training & Fine-tuning

Build and maintain pipelines for training and fine-tuning open-source foundation models for the Vectro Series.

Implement supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), LoRA/QLoRA and domain adaptation techniques; support continued pre-training where required.

Manage training configurations, checkpoints, model versions and experiment tracking.

Optimize training workflows for computational efficiency, reproducibility and reliability.

Translate model development approaches defined with the Head of AI and Research Partners into practical, scalable training pipelines.

2. Training Data & Tokenization

Build data pipelines to collect, clean, filter, normalize, deduplicate and validate training data.

Develop instruction-tuning, SFT and domain-specific datasets for financial-services and enterprise use cases.

Implement tokenization workflows, tokenizer configuration, vocabulary management, sequence packing, truncation and padding.

Address data representation challenges involving financial terminology, numerical values and multilingual content.

Maintain dataset versioning, lineage and quality controls, incorporating identified data gaps and relevant model failures into future training data.

3. Golden Dataset & Model Evaluation

Build and maintain QubeLabs' Golden Dataset to evaluate key model capabilities and target use cases.

Develop automated evaluation pipelines and reusable evaluation harnesses, combining automated metrics with human assessment where appropriate.

Evaluate accuracy, reasoning, factuality, instruction following, safety, multilingual performance and financial-domain capabilities.

Establish consistent evaluation methodology and regression tests to measure changes across model versions.

4. Benchmarking & Error Analysis

Evaluate Vectro against relevant public benchmarks and proprietary QubeLabs BFSI/enterprise benchmarks.

Compare results with relevant open-source and commercial models using consistent evaluation conditions.

Analyze failures, identify root causes across data, training and model behavior, and translate findings into actionable improvements.

Maintain reproducible benchmark results and reports to track model strengths, limitations and progress.

5. Model Quality & Release

Define model release quality gates and ensure each major release meets agreed evaluation and benchmark criteria.

Support red-teaming, robustness and safety testing.

Coordinate with the Head of AI, Research Partners and engineering teams to ensure validated model improvements are suitable for integration into production systems.

Experience

3–5 years in Machine Learning, Deep Learning, NLP, Generative AI or related fields.

Hands-on experience training or fine-tuning LLMs using open-source foundation models.

Practical experience building training pipelines, preparing datasets and evaluating model performance.

Experience with GPU-based training; distributed training is an advantage.

Financial-services, multilingual AI or enterprise AI experience is preferred.

Required Skills

LLM & Machine Learning: Large Language Models, Transformer architectures, Generative AI, NLP, Deep Learning, SFT, PEFT, LoRA/QLoRA, fine-tuning and continued pre-training.

Frameworks & Infrastructure: Python, PyTorch, Hugging Face Transformers, Hugging Face Tokenizers, GPU computing, experiment tracking and ML pipelines.

Data & Tokenization: Tokenization, tokenizer configuration, text preprocessing, sequence packing, dataset construction, data quality, versioning and lineage.

Evaluation & Benchmarking: Golden datasets, LLM evaluation, evaluation harnesses, automated and human evaluation, LLM-as-a-Judge, public and domain-specific benchmarks, regression testing and error analysis.

The role requires the ability to build both the model training pipeline and the measurement system that determines whether the model has actually improved.

What Success Looks Like

Reliable and reproducible training and fine-tuning pipelines for the Vectro Series.

High-quality training datasets, tokenization workflows and a comprehensive Golden Dataset.

Automated evaluation and benchmarking infrastructure with measurable model quality gates.

Clear, reproducible evidence of model performance against public, proprietary BFSI and relevant competing-model benchmarks.

A continuous improvement loop that converts evaluation findings into measurable gains across successive Vectro releases.

Key Performance Indicators

Training pipeline reliability and experiment throughput.

Training data quality and coverage.

Golden Dataset and evaluation coverage.

Benchmark reproducibility and model performance improvement.

Accuracy of error diagnosis and effectiveness of regression detection.

Time required to complete training, evaluation and benchmarking cycles.

Compliance with model release quality criteria.

Why Join QubeLabs?

Build the proprietary Vectro Series of enterprise and financial-services language models.

Develop model training, evaluation and benchmarking infrastructure from the ground up.

Work closely with the Head of AI and Research Partners.

Solve real-world AI challenges across financial services.

Develop proprietary datasets, benchmarks and model improvement systems.

Contribute directly to production AI systems serving customers across India and Europe.

Preferred Candidate Profile

B.Tech, M.Tech, MS or PhD in Computer Science, AI, ML, Mathematics or a related field.

3–5 years of relevant experience with strong practical skills in Python, PyTorch, LLM fine-tuning and training data pipelines.

Working knowledge of tokenization, evaluation datasets, benchmarking and regression testing.

Experience with open-source models such as Llama, Qwen, Mistral or similar is preferred.

Ability to independently build reliable model training and evaluation systems from scratch.

QubeLabs is an equal opportunity employer and values diversity and inclusion. We encourage applications from qualified candidates regardless of background or identity.

Role Overview

We are looking for an experienced professional to evaluate AI-generated responses for conversational AI agents used in travel-related products. The role focuses on assessing response accuracy, relevance, completeness, and quality to improve AI performance.

Key Responsibilities

• Evaluate AI-generated responses against predefined guidelines and quality standards.
• Perform data annotation, labeling, classification, and quality reviews.
• Identify incorrect, misleading, irrelevant, or incomplete responses and potential AI hallucinations.
• Analyze evaluation data to identify recurring errors, quality gaps, and improvement opportunities.
• Track quality metrics, document findings, and recommend improvements.
• Use Excel/Google Sheets and basic SQL, where applicable, to analyze evaluation data.
• Collaborate with AI and Machine Learning teams to improve conversational AI performance.

Required Skills

• 5–8 years of experience in Data Annotation, AI Evaluation, Content Quality, Search Relevance, or Quality Assurance.
• Strong analytical, problem-solving, and attention-to-detail skills.
• Experience applying evaluation guidelines and quality standards consistently.
• Proficiency in Excel or Google Sheets.
• Good written communication and documentation skills.
• Understanding of quality metrics and data analysis.

Good to Have

• Exposure to Generative AI, LLMs, conversational AI, or AI response evaluation.
• Experience with Human-in-the-Loop (HITL), AI training data validation, or hallucination detection.
• Basic SQL knowledge.
• Experience in travel, hospitality, or search quality evaluation.

Strong AI Engineer / Machine Learning Engineer profiles.

2

Mandatory (Experience 1) – Must have minimum 3+ years of hands-on experience in Data Science, Machine Learning, Applied AI, NLP, Deep Learning, or Generative AI solutions.

3

Mandatory (Experience 2) – Must have strong hands-on experience in Python programming, SQL, data analysis, feature engineering, model development, and production-grade ML applications.

4

Mandatory (Experience 3) – Must have experience working with Machine Learning and Deep Learning frameworks such as PyTorch, TensorFlow, Keras, Scikit-learn, or equivalent.

5

Mandatory (Experience 4) – Must have hands-on experience working on NLP, embeddings, semantic search, text classification, document understanding, recommendation systems, or similar AI/ML use cases.

6

Mandatory (Experience 5) – Must have experience working with Large Language Models (LLMs) such as GPT, Llama, Mistral, Claude, Gemini, Phi, or similar foundation models.

7

Mandatory (Experience 6) – Must have hands-on experience building or implementing RAG (Retrieval Augmented Generation) systems, vector search, knowledge retrieval, embeddings, chunking, indexing, or semantic retrieval solutions.

8

Mandatory (Experience 7) – Must have experience working with Git, CI/CD practices, production environments, and scalable AI/ML systems.

9

Mandatory (CTC) – The CTC breakup offered will be 75% fixed + 25% variable, as per company policy.

10

Mandatory (Age) - Candidate's Age should be below 28 Years