Cutshort logo
MeltPlan logo
AI System QA Engineer – Large Language Models (Evaluation Testing)
AI System QA Engineer – Large Language Models (Evaluation Testing)
MeltPlan's logo

AI System QA Engineer – Large Language Models (Evaluation Testing)

Gaurav Bhardwaj's profile picture
Posted by Gaurav Bhardwaj
6 - 10 yrs
₹25L - ₹35L / yr
Bengaluru (Bangalore)
Skills
Data-driven testing
Playwright
DeepEval
Ragas
LLM Evaluation Frameworks
MLOps

MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. While design software optimizes use and aesthetics and construction software optimizes execution and control, MeltPlan is building the missing layer - software that optimizes decisions and tradeoffs upstream, before scope is locked, procurement begins, and change orders become inevitable. MeltPlan’s long-term goal is to help teams make construction “boring” by making planning more intense: surfacing constraints and tradeoffs early, aligning stakeholders before plans are frozen, and reducing the need for late-stage redlines, rework, and change orders.


MeltPlan is founded by operators who have built at scale. Kanav previously co-founded Innovaccer, a $3Bn healthtech company focused on making US healthcare more affordable and accessible. He’s now applying that systems-level thinking to construction.He’s joined by Tanmaya Kala, former Project Executive at DPR Construction, who led large commercial, healthcare, and life sciences projects. We combine deep tech scale with real construction execution.


What This Role Really is :


We are seeking a detail-oriented and technically strong AI QA Engineer to ensure the quality, reliability, and performance of Large Language Model (LLM)-based systems. In this role, you will be responsible for designing and executing test strategies, validating model outputs, and building evaluation frameworks to enhance the accuracy, safety, and overall performance of AI-driven applications.We would particularly value candidates who have hands-on experience in developing evaluation frameworks (evals) for AI systems, along with strong expertise in comprehensive system testing and quality assurance practices.You are responsible for making MeltPlan work in the real world.


What You'll Do:


  • Design, develop, and execute evaluation frameworks (Evals) for Large Language Models (LLMs) and AI systems.
  • Perform end-to-end system testing, regression testing, and performance testing for AI-driven applications.
  • Validate model outputs for accuracy, consistency, safety, hallucination detection, and edge cases.
  • Build automated test pipelines and quality benchmarks for AI systems.
  • Collaborate closely with AI/ML engineers, product teams, and platform engineers to improve system reliability.
  • Analyze failures, identify root causes, and provide actionable feedback to improve model behavior.
  • Develop datasets, prompts, and testing scenarios to measure model performance across multiple use cases.
  • Monitor production performance and continuously improve evaluation metrics and testing standards.
  • Ensure compliance with responsible AI and quality assurance best practices.


What We're looking for: 


  • Bachelor’s degree in Computer Science, Engineering, or related field
  • 5–9 years of experience in QA/testing, preferably in AI/ML or data-driven systems
  • Strong experience in AI/LLM evaluation frameworks and system testing.
  • Hands-on experience with automated testing methodologies and QA processes.
  • Familiarity with prompt engineering, AI benchmarking, and model validation techniques.
  • Experience working with Python and testing frameworks.
  • Understanding of LLM behaviors, hallucinations, prompt injection risks, and AI safety concepts.
  • Exposure to tools/frameworks such as OpenAI Evals, LangSmith, DeepEval, Promptfoo, or similar platforms is preferred.
  • Strong analytical and debugging skills with attention to detail.
  • Excellent collaboration and communication skills.
  • Familiarity with Large Language Models and Generative AI concepts
  • Experience with API testing tools (e.g., Postman) and automation frameworks
  • Understanding of NLP concepts such as tokenization, embeddings, and text generation
  • Strong analytical and problem-solving skills
  • Experience testing AI/ML models or data pipelines
  • Experience with prompt engineering and prompt testing
  • Familiarity with cloud platforms (AWS, GCP, or Azure)
  • Exposure to AI safety, bias detection, and model governance


Bonus if you have:


  • Have worked in construction or on project sites
  • Have startup experience
  • Experience working with Generative AI or conversational AI products.
  • Knowledge of CI/CD pipelines and automation workflows.
  • Prior experience in performance testing and monitoring distributed systems.
  • Understanding of AI product lifecycle and production deployment environments.


We’re not looking for someone who waits for clean requirements.We’re looking for someone who thrives in the mess and turns it into systems.


Why meltplan

  • Massive industry, real-world impact
  • High ownership from day one
  • Small team, zero bureaucracy
  • Competitive comp + meaningful equity
Read more
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos

About MeltPlan

Founded :
2024
Type :
Product
Size :
20-100
Stage :
Raised funding

About

MeltPlan is the AI-native planning software for construction – automate takeoffs with AI-assisted service, ensure building code compliance, and streamline cost planning with transparent, expert AI that shows its work. Designed for architects, engineers, contractors, and inspectors.
Read more

Company social profiles

blog

Similar jobs

Wissen Technology
at Wissen Technology
4 recruiters
Akansha Sharma
Posted by Akansha Sharma
Bengaluru (Bangalore)
4 - 6 yrs
Best in industry
Generative AI
MLOps
Data engineering
Big Data


Notice Period - 0-15 days Max

Apply only who are currently in Karnataka

F2F interview

Interview - 4 rounds


Job Title: AI Specialist

Company Overview: We are the Technology Center of Excellence for Long Arc Capital

which provides growth capital to businesses with a sustainable competitive advantage and 

a strong management team with whom we can partner to build a category leader. We focus 

on North American and European companies where technology is transforming traditional 

business models in the Financial Services, Business Services, Technology, Media and 

Telecommunications sectors.

As part of our mission to leverage AI for business innovation, we are establishing AI COE to 

develop Generative AI (GenAI) and Agentic AI solutions that enhance decision-making, 

automation, and user experiences.

Job Overview: We are seeking dynamic and talented individuals to join our AI COE. This 

team will focus on developing advanced AI models, integrating them into our cloud-based 

platform, and delivering impactful solutions that drive efficiency, innovation, and customer 

value.

Key Responsibilities:

• As a Full Stack AI Engineer, research, design, and develop AI solutions for text, 

image, audio, and video generation

• Build and deploy Agentic AI systems for autonomous decision-making across 

business outcomes and enhancing associate productivity. 

• Work with domain experts to design and fine-tune AI solutions tailored to portfoliospecific challenges.

• Partner with data engineers across portfolio companies to –

o Preprocess large datasets and ensure high-quality input for training AI 

models.

o Develop scalable and efficient AI pipelines using frameworks like 

TensorFlow, PyTorch, and Hugging Face.

• Implement MLOps best practices for AI model deployment, versioning, and 

monitoring using tools like MLflow and Kubernetes.

• Ensure AI solutions adhere to ethical standards, comply with regulations (e.g., 

GDPR, CCPA), and mitigate biases.

• Design intuitive and user-friendly interfaces for AI-driven applications, collaborating 

with UX designers and frontend developers.

Internal Use Only

• Stay up to date with the latest AI research and tools and evaluate their applicability 

to our business needs.

Key Qualifications:

Technical Expertise:

• Proficiency in full stack application development (specifically using Angular, React).

• Expertise in backend technologies (Django, Flask) and cloud platforms (AWS 

SageMaker/Azure AI Studio).

• Proficiency in deep learning frameworks (TensorFlow, PyTorch, JAX).

• Proficiency with Large Language Models (LLMs) and generative AI tools (e.g., OpenAI 

APIs, LangChain, Stable Diffusion).

• Solid understanding of data engineering workflows, including ETL processes and 

distributed computing tools (Apache Spark, Kafka).

• Experience with data pipelines, big data processing, and database management 

(SQL, NoSQL).

• Knowledge of containerization (Docker) and orchestration (Kubernetes) for scalable 

AI deployment.

• Familiarity with CI/CD pipelines and automation tools (Terraform, Jenkins).

• Good understanding of AI ethics, bias mitigation, and compliance standards.

• Excellent problem-solving abilities and innovative thinking.

• Strong collaboration and communication skills, with the ability to work in crossfunctional teams.

• Proven ability to work in a fast-paced and dynamic environment.

Preferred Qualifications:

• Advanced studies in Artificial Intelligence, or a related field.

• Experience with reinforcement learning, multi-agent systems, or autonomous 

decision-making

Read more
Product based company
Product based company
Agency job
via Vmultiply solutions by Mounica Buddharaju
Bengaluru (Bangalore)
3 - 5 yrs
₹10L - ₹15L / yr
skill iconReact.js
skill iconJavascript
skill iconHTML/CSS
skill iconNextJs (Next.js)

JD : 

React. Js Developer


Skill Sets: HTML, CSS, JS , Typescript, Nextjs

Location : Bangalore , Complete WFO


Address: 

Embassy Tech Village Rd, Devarabisanahalli, Bellandur, Bengaluru, Karnataka 560103, India.


Candidate Persona:

Need Product based Background Folks only or someone who has worked on product projects. 

Resume is extremely crucial here, if roles and responsibilities, Education details and company names are not mentioned properly in utmost details then they will not be considered further.

Communication skills- Good 

Btech graduated ONLY

Skills that should be mentioned in the CV - HTML, css, js, typescript, Data structures and algorithm. 

Candidates should have a linkedin ID. 

Candidates should have some leet code or hackerrank links (preferred)



Note : candidates who are ready for all the rounds can apply.

Read more
US Based MNC
US Based MNC
Agency job
via People First Consultants by Naveed Mohd
Chennai
9 - 16 yrs
₹16L - ₹36L / yr
skill iconNodeJS (Node.js)
skill iconReact.js
skill iconAngular (2+)
skill iconAngularJS (1.x)
MERN Stack
+5 more
DESIRED SKILLS AND EXPERIENCE
  •  Strong analytical and problem-solving skills
  •  Ability to work independently, learn quickly and be proactive
  •  10-14 years of hands-on experience working on Web Full Stack technologies, with at least 4-6 years of experience developing applications with/on React/NextJS, NoSQL, REST APIs
  •  Proficiency in JavaScript/TypeDcript (ES6), NodeJS, HTML5, CSS3, CSS Preprocessors, Webpack, Gulp
  •  Client-side scripting and JavaScript frameworks – jQUery, ReactJS, Redux, Babel, JSX
  •  Experience in designing high-performance REST APIs and associated data structures
  •  Familiarity with developing microservices using containerization technologies such as docker, Kubernetes, etc.
  •  Working knowledge of git and using branches for development
Read more
Auraglobal
Raveen Prasad
Posted by Raveen Prasad
Chennai
3 - 8 yrs
₹5L - ₹8L / yr
skill iconReact.js
skill iconReact Native
skill iconJavascript
Communication Skills
Team Management
Designation: React Native and React JS developers.

As part of our Developer team you will get high quality experience in a young, high energy, friendly, fun and professional environment where people work together as a family. Our employees are constantly learning, upgrading their skills, developing their competences and finding ways to fully use their potential.

Aura Global Developers professionals work with innovative technologies, build world-class products and cooperate with global customers. Being part of Aura Global means working in an independent, well-organized and collaborative team.

SKILLS NEEDED - React Native, React JS, GraphQL, Postgres SQL

-3 to 7 years of experience is required.

-Should be able to handel a team.

-Designing and developing user interfaces using React JS best practices.

-Adapting interface for modern internet applications using the latest front-end technologies.

-Writing JavaScript, CSS, and HTML.

-Developing product analysis tasks.
-Developing application codes and unit tests in React Native and GraphQL Services
-Consulting with the design team.
-Ensuring high performance of applications and providing support.
-Ensuring optimal performance of the central database and responsiveness to front-end requests
-Collaborating with Back-end developers on the integration of elements
-Designing customer-facing UI
-Developing high-performance applications by writing testable, reusable, and efficient code
-Implementing effective security protocols, data protection measures, and storage solutions
-Running diagnostic tests, repairing defects, and providing technical support

-Technical Experience: React JS, React Native, GraphQL.

-Good Analytical skill to analyze Product Requirements, Agile Methodology and Process Knowledge of Banking Payment Gateway Domain is added advantage Actively participate in Agile Daily stand up and Jira

-Professional Attributes Able to work independently

-Educational Qualification: BE

Contact: SEVEN THREE FIVE EIGHT THREE THREE SEVEN FOUR NINE FIVE.
Read more
Synapsica Technologies Pvt Ltd
at Synapsica Technologies Pvt Ltd
6 candid answers
1 video
Human Resources
Posted by Human Resources
Bengaluru (Bangalore)
1 - 4 yrs
₹10L - ₹20L / yr
Product Management
Product Manager
Systems Development Life Cycle (SDLC)
STLC

We are looking for a passionate, highly driven, intrinsically motivated Associate Product Manager who wants to join a high-growth startup, learn something new everyday and join of one the most energetic and speedy Product Teams in health-tech!



Responsibilities

 

  • Building and executing new initiatives and roadmaps for retention and increasing customer lifetime value.
  • Understand the healthcare market, customers and build business cases for new product opportunities.
  • Listen to the users on a regular basis and figure out the opportunities to solve their problems.
  • Manage product lifecycle from ideation to launch and beyond which would include liaising with multiple stake-holders.
  • Work closely with partners and clients from 4 continents to localize the product builds as per their need and ensure adoption of the new features.
  • Use data, creativity, and experimentation to constantly improve the product experience

Requirements


  • Have 1-4 years of experience in building and managing large-scale enterprise products.
  • Have prior experience in the development team and understand how modern development frameworks function.
  • Are passionate about translating customer needs to usable design and process flows for growth levers and optimization.
  • Have Good understanding of funnels, agile & sprints, and wireframes.
  • Display the attitude to be comfortable with ambiguity and have the skill to transform broad ideas into action plans and display Empathy towards users and also your colleagues.
  • Understand how the application functions and the technicalities around it.
  • Have familiarity with tools like Google data studio, Firebase, Google Analytics, Product Analytics tools, and Figma.
  • Love analytics and very frequent experimentation
Read more
Our client company is into E-commerce.(I1)
Our client company is into E-commerce.(I1)
Agency job
via Multi Recruit by Ashwini Miniyar
Bengaluru (Bangalore), Hyderabad, Chennai, Pune, Noida, Delhi
8 - 13 yrs
₹20L - ₹40L / yr
C#.Net
skill iconC#
skill iconReact.js
skill icon.NET
Dot Net
+1 more

Our client is looking for a Senior/Lead Full Stack .Net Developer for their headquarter office which is located in San Francisco, California. Looking for top-notch developers who are passionate and well-versed with modern technologies like .Net Core, MVC, SQL, and JS frameworks.

This position gives you an opportunity to lead exciting projects, work directly with teams in the US and with super smart people that you can learn from, and contribute to their knowledge.

Requirement:

  • 7-15 years of experience in software development.
  • Hands-on design and coding are required.
  • Expertise in React.JS.
  • Good knowledge of database concepts and Microsoft SQL Server.
  • Highly proficient in modern JavaScript, HTML, CSS, ReactJs, and in one or more libraries for state management (e.g. Redux)
  • Solid foundation in computer science, with strong competencies in data structures, design patterns, concurrency, algorithms, and software design
  • Research and evaluate new software, frameworks, and techniques to provide recommendations to the division.
  • Design and develop robust and scalable software components.
  • Strong analytical and troubleshooting skills.
  • Bachelor's degree in Computer Science or a technical field.


Perks & Benefits:

  • Friendly, talented, collaborative, and entrepreneurial team
  • Competitive and comprehensive benefits and perks
  • Generous holiday and PTO policies
  • Training and development opportunities and allowance
  • Fun and inclusive digital, and (in the future) in-person events
  • Employee groups - DEI committee, fun committee, wellness group, and more
  • Flexible remote work


About company

The company is the industry-leading collection of online destinations (including Astrology.com, Horoscope.com, Keen, and PsychicCenter) providing spiritual guidance on love, relationships, career, health, and life overall. They are passionate about connecting people with the world's best advisors (including psychics, tarot card readers, life coaches, and more) and content to empower everyone to live happier lives.

A team of over 100 employees is powered by their diverse perspectives and company core values:

  • They are humble. They believe the best result is achieved by leveraging others perspectives
  • They think like owners. They make decisions that optimize for the greater good of the organization
  • They challenge limiting beliefs. They are at their best when they identify and shatter the status quo expectations.


The company is an equal opportunity workplace and is an affirmative action employer. They are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity, or Veteran status.

 

Read more
SaltNow
at SaltNow
1 recruiter
Shadab Khatib
Posted by Shadab Khatib
Remote, Belgaum, Hubli, Dharwad, Bengaluru (Bangalore), Mumbai
1 - 5 yrs
₹0.1L - ₹0.2L / yr
Presales
Business Development
Client Servicing
Sales
Salesforce
Responsibilities and Duties Familiar with software and IT services and products business. Analysing customer requirements Pricing the business and customer requirements Achieving sales targets by acquiring new clients and growing business from existing clients Maintaining intuitive databases of existing and new clients using existing CRM software Have a passion for increasing sales and closing deals particularly with overseas clients. Business development activities for Android App, Mobile Applications, ERPs, Websites and other software related products on web and mobile, iot and other similar industry aspects Requirements Strong verbal and written English communication skills. Bachelors or Equivalent MBA preferred Previous successful experience with business development Operating from abroad or India both preferred. Must be able to work with a small team. Salary is given in % of business generated by the person.
Read more
SecurEyes
at SecurEyes
1 recruiter
Preeti Paul
Posted by Preeti Paul
Bengaluru (Bangalore)
8 - 13 yrs
₹10L - ₹12L / yr
MS SQLServer
DBA
Microsoft SQL Server DBA
MySQL DBA
Database migration

Job Title – Sr. Database Developer and DBA MS SQL Server

Job Location – Bangalore

Experience – 8 to 10 years

CTC – 10-12 LPA

Job Description

  • 8-10 years of industry experience with an appropriate University technology degree (BE,BTech,MCA)
  • Should have extensive Hands-on experience in managing RDBMS Production and application DBA for MS-SQL Database.
  • Must be proficient in MS-SQL programming and can handle complex logic.
  • Knowledge of data structure and algorithms is a plus.
  • Must have good experience in SSIS and SSRS using MSSQL Sever 2016 or later
  • Monitoring, maintaining, Query tuning, performance tuning, cluster management for all RDBMS Production and application (MS-SQL).
  • Database server administration with focus on security, automation, tuning, optimization, and standardization of new and deployed systems.
  • Be able to work as a DBA lead and individual contributor for RDBMS platform.
  • Will be responsible for database server planning with regards to scalability, redundancy, and data preservation of backend systems.
  • Will develop High Availability Clustered server/network RDBMS topologies.
  • Write comprehensive documentation, help develop tools to monitor systems performance, and work on optimization and tuning of various systems.
  • Working with Production support teams on troubleshooting Incidents on a 24x7 basis. Looking for Sr. Sql Server DBA Who Would Provide Technical Support In Analysis, Design, Testing, And Deployment Of All Database Platform.
Read more
IT Software MNC
IT Software MNC
Agency job
via Suven Consultants by Neetu Sani
Kochi (Cochin)
10 - 18 yrs
₹15L - ₹20L / yr
skill icon.NET
Technical Architecture
skill iconC#
Web API
skill iconAngular (2+)
+11 more
  • Strong object oriented concepts. Experience in structure object oriented modelling with preferred expertise in using tools like Enterprise Architect or similar. Experience in working with TOGAF standards is preferred.
  • Experience in architecting the technical scale and scope of high volume, scalable enterprise software solutions including logical and physical landscape requirements with specific attention to design, development, and deployment strategies
  • Capability to adapt, learn and work with multiple technology platforms. 
  • Knowledge in Application Security including Information security principles & realization, OWASP & PCI DSS Compliance ( Security Design & Technology Skills )
  • In depth knowledge and experience in large scale database management, data modelling and database design in RDBMS and NoSQL.
  • Experience in recommending and implementing DevOps tools for enterprise projects.
  • Capability to evaluate tools, technologies and processes, including assessing their strategic benefit in the solution.
  • Willingness to work hands-on with engineers to review, troubleshoot coding problems quickly and efficiently.
  • Expertise in following technologies – ASP.Net MVC, Web API, ASP.Net Core, Entity Framework, Entity Framework Core, ASP.Net Identity, REST
  • Experience in implementing various application deployment models and monitoring the server infrastructure using industry standard tools.
  • Experience in docker based deployment models.
  • Experience in architecting, developing and deploying cloud based (One or more among AWS, Azure, Google Cloud) enterprise solutions.
  • Experience in designing and developing micro-services based applications. 
  • Experience in designing and developing solutions with TDD (Test Driven Development)
Read more
Koramangala, Bengaluru (Bangalore)
6 - 7 yrs
₹4L - ₹4L / yr
skill icon.NET
Technical Training
Learning & Development
Research & Development
Education & Research Management
We are looking for a Training Specialist to enhance the competencies of individual employees by designing and conducting training programs that will boost employees workplace performance in alliance with company’s core values. You will be responsible for performing training needs assessments, designing and delivering curriculum and learning materials and for managing all phases of training interventions. Responsibilities Identify training needs by evaluating strengths and weaknesses Translate requirements into trainings that will groom employees for the next step of their career path Build annual training program and prepare teaching plans Develop or oversee the production of classroom handouts, instructional materials, aids and manuals Direct structured learning experiences and monitor their quality results Acclimate new hires to the business and conduct orientation sessions Deliver training courses Assess training effectiveness to ensure incorporation of taught skills and techniques into employees work behavior Periodically evaluate ongoing programs to ensure that they reflect any changes Stay abreast of the new trends and tools in employee development Requirements Proven experience in designing multiple training events in a corporate setting Extensive knowledge of instructional design theory and learning principles Proven ability to master the full training cycle Adequate knowledge of learning management software Familiarity with traditional and modern training methods, tools and techniques Familiarity with talent management and succession planning Ability to conduct cost-benefit analysis and calculate training ROI Sound decision making and organisational skills Ability to present complex information to a variety of audiences Proficiency in MS Office and in database software BS degree in Education, Training, HR or related field
Read more
Why apply to jobs via Cutshort
people_solving_puzzle
Personalized job matches
Stop wasting time. Get matched with jobs that meet your skills, aspirations and preferences.
people_verifying_people
Verified hiring teams
See actual hiring teams, find common social connections or connect with them directly.
ai_chip
Move faster with AI
We use AI to get you faster responses, recommendations and unmatched user experience.
Did not find a job you were looking for?
icon
Search for relevant jobs from 10000+ companies such as Google, Amazon & Uber actively hiring on Cutshort.
companies logo
companies logo
companies logo
companies logo
companies logo
Get to hear about interesting companies hiring right now
Company logo
Company logo
Company logo
Company logo
Company logo
Linkedin iconFollow Cutshort
Users love Cutshort
Read about what our users have to say about finding their next opportunity on Cutshort.
Shubham Vishwakarma's profile image

Shubham Vishwakarma

Full Stack Developer - Averlon
I had an amazing experience. It was a delight getting interviewed via Cutshort. The entire end to end process was amazing. I would like to mention Reshika, she was just amazing wrt guiding me through the process. Thank you team.
Companies hiring on Cutshort
companies logos