AI System QA Engineer – Large Language Models (Evaluation Testing)
at MeltPlan
MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. While design software optimizes use and aesthetics and construction software optimizes execution and control, MeltPlan is building the missing layer - software that optimizes decisions and tradeoffs upstream, before scope is locked, procurement begins, and change orders become inevitable. MeltPlan’s long-term goal is to help teams make construction “boring” by making planning more intense: surfacing constraints and tradeoffs early, aligning stakeholders before plans are frozen, and reducing the need for late-stage redlines, rework, and change orders.
MeltPlan is founded by operators who have built at scale. Kanav previously co-founded Innovaccer, a $3Bn healthtech company focused on making US healthcare more affordable and accessible. He’s now applying that systems-level thinking to construction.He’s joined by Tanmaya Kala, former Project Executive at DPR Construction, who led large commercial, healthcare, and life sciences projects. We combine deep tech scale with real construction execution.
What This Role Really is :
We are seeking a detail-oriented and technically strong AI QA Engineer to ensure the quality, reliability, and performance of Large Language Model (LLM)-based systems. In this role, you will be responsible for designing and executing test strategies, validating model outputs, and building evaluation frameworks to enhance the accuracy, safety, and overall performance of AI-driven applications.We would particularly value candidates who have hands-on experience in developing evaluation frameworks (evals) for AI systems, along with strong expertise in comprehensive system testing and quality assurance practices.You are responsible for making MeltPlan work in the real world.
What You'll Do:
- Design, develop, and execute evaluation frameworks (Evals) for Large Language Models (LLMs) and AI systems.
- Perform end-to-end system testing, regression testing, and performance testing for AI-driven applications.
- Validate model outputs for accuracy, consistency, safety, hallucination detection, and edge cases.
- Build automated test pipelines and quality benchmarks for AI systems.
- Collaborate closely with AI/ML engineers, product teams, and platform engineers to improve system reliability.
- Analyze failures, identify root causes, and provide actionable feedback to improve model behavior.
- Develop datasets, prompts, and testing scenarios to measure model performance across multiple use cases.
- Monitor production performance and continuously improve evaluation metrics and testing standards.
- Ensure compliance with responsible AI and quality assurance best practices.
What We're looking for:
- Bachelor’s degree in Computer Science, Engineering, or related field
- 5–7 years of experience in QA/testing, preferably in AI/ML or data-driven systems
- Strong experience in AI/LLM evaluation frameworks and system testing.
- Hands-on experience with automated testing methodologies and QA processes.
- Familiarity with prompt engineering, AI benchmarking, and model validation techniques.
- Experience working with Python and testing frameworks.
- Understanding of LLM behaviors, hallucinations, prompt injection risks, and AI safety concepts.
- Exposure to tools/frameworks such as OpenAI Evals, LangSmith, DeepEval, Promptfoo, or similar platforms is preferred.
- Strong analytical and debugging skills with attention to detail.
- Excellent collaboration and communication skills.
- Familiarity with Large Language Models and Generative AI concepts
- Experience with API testing tools (e.g., Postman) and automation frameworks
- Understanding of NLP concepts such as tokenization, embeddings, and text generation
- Strong analytical and problem-solving skills
- Experience testing AI/ML models or data pipelines
- Experience with prompt engineering and prompt testing
- Familiarity with cloud platforms (AWS, GCP, or Azure)
- Exposure to AI safety, bias detection, and model governance
Bonus if you have:
- Have worked in construction or on project sites
- Have startup experience
- Experience working with Generative AI or conversational AI products.
- Knowledge of CI/CD pipelines and automation workflows.
- Prior experience in performance testing and monitoring distributed systems.
- Understanding of AI product lifecycle and production deployment environments.
We’re not looking for someone who waits for clean requirements.We’re looking for someone who thrives in the mess and turns it into systems.
Why meltplan
- Massive industry, real-world impact
- High ownership from day one
- Small team, zero bureaucracy
- Competitive comp + meaningful equity
Shubham Vishwakarma
Full Stack Developer - Averlon
About MeltPlan
About
Company social profiles
Similar jobs
XMLPosition: Oracle Integration Cloud (OIC) Consultant
Experience: 10+ Years
Notice Period: Immediate to Maximum 30 Days
Employment Type: Full-Time
About the Role
We are seeking an experienced Oracle Integration Cloud (OIC) Consultant with strong expertise in designing, developing, and supporting enterprise integrations across Oracle Cloud and third-party applications.
Key Responsibilities
· Design, develop, test, and deploy integrations using Oracle Integration Cloud (OIC).
· Develop and maintain REST/SOAP web services and orchestrations.
· Build file-based, event-driven, and real-time integrations.
· Work on OIC adapters such as ERP, HCM, SOAP, REST, FTP, DB, and Stage File adapters.
· Troubleshoot and resolve integration issues.
· Participate in client discussions and architecture reviews.
Required Skills
· 10+ years of IT experience with strong hands-on experience in Oracle Integration Cloud (OIC).
· Strong experience integrating Oracle Fusion Applications, Oracle EBS, and third-party systems.
· Expertise in REST APIs, SOAP Web Services, XML, XSLT, and JSON.
· Good understanding of Oracle SaaS data models and business processes.
· Strong debugging and performance tuning skills.
Preferred Qualifications
· Experience with Oracle Visual Builder (VBCS/VB Studio) is an added advantage.
· Knowledge of Oracle ATP/ADW, OCI services, or SOA Suite is preferred.
· Oracle OIC Certification preferred.
Candidate Criteria
· Candidate should be available to join within a maximum of 30 days.
· Ability to work independently in a fast-paced consulting environment.
Soft Skills
· Strong analytical and problem-solving abilities.
· Excellent verbal and written communication skills.
· Strong stakeholder and client management capabilities.
React.jsPosition: React Js Developer
Location: Chennai
Job Specification
- Research, design, implement and manage front end applications Integrating and testing with Restful APIs.
- Identify the areas for modification in existing programs and subsequently developing these modifications
- Writing and implementing efficient code Determining operational practicality
- Work in an agile team Maintaining and upgrading existing systems.
- Delivering a complete front end application from scratch or extending existing application
- Proficiency in React JS and React Native is a must ensure high performance on Mobile and desktop. Writing high quality JavaScript, HTML and CSS.
- Work experience in creating progressive and responsive web applications ensures the best possible performance, quality and responsiveness of the web application.
- Identify and correct bottlenecks and fix bugs.
- Familiarity with RESTful APIs to connect to back-end services and data parsing using XML, JSON.
- Thorough understanding of the responsibilities of the platform, database API, caching layer, proxies and other web services used in the system.
- Configuration, build and test scripts for Continuous Integration environments. Proficiency in Git.
If you posses the above mentioned required skill sets & want to grow along with our fast paced growing Organisation then please share your updated CV.
What will you do?
● Partner closely with our partners in product management to understand the customers who use it, empathize with them, and imagine creative ways we can make their insurance experience dramatically better.
● Transforming complex requirements into simple understandable and usable experiences.
● Defining user flows, information architecture, lofi and visual design.
● Validating your solutions before development.
● Identifying success metrics for design projects and keeping an active track of it.
● Creating high-quality prototypes and detailed specs to effectively communicate your designs to the development team and other stakeholders.
● Always keeping in mind our customers and their interests while designing.
● Review the performance of the release and identify learnings and build hypotheses.
● Constantly pitching in new ideas to improve product experiences/adoption/other key areas.
What are we looking for?
● 3+ years proven track record of shipping high-quality mobile and web experiences.
● Strong portfolio showcasing your previous works.
● Strong proficiency in designing for Android, iOS, and Web platforms.
● Ability to see a holistic picture of the product and take necessary design decisions.
● Ability to understand the development aspects of designs.
● Ability to take ownership of design projects.
● Customer obsession and strong UX focus.
● Clean visual design sense.
● Ability to contribute to design systems.
● You’ll love to be challenged and constantly learn new things.
● You’ll love working with teams and believe strong teams make a stronger impact.
CTC up to 30 Lacs
JavaRole / Purpose - Lead Developer - API and Microservices
Must have a strong hands-on development track record building integration utilizing a variety of integration products, tools, protocols, technologies, and patterns.
- Must have an in-depth understanding of SOA/EAI/ESB concepts, SOA Governance, Event-Driven Architecture, message-based architectures, file sharing, and exchange platforms, data virtualization and caching strategies, J2EE design patterns, frameworks
- Should possess experience with at least one of middleware technologies (Application Servers, BPMS, BRMS, ESB & Message Brokers), Programming languages (e.g. Java/J2EE, JavaScript, COBOL, C), Operating Systems (e.g. Windows, Linux, MVS), and Databases (DB2, MySQL, No SQL Databases like MongoDB, Cassandra, Hadoop, etc.)
- Must have experience implementing API Service architectures (SOAP, REST) using any of the market-leading API Management tools such as Apigee and frameworks such as Spring Boot for Microservices
- Should have Advanced skills in implementing API Service architectures (SOAP, REST) using any of the market-leading API Management tools such as Apigee or similar frameworks such as Spring Boot for Microservices
- Appetite to manage large-scale projects and multiple tracks
- Experience and knowhow of the e-commerce domain and retail experience are preferred
- Good communication & people managerial skills
As a Sales Executive, you will pursue new opportunities in the small and medium-sized business sector through qualifying leads, building relationships, and scheduling appointments with prospects.
Responsibilities:
- Onboarding new customers (micro, small and medium businesses) with vehicle operations.
- Educating clients on the benefits and value addition of GPS technology for improving fleet operations.
Laravel
MongoDB
Amazon Web Services (AWS)Technical Manager: We are looking for someone who can report to the firm’s chief technology officer, the technical lead is a full-time position that participates in all phases of the project lifecycle and consistently delivers business value to our clients. The technical lead is a leader within our client-facing project delivery team structure and sets best practices, approach and direction for the technical aspects of our practice. The ideal candidate has a passion for the web and higher education and is a champion for Laravel or similar frameworks. And guide our passionate Software Developers to design, develop robust products. Candidates should be open minded to new solutions, forward thinking and strong ability to adapt. As a Lead, you’ll work closely with our engineers to ensure system consistency and improve user experience. Ultimately, you should be able to lead, develop and maintain functional and stable web applications to meet our company’s needs along with managing the team.
Primary Responsibilities
In partnership with the strategist and CTO, co-leads the strategy phase of client projects including discovery, evaluating and recommending technical solution alternatives and howe our product can be the solution to the various potential issues. Devising a cohesive strategy/approach for the project, and writing a top-quality strategy report.
-
Participates in pre-project business development activities including writing proposals, developing price estimates, and delivering pitches for new work.
-
Participates in pre-project business development activities including writing proposals, developing price estimates, and delivering pitches for new work.
-
In partnership with the other technical lead, sets direction for all aspects of our technology practice.
-
Leads and advances our growing Laravel practice.
-
Responsible for various aspects of ongoing support and maintenance engagements. Partners with project managers and web developers to successfully manage these client relationships
- Understand requirements & plan the architecture for frontend/backend of the application
- Plan out the contracts & APIs for communication of various components.
- Work on integration of any external libraries and tools as per required by the application needs
- Working with modern tools to make the application more robust, efficient and scalable
Client Management Responsibilities:
-
Understands each client’s organizational goals and objectives.
-
Develops lasting relationships with client personnel that foster client ties.
-
Seeks opportunities to increase customer satisfaction and deepen client relationships.
-
Delivers training to client personnel of various skill levels and technical capabilities.
Communication Responsibilities:
- Delivers engaging, informative and well-organized presentations.
- Resolves and/or escalates issues in a timely fashion.
- Understands how to tactfully communicate difficult/sensitive information.
- Demonstrates strong interpersonal skills.
- Elicits cooperation from a variety of sources including Froogal management and client
team members.
- Comfortable using teleconferencing and web-based technology to communicate with our
Clients.
Other Responsibilities:
- Helps team members progress toward their professional development goals.
- Defines and disseminates technology best practices.
- Ensures we are proactive in our pursuit of new solutions and innovation within our
technology practice.
- Advances the firm’s thought leadership, specifically around web technologies, via the
mStoner blog, social media outlets and industry events and conferences.
Requirements
- Strong understanding of Object-Oriented Programming & its implementation in any programming language. Knowledge of PHP and/or JS is a plus
- Proven industry experience as a Tech Lead
- Experience/Knowledge of microservice architecture
- Fluency and Hands-on Experience on Laravel 5+ PHP Framework (specially working with Eloquent Models/Collections) and/or React/Redux is a plus.
- Excellent knowledge of relational databases such as MySQL and/or NoSql based DB such as MongoDB.
- Experience in JavaScript. Vue JS/React JS is a plus
Javascript
PHP
Python
Ruby on Rails (ROR)
Adobe IllustratorSelected Candidates will be work on the following activities.
1. Promotion Ads
2. Icon Design
3. Product Design
4. Social Media Ads
5. Web Banners
6. Templates design
7. Packaging Design
8. Photography
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon