AI System QA Engineer – Large Language Models (Evaluation Testing)
at MeltPlan
MeltPlan is building the “planning engine” for the $14 Tn construction industry, an AI system designed specifically to optimize decisions before construction begins. While design software optimizes use and aesthetics and construction software optimizes execution and control, MeltPlan is building the missing layer - software that optimizes decisions and tradeoffs upstream, before scope is locked, procurement begins, and change orders become inevitable. MeltPlan’s long-term goal is to help teams make construction “boring” by making planning more intense: surfacing constraints and tradeoffs early, aligning stakeholders before plans are frozen, and reducing the need for late-stage redlines, rework, and change orders.
MeltPlan is founded by operators who have built at scale. Kanav previously co-founded Innovaccer, a $3Bn healthtech company focused on making US healthcare more affordable and accessible. He’s now applying that systems-level thinking to construction.He’s joined by Tanmaya Kala, former Project Executive at DPR Construction, who led large commercial, healthcare, and life sciences projects. We combine deep tech scale with real construction execution.
What This Role Really is :
We are seeking a detail-oriented and technically strong AI QA Engineer to ensure the quality, reliability, and performance of Large Language Model (LLM)-based systems. In this role, you will be responsible for designing and executing test strategies, validating model outputs, and building evaluation frameworks to enhance the accuracy, safety, and overall performance of AI-driven applications.We would particularly value candidates who have hands-on experience in developing evaluation frameworks (evals) for AI systems, along with strong expertise in comprehensive system testing and quality assurance practices.You are responsible for making MeltPlan work in the real world.
What You'll Do:
- Design, develop, and execute evaluation frameworks (Evals) for Large Language Models (LLMs) and AI systems.
- Perform end-to-end system testing, regression testing, and performance testing for AI-driven applications.
- Validate model outputs for accuracy, consistency, safety, hallucination detection, and edge cases.
- Build automated test pipelines and quality benchmarks for AI systems.
- Collaborate closely with AI/ML engineers, product teams, and platform engineers to improve system reliability.
- Analyze failures, identify root causes, and provide actionable feedback to improve model behavior.
- Develop datasets, prompts, and testing scenarios to measure model performance across multiple use cases.
- Monitor production performance and continuously improve evaluation metrics and testing standards.
- Ensure compliance with responsible AI and quality assurance best practices.
What We're looking for:
- Bachelor’s degree in Computer Science, Engineering, or related field
- 5–7 years of experience in QA/testing, preferably in AI/ML or data-driven systems
- Strong experience in AI/LLM evaluation frameworks and system testing.
- Hands-on experience with automated testing methodologies and QA processes.
- Familiarity with prompt engineering, AI benchmarking, and model validation techniques.
- Experience working with Python and testing frameworks.
- Understanding of LLM behaviors, hallucinations, prompt injection risks, and AI safety concepts.
- Exposure to tools/frameworks such as OpenAI Evals, LangSmith, DeepEval, Promptfoo, or similar platforms is preferred.
- Strong analytical and debugging skills with attention to detail.
- Excellent collaboration and communication skills.
- Familiarity with Large Language Models and Generative AI concepts
- Experience with API testing tools (e.g., Postman) and automation frameworks
- Understanding of NLP concepts such as tokenization, embeddings, and text generation
- Strong analytical and problem-solving skills
- Experience testing AI/ML models or data pipelines
- Experience with prompt engineering and prompt testing
- Familiarity with cloud platforms (AWS, GCP, or Azure)
- Exposure to AI safety, bias detection, and model governance
Bonus if you have:
- Have worked in construction or on project sites
- Have startup experience
- Experience working with Generative AI or conversational AI products.
- Knowledge of CI/CD pipelines and automation workflows.
- Prior experience in performance testing and monitoring distributed systems.
- Understanding of AI product lifecycle and production deployment environments.
We’re not looking for someone who waits for clean requirements.We’re looking for someone who thrives in the mess and turns it into systems.
Why meltplan
- Massive industry, real-world impact
- High ownership from day one
- Small team, zero bureaucracy
- Competitive comp + meaningful equity
Shubham Vishwakarma
Full Stack Developer - Averlon
About MeltPlan
About
Company social profiles
Similar jobs
Java
Spring BootJob Title : Senior Software Engineer
Location : Gurugram (Full-time)
Job Description :
- We are looking for a skilled and experienced Java Engineer to join our dynamic team.
- The ideal candidate will have 4 to 10 Years of hands-on experience in software development, with a proven experience as Java Developer with a strong focus on Spring and Spring Boot, Relational Databases and AWS technologies.
- Strong understanding of monolithic & microservices architecture. You will play a crucial role in designing, developing, and maintaining our applications.
- Ensuring their performance, quality and responsiveness.
Key Responsibilities :
● Design, develop, and maintain scalable applications using Java, Spring and Spring Boot.
● Develop and manage relational databases. Should be able to write complex SQL queries.
● Ensure the best possible performance, quality, and responsiveness of the applications.
● Have strong debugging skills to identify bottlenecks and bugs, and devise solutions to mitigate and address these issues.
● Deploy, manage, and scale applications on AWS.
● Conduct code reviews and integration testing to ensure software quality and reliability.
● Collaborate with front-end developers to integrate user-facing elements with server-side logic.
● Collaborate with cross-functional teams to define, design, and ship new features.
● Stay updated with emerging technologies and industry trends.
Qualifications :
● Bachelor’s degree in Computer Science, Information Technology, or a related field.
● 4 to 10 Years of experience in software development.
● Proficient in Java 8+, Spring and Spring Boot.
● Experience with PostgreSQL, MySQL or other relational databases.
● Experience with microservices architecture.
● Understanding of Rest API design and development.
● Hands on experience with Unit Testing frameworks such as Junit, Mockito.
● Experience with version control systems such as Git.
● Solid understanding of object-oriented programming.
● Strong problem-solving skills and attention to detail.
● Excellent communication and teamwork skills.
Preferred Qualifications :
● Strong knowledge of AWS services and best practices.
● Knowledge of containerization technologies like Docker and Kubernetes.
● Familiarity with CI/CD pipelines and DevOps practices.
We are looking for a performance-driven Ecommerce Marketplace Manager / Amazon Ads Expert to lead and optimize marketplace advertising and growth strategies. The ideal candidate will have hands-on experience managing Amazon Ads and other marketplace platforms, with a strong focus on ROI, data analysis, and revenue growth.
Key Responsibilities
- Plan, execute, and optimize advertising campaigns across Amazon Ads and Flipkart Ads
- Develop data-driven strategies to improve ROAS, conversions, and overall revenue
- Manage and scale paid campaigns including search, display, and retargeting ads
- Monitor, analyze, and report campaign performance; derive actionable insights
- Coordinate with Category Managers and Key Account Managers (KAMs) for marketplace deals and growth strategies
- Conduct competitor analysis and identify opportunities for new product trends, pricing, and optimization
- Collaborate with creative teams to enhance ad creatives, product listings, and landing pages
- Track market trends and competitor activities to stay ahead in performance marketing
- Manage and allocate advertising budgets effectively to maximize ROI
Required Skills & Qualifications
- Proven experience in managing Amazon Ads (mandatory)
- Strong analytical skills with hands-on experience in performance marketing tools
- Ability to manage multiple campaigns and achieve aggressive growth targets
- Data-driven mindset with a focus on experimentation and continuous improvement
- Excellent communication and teamwork skills
About Trusttech:
We, at Trusttech Solutions, comprise a team of experts who are passionate about what they do. Developing websites and mobile apps, we aim at turning each of our clients business ideas into a reality that has a strong hold in the marketplace with designs resistant to time and competition. Serving a number of clients,mostly international ones, we are well-versed in catering to the needs of customers from different time-zones. Pledged to provide excellent customer service, we strive hard to make on time project deliveries while ensuring their efficiency.
At Trusttech, we focus on delivering excellence by fostering a culture of innovation, collaboration, and growth. Our team of highly skilled professionals works on strategic projects that create real impact, making Trusttech the preferred offshore partner for businesses worldwide.
Job Title: Motion Graphics Designer (Full-Time, In-House)
We are looking for a creative and performance-driven Motion Graphics Designer to join our marketing team. This is a full-time, in-house role focused on producing engaging visual content that drives lead generation, brand visibility, and campaign performance.
The ideal candidate should be passionate about short-form video content, digital advertising creatives, and market-driven visual storytelling. Your work will play a key role in supporting our marketing campaigns across multiple digital platforms.
Key Responsibilities
- Create daily short-form video content for digital platforms including:
- Instagram Reels
- TikTok
- YouTube Shorts
- Design and produce high-performing advertising creatives for platforms such as:
- Meta Ads (Facebook & Instagram)
- Google Ads
- Other digital advertising channels
- Develop high-converting motion graphics and ad creatives related to financial markets such as forex, gold, oil, and cryptocurrency trading.
- Produce branding animations, promotional videos, and campaign visuals to support marketing initiatives.
- Create trend-based content by closely following market updates and social media trends to keep creatives fresh and engaging.
- Adapt and optimize video creatives into multiple formats suitable for different platforms, including:
- Vertical (9:16)
- Square (1:1)
- Landscape (16:9)
- Collaborate closely with the digital marketing team on:
- Campaign concepts and creative strategy
- A/B testing of ad creatives
- Creative performance analysis and optimization
Requirements
- Proven experience in motion graphics and video editing.,
- Strong knowledge of Adobe After Effects, Premiere Pro, and other relevant design tools.
- Experience creating short-form social media videos and advertising creatives.
- Understanding of digital marketing and performance-based creative production.
- Ability to work in a fast-paced environment and meet daily content requirements.
- Strong attention to detail and creativity.
Job Type
Full-time, In-House
- Experience: 3 + years
- Education:
- Bachelor's (Preferred)
- Work Location: Kakkanad, Kochi
About Trusttech's Vision:
Trusttech is committed to empowering global businesses with innovative solutions, unmatched expertise, and a highly skilled workforce. We deliver results that enable our clients to grow, scale, and lead their industries.
About Us
Toronto Boutique Apartments (TBA) is a high-growth property management and corporate housing company managing 200+ furnished rental units. We operate at the intersection of hospitality, real estate, and tech, with a growing focus on automation and AI.
We're seeking a Performance Marketing Manager to own and optimize our live paid ad campaigns, improve SEO across landing pages, and explore the strategic use of generative AI in marketing execution.
Role Overview
This is not a strategy-only role—we’re looking for someone who can manage our existing campaigns and continuously improve performance through data, experimentation, and automation. You’ll work alongside our leadership team to scale tenant acquisition, owner outreach, and corporate partnerships.
Responsibilities
Paid Ads (Google, Meta)
- Manage and optimize active Google Ads and Meta campaigns across multiple segments
- Monitor spend, bidding strategy, and audience targeting
- Run multivariate tests (copy, creative, landing pages)
- Improve cost-per-lead and drive qualified conversions
SEO
- Improve on-page SEO and internal linking across TBA websites
- Identify technical SEO issues and coordinate fixes
- Work with content writers to build targeted, rankable pages
- Track performance using SEMrush, Google Analytics, and GSC
AI-Enhanced Marketing Execution
- Use generative AI tools (e.g., ChatGPT, Claude) to:
- Generate and iterate on ad copy, landing page variations, and SEO meta content
- Speed up content production and testing
- Support dynamic content personalization and retargeting workflows
Reporting and Analytics
- Build concise weekly reports on ad performance, SEO health, and test results
- Propose and implement data-backed improvements
Qualifications
- 3+ years of experience managing paid ad accounts and SEO projects
- Strong command of Google Ads, Meta Ads, SEMrush, and GSC
- Proven ability to optimize CPL, ROAS, and organic traffic
- Comfortable using generative AI tools in marketing workflows
- Bonus: Experience with real estate, property tech, or hospitality verticals
Why Join Us
- Clear KPIs and real impact from day one
- Freedom to experiment, iterate, and scale what works
- Small, fast-moving team with ownership and visibility
- Opportunities to explore AI-based marketing automation in a live business setting
.BDA and BDE
ROLES
We're seeking a qualified sales associate to sell annual car and bike Scotty subscription products that our customers have grown to rely on. The sales associate will utilize their skills to generate high-quality leads, build strong relationships with customers, and close deals. The ideal candidate has skills and demonstrates the ability to showcase our offerings compellingly.
Job Location: Work from home
Selection process: HR Round followed by group discussion and sales manager round.
Qualification: Any Graduate/Post Graduate
Salary offered: As per industry standards
Working days: 6 days Sunday is off
Shifts:10:00am -7:00pm
Mandatory language: English
Laptop/wi-fi: candidates are to use their own laptops,
Additional Compensation: If applicable, this will be decided on the basis of your designation.
Coordinate office activities and operations to secure efficiency and compliance to company policies
Supervise administrative staff and divide responsibilities to ensure performance
Manage agendas / travel arrangements / appointments etc. for the upper management
Manage phone calls and correspondence (e-mail, letters, packages etc.)
Support budgeting and bookkeeping procedures
Create and update records and databases with personnel, financial and other data
Track stocks of office supplies and place orders when necessary
Submit timely reports and prepare presentations / proposals as assigned
Assist colleagues whenever necessary
Requirements and skills
Proven experience as an office administrator, office assistant or relevant role
Outstanding communication and interpersonal abilities
Excellent organizational and leadership skills
Familiarity with office management procedures and basic accounting principles
Excellent knowledge of MS Office and office
Python
Django
Flask
Amazon Web Services (AWS)We are looking for an experienced Python developer who can help create dynamic software applications for our clients with their skill set. In this role, you will be responsible for gathering requirements from clients and accordingly write and test scalable code, and develop back-end components.
Technologies worked on:
Python - Django/Flask/FastAPI, Pytest/Unittest, AWS services.
Database - Postgresql/Mysql or NoSQL databases.
Requirement Description:
• Experience in designing,implementation & testing of Python Applications.
• Must have knowledge of at least one of python web framework Django/Flask/FastAPI and any of unit test frameworks(Pytest/Unittest).
• Should have a solid understanding of object-oriented programming (OOP).
• Well-experienced to perform Unit Testing and Integration Testing
• Have good experience in Agile based development approach.
• Expertise in developing enterprise-level web applications and REST/GRAPHQL APIs using
MicroServices, with demonstrable production-scale experience.
• Demonstrate strong design and programming skills ,writing optimized code.
• Working knowledge with SQL(MySQL,Postgresql etc) is mandatory and knowledge on NoSQL databases can be an addon.
• Understand Architecture Requirements and ensure effective design, development, validation, and support activities.
• Understanding of core AWS services, uses and basic AWS architecture best practices.
• Proficiency in developing, deploying, and debugging cloud-based applications using AWS.
• Ability to use the AWS service APIs, AWS CLI, and SDKs to write applications
• Ability to identify key features of AWS services while designing a solution.
• Identify bottlenecks and bugs, and recommend solutions by comparing the advantages and disadvantages of custom development
• Should contribute to team meetings, troubleshooting development and production problems across multiple environments and operating platforms
• Execute strong collaboration and communication skills within distributed project teams.
• Responsible for quality and timely deliverables for each given task.
• Knowledge on frontend technologies could be addon.
Data Science
Deep Learning
R ProgrammingPythonData Scientist
Requirements
● B.Tech/Masters in Mathematics, Statistics, Computer Science or another
quantitative field
● 2-3+ years of work experience in ML domain ( 2-5 years experience )
● Hands-on coding experience in Python
● Experience in machine learning techniques such as Regression, Classification,
Predictive modeling, Clustering, Deep Learning stack, NLP
● Working knowledge of Tensorflow/PyTorch
Optional Add-ons-
● Experience with distributed computing frameworks: Map/Reduce, Hadoop, Spark
etc.
● Experience with databases: MongoDB
PHP
CodeIgniter
JavascriptAmazon Web Services (AWS)· Knowledge on Angular and Node.Js(Stack) would be beneficial
· Demonstrated proficiency with LAMP architecture (Linux, Apache, MySQL, PHP)
· Demonstrated proficiency with scripting – Python/Perl.
· Must be well versed in MVC architecture.
· Hands on experience with MySQL (Indexing, Scaling)
· Good understanding of RESTful API development
· Good in Bootstrap and Responsive web development.
· Expert level Object Oriented Design, and MVC (Multi-Tier Architecture/ Framework)
· Good understanding of R Programming, Solr, Sphinx a plus
· Good understanding of NoSQL, MongoDB a plus
Follow CutshortShubham Vishwakarma
Full Stack Developer - Averlon