Job Title: Principal Automation Engineer (AI)
Deltek is seeking a Principal Automation Engineer with deep expertise in AI-native test automation to help shape the quality engineering foundation Deltek’s next-generation, AI-first ERP platform for project-based businesses. This is not a role for someone who automates feature regression. It is a role for someone who can harness AI tools to build intelligent automation frameworks that reason, adapt, and self-heal.
You will be the automation architect behind Deltek’s in-house AI-native test automation platform combining Playwright with LLM-powered agents (Planner, Generator, Healer). You will extend, evolve, and industrialize this framework, integrating AI tools at every layer: test generation, self-healing selectors, LLM-as-a-Judge evaluation, and CI/CD-gated quality pipelines.
If you are fluent in Playwright, agentic AI workflows, and modern test engineering — and want to build something genuinely new rather than maintain legacy frameworks — we invite you to join our team. ERP domain knowledge is a strong plus and will accelerate your impact.
Responsibilities:
Architect and evolve the AI-native automation framework — extending Playwright-based agents with LLM-powered planning, test generation, and self-healing capabilities.
Use AI tools extensively (Claude, GitHub Copilot, LLM APIs) to design, generate, and augment automation suites — reducing human authoring effort while increasing scenario coverage.
Build and maintain Playwright agent pipelines for end-to-end workflow automation across Deltek’s Projects, Workforce Management, and Financials modules.
Integrate LLM-as-a-Judge (LLMaaJ) evaluation into the test pipeline to automatically score AI-generated outputs, detect hallucinations, and validate response quality against golden datasets.
Design and implement AI safety and correctness test cases: hallucination detection, bias testing, output guardrail validation, and behavioral consistency across edge cases.
Own the CI/CD automation pipeline (GitHub Actions / Azure DevOps) for AI-enabled releases — including regression gates, model-response validation, and automated quality dashboards in ReportPortal and Grafana.
Validate AI/ML outputs including prediction accuracy, recommendation relevance, natural-language responses, and inference API payloads.
Build and maintain golden datasets for AI drift detection, regression baselines, and LLM evaluation benchmarks.
Collaborate with Product Managers, AI/ML Engineers, and QE leads to define AI feature release quality gates and automation coverage targets.
Mentor QE team members on AI-assisted automation patterns, agentic testing concepts, and framework best practices.
Contribute to test strategy for data migration validation of schema fidelity and record correctness.
Qualifications:
BS/MS degree in Computer Science, Software Engineering, or a related field.
Relevant certifications in software quality, AI/ML, or cloud engineering are advantageous.
Experience:
8+ years of experience in test automation engineering, with at least 3+ years working with AI/LLM-based systems or agentic automation frameworks.
Proven hands-on experience with Playwright — including Playwright agents, fixtures, and API testing integration.
Demonstrated experience using AI tools (Claude API, OpenAI, GitHub Copilot, or equivalent) to accelerate test authoring, framework design, or output evaluation.
Track record of designing and implementing AI-based automation solutions — not just using automation tools, but building the frameworks others use.
Experience integrating automation into CI/CD pipelines (GitHub Actions, Jenkins, or Azure DevOps).
Experience with performance, scalability, or data-drift testing of AI features in production or pre-production ERP contexts.
ERP domain knowledge (Project Accounting, Financials, Payroll, Time & Expense) is a strong plus and will significantly accelerate onboarding and impact.
Good-to-Have Skills:
Familiarity with Ajera, Costpoint, Vantagepoint, or comparable project-based ERP systems.
Understanding of LLM fine-tuning, RAG pipelines, vector databases, and embeddings from a QA/validation perspective.
Experience building or working with self-healing automation frameworks or AIOps tooling.
Exposure to security testing for AI systems — prompt injection, output sanitization, guardrail bypass testing.
Familiarity with data privacy and compliance frameworks in AI-enabled enterprise software.
Technical Qualifications:
Deep, hands-on proficiency with Playwright — including agentic patterns, multi-step workflow automation, and integration with LLM backends.
Proficiency in TypeScript and/or Python for building automation frameworks, AI evaluation utilities, prompt-testing harnesses, and data-driven test pipelines.
Strong understanding of LLM/ML concepts from a QA perspective: prompt engineering, hallucination detection, output scoring, explainability validation, behavioral consistency testing.
Experience with REST and GraphQL API testing, including automated evaluation of LLM inference API payloads and AI-generated JSON responses.
Familiarity with ReportPortal, Grafana, or equivalent for test execution dashboards and quality metric visualization.
Strong SQL skills for data validation, training dataset verification, and ERP data pipeline testing.
Working knowledge of GitHub Actions and Azure DevOps (ADO/TFS) for CI/CD pipeline integration and issue tracking.
Good understanding of Agile/Scrum practices and AI model release cycles — shadow mode, A/B comparison, phased rollout validation.
Soft Skills:
Framework-builder mindset: thinks in systems, not scripts — builds what others use rather than executing what others built.
Strong communication skills: able to explain AI validation concepts clearly to engineers, product managers, and QE team members.
High ownership and self-direction: identifies automation gaps proactively and drives coverage improvements without waiting to be asked.
Collaborative and generous with knowledge: invests in mentoring team members and raising the team’s automation maturity.
Continuous learner: actively tracks the evolving AI tooling ecosystem and brings new techniques into the framework.
Able to manage multiple priorities in a fast-paced, distributed team environment.
Join Deltek and be at the forefront of how modern ERP quality engineering is done. You will help build the AI driven current automation framework into an industry-leading AI-native automation platform — combining Playwright agents, LLM-powered test generation, self-healing infrastructure, and intelligent quality gates.
Follow Cutshort