We built an automated QA/eval engine for agent prompts. Help us test it out!

Reddit r/AI_Agents Tools

Summary

Built an automated QA/eval engine for agent prompts called Baseline that treats prompts like software for regression testing, allowing non-coders to define rubrics and automatically optimize prompts. Currently in limited beta with a 30-day free trial.

Anyone else trapped in the "guessing loop"? You change a word, read a couple of answers, and hope it holds. But there’s no number, no proof, and no way to know if you just quietly degraded the output for a dozen other tasks. I got tired of this, so my team built Baseline It basically treats your AI agent's prompts like software that needs real regression testing, but without requiring any code or engineering hours. Here’s the breakdown of how it works in the attached video: Define what "good" means. You write a plain-language rubric weighted by what matters most to your agent (like Accuracy, Tone, and Resolution). The Optimization Engine: Baseline tunes the prompt for you. It automatically runs your prompt across your eval dataset, scores it, rewrites the instructions, and re-scores until it finds the winning version. Quality Backed by Proof: The engine can pull an output score from a 0.62 baseline all the way up to a 0.94 optimized state during your coffee break. We’re running a limited beta right now to get feedback from people who are actively building and breaking AI agents. Check out the link the comments. If you want to test it out, DM me and I’ll send you an access code that unlocks a 30-day free trial. We need to get some other eyes and testers on this. Happy to take any feedback!
Original Article

Similar Articles

How are teams handling prompt QA at scale?

Reddit r/AI_Agents

A practitioner at a company handling ~40k conversations/month describes the bottleneck of manual prompt QA and asks how teams are using automated systems to detect regressions and user frustration in production.

An Empirical Study of Automating Agent Evaluation

arXiv cs.CL

This paper introduces EvalAgent, a system that automates the evaluation of AI agents by encoding domain-specific expertise, addressing the limitations of standard coding assistants in this task. It also presents AgentEvalBench, a benchmark for testing evaluation pipelines, and demonstrates significant improvements in evaluation reliability.