We built an automated QA/eval engine for agent prompts. Help us test it out!

Reddit r/AI_Agents Tools

Summary

Built an automated QA/eval engine for agent prompts called Baseline that treats prompts like software for regression testing, allowing non-coders to define rubrics and automatically optimize prompts. Currently in limited beta with a 30-day free trial.

Anyone else trapped in the "guessing loop"? You change a word, read a couple of answers, and hope it holds. But there’s no number, no proof, and no way to know if you just quietly degraded the output for a dozen other tasks. I got tired of this, so my team built Baseline It basically treats your AI agent's prompts like software that needs real regression testing, but without requiring any code or engineering hours. Here’s the breakdown of how it works in the attached video: Define what "good" means. You write a plain-language rubric weighted by what matters most to your agent (like Accuracy, Tone, and Resolution). The Optimization Engine: Baseline tunes the prompt for you. It automatically runs your prompt across your eval dataset, scores it, rewrites the instructions, and re-scores until it finds the winning version. Quality Backed by Proof: The engine can pull an output score from a 0.62 baseline all the way up to a 0.94 optimized state during your coffee break. We’re running a limited beta right now to get feedback from people who are actively building and breaking AI agents. Check out the link the comments. If you want to test it out, DM me and I’ll send you an access code that unlocks a 30-day free trial. We need to get some other eyes and testers on this. Happy to take any feedback!
Original Article

Similar Articles

Promptic

Product Hunt

Promptic is a platform that optimizes GenAI applications for quality and cost by benchmarking models, tuning prompts and agents, and integrating with CI systems and dashboards.

QAgent

Product Hunt

QAgent automates quality assurance for AI agents by scoring correctness, detecting hallucinations against ground truth, verifying policy adherence, and benchmarking RAG to prevent bad responses from reaching customers.

How are teams handling prompt QA at scale?

Reddit r/AI_Agents

A practitioner at a company handling ~40k conversations/month describes the bottleneck of manual prompt QA and asks how teams are using automated systems to detect regressions and user frustration in production.