Tag
A discussion comparing LLM evaluation and observability tools (LangSmith, Weave, Phoenix, Braintrust, Galileo, Opik) for fixing prompt failures and introducing an open-source platform that integrates the full eval-to-fix loop on a single trace.
Built an automated QA/eval engine for agent prompts called Baseline that treats prompts like software for regression testing, allowing non-coders to define rubrics and automatically optimize prompts. Currently in limited beta with a 30-day free trial.
The author asks how developers are regression-testing AI agent workflows, noting common failure modes and sharing their work on adding eval support to Runme for recording tasks, scoring trajectories, and comparing against baselines.
The article discusses various techniques to make LLM-powered agents more deterministic, such as golden sets, guardrails, consensus mechanisms, regression tests, coded logic, and hyperparameter tuning, and asks for additional successful methods.
This paper introduces layer-isolated evaluation for LLM agents, decomposing a production agent into architectural layers each tested with a deterministic, no-LLM harness. It demonstrates that per-slice baseline testing localizes regressions that aggregate metrics mask, validated by controlled regression injections across multiple tenants.
replayd is an open source Python tool that captures failed AI agent runs and replays them as regression tests to prevent regressions from returning after changes.
An archive called Agent Fail Museum documents recurring AI failure patterns and provides regression test drafts for submitted failures, aiming to prevent repeat incidents.
TesterArmy is a QA tool to test websites and mobile apps across browsers and devices, catching regressions and generating tests from natural language.