@no_stp_on_snek: FIELD NOTES: AI and QA pointed an agent at my test suite with one question. not "do the tests pass." can this test dete…
Summary
A field note argues that a green test suite doesn't prove tests are meaningful; AI agents should be asked to make tests fail to verify they actually catch bugs.
View Cached Full Text
Cached at: 08/09/26, 03:17 AM
FIELD NOTES: AI and QA
pointed an agent at my test suite with one question. not “do the tests pass.” can this test detect the bug it exists to catch?
17 test files across two repos couldn’t. assertions written so loose nothing ever trips them. green for months, and everyone downstream trusted them.
a green check means a test ran and didn’t complain. says nothing about a test that has no way to complain.
so now i don’t ask for coverage or pass rate first. i ask to see the test fail. can’t show me? it’s not a test. it’s a decoration that lights up green.
Similar Articles
What your agent's green test suite actually proves
This article argues that standard test suites with fixed inputs and expected outputs are insufficient for AI agents due to infinite input spaces and non-deterministic behavior, advocating for property-based testing instead.
Your AI Agent is one bad prompt away from ruining your brand (And why traditional QA is useless)
The article argues that traditional chatbot QA is broken because it only tests happy paths, and proposes using an AI-powered user simulator that attacks the bot with diverse personas and edge cases to find vulnerabilities before deployment.
A right answer from your agent doesn't mean it did the right thing
The article discusses the pitfalls of evaluating AI agents solely based on their final answers, emphasizing the importance of inspecting intermediate steps, tool calls, and reasoning to catch confidently wrong outputs. It suggests using automated scoring and trace replays to measure and improve agent behavior.
It's impossible to test your own agent. I tried and failed.
A developer's personal account of the difficulty in objectively evaluating their own AI agent's performance, highlighting the pitfalls of self-testing and the value of unexpected, real-world benchmarks.
Agent failures should become evals, not just traces
Advocates for treating agent failures as evaluation benchmarks rather than just trace logs, emphasizing the need for systematic testing of AI agent behaviors.