I stopped using 100% as the success criterion for my coding-agent evals

I stopped trying to get my coding-agent eval suite to 100%. Not because the remaining failures were acceptable, but because after 19 full runs it became clear that "three perfect runs in a row" was measuring luck more than quality. Keep the Why currently has 88 eval cases. Each case runs a real coding-agent session against a fixture project, then an LLM judge grades the transcript and resulting file changes. The target for this release was three consecutive 88/88 runs. What actually happened: 98.6% of all case runs passed roughly one case failed per full run it was usually a different case each time those cases had often passed 20 or 30 times before neither the agent nor the judge can be seeded At that point, three clean runs in a row are dice, not a property of the skill. Worse, chasing isolated failures started making the skill less stable. Of the sentences I added to fix cases that had failed once, three caused a neighbouring case that had never failed before to flip. So I changed the release criterion. A release series now passes when: every case passes at least 2 of 3 runs no individual run contains more than one failed case That gives failures meaning: one isolated failure can be sampling noise the same case failing twice points to a wording problem several failures in one run point to a regression 0.17.0 produced: 87/88 88/88 87/88 86 of 88 cases passed all three runs. The other useful result came from the same process. The skill had six prose rules controlling whether the agent should ask for confirmation or write directly. They kept influencing each other in unexpected ways. I replaced them with a 6x3 decision table. The result is 21% shorter and produced no decision flip across two complete eval series. This has changed how I think about evals for probabilistic agent behavior. A perfect run can be a nice observation, but requiring repeated perfection may be a bad acceptance criterion when both the system under test and the evaluator are sampled. If someone finds a way to phrase the skill so that it reliably passes 88/88 three times without weakening the cases, I'd genuinely like to see the PR.