Google's agent team says test discrete behaviors, not just benchmark scores. How do you catch an agent getting worse after a prompt, tool or model change?

Google's developer blog (Sept 9, 2026) has a post on evaluating coding-agent harnesses. Its argument: end-to-end benchmarks like Terminal-Bench and DeepSWE can move a composite score by a few points without telling you why. Behavioral evals check discrete, observable actions instead, for example whether the agent asks a clarifying question on an underspecified prompt, or runs the local validator before it declares a build-file change done. They say the main job of the suite is guarding against regressions: confirming that a prompt tweak, a tool schema change or a model upgrade didn't make the agent worse overall. Their suggested start is small: pick one recent failure, write a check for it, and run batches because single runs are noisy. OpenAI's Feb 2026 harness engineering post attacks the same problem from the build side. When the agent struggled, they treated it as a missing capability and fed the fix back into the repo as docs, guardrails or tooling. Both are about coding agents with specific tooling, so I don't know how far this transfers to other kinds of agents. Questions for people running agents in production: What was the first failure you turned into a repeatable check? Do you assert on tool calls and steps, or judge outcomes (LLM-as-judge)? How do you deal with run-to-run noise? Sources: Google Developers Blog, "The Anatomy of Harness Engineering" (Sept 9, 2026); OpenAI, "Harness engineering: leveraging Codex in an agent-first world" (Feb 11, 2026).