What your agent's green test suite actually proves

Something I keep running into when people start shipping agents, they write a test suite the way they would test any other code, a set of inputs with expected outputs, it goes green, and they treat the agent as covered. It isn't, and not by a little. Two reasons it breaks. The input space for normal code is something you can mostly enumerate, the branches are finite and you can hit them. An agent's input is open text, so your fifty cases are fifty points in a space that is effectively infinite, and the trouble usually lands in the part you never wrote a case for. On top of that the same input does not give you the same run, so a case that passed today is a probabilistic statement and not a guarantee. So a green suite on an agent means it worked on those exact strings, on that run. That is a much weaker claim than green means on a normal codebase, and people read it as the same claim. What has been more honest for me is testing across a distribution of inputs and checking properties that should always hold, things like it never calls the same tool twice in a row or never emits an action outside the allowlist, rather than asserting one exact output. For people shipping agents, are you testing fixed cases, or something closer to a property check?