the success rate number on computer use agents is the wrong one to watch

I spent way too long optimizing the single-run task success rate on my desktop agent, basically the number you'd put in a demo. turns out that's the vanity metric. the thing that actually predicts whether i still reach for it next week is recovery rate: what happens on the step right after it clicks the wrong button. every multi-step agent takes a wrong action eventually. the usable ones notice the state didn't change the way they expected and back out. the demo-grade ones just barrel ahead. and since i drive native mac apps through the accessibility tree instead of screenshots, that failure is silent, the tree goes stale and the next action quietly lands on nothing. So now i track two numbers, first-try success and recover-from-a-wrong-action success, and the second is the only one that moved whether it actually finishes my work. no benchmark i've seen even reports the recovery case. they all score the clean first attempt, which is the one part of real usage that almost never happens. written with ai