A model can give the right answer while the agent still fails the task

A lot of agent evaluations seem to score only the final answer. That makes sense for chat, but it feels incomplete once the agent is expected to act on external systems. Consider a paid API workflow. The model can make the correct decision and still fail because: the authorization expired before execution the payment completed but the request timed out the service accepted the request but the response was lost a retry created a duplicate charge the final result could not be retrieved later I have started thinking about two separate scoreboards: Decision quality: Was the answer or choice correct? Execution integrity: Was the intended external action completed exactly once and later verified? For execution, I track a small state machine: proposed → authorized → executed → acknowledged → verified. An agent should not claim completion just because it reached the first or third state. How are people benchmarking this in practice? If an agent reaches the correct answer but fails the external action, do you count that as a failure, partial success, or a separate metric entirely?