Production agent evals should test incident replay not just task success

Most agent evals I see still measure whether the agent completed the happy path task. That is useful but for production I think the more important eval is can an operator reconstruct what happened when the run went wrong For every failed or partially completed run I would want to answer - What was the durable state before the failure - Which tool call changed the outside world - What exact payload or diff was sent - Was there an idempotency key or external receipt - Which evidence did the model use - Could the next operator safely resume retry compensate or abandon If the answer is no the agent may have passed the demo but failed the production eval. This changes how I think about observability. Traces are not enough if they are just a wall of spans. The trace has to become a recovery artifact state decision external receipt policy result and owner. Curious how teams here are evaluating production agents. Are you measuring task success only or do you have failure replay and resume evals too