what do u actually check when every span is green but the agent still did the wrong thing?

curious to see how people handle this one in production not the easy version where a tool throws an exception or some node obviously fails im talking about when the whole run looks healthy llm call returned retrieval returned docs tool returned 200 state update says success verification says pass final step completes and the actual outcome is still wrong lol ive heard a bunch of different versions of this happening tool says success but the response body quietly contains an error the right field exists but it has the wrong account or tenant id retrieval returned perfectly valid docs but they were the wrong docs for that request state got written successfully but the next step read something different configured model says one thing but a fallback/provider actually served another something got verified but the check was looking at the wrong thing and thats where i feel like “status = success” stops telling u much so when everything is technically green what evidence do u reach for next? do u compare against a known good run? start tracing state backwards? check exactly what each downstream step consumed? verify the read after the write instead of trusting the write response? look at effective model/provider/cache info? or is there some piece of data u wish every trace captured because without it ur basically guessing? im especially curious about the last one whats the thing that usually isnt in the trace when u need it most? anything youve built is also welcome just link it below id love to see solutions free pls dont just promote slop thanks