replay-benchmark

This paper demonstrates that replay-based static evaluation of model switching in LLM agents is fundamentally flawed: when swapping models mid-trajectory, the environment and subsequent actions diverge dramatically from logged trajectories, invalidating most benchmark results. The authors propose branching rollouts as a more faithful evaluation method and release their harness and trajectories.