order-inconsistency

This paper tests whether persisting evidence generated in one call and using it as the exclusive input for a later verdict (evidence locking) improves or harms LLM-as-judge evaluation. Across 24,000 judgments, they find that evidence locking reduces agreement with human preferences by 4-6 points and increases answer-order inconsistency compared to structured one-call judging.