llm-oversight

This study uses signal detection theory to analyze how procedural traces affect LLM overseers, finding that detailed traces shift decision criteria towards rejection and increase false alarms.

This paper from Carnegie Mellon researchers shows that giving an LLM judge more compute doesn't fix oversight failures when it must check many requirements in one call. It proposes sharding—dividing requirements into smaller groups handled by separate calls—which improves accuracy, resists presentation-based adversarial attacks, and can make a weaker sharded judge match a more capable holistic judge.