Tag
Stanford researchers develop a Bayesian variance-decomposition framework using Generalizability theory to diagnose reliability in agent benchmarks, showing that adding more tasks cannot fix uncertainty caused by scaffold coverage and that pooling diverse benchmarks is a cheaper path to reliable rankings.
Researchers show agent leaderboards rank task specialization rather than capability, with agent main effects accounting for under 3% of variance across three benchmarks, and propose a Deployment Decision Reliability framework.
This paper applies Generalizability Theory to agent benchmarks, showing leaderboards rank specialization rather than capability, and proposes a framework (DDR) for sizing reliable deployment evaluations.