Tag
This paper unifies selection bias and order sensitivity in multiple-choice LLM evaluation by showing the pooled selection-rate standard deviation factorizes into flip rate and positional skew. Across 41 checkpoints and 12 benchmarks, it shows prior correction mainly reduces positional skew while ordering aggregation mainly reduces the flip rate, recommending both be reported.
这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率（flip rate）和位置偏斜（positional skew）两个因素，并发现先验校正主要降低位置偏斜，而顺序聚合主要降低翻转率，建议同时报告这两个因素以区分缓解效果。
This paper shows that chain-of-thought instructions can distort evaluation in vision-language models when answer decoding happens before reasoning generation, leading to significant accuracy drops and selection bias.
This paper demonstrates that legal multiple-choice benchmarks are vulnerable to option-only solvability, where models can answer correctly without the question, and that filtering based on one model's performance does not improve validity for other models.