Tag
This paper unifies selection bias and order sensitivity in multiple-choice LLM evaluation by showing the pooled selection-rate standard deviation factorizes into flip rate and positional skew. Across 41 checkpoints and 12 benchmarks, it shows prior correction mainly reduces positional skew while ordering aggregation mainly reduces the flip rate, recommending both be reported.
这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率（flip rate）和位置偏斜（positional skew）两个因素，并发现先验校正主要降低位置偏斜，而顺序聚合主要降低翻转率，建议同时报告这两个因素以区分缓解效果。
This paper examines how the order of conflicting evidence in multimodal large language models affects judgments, revealing cross-modal evidence noncommutativity where placing perceptual evidence later increases model reliance on it.
This paper investigates whether label-free strategies for multiple-choice benchmarks can remove option-order sensitivity in large language models, finding that neither two-stage prompting nor independent hypothesis scoring reliably improves accuracy.
The paper tests label-free strategies to reduce positional bias in multiple-choice benchmarks but finds they do not reliably improve accuracy or reduce bias. Only showing all options with an LLM matcher preserves baseline performance, while cyclic permutation sometimes helps.
Introduces CDR-Bench, a benchmark with 3,462 tasks to evaluate LLMs' ability to faithfully execute compositional, order-sensitive data refinement recipes. Experiments on 10+ LLMs reveal significant performance degradation in compositional and order-sensitive settings, highlighting a lack of procedural faithfulness.
A paper accepted at ICML 2026 introduces predictable hallucination via an information-budget abstention gate, and releases ntkMirror, a training-free open-weight implementation that reduces hallucination by abstaining when information is insufficient, achieving 0.0–0.7% hallucination at ~24% abstention.