Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
Summary
The paper tests label-free strategies to reduce positional bias in multiple-choice benchmarks but finds they do not reliably improve accuracy or reduce bias. Only showing all options with an LLM matcher preserves baseline performance, while cyclic permutation sometimes helps.
View Cached Full Text
Cached at: 08/18/26, 11:54 PM
Paper page - Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
Source: https://huggingface.co/papers/2608.11947
Abstract
Preventing models from seeing option labels during answering does not reliably reduce positional bias or improve multiple-choice accuracy, and only showing all options with an LLM matcher preserves baseline performance.
Multiple-choice benchmarksare widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity tooption order, which makes them unreliable measures of model knowledge. In this paper, we test whether preventing a model from seeing option labels while committing to an answer removespositional influenceand, in turn, improves performance. We evaluate two different strategies for mitigating bias. The first uses ageneration-then-matchingapproach, and the second scores options in isolation, which is positionally unbiased by construction. Neither reliably improves accuracy. A complete decomposition shows that the bottleneck is withholding options, not the matching step. The only configuration that consistently matches the baseline is the one that shows the model all options paired with anLLM matcher. However, eliminatingpositional influenceentirely still does not reliably yield accuracy gains, whilecyclic permutationoften improves them. For two-stage prompting, an aggregate measure ofrecall imbalanceand a direct per-question measure oforder sensitivityboth fail to show reliable debiasing.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2608\.11947
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.11947 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.11947 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.11947 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
This paper investigates whether label-free strategies for multiple-choice benchmarks can remove option-order sensitivity in large language models, finding that neither two-stage prompting nor independent hypothesis scoring reliably improves accuracy.
Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning
Introduces a curation-free metric (Triangulated Preference Shift) to isolate and quantify lexical biases induced during preference learning in LLMs, without manual curation, across six model families.
Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels
This paper studies how post-training quantization introduces new biases in instruction-tuned LLMs, finding that 3-bit precision causes 6–21% of previously unbiased items to develop stereotypes, while standard metrics like perplexity fail to detect this degradation.
Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative
The paper analyzes length bias in likelihood-based multiple-choice evaluation, showing that standard and length-normalized accuracy introduce biases toward shorter and longer answers respectively. It proposes Bayesian accuracy, a scoring rule that uses an explicit prior over answer length to remove linear length effects, demonstrating consistently lower empirical bias across benchmarks.
Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
This paper investigates how LLMs' answers change under meaning-preserving paraphrases, finding that instance-level behavior is unstable (flip rates >23%) and that single-prompt accuracy masks substantial inconsistency, while a self-paraphrasing strategy can partially recover latent knowledge.