One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Hugging Face Daily Papers Papers
multiple-choice-evaluation selection-bias order-sensitivity model-bias benchmarking mitigation-methods

Summary

这篇论文将多选题评测中的选择偏差和顺序敏感性分解为翻转率（flip rate）和位置偏斜（positional skew）两个因素，并发现先验校正主要降低位置偏斜，而顺序聚合主要降低翻转率，建议同时报告这两个因素以区分缓解效果。

Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in answers across option orderings). We connect these perspectives through the pooled selection-rate standard deviation (pooled SD), which measures how unevenly option letters are selected across questions and orderings. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two label-free factors: the flip rate and the positional skew. The flip rate is the fraction of questions whose chosen option changes across orderings; the positional skew is the standard deviation of option-letter selection rates among those questions. Across 41 checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while the positional skew tracks the estimated option-symbol prior. Prior correction for selection bias mainly reduces the positional skew. Aggregating each ordering with its reversal, a mitigation for order sensitivity, mainly reduces the flip rate. Both reduce the range of accuracy across correct-answer letters by similar amounts, but aggregation leaves fewer model-rank reversals. Applying prior correction before aggregation reduces both factors and further narrows this accuracy range. However, aggregation lowers the flip rate partly by settling questions on the same wrong option. We therefore recommend reporting both factors to distinguish mitigation effects, alongside accuracy to distinguish stability from correctness.
Original Article
View Cached Full Text

Cached at: 10/16/26, 11:03 AM

Paper page - One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

Source: https://huggingface.co/papers/2610.20993

Abstract

Selection bias and order sensitivity in multiple-choice models are decomposed into flip rate and positional skew via pooled standard deviation, showing that prior correction and aggregation target different factors and should be reported together with accuracy.

Reordering multiple-choice options can change a model’s answer, an effect studied asselection bias(a preference for certain option letters) and asorder sensitivity(changes in answers across option orderings). We connect these perspectives through thepooled selection-rate standard deviation(pooled SD), which measures how unevenly option letters are selected across questions and orderings. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two label-free factors: theflip rateand thepositional skew. Theflip rateis the fraction of questions whose chosen option changes across orderings; thepositional skewis the standard deviation of option-letter selection rates among those questions. Across 41 checkpoints and 12 benchmarks, theflip ratetracks accuracy and output uncertainty, while thepositional skewtracks the estimated option-symbol prior.Prior correctionforselection biasmainly reduces thepositional skew. Aggregating each ordering with its reversal, a mitigation fororder sensitivity, mainly reduces theflip rate. Both reduce the range of accuracy across correct-answer letters by similar amounts, butaggregationleaves fewer model-rank reversals. Applyingprior correctionbeforeaggregationreduces both factors and further narrows this accuracy range. However,aggregationlowers theflip ratepartly by settling questions on the same wrong option. We therefore recommend reporting both factors to distinguish mitigation effects, alongside accuracy to distinguish stability from correctness.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.20993

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.20993 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.20993 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.20993 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

One Bias Score, Two Factors: How Mitigations Affect Option Bias in Multiple-Choice Evaluation

arXiv cs.CL

This paper unifies selection bias and order sensitivity in multiple-choice LLM evaluation by showing the pooled selection-rate standard deviation factorizes into flip rate and positional skew. Across 41 checkpoints and 12 benchmarks, it shows prior correction mainly reduces positional skew while ordering aggregation mainly reduces the flip rate, recommending both be reported.

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Hugging Face Daily Papers

The paper tests label-free strategies to reduce positional bias in multiple-choice benchmarks but finds they do not reliably improve accuracy or reduce bias. Only showing all options with an LLM matcher preserves baseline performance, while cyclic permutation sometimes helps.

Position Bias Correction is Insufficient for One-Pass Attention Sorting

arXiv cs.CL

This paper tests the hypothesis that correcting position bias in attention scores can enable single-pass document sorting for long-context QA. The authors find that while debiasing helps moderately, it does not match the accuracy of iterative sorting, indicating that repeated reordering provides additional benefits beyond bias correction alone.

More Thinking, More Bias: Length-Driven Position Bias in Reasoning Models

arXiv cs.AI

This research paper investigates position bias in reasoning models, finding that bias scales with the length of the reasoning trajectory rather than being eliminated by 'more thinking.' The study provides causal evidence and a diagnostic toolkit for auditing this length-driven bias in multiple-choice QA evaluations.

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

arXiv cs.CL

This paper investigates whether label-free strategies for multiple-choice benchmarks can remove option-order sensitivity in large language models, finding that neither two-stage prompting nor independent hypothesis scoring reliably improves accuracy.