Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative

# Accuracy and Normalized Accuracy under Length Bias: Analysis, Guidelines, and a Bayesian Alternative
Source: [https://arxiv.org/html/2607.12767](https://arxiv.org/html/2607.12767)
###### Abstract

Multiple\-choice benchmarks that rank candidate completions by conditional log\-probability suffer from a length bias: because log\-probabilities sum over tokens, longer answers tend to be penalized relative to shorter ones in practice\. A common mitigation is to normalize scores by completion length, but we show empirically that this heuristic frequently over\-corrects, introducing a bias toward longer answers instead\. We first analyze these scoring rules, characterizing when standard and length\-normalized accuracy are appropriate and how their length biases depend on the distribution of completion lengths\. Motivated by this analysis, we introduce*Bayesian accuracy*, a scoring rule that computes the posterior probability of each candidate under an explicit prior over answer length, thereby removing linear length effects\. Bayesian accuracy is a drop\-in replacement for likelihood\-based multiple\-choice evaluation, requires no additional forward passes, and consistently exhibits lower empirical length bias than both standard and length\-normalized accuracy across benchmarks and few\-shot settings\.

Machine Learning, ICML

## 1Introduction

Large language models \(LLMs\) have achieved strong performance on a wide range of natural language processing tasks, from factual question answering to multi\-step reasoning\(Brown et al\.,[2020](https://arxiv.org/html/2607.12767#bib.bib6)\)\. A common way to evaluate such models is via multiple\-choice benchmarks: the model is given a prompt together with a small set of candidate completions, which are ranked according to the conditional log\-probability the model assigns to each candidate\. The candidate with the highest score is taken as the model’s answer, and accuracy is computed as the fraction of prompts for which this highest\-scoring candidate is correct\. Prominent examples of this likelihood\-based multiple\-choice evaluation include MMLU\(Hendrycks et al\.,[2020](https://arxiv.org/html/2607.12767#bib.bib13)\), HellaSwag\(Zellers et al\.,[2019](https://arxiv.org/html/2607.12767#bib.bib21)\), and WinoGrande\(Sakaguchi et al\.,[2021](https://arxiv.org/html/2607.12767#bib.bib20)\)\.

This way of evaluating models has a key advantage: by restricting the space of possible outputs to a fixed set of benchmark\-provided completions, it turns open\-ended generation into a simple answer\-checking problem\. The evaluator no longer has to parse or interpret arbitrary text, it only needs to compare scores over a small, known candidate set\. This is particularly useful for pretrained base models that have not been instruction\-tuned and may not reliably follow templates or formatting instructions in free\-form generation settings\. Because multiple\-choice evaluation only requires the model to assign probabilities to the given candidates and select the most likely one, it sidesteps these formatting issues and yields a simple, scalable metric that is insensitive to output format\.

A key property of these benchmarks is that the total log\-likelihood of a completion is typically given by the sum of its token\-wise log\-probabilities\. Because this score is additive over tokens, any per\-token uncertainty or modeling error is accumulated with length: even when the model “knows” the correct answer, it is likely to still incur a non\-zero loss on each token, so longer answers have more opportunities to accumulate negative log\-probability\. Empirically, as will be shown in this paper, the total log\-likelihoods often decrease approximately linearly with completion length, inducing a systematic*length bias*towards shorter candidates\.

A widely used heuristic to mitigate this bias is to normalize the total log\-likelihood by the length of the completion, for example by dividing by the number of tokens or bytes\(Brown et al\.,[2020](https://arxiv.org/html/2607.12767#bib.bib6); Gao et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib10)\)\. Length\-normalized scores can be interpreted as \(the negative of\) an average per\-token cross\-entropy and are monotonically related to perplexity\. In practice, however, we find that such normalization frequently*over\-corrects*: scores become biased in the opposite direction, favoring longer completions instead\. Moreover, these effects are not uniform across benchmarks or task formats, raising two questions: \(I\) when is it appropriate to use unnormalized versus length\-normalized accuracy, and \(II\) is there an alternative that behaves robustly across tasks?

Beyond length normalization, several alternative scoring rules have been proposed that adjust for unconditional priors or rescale scores, most notably pointwise mutual information \(PMI\) and asymmetric normalized PMI \(ANPMI\); we review these metrics and their limitations as baselines in Section[2\.2](https://arxiv.org/html/2607.12767#S2.SS2)\.

In this work, we study length bias in likelihood\-based multiple\-choice benchmarks and propose a simple Bayesian alternative to both standard and length\-normalized accuracy\. Our contributions are as follows:

1. 1\.We formalize length bias in log\-likelihood\-based benchmarks by defining it as the average Kendall rank correlation coefficient between candidate lengths and their scores\.
2. 2\.Using this formalization, we empirically characterize which benchmarks and scoring rules exhibit substantial length bias and how this depends on model and training setup\.
3. 3\.Motivated by the empirical observation that, for standard subword tokenizers used by contemporary autoregressive LLMs and across a representative range of multiple\-choice benchmarks, total log\-likelihoods exhibit a dominant approximately affine dependence on completion length, we introduce*Bayesian accuracy*, a scoring rule that incorporates an explicit length prior and removes this first\-order length trend, while remaining a drop\-in replacement for existing likelihood\-based evaluations\.
4. 4\.Across benchmarks and few\-shot settings, Bayesian accuracy yields lower length bias than standard and length\-normalized accuracy, without additional forward passes\.

## 2Background

### 2\.1Notation

We evaluate a language modelfθf\_\{\\theta\}on multiple\-choice datasets of the form

𝒟=\{\(x\(k\),C\(k\),y\(k\)\)\}k=1K,\\displaystyle\\mathcal\{D\}=\\left\\\{\\bigl\(x^\{\(k\)\},C^\{\(k\)\},y^\{\(k\)\}\\bigr\)\\right\\\}\_\{k=1\}^\{K\},\(1\)wherex\(k\)x^\{\(k\)\}is the prompt,C\(k\)=\{cm\(k\)\}m=1MkC^\{\(k\)\}=\\\{c^\{\(k\)\}\_\{m\}\\\}\_\{m=1\}^\{M\_\{k\}\}is the set of candidate completions, andy\(k\)∈C\(k\)y^\{\(k\)\}\\in C^\{\(k\)\}is the ground\-truth answer\. These completions are of lengthnm\(k\)≡\|cm\(k\)\|n\_\{m\}^\{\(k\)\}\\equiv\|c^\{\(k\)\}\_\{m\}\|measured either in bytes or tokens\. For a prompt–completion pair\(x,c\)\(x,c\), we write

ℓθ​\(c∣x\)≡log⁡Pθ​\(c∣x\)\\displaystyle\\ell\_\{\\theta\}\(c\\mid x\)\\equiv\\log P\_\{\\theta\}\(c\\mid x\)\(2\)for the total conditional log\-probability assigned by the model, obtained by summing token\-wise log\-probabilities over the completion\.

A*scoring function*SStransforms these log\-likelihoods into scores

s​\(c∣x\)=S​\(ℓθ​\(c∣x\)\),\\displaystyle s\(c\\mid x\)=S\\bigl\(\\ell\_\{\\theta\}\(c\\mid x\)\\bigr\),\(3\)that are used to determine the model’s prediction by taking the completion with the highest score

y^\(k\)=arg⁡maxc∈C\(k\)⁡s​\(c∣x\(k\)\)\.\\displaystyle\\hat\{y\}^\{\(k\)\}=\\arg\\max\_\{c\\in C^\{\(k\)\}\}s\(c\\mid x^\{\(k\)\}\)\.\(4\)The resulting accuracy for a givenSSis the fraction of correct predictions

AccS​\(fθ;𝒟\)=1K​∑k=1K𝕀​\[y^\(k\)=y\(k\)\]\.\\displaystyle\\text\{Acc\}\_\{S\}\(f\_\{\\theta\};\\mathcal\{D\}\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{I\}\\bigl\[\\hat\{y\}^\{\(k\)\}=y^\{\(k\)\}\\bigr\]\.\(5\)

### 2\.2Existing Metrics

#### 2\.2\.1Standard \(unnormalized\) accuracy

The simplest choice is to use the total conditional log\-likelihood directly as the score, in which case the scoring function is the identity function:

Standard:Ssta​\(ℓθ​\(c∣x\)\)=ℓθ​\(c∣x\)\.\\displaystyle S\_\{\\text\{sta\}\}\(\\ell\_\{\\theta\}\(c\\mid x\)\)=\\ell\_\{\\theta\}\(c\\mid x\)\.\(6\)In this case, the selected completion is the one the model assigns the highest overall probability under the conditional distributionPθ\(⋅∣x\)P\_\{\\theta\}\(\\cdot\\mid x\)to, i\.e\., which completion is \(according to the model\) most likely to follow from the prompt\. Unless otherwise stated, we refer to*standard accuracy*as the accuracy obtained whenS=SstaS=S\_\{\\text\{sta\}\}\.

#### 2\.2\.2Length\-normalized accuracy

A common way of attempting to correct for length bias is to normalize the log\-likelihoods by a measure of completion length, typically the number of tokens, characters, or bytes\(Brown et al\.,[2020](https://arxiv.org/html/2607.12767#bib.bib6); Gao et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib10)\)\. Normalizing by the number of tokens has the advantage that it directly corresponds to text units the model processes and therefore to the number of log\-probability terms being summed\. The resulting score can be interpreted as the negative average token\-wise cross\-entropy, or equivalently, the negative log perplexity:

Token\-normalized:Stok​\(ℓθ​\(c∣x\)\)=ℓθ​\(c∣x\)ntok\.\\displaystyle S\_\{\\text\{tok\}\}\(\\ell\_\{\\theta\}\(c\\mid x\)\)=\\frac\{\\ell\_\{\\theta\}\(c\\mid x\)\}\{n\_\{\\text\{tok\}\}\}\.\(7\)
A drawback of token\-based normalization is that the metric becomes dependent on the tokenizer, which complicates comparisons between models with different tokenization schemes\. This can be mitigated by normalizing by the number of bytes \(or characters\):

Byte\-normalized:Sbyte​\(ℓθ​\(c∣x\)\)=ℓθ​\(c∣x\)nbyte\.\\displaystyle S\_\{\\text\{byte\}\}\(\\ell\_\{\\theta\}\(c\\mid x\)\)=\\frac\{\\ell\_\{\\theta\}\(c\\mid x\)\}\{n\_\{\\text\{byte\}\}\}\.\(8\)The latter variant is widely used in practice\(Gao et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib10); Habib et al\.,[2023](https://arxiv.org/html/2607.12767#bib.bib12)\)\.

#### 2\.2\.3Pointwise mutual information \(PMI\)

Pointwise mutual information \(PMI\)\(Fano & Hawkins,[1961](https://arxiv.org/html/2607.12767#bib.bib9)\)adjusts the conditional log\-likelihoods by subtracting the unconditional log\-likelihoods of the completions:

PMI:SPMI​\(ℓθ​\(c∣x\)\)=ℓθ​\(c∣x\)−ℓθ​\(c\),\\displaystyle S\_\{\\text\{PMI\}\}\(\\ell\_\{\\theta\}\(c\\mid x\)\)=\\ell\_\{\\theta\}\(c\\mid x\)\-\\ell\_\{\\theta\}\(c\),\(9\)whereℓθ​\(c\)≡log⁡Pθ​\(c\)\\ell\_\{\\theta\}\(c\)\\equiv\\log P\_\{\\theta\}\(c\)is the log\-probability of the completion without conditioning on the prompt\. For brevity, we slightly abuse notation and omit the explicit dependence ofℓθ​\(c\)\\ell\_\{\\theta\}\(c\)on its inputs\. Intuitively, PMI measures the increase in log\-probability that the promptxxprovides for completioncc, and by doing so removes any prior preference the model might assign to a completion regardless of the context\.

Although not as widely used as standard or length\-normalized accuracy, PMI\-based scoring has appeared in several recent studies\(Askell et al\.,[2021](https://arxiv.org/html/2607.12767#bib.bib3); Biderman et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib5)\)\. A practical limitation is that PMI requires evaluatingℓθ​\(c\)\\ell\_\{\\theta\}\(c\)for every completion in addition toℓθ​\(c∣x\)\\ell\_\{\\theta\}\(c\\mid x\), doubling the number of forward passes needed for evaluation\. Moreover, by subtracting the unconditional log\-probability of a completion, PMI removes global priors over completions, indirectly mitigating some length\-related biases without explicitly performing length normalization\.

#### 2\.2\.4Asymmetric normalized PMI \(ANPMI\)

Cho et al\. \([2025](https://arxiv.org/html/2607.12767#bib.bib7)\)observe that PMI is bounded from above by−ℓθ​\(c\)\-\\ell\_\{\\theta\}\(c\)\. This happens when the prompt completely determines the completion, which causes the conditional log\-likelihood to approach zero \(see Equation[9](https://arxiv.org/html/2607.12767#S2.E9)\)\. Consequently, different completions have different maximum attainable PMI values depending on their unconditional probabilities, which can distort comparisons between candidates with substantially different priors\. Asymmetric normalized PMI \(ANPMI\) addresses this by normalizing PMI by−ℓθ​\(c\)\-\\ell\_\{\\theta\}\(c\):

ANPMI:SANPMI​\(ℓθ​\(c∣x\)\)=−ℓθ​\(c∣x\)−ℓθ​\(c\)ℓθ​\(c\)\.\\displaystyle S\_\{\\text\{ANPMI\}\}\(\\ell\_\{\\theta\}\(c\\mid x\)\)=\-\\frac\{\\ell\_\{\\theta\}\(c\\mid x\)\-\\ell\_\{\\theta\}\(c\)\}\{\\ell\_\{\\theta\}\(c\)\}\.\(10\)With this normalization, ANPMI is upper\-bounded by11for all completions, regardless of their prior probability\.

In practice, ANPMI inherits the computational overhead of PMI, since it also requires unconditional log\-likelihoodsℓθ​\(c\)\\ell\_\{\\theta\}\(c\)for all completions\. In addition, it can be unstable or undefined for completions with extremely high prior probability \(i\.e\.,ℓθ​\(c\)≈0\\ell\_\{\\theta\}\(c\)\\approx 0\) and is poorly defined for single\-token completions in models that do not employ an explicit beginning\-of\-sentence token\. For these reasons, both PMI and ANPMI serve as useful baselines, but they are not ideal as default scoring rules in large\-scale benchmark evaluations\. Our proposed Bayesian accuracy, in contrast, only requires conditional log\-likelihoods and is explicitly designed to control length bias\.

## 3Experiments

### 3\.1Measuring Length Bias

We quantify the length bias induced by a scoring functionSSon a multiple\-choice dataset by measuring the rank correlation between candidate lengths and their scores within each example: Consider a single example with promptxxand candidate setC=\{cm\}m=1MC=\\\{c\_\{m\}\\\}\_\{m=1\}^\{M\}\. For a modelfθf\_\{\\theta\}and scoring functionSS, the score is defined as

si=S​\(ℓθ​\(ci∣x\)\)\.\\displaystyle s\_\{i\}=S\\bigl\(\\ell\_\{\\theta\}\(c\_\{i\}\\mid x\)\\bigr\)\.\(11\)Throughout the rest of this paper, we primarily use byte length, because of its common usage and desirable tokenizer\-independence property\. However, length bias can be defined analogously by swapping out byte length for token length\. We report additional token\-length results in the appendix, which result in the same conclusions\.

For a single sample, we measure the correlation between the set of scores\{s1,…,sM\}\\\{s\_\{1\},\\dots,s\_\{M\}\\\}and the set of lengths\{n1,…,nM\}\\\{n\_\{1\},\\dots,n\_\{M\}\\\}using Kendall’s rank correlation coefficientτ\\tau\(Kendall,[1938](https://arxiv.org/html/2607.12767#bib.bib16)\)\. For each unordered pair\(i,j\)\(i,j\)with1≤i<j≤M1\\leq i<j\\leq M, we compare their scores and lengths:

- •The pair is*concordant*if\(si−sj\)​\(ni−nj\)\>0\(s\_\{i\}\-s\_\{j\}\)\(n\_\{i\}\-n\_\{j\}\)\>0, i\.e\., the candidate with the higher score is also longer\.
- •The pair is*discordant*if\(si−sj\)​\(ni−nj\)<0\(s\_\{i\}\-s\_\{j\}\)\(n\_\{i\}\-n\_\{j\}\)<0, i\.e\., the candidate with the higher score is shorter\.
- •The pair is*tied in scores*ifsi=sjs\_\{i\}=s\_\{j\}\.
- •The pair is*tied in lengths*ifni=njn\_\{i\}=n\_\{j\}\.

LetM0=M​\(M−1\)/2M\_\{0\}=M\(M\-1\)/2be the total number of unordered pairs of candidates, and letMcM\_\{c\}andMdM\_\{d\}be the numbers of concordant and discordant pairs, respectively\. The numbers of tied pairs are given by

Ts\\displaystyle T\_\{s\}=∑1≤i<j≤M𝕀​\[si=sj\],\\displaystyle=\\sum\_\{1\\leq i<j\\leq M\}\\mathbb\{I\}\[s\_\{i\}=s\_\{j\}\],\(12\)Tb\\displaystyle T\_\{b\}=∑1≤i<j≤M𝕀​\[ni=nj\],\\displaystyle=\\sum\_\{1\\leq i<j\\leq M\}\\mathbb\{I\}\[n\_\{i\}=n\_\{j\}\],\(13\)where𝕀​\[⋅\]\\mathbb\{I\}\[\\cdot\]is the indicator function\. Using these quantities, Kendall’sτb\\tau\_\{b\}is then computed as

τ​\(S;C\)=Mc−Md\(M0−Ts\)​\(M0−Tb\)\.\\displaystyle\\tau\(S;C\)=\\frac\{M\_\{c\}\-M\_\{d\}\}\{\\sqrt\{\(M\_\{0\}\-T\_\{s\}\)\(M\_\{0\}\-T\_\{b\}\)\}\}\.\(14\)Intuitively, the numerator captures the net agreement between the score and length orderings: it is positive when pairs where the higher\-scoring candidate is longer dominate, and negative when higher scores tend to go to shorter candidates\. The denominator normalizes by the number of pairs whose relative ordering is actually determined once ties in scores and lengths are removed, ensuring thatτ​\(S;C\)∈\[−1,1\]\\tau\(S;C\)\\in\[\-1,1\]\. Negative values indicate that higher scores are typically assigned to*shorter*candidates, positive values indicate a tendency to favor*longer*candidates, and values near zero correspond to little or no monotonic association between score and length\. When all candidates have identical scores or identical lengths, no meaningful ordering exists and the denominator is zero, soτ​\(S;C\)\\tau\(S;C\)is undefined\.

The overall length bias of a scoring functionSSon dataset𝒟\\mathcal\{D\}is defined as the average Kendall correlation across all samples for which it is defined\. Let

Idef≡\{k∈\{1,…,K\}∣τ​\(S;C\(k\)\)​is defined\}\\displaystyle I\_\{\\text\{def\}\}\\equiv\\left\\\{k\\in\\\{1,\\dots,K\\\}\\mid\\tau\\left\(S;C^\{\(k\)\}\\right\)\\text\{ is defined\}\\right\\\}\(15\)be the set of indices with a well\-definedτ\\tau\. Then, the overall length bias of the dataset is computed as the average overτ\\tau\-defined samples:

τ​\(S,𝒟\)≡1\|Idef\|​∑k∈Idefτ​\(S;C\(k\)\)\.\\displaystyle\\tau\(S,\\mathcal\{D\}\)\\equiv\\frac\{1\}\{\|I\_\{\\text\{def\}\}\|\}\\sum\_\{k\\in I\_\{\\text\{def\}\}\}\\tau\(S;C^\{\(k\)\}\)\.\(16\)

### 3\.2Benchmarks and Models

We evaluate on a suite of multiple\-choice benchmarks: ARC\(Clark et al\.,[2018](https://arxiv.org/html/2607.12767#bib.bib8)\), ARC German\(LAION,[2024](https://arxiv.org/html/2607.12767#bib.bib17)\), HellaSwag\(Zellers et al\.,[2019](https://arxiv.org/html/2607.12767#bib.bib21)\), MMLU\(Hendrycks et al\.,[2020](https://arxiv.org/html/2607.12767#bib.bib13)\), OpenbookQA\(Mihaylov et al\.,[2018](https://arxiv.org/html/2607.12767#bib.bib18)\), SciQ\(Johannes Welbl,[2017](https://arxiv.org/html/2607.12767#bib.bib15)\), and WinoGrande\(Sakaguchi et al\.,[2021](https://arxiv.org/html/2607.12767#bib.bib20)\)\. For MMLU we use two formats\. In*MMLU Full\-Text*, the prompt contains the question and answer options, and each candidate completion is the full answer text\. In*MMLU Cloze*, the option list is removed and the model instead fills in an answer span directly in the context\. These variants let us probe how length bias changes when moving from single\-symbol labels to heterogeneous multi\-token completions\. We run all experiments in a zero\-shot setting; prompt templates and preprocessing details are given in the appendix\.

We evaluate open\-weight models from the LLaMA, Qwen, Mistral, Pythia, and SmolLM families, covering parameter counts from 135M to 70B in both base and instruction\-tuned configurations\(Grattafiori et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib11); Jiang et al\.,[2023](https://arxiv.org/html/2607.12767#bib.bib14); Biderman et al\.,[2023](https://arxiv.org/html/2607.12767#bib.bib4); Allal et al\.,[2024](https://arxiv.org/html/2607.12767#bib.bib2); Qwen Team,[2025](https://arxiv.org/html/2607.12767#bib.bib19)\)\.

Unless otherwise stated, experiments in the main text use the zero\-shot setting; few\-shot results are reported in the appendix\. In the few\-shot setting, we use one fixed exemplar set per prompt configuration, and estimateb^\\hat\{b\}separately for each model–dataset–prompt configuration, including the corresponding few\-shot prompt\.

All evaluations are run using the Aleph Alpha Eval Framework\(Aleph Alpha Research,[2025](https://arxiv.org/html/2607.12767#bib.bib1)\)\.

### 3\.3Empirical Length Bias of Existing Scoring Rules

We begin by analyzing how completion length relates to log\-likelihood under the standard and length\-normalized scoring rules, and how these score\-length curves align with the measured length biases\. This lets us check how well the assumed linear scaling of log\-likelihood with length holds in practice and how these trends translate into the rank\-based length biasτ​\(S,𝒟\)\\tau\(S,\\mathcal\{D\}\), as defined in Section[3\.1](https://arxiv.org/html/2607.12767#S3.SS1)\.

#### 3\.3\.1Standard Accuracy

Figure[1](https://arxiv.org/html/2607.12767#S3.F1)plots, for each benchmark, the average conditional log\-likelihoodℓθ​\(c∣x\)\\ell\_\{\\theta\}\(c\\mid x\)as a function of completion byte lengthnbyten\_\{\\text\{byte\}\}, aggregated over all models and candidates and binned for readability\.

![Refer to caption](https://arxiv.org/html/2607.12767v1/x1.png)Figure 1:Average conditional log\-likelihoodℓθ​\(c∣x\)\\ell\_\{\\theta\}\(c\\mid x\)versus completion byte lengthnbyten\_\{\\text\{byte\}\}, aggregated over all models\. For readability, completion lengths are restricted to a maximum of 120 bytes and results are binned\.Across almost all benchmarks we observe a pronounced, approximately linear decrease of total log\-likelihood with length\. The main exception is MMLU Full\-Text, whose slope is close to zero at the aggregate level\. This behavior matches the intuition that any non\-zero per\-token loss accumulates across tokens and therefore induces a systematic preference for shorter candidates\. It is worth noting that the fitted lines also exhibit a substantial positive intercept at length one\. This seems to indicate that the first token of the completion is noticeably harder to predict than subsequent tokens\. Initial tokens must both start a new span \(often with weaker local context than mid\-span tokens\)\. Subsequent tokens then benefit from this additional context and incur a smaller average per\-token loss\.

Table[1](https://arxiv.org/html/2607.12767#S3.T1)summarizes the actual length biases for standard accuracy using the average Kendall correlationτ​\(Ssta,𝒟\)\\tau\(S\_\{\\text\{sta\}\},\\mathcal\{D\}\)\. For most benchmarks and models,τ\\tauis clearly negative, confirming our suspicion that unnormalized likelihood tends to favor shorter candidates\. The effect is particularly strong on HellaSwag and the ARC benchmarks, where candidate lengths vary substantially, and weakest on SciQ and WinoGrande, whoseτ\\tauvalues are close to zero despite visible slopes in Figure[1](https://arxiv.org/html/2607.12767#S3.F1)\. This illustrates that a sloped score\-versus\-length curve does not necessarily imply a strong per\-example rank correlation: length bias depends not only on the average slope but also on how candidate lengths are distributed within each question\. Within each model family, the magnitude of negativeτ\\tautends to shrink with model size, indicating that larger models are somewhat less sensitive to length artifacts under standard scoring, although the bias remains clearly non\-zero\.

#### 3\.3\.2Multiple\-Choice with Locally Determined Answers

MMLU Full\-Text provides an illustrative example\. Here, each candidate completion is the full answer sentence, but the question and the answer options are already present in the prompt\. When we decompose the total log\-likelihood into per\-token contributions, we find that the*first*one or two answer tokens account for almost all of the variation between candidates\. Once the model has committed to a specific option, the remainder of the answer is largely a deterministic copy of text already appearing in the prompt and therefore carries much higher conditional probabilities and much smaller per\-token loss\.

In other words, for these “locally determined” answers the candidate identity is effectively fixed by the first few tokens, and the remaining tokens contribute an almost identical offset to the log\-likelihoods of all candidates\. Consequently, the effective slope is close to zero\. This explains why MMLU Full\-Text shows little aggregate length dependence and near\-zero length bias under standard accuracy despite substantial variation in absolute completion lengths\.

A second situation in which standard accuracy is effectively insensitive to length is when all completions for a given example have \(almost\) the same length\. Intuitively, if the shortest and longest candidates differ only slightly, then they all sum log\-probabilities over nearly the same number of tokens/bytes, so the length\-dependent part of the score is almost identical across candidates\.

We summarize these observations as a simple rule of thumb:

When is standard accuracy safe to use?Standard \(unnormalized\) accuracy is mainly appropriate when \(i\) the answer text already appears in the prompt and the first one or two completion tokens effectively fix the option, or \(ii\) all candidates for a question have very similar lengths, with single\-letter prediction benchmarks as an extreme case\.

Table 1:Length bias for standard accuracy\.
#### 3\.3\.3Normalized Accuracy

We now repeat the analysis for byte\-normalized scoresSbyteS\_\{\\text\{byte\}\}, plottingℓθ​\(c∣x\)/nbyte\\ell\_\{\\theta\}\(c\\mid x\)/n\_\{\\text\{byte\}\}againstnbyten\_\{\\text\{byte\}\}in Figure[2](https://arxiv.org/html/2607.12767#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2607.12767v1/x2.png)Figure 2:Average byte\-normalized conditional log\-likelihoodℓθ​\(c∣x\)/nbyte\\ell\_\{\\theta\}\(c\\mid x\)/n\_\{\\text\{byte\}\}versus completion byte length, aggregated over all models\. For readability, completion lengths are restricted to a maximum of 120 bytes and results are binned\.Naïve length normalization largely flattens the global trend over medium\-length completions: the strong negative slopes in Figure[1](https://arxiv.org/html/2607.12767#S3.F1)are substantially reduced\. However, for short completions we often observe a sharp decrease inℓθ​\(c∣x\)/nbyte\\ell\_\{\\theta\}\(c\\mid x\)/n\_\{\\text\{byte\}\}as a function of length\. This suggests that normalization can over\-correct the raw length bias: very short completions become relatively under\-scored compared to slightly longer ones, inducing a systematic bias*towards*longer answers instead\.

Table[2](https://arxiv.org/html/2607.12767#S3.T2)quantifies this effect\. In contrast to Table[1](https://arxiv.org/html/2607.12767#S3.T1), the Kendall correlationsτ​\(Sbyte,𝒟\)\\tau\(S\_\{\\text\{byte\}\},\\mathcal\{D\}\)are uniformly positive across benchmarks and models, often with comparable or even larger magnitude than the negative biases under standard accuracy\. On datasets such as ARC, ARC German, OpenbookQA, SciQ and WinoGrande, length\-normalization turns a modest negative or near\-zeroτ\\tauinto a substantial positive one, indicating that normalization introduces a strong length bias where there was little before\. For HellaSwag and MMLU Full\-Text, byte\-normalization partially corrects the strong negative bias of standard accuracy, bringingτ\\taucloser to zero for base models, but overshoots for instruction\-tuned models, which show markedly larger positiveτ\\tauon most benchmarks\.

Table 2:Length bias of byte\-normalized accuracy\.These mixed outcomes make it hard to state a simple, structural condition, analogous to locally determined answers or small within\-example length variation forSstaS\_\{\\text\{sta\}\}—under which length\-normalized accuracy is guaranteed to be safe\. Instead, its behavior appears to depend sensitively on the dataset templating and the specific model family\. We therefore summarize its practical use in a rule\-of\-thumb box:

When is length\-normalized accuracy safe to use?The results do not reveal a simple heuristic that guarantees low length bias under byte\- or token\-normalized accuracy\. We therefore do not view normalized accuracy as a safe default and only recommend it once low length bias has been empirically confirmed\. Instead, we suggest using the length\-corrected Bayesian alternative introduced in the next section\.

## 4Bayesian Accuracy

### 4\.1A Generative Model for Log\-Likelihoods

We now formalize the empirical observation from Section[3\.3](https://arxiv.org/html/2607.12767#S3.SS3)that, within a benchmark, total conditional log\-likelihoods of standard subword tokenizers scale approximately linearly with completion length\. For each examplekkwith promptx\(k\)x^\{\(k\)\}and candidate setC\(k\)=\{cm\(k\)\}m=1MkC^\{\(k\)\}=\\\{c^\{\(k\)\}\_\{m\}\\\}\_\{m=1\}^\{M\_\{k\}\}, writeℓm\(k\)≡ℓθ​\(cm\(k\)∣x\(k\)\)\\ell^\{\(k\)\}\_\{m\}\\equiv\\ell\_\{\\theta\}\(c^\{\(k\)\}\_\{m\}\\mid x^\{\(k\)\}\)for the total conditional log\-likelihood, and letnm\(k\)n^\{\(k\)\}\_\{m\}denote the byte length ofcm\(k\)c^\{\(k\)\}\_\{m\}\.

A simple but flexible description of the observed score\-length relationship is the hierarchical linear model

ℓm\(k\)=αk\+βk​nm\(k\)\+εm\(k\),\\displaystyle\\ell^\{\(k\)\}\_\{m\}=\\alpha\_\{k\}\+\\beta\_\{k\}n^\{\(k\)\}\_\{m\}\+\\varepsilon^\{\(k\)\}\_\{m\},\(17\)where

- •αk\\alpha\_\{k\}is a prompt\-specific offset capturing the overall difficulty of itemkkand the effect of the first completion token,
- •βk\\beta\_\{k\}is a prompt\-specific average per\-byte contribution to the log\-likelihood, i\.e\., the local slope of the length trend for examplekk,
- •εm\(k\)\\varepsilon^\{\(k\)\}\_\{m\}aggregates token\-level fluctuations around this trend and has zero mean with a variance that grows linearly in the length\.

We treat\(αk,βk\)\(\\alpha\_\{k\},\\beta\_\{k\}\)as draws from a population distribution with finite moments𝔼​\[αk\]=a\\mathbb\{E\}\[\\alpha\_\{k\}\]=a,𝔼​\[βk\]=b\\mathbb\{E\}\[\\beta\_\{k\}\]=band variancesVar​\[αk\]=σα2\\mathrm\{Var\}\[\\alpha\_\{k\}\]=\\sigma\_\{\\alpha\}^\{2\},Var​\[βk\]=σβ2\\mathrm\{Var\}\[\\beta\_\{k\}\]=\\sigma\_\{\\beta\}^\{2\}\. Marginalizing over prompts yields a linear trend:

𝔼​\[ℓm\(k\)∣nm\(k\)=n\]\\displaystyle\\mathbb\{E\}\[\\ell^\{\(k\)\}\_\{m\}\\mid n^\{\(k\)\}\_\{m\}=n\]=a\+b​n,\\displaystyle=a\+bn,\(18\)and variance that scales quadratically in the length:

Var​\[ℓm\(k\)∣nm\(k\)=n\]=𝒪​\(n2\)\.\\displaystyle\\mathrm\{Var\}\[\\ell^\{\(k\)\}\_\{m\}\\mid n^\{\(k\)\}\_\{m\}=n\]=\\mathcal\{O\}\(n^\{2\}\)\.\(19\)This is in line with the empirical trends in Figure[1](https://arxiv.org/html/2607.12767#S3.F1)and Figure[3](https://arxiv.org/html/2607.12767#A1.F3)in the Appendix\. The quadratic behavior stems from variation in the per\-example slopesβk\\beta\_\{k\}: ifβk\\beta\_\{k\}were constant, only the linearn​σ2n\\sigma^\{2\}term would remain\.

If we use the rawℓm\(k\)\\ell^\{\(k\)\}\_\{m\}to rank candidates, the average slopebbinduces a systematic preference for shorter completions whenb<0b<0or longer completions whenb\>0b\>0\. Our goal is to remove this*global*length trend while preserving genuine, context\-dependent preferences among candidates of similar length\. Rather than dividing by length, which rescales both signal and noise, we instead introduce an explicit prior over completion lengths and absorb the linear trend into that prior\.

### 4\.2Bayesian Correction

LetPθ​\(c∣x\)P\_\{\\theta\}\(c\\mid x\)denote the model’s conditional distribution over completions andℓθ​\(c∣x\)=log⁡Pθ​\(c∣x\)\\ell\_\{\\theta\}\(c\\mid x\)=\\log P\_\{\\theta\}\(c\\mid x\)the corresponding log\-likelihood\. We introduce a prior over completion lengths,Pprior​\(n\)P\_\{\\text\{prior\}\}\(n\), and define a posterior over completions

Ppost​\(c∣x\)∝Pθ​\(c∣x\)​Pprior​\(n\)\.\\displaystyle P\_\{\\text\{post\}\}\(c\\mid x\)\\propto P\_\{\\theta\}\(c\\mid x\)\\,P\_\{\\text\{prior\}\}\(n\)\.\(20\)Taking the logarithm of the posterior distribution gives

log⁡Ppost​\(c∣x\)=ℓθ​\(c∣x\)\+log⁡Pprior​\(n\)−log⁡Z​\(x\),\\displaystyle\\log P\_\{\\text\{post\}\}\(c\\mid x\)=\\ell\_\{\\theta\}\(c\\mid x\)\+\\log P\_\{\\text\{prior\}\}\(n\)\-\\log Z\(x\),\(21\)whereZ​\(x\)Z\(x\)is a normalization constant that does not depend onccand therefore does not affect the argmax over candidates\.

The linear relation \([18](https://arxiv.org/html/2607.12767#S4.E18)\) implies that the length prior is exponential in length:

Pprior​\(n\)∝exp⁡\(−b​n\),\\displaystyle P\_\{\\text\{prior\}\}\(n\)\\propto\\exp\\bigl\(\-b\\,n\\bigr\),\(22\)with length slopeb∈ℝb\\in\\mathbb\{R\}\. Since benchmark evaluation only compares a finite set of candidates, we only need these relative length weights within each candidate set\. Plugging this into the posterior and dropping terms that are constant across candidates yields the*length\-debiased log\-likelihood score*:

SBayes​\(c∣x;b\)\\displaystyle S\_\{\\text\{Bayes\}\}\(c\\mid x;b\)≡ℓ~θ​\(c∣x;b\)≡ℓθ​\(c∣x\)−b​n\.\\displaystyle\\equiv\\tilde\{\\ell\}\_\{\\theta\}\(c\\mid x;b\)\\equiv\\ell\_\{\\theta\}\(c\\mid x\)\-b\\,n\.\(23\)Becauseℓ~θ\\tilde\{\\ell\}\_\{\\theta\}is obtained by a simple affine transformation of the same conditional log\-likelihoods used for standard accuracy, Bayesian accuracy can be used as a drop\-in replacement that does not require any additional model evaluations\. It should be noted that this correction targets length\-dependent priors induced by additive log\-likelihood accumulation, but does not remove other answer priors such as preferences for frequent words, phrases, or syntactic forms\.

### 4\.3Estimating the Length Decay Factor

To apply Bayesian accuracy, we estimate one global length\-decay factorbbfor each model–dataset–prompt configuration\. Rather than fitting a pooled regression of log\-likelihood on length, we estimatebbfrom within\-question variation\. This removes prompt\-specific offsetsαk\\alpha\_\{k\}, which would otherwise confound the slope whenever easier or harder prompts systematically contain longer candidates\.

Concretely, for each question we center candidate lengths and log\-likelihoods within that question and fit the resulting fixed\-effect slope:

b^\\displaystyle\\hat\{b\}=∑kMk​∑i=1Mk\(ni\(k\)−n¯\(k\)\)​\(ℓi\(k\)−ℓ¯\(k\)\)∑kMk​∑i=1Mk\(ni\(k\)−n¯\(k\)\)2\.\\displaystyle=\\frac\{\\displaystyle\\sum\_\{k\}M\_\{k\}\\sum\_\{i=1\}^\{M\_\{k\}\}\\bigl\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}^\{\(k\)\}\\bigr\)\\bigl\(\\ell^\{\(k\)\}\_\{i\}\-\\bar\{\\ell\}^\{\(k\)\}\\bigr\)\}\{\\displaystyle\\sum\_\{k\}M\_\{k\}\\sum\_\{i=1\}^\{M\_\{k\}\}\\bigl\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}^\{\(k\)\}\\bigr\)^\{2\}\}\.\(24\)This estimator is equivalent to regressing pairwise log\-likelihood differences on pairwise length differences, but the centered form is linear in the number of candidates per question\. When calibration data contain little within\-question length variation,b^\\hat\{b\}can be noisy; in that case we use the same prompt configuration as the evaluation set and can optionally stabilize the estimate with held\-out calibration, pooling across related templates, or shrinkage towardb=0b=0\. Appendix[A\.1](https://arxiv.org/html/2607.12767#A1.SS1)gives the full derivation and Algorithm[1](https://arxiv.org/html/2607.12767#alg1)gives the implementation\.

### 4\.4Results

For each model–dataset pair we estimate a separateb^\\hat\{b\}and rescore candidates with the length\-corrected log\-likelihoodℓ~θ​\(c∣x;b^\)\\tilde\{\\ell\}\_\{\\theta\}\(c\\mid x;\\hat\{b\}\)\. Table[3](https://arxiv.org/html/2607.12767#S4.T3)reports the resulting Kendall correlationsτ​\(S,𝒟\)\\tau\(S,\\mathcal\{D\}\)between candidate length and score under Bayesian accuracy\.

Table 3:Length bias under Bayesian accuracy\.Bayesian accuracy brings the rank\-correlation between candidate length and score close to zero: almost all entries in Table[3](https://arxiv.org/html/2607.12767#S4.T3)satisfy\|τ\|≤0\.1\|\\tau\|\\leq 0\.1\. This supports Bayesian accuracy as a robust scoring rule that substantially reduces length bias without requiring additional forward passes and while preserving meaningful within\-item preferences among candidates\.

### 4\.5Comparison to baselines

We now compare Bayesian accuracy to standard accuracy, byte\-normalized accuracy, PMI, and ANPMI in terms of their overall length bias\. For each model and scoring ruleSS, we summarize length bias by the mean absolute correlation

\|τ\|¯≡1\|ℬ\|​∑𝒟∈ℬ\|τ​\(S,𝒟\)\|,\\displaystyle\\overline\{\|\\tau\|\}\\equiv\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{\\mathcal\{D\}\\in\\mathcal\{B\}\}\\bigl\|\\tau\(S,\\mathcal\{D\}\)\\bigr\|,\(25\)averaged over the benchmarksℬ\\mathcal\{B\}from Section[3\.3](https://arxiv.org/html/2607.12767#S3.SS3)\. Table[4](https://arxiv.org/html/2607.12767#S4.T4)reports\|τ\|¯\\overline\{\|\\tau\|\}per model and metric \(lower is better\)\.

Table 4:Average absolute length bias\|τ\|¯\\overline\{\|\\tau\|\}per model and scoring rule, aggregated over all benchmarks\.*Standard*and*Norm*correspond to unnormalized and byte\-normalized accuracy, respectively\. Lower is better\.Byte\-normalized accuracy \(*Norm*\) exhibits the largest mean absolute length bias, often substantially worse than standard accuracy\. PMI and ANPMI reduce bias relative to both, and ANPMI consistently improves on PMI, but both require unconditional log\-likelihoods for every candidate, doubling evaluation cost and still leaving non\-trivial residual bias\. Bayesian accuracy attains the smallest or near\-smallest\|τ\|¯\\overline\{\|\\tau\|\}for all models: values in the range0\.030\.03–0\.070\.07correspond to roughly a4×4\\times–8×8\\timesreduction in length bias relative to normalized accuracy and typically a several\-fold improvement over PMI and ANPMI, for both base and instruction\-tuned models\. We therefore recommend Bayesian accuracy as a simple length\-corrected default, reserving standard accuracy for regimes where length is provably irrelevant and using normalized accuracy only when its low bias has been empirically verified\.

## 5Conclusion

Multiple\-choice benchmarks that score candidates by summed conditional log\-likelihood are convenient and widely used, but this additivity induces systematic length bias: across models and datasets we find that total log\-likelihoods scale approximately linearly with completion length, so standard accuracy tends to prefer shorter answers, while naïve length\-normalization often over\-corrects and favors longer ones instead\. We quantify these effects via Kendall’sτ\\taubetween candidate lengths and scores within each example, which highlights when length can be ignored \(locally determined answers or near\-equal\-length candidates\) and when both standard and normalized accuracy become unreliable\.

Motivated by the empirical score–length relationship, we introduce Bayesian accuracy: a drop\-in replacement for standard accuracy that incorporates an explicit exponential prior over completion lengths and subtracts a learned global length trend\. Bayesian accuracy only uses conditional log\-likelihoods, requires no additional forward passes, and consistently reduces measured length bias by a large factor compared to standard, normalized, PMI, and ANPMI\-based scoring across benchmarks and models\. This addresses a measurable nuisance effect in likelihood\-based evaluation, while broader questions of evaluation faithfulness, such as agreement with downstream capabilities, remain important directions for future work\. We therefore view Bayesian accuracy as a simple, robust default for likelihood\-based multiple\-choice evaluation\.

## Acknowledgements

I thank Tom Burns for many helpful brainstorming sessions and early discussions that shaped the ideas in this paper\.

## Impact Statement

This work proposes a simple correction for length bias in likelihood\-based multiple\-choice evaluation of language models\. More reliable scoring rules can change comparative model rankings and encourage fairer, more robust benchmark design, but they do not address other sources of bias or harmful behavior in the underlying models, which must be studied separately\.

## Financial Conflict of Interest

The author declares no financial or other substantive conflicts of interest that could reasonably be perceived to influence this work\.

## References

- Aleph Alpha Research \(2025\)Aleph Alpha Research\.Aleph alpha eval framework, 2025\.URL[https://github\.com/Aleph\-Alpha\-Research/eval\-framework](https://github.com/Aleph-Alpha-Research/eval-framework)\.
- Allal et al\. \(2024\)Allal, L\. B\., Lozhkov, A\., Bakouch, E\., von Werra, L\., and Wolf, T\.Smollm \- blazingly fast and remarkably powerful, 2024\.
- Askell et al\. \(2021\)Askell, A\., Bai, Y\., Chen, A\., Drain, D\., Ganguli, D\., Henighan, T\., Jones, A\., Joseph, N\., Mann, B\., DasSarma, N\., et al\.A general language assistant as a laboratory for alignment\.*arXiv preprint arXiv:2112\.00861*, 2021\.
- Biderman et al\. \(2023\)Biderman, S\., Schoelkopf, H\., Anthony, Q\. G\., Bradley, H\., O’Brien, K\., Hallahan, E\., Khan, M\. A\., Purohit, S\., Prashanth, U\. S\., Raff, E\., et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning*, pp\. 2397–2430\. PMLR, 2023\.
- Biderman et al\. \(2024\)Biderman, S\., Schoelkopf, H\., Sutawika, L\., Gao, L\., Tow, J\., Abbasi, B\., Aji, A\. F\., Ammanamanchi, P\. S\., Black, S\., Clive, J\., et al\.Lessons from the trenches on reproducible evaluation of language models\.*arXiv preprint arXiv:2405\.14782*, 2024\.
- Brown et al\. \(2020\)Brown, T\., Mann, B\., Ryder, N\., Subbiah, M\., Kaplan, J\. D\., Dhariwal, P\., Neelakantan, A\., Shyam, P\., Sastry, G\., Askell, A\., et al\.Language models are few\-shot learners\.*Advances in neural information processing systems*, 33:1877–1901, 2020\.
- Cho et al\. \(2025\)Cho, G\., So, Y\., and Lee, J\.Anpmi: Assessing the true comprehension capabilities of llms for multiple choice questions\.*arXiv preprint arXiv:2502\.18798v3*, 2025\.
- Clark et al\. \(2018\)Clark, P\., Cowhey, I\., Etzioni, O\., Khot, T\., Sabharwal, A\., Schoenick, C\., and Tafjord, O\.Think you have solved question answering? try arc, the ai2 reasoning challenge\.*arXiv:1803\.05457v1*, 2018\.
- Fano & Hawkins \(1961\)Fano, R\. M\. and Hawkins, D\.Transmission of information: A statistical theory of communications\.*American Journal of Physics*, 29\(11\):793–794, 1961\.
- Gao et al\. \(2024\)Gao, L\., Tow, J\., Abbasi, B\., Biderman, S\., Black, S\., DiPofi, A\., Foster, C\., Golding, L\., Hsu, J\., Le Noac’h, A\., Li, H\., McDonell, K\., Muennighoff, N\., Ociepa, C\., Phang, J\., Reynolds, L\., Schoelkopf, H\., Skowron, A\., Sutawika, L\., Tang, E\., Thite, A\., Wang, B\., Wang, K\., and Zou, A\.The language model evaluation harness, 07 2024\.URL[https://zenodo\.org/records/12608602](https://zenodo.org/records/12608602)\.
- Grattafiori et al\. \(2024\)Grattafiori, A\., Dubey, A\., Jauhri, A\., Pandey, A\., Kadian, A\., Al\-Dahle, A\., Letman, A\., Mathur, A\., Schelten, A\., Vaughan, A\., et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Habib et al\. \(2023\)Habib, N\., Fourrier, C\., Kydlíček, H\., Wolf, T\., and Tunstall, L\.Lighteval: A lightweight framework for llm evaluation, 2023\.URL[https://github\.com/huggingface/lighteval](https://github.com/huggingface/lighteval)\.
- Hendrycks et al\. \(2020\)Hendrycks, D\., Burns, C\., Basart, S\., Zou, A\., Mazeika, M\., Song, D\., and Steinhardt, J\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*, 2020\.
- Jiang et al\. \(2023\)Jiang, A\. Q\., Sablayrolles, A\., Mensch, A\., Bamford, C\., Chaplot, D\. S\., de las Casas, D\., Bressand, F\., Lengyel, G\., Lample, G\., Saulnier, L\., Lavaud, L\. R\., Lachaux, M\.\-A\., Stock, P\., Scao, T\. L\., Lavril, T\., Wang, T\., Lacroix, T\., and Sayed, W\. E\.Mistral 7b, 2023\.URL[https://arxiv\.org/abs/2310\.06825](https://arxiv.org/abs/2310.06825)\.
- Johannes Welbl \(2017\)Johannes Welbl, Nelson F\. Liu, M\. G\.Crowdsourcing multiple choice science questions\.2017\.
- Kendall \(1938\)Kendall, M\. G\.A new measure of rank correlation\.*Biometrika*, 30\(1\-2\):81–93, 1938\.
- LAION \(2024\)LAION\.Arcchallenge\_de: German translation of ai2 reasoning challenge \(arc\)\.[https://huggingface\.co/datasets/LeoLM/ArcChallenge\_de](https://huggingface.co/datasets/LeoLM/ArcChallenge_de), 2024\.German translation of the ARC Challenge dataset\.
- Mihaylov et al\. \(2018\)Mihaylov, T\., Clark, P\., Khot, T\., and Sabharwal, A\.Can a suit of armor conduct electricity? a new dataset for open book question answering\.In*EMNLP*, 2018\.
- Qwen Team \(2025\)Qwen Team\.Qwen3 technical report, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Sakaguchi et al\. \(2021\)Sakaguchi, K\., Bras, R\. L\., Bhagavatula, C\., and Choi, Y\.Winogrande: An adversarial winograd schema challenge at scale\.*Communications of the ACM*, 64\(9\):99–106, 2021\.
- Zellers et al\. \(2019\)Zellers, R\., Holtzman, A\., Bisk, Y\., Farhadi, A\., and Choi, Y\.Hellaswag: Can a machine really finish your sentence?*arXiv preprint arXiv:1905\.07830*, 2019\.

## Appendix AAppendix

### A\.1Details of Estimating the Length Decay Factor

A naïve way to estimate the global slopebbwould be to pool all candidates and regressℓm\(k\)\\ell^\{\(k\)\}\_\{m\}directly onnm\(k\)n^\{\(k\)\}\_\{m\}using ordinary least squares\. This ignores the grouping by prompts\. If prompts with larger offsetsαk\\alpha\_\{k\}systematically induce longer or shorter candidates, the pooled slope absorbs both the true length effect and prompt\-level correlations:

𝔼​\[b^OLS\]=b\+𝔼​\[Cov​\(α,n\)\]Var​\(n\)\.\\displaystyle\\mathbb\{E\}\[\\hat\{b\}\_\{\\mathrm\{OLS\}\}\]=b\+\\frac\{\\mathbb\{E\}\[\\mathrm\{Cov\}\(\\alpha,n\)\]\}\{\\mathrm\{Var\}\(n\)\}\.\(26\)
To avoid this, we estimatebbusing only within\-example differences, which cancel the prompt\-specific intercepts\. For each examplekkand candidate pair\(i,j\)\(i,j\), define

Δ​ℓi​j\(k\)\\displaystyle\\Delta\\ell^\{\(k\)\}\_\{ij\}=ℓθ​\(ci\(k\)∣x\(k\)\)−ℓθ​\(cj\(k\)∣x\(k\)\),\\displaystyle=\\ell\_\{\\theta\}\(c^\{\(k\)\}\_\{i\}\\mid x^\{\(k\)\}\)\-\\ell\_\{\\theta\}\(c^\{\(k\)\}\_\{j\}\\mid x^\{\(k\)\}\),\(27\)Δ​ni​j\(k\)\\displaystyle\\Delta n^\{\(k\)\}\_\{ij\}=ni\(k\)−nj\(k\)\.\\displaystyle=n^\{\(k\)\}\_\{i\}\-n^\{\(k\)\}\_\{j\}\.\(28\)Under the linear model in Equation[17](https://arxiv.org/html/2607.12767#S4.E17), these differences satisfy

Δ​ℓi​j\(k\)=βk​Δ​ni​j\(k\)\+εi​j\(k\),\\displaystyle\\Delta\\ell^\{\(k\)\}\_\{ij\}=\\beta\_\{k\}\\,\\Delta n^\{\(k\)\}\_\{ij\}\+\\varepsilon^\{\(k\)\}\_\{ij\},\(29\)whereεi​j\(k\)≡εi\(k\)−εj\(k\)\\varepsilon^\{\(k\)\}\_\{ij\}\\equiv\\varepsilon^\{\(k\)\}\_\{i\}\-\\varepsilon^\{\(k\)\}\_\{j\}is zero\-mean\. Treating theβk\\beta\_\{k\}as noisy draws around a common meanbb, the corresponding pairwise least\-squares estimator is

b^=∑k∑i,jΔ​ℓi​j\(k\)​Δ​ni​j\(k\)∑k∑i,j\(Δ​ni​j\(k\)\)2\.\\displaystyle\\hat\{b\}=\\frac\{\\sum\_\{k\}\\sum\_\{i,j\}\\Delta\\ell^\{\(k\)\}\_\{ij\}\\,\\Delta n^\{\(k\)\}\_\{ij\}\}\{\\sum\_\{k\}\\sum\_\{i,j\}\\bigl\(\\Delta n^\{\(k\)\}\_\{ij\}\\bigr\)^\{2\}\}\.\(30\)The symmetryΔ​ni​j\(k\)=−Δ​nj​i\(k\)\\Delta n^\{\(k\)\}\_\{ij\}=\-\\Delta n^\{\(k\)\}\_\{ji\}implies that these differences are automatically mean\-centered\. The next subsection rewrites the same estimator in the computationally cheaper centered form used in Equation[24](https://arxiv.org/html/2607.12767#S4.E24)\.

### A\.2Mean\-Centered Form of the Length Decay Estimator

The pairwise estimator in \([30](https://arxiv.org/html/2607.12767#A1.E30)\) is naturally written as a double sum over all candidate pairs, which scales asO​\(Mk2\)O\(M\_\{k\}^\{2\}\)in the number of candidatesMkM\_\{k\}for each prompt\. In our settingMkM\_\{k\}is typically small, so the quadratic scaling is not a fundamental bottleneck, but for larger candidate sets it is helpful to avoid quadratic dependence\. In this subsection we show that such pairwise sums can be rewritten exactly as single sums over mean\-centered quantities, reducing the per\-prompt computational cost fromO​\(Mk2\)O\(M\_\{k\}^\{2\}\)toO​\(Mk\)O\(M\_\{k\}\)while leaving the value of the estimator unchanged\.

Letx1,…,xnx\_\{1\},\\dots,x\_\{n\}andy1,…,yny\_\{1\},\\dots,y\_\{n\}be real numbers, and define the means

x¯=1n​∑i=1nxi,y¯=1n​∑i=1nyi\.\\displaystyle\\bar\{x\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}x\_\{i\},\\qquad\\bar\{y\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}y\_\{i\}\.\(31\)Consider

S≡∑i,j=1n\(xi−xj\)​\(yi−yj\)\.\\displaystyle S\\equiv\\sum\_\{i,j=1\}^\{n\}\(x\_\{i\}\-x\_\{j\}\)\(y\_\{i\}\-y\_\{j\}\)\.\(32\)Expanding the product inside the sum gives

S\\displaystyle S=∑i,j\(xi​yi−xi​yj−xj​yi\+xj​yj\)\\displaystyle=\\sum\_\{i,j\}\\bigl\(x\_\{i\}y\_\{i\}\-x\_\{i\}y\_\{j\}\-x\_\{j\}y\_\{i\}\+x\_\{j\}y\_\{j\}\\bigr\)\(33\)=2​n​∑ixi​yi−2​\(∑ixi\)​\(∑iyi\)\\displaystyle=2n\\sum\_\{i\}x\_\{i\}y\_\{i\}\-2\\Bigl\(\\sum\_\{i\}x\_\{i\}\\Bigr\)\\Bigl\(\\sum\_\{i\}y\_\{i\}\\Bigr\)\(34\)=2​n​\(∑ixi​yi−n​x¯​y¯\)\.\\displaystyle=2n\\left\(\\sum\_\{i\}x\_\{i\}y\_\{i\}\-n\\bar\{x\}\\,\\bar\{y\}\\right\)\.\(35\)Finally, note that

∑i=1n\(xi−x¯\)​\(yi−y¯\)=∑ixi​yi−n​x¯​y¯,\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\bar\{x\}\)\(y\_\{i\}\-\\bar\{y\}\)=\\sum\_\{i\}x\_\{i\}y\_\{i\}\-n\\bar\{x\}\\,\\bar\{y\},so we arrive at the identity

∑i,j=1n\(xi−xj\)​\(yi−yj\)\\displaystyle\\sum\_\{i,j=1\}^\{n\}\(x\_\{i\}\-x\_\{j\}\)\(y\_\{i\}\-y\_\{j\}\)=2​n​∑i=1n\(xi−x¯\)​\(yi−y¯\)\.\\displaystyle=2n\\sum\_\{i=1\}^\{n\}\(x\_\{i\}\-\\bar\{x\}\)\(y\_\{i\}\-\\bar\{y\}\)\.\(36\)Thus, a quadratic double sum over all pairs\(i,j\)\(i,j\)can be replaced by a linear single sum over mean\-centered terms\(xi−x¯\)​\(yi−y¯\)\(x\_\{i\}\-\\bar\{x\}\)\(y\_\{i\}\-\\bar\{y\}\)\. Substituting these expressions into the definition ofb^\\hat\{b\}and cancelling the common factor22yields the equivalent mean\-centered form used in the main text

b^\\displaystyle\\hat\{b\}=∑kMk​∑i=1Mk\(ni\(k\)−n¯\(k\)\)​\(ℓi\(k\)−ℓ¯\(k\)\)∑kMk​∑i=1Mk\(ni\(k\)−n¯\(k\)\)2\.\\displaystyle=\\frac\{\\displaystyle\\sum\_\{k\}M\_\{k\}\\sum\_\{i=1\}^\{M\_\{k\}\}\\bigl\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}^\{\(k\)\}\\bigr\)\\bigl\(\\ell^\{\(k\)\}\_\{i\}\-\\bar\{\\ell\}^\{\(k\)\}\\bigr\)\}\{\\displaystyle\\sum\_\{k\}M\_\{k\}\\sum\_\{i=1\}^\{M\_\{k\}\}\\bigl\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}^\{\(k\)\}\\bigr\)^\{2\}\}\.\(37\)

### A\.3Algorithm

Algorithm 1Estimate global length decaybbvia within\-prompt centeringInput:For each question

kk, candidate lengths

ni\(k\)n^\{\(k\)\}\_\{i\}and log\-likelihoods

ℓi\(k\)\\ell^\{\(k\)\}\_\{i\}
Output:Estimated length decay factor

b^\\hat\{b\}
num←0\\textit\{num\}\\leftarrow 0

den←0\\textit\{den\}\\leftarrow 0

foreach question

kkdo

M←M\\leftarrownumber of candidates for question

kk
if

M≤1M\\leq 1orn1\(k\)=n2\(k\)=⋯=nM\(k\)n\_\{1\}^\{\(k\)\}=n\_\{2\}^\{\(k\)\}=\\dots=n\_\{M\}^\{\(k\)\}then

continue\{No within\-question length variation\}

endifn¯←1M​∑i=1Mni\(k\)\\bar\{n\}\\leftarrow\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}n^\{\(k\)\}\_\{i\}ℓ¯←1M​∑i=1Mℓi\(k\)\\bar\{\\ell\}\\leftarrow\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\ell^\{\(k\)\}\_\{i\}cov←M​∑i=1M\(ni\(k\)−n¯\)​\(ℓi\(k\)−ℓ¯\)\\textit\{cov\}\\leftarrow M\\sum\_\{i=1\}^\{M\}\\left\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}\\right\)\\left\(\\ell^\{\(k\)\}\_\{i\}\-\\bar\{\\ell\}\\right\)var←M​∑i=1M\(ni\(k\)−n¯\)2\\textit\{var\}\\leftarrow M\\sum\_\{i=1\}^\{M\}\\left\(n^\{\(k\)\}\_\{i\}\-\\bar\{n\}\\right\)^\{2\}num←num\+cov\\textit\{num\}\\leftarrow\\textit\{num\}\+\\textit\{cov\}den←den\+var\\textit\{den\}\\leftarrow\\textit\{den\}\+\\textit\{var\}

endfor

if

den=0\\textit\{den\}=0then

b^←0\\hat\{b\}\\leftarrow 0\{No length variation across any question\}

else

b^←num/den\\hat\{b\}\\leftarrow\\textit\{num\}/\\textit\{den\}

endif

return

b^\\hat\{b\}

### A\.4Benchmark Details

##### ARC\.

The AI2 Reasoning Challenge \(ARC\) is a multiple\-choice question answering dataset of 7,787 science questions from U\.S\. grade\-school exams, split into an “Easy” and a more difficult “Challenge” subset\. It is designed to require non\-trivial scientific and commonsense reasoning rather than simple pattern matching or retrieval\.

##### HellaSwag\.

HellaSwag is an adversarial commonsense benchmark in which models must choose the most plausible continuation of a short narrative or instructional context from four options\. The roughly 70,000 examples are built from video captions and how\-to articles, with distractor endings crafted to look fluent yet be semantically implausible to humans\.

##### MMLU\.

The Massive Multitask Language Understanding \(MMLU\) benchmark aggregates 15,908 four\-way multiple\-choice questions across 57 academic subjects, spanning humanities, social and natural sciences, and professional domains\. It is widely used as a proxy for broad world knowledge and domain\-specific reasoning in language models\.

##### MMLU \(full\-text answer\)\.

In our*MMLU Full\-Text*variant, the underlying questions and answer options are unchanged, but the model is asked to output the full textual answer \(e\.g\., “Paris”\) rather than the option label \(“C”\)\. Accuracy is computed by exact\-match comparison between the generated answer and the corresponding gold option string\.

##### MMLU \(cloze\)\.

In the*MMLU cloze*variant, the multiple\-choice options are removed from the prompt and the problem is phrased as a short\-answer or fill\-in\-the\-blank question\. The model must produce the answer directly; at evaluation time we map the completion back onto the original MMLU answer key \(after minor normalization\) to determine correctness\.

##### SciQ\.

SciQ is a four\-option multiple\-choice benchmark of about 14,000 general science questions modeled on elementary and middle\-school exams\. Questions often come with associated source passages, and the dataset is commonly used as a mid\-difficulty science QA task between simple fact recall and more demanding benchmarks such as ARC\.

##### WinoGrande\.

WinoGrande is a large\-scale pronoun resolution benchmark inspired by the Winograd Schema Challenge\. Each example is a short sentence with an ambiguous pronoun and two candidate antecedents; the model must decide which option makes the sentence coherent, probing fine\-grained commonsense reasoning while controlling for dataset artifacts\.

### A\.5Benchmark Prompt Examples

The following synthetic examples illustrate the prompt formats used for each benchmark in our evaluation\. They are not taken from the original test sets\.

- •The quotation marks are not part of the text, they are there to highlight any potential white spacing
- •Templating is highlighted in green

#### A\.5\.1ARC

User:“Question:Which is an inherited characteristic on a horse?” Assistant:“Answer:” Options:

- •“ a long mane”
- •“ a steel shoe”
- •“ a missing tooth”
- •“ a bruised leg”

#### A\.5\.2ARC German

User:“Frage:Welcher Prozess im Kohlenstoffkreislauf dauert am längsten?” Assistant:“Antwort:” Options:

- •“ Emission von Abfall”
- •“ Atmung bei Tieren”
- •“ Photosynthese bei Pflanzen”
- •“ Bildung von fossilen Brennstoffen”

#### A\.5\.3HellaSwag

User:“Cutting the grass: A man is kneeling down on grass\. He” Options:

- •“ uses a polishing brush on a shoe\.”
- •“ has a heavy work out machine in his arms\.”
- •“ is using a green brush to clean off the grass\.”
- •“ is clipping the grass with large scissors\.”

#### A\.5\.4MMLU Full\-Text

User:“The following are multiple choice questions \(with possible answers\) about machine learning\. Answer with the full text of the correct answer\.

Question:Suppose your model is overfitting\. Which of the following is NOT a valid way to try and reduce the overfitting? Possible answers: \-Increase the amount of training data\. \-Improve the optimisation algorithm being used for error minimisation\. \-Decrease the model complexity\. \-Reduce the noise in the training data\.” Assistant:“Answer:” Options:

- •“ Increase the amount of training data\.”
- •“ Improve the optimisation algorithm being used for error minimisation\.”
- •“ Decrease the model complexity\.”
- •“ Reduce the noise in the training data\.”

#### A\.5\.5MMLU Cloze

User:“The following are multiple choice questions \(with possible answers\) about machine learning\. Answer with the full text of the correct answer\.

Question:Suppose your model is overfitting\. Which of the following is NOT a valid way to try and reduce the overfitting?” Assistant:“Answer:” Options:

- •“ Increase the amount of training data\.”
- •“ Improve the optimisation algorithm being used for error minimisation\.”
- •“ Decrease the model complexity\.”
- •“ Reduce the noise in the training data\.”

#### A\.5\.6OpenbookQA

User:“Climate change has sped up dramatically because” Options:

- •“ CO2 production has accelerated”
- •“ a rapid decline the production of carbon dioxide”
- •“ oxygen levels have spiked”
- •“ Fe has been present in an overabundance”

#### A\.5\.7SciQ

User:“Question:Anemia is a disease that affects what?” Assistant:“Answer:” Options:

- •“ Brain”
- •“ Heart”
- •“ Kidney”
- •“ Blood”

#### A\.5\.8WinoGrande

User:“Felicia ran out of shirts and borrowed one from Patricia, but” Options:

- •“ Felicia didn’t ask permission ahead of time\.”
- •“ Patricia didn’t ask permission ahead of time\.”

### A\.6Few\-shot Results

Table 5:Length bias for few\-shot standard \(unnormalized\) accuracy\. Entries are Kendall’sτ\\taubetween candidate length and score for each model–benchmark pair \(negative values indicate a tendency to favor shorter completions\)\.Table 6:Length bias for few\-shot byte\-normalized accuracy\. Entries are Kendall’sτ\\taubetween candidate length and score; positive values indicate a tendency to favor longer completions\.Table 7:Length bias for few\-shot Bayesian accuracy\. Entries are Kendall’sτ\\taubetween candidate length and Bayesian\-corrected score\.
### A\.7\(AN\)PMI Results

Table 8:Length bias for zero\-shot ANPMI scoring\. Entries are Kendall’sτ\\taubetween candidate length and ANPMI score\.Table 9:Length bias for zero\-shot PMI scoring\. Entries are Kendall’sτ\\taubetween candidate length and PMI score\.
### A\.8Variance Scaling

![Refer to caption](https://arxiv.org/html/2607.12767v1/x3.png)Figure 3:Standard deviation of conditional log\-likelihoodℓθ​\(c∣x\)/nbyte\\ell\_\{\\theta\}\(c\\mid x\)/n\_\{\\text\{byte\}\}versus completion byte length, aggregated over all models\. For readability, completion lengths are restricted to a maximum of 120 bytes and results are binned\.
### A\.9Token\-Level Length Bias Results

Table 10:Average absolute length bias\|τ\|¯\\overline\{\|\\tau\|\}per model and scoring rule when length is measured in tokens, aggregated over all benchmarks\.*Standard*is unnormalized accuracy, while*Norm*and*Bayes*use token length for normalization and length correction, respectively\. Lower is better\.