On the Position Bias of Large Language Models: A Comprehensive Study

# On the Position Bias of Large Language Models: A Comprehensive Study
Source: [https://arxiv.org/html/2610.22807](https://arxiv.org/html/2610.22807)
Yining Li Huayang Li Lemao Liu Taro Watanabe††thanks:Correspondence to Yining Li \([liyn9923@mails\.jlu\.edu\.cn](mailto:[email protected])\) and Huayang Li \([li\.huayang\.lh6@is\.naist\.jp](mailto:[email protected])\)\.

###### Abstract

Recent studies have shown that position bias remains a critical vulnerability in large language models \(LLMs\), undermining both the integrity of multiple\-choice evaluations and the reliability of retrieval\-augmented generation\. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English\-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored\. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion\-level to trillion\-level parameters, covering both multiple\-choice and multi\-document tasks in five languages\. Our experiments reveal that while position bias remains noticeable in small\-scale models and Non\-English languages, increasing model scale and employing standard Chain\-of\-Thought prompting significantly mitigate this issue\. Moreover, we demonstrate that emerging*thinking models*with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability\.

## 1Introduction

Large language models \(LLMs\) have achieved remarkable success across a wide spectrum of natural language processing tasks, demonstrating capabilities that often rival or surpass human performance[Brown et al\. \(2020\)](https://arxiv.org/html/2610.22807#bib.bib20);[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.22807#bib.bib18);[Comanici et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib17);[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib31);[Yang et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib41);[OpenAI \(2025\)](https://arxiv.org/html/2610.22807#bib.bib19)\. By scaling model parameters and training data, LLMs have evolved from simple statistical predictors into sophisticated systems capable of complex reasoning[Jaech et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib26);[Guo et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib27)and multilingual communication[Jiao et al\. \(2023\)](https://arxiv.org/html/2610.22807#bib.bib16);[Wu et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib15)\.

Despite these advancements, LLMs continue to exhibit counter\-intuitive vulnerabilities that undermine their reliability\. One prominent issue is*position bias*, where a model’s performance fluctuates significantly based on the sequential order of input information rather than its semantic content[McIlroy\-Young et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib37);[Yu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib38);[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib29);[Brown and McIlroy\-Young \(2025\)](https://arxiv.org/html/2610.22807#bib.bib2);[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib40)\. Prior research has identified two primary manifestations of this phenomenon\. In multiple\-choice tasks, models often show systematic preferences for specific option labels \(e\.g\., favoring option “\(B\)”\)[McIlroy\-Young et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib37)\. In knowledge\-intensive and long\-context tasks, models struggle to retrieve relevant documents placed in the middle of long input sequences, a phenomenon commonly referred to as*Lost\-in\-the\-Middle*[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42)\.

Position bias poses two major risks\. First, it threatens the validity of LLM evaluation\. Multiple\-choice question answering \(MCQA\) benchmarks[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2610.22807#bib.bib39);[Zellers et al\. \(2019\)](https://arxiv.org/html/2610.22807#bib.bib22);[Suzgun et al\. \(2023\)](https://arxiv.org/html/2610.22807#bib.bib36)remain a widely used standard for assessing general intelligence\. If models achieve high scores by exploiting positional preference rather than genuine understanding, reported progress may be misleading\. Second, severe position bias undermines real\-world applicability, particularly in retrieval\-augmented generation \(RAG\) and long\-context systems[Lewis et al\. \(2020b\)](https://arxiv.org/html/2610.22807#bib.bib24);[Ma et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib21);[Hsieh et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib23)\. If a model systematically favors information at specific positions, system accuracy and trustworthiness can be fundamentally compromised\. Consequently, a thorough assessment of position bias is critical for conducting rigorous evaluations and drawing reliable conclusions within the LLM research\.

However, a comprehensive picture of position bias in modern LLMs remains elusive, leaving it unclear how close current systems are to achieving permutation\-invariant behavior\. Existing literature exhibits several notable limitations\. \(1\)Limited model scope: many studies[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib29);[McIlroy\-Young et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib37);[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib40)focus on earlier model families, such as LLaMA\-2[Touvron et al\. \(2023\)](https://arxiv.org/html/2610.22807#bib.bib35)and GPT\-3\.5[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.22807#bib.bib18), and fail to capture the behavior of frontier models trained with more advanced architectures[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib31);[Chung et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib33)and higher\-quality data[Wang et al\. \(2025b\)](https://arxiv.org/html/2610.22807#bib.bib34)\. \(2\)Neglect of new inference paradigms: recent innovations, such as*thinking models*capable of long\-form reasoning \(i\.e\., System 2\-style inference\)[Guo et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib27);[Jaech et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib26), represent a paradigm shift in model behavior, yet their impact on position bias remains largely unexamined\. \(3\)English\-centric evaluation: most analyses are conducted exclusively in English, leaving typologically diverse languages \(e\.g\., Arabic, Chinese, and Japanese\) underexplored\.

In this work, we address these gaps through a systematic and large\-scale investigation\. We evaluate a diverse set of models spanning a wide range of scales, from Qwen3\-1\.7B to DeepSeek\-V4 1\.6T, across both multiple\-choice and multi\-document question answering tasks in five languages\. Our experiments yield several key findings\. First, we confirm that position bias is pronounced in small\-scale models \(i\.e\., fewer than3232B parameters\) and in non\-English languages\. However, increasing model size substantially alleviates this issue, with frontier models such as GPT\-5\.2 and DeepSeek\-V4 exhibiting markedly greater robustness\. Second, we show that prompting strategies play a critical role: even simple zero\-shot Chain\-of\-Thought prompting consistently reduces position bias\. Finally, we find that thinking models, such as DeepSeek\-V3\.2 with thinking mode enabled, are almost immune to position bias, suggesting that extended reasoning effectively decouples answer generation from positional bias\.

Our contributions are summarized as follows:

- •We present a comprehensive evaluation of position bias across both multiple\-choice and multi\-document settings, spanning five languages \(English, Chinese, Arabic, Spanish, and Japanese\) and model scales from 1\.7B parameters to frontier models such as DeepSeek\-V4 and GPT\-5\.2\.
- •We systematically analyze the factors influencing position bias, demonstrating that model scaling and standard Chain\-of\-Thought prompting significantly mitigate the issue\.
- •We show that the emerging class of thinking models, despite their computational cost, provides a robust solution to position bias, achieving near\-uniform performance regardless of information ordering and pointing toward a general way for robust LLM design\.

## 2Problem Formulation

In this section, we formally define the task of quantifying position bias in Large Language Models \(LLMs\)\.

### 2\.1General Definition of Position Bias

Given an input contextxxsampled from a dataset𝒟\\mathcal\{D\}, the LLMfθf\_\{\\theta\}generates an answer textyyfor it\. We consider tasks where the input contextxxis composed of a queryqqand a set ofKKinformation segments𝒮=\{s1,s2,…,sK\}\\mathcal\{S\}=\\\{s\_\{1\},s\_\{2\},\\dots,s\_\{K\}\\\}\. These segments contain the necessary information to respond toqq, e\.g\., retrieved documents[Lewis et al\. \(2020b\)](https://arxiv.org/html/2610.22807#bib.bib24)or answer options[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2610.22807#bib.bib39)\. Letπ\\pibe a permutation over the indices\{1,…,K\}\\\{1,\\dots,K\\\}\. The input context under permutationπ\\piis defined as:

xπ=ϕ⁡\(q,sπ⁡\(1\),sπ⁡\(2\),…,sπ⁡\(K\)\),x\_\{\\pi\}=\\phi\(q,s\_\{\\pi\(1\)\},s\_\{\\pi\(2\)\},\\dots,s\_\{\\pi\(K\)\}\),whereϕ⁡\(⋅\)\\phi\(\\cdot\)is a prompt function that linearizes these segments into a sequence\. Ideally, an LLM should satisfy permutation invariance with respect to the semantic content of𝒮\\mathcal\{S\}\. That is, performance offϕf\_\{\\phi\}should remain stable regardless of the order of the segments[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42);[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30):

∑x∼𝒟M⁡\(f⁡\(xπ\),y∗\)≈∑x∼𝒟M⁡\(f⁡\(xπ′\),y∗\),\\sum\_\{x\\sim\\mathcal\{D\}\}M\\big\(f\(x\_\{\\pi\}\),y^\{\*\}\\big\)\\approx\\sum\_\{x\\sim\\mathcal\{D\}\}M\\big\(f\(x\_\{\\pi^\{\\prime\}\}\),y^\{\*\}\\big\),\(1\)whereMMis the evaluation metric, e\.g\., exact\-match \(EM\) score,y∗y^\{\*\}is the golden truth for the inputxx, andπ\\piandπ′\\pi^\{\\prime\}are two different permutations for𝒮\\mathcal\{S\}\.

Theposition biasoccurs when the model’s performance fluctuates significantly based on the specific positionkkassigned to the ground\-truth segments∗s^\{\*\}within the linearized sequence\.

### 2\.2Multi\-Choice Tasks

Many fundamental evaluation for benchmarking the performance of LLMs are in the multi\-choice form[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2610.22807#bib.bib39);[Zellers et al\. \(2019\)](https://arxiv.org/html/2610.22807#bib.bib22)\. If most LLMs face a severe position bias issue in these tasks, a systematical reshaping for LLM evaluation may be necessary\.

In the multi\-choice \(MC\) setting, the LLM is presented with a questionqqalong with a set of candidate answer options𝒮MC=\{s1,…,sK\}\\mathcal\{S\}\_\{\\textrm\{MC\}\}=\\\{s\_\{1\},\\dots,s\_\{K\}\\\}, typically labeled from “\(A\)” to “\(D\)”, where position bias manifests as a preference for certain option orderings independent of answer content\. Lets∗s^\{\*\}denote the ground option\. To analyze position bias, we shift the positionkkofs∗s^\{\*\}across all positions in\{1,…,K\}\\\{1,\\dots,K\\\}, yielding a permutation set𝒫MC=\{π1,…,πK\}\\mathcal\{P\}\_\{\\textrm\{MC\}\}=\\\{\\pi^\{1\},\\dots,\\pi^\{K\}\\\}, whereπk\\pi^\{k\}is a permutation of\{1,…,K\}\\\{1,\\dots,K\\\}with the golden answer at thekk\-th position, i\.e\.,sk=s∗s\_\{k\}=s^\{\*\}\.

### 2\.3Multi\-Doc Tasks

Correctly using the information of documents in the context is vital for many LLM applications, e\.g\., retrieval\-augmented generation[Lewis et al\. \(2020b\)](https://arxiv.org/html/2610.22807#bib.bib24)and long\-context modeling[Hsieh et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib23)\. Investigating the position bias in this kind of tasks is also important\.

Different from the the multi\-choice tasks, where the options in𝒮MC\\mathcal\{S\}\_\{\\mathrm\{MC\}\}are generally short and close to the question, the multi\-doc tasks integrate long and independent documents into the context\. In this setting, the LLM is presented with a set of long documents𝒮MD=\{s1,…,sK\}\\mathcal\{S\}\_\{\\textrm\{MD\}\}=\\\{s\_\{1\},\\dots,s\_\{K\}\\\}before the questionqq, where only one golden documents∗∈𝒮s^\{\*\}\\in\\mathcal\{S\}contain the answer forqq\. We shift the positionkkofs∗s^\{\*\}across sampled positions similarly, resulting a permutation setPMD\{P\}\_\{\\textrm\{MD\}\}\.

### 2\.4Evaluation Metrics

The main evaluation metricMMwe used to measure the performance of modelfϕf\_\{\\phi\}is the exact\-match \(EM\)[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42);[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[Hsieh et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib23)under a specific permutationπ\\pi\.

eπ,θ=1\|𝒟\|​∑x∼𝒟M⁡\(f⁡\(xπ\),y∗\)\.e\_\{\\pi,\\theta\}=\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{x\\sim\\mathcal\{D\}\}M\\big\(f\(x\_\{\\pi\}\),y^\{\*\}\\big\)\.\(2\)We also report the largest performance difference offϕf\_\{\\phi\}is at different positions:

Δθ=maxπ∈𝒫⁡eπ,θ−minπ∈𝒫⁡eπ,θ,\\Delta\_\{\\theta\}=\\max\_\{\\pi\\in\\mathcal\{P\}\}e\_\{\\pi,\\theta\}\-\\min\_\{\\pi\\in\\mathcal\{P\}\}e\_\{\\pi,\\theta\},\(3\)where𝒫\\mathcal\{P\}is the permutation set\.

## 3Multi\-Choice Question Answering

\(a\)0\-shot DA\(b\)0\-shot CoT\(c\)5\-shot DA\(d\)5\-shot CoT
Figure 1:Position bias on English Multi\-Choice tasks\. Results are reported on MMLU benchmark[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2610.22807#bib.bib39)using non\-thinking models\. The label of golden answer is shifted from “\(A\)” to “\(D\)” for all data, and the options are positioned in alphabetical order of the labels, i\.e\., “\(A\)” appearing first and “\(D\)” last\.### 3\.1Setup

The MMLU dataset111[https://huggingface\.co/datasets/cais/mmlu](https://huggingface.co/datasets/cais/mmlu)[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2610.22807#bib.bib39), which comprises 57 subtasks, is widely used to benchmark the general intelligence of large language models \(LLMs\)\. Each question includes four answer options labeled “\(A\)” through “\(D\)”, only one of which is correct\. In the original benchmark, the position of the correct answer is randomly shuffled to ensure an even distribution across all option positions\.

In contrast, in our experiments, we evaluate the position bias of LLMs by forcing the correct answer to appear at a fixed position \(one of “\(A\)” to “\(D\)”\) for all questions\. The re\-labeled option list is positioned in alphabetical order of the labels, with option “\(A\)” appearing first and “\(D\)” last\. We then visualize and compare the model’s performance across runs with different fixed answer positions\. This procedure yields14,052×4=56,16814\{,\}052\\times 4=56\{,\}168test examples per run on MMLU benchmark, covering all 57 subtasks\.

In addition to evaluation on the MMLU dataset in English, we are also curious about the position bias in non\-English settings\. Therefore, we leverage the MMMLU222[https://huggingface\.co/datasets/openai/MMMLU](https://huggingface.co/datasets/openai/MMMLU), which is parallel to MMLU, to evaluate on four additional languages: Spanish \(alphabetic and typologically closer to English\), Arabic \(right\-to\-left script\), and Chinese and Japanese \(representing logographic and mixed\-script systems\)\. We sample 25 data points in each sub\-task, resulting25×57×4=5,70025\\times 57\\times 4=5,700test examples333Preliminary study on English data show that evaluation on the down\-sampled dataset yields consistent observations as the those on full dataset\.for each Non\-English language\.

In our experiments, we consider four commonly used evaluation settings:

0\-shot DAThe model is prompted to directly answer \(DA\) by selecting an option label\.

5\-shot DAWe include the 5\-shot DA examples provided by[Chung et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib33)in the prompt\.

0\-shot CoTWe add the text “Let’s think step by step” into the prompt to elicit Chain\-of\-Thought \(CoT\) reasoning[Wei et al\. \(2022\)](https://arxiv.org/html/2610.22807#bib.bib32)before producing the final answer\.

5\-shot CoTWe include 5\-shot CoT examples in the prompt to enhance the reasoning process, following[Chung et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib33)\.

To systematically assess the severity of position bias, we conduct experiments on models spanning a wide range of scales: Qwen3\-1\.7B, Qwen3\-8B, Qwen3\-32B, Qwen3\-30B\-MoE, Qwen3\-235B\-MoE[Yang et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib41), DeepSeek\-V3\.2 with 685B parameters[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib31), DeepSeek\-V4 1\.6T and GPT\-5\.2444[https://platform\.openai\.com/docs/models/gpt\-5\.2](https://platform.openai.com/docs/models/gpt-5.2)\. We use the official APIs provided by Aliyun555[https://modelstudio\.console\.alibabacloud\.com/](https://modelstudio.console.alibabacloud.com/), DeepSeek666[https://api\-docs\.deepseek\.com](https://api-docs.deepseek.com/), and OpenAI777[https://platform\.openai\.com/docs/models/gpt\-5\.2](https://platform.openai.com/docs/models/gpt-5.2)to access Qwen3, DeepSeek V3\.2 and V4, and GPT\-5\.2, respectively\. For all models, we set the temperature to 0 during generation and evaluate performance using the exact\-match score between the predicted and golden\-truth option labels\.

Table 1:The performance differenceΔθ\\Delta\_\{\\theta\}of LLMs on English Multi\-Choice tasks\. Results are reported on MMLU using non\-thinking models\.
### 3\.2Main Results

As shown in Figure[1](https://arxiv.org/html/2610.22807#S3.F1), we compare model performance when the golden truth is forced to appear in only one of the four options\. We also report the largest performance difference \(Equation[3](https://arxiv.org/html/2610.22807#S2.E3)\) for each model in Table[1](https://arxiv.org/html/2610.22807#S3.T1)\.

#### Impact of Model Scale

In many previous works, similar evaluations have mainly been conducted on a limited range of models[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[Brown and McIlroy\-Young \(2025\)](https://arxiv.org/html/2610.22807#bib.bib2);[Reif and Schwartz \(2024\)](https://arxiv.org/html/2610.22807#bib.bib4)\. However, it is important to understand how severe the position bias issue is across models of different scales\.

We observe that the small\-scale Qwen3\-1\.7B fluctuates significantly when the position of the golden answer changes\. For example, the EM score drops by 56\.9 points when the position changes from “\(A\)” to “\(B\)” in Figure[1\(c\)](https://arxiv.org/html/2610.22807#S3.F1.sf3)\. This non\-uniform performance is noticeably alleviated in mid\-scale models, such as Qwen3\-32B\. Moreover, the two largest models, i\.e\., Deepseek\-V4\-Pro and GPT5\.2, achieve much more stable performance across all four positions\. These observations demonstrate that model capacity is strongly related to the severity of position bias; specifically, scaling up model size can significantly mitigate this issue\.

\(a\)Chinese\(b\)Arabic\(c\)Japanese\(d\)Spanish
Figure 2:Position bias on Multilingual Multi\-Choice tasks\. Results are reported on MMMLU benchmark using non\-thinking models and 0\-shot CoT template\. The label of golden answer is shifted from “\(A\)” to “\(D\)” for all data, and the options are positioned in alphabetical order of the labels, i\.e\., “\(A\)” appearing first and “\(D\)” last\.
#### Impact of Few\-Shot Examples

Few\-shot examples are widely used to guide models toward generating higher\-quality answers\. However, the few\-shot examples themselves may introduce additional position bias; for instance, the option labels of golden answers may not be evenly distributed[Chung et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib33)\.

While most models improve overall in the 5\-shot DA setting \(Figures[1\(a\)](https://arxiv.org/html/2610.22807#S3.F1.sf1),[1\(c\)](https://arxiv.org/html/2610.22807#S3.F1.sf3)\), Qwen3\-1\.7B, Qwen3\-30B\-MoE, and even Deepseek\-V3\.2 exhibit more severe position bias when using the 5\-shot DA examples provided by FLAN[Chung et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib33)\. This suggests that position bias may not originate solely from the model, but can also be influenced by the information provided in the context\. But, the influence of few\-shot examples on frontier models with trillion parameters, such as Deepseek V4 and GPT\-5\.2, is marginal\.

#### Impact of CoT

Chain\-of\-Thought \(CoT\) prompting is widely used in practice and may alleviate position bias by encouraging the model to reason more thoroughly before answering\.

As shown in Figure[1\(b\)](https://arxiv.org/html/2610.22807#S3.F1.sf2), simply adding the phrase “Let’s think step by step” to the prompt significantly reduces position bias for almost all models, resulting in a more uniform performance distribution across different option labels\. In Figure[1\(d\)](https://arxiv.org/html/2610.22807#S3.F1.sf4), we observe consistently low\-level position bias when 5\-shot examples with CoT are provided\. However, the 5\-shot CoT examples do not improve overall performance, indicating that the modern LLMs have learned high\-quality reasoning abilities\.

Table 2:The performance differenceΔθ\\Delta\_\{\\theta\}of LLMs on Multilingual Multi\-Choice tasks\. Results are reported on MMMLU benchmark using non\- thinking models and 0\-shot CoT prompt\.

### 3\.3Results on Multilingual Data

Prior work on position bias has focused primarily on English[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib29);[Tripathi et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib28);[Tian et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib14), leaving its severity in other languages underexplored\. We therefore examine four languages: Spanish \(similar to English\), Arabic \(right\-to\-left order\), and Chinese and Japanese \(logographic and mixed\-script systems\)\.

As shown in Figure[2](https://arxiv.org/html/2610.22807#S3.F2)and Table[2](https://arxiv.org/html/2610.22807#S3.T2), even with the 0\-shot CoT method, many LLMs exhibit severe position bias on non\-English data\. Specifically, the Qwen3\-30B\-MoE model, which achieves almost uniform performance in English \(Figure[1\(b\)](https://arxiv.org/html/2610.22807#S3.F1.sf2)\), shows noticeable performance fluctuations in Arabic, Japanese, and Spanish\. Even the Deepseek\-V3\.2 and GPT\-5\.2 models demonstrate more pronounced position bias in non\-English languages\.

### 3\.4Results on the Thinking Model

The*thinking model*[Guo et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib27);[Jaech et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib26), which generates long\-form reasoning trajectory before answering, has achieved considerable success\. Since 0\-shot CoT has shown positive effects on English data, we use the thinking model to evaluate whether enhanced CoT can further alleviate the position bias issue\.

As shown in Figure[3](https://arxiv.org/html/2610.22807#S3.F3), Deepseek\-V3\.2\-Thinking exhibits less position bias than its standard counterpart\. Since Deepseek\-V4 shows nearly uniform performance even without reasoning, there has little room to further reduce the position bias with thinking mode\. Interestingly, DeepSeek V4 achieves a lower EM score in thinking mode compared to non\-thinking mode\. We hypothesize that this is driven by the "over\-thinking" phenomenon on simple questions[Aggarwal et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib13)\.

\(a\)Chinese\(b\)Arabic\(c\)Spanish\(d\)Japanese
Figure 3:Position bias on Multilingual Multi\-Choice tasks\. We use the Deepseek\-V3\.2, Deepseek\-V4\-Pro and the 0\-shot CoT tempalte for evaluation\.

## 4Multi\-Document Tasks

The*Lost\-in\-the\-Middle*\(LITM\) effect[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42)describes the counter\-intuitive performance degradation observed when relevant document is placed in the middle of a prompt\. In this section, we comprehensively evaluate on modern LLMs to determine whether this phenomenon remains a significant challenge\.

### 4\.1Setup

\(a\)English\(b\)Chinese\(c\)Arabic\(d\)Spanish
Figure 4:Position bias on Multi\-Document QA\.We use the MLQA dataset888[https://huggingface\.co/datasets/facebook/mlqa](https://huggingface.co/datasets/facebook/mlqa)[Lewis et al\. \(2020a\)](https://arxiv.org/html/2610.22807#bib.bib25)to analyze whether the effect persists cross\-linguistically\. Following the setup of the Lost in the Middle dataset999[https://github\.com/nelson\-liu/lost\-in\-the\-middle](https://github.com/nelson-liu/lost-in-the-middle)provided by[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42), we construct N=20 documents per question with golden documents placed at positions 0, 4, 9, 14, and 19, sampling 500 instances per language \(2,500 examples per run\)\. Details of the dataset construction are provided in Appendix[B](https://arxiv.org/html/2610.22807#A2)\.

Following the experiments in Section[3](https://arxiv.org/html/2610.22807#S3), we evaluate models across a variety of scales: Qwen3\-1\.7B, Qwen3\-8B, Qwen3\-32B, Qwen3\-30B\-MoE, Qwen3\-235B\-MoE[Yang et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib41), DeepSeek\-V3\.2,[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib31), DeepSeek\-V4 1\.6T, and GPT\-5\.2\. Following[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42), we use Exact Match \(EM\) as our primary evaluation metric\.

### 4\.2Main Results

#### Impact of Model Scale

As shown in Table[3](https://arxiv.org/html/2610.22807#S4.T3), we calculate the largest performance difference \(Δθ\\Delta\_\{\\theta\}\), as defined in Eq\.[3](https://arxiv.org/html/2610.22807#S2.E3), across all positions for each model\. The overall trend indicates thatΔθ\\Delta\_\{\\theta\}values decrease as the number of model parameters increases, although a few outliers exist\. Many small\-scale models still exhibit strong position bias; for instance, the performance differences for Qwen3\-1\.7B and Qwen3\-8B are 9\.8 and 6\.2 points in Spanish, respectively\. In contrast, the performance differences for large\-scale models, such as Qwen3\-235B\-MoE, DeepSeek V3\.2 and V4, and GPT\-5\.2, are maintained within 5 points across most languages other than Arabic\.

Table 3:The performance differenceΔθ\\Delta\_\{\\theta\}of LLMs on multi\-document tasks\. Results are reported on non\-thinking models\.
#### Impact of multilingual data

In our setting, various models exhibit highly consistent performance on English datasets\. However, in non\-English contexts, the models display significant disparity\. According to Figure 4, when the position of the golden document is fixed, the performance gap between most models \(with the exception of DeepSeek\-V3\.2\) remains within a narrow margin of 8 percentage points\. Visually, this is reflected by the high degree of overlap among the different curves in the English plots\. The curves in Figure 4 for non\-English data are notably more scattered, indicating that multilingual capabilities vary significantly across different models\.For additional results and detailed analysis, see Appendix[A](https://arxiv.org/html/2610.22807#A1)\.

### 4\.3Results on Reasoning Models

Inspired by the low position bias achieved by reasoning models[Guo et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib27);[Liu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib31);[Jaech et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib26), we further investigate whether such "thinking" models can alleviate position bias in multi\-document QA\. As shown in Figure[5](https://arxiv.org/html/2610.22807#S4.F5), the reasoning models exhibit more uniform performance patterns across positions compared to non\-reasoning models\. This observation suggests that allocating more tokens for explicit reasoning is an effective general approach to mitigating the position bias issue\. For additional analysis, see Appendix[C](https://arxiv.org/html/2610.22807#A3)\.

\(a\)English\(b\)Chinese\(c\)Arabic\(d\)Spanish
Figure 5:Position bias on Multi\-Document tasks\. We use the Deepseek\-V3\.2 and Deepseek\-V4\-Pro for evaluation\.

## 5Related Work

The issue of*position bias*was first discussed in the context of search engines, where users may be influenced by the positions of documents regardless of their content relevance[Joachims et al\. \(2017\)](https://arxiv.org/html/2610.22807#bib.bib12);[Aslanyan and Porwal \(2019\)](https://arxiv.org/html/2610.22807#bib.bib11)\.

Interestingly,[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42)found similar behavior in large language models \(LLMs\)\. Concretely, in retrieval\-augmented generation[Lewis et al\. \(2020b\)](https://arxiv.org/html/2610.22807#bib.bib24), the order of retrieved documents, which is permutation\-invariant to humans, may significantly bias model outputs\. This bias can lead to a performance drop of more than 20 points when the golden document is placed in the middle of the context for GPT\-3\.5[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.22807#bib.bib18)\. Many efforts have been devoted to understanding this phenomenon[Salvatore et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib9);[Wu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib8);[Hsieh et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib6);[Tian et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib14)and alleviating it[Wang et al\. \(2025a\)](https://arxiv.org/html/2610.22807#bib.bib29);[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[He et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib10);[Yu et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib38)\.

Recent literature has shown that position bias is not only in the multi\-document setting but also in multi\-choice question answering\. Specifically, permutations of answer options can significantly affect model performance without regard to content[Wei et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib3);[Wang et al\. \(2025c\)](https://arxiv.org/html/2610.22807#bib.bib30);[Dimino et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib7);[Zhao et al\. \(2024\)](https://arxiv.org/html/2610.22807#bib.bib5);[Reif and Schwartz \(2024\)](https://arxiv.org/html/2610.22807#bib.bib4);[Brown and McIlroy\-Young \(2025\)](https://arxiv.org/html/2610.22807#bib.bib2);[Tripathi et al\. \(2025\)](https://arxiv.org/html/2610.22807#bib.bib28)\.[Tian et al\. \(2025b\)](https://arxiv.org/html/2610.22807#bib.bib1)further find that this issue also exists in multimodal language models\.

The large performance gaps across different permutations observed in previous work are counter\-intuitive and motivate our study\. First, multi\-choice evaluation is widely used to benchmark the intelligence of LLMs\. If position bias remains severe, the community may need to rethink evaluation protocols\. Second, since the multi\-document setting is widely adopted in many LLM applications[Lewis et al\. \(2020b\)](https://arxiv.org/html/2610.22807#bib.bib24);[Hsieh et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib23), severe position bias can have a strong negative impact on system accuracy and trustworthiness\. These concerns motivate us to conduct a comprehensive study of position bias across models, rather than mitigating the issue in limited settings, such as specific models like LLaMA\-2[Touvron et al\. \(2023\)](https://arxiv.org/html/2610.22807#bib.bib35)or GPT\-3\.5[Ouyang et al\. \(2022\)](https://arxiv.org/html/2610.22807#bib.bib18)\.

## 6Conclusions

This work conducts a comprehensive evaluation of modern LLMs to assess the severity of position bias in large language models\. First, we confirm that small\-scale LLMs, i\.e\., models with fewer than3232B parameters, generally exhibit more severe position bias across all the tested languages\. In contrast, scaling up model size and allowing models to reason with more tokens significantly alleviates this issue\. We also find that although few\-shot examples can improve overall performance, enforcing a uniform distribution of option labels in few\-shot demonstrations may exacerbate position bias on LLMs less than 1T parameters\. Additionally, when modern*thinking*mode \(i\.e\., generating a long reasoning chain before answering\) is enabled, LLMs achieve nearly uniform performance that is largely unaffected by the position of information segments\.

We believe these findings provide many insights for the LLM community\. First, evaluation protocols for small\-scale models should be carefully designed, as severe answer\-option bias may fail to reliably reflect model intelligence\. Second, although scaling model size and test\-time computation helps mitigate position bias, efficient solutions remain appealing in computation\-constrained scenarios\. For a detailed discussion of the limitations, please refer to Appendix[D](https://arxiv.org/html/2610.22807#A4)\. Ethical considerations are also addressed in Appendix[E](https://arxiv.org/html/2610.22807#A5)\.

## References

- Aggarwalet al\.\(2025\)P\. Aggarwal, S\. Kim, J\. Lanchantin, S\. Welleck, J\. Weston, I\. Kulikov, and S\. SahaOptimalthinkingbench: evaluating over and underthinking in llms\.arXiv preprint arXiv:2508\.13141\.Cited by:[§3\.4](https://arxiv.org/html/2610.22807#S3.SS4.p2.1)\.
- Aslanyan and Porwal \(2019\)G\. Aslanyan and U\. PorwalPosition bias estimation for unbiased learning\-to\-rank in ecommerce search\.InString Processing and Information Retrieval: 26th International Symposium, SPIRE 2019, Segovia, Spain, October 7–9, 2019, Proceedings,Berlin, Heidelberg,pp\. 47–64\.External Links:ISBN 978\-3\-030\-32685\-2,[Link](https://doi.org/10.1007/978-3-030-32686-9_4),[Document](https://dx.doi.org/10.1007/978-3-030-32686-9%5F4)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p1.1)\.
- Brown and McIlroy\-Young \(2025\)K\. Brown and R\. McIlroy\-YoungOrder independence with finetuning\.InICLR 2025 Workshop on Bidirectional Human\-AI Alignment,External Links:[Link](https://openreview.net/forum?id=08E6XX0Yen)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§3\.2](https://arxiv.org/html/2610.22807#S3.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1)\.
- Chunget al\.\(2024\)H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[item 5\-shot DA](https://arxiv.org/html/2610.22807#S3.I1.ix2.p1.1),[item 5\-shot CoT](https://arxiv.org/html/2610.22807#S3.I1.ix4.p1.1),[§3\.2](https://arxiv.org/html/2610.22807#S3.SS2.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.22807#S3.SS2.SSS0.Px2.p2.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1)\.
- Diminoet al\.\(2025\)F\. Dimino, K\. Saxena, B\. Sarmah, and S\. PasqualiTracing positional bias in financial decision\-making: mechanistic insights from qwen2\. 5\.InProceedings of the 6th ACM International Conference on AI in Finance,pp\. 96–104\.Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§3\.4](https://arxiv.org/html/2610.22807#S3.SS4.p1.1),[§4\.3](https://arxiv.org/html/2610.22807#S4.SS3.p1.1)\.
- Heet al\.\(2024\)J\. He, K\. Pan, X\. Dong, Z\. Song, Y\. Liu, Q\. Sun, Y\. Liang, H\. Wang, E\. Zhang, and J\. ZhangNever lost in the middle: mastering long\-context question answering with position\-agnostic decompositional training\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 13628–13642\.External Links:[Link](https://aclanthology.org/2024.acl-long.736/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.736)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22807#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2610.22807#S2.SS2.p1.1),[Figure 1](https://arxiv.org/html/2610.22807#S3.F1),[§3\.1](https://arxiv.org/html/2610.22807#S3.SS1.p1.1)\.
- Hsiehet al\.\(2024a\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.22807#S2.SS3.p1.1),[§2\.4](https://arxiv.org/html/2610.22807#S2.SS4.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p4.1)\.
- Hsiehet al\.\(2024b\)C\. Hsieh, Y\. Chuang, C\. Li, Z\. Wang, L\. Le, A\. Kumar, J\. Glass, A\. Ratner, C\. Lee, R\. Krishna, and T\. PfisterFound in the middle: calibrating positional attention bias improves long context utilization\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14982–14995\.External Links:[Link](https://aclanthology.org/2024.findings-acl.890/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.890)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Jaechet al\.\(2024\)A\. Jaech, A\. Kalai, A\. Lerer, A\. Richardson, A\. El\-Kishky, A\. Low, A\. Helyar, A\. Madry, A\. Beutel, A\. Carney,et al\.Openai o1 system card\.arXiv preprint arXiv:2412\.16720\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§3\.4](https://arxiv.org/html/2610.22807#S3.SS4.p1.1),[§4\.3](https://arxiv.org/html/2610.22807#S4.SS3.p1.1)\.
- Jiaoet al\.\(2023\)W\. Jiao, W\. Wang, J\. Huang, X\. Wang, S\. Shi, and Z\. TuIs chatgpt a good translator? yes with gpt\-4 as the engine\.arXiv preprint arXiv:2301\.08745\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1)\.
- Joachimset al\.\(2017\)T\. Joachims, A\. Swaminathan, and T\. SchnabelUnbiased learning\-to\-rank with biased feedback\.InProceedings of the tenth ACM international conference on web search and data mining,pp\. 781–789\.Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p1.1)\.
- Lewiset al\.\(2020a\)P\. Lewis, B\. Oguz, R\. Rinott, S\. Riedel, and H\. SchwenkMLQA: evaluating cross\-lingual extractive question answering\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 7315–7330\.Cited by:[§4\.1](https://arxiv.org/html/2610.22807#S4.SS1.p1.1)\.
- Lewiset al\.\(2020b\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1),[§2\.1](https://arxiv.org/html/2610.22807#S2.SS1.p1.1),[§2\.3](https://arxiv.org/html/2610.22807#S2.SS3.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1),[§5](https://arxiv.org/html/2610.22807#S5.p4.1)\.
- Liuet al\.\(2025\)A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.22807#S3.SS1.p5.1),[§4\.1](https://arxiv.org/html/2610.22807#S4.SS1.p2.1),[§4\.3](https://arxiv.org/html/2610.22807#S4.SS3.p1.1)\.
- Liuet al\.\(2024a\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[Appendix A](https://arxiv.org/html/2610.22807#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1)\.
- Liuet al\.\(2024b\)N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. LiangLost in the Middle: How Language Models Use Long Contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[Appendix B](https://arxiv.org/html/2610.22807#A2.p1.1),[Appendix B](https://arxiv.org/html/2610.22807#A2.p2.1),[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.22807#S2.SS1.p1.2),[§2\.4](https://arxiv.org/html/2610.22807#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2610.22807#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.22807#S4.SS1.p2.1),[§4](https://arxiv.org/html/2610.22807#S4.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Maet al\.\(2025\)D\. Ma, Y\. Wang, and T\. LanBlock\-attention for efficient prefilling\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=7zNYY1E2fq)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1)\.
- McIlroy\-Younget al\.\(2024\)R\. McIlroy\-Young, K\. Brown, C\. Olson, L\. Zhang, and C\. DworkOrder\-independence without fine tuning\.Advances in Neural Information Processing Systems37,pp\. 72818–72839\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1)\.
- OpenAI \(2025\)OpenAIGPT\-5 system card\.Note:Accessed: 2026\-01\-04External Links:[Link](https://cdn.openai.com/gpt-5-system-card.pdf)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1),[§5](https://arxiv.org/html/2610.22807#S5.p4.1)\.
- Reif and Schwartz \(2024\)Y\. Reif and R\. SchwartzBeyond performance: quantifying and mitigating label bias in LLMs\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 6784–6798\.External Links:[Link](https://aclanthology.org/2024.naacl-long.378/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.378)Cited by:[§3\.2](https://arxiv.org/html/2610.22807#S3.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Salvatoreet al\.\(2025\)N\. Salvatore, H\. Wang, and Q\. ZhangLost in the middle: an emergent property from information retrieval demands in llms\.arXiv preprint arXiv:2510\.10276\.Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou,et al\.Challenging big\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 13003–13051\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1)\.
- Tianet al\.\(2025a\)R\. Tian, Y\. Li, Y\. Fu, S\. Deng, Q\. Luo, C\. Qian, S\. Wang, X\. Cong, Z\. Zhang, Y\. Wu, Y\. Lin, H\. Wang, and X\. LiuDistance between relevant information pieces causes bias in long\-context LLMs\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 521–533\.External Links:[Link](https://aclanthology.org/2025.findings-acl.28/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.28),ISBN 979\-8\-89176\-256\-5Cited by:[§3\.3](https://arxiv.org/html/2610.22807#S3.SS3.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Tianet al\.\(2025b\)X\. Tian, S\. Zou, Z\. Yang, and J\. ZhangIdentifying and mitigating position bias of multi\-image vision\-language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 10599–10609\.Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§5](https://arxiv.org/html/2610.22807#S5.p4.1)\.
- Tripathiet al\.\(2025\)T\. Tripathi, M\. Wadhwa, G\. Durrett, and S\. NiekumPairwise or pointwise? evaluating feedback protocols for bias in LLM\-based evaluation\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=uyX5Vnow3U)Cited by:[§3\.3](https://arxiv.org/html/2610.22807#S3.SS3.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Wanget al\.\(2025a\)Y\. Wang, F\. Xiong, Y\. Wang, L\. Li, X\. Chu, and D\. D\. ZengPOSITION BIAS MITIGATES POSITION BIAS: mitigate position bias through inter\-position knowledge distillation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 1495–1512\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.78/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.78),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§1](https://arxiv.org/html/2610.22807#S1.p4.1),[§3\.3](https://arxiv.org/html/2610.22807#S3.SS3.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Wanget al\.\(2025b\)Z\. Wang, F\. Zhou, X\. Li, and P\. LiuOctothinker: mid\-training incentivizes reinforcement learning scaling\.arXiv preprint arXiv:2506\.20512\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p4.1)\.
- Wanget al\.\(2025c\)Z\. Wang, H\. Zhang, X\. Li, K\. Huang, C\. Han, S\. Ji, S\. M\. Kakade, H\. Peng, and H\. JiEliminating position bias of language models: a mechanistic approach\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fvkElsJOsN)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.22807#S2.SS1.p1.2),[§2\.4](https://arxiv.org/html/2610.22807#S2.SS4.p1.1),[§3\.2](https://arxiv.org/html/2610.22807#S3.SS2.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2610.22807#S3.SS3.p1.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1),[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[item 0\-shot CoT](https://arxiv.org/html/2610.22807#S3.I1.ix3.p1.1)\.
- Weiet al\.\(2024\)S\. Wei, C\. Wu, H\. Huang, and H\. ChenUnveiling selection biases: exploring order and token sensitivity in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5598–5621\.External Links:[Link](https://aclanthology.org/2024.findings-acl.333/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.333)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.
- Wuet al\.\(2024\)M\. Wu, J\. Xu, and L\. WangTransAgents: build your translation company with language agents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,D\. I\. Hernandez Farias, T\. Hope, and M\. Li \(Eds\.\),Miami, Florida, USA,pp\. 131–141\.External Links:[Link](https://aclanthology.org/2024.emnlp-demo.14/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-demo.14)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1)\.
- Wuet al\.\(2025\)X\. Wu, Y\. Wang, S\. Jegelka, and A\. JadbabaieOn the emergence of position bias in transformers\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=YufVk7I6Ii)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p1.1),[§3\.1](https://arxiv.org/html/2610.22807#S3.SS1.p5.1),[§4\.1](https://arxiv.org/html/2610.22807#S4.SS1.p2.1)\.
- Yuet al\.\(2025\)Y\. Yu, H\. Jiang, X\. Luo, Q\. Wu, C\. Lin, D\. Li, Y\. Yang, Y\. Huang, and L\. QiuMitigate position bias in LLMs via scaling a single hidden states channel\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 6092–6111\.External Links:[Link](https://aclanthology.org/2025.findings-acl.316/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.316),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p2.1),[§5](https://arxiv.org/html/2610.22807#S5.p2.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§1](https://arxiv.org/html/2610.22807#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.22807#S2.SS2.p1.1)\.
- Zhaoet al\.\(2024\)X\. Zhao, K\. Wang, and W\. PengMeasuring the inconsistency of large language models in preferential ranking\.InProceedings of the 1st Workshop on Towards Knowledgeable Language Models \(KnowLLM 2024\),S\. Li, M\. Li, M\. J\. Zhang, E\. Choi, M\. Geva, P\. Hase, and H\. Ji \(Eds\.\),Bangkok, Thailand,pp\. 171–176\.External Links:[Link](https://aclanthology.org/2024.knowllm-1.14/),[Document](https://dx.doi.org/10.18653/v1/2024.knowllm-1.14)Cited by:[§5](https://arxiv.org/html/2610.22807#S5.p3.1)\.

## Appendix AAdditional Results

#### The U\-Shape

As illustrated in Figure[4](https://arxiv.org/html/2610.22807#S4.F4), we observe that the performance of most models in multi\-document QA exhibits a distinct “U\-shape” pattern\. Specifically, placing the golden document at the beginning or the end of the context generally yields better performance than placing it in the middle\. This observation is consistent with the findings of[Liu et al\. \(2024a\)](https://arxiv.org/html/2610.22807#bib.bib40)\. However, this contrasts with our observations in multiple\-choice tasks, where the first position generally achieves significantly higher performance than all others\.

## Appendix BDataset Construction Details

We adopt the experimental setup of[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42), but since LITM only contains English data, we use the MLQA dataset for multilingual evaluation\. In the LITM dataset, each question is associated withKKindependent documents, only one of which \(the "golden documeLITMnt"\) contains the answer\. We conduct our experiments using a configuration ofN=20N=20documents, placing the golden document at positions 0, 4, 9, 14, and 19\.

Because MLQA provides three\-way alignment among <question, answer, golden doc\>, we randomly sampleN−1N\-1irrelevant documents from the same dataset for each question to simulate the setup in[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.22807#bib.bib42)\. Following the LITM configuration, we construct datasets for English, Arabic, Chinese, and Spanish withN=20N=20and golden documents at positions 0, 4, 9, 14, and 19\. For each language, we randomly sample 500 data points consisting of <question, answer, docs\> and evaluate model performance at each of the five golden document positions\. Consequently, each experimental run has5×500=2,5005\\times 500=2,500examples\.

## Appendix CAdditional Analysis of Thinking Models

We further analyzed why thinking models \(long CoT\) outperform non\-thinking models in the multi\-document scenario\. To this end, we utilized the thinking and non\-thinking models of the DeepSeek\-V3\.2 for a controlled comparison\. We categorize the cases where thinking models exhibit an advantage into three types of errors made by the non\-thinking baseline, which the thinking models successfully resolve\.

Table 4:Error type distribution among cases where thinking models outperform non\-thinking models\.We find that in 66\.1% of cases \(Type 1 and Type 2 combined\), the thinking model’s improved performance is attributed to its ability to comprehensively summarize multiple relevant documents, analyze and compare conflicting information, and better identify relevant documents\.

### C\.1Definition of Error Types

We define the three error types as follows:

Type 1 — Conflicting Docs\.The non\-thinking model fails to compare or reconcile conflicting information presented in the context\.

Type 2 — Relevance Identification\.Although model outputs frequently cite supporting documents, the non\-thinking model does not include the most relevant one among them\.

Type 3 — Unknown\.There are no explicit clues in the model\-generated answer or the reference answer to explain why the thinking model performs better\.

### C\.2Data Selection Criteria

For each question, with the golden document placed at positions 0, 4, 9, 14, and 19, five completely incorrect responses received a score of 0\. Each additional correct response earned \+1 point, yielding a maximum score of 5\. We identified 65 questions where the thinking model outperformed the non\-thinking model\.

## Appendix DLimitations

Our study is primarily limited by the scope of language and prompt variety\. While we evaluate five diverse languages, the results may not generalize to extremely low\-resource languages with distinct syntactic structures\. Finally, while thinking effectively mitigate bias, they do so at the cost of significantly increased inference\-time computation and latency, which may be impractical for real\-time applications\.

## Appendix EEthical Considerations

The primary ethical concern of this work involves the environmental impact and computational cost of evaluating extremely large\-scale models across multiple permutations and reasoning modes\. To address this, we utilized down\-sampled datasets where consistent trends were observed\. Additionally, our findings highlight that position bias can lead to misleading performance; failing to account for this bias could result in the deployment of unreliable systems in high\-stakes domains\. We encourage the community to use our findings to develop more robust and fair evaluation protocols rather than to manipulate model performance through strategic information placement\.