Position Bias Correction is Insufficient for One-Pass Attention Sorting

# Position Bias Correction is Insufficient for One-Pass Attention Sorting
Source: [https://arxiv.org/html/2606.27793](https://arxiv.org/html/2606.27793)
FARS, Qiong Tang222Equal contribution; human authors listed in alphabetical order by first name\.,Xiangkun Hu222Equal contribution; human authors listed in alphabetical order by first name\.,Xiangyang Liu222Equal contribution; human authors listed in alphabetical order by first name\.,Yiran Chen222Equal contribution; human authors listed in alphabetical order by first name\.,Yunfan Shao222Equal contribution; human authors listed in alphabetical order by first name\. Analemma fars@analemma\.ai

###### Abstract

Long\-context language models suffer from position bias, where information in middle positions is under\-utilized\. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple sort\-and\-generate cycles increase deployment cost\. We hypothesize that position bias is the primary bottleneck and propose Debiased One\-Pass Attention Sorting, which estimates a per\-prompt position\-bias curve from the low\-attention majority of documents and uses it to correct raw attention scores \(via subtraction or division\) to enable single\-pass sorting\. Our experiments on two models refute this hypothesis in the tested setting: on LLaMA\-2\-7B\-32K\-Instruct, debiasing produces identical results to uncalibrated single\-pass sorting \(94\.83% containment accuracy\), while on YaRN\-Llama\-2\-7b\-64k, debiasing improves accuracy by 8\.67 percentage points but remains 14\.84pp behind iterative sorting, closing only 37% of the gap\. These results suggest that position\-bias correction is insufficient to match iterative sorting, and that repeated reordering provides additional benefits beyond bias correction\.

> Disclosure:This paper was produced by FARS \(Fully Automated Research System\)111[https://analemma\.ai/fars/](https://analemma.ai/fars/), which autonomously performed the ideation, literature review, experiment design and execution, result analysis, and manuscript composition\. The accompanying code is publicly available\.222[https://gitlab\.com/fars\-a/calib\-attnsort\-onepass](https://gitlab.com/fars-a/calib-attnsort-onepass)The human authors contributed review and minor editorial revisions\. They have verified the authenticity of all cited references and confirmed that all reported experimental results originate from actual code execution\. Readers should be aware that the prose and presentation of this manuscript are primarily machine\-generated and may not meet the standards of fully human\-authored work\.

## 1Introduction

Long\-context language models are increasingly deployed in retrieval\-augmented generation \(RAG\) and multi\-document question answering, where models must identify and extract relevant information from extensive contexts\. However, these models exhibit systematic position bias: they tend to over\-attend to information at the beginning or end of the context while under\-utilizing middle positions, a phenomenon known as “lost\-in\-the\-middle”\(Liuet al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib1)\)\. This bias significantly degrades performance when relevant information appears in unfavorable positions\.

Attention Sorting\(Peysakhovich and Lerer,[2023](https://arxiv.org/html/2606.27793#bib.bib14)\)addresses this limitation through iterative document reordering based on attention patterns\. By measuring how much attention each document receives during generation and placing high\-attention documents at the end \(where recency\-biased models attend more effectively\), the method substantially improves long\-context QA accuracy\. However, Attention Sorting requireskkiterations of sorting, each followed by answer generation, significantly increasing latency for long contexts\.

We hypothesize that iterative sorting is primarily needed because raw attention scores are confounded by position bias: a relevant document far from the end may not receive sufficient attention in a single pass to be placed optimally\. If we can explicitly estimate and correct this position\-dependent bias, a single sorting pass should suffice to match iterative sorting accuracy\. To test this hypothesis, we proposeDebiased One\-Pass Attention Sorting, which estimates a per\-prompt position\-bias curve from the low\-attention majority of documents and uses it to correct raw attention scores \(via subtraction or division\), requiring only one sort pass plus one generation pass\.

Our experiments on SynthWiki@28K with two models of different bias characteristics refute this hypothesis in the tested setting\. On LLaMA\-2\-7B\-32K\-Instruct, debiasing produces identical results to uncalibratedk=1k=1sorting \(94\.83% accuracy\), with zero improvement across 600 examples\. On YaRN\-Llama\-2\-7b\-64k, which exhibits severe recency bias, debiasing improves accuracy by 8\.67 percentage points over uncalibratedk=1k=1but remains 14\.84pp behindk=5k=5sorting, closing only 37% of the gap\.

Our contributions are threefold\. First, we propose and evaluate debiased one\-pass attention sorting, a method that estimates and corrects for position bias in attention scores to enable single\-pass document reordering\. Second, we show that this correction is unnecessary for a strongk=1k=1LLaMA\-2\-7B\-32K\-Instruct setting but beneficial for a high\-bias YaRN setting, making the effect sharply model\-dependent\. Third, we provide evidence that position\-bias correction alone does not recover iterative sorting on YaRN, suggesting that iterative sorting provides benefits beyond bias correction that future work should investigate\.

## 2Related Work

### 2\.1Position Bias in LLMs

Large language models exhibit systematic position bias when processing long contexts\.Liuet al\.\([2024](https://arxiv.org/html/2606.27793#bib.bib1)\)identified the “lost\-in\-the\-middle” phenomenon, demonstrating that LLMs perform best when relevant information appears at the beginning or end of the input, with significant performance degradation for middle positions\. This U\-shaped attention pattern has been attributed to intrinsic attention bias, where tokens at extreme positions receive disproportionately high attention regardless of their semantic relevance\(Hsiehet al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib2)\)\.Yiet al\.\([2025](https://arxiv.org/html/2606.27793#bib.bib4)\)characterized this as the “attention basin” phenomenon, showing that shallow attention layers play a critical role in determining which positions receive attention\.Qianget al\.\([2025](https://arxiv.org/html/2606.27793#bib.bib3)\)further argue that initial\-token saliency can contribute to U\-shaped attention bias and propose directly scaling initial\-token attention weights\. Separately,Xiaoet al\.\([2024](https://arxiv.org/html/2606.27793#bib.bib5)\)discovered “attention sinks”—initial tokens that absorb excess attention mass even when semantically unimportant—which enables efficient streaming inference but also contributes to position\-dependent attention allocation\. Our method differs from these calibration approaches by estimating a prompt\-specific document\-level bias curve from the same context rather than intervening on token\-level attention weights or positional encodings\.

### 2\.2Context Window Extension

Extending the context window of pretrained LLMs has been an active research area\. Rotary Position Embedding \(RoPE\)\(Suet al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib13)\)encodes absolute positions using rotation matrices while incorporating relative position information in self\-attention, enabling flexible sequence lengths\. However, RoPE\-based models fail to generalize beyond their training sequence length\. Position Interpolation\(Chenet al\.,[2023](https://arxiv.org/html/2606.27793#bib.bib6)\)addresses this by linearly down\-scaling position indices to fit within the original context window, extending LLaMA models to 32K tokens with minimal fine\-tuning\. YaRN\(Penget al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib7)\)improves upon this with a compute\-efficient extension method requiring 10×\\timesfewer tokens and 2\.5×\\timesfewer training steps, enabling context windows up to 128K tokens\. More recently, LongRoPE\(Dinget al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib8)\)extends context windows to 2M tokens through progressive extension and non\-uniform positional interpolation\. These methods focus on enabling longer contexts but do not directly address the position bias that affects information utilization within those contexts\.

### 2\.3Attention\-Based Document Ranking

Several methods leverage attention patterns for document ranking in retrieval\-augmented generation\. Attention Sorting\(Peysakhovich and Lerer,[2023](https://arxiv.org/html/2606.27793#bib.bib14)\)iteratively reorders documents based on the attention they receive during generation, placing high\-attention documents at the end where recency\-biased models attend more effectively\.Chenet al\.\([2025](https://arxiv.org/html/2606.27793#bib.bib15)\)propose in\-context re\-ranking \(ICR\), which uses attention pattern changes caused by the query for efficient zero\-shot re\-ranking with only two forward passes\. To address inconsistencies in LLM\-based ranking,Zenget al\.\([2024](https://arxiv.org/html/2606.27793#bib.bib16)\)introduce LLM\-RankFusion, which mitigates order and transitive inconsistencies through calibration and ranker aggregation\.Tanget al\.\([2024](https://arxiv.org/html/2606.27793#bib.bib17)\)propose permutation self\-consistency, which marginalizes over different list orderings to produce order\-independent rankings\. More recently,Jinet al\.\([2025](https://arxiv.org/html/2606.27793#bib.bib18)\)show that simple retrieval reordering can mitigate hard negatives and lost\-in\-the\-middle effects in long\-context RAG\. Our work builds on Attention Sorting but investigates whether explicit position\-bias correction can reduce the number of required sorting iterations\.

## 3Method

### 3\.1Problem Setup

We consider long\-context extractive question answering, where a model receives a queryqqandNNdocuments𝒟=\{d1,…,dN\}\\mathcal\{D\}=\\\{d\_\{1\},\\ldots,d\_\{N\}\\\}concatenated into a single prompt, with exactly one document containing the answer\. Due to position bias\(Liuet al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib1)\), LLMs tend to over\-attend to documents near the beginning or end of the context while under\-utilizing middle positions\. For models exhibiting recency bias, placing the relevant document at the end of the context improves answer accuracy\. The goal is to reorder documents so that the relevant document appears in a high\-attention position \(typically the end\) before generating the answer\.

### 3\.2Attention Sorting Baseline

Attention Sorting\(Peysakhovich and Lerer,[2023](https://arxiv.org/html/2606.27793#bib.bib14)\)addresses position bias through iterative document reordering based on attention patterns\. For each documentdid\_\{i\}occupying token spanSiS\_\{i\}, the method computes a raw attention massaia\_\{i\}by aggregating attention weights from the first generated token across all layersℓ\\elland headshh:

ai=∑ℓ=1L∑h=1H∑t∈SiAtℓ,h,a\_\{i\}=\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{h=1\}^\{H\}\\sum\_\{t\\in S\_\{i\}\}A^\{\\ell,h\}\_\{t\},\(1\)whereAtℓ,hA^\{\\ell,h\}\_\{t\}is the attention weight from the first output token to input tokenttat layerℓ\\elland headhh\. Documents are then sorted by ascendingaia\_\{i\}, placing high\-attention documents at the end\. This process repeats forkkiterations, with an answer generated after each iteration; the final iteration’s answer is used\. Fork=5k=5, this results in 5 sort\-and\-generate cycles, significantly increasing latency for long contexts compared to a single generation pass\.

### 3\.3Debiased One\-Pass Sorting

We hypothesize that iterative sorting is primarily needed because raw attention scoresaia\_\{i\}are confounded by position bias: a relevant document far from the end may not receive sufficient attention in a single pass to be placed optimally\. If we can explicitly estimate and remove this position\-dependent bias, a single sorting pass should suffice\.

![Refer to caption](https://arxiv.org/html/2606.27793v1/figures/framework_overview.png)Figure 1:Overview of standard iterative Attention Sorting \(k=5k=5, 5 sort\-and\-generate cycles\) versus the proposed Debiased One\-Pass approach \(k=1k=1, 1 sort \+ 1 generation pass\)\. The proposed method estimates a position\-bias curve from the low\-attention majority of documents and uses it to correct raw attention scores for document reranking\.Our proposed method, illustrated in Figure[1](https://arxiv.org/html/2606.27793#S3.F1), operates as follows\. First, we extract raw attention massesaia\_\{i\}for each document using a single decode step, identical to the first iteration of standard Attention Sorting\. Second, we estimate a position\-bias curveb^​\(p\)\\hat\{b\}\(p\)from the attention\-position pairs\(pi,ai\)\(p\_\{i\},a\_\{i\}\)within the same prompt, wherepip\_\{i\}denotes documentii’s position index\. Under the assumption that most documents are distractors with near\-zero relevance, we trim the topα\\alphafraction of documents by attention to exclude likely relevant documents, bin the remaining positions intoBBequal\-width bins, compute an aggregate statistic \(median or mean\) within each bin, and linearly interpolate to obtain a continuous bias curveb^​\(p\)\\hat\{b\}\(p\)\. Third, we compute debiased scoressi=ai−b^​\(pi\)s\_\{i\}=a\_\{i\}\-\\hat\{b\}\(p\_\{i\}\)\(additive mode\) orsi=ai/max⁡\(b^​\(pi\),ϵ\)s\_\{i\}=a\_\{i\}/\\max\(\\hat\{b\}\(p\_\{i\}\),\\epsilon\)\(divisive mode, withϵ=10−10\\epsilon=10^\{\-10\}in our implementation\)\. Finally, we use the debiased scores to adjust the document ordering and generate the answer from the reordered prompt\.

The choice of hyperparameters \(α\\alpha,BB, aggregation function, debiasing mode\) and reordering strategy was tuned per model on the same evaluation data reported in our results, as described in Section[4](https://arxiv.org/html/2606.27793#S4)\. This approach requires only one sort pass plus one generation pass, matchingk=1k=1Attention Sorting in compute while aiming to achievek=5k=5accuracy through explicit bias correction rather than iterative refinement\.

## 4Experiments

### 4\.1Experimental Setup

We evaluate debiased one\-pass attention sorting on the SynthWiki benchmark\(Peysakhovich and Lerer,[2023](https://arxiv.org/html/2606.27793#bib.bib14)\), which contains synthetic long\-context extractive QA instances with one gold document among many distractors\. Following the original setup, we construct contexts of approximately 28K tokens by sampling distractor documents, replacing one with the gold document at a random position\.

We test two models with different position bias characteristics: LLaMA\-2\-7B\-32K\-Instruct333[https://huggingface\.co/togethercomputer/Llama\-2\-7B\-32K\-Instruct](https://huggingface.co/togethercomputer/Llama-2-7B-32K-Instruct), an instruction\-tuned variant of Llama 2\(Touvronet al\.,[2023](https://arxiv.org/html/2606.27793#bib.bib19)\)with a 32K context window and moderate position bias, and YaRN\-Llama\-2\-7b\-64k\(Penget al\.,[2024](https://arxiv.org/html/2606.27793#bib.bib7)\), a context\-extended model without instruction tuning that exhibits severe recency bias\. We compare four methods: \(1\) No Sorting \(vanilla generation\), \(2\) Attention Sortk=1k=1\(single\-pass sorting by raw attention, then generation\), \(3\) Debiasedk=1k=1\(our proposed method, same compute ask=1k=1\), and \(4\) Attention Sortk=5k=5\(iterative sorting baseline\)\. In thek=5k=5condition, the implementation generates an answer after each sorting iteration and uses the final iteration’s answer, resulting in 5 sort\-and\-generate cycles\.444This matches the original Attention Sorting implementation, which evaluates intermediate answers at each iteration\.We use greedy decoding and report normalized substring\-match accuracy \(the prediction is counted as correct if any acceptable gold answer appears as a substring of the lowercased model response\) across 200 examples per seed with 3 random seeds \(42, 123, 456\)\.

### 4\.2Main Results

Table 1:Results on SynthWiki@28K with LLaMA\-2\-7B\-32K\-Instruct\. Under the selected minimal\-swap configuration, debiasedk=1k=1produces identical results to uncalibratedk=1k=1\. Best inbold\.Table 2:Results on SynthWiki@28K with YaRN\-Llama\-2\-7b\-64k\. Debiasedk=1k=1improves \+8\.67pp over uncalibratedk=1k=1but remains 14\.84pp behindk=5k=5, closing only 37% of the gap\. Best inbold, second\-bestunderlined\.Table[1](https://arxiv.org/html/2606.27793#S4.T1)presents results on LLaMA\-2\-7B\-32K\-Instruct\. Debiasedk=1k=1produces identical results to uncalibratedk=1k=1across all 600 examples\. This does not prove that no debiasing intervention could affect this model, but shows that our prompt\-level correction adds no value over raw attention sorting in this setting\. The gap betweenk=1k=1\(94\.83%\) andk=5k=5\(95\.50%\) is only 0\.67 percentage points, within one standard deviation\.

Table[2](https://arxiv.org/html/2606.27793#S4.T2)shows results on YaRN\-Llama\-2\-7b\-64k, which exhibits severe recency bias\. Here, debiasedk=1k=1achieves 55\.83% accuracy, improving \+8\.67pp over uncalibratedk=1k=1\(47\.17%\)\. However, it remains 14\.84pp behindk=5k=5\(70\.67%\), closing only 37% of the gap\. This shows that while debiasing provides meaningful improvement on high\-bias models, it is insufficient to match iterative sorting\.

The debiased method was configured per model\. For LLaMA, we use minimal\-swap with additive debiasing \(α=0\.05\\alpha=0\.05,B=20B=20, median\); for YaRN, we use full\-sort with divisive debiasing \(α=0\.005\\alpha=0\.005,B=40B=40, mean\)\. These hyperparameters were selected on the same evaluation set reported here, which may overestimate the method’s effectiveness on new data\.

The effectiveness of position\-bias correction is highly model\-dependent\. On LLaMA\-2\-7B\-32K\-Instruct, uncalibratedk=1k=1already places the gold document at mean position 164\.86 out of approximately 166 documents, leaving minimal room for bias correction to improve\. On YaRN, the severe recency bias means raw attention scores are significantly corrupted by position, making debiasing beneficial but still insufficient\.

### 4\.3Analysis: Iterative Sorting Progression

![Refer to caption](https://arxiv.org/html/2606.27793v1/figures/iterative_progression.png)Figure 2:Accuracy and mean gold document position across sorting iterations \(k=1k=1tok=5k=5\) on LLaMA\-2\-7B\-32K\-Instruct\. Most improvement occurs in the first two iterations, with accuracy increasing from 94\.83% \(k=1k=1\) to 95\.83% \(k=2k=2\), then plateauing\. Gold document position converges to∼\\sim165\.77 out of∼\\sim166 documents\.Figure[2](https://arxiv.org/html/2606.27793#S4.F2)shows the progression of accuracy and gold document position across sorting iterations on LLaMA\-2\-7B\-32K\-Instruct\. Most improvement occurs in the first two iterations: accuracy increases from 94\.83% atk=1k=1to 95\.83% atk=2k=2, then plateaus with minor fluctuations throughk=5k=5\(final accuracy 95\.50%\)\. The gold document position converges rapidly, reaching 164\.85 afterk=1k=1and stabilizing at approximately 165\.77 byk=5k=5\.

These results suggest that iterative sorting may provide benefits beyond position\-bias correction\. Even after the gold document is placed near the optimal position, additional iterations continue to improve accuracy slightly, possibly from attention context refinement or noise reduction across passes\. However, we do not directly measure these mechanisms, and the 0\.67pp LLaMA gap is within seed variation\. The stronger evidence is the 14\.84pp YaRN gap after debiasing, which motivates future analysis of attention\-pattern changes across sorting iterations\.

## 5Conclusion

We proposed debiased one\-pass attention sorting, which estimates and corrects for position bias in raw attention scores to enable single\-pass document reordering\. Our experiments reveal that position\-bias correction is insufficient to match iterative sorting: on LLaMA\-2\-7B\-32K\-Instruct, debiasing has zero effect, while on YaRN\-Llama\-2\-7b\-64k, it improves accuracy by 8\.67pp but remains 14\.84pp behindk=5k=5sorting\. These results suggest that iterative sorting provides benefits beyond bias correction, and that future work should investigate the mechanisms underlying iterative sorting’s effectiveness\. Our findings indicate that debiasing should be applied selectively to high\-bias models\. Limitations include evaluation on only two models and one benchmark; broader evaluation across diverse models and tasks would strengthen these conclusions\.

## References

- Attention in large language models yields efficient zero\-shot re\-rankers\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2606.27793#S2.SS3.p1.1)\.
- S\. Chen, S\. Wong, L\. Chen, and Y\. Tian \(2023\)Extending context window of large language models via positional interpolation\.ArXivabs/2306\.15595\.Cited by:[§2\.2](https://arxiv.org/html/2606.27793#S2.SS2.p1.2)\.
- Y\. Ding, L\. Zhang, C\. Zhang, Y\. Xu, N\. Shang, J\. Xu, F\. Yang, and M\. Yang \(2024\)LongRoPE: extending llm context window beyond 2 million tokens\.ArXivabs/2402\.13753\.Cited by:[§2\.2](https://arxiv.org/html/2606.27793#S2.SS2.p1.2)\.
- C\. Hsieh, Y\. Chuang, C\. Li, Z\. Wang, L\. Le, A\. Kumar, J\. Glass, A\. Ratner, C\. Lee, R\. Krishna, and T\. Pfister \(2024\)Found in the middle: calibrating positional attention bias improves long context utilization\.pp\. 14982–14995\.Cited by:[§2\.1](https://arxiv.org/html/2606.27793#S2.SS1.p1.1)\.
- B\. Jin, J\. Yoon, J\. Han, and S\. O\. Arik \(2025\)Long\-context llms meet rag: overcoming challenges for long inputs in rag\.InInternational Conference on Learning Representations,Cited by:[§2\.3](https://arxiv.org/html/2606.27793#S2.SS3.p1.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.Cited by:[§1](https://arxiv.org/html/2606.27793#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.27793#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2606.27793#S3.SS1.p1.3)\.
- B\. Peng, J\. Quesnelle, H\. Fan, and E\. Shippole \(2024\)YaRN: efficient context window extension of large language models\.InInternational Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2606.27793#S2.SS2.p1.2),[§4\.1](https://arxiv.org/html/2606.27793#S4.SS1.p2.5)\.
- A\. Peysakhovich and A\. Lerer \(2023\)Attention sorting combats recency bias in long context language models\.ArXivabs/2310\.01427\.Cited by:[§1](https://arxiv.org/html/2606.27793#S1.p2.1),[§2\.3](https://arxiv.org/html/2606.27793#S2.SS3.p1.1),[§3\.2](https://arxiv.org/html/2606.27793#S3.SS2.p1.5),[§4\.1](https://arxiv.org/html/2606.27793#S4.SS1.p1.1)\.
- Z\. Qiang, S\. Zhao, H\. Wang, B\. Qin, and T\. Liu \(2025\)Uncovering the role of initial saliency in u\-shaped attention bias: scaling initial token weight for enhanced long\-text processing\.ArXivabs/2512\.13109\.Cited by:[§2\.1](https://arxiv.org/html/2606.27793#S2.SS1.p1.1)\.
- J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§2\.2](https://arxiv.org/html/2606.27793#S2.SS2.p1.2)\.
- R\. Tang, X\. Zhang, X\. Ma, J\. Lin, and F\. Ture \(2024\)Found in the middle: permutation self\-consistency improves listwise ranking in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 2327–2340\.Cited by:[§2\.3](https://arxiv.org/html/2606.27793#S2.SS3.p1.1)\.
- H\. Touvron, L\. Martin, K\. R\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. Canton\-Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. M\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.ArXivabs/2307\.09288\.Cited by:[§4\.1](https://arxiv.org/html/2606.27793#S4.SS1.p2.5)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2606.27793#S2.SS1.p1.1)\.
- Z\. Yi, D\. Zeng, Z\. Ling, H\. Luo, Z\. Xu, W\. Liu, J\. Luan, W\. Cao, and Y\. Shen \(2025\)Attention basin: why contextual position matters in large language models\.ArXivabs/2508\.05128\.Cited by:[§2\.1](https://arxiv.org/html/2606.27793#S2.SS1.p1.1)\.
- Y\. Zeng, O\. Tendolkar, R\. Baartmans, Q\. Wu, H\. Wang, and L\. Chen \(2024\)LLM\-rankfusion: mitigating intrinsic inconsistency in llm\-based ranking\.ArXivabs/2406\.00231\.Cited by:[§2\.3](https://arxiv.org/html/2606.27793#S2.SS3.p1.1)\.