Sentence-Level Contextual Entrainment in Large Language Models

# Sentence-Level Contextual Entrainment in Large Language Models
Source: [https://arxiv.org/html/2606.24077](https://arxiv.org/html/2606.24077)
###### Abstract

Contextual entrainment, which is a newly discovered phenomenon in large language models \(LLMs\), refers to the tendency of a model to assign higher probabilities to tokens that appear in its context\. In this work, we extend this phenomenon from the token level to the sentence level by examining the per\-token mean log\-probability of a sentence instead of the probabilities of individual tokens\. We investigate sentence\-level contextual entrainment across 26 LLMs from seven families and two datasets, which cover both subjective and objective tasks\. We find that sentence\-level contextual entrainment exists\. This means that the sentences in the prompt \(even if they are counterfactual statements\) can significantly increase their probability during model inference time\. As the model size increases, contextual entrainment gradually decreases\. We also find that contextual entrainment is controlled by 2% to 4% of the attention heads\. Turning off these attention heads can effectively mitigate contextual entrainment without hurting the model’s performance\.111Our code is available at[https://github\.com/ku\-nlp/Sentence\-Level\_Contextual\_Entrainment\_in\_LLMs](https://github.com/ku-nlp/Sentence-Level_Contextual_Entrainment_in_LLMs)\.

## 1Introduction

Large language models \(LLMs\) have demonstrated remarkable in\-context learning \(ICL\) capabilities, enabling them to effectively leverage contextual information provided in the prompt without any parameter updatesBrownet al\.\([2020](https://arxiv.org/html/2606.24077#bib.bib1)\)\. Due to its simplicity, flexibility, and outstanding empirical performance, ICL has become an important approach for numerous natural language processing tasks, spanning a wide range of domains from classificationZhaoet al\.\([2021](https://arxiv.org/html/2606.24077#bib.bib5)\), question\-answeringLiet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib44)\), reasoningWeiet al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib3)\), and code generationChenet al\.\([2021](https://arxiv.org/html/2606.24077#bib.bib4)\)\.

To understand how ICL works,Daiet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib45)\)explain it as implicitly performing gradient descent, with the model acting as a meta\-optimizer that produces meta\-gradients from the demonstration examples\. From a circuit perspective, this ability has been traced to induction heads that complete patterns by copying relevant tokens from the context as the model’s responseOlssonet al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib30)\); Crosbie and Shutova \([2025](https://arxiv.org/html/2606.24077#bib.bib31)\)\. These studies mostly explain how models benefit from contextual information in prompts; however, how models misuse contextual information in prompts is comparatively less understood\.

Recently,Niuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)uncovered a striking phenomenon they term*contextual entrainment*: an LLM systematically increases the probabilities of any token that has appeared in the context, including tokens that have no semantic connection to the subsequent query\. As shown in Figure[1\(a\)](https://arxiv.org/html/2606.24077#S1.F1.sf1), given the context “Paris is part of France\.” followed by the query “Which country is Tokyo in?” the next token probability of “France” \(a token from the context\) rises sharply above its no\-context baseline, even though the correct next token is “Japan\.” Through a differentiable masking analysis,Niuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)traced this phenomenon to a small set of attention heads and demonstrated that setting their outputs to zero can reduce entrainment\.

![Refer to caption](https://arxiv.org/html/2606.24077v1/x1.png)\(a\)Token\-level contextual entrainment\(Niuet al\.,[2025](https://arxiv.org/html/2606.24077#bib.bib6)\): at the next\-token position, in\-prompt tokens \(e\.g\., “France”\) receive a large probability increase\.
![Refer to caption](https://arxiv.org/html/2606.24077v1/x2.png)\(b\)Sentence\-level contextual entrainment \(Ours\): over candidate*full\-sentence*continuations, the in\-prompt sentence \(“Paris is part of France\.”\) receives a large probability increase\.

Figure 1:Examples of token\-level and sentence\-level contextual entrainment\.Niuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)’s analysis, however, is restricted to the token level: it quantifies the increase in the probabilities of a single token at the position of the next token predicted by the model\. Sentences are the more common unit of input and output in LLM use; we therefore extend token\-level contextual entrainment to the sentence level\. As shown in Figure[1\(b\)](https://arxiv.org/html/2606.24077#S1.F1.sf2), under the same prompt above, the entire sentence “Paris is part of France\.” receives a sharp probability increase as a candidate continuation—even though it is factually incorrect about Tokyo—while the probability of the correct answer “Tokyo is part of Japan\.” is decreased\.

Our work differs from prior studiesNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\); Kukrejaet al\.\([2026](https://arxiv.org/html/2606.24077#bib.bib28)\)in the following main points\. First, we study contextual entrainment at the sentence\-level rather than the token\-level, an extension that more closely matches how information accumulates during realistic generation\. Second, whereasNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)reports results for a single modelGrattafioriet al\.\([2024](https://arxiv.org/html/2606.24077#bib.bib18)\)and the panel ofKukrejaet al\.\([2026](https://arxiv.org/html/2606.24077#bib.bib28)\)is limited to the older Cerebras\-GPTDeyet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib33)\)and PythiaBidermanet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib34)\)families, we run experiments on 26 models spanning seven families\. Third, existing work measures entrainment only on factual recall tasks such as LREHernandezet al\.\([2024](https://arxiv.org/html/2606.24077#bib.bib8)\), while we introduce the subjective task WVSHaerpferet al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib9)\)as a second probe\. Fourth, beyond the entrainment heads tied to individual relations, we identify a set of shared heads common to all relations\. We are the first to identify heads that generalize across relations\.

We focus on the following three research questions\.RQ1:Does the contextual entrainment phenomenon also exist at the sentence level? We investigate this in both subjective and objective tasks\.RQ2:How does contextual entrainment scale with model size? As LLMs encode sentences through contextual representations whose richness scales with model capacity, sentence\-level entrainment may vary with model size\.RQ3:Do LLMs have a small set of entrainment heads that can be turned off to mitigate contextual entrainment without hurting task performance?

Our main contributions are as follows:1\)We extend token\-level contextual entrainment to the sentence level by representing the probability of the model’s response with its per\-token mean log\-probability \(§[2](https://arxiv.org/html/2606.24077#S2)\)\.2\)We extend differentiable attention head masking to the sentence level \(§[3](https://arxiv.org/html/2606.24077#S3)\)\.3\)Our experiments demonstrate that sentence\-level contextual entrainment exists; this entrainment persists even when the sentence in the prompt is a counterfactual statement associated with the query \(§[5\.1](https://arxiv.org/html/2606.24077#S5.SS1)\)\.4\)By analyzing different sizes of four model families, we find that the contextual entrainment phenomenon is related to model size: as model size increases, contextual entrainment gradually decreases\. In contrast, the distraction on the response that does not appear in the context increases with model size \(§[5\.2](https://arxiv.org/html/2606.24077#S5.SS2)\)\.5\)We identify a sparse set of shared heads: turning off only 2% to 4% of the attention heads can effectively mitigate contextual entrainment without hurting model performance \(§[5\.3](https://arxiv.org/html/2606.24077#S5.SS3)\)\.

## 2Background and Formalisation

### 2\.1Notations

Letℳ\\mathcal\{M\}be an LLM parameterized byθ\\theta, with vocabulary𝒱\\mathcal\{V\}\. For any token sequencexx,ℳ\\mathcal\{M\}produces logitsz=hθ​\(x\)z=h\_\{\\theta\}\(x\), and we write its predictive distribution asπ=Softmax​\(z\)\\pi=\\texttt\{Softmax\}\(z\), withlog⁡π​\(w∣x\)\\log\\pi\(w\\mid x\)denoting the log\-probability assigned to a candidate next tokenw∈𝒱w\\in\\mathcal\{V\}\.

For a token sequencey=\(y1,…,yL\)y=\(y\_\{1\},\\dots,y\_\{L\}\), the model’s log\-probability in generatingyyafter the token sequencexxdecomposes by the chain rule:

log⁡π​\(y∣x\)=∑i=1Llog⁡π​\(yi∣x,y<i\),\\log\\pi\(y\\mid x\)=\\sum\_\{i=1\}^\{L\}\\log\\pi\(y\_\{i\}\\mid x,y\_\{<i\}\),\(1\)wherey<i=\(y1,…,yi−1\)y\_\{<i\}=\(y\_\{1\},\\dots,y\_\{i\-1\}\)andy<1y\_\{<1\}are empty\.

#### Inputs

The input can be split into two parts:

- •A queryqq, such as “Tokyo is part of” \(the next\-token completion style introduced byNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)\) or “Which country is Tokyo in?” \(for our sentence\-level queries\);
- •A contextcc, an additional segment prepended toqq, such as “Paris is part of France\.”

We denote the prompt bypp, formed by concatenatingccandqqin that order, and use the comma notationc,qc,qinside conditioning expressions to denote this concatenation\.222Several notations for sequence concatenation appear in the existing work \(e\.g\.,Liu and Chu \([2025](https://arxiv.org/html/2606.24077#bib.bib10)\)uses⊕\\oplus\), but we follow the comma convention adopted inNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)\.The contrast betweenlogπ\(⋅∣q\)\\log\\pi\(\\cdot\\mid q\)andlogπ\(⋅∣c,q\)\\log\\pi\(\\cdot\\mid c,q\)is the central object of study throughout this work\.

### 2\.2Token\-Level Contextual Entrainment

Let𝒯​\(c\)\\mathcal\{T\}\(c\)be the set of tokens that appear in the contextcc:

𝒯​\(c\)=\{w∈𝒱:w​appears at least once in​c\}\.\\mathcal\{T\}\(c\)=\\\{w\\in\\mathcal\{V\}:w\\text\{ appears at least once in \}c\\\}\.\(2\)The contextual entrainment phenomenon concerns how tokenwwin𝒯​\(c\)\\mathcal\{T\}\(c\)affects the log\-probability of the model’s next token\. The log\-probability increase at the token level can be expressed as:

Δ​log⁡π​\(w∣c,q\):=log⁡π​\(w∣c,q\)−log⁡π​\(w∣q\)\.\\Delta\\log\\pi\(w\\mid c,q\):=\\log\\pi\(w\\mid c,q\)\-\\log\\pi\(w\\mid q\)\.\(3\)Eq\. \([3](https://arxiv.org/html/2606.24077#S2.E3)\) measures the change in the log\-probability of generatingw∈cw\\in cwhen contextccis concatenated before queryqq\. Positive values indicate that concatenating the contextccincreases the probability of the next tokenww\.

Next, we formalize the token\-level contextual entrainment phenomenon:

###### Hypothesis 1\.

For every tokenw∈𝒯​\(c\)w\\in\\mathcal\{T\}\(c\), the log\-probability of the model’s next tokenwwis systematically increased by concatenating the contextccto the queryqq:

∀w∈𝒯\(c\):𝔼\[Δlogπ\(w∣c,q\)\]\>0,\\forall\\,w\\in\\mathcal\{T\}\(c\):\\quad\\mathbb\{E\}\[\\Delta\\log\\pi\(w\\mid c,q\)\]\>0,\(4\)where the expectation is taken over the data distribution of\(c,q\)\(c,q\)pairs\.

This restatesNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)’s finding in log\-probability form\. The original definition is expressed in terms of probabilities; for comparisons at the token level, the two are equivalent\.

### 2\.3Sentence\-Level Contextual Entrainment

We now extend Hypothesis[1](https://arxiv.org/html/2606.24077#Thmhypothesis1)from a single token to a sentence\. Lety=\(y1,…,yL\)∈𝒱y=\(y\_\{1\},\\dots,y\_\{L\}\)\\in\\mathcal\{V\}be a sentence of tokens that appears as a contiguous substring of the contextcc\(soyi∈𝒯​\(c\)y\_\{i\}\\in\\mathcal\{T\}\(c\)for everyii\)\. The quantity of interest is how the model’s log\-probability in generatingyyas a continuation changes when the prompt is extended fromqqtoc,qc,q\. The log\-probability increase at the sentence level can be defined as:

Δ​log⁡π​\(y∣c,q\):=log⁡π​\(y∣c,q\)−log⁡π​\(y∣q\)\.\\Delta\\log\\pi\(y\\mid c,q\):=\\log\\pi\(y\\mid c,q\)\-\\log\\pi\(y\\mid q\)\.\(5\)We focus on cases where the responseyyis exactly the contextccand investigate whether the model’s log\-probability in generatingyyalso increases in the expectation when the contextccis concatenated before the queryqq\.

Applying Eq\. \([1](https://arxiv.org/html/2606.24077#S2.E1)\) to each term in Eq\. \([5](https://arxiv.org/html/2606.24077#S2.E5)\), and then subtracting them:

Δ​log⁡π​\(y∣c,q\)\\displaystyle\\Delta\\log\\pi\(y\\mid c,q\)=∑i=1L\[logπ\(yi∣c,q,y<i\)\\displaystyle=\\sum\_\{i=1\}^\{L\}\\Big\[\\log\\pi\(y\_\{i\}\\mid c,q,y\_\{<i\}\)\(6\)−logπ\(yi∣q,y<i\)\]\.\\displaystyle\-\\log\\pi\(y\_\{i\}\\mid q,y\_\{<i\}\)\\Big\]\.This means that the increase in log\-probability at the sentence level is exactly equal to the sum of the increases in log\-probability for each token in the sentence\. Each term in Eq\. \([6](https://arxiv.org/html/2606.24077#S2.E6)\) takes the form of a token\-level increase, where the prompt isc,q,y<ic,q,y\_\{<i\}:

Δ​log⁡π​\(yi∣c,q,y<i\)\\displaystyle\\Delta\\log\\pi\(y\_\{i\}\\mid c,q,y\_\{<i\}\)=log⁡π​\(yi∣c,q,y<i\)\\displaystyle=\\log\\pi\(y\_\{i\}\\mid c,q,y\_\{<i\}\)\(7\)−log⁡π​\(yi∣q,y<i\)\.\\displaystyle\-\\log\\pi\(y\_\{i\}\\mid q,y\_\{<i\}\)\.
###### Proposition 1\.

If the model’s responseyyequals \(or is a subsequence of\) the contextccin the promptp=\(c,q\)p=\(c,q\), then under Hypothesis[1](https://arxiv.org/html/2606.24077#Thmhypothesis1),

𝔼​\[Δ​log⁡π​\(y∣c,q\)\]\>0\.\\mathbb\{E\}\[\\Delta\\log\\pi\(y\\mid c,q\)\]\>0\.\(8\)

###### Proof sketch\.

Given thatyyequals \(or is a subsequence of\)cc, every tokenyiy\_\{i\}belongs to𝒯​\(c\)\\mathcal\{T\}\(c\)\. For each positionii, Hypothesis[1](https://arxiv.org/html/2606.24077#Thmhypothesis1)applied with tokenw=yiw=y\_\{i\}and promptp=c,q,y<ip=c,q,y\_\{<i\}\(notingyi∈𝒯​\(c\)⊆𝒯​\(p\)y\_\{i\}\\in\\mathcal\{T\}\(c\)\\subseteq\\mathcal\{T\}\(p\)\) gives𝔼​\[Δ​log⁡π​\(yi∣c,q,y<i\)\]\>0\\mathbb\{E\}\[\\Delta\\log\\pi\(y\_\{i\}\\mid c,q,y\_\{<i\}\)\]\>0, i\.e\., each summand in Eq\. \([6](https://arxiv.org/html/2606.24077#S2.E6)\) is positive in expectation\. Applying the expectation linearly to Eq\. \([6](https://arxiv.org/html/2606.24077#S2.E6)\) yields:

𝔼​\[Δ​log⁡π​\(y∣c,q\)\]\\displaystyle\\mathbb\{E\}\[\\Delta\\log\\pi\(y\\mid c,q\)\]\(9\)=∑i=1L𝔼​\[Δ​log⁡π​\(yi∣c,q,y<i\)\]\>0\.\\displaystyle\\quad=\\sum\_\{i=1\}^\{L\}\\mathbb\{E\}\[\\Delta\\log\\pi\(y\_\{i\}\\mid c,q,y\_\{<i\}\)\]\>0\.∎

## 3Sentence\-Level Entrainment Head Discovery

FollowingNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)we identify the attention heads responsible for contextual entrainment via differentiable maskingDe Caoet al\.\([2020](https://arxiv.org/html/2606.24077#bib.bib11)\)\. A learnable Gumbel\-sigmoid gateml,h∈\{0,1\}m\_\{l,h\}\\in\\\{0,1\\\}\(Janget al\.,[2017](https://arxiv.org/html/2606.24077#bib.bib12)\)is attached to every attention headhhat every layerll, scaling the head’s output; settingml,h=0m\_\{l,h\}=0multiplies that head’s contribution to the residual stream by zero\. During training, the gate is computed as

ml,h=𝟙​\[σ​\(ℓl,h\+gτ\)\>12\],m\_\{l,h\}=\\mathds\{1\}\\\!\\left\[\\sigma\\\!\\left\(\\frac\{\\ell\_\{l,h\}\+g\}\{\\tau\}\\right\)\>\\frac\{1\}\{2\}\\right\],\(10\)whereσ​\(x\)=11\+e−x\\sigma\(x\)=\\frac\{1\}\{1\+e^\{\-x\}\}is the sigmoid function,ggis Logistic noise,τ∈\(0,∞\)\\tau\\in\(0,\\infty\)is a temperature hyperparameter, and𝟙​\[⋅\]\\mathds\{1\}\[\\cdot\]is the indicator function; the gradient bypasses the discretization through the straight\-through estimator\(Bengioet al\.,[2013](https://arxiv.org/html/2606.24077#bib.bib43)\)\. At inference time, we deterministically setml,h=𝟙​\[ℓl,h\>0\]m\_\{l,h\}=\\mathds\{1\}\[\\ell\_\{l,h\}\>0\]\. In this section, we use the factual statements as the context, denoted ascc\. The model’s no\-context natural response isrr, evaluatingccandrrunder the same with\-context prompt:Lc¯=1\|c\|​∑i=1\|c\|log⁡π​\(ci∣c,q,c<i\)\\overline\{L\_\{c\}\}=\\frac\{1\}\{\|c\|\}\\sum\_\{i=1\}^\{\|c\|\}\\log\\pi\(c\_\{i\}\\mid c,q,c\_\{<i\}\)andLr¯=1\|r\|​∑i=1\|r\|log⁡π​\(ri∣c,q,r<i\)\\overline\{L\_\{r\}\}=\\frac\{1\}\{\|r\|\}\\sum\_\{i=1\}^\{\|r\|\}\\log\\pi\(r\_\{i\}\\mid c,q,r\_\{<i\}\)\. The mask logitsℓl,h\\ell\_\{l,h\}are trained to minimize:

ℒ\\displaystyle\\mathcal\{L\}=softplus​\(Lc¯−Lr¯\)\\displaystyle=\\mathrm\{softplus\}\(\\overline\{L\_\{c\}\}\-\\overline\{L\_\{r\}\}\)\(11\)\+α⋅KL​\(Pnm∥Pm\)\+λ⋅1\|H\|​∑l,h\(1−σ​\(ℓl,h\)\)\.\\displaystyle\+\\alpha\\cdot\\mathrm\{KL\}\\\!\\bigl\(P\_\{\\text\{nm\}\}\\,\\\|\\,P\_\{\\text\{m\}\}\\bigr\)\+\\lambda\\cdot\\tfrac\{1\}\{\|H\|\}\\sum\_\{l,h\}\\\!\\bigl\(1\-\\sigma\(\\ell\_\{l,h\}\)\\bigr\)\.We wrapLc¯−Lr¯\\overline\{L\_\{c\}\}\-\\overline\{L\_\{r\}\}with a softplus functionDugaset al\.\([2000](https://arxiv.org/html/2606.24077#bib.bib38)\)so that the mask is no longer modified onceLc¯<Lr¯\\overline\{L\_\{c\}\}<\\overline\{L\_\{r\}\}, as continued modification in this regime harms the decoderGaoet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib39)\)\. Put simply, the optimization halts as soon as the model prefersrrovercc\.PnmP\_\{\\text\{nm\}\}is the next\-token distribution of the model without context, andPmP\_\{\\text\{m\}\}is the distribution obtained under the same condition with the current mask applied\. The KL divergenceKullback and Leibler \([1951](https://arxiv.org/html/2606.24077#bib.bib42)\)pullsPnmP\_\{\\text\{nm\}\}andPmP\_\{\\text\{m\}\}closer to each other, a smaller value indicates a smaller discrepancy between the two distributionsLiu and Hou \([2023](https://arxiv.org/html/2606.24077#bib.bib40)\); Liu \([2024](https://arxiv.org/html/2606.24077#bib.bib41)\)\.ℓl,h\\ell\_\{l,h\}is the learnable mask logit of thehh\-th attention head in thell\-th layer\. Hereσ​\(ℓl,h\)\\sigma\(\\ell\_\{l,h\}\)denotes the probability that the head is retained, while1−σ​\(ℓl,h\)1\-\\sigma\(\\ell\_\{l,h\}\)denotes the probability that it is turned off\. The third term encourages the model to turn off fewer heads\.α\\alphaandλ\\lambdaare hyperparameters\.

## 4Experimental Settings

### 4\.1Datasets

We evaluate the contextual entrainment phenomenon on two datasets, covering both objective and subjective tasks:

#### Linearity of Relation Decoding\(LRE; Hernandezet al\.,[2024](https://arxiv.org/html/2606.24077#bib.bib8)\)

LRE contains 47 relations across four categories \(factual associations,commonsense knowledge,implicit biases,linguistic knowledge\), each contains facts in the triplet format:⟨s​o​u​r​c​e,t​a​r​g​e​t,r​e​l​a​t​i​o​n⟩\\langle source,target,relation\\rangle\. For example,⟨T​o​k​y​o,J​a​p​a​n,c​i​t​y​i​n​c​o​u​n​t​r​y⟩\\langle Tokyo,Japan,city\\ in\\ country\\ranglecorresponds to the fact thatTokyo is part of Japan\. We apply the filtering rules fromNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)to keep only 16 relations covering 3,688 samples\. For each retained relation, we keep at most 50 samples: relations with more than 50 instances are downsampled to 50, and the others are kept in full\. If the resulting count is odd, one sample is discarded\. The remaining samples of each relation are then split in a 1:1 ratio into a context pool \(used as the context\) and a query pool \(used as the query\)\. This results in 594 triples in total: 297 in the context pool and 297 in the query pool\. Each triple can be assembled into a context and a query for the contextual entrainment experiment\.

#### World Values Survey\(WVS; Haerpferet al\.,[2022](https://arxiv.org/html/2606.24077#bib.bib9)\)

The WVS is a global survey of human values; we use the publicly released Wave 7 \(2017–2022\), which contains opinion items on social, political, and ethical topics\. We utilize the dataset filtered fromLiuet al\.\([2026](https://arxiv.org/html/2606.24077#bib.bib14)\)’s work\. After removing items whose answer scales were not naturally binary, we retained 135 items\. Each item is paired with two opposing opinion statements generated by GPT\-5\.4:333[https://openai\.com/](https://openai.com/)a support statement to express the positive opinion, and an oppose statement to express the negative opinion\. We split these 135 items into a context pool containing 5 items and a test set containing 130 items\.

### 4\.2Prompts

In this section, we introduce the exact prompt formats used in our experiments\. We use a consistent prompt format in LRE and WVS: A prompt consists of two components: \(i\)Contextand \(ii\)Query\. Then, the model is required to generate aResponsein the form of a sequence of tokens\.

For the LRE dataset, the prompt is as follows:

\# Context Paris is part of France\. \# Query Which country is Tokyo in? \# Response <Response\>

The role markers \(\# Context,\# Query,\# Response\) are illustrative only and are not part of the actual prompt\. The query is a question that asks for the target of a relation instance, and the model is required to generate the answer as the response\. We use two types of context\. The context is a relation statement expressed with a fixed sentence template\. We construct two types of relation statements for each query: a factual statement and a counterfactual statement\. A factual statement comes from the same relation as the query, but itssourceandtargetare different from those in the query; for example, when the query is “Which country is Tokyo in?”, the context could be “Paris is part of France\.” A counterfactual statement keeps thesourceof the query but replaces thetargetwith an incorrect one, such as “Tokyo is part of France\.”

For the WVS dataset, the prompt is as follows:

\# Context I would mention immigrants or foreign workers because I want neighbors who share my language, customs, and way of life, so daily contact feels easier and less stressful\. \# Query How important is family in your life? \# Response <Response\>

The queryqqis to ask about the respondent’s opinion, e\.g\., “How important is family in your life?” The context is an opinion statement used to respond to the queryq′q^\{\\prime\}; the opinion statement is either a support statement or an oppose statement for the queryq′q^\{\\prime\}\. The context can also be removed to obtain a no\-context baseline\.

#### Response Categories

To systematically characterize the effect of the contextccon model generation, we partition the evaluated responsesrrfor each queryqqinto three categories:

- •Context response: the response is identical to the context \(r=cr=c\); that is, the model is required to reproduce a sentence that appeared in the prompt;
- •Correct response: the response is the correct answer to the queryqqand is not equal to the context \(r≠cr\\neq candrr= correct\);444In our work, the context is never the correct answer\.
- •Incorrect response: the response neither equals the context nor is the correct answer \(r≠cr\\neq candr≠r\\neqcorrect\)\.

The exact form of the correct response depends on the dataset\. For LRE, it comprises both the gold response to the query and the no\-context natural response produced by the model itself\. For WVS, it comprises the matching support or oppose statement associated with the query and the no\-context natural response produced by the model\.

The exact form of the incorrect response also depends on the dataset\. For LRE, it is the alternative of the pair \{factual statement, counterfactual statement\} not used as the context\. For WVS, it is the alternative of the pair \{support statement, oppose statement\} that is not used as the context\.

### 4\.3Metrics

#### Contextual Entrainment

We measure this effect as the difference in the log\-probability that the model’s response is sentencess, under two prompting conditions: when the prompt is the concatenation of contextcc\(the same as the sentencess\) and queryqq, versus when the prompt consists of the queryqqalone\. Formally,

ℰ​\(c∣c,q\)\\displaystyle\\mathcal\{E\}\(c\\mid c,q\)=1\|c\|​∑i=1\|c\|log⁡π​\(ci∣c,q,c<i\)\\displaystyle=\\frac\{1\}\{\|c\|\}\\sum\_\{i=1\}^\{\|c\|\}\\log\\pi\(c\_\{i\}\\mid c,q,c\_\{<i\}\)\(12\)−1\|c\|​∑i=1\|c\|log⁡π​\(ci∣q,c<i\),\\displaystyle\-\\frac\{1\}\{\|c\|\}\\sum\_\{i=1\}^\{\|c\|\}\\log\\pi\(c\_\{i\}\\mid q,c\_\{<i\}\),where\|c\|\|c\|denotes the number of tokens in the contextcc, andcic\_\{i\}is itsii\-th token, so the average is taken over the\|c\|\|c\|teacher\-forced per\-token log\-probabilities ofcc\.

#### Distraction

Whereas Eq\. \([12](https://arxiv.org/html/2606.24077#S4.E12)\) measures the tendency of the model to reproduce the context sentence itself, distraction measures the effect of the contextccon the log\-probability of a responserrthat is not equal to the context \(r≠cr\\neq c\)\. Formally,

𝒟​\(r∣c,q\)\\displaystyle\\mathcal\{D\}\(r\\mid c,q\)=1\|r\|​∑i=1\|r\|log⁡π​\(ri∣c,q,r<i\)\\displaystyle=\\frac\{1\}\{\|r\|\}\\sum\_\{i=1\}^\{\|r\|\}\\log\\pi\(r\_\{i\}\\mid c,q,r\_\{<i\}\)\(13\)−1\|r\|​∑i=1\|r\|log⁡π​\(ri∣q,r<i\),\\displaystyle\-\\frac\{1\}\{\|r\|\}\\sum\_\{i=1\}^\{\|r\|\}\\log\\pi\(r\_\{i\}\\mid q,r\_\{<i\}\),where the responserrmay correspond to the correct or reasonable answer to the query or any other candidate sentence distinct from the context\. Note that our usage is broader than the behavioral notion of distraction in prior work\(Shiet al\.,[2023](https://arxiv.org/html/2606.24077#bib.bib35)\): Our distraction metric is a neutral measure of how the context shifts the probability of a response not equal to the context, and distraction in the conventional sense corresponds to the special case where distraction is negative for a correct response\.

### 4\.4Models

All experiments use open\-weight decoder\-only LLMs from Hugging Face\. For RQ1 and RQ3, we use Gemma\-2\-9BTeamet al\.\([2024](https://arxiv.org/html/2606.24077#bib.bib16)\), Llama\-3\.1\-8BGrattafioriet al\.\([2024](https://arxiv.org/html/2606.24077#bib.bib18)\), and Mistral\-7BJianget al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib20)\)\. Because the sentences we study are produced by models and may therefore be affected by instruction tuningKuribayashiet al\.\([2024](https://arxiv.org/html/2606.24077#bib.bib26)\), we compare the base and instruction\-tuned variants of three model families\. For RQ2, we use multiple sizes of four model families: Llama\-2\-7B/13B/70BTouvronet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib19)\), Qwen2\.5\-0\.5B/1\.5B/3B/7B/14B/32B/72BQwenet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib21)\), Qwen3\-0\.6B/1\.7B/4B/8B/14B/32BYanget al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib22)\), and Gemma\-3\-1B/4B/12B/27BTeamet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib17)\)\. We disable Qwen3’s thinking mode at inference time so that the scored continuation does not include the reasoning trace\.

### 4\.5Mask Training

We optimize the mask logits with AdamWLoshchilov and Hutter \([2017](https://arxiv.org/html/2606.24077#bib.bib27)\)at a learning rate of 0\.1, withα=1\.0\\alpha=1\.0andλ=2\.0\\lambda=2\.0for at most 500 epochs, with early stopping once the development set shows no improvement over 20 consecutive epochs\. For each relation, we split the samples into training, development, and test sets at an 80/10/10 ratio, and within every set, we form\(c,q\)\(c,q\)combinations by pairing each sample’s statement as the contextccwith another sample’s question as the queryqq\. We train a separate mask for each LRE relation, so each model produces 16 relation\-specific masks \(we call these masked headsper\-relation heads\)\. From these 16 per\-relation masks, we derive a single set ofshared headsfor each model: the heads turned off by at least 8 of the 16 per\-relation masks, i\.e\., the heads consistently identified as entrainment\-relevant across relations\.

## 5Experiments

### 5\.1Sentence\-Level Contextual Entrainment \(RQ1\)

Table 1:Model accuracy on the LRE dataset \(RQ1\)\. “no\-context” denotes the model’s natural response without context; “factual statement” uses the factual context; “counterfactual statement” uses the counterfactual context; “self” uses the model’s own “no\-context” natural response as context\.Basedenotes the base model andITthe instruction\-tuned model\.Blue shadingmarks the context condition with the smallest effect on accuracy\.Red shadingmarks the condition that reduces accuracy the most\.Table 2:Model opinion consistency on the WVS Dataset \(RQ1\)\.ρyx\\rho\_\{y\}^\{x\}denotes the consistency of opinion polarity between the model’s natural response \(of polarityxx\) given contextyyand its natural response without context;csc\_\{s\}andcoc\_\{o\}denote contexts that support and oppose the statement, whilensn\_\{s\}andnon\_\{o\}denote whether the model’s natural response is closer to the supporting or the opposing statement relative to queryqq\. “self” uses the model’s own “no\-context” natural response as context\. Each cell is the percentage of queries whose with\-context opinion still matches the no\-context opinion\.Yellow shadingmarks the context condition with the smallest effect on opinion consistency\.Gray shadingmarks the condition with the largest effect \(the context whose polarity opposes the model’s natural response\)\.![Refer to caption](https://arxiv.org/html/2606.24077v1/x3.png)Figure 2:Sentence\-level contextual entrainment on Llama\-3\.1\-8B \(RQ1\)\.Contextindicates the context type, which supports two settings: factual and counterfactual \(counter\-\);Responseindicates the text on which the log probability is computed\. Theredbars represent contextual entrainment, and thebluebars represent distraction\.#### Method\.

We examine how the mean log\-probability that the model assigns to the response tokens changes when the contextccis concatenated to the queryqq\. For the LRE dataset, we compute accuracy through string matching between the gold response and the model’s natural response\. To evaluate the model’s natural response on the WVS dataset, we classify its polarity through the OpenAI embedding API \(text\-embedding\-3\-small\),555[https://platform\.openai\.com/docs/guides/embeddings](https://platform.openai.com/docs/guides/embeddings)computing the cosine similarity between the natural response and the support and oppose statements associated with the query\. Then, we assign the polarity of the statement with the higher similarity as the polarity of the natural response\.

#### Results\.

Figure[2](https://arxiv.org/html/2606.24077#S5.F2)reports the mean log\-probability shift of Llama\-3\.1\-8B on the LRE dataset\. Results on other models and the WVS dataset also show the same pattern\. Table[1](https://arxiv.org/html/2606.24077#S5.T1)shows the effect of context on the accuracy of the model’s natural response\.

\(i\) Sentence\-level contextual entrainment exists, including under counterfactual statements\.In Figure[2](https://arxiv.org/html/2606.24077#S5.F2), the cases where the response equals the context are instances of contextual entrainment\. We find that sentence\-level contextual entrainment exists, meaning that a sentence appearing in the prompt receives a higher probability in the model response\. Factual and counterfactual statements also raise each other’s probability, and this effect can even exceed the probability gain on the gold answer\. This is counterintuitive, as it indicates that counterfactual statements \(incorrect examples\) can also increase the probability that the model continues to generate the correct responses\. The observation is consistent with prior work reporting that few\-shot demonstrations with incorrect labels still improve model performanceMinet al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib2)\); Liu and Chu \([2025](https://arxiv.org/html/2606.24077#bib.bib10)\)\. The probability of the model’s natural response is distracted, regardless of whether the context is factual or counterfactual, which confirms that the influence of the context on the model output genuinely exists\.

![Refer to caption](https://arxiv.org/html/2606.24077v1/x4.png)\(a\)Support statement\.
![Refer to caption](https://arxiv.org/html/2606.24077v1/x5.png)\(b\)Counter\- and Factual\.
![Refer to caption](https://arxiv.org/html/2606.24077v1/x6.png)\(c\)Support response\.
![Refer to caption](https://arxiv.org/html/2606.24077v1/x7.png)\(d\)Oppose statement\.

Figure 3:Model\-size effects on sentence\-level contextual entrainment \(RQ2\)\. The context for \(a\), \(c\), and \(d\) issupport statement; the caption represents the response\. \(b\) is that the response equals the context, under factual and counterfactual contexts\. Model family encoding:Llama\-2,Qwen3,Qwen2\.5,Gemma\-3\. The y\-axis shows the “Δ\\DeltaMean Log Probability” under a specific context and response setting\.\(ii\) The shifts in log\-probability translate into behavior\-level changes in correctness under free generation\.Tables[1](https://arxiv.org/html/2606.24077#S5.T1)and[2](https://arxiv.org/html/2606.24077#S5.T2)report how different context types affect the model’s natural response: accuracy on the LRE dataset and polarity consistency on the WVS dataset\. On the LRE dataset, every model decreases a substantial portion of its accuracy under factual context; the decrease is larger under counterfactual context, where Llama\-3\.1\-8B drops from 79\.4% to 22\.4%; accuracy falls even when the context is the model’s own natural response\. On the WVS dataset, the consistency of the opinion’s polarity decreases most when the polarity of the opinion statement in the context conflicts with that of the model’s natural response\. However, even a same\-polarity context, or one that reuses the model’s natural response itself, still affects the model’s opinion consistency, and natural responses that take an opposing polarity are more susceptible\. Taken together, the context is not a spurious effect at the log\-probability level but instead affects the opinion polarity of the model’s generation\.

\(iii\) Instruction\-tuning mitigates the effects of context\.As shown in Table[1](https://arxiv.org/html/2606.24077#S5.T1), instruction tuning substantially mitigates the influence of context on model performance on the LRE dataset: the base\-model drops of \-25\.5%, \-31\.9%, and \-13\.5% shrink to \-15\.3%, \-9\.5%, and \-4\.5% after tuning\. Yet, although the instruction\-tuned models cut the average degradation by 40% to 70% within each family, they never eliminate it\. On WVS \(Table[2](https://arxiv.org/html/2606.24077#S5.T2)\), instruction tuning yields inconsistent effects across families relative to the base models, lowering consistency on Gemma\-2\-9B while raising it on Llama\-3\.1\-8B and Mistral\-7B; the highest average consistency it attains is only 80\.3%\. Contextual entrainment, therefore, persists after instruction tuning, which shows that it is not an artifact of an insufficiently aligned base model\.

### 5\.2Model\-size Effects on Contextual Entrainment \(RQ2\)

#### Motivation\.

Because LLMs are contextual representation modelsVaswaniet al\.\([2017](https://arxiv.org/html/2606.24077#bib.bib23)\); Devlinet al\.\([2019](https://arxiv.org/html/2606.24077#bib.bib24)\); Radfordet al\.\([2018](https://arxiv.org/html/2606.24077#bib.bib25)\), sentence\-level contextual entrainment, unlike token\-level contextual entrainment, depends on the representational capacity of the model\. In this section, we therefore report the contextual entrainment and distraction for 20 models of varying sizes drawn from four model families \(Llama\-2, Qwen2\.5, Qwen3, and Gemma\-3\), evaluated across a range of context and response settings\.

#### Results\.

Figure[3](https://arxiv.org/html/2606.24077#S5.F3)reports how the log\-probability change of the response, induced by the appearance of the context in the prompt, varies across three context and response settings on the LRE and WVS dataset\. A larger value indicates that the context contributes more to raising the probability of the response\.

\(i\) Contextual entrainment decreases as model size increases\.Figure[3\(a\)](https://arxiv.org/html/2606.24077#S5.F3.sf1)shows that contextual entrainment decreases as the model size increases: on Qwen3, it falls from 3\.91 at 0\.6B to 3\.35 at 32B\. On LRE \(Figure[3\(b\)](https://arxiv.org/html/2606.24077#S5.F3.sf2)\), the factual context follows the same decreasing trend, whereas the counterfactual context deviates from this pattern, with the value instead increasing slightly with model size\. This indicates that the model resists repeating counterfactual context, and this resistance is more pronounced in smaller models\.

Table 3:Contextual entrainment averaged over 16 LRE relations \(RQ3\)\. “Unmasked” is the original model without any attention heads masked\. “Random head” is averaged over 10 seeds of random head sets matched in size to the set of shared heads\. “Per\-relation head” uses each relation’s own trained mask; “Shared head” uses the shared head mask \(2%\-4% of attention heads\)\.Green shadingmarks the per\-relation mask, which almost entirely eliminates contextual entrainment;Orange shadingmarks the shared head mask, which roughly halves entrainment while preserving model performance\.Table 4:Free\-generation accuracy on the 16 LRE relations \(RQ3\)\. “w/o context” is the bare query \(capability preservation\); “w/ context” is the query preceded by the factual statement \(entrainment robustness\)\. Subscripts give the change in accuracy relative to the “Unmasked” row\.Boldmarks the highest free\-generation accuracy among the three masking strategies\.\(ii\) Distraction increases as model size increases\.Figures[3\(c\)](https://arxiv.org/html/2606.24077#S5.F3.sf3)and[3\(d\)](https://arxiv.org/html/2606.24077#S5.F3.sf4)illustrate this trend in two types of distracted responses\. For the response matching the query \(Figure[3\(c\)](https://arxiv.org/html/2606.24077#S5.F3.sf3)\), the influence of the support statement on the support response increases with model size: on Qwen3, it rises from 0\.06 at 0\.6B to 0\.27 at 32B, and the same trend holds for Qwen2\.5, Gemma\-3, and Llama\-2; For a response that neither appears in the prompt nor matches the query \(Figure[3\(d\)](https://arxiv.org/html/2606.24077#S5.F3.sf4)\), the probability gain likewise increases with model size within every model family, from roughly 0\.4 to 0\.6 at the smallest size to roughly 0\.8 to 1\.0 at the largest\. We interpret this as follows: as the context and the response are expressions drawn from a similar topic, a larger model is better able to extract topical or stylistic features from the context and use them to raise the probability of a topically similar response; when the response equals the context, however, token\-level entrainment overrides this representational effect\. We can conclude thatsmaller models rely more on token\-level contextual entrainment, while larger models rely more on representation\-level contextual entrainment\.

### 5\.3Entrainment Heads \(RQ3\)

#### Results\.

After identifying the entrainment heads through differentiable masking, we validate whether turning them off suppresses contextual entrainment and at what cost to task performance\.

\(i\) Only 2%\-4% of attention heads suffice to suppress contextual entrainment\.Table[3](https://arxiv.org/html/2606.24077#S5.T3)shows that a random masking baseline that uses the same number of attention heads barely affects contextual entrainment \(2\.01 on average, essentially unchanged from the unmasked model\)\. Per\-relation masking, in contrast, almost entirely eliminates it, driving contextual entrainment down to roughly zero \(\+0\.05 on average, slightly negative on several models\)\. Masking the shared heads roughly halves the contextual entrainment of every model: when no heads are masked, the average contextual entrainment is 2\.02; however, when only 2%–4% of the shared heads are masked, this value drops to 1\.11\. The shared entrainment heads are therefore both sparse and general across relations\.

\(ii\) Masking the shared heads barely hurts model performance\.Table[4](https://arxiv.org/html/2606.24077#S5.T4)shows how masking heads affects accuracy in the no\-context and with\-context settings\. In the no\-context setting, masking the shared heads costs only a few percentage points of accuracy on average \(\-2\.3\), well below the loss incurred by per\-relation masking \(\-6\.8\) and by a random mask of the same size \(\-4\.1\)\. The performance drop primarily occurred in the base models, whereas the instruction\-tuned variants across all three families retain essentially all of their no\-context accuracy when the shared heads are masked\. Shared heads can therefore be disabled at deployment without retraining or sacrificing the performance of the underlying model\. In the with\-context setting, the unmasked model is distracted by the context, and its accuracy falls below the no\-context level\. Masking the shared heads improves with\-context accuracy on five of the six models \(\+1\.5 to \+7\.9\), and the recovery is most complete on the instruction\-tuned variants, whose accuracy returns to within a few percentage points of the no\-context ceiling\. A random mask of the same size produces no such recovery, which confirms that the effect comes from these specific heads rather than from the general perturbation of removing an arbitrary 2% to 4% of the heads\.

## 6Related Work

#### Contextual Entrainment and Distraction\.

LLMs are known to be susceptible to distraction: irrelevant or misleading content in the prompt can derail an otherwise correct prediction\.Shiet al\.\([2023](https://arxiv.org/html/2606.24077#bib.bib35)\)shows that adding a single irrelevant clause to grade\-school math problems substantially degrades the accuracy of models that otherwise solve them reliably\. This vulnerability has drawn particular attention in retrieval\-augmented generation, where retrieved passages are inevitably noisy: irrelevant or distracting documents in the context degrade answer quality, and models need to be explicitly hardened against them\(Yoranet al\.,[2023](https://arxiv.org/html/2606.24077#bib.bib36); Cuconasuet al\.,[2024](https://arxiv.org/html/2606.24077#bib.bib37)\)\. The Contextual Entrainment phenomenon describes how an LLM increases the probability of any token that appears in the prompt, regardless of its relevance to the query\.Niuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)attributes this entrainment to a small set of entrainment heads\. While their finding is novel, they experiment on only a single model, Llama\-3\.1\-8B\.Kukrejaet al\.\([2026](https://arxiv.org/html/2606.24077#bib.bib28)\)instead studies how contextual entrainment scales across models of different sizes, yet their analysis is confined to small and medium models\. BothNiuet al\.\([2025](https://arxiv.org/html/2606.24077#bib.bib6)\)andKukrejaet al\.\([2026](https://arxiv.org/html/2606.24077#bib.bib28)\)measure the logit increases of a single next\-token candidate, whereas in actual generation, entrainment accumulates over an entire sentence\. Prior work also tests entrainment only on the LRE dataset, which leaves open whether the phenomenon shapes subjective behavior as well\. We add the WVS dataset as a second probe and show that entrainment impacts both factual answers and expressed opinions\.

#### Attention Heads and Mechanistic Interpretability\.

Mechanistic interpretability has identified attention\-head circuits that implement specific functions in LLMs\.Elhageet al\.\([2021](https://arxiv.org/html/2606.24077#bib.bib29)\)andOlssonet al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib30)\)identify induction heads, which complete patterns of the form “A​B​…​A→BAB\\ldots A\\to B” and drive in\-context learning, andCrosbie and Shutova \([2025](https://arxiv.org/html/2606.24077#bib.bib31)\)extend this analysis to real\-world LLMs\. Beyond pattern completion, narrower circuits have been localized to specific tasks, as in the indirect object identification circuit ofWanget al\.\([2022](https://arxiv.org/html/2606.24077#bib.bib32)\)\. A common methodology is differentiable masking through a Gumbel\-softmax relaxation, which attaches a learnable gate to each component in order to discover the minimal set of components that explains a target behavior\. Most prior applications aim to explain a correct prediction\. We instead apply attention\-head masking as a suppressive intervention, searching for a minimal set of heads whose removal eliminates an unwanted behavior, namely sentence\-level contextual entrainment, and we ask whether the heads found in this way are general across relations\.

## 7Conclusion

We extended token\-level contextual entrainment to sentence\-level and studied it on an objective task \(LRE\) and a subjective task \(WVS\)\. Extensive experiments show that the model increases the probability that the context in the prompt, including counterfactual context, appears in its continuation\. When the context is an opinion statement, the model increases the probability of generating opinion statements of both the same and opposite polarity in its continuation\. Contextual entrainment varies predictably with model size: it decreases as the model size increases, whereas the probability assigned to query\-relevant response candidates increases\. Contextual entrainment is controlled by a small set of shared heads\. Masking these heads mitigates contextual entrainment by about half while leaving model performance essentially unchanged, which makes the shared heads a concrete and minimal intervention point for mitigating the phenomenon\.

## References

- Y\. Bengio, N\. Léonard, and A\. Courville \(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.16)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational conference on machine learning,pp\. 2397–2430\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p1.1)\.
- J\. Crosbie and E\. Shutova \(2025\)Induction heads as an essential mechanism for pattern matching in in\-context learning\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5034–5096\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p2.1),[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px2.p1.1)\.
- F\. Cuconasu, G\. Trappolini, F\. Siciliano, S\. Filice, C\. Campagnano, Y\. Maarek, N\. Tonellotto, and F\. Silvestri \(2024\)The power of noise: redefining retrieval for rag systems\.InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 719–729\.Cited by:[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px1.p1.1)\.
- D\. Dai, Y\. Sun, L\. Dong, Y\. Hao, S\. Ma, Z\. Sui, and F\. Wei \(2023\)Why can gpt learn in\-context? language models secretly perform gradient descent as meta\-optimizers\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4005–4019\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p2.1)\.
- N\. De Cao, M\. Schlichtkrull, W\. Aziz, and I\. Titov \(2020\)How do decisions emerge across layers in neural models? interpretation with differentiable masking\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 3243–3255\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.4)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)Bert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§5\.2](https://arxiv.org/html/2606.24077#S5.SS2.SSS0.Px1.p1.1)\.
- N\. Dey, G\. Gosal, H\. Khachane, W\. Marshall, R\. Pathria, M\. Tom, J\. Hestness,et al\.\(2023\)Cerebras\-gpt: open compute\-optimal language models trained on the cerebras wafer\-scale cluster\.arXiv preprint arXiv:2304\.03208\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1)\.
- C\. Dugas, Y\. Bengio, F\. Bélisle, C\. Nadeau, and R\. Garcia \(2000\)Incorporating second\-order functional knowledge for better option pricing\.Advances in neural information processing systems13\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.31)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread1\(1\),pp\. 12\.Cited by:[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.31)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1),[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- C\. Haerpfer, R\. Inglehart, A\. Moreno, C\. Welzel, K\. Kizilova, J\. Diez\-Medrano, M\. Lagos, P\. Norris, E\. Ponarin, and B\. Puranen \(2022\)World values survey: round seven – country\-pooled datafile version 6\.0\.JD Systems Institute & WVSA Secretariat,Madrid, Spain & Vienna, Austria\.Note:DatasetExternal Links:[Document](https://dx.doi.org/10.14281/18241.24),[Link](https://doi.org/10.14281/18241.24)Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.24077#S4.SS1.SSS0.Px2)\.
- E\. Hernandez, A\. Sen Sharma, T\. Haklay, K\. Meng, M\. Wattenberg, J\. Andreas, Y\. Belinkov, and D\. Bau \(2024\)Linearity of relation decoding in transformer language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 10504–10526\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1),[§4\.1](https://arxiv.org/html/2606.24077#S4.SS1.SSS0.Px1)\.
- E\. Jang, S\. Gu, and B\. Poole \(2017\)Categorical reparametrization with gumble\-softmax\.InInternational Conference on Learning Representations \(ICLR 2017\),Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.4)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- D\. Kukreja, K\. Sah, G\. Gupta, A\. Anand, R\. R\. Shah, Z\. Wang, A\. B\. Ng, and E\. Cambria \(2026\)Better and worse with scale: how contextual entrainment diverges with model size\.External Links:2604\.13275,[Link](https://arxiv.org/abs/2604.13275)Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p5.1),[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px1.p1.1)\.
- S\. Kullback and R\. A\. Leibler \(1951\)On information and sufficiency\.The annals of mathematical statistics22\(1\),pp\. 79–86\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.31)\.
- T\. Kuribayashi, Y\. Oseki, and T\. Baldwin \(2024\)Psychometric predictive power of large language models\.InFindings of the Association for Computational Linguistics: NAACL 2024,pp\. 1983–2005\.Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- T\. Li, X\. Ma, A\. Zhuang, Y\. Gu, Y\. Su, and W\. Chen \(2023\)Few\-shot in\-context learning on knowledge base question answering\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 6966–6980\.External Links:[Link](https://aclanthology.org/2023.acl-long.385/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.385)Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p1.1)\.
- Y\. Liu and C\. Chu \(2025\)Do LLMs align human values regarding social biases? judging and explaining social biases with LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 21591–21628\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1178/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1178),ISBN 979\-8\-89176\-335\-7Cited by:[§5\.1](https://arxiv.org/html/2606.24077#S5.SS1.SSS0.Px2.p2.1),[footnote 2](https://arxiv.org/html/2606.24077#footnote2)\.
- Y\. Liu and Y\. Hou \(2023\)Mining effective features using quantum entropy for humor recognition\.InFindings of the Association for Computational Linguistics: EACL 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 2048–2053\.External Links:[Link](https://aclanthology.org/2023.findings-eacl.152/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.152)Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.31)\.
- Y\. Liu, M\. Kaneko, and C\. Chu \(2026\)On the alignment of large language models with global human opinion\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 37673–37681\.Cited by:[§4\.1](https://arxiv.org/html/2606.24077#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Liu \(2024\)Robust evaluation measures for evaluating social biases in masked language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 18707–18715\.Cited by:[§3](https://arxiv.org/html/2606.24077#S3.p1.31)\.
- I\. Loshchilov and F\. Hutter \(2017\)Decoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§4\.5](https://arxiv.org/html/2606.24077#S4.SS5.p1.5)\.
- S\. Min, X\. Lyu, A\. Holtzman, M\. Artetxe, M\. Lewis, H\. Hajishirzi, and L\. Zettlemoyer \(2022\)Rethinking the role of demonstrations: what makes in\-context learning work?\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 11048–11064\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.759/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.759)Cited by:[§5\.1](https://arxiv.org/html/2606.24077#S5.SS1.SSS0.Px2.p2.1)\.
- J\. Niu, X\. Yuan, T\. Wang, H\. Saghir, and A\. H\. Abdi \(2025\)Llama see, llama do: a mechanistic perspective on contextual entrainment and distraction in LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16218–16239\.External Links:[Link](https://aclanthology.org/2025.acl-long.791/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.791),ISBN 979\-8\-89176\-251\-0Cited by:[1\(a\)](https://arxiv.org/html/2606.24077#S1.F1.sf1),[1\(a\)](https://arxiv.org/html/2606.24077#S1.F1.sf1.3.2),[§1](https://arxiv.org/html/2606.24077#S1.p3.1),[§1](https://arxiv.org/html/2606.24077#S1.p4.1),[§1](https://arxiv.org/html/2606.24077#S1.p5.1),[1st item](https://arxiv.org/html/2606.24077#S2.I1.i1.p1.1),[§2\.2](https://arxiv.org/html/2606.24077#S2.SS2.p3.1),[§3](https://arxiv.org/html/2606.24077#S3.p1.4),[§4\.1](https://arxiv.org/html/2606.24077#S4.SS1.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px1.p1.1),[footnote 2](https://arxiv.org/html/2606.24077#footnote2)\.
- C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, S\. Johnston, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah \(2022\)In\-context learning and induction heads\.External Links:2209\.11895,[Link](https://arxiv.org/abs/2209.11895)Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p2.1),[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px2.p1.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- A\. Radford, K\. Narasimhan, T\. Salimans, I\. Sutskever,et al\.\(2018\)Improving language understanding by generative pre\-training\.Cited by:[§5\.2](https://arxiv.org/html/2606.24077#S5.SS2.SSS0.Px1.p1.1)\.
- F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. Zhou \(2023\)Large language models can be easily distracted by irrelevant context\.InInternational Conference on Machine Learning,pp\. 31210–31227\.Cited by:[§4\.3](https://arxiv.org/html/2606.24077#S4.SS3.SSS0.Px2.p1.4),[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px1.p1.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. C\. Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. Scialom \(2023\)Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§5\.2](https://arxiv.org/html/2606.24077#S5.SS2.SSS0.Px1.p1.1)\.
- K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. Steinhardt \(2022\)Interpretability in the wild: a circuit for indirect object identification in gpt\-2 small\.arXiv preprint arXiv:2211\.00593\.Cited by:[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.4](https://arxiv.org/html/2606.24077#S4.SS4.p1.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2023\)Making retrieval\-augmented language models robust to irrelevant context\.arXiv preprint arXiv:2310\.01558\.Cited by:[§6](https://arxiv.org/html/2606.24077#S6.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh \(2021\)Calibrate before use: improving few\-shot performance of language models\.InInternational conference on machine learning,pp\. 12697–12706\.Cited by:[§1](https://arxiv.org/html/2606.24077#S1.p1.1)\.