从复制到组合：语言模型如何发展出上下文学习能力

# How Language Models Develop In-Context Learning
Source: [https://arxiv.org/html/2610.19113](https://arxiv.org/html/2610.19113)
## From Copying to Composition: How Language Models Develop In\-Context Learning

Minglai Yang Xinyu Guo Renxiong Wang Sunjie Hou Daniel Yue Zhang Yunzhong He††thanks:Corresponding author\.

###### Abstract

In\-context learning \(ICL\) allows language models to adapt their behavior from instructions and demonstrations in a prompt, but accuracy alone does not reveal how contextual information controls their predictions\. We study the acquisition and trainability of ICL with a controlled diagnostic that separates copying a nearby demonstration from integrating information across the prompt\. Across 17 models, about a third prefer the copied answer, a few are undecided, and the rest prefer the rule\-consistent answer, which goes with higher accuracy and with control by the requested value rather than the closest conflicting example\. Checkpoint analyses show that models begin prioritizing task\-relevant prompt tokens over conflicting examples long before their accuracy improves beyond chance level\. In matched training experiments, synthetic documents that define endings and then query them shift models toward compositional responses on the controlled diagnostic, whereas fine\-tuning on definition exercises built from rewritten real documents improves accuracy when prompts reassign label meanings\. The results suggest that ICL is not acquired as a single, uniform capability\. Instead, prompt control, answer preference, construction without candidates, and transfer can develop along distinct trajectories\.

Figure 1:Study overview\. \(a\) The task makes copying and applying prompt definitions predict different answers\. \(b\) Across pretraining checkpoints \(OLMo\-2\-7B shown\), the requested value overtakes the closest ending as the source of the answer \(star\) long before accuracy crosses chance \(dashed line\)\. \(c\) Fine\-tuning SmolLM2\-1\.7B on definition exercises rewritten from real documents raises accuracy on three unseen datasets with made\-up or exchanged labels \(pale bars: a matched fine\-tune on the original documents, solid extensions: the gain from the exercises\)\.## 1Introduction

Language models are increasingly expected to adapt at inference time, from instructions, demonstrations, and temporary conventions in the prompt rather than from parameter updates\. This ability is commonly described as in\-context learning \(ICL\)\. We know much less about how this capacity is acquired during training than about how well trained models display it\. Understanding ICL acquisition requires tracking how models shift their reliance on contextual information throughout training\. Knowing when accuracy improves is not enough\.

The same answer can result from different uses of the context: a model may reuse the output pattern of a demonstration similar to the query, or integrate mappings distributed across the prompt and apply them to a new combination\. In ordinary ICL evaluations these behaviors predict the same output, so correctness shows that the context was useful but not how it was used\. Studying the acquisition of ICL therefore needs a diagnostic in which different ways of using the prompt make different predictions\.

These observations motivate a developmental question that final\-answer accuracy alone cannot resolve: how does a model’s use of different sources of contextual information change during training? We track four aspects of ICL that, as we show, develop on different schedules: which prompt source controls the answer, which answer the model prefers, whether it can construct that answer without candidates, and whether a trained change transfers to other evaluations\.

To make these aspects distinguishable, we construct a controlled task inspired by work on compositional generalization and made\-up\-word tests\([Hupkes et al\., 2020](https://arxiv.org/html/2610.19113#bib.bib34);[Berko, 1958](https://arxiv.org/html/2610.19113#bib.bib1)\)\. Each prompt defines endings for two binary attributes, demonstrates three of the four possible combinations on made\-up words, and asks for the missing combination on a new word \(Figure[1](https://arxiv.org/html/2610.19113#S0.F1)\)\. Combining the two prompt\-defined mappings yields the rule\-consistent \(correct\) answer, whereas reusing the conflicting ending from the demonstrated combination that shares the first requested attribute yields a distinct copied answer\. Because the assignments change in every prompt, the task cannot be solved using a fixed word–ending mapping\. We measure the copied answer’s probability normalized over these two candidates, denoting it byP⁡\(copy\)P\(\\mathrm\{copy\}\)\. Interventions on the conflicting ending in the closest example or on the requested value in the question measure which source controls the candidate\-score difference, and activation patching\([Heimersheim and Nanda, 2024](https://arxiv.org/html/2610.19113#bib.bib38)\)and the disabling of selected attention heads\([Wang et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib51)\)trace that control to internal states and heads\.

Across 17 models, the diagnostic sorts answer preferences from copying the nearest demonstration to producing the correct form, and the composing models score higher and generate the form without candidates\. Across pretraining checkpoints, sensitivity to the requested value overtakes sensitivity to the conflicting example hundreds of billions of tokens before accuracy rises above chance\. Most other checkpoint series that reach composition show the same ordering, OLMo\-3 and Comma on every criterion \(Section[5](https://arxiv.org/html/2610.19113#S5)\)\.

Matched continued\-training experiments then show which training signals change these behaviors and where the improvements transfer\. Training on synthetic documents that define endings and then query them reverses the preferred answer on the controlled task and lets models generate the missing form\. Fine\-tuning on definition exercises built from rewritten real documents improves performance on evaluations that assign new meanings to labels\. Each signal transfers the use of the prompt it rewards, so the exercises leave the controlled task unchanged\.

Our work makes three contributions\. First, we introduce a controlled diagnostic in which two ways of using the context predict different answers\. Second, we measure answer preference and prompt dependence across 17 models and multiple checkpoint series showing that prompt control shifts long before accuracy crosses chance\. Third, matched continued\-training experiments show that the compositional answer must be built rather than unblocked: suppressing copy\-favoring heads shifts the balance between candidates, training on synthetic documents that define endings and then query them creates the missing answer, and each signal transfers the use of the prompt it rehearses \(Section[6](https://arxiv.org/html/2610.19113#S6)\)\.

## 2Related work

#### In\-context learning and composition\.

Accounts of in\-context learning include copying through induction\-like mechanisms\([Olsson et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib6);[Bietti et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib50);[Reddy, 2024](https://arxiv.org/html/2610.19113#bib.bib56);[Akyürek et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib7)\), inference over a latent task, and an update during the forward pass\([Xie et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib41);[von Oswald et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib42)\)\. ICL behavior varies with training distribution and stage\([Chan et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib43);[Singh et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib48)\)\. Controlled tasks show strategies between repetition and exact rule use\([Niu et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib8);[Bhattamishra et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib57)\), and models can recognize a task without learning it\([Yang et al\., 2025b](https://arxiv.org/html/2610.19113#bib.bib25);[Kim, 2025](https://arxiv.org/html/2610.19113#bib.bib24)\)\. Our task connects this issue to compositional generalization and made\-up\-word tests\([Hupkes et al\., 2020](https://arxiv.org/html/2610.19113#bib.bib34);[Lake and Baroni, 2018](https://arxiv.org/html/2610.19113#bib.bib19);[Kim and Linzen, 2020](https://arxiv.org/html/2610.19113#bib.bib44);[Lake and Baroni, 2023](https://arxiv.org/html/2610.19113#bib.bib55);[Berko, 1958](https://arxiv.org/html/2610.19113#bib.bib1);[Weissweiler et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib2)\), and to work on prompt\-defined labels\([Wei et al\., 2023a](https://arxiv.org/html/2610.19113#bib.bib63);[Wei et al\., 2023b](https://arxiv.org/html/2610.19113#bib.bib62)\)\. Morphology offers a long\-studied case of the same contrast between applying a rule and analogizing to stored forms\([Pinker and Prince, 1988](https://arxiv.org/html/2610.19113#bib.bib27);[Albright and Hayes, 2003](https://arxiv.org/html/2610.19113#bib.bib28)\), and recent work asks which of the two language models do\([Hofmann et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib4);[Anh et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib3);[Ismayilzada et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib5)\)\. Tokenization also shapes what counts as a morpheme for a model\([Sennrich et al\., 2016](https://arxiv.org/html/2610.19113#bib.bib71);[Rust et al\., 2021](https://arxiv.org/html/2610.19113#bib.bib35)\), which our Quechua results bear out\. We make copying a similar example and applying the supplied mappings predict different answers, then compare their scores and intervene on the supporting prompt tokens\. Attention weights provide incomplete explanations of model behavior\([Jain and Wallace, 2019](https://arxiv.org/html/2610.19113#bib.bib39);[Serrano and Smith, 2019](https://arxiv.org/html/2610.19113#bib.bib40)\)\. Our interventions directly test which prompt tokens change the candidate scores\.

#### Mechanisms of context use\.

Mechanistic studies locate parts of in\-context behavior in attention heads\. Induction and retrieval heads copy from the context\([Olsson et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib6);[Wu et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib60)\), a copy\-suppression head counteracts them\([McDougall et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib49)\), token\-level and concept\-level copying travel separate routes\([Feucht et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib59)\), and a set of demonstrations can be summarized into a task or function vector\([Hendel et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib45);[Todd et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib46)\)\. Our interventions follow this tradition of patching and ablation\([Geiger et al\., 2021](https://arxiv.org/html/2610.19113#bib.bib31);[Meng et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib37);[Goldowsky\-Dill et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib47);[Wu et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib32)\), but we ask a different question: which prompt span controls the answer at each stage of training\.

#### Development during training\.

Checkpoint studies usually measure abilities by accuracy gains\([Chen et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib16);[Evanson et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib15);[Tigges et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib17);[Arora et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib18)\)\. Work on combined abilities finds that they emerge after their components\([Liu et al\., 2026](https://arxiv.org/html/2610.19113#bib.bib20)\), while other studies track reasoning, arithmetic, grammar, and internal components across training\([Ye et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib22);[Zhao et al\., 2026](https://arxiv.org/html/2610.19113#bib.bib26);[Chang and Bergen, 2022](https://arxiv.org/html/2610.19113#bib.bib14);[Pan et al\., 2026](https://arxiv.org/html/2610.19113#bib.bib23)\)\. We instead track copied\-versus\-correct scores and prompt\-token dependence at every checkpoint\. We extend work that localizes or modifies mechanisms for context use\([Yin et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib58);[Prakash et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib61)\)to the training trajectory\.

## 3Task and measurements

#### Task construction\.

Figure[1](https://arxiv.org/html/2610.19113#S0.F1)a shows one prompt\. Each prompt assigns a fresh ending to each of four feature values, shows three of the four value combinations, and asks for the missing combination on a new stem\. In the example, applying the definitions giveswug\-or\-nak, whereas copying the closest example,big one zud = zud\-or\-um, giveswug\-or\-um\. The words, endings, and mappings are sampled anew for every prompt\. A second scheme uses short codes without a made\-up word, and a reverse\-order version serves as a control \(Appendices[A](https://arxiv.org/html/2610.19113#A1)and[C](https://arxiv.org/html/2610.19113#A3)\)\.

#### Formalization and scores\.

Let𝒰=\{u0,u1\}\\mathcal\{U\}=\\\{u\_\{0\},u\_\{1\}\\\}and𝒱=\{v0,v1\}\\mathcal\{V\}=\\\{v\_\{0\},v\_\{1\}\\\}be the feature domains\. Each prompt defines a mapϕ:𝒰⊔𝒱→ℰ\\phi:\\mathcal\{U\}\\sqcup\\mathcal\{V\}\\rightarrow\\mathcal\{E\}from their four values to four endings, with

fϕ​\(s,u,v\)=s∥ϕ⁡\(u\)∥ϕ⁡\(v\),f\_\{\\phi\}\(s,u,v\)=s\\mathbin\{\\\|\}\\phi\(u\)\\mathbin\{\\\|\}\\phi\(v\),\(1\)where∥\\\|denotes concatenation\. Equation[1](https://arxiv.org/html/2610.19113#S3.E1)composes the two prompt\-defined endings in feature order\. The query requests a held\-out pair\(u⋆,v⋆\)\(u^\{\\star\},v^\{\\star\}\)on a new stems⋆s^\{\\star\}\. Ifv¯⋆\\bar\{v\}^\{\\star\}is the other value in𝒱\\mathcal\{V\}, the closest example supplies\(u⋆,v¯⋆\)\(u^\{\\star\},\\bar\{v\}^\{\\star\}\)\. In Figure[1](https://arxiv.org/html/2610.19113#S0.F1)a,u⋆=bigu^\{\\star\}=\\textit\{big\},v⋆=manyv^\{\\star\}=\\textit\{many\}, andv¯⋆=one\\bar\{v\}^\{\\star\}=\\textit\{one\}\. The rule\-consistent and copied candidates are therefore

yr=fϕ​\(s⋆,u⋆,v⋆\),yc=fϕ​\(s⋆,u⋆,v¯⋆\)\.y\_\{r\}=f\_\{\\phi\}\(s^\{\\star\},u^\{\\star\},v^\{\\star\}\),\\qquad y\_\{c\}=f\_\{\\phi\}\(s^\{\\star\},u^\{\\star\},\\bar\{v\}^\{\\star\}\)\.\(2\)The candidates in Equation[2](https://arxiv.org/html/2610.19113#S3.E2)share the stem and first ending and differ only where copying conflicts with the requested value\.

For promptxx, letpr=pθ​\(yr∣x\)p\_\{r\}=p\_\{\\theta\}\(y\_\{r\}\\mid x\)andpc=pθ​\(yc∣x\)p\_\{c\}=p\_\{\\theta\}\(y\_\{c\}\\mid x\)\. We measure their competition by

m\\displaystyle m=log⁡pr−log⁡pc,\\displaystyle=\\log p\_\{r\}\-\\log p\_\{c\},\(3\)P⁡\(copy\)\\displaystyle P\(\\mathrm\{copy\}\)=pcpc\+pr=σ⁡\(−m\)\.\\displaystyle=\\frac\{p\_\{c\}\}\{p\_\{c\}\+p\_\{r\}\}=\\sigma\(\-m\)\.Positivemm, equivalentlyP⁡\(copy\)<\.5P\(\\mathrm\{copy\}\)<\.5, favors the correct form\. Four\-choice accuracy additionally requiresyry\_\{r\}to outrank all four combinations, while exact generation supplies no candidates\.

#### Blocking and the switch of control\.

We block attention from the answer position to selected tokens without deleting them\([Geva et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib36)\)\(Figure[5](https://arxiv.org/html/2610.19113#S4.F5)a\)\. LetKSK\_\{S\}denote this intervention for spanSSandΔS​m=mθ​\(KS​x\)−mθ​\(x\)\\Delta\_\{S\}m=m\_\{\\theta\}\(K\_\{S\}x\)\-m\_\{\\theta\}\(x\)\. For the closest endingSeS\_\{e\}, the conflicting ending in the closest example \(\-um\), and the requested valueSqS\_\{q\}\(many\), define

di,τ\\displaystyle d\_\{i,\\tau\}=\|ΔSe​mi,τ\|−\|ΔSq​mi,τ\|,\\displaystyle=\|\\Delta\_\{S\_\{e\}\}m\_\{i,\\tau\}\|\-\|\\Delta\_\{S\_\{q\}\}m\_\{i,\\tau\}\|,\(4\)Dτ\\displaystyle D\_\{\\tau\}=1n​∑i=1ndi,τ\.\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}d\_\{i,\\tau\}\.PositiveDτD\_\{\\tau\}in Equation[4](https://arxiv.org/html/2610.19113#S3.E4)indicates greater control by the closest ending, and negativeDτD\_\{\\tau\}greater control by the requested value\. Across checkpointsτ1<⋯<τJ\\tau\_\{1\}<\\cdots<\\tau\_\{J\}, the switch of control is the first adjacent bracket\[τj−1,τj\]\[\\tau\_\{j\-1\},\\tau\_\{j\}\]in which meanDDchanges from positive to negative\. Other prompt lines and unrelated tokens serve as blocking controls\.

#### Activation patching at the word naming the requested value\.

We run a second prompt that asks for the other value of the same attribute\. Following activation\-patching methods\([Heimersheim and Nanda, 2024](https://arxiv.org/html/2610.19113#bib.bib38)\), after layer 4 we replace the original run’s internal representation at that word with the representation from the second run, then let the remaining layers continue\. We score success strictly: the model must write the full form required by the replacement value\. Targets shown in the prompt are reported separately from never\-shown ones, since never\-shown targets alone require constructing a new form \(Figure[13](https://arxiv.org/html/2610.19113#A7.F13); layer choice: Appendices[G](https://arxiv.org/html/2610.19113#A7)and[N](https://arxiv.org/html/2610.19113#A14)\)\.

#### Models\.

We test released Pythia and OLMo models\([Biderman et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib9);[Team OLMo et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib10)\)and Qwen and Gemma families\([Yang et al\., 2025a](https://arxiv.org/html/2610.19113#bib.bib11);[Qwen et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib12);[Gemma Team et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib13)\)\. To follow changes during training, we use released intermediate checkpoints and public training trajectories\([Groeneveld et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib52);[Muennighoff et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib53);[Liu et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib54)\)\. Continued\-training experiments restart intermediate checkpoints of SmolLM2\-360M and SmolLM2\-1\.7B\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib66)\)on a public proxy of their corpus \(Appendices[L](https://arxiv.org/html/2610.19113#A12)and[M](https://arxiv.org/html/2610.19113#A13)\)\. Definition\-exercise fine\-tunes use SmolLM2\-1\.7B and SmolLM3\-3B\([Hugging Face, 2025](https://arxiv.org/html/2610.19113#bib.bib69)\)\(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\. Low\-rank adapters are trained on five released 7–8B bases \(Qwen2\.5\-7B, OLMo\-2\-7B, Llama\-3\.1\-8B, Comma\-v0\.1\-1T, RedPajama\-INCITE\-7B; Figure[8](https://arxiv.org/html/2610.19113#S6.F8)\)\. A direct task fine\-tune uses Pythia\-1\.4B \(Appendix[I](https://arxiv.org/html/2610.19113#A9)\)\. Code\-trained Qwen variants\([Hui et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib21)\)serve one control \(Appendix[C](https://arxiv.org/html/2610.19113#A3)\)\.

## 4Answer preference reflects prompt control

![Refer to caption](https://arxiv.org/html/2610.19113v1/fig_fingerprint.png)Figure 2:Copied\-answer probability \(left\) and four\-choice accuracy \(right\) for 17 models, grouped by the\.70\.70and\.30\.30thresholds; marker shape gives the family \(full scores in Table[2](https://arxiv.org/html/2610.19113#A3.T2)\)\.![Refer to caption](https://arxiv.org/html/2610.19113v1/fig_knockout_col.png)Figure 3:Change inmmafter blocking the closest ending or the requested value, model by model \(all blocked spans: Appendix[D](https://arxiv.org/html/2610.19113#A4)\)\.![Refer to caption](https://arxiv.org/html/2610.19113v1/fig2_interventions.png)Figure 4:Blocking a single prompt span reverses the preferred answer: meanmmwith the full prompt and after blocking each span, for high\-copy Pythia\-2\.8B and low\-copy Qwen3\-8B\.Figure 5:Attention analysis\. \(a\) The answer position reads two prompt spans, the conflicting ending of the closest example \(favors the copied answer\) and the requested value \(favors the correct one\)\. We block attention to a span, patch the requested\-value state from a prompt asking for the other value, and read out the change inmm\. Direct logit attribution picks the five heads pushing hardest toward each answer, which are silenced and compared with random sets\. \(b\) Blocking a span movesmmfar more than silencing either set \(Figure[3](https://arxiv.org/html/2610.19113#S4.F3)\)\.Three groups of models\.The models spread across the copying\-to\-composition range\. Six assign at least\.70\.70of the copied and correct candidates’ combined probability to the copied answer, eight assign at most\.30\.30, and three fall between these thresholds \(Figure[2](https://arxiv.org/html/2610.19113#S4.F2); Table[2](https://arxiv.org/html/2610.19113#A3.T2)\)\. Candidate preference tracks the ability to construct the missing form: the correct answer rarely ranks first in the high\-copy group and usually does in the low\-copy group, with exact generation following the same ordering\. The grouping is not a size effect: Pythia\-12B copies while Qwen3\-1\.7B and Gemma\-3\-4B compose \(Figure[2](https://arxiv.org/html/2610.19113#S4.F2)\)\.

Arbitrary codes and a token\-aligned Southern Quechua version preserve the model ordering\. Removing the closest combined example selectively weakens the copied candidate, while deleting unrelated lines has little effect \(Appendices[F](https://arxiv.org/html/2610.19113#A6)and[D](https://arxiv.org/html/2610.19113#A4); Table[4](https://arxiv.org/html/2610.19113#A4.T4)\)\. These controls tie the preference to the competing information in the prompt rather than to wording, suffix inventory, or prompt length \(Figure[10](https://arxiv.org/html/2610.19113#A3.F10)\)\.

#### The controlling prompt source differs between the groups\.

Blocking the closest ending has the larger effect in high\-copy models, whereas blocking the requested value has the larger effect in low\-copy models \(Figure[3](https://arxiv.org/html/2610.19113#S4.F3)\)\. In the high\-copy Pythia models the closest\-ending effect is about twice the requested\-value effect\. In low\-copy models the requested\-value effect is several times the closest\-ending effect\. Figure[4](https://arxiv.org/html/2610.19113#S4.F4)shows one model per group \(all blocked spans: Appendix[D](https://arxiv.org/html/2610.19113#A4), Table[3](https://arxiv.org/html/2610.19113#A4.T3)\)\.

The three models between the probability thresholds behave as an intermediate group: their two candidates receive similar probability and, for the models measured, the two effects are much closer than in either of the two main groups \(Table[3](https://arxiv.org/html/2610.19113#A4.T3)\), so the balance between the candidates and the controlling span both vary continuously\.

#### Models can read the query without constructing the missing answer\.

Patching the requested\-value state redirects every model toward the corresponding combination when that combination already appears in the prompt, but when it requires the unshown combination the Pythia models rarely follow, whereas the other models follow on a substantial share of prompts \(Figure[13](https://arxiv.org/html/2610.19113#A7.F13)\)\. High\-copy models thus represent enough of the query to produce a shown combination but fail to construct the missing one\.

Head\-level analyses \(Figure[5](https://arxiv.org/html/2610.19113#S4.F5)\) place the behavior above the level of single heads\. Copy\-favoring head sets transfer across prompt samples in several models and lowerP⁡\(copy\)P\(\\mathrm\{copy\}\)when disabled, yet the preference survives their removal, and no five\-head set accounts for most of the token\-blocking effect\. Blocking, not raw attention weights, tracks the developmental change reported next \(Appendix[K](https://arxiv.org/html/2610.19113#A11)\)\.

## 5Prompt control changes before composition

![Refer to caption](https://arxiv.org/html/2610.19113v1/fig_traj_col.png)Figure 6:Prompt control across pretraining checkpoints, shown for OLMo\-2\-7B, the most densely sampled series \(other series: Appendix[E](https://arxiv.org/html/2610.19113#A5)\)\. The requested\-value effect overtakes the closest\-ending effect between 336B and 630B tokens \(star\)\. Four\-choice accuracy crosses chance much later \(dashed line; Table[1](https://arxiv.org/html/2610.19113#A2.T1)\)\.The checkpoints confirm what the cross\-model contrast predicts: control of the answer changes before the model can construct the missing form\.

Control switches first\.Early checkpoints are governed by the closest ending\. Blocking that ending changes the copied\-versus\-correct margin far more than blocking the requested value\. The balance reverses between 336B and 630B tokens \(Figure[6](https://arxiv.org/html/2610.19113#S5.F6); also Figure[1](https://arxiv.org/html/2610.19113#S0.F1)b\)\. At the end of this interval the two candidates receive similar probability and four\-choice accuracy has yet to reach chance: prompt control has shifted before the correct answer becomes reliable\.

Accuracy follows\.Accuracy improves later\. The fitted four\-choice trajectory crosses chance at about 1\.5T tokens on the original items and 1\.8T on the re\-test items \(Table[1](https://arxiv.org/html/2610.19113#A2.T1); Appendix[J](https://arxiv.org/html/2610.19113#A10)\), well after the two blocking effects cross\. The parts of the task are already in place much earlier: the model can copy the requested stem, recover each feature mapping, and generate combinations shown in the prompt hundreds of billions of tokens before it ranks the missing combination first\. The long middle stage therefore has every part of the task in place, with composition of those parts the last piece to arrive\.

Exact generation of the missing form arrives last, at about 3\.6T tokens \(Table[1](https://arxiv.org/html/2610.19113#A2.T1)\): parts, then control, then selection, then generation\.

Other training series\.The ordering recurs in runs from other labs and recipes\. Of sixteen released series, Marin\-8B, Comma\-v0\.1\-1T, SmolLM3\-3B, and OLMo\-3\-7B end above the accuracy criterion, Apertus is partial, and eleven stay below it\. OLMo\-3 and Comma satisfy every ordering criterion, shifting hundreds of billions of tokens before their first above\-chance checkpoints\. SmolLM3 also shifts before accuracy rises, with the closest\-example effect later regaining strength\. Within the OLMo family, OLMo\-2\-13B shifts early but briefly reverses, and the older OLMo\-7B\-0424 run changes sign repeatedly and turns negative for good only after its fitted accuracy onset\. Runs that fail to compose never sustain control by the requested value and keep the copied answer\. In every run that reaches composition,DDfirst turns negative by the time accuracy crosses chance, and no below\-chance run sustains the switch \(Appendix[E](https://arxiv.org/html/2610.19113#A5); Table[6](https://arxiv.org/html/2610.19113#A5.T6)\)\.

The endpoint comparison links this transition to eventual success: trajectories that exceed chance finish with much larger requested\-value effects than those that remain below chance, and in the successful runsDDfalls below zero before or by the time four\-choice accuracy exceeds chance, whereas runs in which the requested value never gains control do not learn the missing combination within the observed training window \(Table[6](https://arxiv.org/html/2610.19113#A5.T6)\)\.

#### The switch shows at the head level and extends beyond it\.

The correct\-favoring heads overtake the copy\-favoring heads in the same interval \(Figure[14](https://arxiv.org/html/2610.19113#A11.F14)\)\. The change is distributed: five heads carry part of the blocking effect, and silencing them moves the balance without creating composition\.

The order is a property of pretraining, not of the task: under direct task fine\-tuning the blocking crossing and the accuracy rise happen together, or accuracy moves first on a denser grid \(Appendix[I](https://arxiv.org/html/2610.19113#A9)\)\. The next section asks which training signals produce the switch and what transfers once they do\.

## 6Training teaches what it practices

The checkpoint results show control of the answer passing from the closest example to the requested value\. We next ask which training signal produces this change\. Synthetic documents that define endings and then query them reverse which prompt source controls the answer and let models produce the missing form\. Exposure to definitions and a penalty on the copy\-favoring heads do neither, and definition exercises built from rewritten real documents instead teach models to follow the labels a prompt assigns\.

### 6\.1Practice, not exposure or suppression, moves control

Exposure to definitions\.Definitions embedded in natural text are not the active ingredient\. After 1B additional tokens on SmolLM2\-1\.7B \(Appendix[M](https://arxiv.org/html/2610.19113#A13)\), continued pretraining on mined passages that define and later use a term, on absent\-term variants, or on dictionary\-prefixed text leavesDDpositive and exact generation at the plain\-continuation level\. All four conditions remain controlled by the closest ending \(Figure[16](https://arxiv.org/html/2610.19113#A13.F16)\)\.

Penalizing copying\.Penalizing the copy\-favoring heads changes the balance between the two candidates without creating the missing answer\. At SmolLM2\-360M, adding to plain continuation a penalty on attention from the answer position to the closest ending, applied to ten heads selected by direct logit attribution for favoring the copied answer, drives those heads’ attention to that ending to near zero\. With the copy route closed,DDstays positive and exact generation stays at zero, while the same penalty on ten random heads leaves both at plain\-continuation levels \(Figure[15](https://arxiv.org/html/2610.19113#A12.F15); Appendices[N](https://arxiv.org/html/2610.19113#A14)and[L](https://arxiv.org/html/2610.19113#A12)\)\.

### 6\.2Synthetic documents reverse copying and transfer what they practice

Synthetic documents\.The synthetic documents reverse both the preferred answer and which prompt source controls it\. These documents have the diagnostic’s own dictionary\-and\-query form: each defines endings for a small feature system, shows some combinations, leaves others unshown, and writes out their answers, with nonce stems, markers, and templates drawn from pools disjoint from the evaluation inventories and no code systems \(Appendix[L](https://arxiv.org/html/2610.19113#A12)\)\. The suffix diagnostic shares the training format, so the code and Quechua versions serve as transfer probes\. Replacing3%3\\,\\%of a continued\-pretraining stream with these documents moves SmolLM2\-1\.7B from copy\-dominant to query\-dominant and lifts exact generation from near zero to near ceiling \(Figure[7](https://arxiv.org/html/2610.19113#S6.F7); Figure[16](https://arxiv.org/html/2610.19113#A13.F16)\)\. By 50M tokensP⁡\(copy\)P\(\\mathrm\{copy\}\)has fallen from\.82\.82to near zero and exact generation has risen to\.56\.56; by the end of training exact generation is near ceiling, whereas plain continuation stays copy\-dominant\.

Figure 7:Continued pretraining of SmolLM2\-1\.7B\. The right panels zoom on the first 50M tokens\. A3%3\\,\\%synthetic mixture reverses copying by the first checkpoint, whereas mined passages that merely contain definitions do not \(Figure[16](https://arxiv.org/html/2610.19113#A13.F16)\)\.The reversal costs nothing measurable on general text: held\-out losses move by less than\.001\.001at both scales whileDDmoves by more than twenty nats\.

Held\-out versus all\-shown\.The SmolLM2\-360M experiments separate query\-resolution practice from rehearsal of the missing cell\. Two synthetic variants share format and data order\. In the held\-out condition each document queries combinations it does not show\. In the all\-shown condition every queried combination already appears on another stem in the same document, so the model never practices an absent cell\. Both synthetic conditions turn toD<0D<0by the first checkpoint, after 50M tokens, and write the missing form on most prompts by the end of training, while plain continuation stays copy\-dominant \(Figure[15](https://arxiv.org/html/2610.19113#A12.F15)\)\. The all\-shown condition matches the held\-out condition, so learning to resolve queries through temporary mappings does not require practicing an absent cell\.

Full fine\-tuning of Pythia\-1\.4B on prompts that show every requested form reproduces the all\-shown result \(Appendix[I](https://arxiv.org/html/2610.19113#A9)\), so the shift holds under full fine\-tuning as well as continued pretraining\.

Low\-rank adapters trained on the same held\-out or all\-shown documents take five released 7–8B bases to or near the ceiling of the diagnostic \(Figure[8](https://arxiv.org/html/2610.19113#S6.F8)\): RedPajama\-7B, the one copy\-dominant base, turns query\-dominant, and the four that already composed deepen their query dependence\. The gain stays with the practiced form: the external composition tasks register no gain under the pre\-registered rule, SCAN\-ICL falls on OLMo\-2 and Llama\-3\.1, and OLMo\-2’s general\-text loss passes the pre\-registered bound \(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\. The one ElementalTask gain arrives with a SCAN loss \(Figure[8](https://arxiv.org/html/2610.19113#S6.F8)b\)\.

Figure 8:LoRA takes exact generation to or near ceiling on five bases, while the external SCAN–Elemental pairs stay outside the shaded quadrant where both would improve: the gain stays with the practiced form\.Transfer follows what was practiced, not composition tasks in general\. At 1\.7B the five external evaluations fall outside the pre\-registered qualification rule, so their synthetic\-versus\-plain differences are reported as descriptive \(Appendix[M](https://arxiv.org/html/2610.19113#A13)\)\. The boundary of transfer is therefore set by the 7–8B adapters, whose bases score well above floor on SCAN\-ICL and ElementalTask\-ICL and whose gains stay with the practiced form under that rule \(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\.

### 6\.3Transfer depends on the supervision

Definition exercises\.Definition exercises teach a different use of the prompt\. A language model rewrote real passages into short exercises that define a label, term, or rule and then pose a query answerable only from that definition \(verified examples in Table[8](https://arxiv.org/html/2610.19113#A14.T8)\)\. Unlike passive exposure to definitions in natural text, every exercise requires an answer determined by a local definition\. Fine\-tuning SmolLM2\-1\.7B on 57M tokens of exercises is compared with the same schedule on the original documents, and both are evaluated on in\-context classification whose class names are replaced by made\-up symbols \(made\-up labels\) or exchanged with one another \(exchanged labels\)\([Wei et al\., 2023a](https://arxiv.org/html/2610.19113#bib.bib63)\), so that accuracy requires following the prompt’s assignment\. Made\-up\-label accuracy rises by17\.617\.6points on TREC,17\.217\.2on DBpedia, and3\.73\.7on Yahoo \(Figure[1](https://arxiv.org/html/2610.19113#S0.F1)c\), and exchanged\-label accuracy rises on all three\. On SmolLM3\-3B the gains hold on TREC and Yahoo, and no counted evaluation loses\. The exercises contain neither these datasets nor their labels\. The transfer is specific to the practiced use of the prompt: a relational\-reasoning suite that also draws definitions from the prompt \(bAbI\-1, bAbI\-15, or CLUTRR\) moves down by a few points and farther tasks move in both directions, so the exercises teach following prompt\-assigned labels rather than prompt\-dependent tasks in general\. \(Appendix[N](https://arxiv.org/html/2610.19113#A14); Figure[17](https://arxiv.org/html/2610.19113#A14.F17)\)\.

The morphology diagnostic is untouched by the exercises:P⁡\(copy\)P\(\\mathrm\{copy\}\)is unchanged and four\-choice accuracy and exact generation stay near floor \(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\. The exercises make label following reusable across datasets and leave the morphology task in place\.

#### The interventions separate suppression, construction, and transfer\.

Suppression moves the margin without producing the missing form\. Training on the synthetic documents turnsDDnegative and lifts exact generation across model families, update methods, and encodings, and its transfer follows the practiced form\. Definition exercises improve prompt\-defined label use elsewhere while leaving the diagnostic unchanged\. What mattered was which use of the prompt the training text repeatedly rewarded\.

## 7Discussion

Figure 9:Four events that accuracy compresses into one, in training order: a prompt source gains control of the answer, the preferred candidate flips \(the only event accuracy registers\), the form is written without choices, and the learned skill transfers to held\-out tasks \(after the training of Section[6](https://arxiv.org/html/2610.19113#S6)\)\.The results separate four events that accuracy compresses into one \(Figure[9](https://arxiv.org/html/2610.19113#S7.F9)\)\. The requested value takes over control from the closest example before accuracy leaves chance\.

Copying and composing from the requested value coexist in the same model\. High\-copy models already use the query to produce a shown combination, direct fine\-tuning strengthens the effect of the requested value without erasing the effect of the closest example, and penalizing the copy\-favoring heads moves the margin, while the missing form appears only once training supplies a way to build it\.

What the all\-shown control shows\.In the all\-shown control, documents in which every queried combination is already shown on another stem cross at the same checkpoint and reach comparable exact generation at 360M\. The tie places the cause in what both conditions share, repeated resolution of queries through temporary mappings, and rules out rehearsal of the absent cell \(Appendix[L](https://arxiv.org/html/2610.19113#A12)\)\.

Developmental checkpoints should be read accordingly: a wrong answer can mark a substantive transition if its score now depends on the query rather than the closest example, and a correct multiple\-choice answer need not establish construction if the form cannot be generated without candidates\.

## 8Conclusion

On a diagnostic that separates copying from composition, in\-context learning develops as control shifts among competing prompt cues\. The requested value takes control of candidate scores before the correct answer wins\. Suppressing copying moves the margin, practice at resolving queries writes the missing form, and training transfers the use of the prompt it repeatedly requires\. Tracking control, preference, construction, and transfer shows what pretraining teaches beyond what accuracy reveals\. Across models, checkpoint series, and training experiments, these four aspects develop on different schedules, so in\-context learning is acquired as several capabilities, each on its own timetable\.

## Limitations

Our diagnostic is a single controlled task that separates copying from composition, and the code and Quechua versions carry the contrast into two further encodings\. In\-context learning covers many other behaviors, and the four measures used here give a way to ask whether they develop in the same order\. The training experiments use models up to 8B parameters and synthetic documents that share the diagnostic’s format, so the transfer they produce is the format\-bound transfer the design predicts\. Larger models and more varied training data are the next tests of this account\. The selected attention heads carry part of the behavior, so the localization rests on the prompt\-span interventions\. The pre\-registered training block, transfer evaluations, and analysis\-plan record are in Appendices[L](https://arxiv.org/html/2610.19113#A12),[N](https://arxiv.org/html/2610.19113#A14), and[P](https://arxiv.org/html/2610.19113#A16)\.

## Ethical Considerations

The Southern Quechua paradigms used in the morphology version of the task come from published linguistic resources\([Adelaar and Muysken, 2004](https://arxiv.org/html/2610.19113#bib.bib33);[Kirov et al\., 2018](https://arxiv.org/html/2610.19113#bib.bib29);[Batsuren et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib30)\)\.

The mined documents come from public collections of web, scientific, mathematical, encyclopedia, forum, and legal text\. We applied no additional screening for personal or offensive content beyond the source collections’ own filtering, so these sources may contain such material\.

AI assistants were used for writing code and editing text\.

## References

- Adelaar and Muysken \(2004\)W\. F\. H\. Adelaar and P\. C\. MuyskenThe languages of the Andes\.Cambridge University Press\.Cited by:[Appendix F](https://arxiv.org/html/2610.19113#A6.SS0.SSS0.Px2.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19113#Sx2.p1.1)\.
- Akyüreket al\.\(2024\)E\. Akyürek, B\. Wang, Y\. Kim, and J\. AndreasIn\-context language learning: architectures and algorithms\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Albright and Hayes \(2003\)A\. Albright and B\. HayesRules vs\. analogy in English past tenses: a computational/experimental study\.Cognition90\(2\),pp\. 119–161\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Anhet al\.\(2024\)D\. Anh, L\. Raviv, and L\. GalkeMorphology matters: probing the cross\-linguistic morphological generalization abilities of large language models through a wug test\.InProceedings of the Workshop on Cognitive Modeling and Computational Linguistics \(CMCL 2024\),pp\. 177–188\.External Links:[Link](https://aclanthology.org/2024.cmcl-1.15/),[Document](https://dx.doi.org/10.18653/v1/2024.cmcl-1.15)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Aroraet al\.\(2024\)A\. Arora, D\. Jurafsky, and C\. PottsCausalGym: benchmarking causal interpretability methods on linguistic tasks\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 14638–14663\.External Links:[Link](https://aclanthology.org/2024.acl-long.785/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.785)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Batsurenet al\.\(2022\)K\. Batsuren, O\. Goldman, S\. Khalifa, N\. Habash, W\. Kieraś, G\. Bella, B\. Leonard, G\. Nicolai, K\. Gorman, Y\. G\. Ate,et al\.UniMorph 4\.0: universal morphology\.InProceedings of the Thirteenth Language Resources and Evaluation Conference \(LREC 2022\),pp\. 840–855\.Cited by:[Appendix F](https://arxiv.org/html/2610.19113#A6.SS0.SSS0.Px2.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19113#Sx2.p1.1)\.
- Ben Allalet al\.\(2025\)L\. Ben Allal, A\. Lozhkov, E\. Bakouch, G\. Martín Blázquez, G\. Penedo, L\. Tunstall, A\. Marafioti, H\. Kydlíček, A\. Piqueres Lajarín, V\. Srivastav, J\. Lochner, C\. Fahlgren, X\. Nguyen, C\. Fourrier, B\. Burtenshaw, H\. Larcher, H\. Zhao, C\. Zakka, M\. Morlon, C\. Raffel, L\. von Werra, and T\. WolfSmolLM2: when smol goes big—data\-centric training of a small language model\.arXiv preprint arXiv:2502\.02737\.External Links:[Link](https://arxiv.org/abs/2502.02737)Cited by:[1st item](https://arxiv.org/html/2610.19113#A12.I1.i1.p1.1),[1st item](https://arxiv.org/html/2610.19113#A13.I1.i1.p1.1),[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.16.7),[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.8.7),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Berko \(1958\)J\. BerkoThe child’s learning of English morphology\.Word14\(2\-3\),pp\. 150–177\.Cited by:[§1](https://arxiv.org/html/2610.19113#S1.p4.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Bhattamishraet al\.\(2024\)S\. Bhattamishra, A\. Patel, P\. Blunsom, and V\. KanadeUnderstanding in\-context learning in transformers and LLMs by learning to learn discrete functions\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Bidermanet al\.\(2023\)S\. Biderman, H\. Schoelkopf, Q\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. van der WalPythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),Proceedings of Machine Learning Research, Vol\.202,pp\. 2397–2430\.External Links:[Link](https://proceedings.mlr.press/v202/biderman23a.html)Cited by:[Appendix A](https://arxiv.org/html/2610.19113#A1.SS0.SSS0.Px5.p1.1),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Biettiet al\.\(2023\)A\. Bietti, V\. Cabannes, D\. Bouchacourt, H\. Jégou, and L\. BottouBirth of a transformer: a memory viewpoint\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- BigScience Workshop \(2024\)BigScience WorkshopBLOOM: a 176b\-parameter open\-access multilingual language model\.Journal of Machine Learning Research25\(422\),pp\. 1–74\.External Links:[Link](https://jmlr.org/papers/v25/23-0581.html)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.13.7)\.
- Chanet al\.\(2022\)S\. C\. Y\. Chan, A\. Santoro, A\. K\. Lampinen, J\. X\. Wang, A\. Singh, P\. H\. Richemond, J\. L\. McClelland, and F\. HillData distributional properties drive emergent in\-context learning in transformers\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Chang and Bergen \(2022\)T\. A\. Chang and B\. K\. BergenWord acquisition in neural language models\.Transactions of the Association for Computational Linguistics10,pp\. 1–16\.External Links:[Link](https://aclanthology.org/2022.tacl-1.1/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00444)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Chenet al\.\(2024\)A\. Chen, R\. Shwartz\-Ziv, K\. Cho, M\. L\. Leavitt, and N\. SaphraSudden drops in the loss: syntax acquisition, phase transitions, and simplicity bias in MLMs\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Evansonet al\.\(2023\)L\. Evanson, Y\. Lakretz, and J\. KingLanguage acquisition: do children and language models follow similar learning stages?\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 12205–12218\.External Links:[Link](https://aclanthology.org/2023.findings-acl.773/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.773)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Feuchtet al\.\(2025\)S\. Feucht, E\. Todd, B\. C\. Wallace, and D\. BauThe dual\-route model of induction\.InConference on Language Modeling \(COLM\),Note:arXiv:2504\.03022Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. LeahyThe pile: an 800GB dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[4th item](https://arxiv.org/html/2610.19113#A13.I1.i4.p1.1)\.
- Geigeret al\.\(2021\)A\. Geiger, H\. Lu, T\. Icard, and C\. PottsCausal abstractions of neural networks\.InAdvances in Neural Information Processing Systems 34 \(NeurIPS 2021\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Gemma Teamet al\.\(2025\)Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière,et al\.Gemma 3 technical report\.arXiv preprint arXiv:2503\.19786\.Cited by:[Appendix A](https://arxiv.org/html/2610.19113#A1.SS0.SSS0.Px5.p1.1),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Gevaet al\.\(2023\)M\. Geva, J\. Bastings, K\. Filippova, and A\. GlobersonDissecting recall of factual associations in auto\-regressive language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/2023.emnlp-main.751/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.751)Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px3.p1.2)\.
- Goldowsky\-Dillet al\.\(2023\)N\. Goldowsky\-Dill, C\. MacLeod, L\. Sato, and A\. AroraLocalizing model behavior with path patching\.arXiv preprint arXiv:2304\.05969\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Groeneveldet al\.\(2024\)D\. Groeneveld, I\. Beltagy, P\. Walsh, A\. Bhagia, R\. Kinney, O\. Tafjord, A\. H\. Jha, H\. Ivison, I\. Magnusson, Y\. Wang, S\. Arora, D\. Atkinson, R\. Authur, K\. R\. Chandu, A\. Cohan, J\. Dumas, Y\. Elazar, Y\. Gu, J\. Hessel, T\. Khot, W\. Merrill, J\. Morrison, N\. Muennighoff, A\. Naik, C\. Nam, M\. E\. Peters, V\. Pyatkin, A\. Ravichander, D\. Schwenk, S\. Shah, W\. Smith, E\. Strubell, N\. Subramani, M\. Wortsman, P\. Dasigi, N\. Lambert, K\. Richardson, L\. Zettlemoyer, J\. Dodge, K\. Lo, L\. Soldaini, N\. A\. Smith, and H\. HajishirziOLMo: accelerating the science of language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15789–15809\.External Links:[Link](https://aclanthology.org/2024.acl-long.841/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.841)Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Heimersheim and Nanda \(2024\)S\. Heimersheim and N\. NandaHow to use and interpret activation patching\.arXiv preprint arXiv:2404\.15255\.Cited by:[§1](https://arxiv.org/html/2610.19113#S1.p4.1),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px4.p1.1)\.
- Hendelet al\.\(2023\)R\. Hendel, M\. Geva, and A\. GlobersonIn\-context learning creates task vectors\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 9318–9333\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.624/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.624)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px5.p1.1)\.
- Hofmannet al\.\(2025\)V\. Hofmann, L\. Weissweiler, D\. R\. Mortensen, H\. Schütze, and J\. B\. PierrehumbertDerivational morphology reveals analogical generalization in large language models\.Proceedings of the National Academy of Sciences122\(19\),pp\. e2423232122\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px3.p1.1)\.
- Huet al\.\(2024\)Y\. Hu, H\. Song, J\. Deng, J\. Wang, J\. Chen, K\. Zhou, Y\. Zhu, J\. Jiang, Z\. Dong, W\. X\. Zhao, and J\. WenYuLan\-Mini: an open data\-efficient language model\.arXiv preprint arXiv:2412\.17743\.External Links:[Link](https://arxiv.org/abs/2412.17743)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.10.7)\.
- Hugging Face \(2024\)Hugging FaceHuggingFaceFW/ablation\-model\-fineweb\-edu model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.15.7)\.
- Hugging Face \(2025\)Hugging FaceSmolLM3\-3B\-Base model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/HuggingFaceTB/SmolLM3-3B-Base)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.4.7),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Huiet al\.\(2024\)B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu, K\. Dang, Y\. Fan, Y\. Zhang, A\. Yang, R\. Men, F\. Huang, B\. Zheng, Y\. Miao, S\. Quan, Y\. Feng, X\. Ren, X\. Ren, J\. Zhou, and J\. LinQwen2\.5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Hupkeset al\.\(2020\)D\. Hupkes, V\. Dankers, M\. Mul, and E\. BruniCompositionality decomposed: how do neural networks generalise?\.Journal of Artificial Intelligence Research67,pp\. 757–795\.Cited by:[§1](https://arxiv.org/html/2610.19113#S1.p4.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Ismayilzadaet al\.\(2025\)M\. Ismayilzada, D\. Circi, J\. Sälevä, H\. Sirin, A\. Köksal, B\. Dhingra, A\. Bosselut, D\. Ataman, and L\. van der PlasEvaluating morphological compositional generalization in large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),External Links:[Link](https://aclanthology.org/2025.naacl-long.59/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.59)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Jaghouaret al\.\(2024\)S\. Jaghouar, J\. M\. Ong, M\. Basra, F\. Obeid, J\. Straube, M\. Keiblinger, E\. Bakouch, L\. Atkins, M\. Panahi, C\. Goddard, M\. Ryabinin, and J\. HagemannINTELLECT\-1 technical report\.arXiv preprint arXiv:2412\.01152\.External Links:[Link](https://arxiv.org/abs/2412.01152)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.14.7)\.
- Jain and Wallace \(2019\)S\. Jain and B\. C\. WallaceAttention is not explanation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 3543–3556\.External Links:[Link](https://aclanthology.org/N19-1357/),[Document](https://dx.doi.org/10.18653/v1/N19-1357)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Kandpalet al\.\(2025\)N\. Kandpal, B\. Lester, C\. Raffel, S\. Majstorovic, S\. Biderman, B\. Abbasi, L\. Soldaini, E\. Shippole, A\. F\. Cooper, A\. Skowron, S\. Longpre, L\. Sutawika, A\. Albalak, Z\. Xu, G\. Penedo, L\. Ben Allal, E\. Bakouch, J\. Pressman, H\. Fan, D\. Stander, G\. Song, A\. Gokaslan, J\. Kirchenbauer, T\. Goldstein, B\. Bartoldson, B\. Kailkhura, and T\. MurrayThe common pile v0\.1: an 8tb dataset of public domain and openly licensed text\.InAdvances in Neural Information Processing Systems,Vol\.38, Main Conference\.External Links:[Document](https://dx.doi.org/10.52202/085713-1918),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/52acc050138d6f40dad6f12f91a4ce22-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.3.7)\.
- Kim \(2025\)C\. KimTask Schema and Binding: A Double Dissociation Study of In\-Context Learning\.arXiv preprint arXiv:2512\.17325\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Kim and Linzen \(2020\)N\. Kim and T\. LinzenCOGS: a compositional generalization challenge based on semantic interpretation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9087–9105\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.731/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.731)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Kirovet al\.\(2018\)C\. Kirov, R\. Cotterell, J\. Sylak\-Glassman, G\. Walther, E\. Vylomova, P\. Xia, M\. Faruqui, S\. J\. Mielke, A\. McCarthy, S\. Kübler, D\. Yarowsky, J\. Eisner, and M\. HuldenUniMorph 2\.0: universal morphology\.InProceedings of the Eleventh International Conference on Language Resources and Evaluation \(LREC 2018\),Miyazaki, Japan\.External Links:[Link](https://aclanthology.org/L18-1293/)Cited by:[Appendix F](https://arxiv.org/html/2610.19113#A6.SS0.SSS0.Px2.p1.1),[Ethical Considerations](https://arxiv.org/html/2610.19113#Sx2.p1.1)\.
- Lake and Baroni \(2018\)B\. M\. Lake and M\. BaroniGeneralization without systematicity: on the compositional skills of sequence\-to\-sequence recurrent networks\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Cited by:[4th item](https://arxiv.org/html/2610.19113#A13.I1.i4.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Lake and Baroni \(2023\)B\. M\. Lake and M\. BaroniHuman\-like systematic generalization through a meta\-learning neural network\.Nature623,pp\. 115–121\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Li and Roth \(2002\)X\. Li and D\. RothLearning question classifiers\.InCOLING 2002: The 19th International Conference on Computational Linguistics,External Links:[Link](https://aclanthology.org/C02-1150/)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1)\.
- Liuet al\.\(2026\)E\. Liu, K\. Sun, M\. Li, I\. Lee, L\. Tjuatja, J\. Huang, and G\. NeubigWhat do language models learn and when? the implicit curriculum hypothesis\.arXiv preprint arXiv:2604\.08510\.Cited by:[4th item](https://arxiv.org/html/2610.19113#A13.I1.i4.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)Z\. Liu, A\. Qiao, W\. Neiswanger, H\. Wang, B\. Tan, T\. Tao, J\. Li, Y\. Wang, S\. Sun, O\. Pangarkar, R\. Fan, Y\. Gu, V\. Miller, Y\. Zhuang, G\. He, H\. Li, F\. Koto, L\. Tang, N\. Ranjan, Z\. Shen, X\. Ren, R\. Iriondo, C\. Mu, Z\. Hu, M\. Schulze, P\. Nakov, T\. Baldwin, and E\. P\. XingLLM360: towards fully transparent open\-source LLMs\.arXiv preprint arXiv:2312\.06550\.Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- LumiOpen \(2024\)LumiOpenViking\-7B model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/LumiOpen/Viking-7B)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.12.7)\.
- Marin Community \(2025\)Marin CommunityMarin\-8B\-Base model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/marin-community/marin-8b-base)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.2.7)\.
- McDougallet al\.\(2023\)C\. McDougall, A\. Conmy, C\. Rushing, T\. McGrath, and N\. NandaCopy suppression: comprehensively understanding an attention head\.arXiv preprint arXiv:2310\.04625\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Menget al\.\(2022\)K\. Meng, D\. Bau, A\. Andonian, and Y\. BelinkovLocating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. HajishirziOLMoE: open mixture\-of\-experts language models\.arXiv preprint arXiv:2409\.02060\.Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Niuet al\.\(2025\)J\. Niu, S\. Dutta, A\. Elshabrawy, H\. Tayyar Madabushi, and I\. GurevychIllusion or algorithm? investigating memorization, emergence, and symbolic processing in in\-context learning\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Olssonet al\.\(2022\)C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, S\. Johnston, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. OlahIn\-context learning and induction heads\.Transformer Circuits Thread\.Cited by:[Appendix K](https://arxiv.org/html/2610.19113#A11.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- OpenLLM\-France \(2025\)OpenLLM\-FranceLucie\-7B model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/OpenLLM-France/Lucie-7B)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.9.7)\.
- OpenLM Research \(2023\)OpenLM ResearchOpenLLaMA\-7B\-v2 model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/openlm-research/open_llama_7b_v2)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.7.7)\.
- Panet al\.\(2026\)L\. Pan, J\. Liang, J\. Ye, M\. Yang, X\. Lu, and F\. ZhuOpening the Black Box: A Survey on the Mechanisms of Multi\-Step Reasoning in Large Language Models\.arXiv preprint arXiv:2601\.14270\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 1525–1534\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1144)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px5.p1.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfFineWeb: decanting the web for the finest text data at scale\.arXiv preprint arXiv:2406\.17557\.Cited by:[4th item](https://arxiv.org/html/2610.19113#A13.I1.i4.p1.1)\.
- Pinker and Prince \(1988\)S\. Pinker and A\. PrinceOn language and connectionism: analysis of a parallel distributed processing model of language acquisition\.Cognition28\(1\-2\),pp\. 73–193\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Prakashet al\.\(2024\)N\. Prakash, T\. Rott Shaham, T\. Haklay, Y\. Belinkov, and D\. BauFine\-tuning enhances existing mechanisms: a case study on entity tracking\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Note:arXiv:2402\.14811Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Qwenet al\.\(2024\)Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Reddy \(2024\)G\. ReddyThe mechanistic basis of data dependence and abrupt learning in an in\-context classification task\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Rustet al\.\(2021\)P\. Rust, J\. Pfeiffer, I\. Vulić, S\. Ruder, and I\. GurevychHow good is your tokenizer? on the monolingual performance of multilingual language models\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://aclanthology.org/2021.acl-long.243/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.243)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 1715–1725\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1162)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Serrano and Smith \(2019\)S\. Serrano and N\. A\. SmithIs attention interpretable?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 2931–2951\.External Links:[Link](https://aclanthology.org/P19-1282/),[Document](https://dx.doi.org/10.18653/v1/P19-1282)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Singhet al\.\(2023\)A\. K\. Singh, S\. C\. Y\. Chan, T\. Moskovitz, E\. Grant, A\. M\. Saxe, and F\. HillThe transient nature of emergent in\-context learning in transformers\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Sinhaet al\.\(2019\)K\. Sinha, S\. Sodhani, J\. Dong, J\. Pineau, and W\. L\. HamiltonCLUTRR: a diagnostic benchmark for inductive reasoning from text\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 4506–4515\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1458)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1)\.
- Socheret al\.\(2013\)R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Ng, and C\. PottsRecursive deep models for semantic compositionality over a sentiment treebank\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,pp\. 1631–1642\.Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px5.p1.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastavaet al\.Beyond the imitation game: quantifying and extrapolating the capabilities of language models\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=uyTL5Bvosj)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px6.p1.1),[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1)\.
- Swiss AI Initiative \(2025\)Swiss AI InitiativeApertus\-8B\-2509 model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/swiss-ai/Apertus-8B-2509)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.6.7)\.
- Team OLMoet al\.\(2024\)Team OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, M\. Guerquin, H\. Ivison, P\. W\. Koh, J\. Liu, S\. Malik, W\. Merrill, L\. J\. V\. Miranda, J\. Morrison, T\. Murray, C\. Nam, V\. Pyatkin, A\. Rangapur, M\. Schmitz, S\. Skjonsberg, D\. Wadden, C\. Wilhelm, M\. Wilson, L\. Zettlemoyer, A\. Farhadi, N\. A\. Smith, H\. Hajishirzi,et al\.2 OLMo 2 furious\.arXiv preprint arXiv:2501\.00656\.Cited by:[Appendix A](https://arxiv.org/html/2610.19113#A1.SS0.SSS0.Px5.p1.1),[Appendix H](https://arxiv.org/html/2610.19113#A8.p1.1),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Team Olmo \(2026\)Team OlmoOlmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.5.7)\.
- Tiggeset al\.\(2024\)C\. Tigges, M\. Hanna, Q\. Yu, and S\. BidermanLLM circuit analyses are consistent across training and scale\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Toddet al\.\(2024\)E\. Todd, M\. L\. Li, A\. Sen Sharma, A\. Mueller, B\. C\. Wallace, and D\. BauFunction vectors in large language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Together Computer \(2023\)Together ComputerRedPajama\-INCITE\-7B\-Base model card\.Note:Hugging Face model cardExternal Links:[Link](https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.11.7)\.
- von Oswaldet al\.\(2023\)J\. von Oswald, E\. Niklasson, E\. Randazzo, J\. Sacramento, A\. Mordvintsev, A\. Zhmoginov, and M\. VladymyrovTransformers learn in\-context by gradient descent\.InProceedings of the 40th International Conference on Machine Learning \(ICML\),pp\. 35151–35174\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)K\. Wang, A\. Variengien, A\. Conmy, B\. Shlegeris, and J\. SteinhardtInterpretability in the wild: a circuit for indirect object identification in GPT\-2 small\.InThe Eleventh International Conference on Learning Representations \(ICLR\),Cited by:[Appendix K](https://arxiv.org/html/2610.19113#A11.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.19113#S1.p4.1)\.
- Weiet al\.\(2023a\)J\. Wei, L\. Hou, A\. Lampinen, X\. Chen, D\. Huang, Y\. Tay, X\. Chen, Y\. Lu, D\. Zhou, T\. Ma, and Q\. V\. LeSymbol tuning improves in\-context learning in language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Note:arXiv:2305\.08298External Links:[Link](https://aclanthology.org/2023.emnlp-main.61/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.61)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1),[§6\.3](https://arxiv.org/html/2610.19113#S6.SS3.p1.1)\.
- Weiet al\.\(2023b\)J\. Wei, J\. Wei, Y\. Tay, D\. Tran, A\. Webson, Y\. Lu, X\. Chen, H\. Liu, D\. Huang, D\. Zhou, and T\. MaLarger language models do in\-context learning differently\.arXiv preprint arXiv:2303\.03846\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Weissweileret al\.\(2023\)L\. Weissweiler, V\. Hofmann, A\. Kantharuban, A\. Cai, R\. Dutt, A\. Hengle, A\. Kabra, A\. Kulkarni, A\. Vijayakumar, H\. Yu, H\. Schuetze, K\. Oflazer, and D\. MortensenCounting the bugs in ChatGPT’s wugs: a multilingual investigation into the morphological capabilities of a large language model\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 6508–6524\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.401/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.401)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Westonet al\.\(2015\)J\. Weston, A\. Bordes, S\. Chopra, A\. M\. Rush, B\. van Merriënboer, A\. Joulin, and T\. MikolovTowards AI\-complete question answering: a set of prerequisite toy tasks\.arXiv preprint arXiv:1502\.05698\.Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px6.p1.1),[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1)\.
- Wuet al\.\(2024\)W\. Wu, Y\. Wang, G\. Xiao, H\. Peng, and Y\. FuRetrieval head mechanistically explains long\-context factuality\.arXiv preprint arXiv:2404\.15574\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2023\)Z\. Wu, A\. Geiger, T\. Icard, C\. Potts, and N\. D\. GoodmanInterpretability at scale: identifying causal mechanisms in Alpaca\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),Cited by:[Appendix G](https://arxiv.org/html/2610.19113#A7.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px2.p1.1)\.
- Xieet al\.\(2022\)S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. MaAn explanation of in\-context learning as implicit bayesian inference\.International Conference on Learning Representations \(ICLR\)\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix A](https://arxiv.org/html/2610.19113#A1.SS0.SSS0.Px5.p1.1),[§3](https://arxiv.org/html/2610.19113#S3.SS0.SSS0.Px5.p1.1)\.
- Yanget al\.\(2025b\)H\. Yang, H\. Cho, and N\. InoueLocalizing Task Recognition and Task Learning in In\-Context Learning via Attention Head Analysis\.arXiv preprint arXiv:2509\.24164\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2025\)J\. Ye, Z\. Yao, Z\. Huang, L\. Pan, J\. Liu, Y\. Bai, A\. Xin, W\. Liu, X\. Che, L\. Hou, and J\. LiHow do Transformers Learn Implicit Reasoning?\.InAdvances in Neural Information Processing Systems 38 \(NeurIPS 2025\),pp\. 73341–73369\.Note:arXiv:2505\.23653External Links:[Document](https://dx.doi.org/10.52202/085713-2208)Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Yinet al\.\(2024\)F\. Yin, X\. Ye, and G\. DurrettLoFiT: localized fine\-tuning on LLM representations\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS\),Note:arXiv:2406\.01563Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.
- Zhanget al\.\(2024\)P\. Zhang, G\. Zeng, T\. Wang, and W\. LuTinyLlama: an open\-source small language model\.arXiv preprint arXiv:2401\.02385\.External Links:[Link](https://arxiv.org/abs/2401.02385)Cited by:[Table 5](https://arxiv.org/html/2610.19113#A5.T5.2.17.7)\.
- Zhanget al\.\(2015\)X\. Zhang, J\. Zhao, and Y\. LeCunCharacter\-level convolutional networks for text classification\.InAdvances in Neural Information Processing Systems,Vol\.28\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2015/file/250cf8b51c773f3f8dc8b4be867a9a02-Paper.pdf)Cited by:[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px5.p1.1),[Appendix N](https://arxiv.org/html/2610.19113#A14.SS0.SSS0.Px7.p1.1)\.
- Zhaoet al\.\(2026\)X\. Zhao, D\. Sharma, R\. Uppaal, and Y\. ZhongShattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic\.arXiv preprint arXiv:2601\.22510\.Cited by:[§2](https://arxiv.org/html/2610.19113#S2.SS0.SSS0.Px3.p1.1)\.

## Contents of the Appendix

- [A](https://arxiv.org/html/2610.19113#A1)Task construction and conventions
- [B](https://arxiv.org/html/2610.19113#A2)Threshold crossings during training
- [C](https://arxiv.org/html/2610.19113#A3)Final scores and prompt\-variant controls
- [D](https://arxiv.org/html/2610.19113#A4)Blocking, deletion, and scoring checks
- [E](https://arxiv.org/html/2610.19113#A5)Training trajectories of seventeen runs
- [F](https://arxiv.org/html/2610.19113#A6)Code and Quechua versions
- [G](https://arxiv.org/html/2610.19113#A7)Activation patching
- [H](https://arxiv.org/html/2610.19113#A8)Stage\-2 OLMo\-2\-7B models
- [I](https://arxiv.org/html/2610.19113#A9)Fine\-tuning a copying model
- [J](https://arxiv.org/html/2610.19113#A10)Re\-test on new items
- [K](https://arxiv.org/html/2610.19113#A11)Selected attention heads
- [L](https://arxiv.org/html/2610.19113#A12)Continued pretraining with synthetic documents and attention penalties \(360M\)
- [M](https://arxiv.org/html/2610.19113#A13)Continued pretraining with synthetic documents and mined passages \(1\.7B\)
- [N](https://arxiv.org/html/2610.19113#A14)Layer sweep, head penalties, adapters, and exercises
- [O](https://arxiv.org/html/2610.19113#A15)Release
- [P](https://arxiv.org/html/2610.19113#A16)Changes to the analysis plans

## Appendix ATask construction and conventions

#### Generator\.

Every prompt draws fresh stems, words, endings and held\-out combination, so no fixed stem–ending mapping solves the task\. A suffix\-scheme stem concatenates one of 36 initial strings, one of 9 vowels and one of 22 final strings\. No stem repeats within a prompt, and the queried stem never appears in the demonstrations\. Endings are drawn without replacement from 30 strings \(*il, nak, or, um, esh, oth, ka, tu, ri, mo, lin, dor, sa, ne, po, ta, ku, zi, fen, gor, lu, wa, ex, an, ir, on, ul, ek, ip, ab*\), and line order is shuffled\. Each prompt draws one first\-feature pair from*small*/*big*,*old*/*new*,*red*/*blue*and*hot*/*cold*, and one second\-feature pair from*one*/*many*,*here*/*there*and*mine*/*yours*\. The code scheme keeps the first\-feature pairs, takes the second from*circle*/*square*,*north*/*south*or*left*/*right*, and uses 20 two\-character symbols with the header “Codebook:” and the separator “=\>”\.

Four fixed prompt sets serve all models and checkpoints, two \(320 items\) for the initial analyses and two for the re\-test on new items \(Appendix[J](https://arxiv.org/html/2610.19113#A10)\)\.

#### A literal prompt\.

One generated prompt, verbatim:

> In this language: one glon = glon\-lu many pliv = pliv\-dor one fooz = fooz\-lu big many swoov = swoov\-an\-dor many bouz = bouz\-dor small one kraix = kraix\-ek\-lu small one plut = plut\-ek\-lu big stoosh = stoosh\-an big one lask = lask\-an\-lu big one twix = twix\-an\-lu big crood = crood\-an small berb = berb\-ek small plaind = plaind\-ek big many doot = doot\-an\-dor small many glev =

The four scored continuations, each preceded by a space, areglev\-ek\-dor\(correct\),glev\-ek\-lu\(the copy, sharing the first ending\-ek\),glev\-an\-dorandglev\-an\-lu\. These are the candidates for four\-choice accuracy and the “ranks first” columns\.

#### Vocabulary\.

“Correct” is the rule\-consistent candidate of Equation[2](https://arxiv.org/html/2610.19113#S3.E2)in Section[3](https://arxiv.org/html/2610.19113#S3), and “copied” is the candidate that repeats the closest example’s second ending\. A generation is labeled by the first word of its first line, split on hyphens\. One ending counts as truncated and a form outside the four combinations as malformed\. Anything else is correct, copied, or another shown combination\. “Closest ending” and “requested value” name the blocked spans\.

#### Intervention conventions\.

Layers are decoder blocks counted from zero\. Activation patching replaces a block’s output at the position of the requested second\-feature value while the prompt is processed, before generation\. The model then takes the most probable token at each step for up to eight new tokens\. A blocked span covers every tokenizer token that overlaps the named string or strings, and blocking the closest ending covers the second\-ending string in both lines of the closest example\.

#### Tokenization\.

All 30 endings are separate tokens after a hyphen in Pythia and Gemma\([Biderman et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib9);[Gemma Team et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib13)\), most are in OLMo\-2, and Qwen merges some hyphens with their endings\([Yang et al\., 2025a](https://arxiv.org/html/2610.19113#bib.bib11);[Team OLMo et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib10)\)\. An ending can span several tokens, so the four forms can differ in token count\. We score the next ending with the correct preceding tokens supplied only where hyphen and ending stay separate \(scoring checks in Appendix[D](https://arxiv.org/html/2610.19113#A4)\)\.

## Appendix BThreshold crossings during training

Table[1](https://arxiv.org/html/2610.19113#A2.T1)dates when each behavior of Section[5](https://arxiv.org/html/2610.19113#S5)crosses a fixed threshold\. Rows use the original items unless marked as the code scheme\. Times are crossings of a fitted curve, not first significant differences\. We fit each mean trajectory overlog10\\log\_\{10\}tokens \(training steps for Pythia\) by least squares to a four\-parameter logistic curve, with both asymptotes between 0 and 1, rate at least \.05, and midpoint within two decades of the observed range\. Intervals are the 2\.5th to 97\.5th percentiles of the finite crossings over 300 resamples of the items with replacement\. A resample whose fitted asymptote stays below the threshold is censored\.

Table 1:Fitted threshold crossings\. “Copy the stem” is the share of generations that begin with the requested novel stem, and “first/second feature correct” is the share of single\-feature probes whose ending matches its prompt definition\. Thresholds follow the first block unless stated\.†Extrapolated beyond 5T tokens, with 8% of resamples censored\.‡Intervals were not retained for these two fits\.∗77–100% of resamples are censored\. The re\-test items give a four\-choice crossing of 1\.79T for OLMo\-2\-7B \(Appendix[J](https://arxiv.org/html/2610.19113#A10)\)\.#### Ordering\.

In OLMo\-2\-7B every component behavior has crossed by 338B, before or during the 336B–630B interval in whichDDchanges sign, while the held\-out form crosses at 1,464B for four\-choice accuracy and 3,586B for exact generation\. OLMo\-2\-13B shows the same order, although its exact\-generation fit extrapolates beyond the observed run\. Pythia\-1\.4B reaches three of the six thresholds\. In the code scheme, held\-out accuracy and generation cross at 143B and 131B, before shown\-combination generation at 815B, so the component\-before\-selection order is established for the suffix scheme only\. The 1\.79T re\-test crossing and the 1,464B crossing come from different items, and neither alters the 336B–630B blocking interval\.

## Appendix CFinal scores and prompt\-variant controls

Table[2](https://arxiv.org/html/2610.19113#A3.T2)lists the scores behind Figure[2](https://arxiv.org/html/2610.19113#S4.F2)\. Five changes to the prompt’s surface form test whether Pythia keeps preferring the copied answer and Qwen3\-1\.7B, Qwen3\-8B and OLMo\-2\-7B the correct one \(Figure[10](https://arxiv.org/html/2610.19113#A3.F10)\)\.

OLMo\-2\-1B \(\.58\.58\), Qwen3\-0\.6B \(\.50\.50\) and Qwen2\.5\-1\.5B \(\.49\.49\) lie between the thresholds\. Qwen2\.5\-0\.5B \(\.86\.86\) is the only non\-Pythia high\-copy model\. The correct form is generated on\.00\.00–\.05\.05of prompts in the high\-copy group and\.30\.30–\.93\.93in the low\-copy group \(Table[2](https://arxiv.org/html/2610.19113#A3.T2)\)\.

Table 2:Final\-checkpoint suffix\-task scores \(two item sets, 320 items\)\.P⁡\(copy\)=pcopied/\(pcopied\+pcorrect\)P\(\\mathrm\{copy\}\)=p\_\{\\mathrm\{copied\}\}/\(p\_\{\\mathrm\{copied\}\}\+p\_\{\\mathrm\{correct\}\}\);\.5\.5means equal probability\. “Ranks first” is the share of prompts where the correct form leads four candidates \(chance\.25\.25\) under summed log\-probabilities; the length\-normalized share differs by at most\.02\.02for every model\. Generation columns omit malformed outputs\. OLMo\-2 7B is the released stage\-2 parameter average \(Appendix[H](https://arxiv.org/html/2610.19113#A8)\)\. Qwen2\.5\-Coder\-1\.5B \(P⁡\(copy\)=\.42P\(\\mathrm\{copy\}\)=\.42\) and Coder\-7B \(\.02\.02\) are omitted\.#### Prompt variants\.

Pythia’sP⁡\(copy\)P\(\\mathrm\{copy\}\)stays at\.66\.66–\.95\.95under every variant keeping the closest example and falls to\.32\.32–\.46\.46with single\-feature lines only \(Figure[10](https://arxiv.org/html/2610.19113#A3.F10)and the supplement\)\. Qwen3\-1\.7B and released OLMo\-2\-7B rank the correct form first on at least\.45\.45of prompts under every variant but combined\-form lines only\. Qwen3\-8B, run only on the reworded template, ranks it first on1\.001\.00\.

Figure 10:Prompt\-variant controls\. \(a\)P⁡\(copy\)P\(\\mathrm\{copy\}\)and \(b\) the fraction of prompts where the correct form ranks first\. Series are Pythia\-410M, Pythia\-1\.4B, Pythia\-2\.8B, Qwen3\-1\.7B, and released OLMo\-2\-7B\. Qwen3\-8B and the OLMo\-2\-7B 336B and 3\.9T checkpoints appear only for the reworded template\. Variants reverse ending order \(B–A\), use one or three example lines per combination, retain only combined\-form or single\-feature lines, or reword the template\. The shaded single\-feature\-only condition has no closest combined example\. Guides mark equal copied/correct probability \(\.5\.5\) and four\-choice chance \(\.25\.25\)\.

## Appendix DBlocking, deletion, and scoring checks

Table[3](https://arxiv.org/html/2610.19113#A4.T3)lists every blocked span per model in bfloat16 on the original items\. The body and Figure[3](https://arxiv.org/html/2610.19113#S4.F3)use float32 re\-test scores \(Precision, below\)\. Figure[4](https://arxiv.org/html/2610.19113#S4.F4)shows one model per group\.

#### Blocked spans\.

Blocking the closest ending raisesmmwherever measured, by\+1\.00\+1\.00to\+5\.34\+5\.34nats, and blocking the requested value lowers it by−1\.59\-1\.59to−10\.51\-10\.51\(Table[3](https://arxiv.org/html/2610.19113#A4.T3)\)\. Pythia’sDDis positive \(\+2\.77\+2\.77to\+3\.75\+3\.75\), OLMo\-2\-1B’s is near zero \(−0\.23\-0\.23\[−0\.74,\+0\.29\-0\.74,\+0\.29\]\), and Qwen3\-0\.6B and all measured low\-copy models have negativeDD\(−1\.36\-1\.36to−9\.44\-9\.44\)\. Single\-feature lines affect every tested model and unrelated lines change little\. Qwen3\-4B’s closest\-ending block was not run\.

Table 3:Blocking selected prompt spans for the held\-out form \(item sets 0 and 1, 320 items, bfloat16;†one set, 160 items\)\. The body and Figure[3](https://arxiv.org/html/2610.19113#S4.F3)quote float32 values on item sets 2 and 3 \(supplement table\)\.mm\(unblocked\) is the full\-prompt correct\-minus\-copied log\-probability in nats; every other entry is the change inmmafter blocking the named span\.*Closest example*: only its ending tokens \(*ending*\) or its two whole lines \(*two lines*\)\.*Question words*: the word naming the requested value \(*asked*\) or the other value \(*other*\)\.*Controls*: two unrelated single\-feature lines \(*unrelated*\), the two single\-feature lines \(*single\-feature lines*\) and only their ending tokens \(*their endings*\)\.DDis the magnitude for*ending*minus that for*asked*, with a 95% item\-bootstrap interval\. – marks a span not run \(Qwen3 4B’s closest\-ending block\) or not tabulated here\.‡Run added after the model set was fixed; float32 and bfloat16 agree to \.02 nats for this model\. Four of the 17 models of Table[2](https://arxiv.org/html/2610.19113#A3.T2)\(Pythia 410M and 6\.9B, Qwen2\.5 1\.5B and 3B\) were not run\.Blocking the closest ending moves Pythia\-2\.8B’smmfrom−2\.1\-2\.1to\+2\.6\+2\.6nats and blocking the requested value moves Qwen3\-8B’s from\+8\.9\+8\.9to−1\.5\-1\.5, reversing the preferred answer in both \(Figure[4](https://arxiv.org/html/2610.19113#S4.F4)and Table[7](https://arxiv.org/html/2610.19113#A10.T7), re\-test items in bfloat16\)\.

#### Deleting lines\.

Table[4](https://arxiv.org/html/2610.19113#A4.T4)repeats the test with lines deleted instead of masked\. Deleting the closest example makes the correct answer preferred in Pythia\-1\.4B, Pythia\-2\.8B and Qwen3\-0\.6B \(\+3\.20\+3\.20,\+3\.30\+3\.30and\+3\.52\+3\.52nats\)\. The gain shrinks as unblockedmmrises, reaching−0\.19\-0\.19in Qwen3\-8B\. Deleting the correct\-ending lines lowersmmeverywhere \(−0\.53\-0\.53to−1\.52\-1\.52\)\. Deleting unrelated lines changesmmby−0\.13\-0\.13and\+0\.05\+0\.05in Pythia and by−0\.44\-0\.44to−0\.89\-0\.89in tested low\-copy models\. Eight of the twelve models in the blocking panel were tested \(Table[4](https://arxiv.org/html/2610.19113#A4.T4)\)\.

Table 4:Deletion effects on item sets 0 and 1, 160 items each\. Fullmmis the unblocked correct\-minus\-copied log\-probability in nats\. Columns 3–5 give the change inmmafter deleting the closest example’s two lines, the two single\-feature lines defining the correct second ending, or two unrelated lines of matched length\.
#### Qwen2\.5\-0\.5B\.

This model was added after the model set was fixed and run on independently generated items\. It is not part of the directional test\. Blocking the closest ending raisesmmby\+3\.25\+3\.25nats and lowersP⁡\(copy\)P\(\\mathrm\{copy\}\)from\.88\.88to\.42\.42, blocking the requested value lowersmmby\.66\.66, and the other question value and unrelated lines move it by\+\.6\+\.6and\+\.1\+\.1\. Its bfloat16DDis\+2\.59\+2\.59\[\+2\.34,\+2\.85\+2\.34,\+2\.85\], positive on88%88\\%of items, and float32 agrees to\.02\.02nats\. The activation patch of Figure[13](https://arxiv.org/html/2610.19113#A7.F13)produces none of the 109 never\-shown targets and a shown combination on\.36\.36and\.31\.31in the two item sets\.

#### Scoring variants\.

Restricting to single\-token endings, scoring only the first token where the forms differ, or length\-normalizing leavesP⁡\(copy\)P\(\\mathrm\{copy\}\)within\.01\.01and changes neither the sign ofDDnor any model’s group\. Item\-level first\-token and summed scores correlate at\.84\.84–1\.01\.0formmand\.90\.90–\.98\.98forDD\. All three put OLMo\-2\-7B’s last positiveDDat 336B and first negative at 630B, and OLMo\-2\-13B’s first sign change at 168B–336B\. Length normalization moves the correct\-ranks\-first fraction by at most\.02\.02in final models and at most\.05\.05across checkpoints \(at 5B, where both scores are near chance\)\. Two other checkpoints move by\.03\.03\. A later analysis that conditionsDDon full\-promptmmmoves the OLMo\-2\-7B sign change to 168B–336B without replacing the unconditioned series \(Appendix[P](https://arxiv.org/html/2610.19113#A16), supplement table\)\.

#### Precision\.

We re\-scoredP⁡\(copy\)P\(\\mathrm\{copy\}\), the blocked\-span changes inmm, andDDin float32 on the re\-test items and the OLMo\-2\-7B and OLMo\-2\-13B checkpoints, which the body and Figure[3](https://arxiv.org/html/2610.19113#S4.F3)use\. In OLMo,P⁡\(copy\)P\(\\mathrm\{copy\}\)changes by at most\.005\.005andDDby at most\.25\.25nats, pooled signs are unchanged, and item\-level signs agree on9494–100%100\\%\. In Pythia, bfloat16 overstates closest\-ending changes by1313–62%62\\%andDDby1212–105%105\\%across models and item sets\. Pooled float32DDfor Pythia\-1\.4B, \-2\.8B and \-12B is\+2\.4\+2\.4,\+2\.1\+2\.1and\+1\.9\+1\.9, against\+3\.3\+3\.3,\+2\.4\+2\.4and\+3\.7\+3\.7in bfloat16\. Signs, groups and checkpoint ordering are unchanged\. On 50 Pythia\-1\.4B items, padded batches and single sequences differ by under\.001\.001nats in float32 and up to2\.62\.6nats per form in bfloat16 \(supplement table\)\.

## Appendix ETraining trajectories of seventeen runs

The body’s pretraining evidence comes from one lab \(Ai2\)\. Here we follow the eleven older public series and six models selected from a screen of sixteen other released models \(Table[5](https://arxiv.org/html/2610.19113#A5.T5)\), asking in each whetherDDturns negative before four\-choice accuracy rises\. Figure[11](https://arxiv.org/html/2610.19113#A5.F11)and Table[6](https://arxiv.org/html/2610.19113#A5.T6)date the crossings\.

#### Runs and screen\.

Seven trajectories end above four\-choice chance: OLMo\-2\-7B, OLMo\-2\-13B, OLMo\-7B\-0424, OLMo\-3\-7B, Marin\-8B\-Base, SmolLM3\-3B and Comma\-v0\.1\-1T\. The other ten end below chance\. The screen keeps models whose correct form ranks first on at least 160 of 320 items\. We follow all four, plus OpenLLaMA\-7B\-v2 \(\.17\.17\) and SmolLM2\-1\.7B \(\.12\.12\), the two low\-accuracy models with the highest correct\-ranks\-first rates\. Apertus\-8B\-2509 \(\.42\.42\) clears only the table’s partial threshold\. OLMo\-3 is another Ai2 run, so it is not independent evidence\.

Released modelcorrect ranksfirst \(of 320\)writescorrectP⁡\(copy\)P\(\\mathrm\{copy\}\)Δ\\Deltaclosest endingDDsourceMarin\-8B\-Base†283 \(0\.88\)\.62\.02\+1\.52\+1\.52\[1\.31,1\.73\]\[1\.31,1\.73\]−6\.19\-6\.19[Marin Community \(2025\)](https://arxiv.org/html/2610.19113#bib.bib81)Comma\-v0\.1\-1T†277 \(0\.87\)\.50\.06\+1\.46\+1\.46\[1\.23,1\.69\]\[1\.23,1\.69\]−6\.40\-6\.40[Kandpal et al\. \(2025\)](https://arxiv.org/html/2610.19113#bib.bib68)SmolLM3\-3B†271 \(0\.85\)\.54\.10\+1\.74\+1\.74\[1\.51,1\.96\]\[1\.51,1\.96\]−5\.10\-5\.10[Hugging Face \(2025\)](https://arxiv.org/html/2610.19113#bib.bib69)OLMo\-3\-7B†265 \(0\.83\)\.43\.05\+1\.63\+1\.63\[1\.40,1\.85\]\[1\.40,1\.85\]−6\.24\-6\.24[Team Olmo \(2026\)](https://arxiv.org/html/2610.19113#bib.bib67)Apertus\-8B\-2509136 \(0\.42\)\.15\.23\+2\.13\+2\.13\[1\.91,2\.36\]\[1\.91,2\.36\]−3\.56\-3\.56[Swiss AI Initiative \(2025\)](https://arxiv.org/html/2610.19113#bib.bib80)OpenLLaMA\-7B\-v2†55 \(0\.17\)\.06\.55\+2\.98\+2\.98\[2\.78,3\.18\]\[2\.78,3\.18\]−0\.20\-0\.20[OpenLM Research \(2023\)](https://arxiv.org/html/2610.19113#bib.bib85)SmolLM2\-1\.7B†38 \(0\.12\)\.05\.64\+3\.92\+3\.92\[3\.71,4\.12\]\[3\.71,4\.12\]\+1\.13\+1\.13[Ben Allal et al\. \(2025\)](https://arxiv.org/html/2610.19113#bib.bib66)Lucie\-7B30 \(0\.09\)\.03\.72\+4\.51\+4\.51\[4\.31,4\.71\]\[4\.31,4\.71\]\+2\.22\+2\.22[OpenLLM\-France \(2025\)](https://arxiv.org/html/2610.19113#bib.bib83)YuLan\-Mini \(2\.4B\)27 \(0\.08\)\.06\.68\+4\.24\+4\.24\[4\.02,4\.45\]\[4\.02,4\.45\]\+0\.92\+0\.92[Hu et al\. \(2024\)](https://arxiv.org/html/2610.19113#bib.bib87)RedPajama\-INCITE\-7B\-Base26 \(0\.08\)\.03\.64\+3\.84\+3\.84\[3\.65,4\.04\]\[3\.65,4\.04\]\+1\.14\+1\.14[Together Computer \(2023\)](https://arxiv.org/html/2610.19113#bib.bib84)Viking\-7B18 \(0\.06\)\.02\.61\+3\.55\+3\.55\[3\.29,3\.78\]\[3\.29,3\.78\]\+0\.53\+0\.53[LumiOpen \(2024\)](https://arxiv.org/html/2610.19113#bib.bib82)BLOOM\-7b114 \(0\.04\)\.02\.76\+2\.96\+2\.96\[2\.75,3\.17\]\[2\.75,3\.17\]\+0\.95\+0\.95[BigScience Workshop \(2024\)](https://arxiv.org/html/2610.19113#bib.bib86)INTELLECT\-1 \(10B\)11 \(0\.03\)\.01\.74\+2\.83\+2\.83\[2\.62,3\.03\]\[2\.62,3\.03\]\+0\.77\+0\.77[Jaghouar et al\. \(2024\)](https://arxiv.org/html/2610.19113#bib.bib79)FineWeb\-Edu ablation model \(1\.82B\)3 \(0\.01\)\.01\.94\+7\.35\+7\.35\[7\.03,7\.64\]\[7\.03,7\.64\]\+6\.48\+6\.48[Hugging Face \(2024\)](https://arxiv.org/html/2610.19113#bib.bib88)SmolLM2\-360M3 \(0\.01\)\.00\.90\+4\.52\+4\.52\[4\.30,4\.76\]\[4\.30,4\.76\]\+3\.80\+3\.80[Ben Allal et al\. \(2025\)](https://arxiv.org/html/2610.19113#bib.bib66)TinyLlama\-1\.1B2 \(0\.01\)\.01\.90\+4\.90\+4\.90\[4\.69,5\.11\]\[4\.69,5\.11\]\+4\.11\+4\.11[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.19113#bib.bib89)Table 5:Screen of 16 released models \(320 items\), sorted by the number of items on which the correct form ranks first\. Rules separate high \(at least 160 correct\), partial \(Wilson lower bound above\.25\.25but fewer than 160 correct\) and low \(neither\)\. “Δ\\Deltaclosest ending” is the change in correct\-minus\-copied log\-probability after blocking that ending, with its 95% interval;DDsubtracts the magnitude after blocking the requested value\.†The six runs followed across their saved intermediate models: the four high models and two contrast series\.
#### Crossings\.

Figure[11](https://arxiv.org/html/2610.19113#A5.F11)a dates the first sampled crossing of each threshold\. In every run ending above chance, the first negativeDDcomes before or at the sampled accuracy crossing, though a first crossing need not last\. No below\-chance run reaches a sustained negativeDD\.

Figure 11:Crossing dates and endpoints of the training trajectories\. \(a\) First sampled crossings ofP⁡\(copy\)<\.5P\(\\mathrm\{copy\}\)<\.5,D<0D<0, and correct\-ranks\-first\>\.25\>\.25, with fitted accuracy crossings shown separately\. “Never marks an unobserved sampled crossing\. Arrows mark fitted crossings beyond the observed run, and Marin is already above threshold at its first public checkpoint\. The numbered checks are a closest\-ending effect below half its peak, an effect that stays below\.8\.8afterD<0D<0, a requested\-value effect that more than doubles, at most one sign change, a copied\-answer probability crossing after the last positiveDD, and a fitted accuracy crossing after the sign change\. OpenLLaMA\-7B\-v2 is measured in optimizer steps: its first negativeDDoccurs at 460k, but it never crosses the probability or accuracy thresholds\. \(b\) Final closest\-ending effect divided by its peak versus final requested\-value effect in nats\. The groups distinguish runs meeting all ordering criteria, other above\-chance runs, and below\-chance runs\. The vertical guide marks half the peak\. Exact sampled and fitted dates are in Table[6](https://arxiv.org/html/2610.19113#A5.T6)\.Table 6:First sampled checkpoint crossing each threshold on 320 independently generated items\. The fitted column is the logistic crossing for the fraction of prompts on which the correct form ranks first \(Appendix[B](https://arxiv.org/html/2610.19113#A2)\); “not reached” marks a fitted crossing beyond the last observed checkpoint\.≤\\leq: Marin\-8B\-Base already exceeds the threshold at its first public checkpoint\. OLMo\-2\-7B’s fitted crossing is the re\-test value, 1\.79T \(Appendix[J](https://arxiv.org/html/2610.19113#A10)\)\. A series may return after its first crossing\.†OpenLLaMA positions are optimizer steps rather than tokens\. The six ordering criteria per run are visible in Figure[11](https://arxiv.org/html/2610.19113#A5.F11)\.
#### Ordering criteria and exceptions\.

OLMo\-2\-7B, OLMo\-3\-7B and Comma\-v0\.1\-1T meet all six checks in the figure caption\. Their sign\-change brackets are 336B–630B, 411B–600B and 482B–545B\. OLMo\-2\-13B meets five but changes sign three times, first at 168B–336B, back above zero at 630B, and again at 630B–839B\. SmolLM3\-3B shifts before accuracy rises, but its closest\-ending effect later climbs back above\.8\.8of its peak\. Marin\-8B\-Base is already past both thresholds at its first public checkpoint, so its ordering cannot be dated\. OpenLLaMA\-7B\-v2’s meanDDdips to−0\.20\-0\.20only at its last checkpoint, and SmolLM2\-1\.7B’s stays positive\.

OLMo\-7B\-0424 is the exception\. Its first negativeDDis at 838B, but it hovers around zero until 2\.3T, while fitted accuracy crosses at 1\.6T and sampled accuracy exceeds\.25\.25at 2\.0T\. The run fails three of the six checks\.

After 1T in OLMo\-2\-7B, the requested\-value effect moves between2\.92\.9and5\.35\.3nats across neighbouring checkpoints, and the closest\-ending effect briefly reaches3\.13\.1at 1\.68T\. The two never re\-cross, andDDstays negative at every later checkpoint \(Figure[6](https://arxiv.org/html/2610.19113#S5.F6)\)\.

#### Endpoints\.

The seven above\-chance runs finish with requested\-value effects of4\.54\.5–7\.77\.7nats and closest\-ending effects at\.28\.28–\.56\.56of their observed peaks\. The ten below\-chance runs finish at\.9\.9–3\.23\.2nats and\.44\.44–\.72\.72of peak \(Figure[11](https://arxiv.org/html/2610.19113#A5.F11)b\)\. Pythia\-1\.4B\-deduped and Amber\-7B do fall below half their peaks \(\.44\.44and\.49\.49\), yet keep positiveDD\(\+3\.1\+3\.1,\+1\.6\+1\.6\) and highP⁡\(copy\)P\(\\mathrm\{copy\}\)\(\.80\.80,\.72\.72\)\. The released OLMo\-2\-1B lies outside its stage\-1 trajectory, atP⁡\(copy\)=\.57P\(\\mathrm\{copy\}\)=\.57andD=−0\.6D=\-0\.6\[−1\.0,−0\.2\-1\.0,\-0\.2\]\. Appendix[H](https://arxiv.org/html/2610.19113#A8)covers the same stage\-1 versus released difference for OLMo\-2\-7B\.

## Appendix FCode and Quechua versions

Two further versions test whether the model ordering of Section[4](https://arxiv.org/html/2610.19113#S4)depends on the invented suffixes or on tokenization \(Figure[12](https://arxiv.org/html/2610.19113#A6.F12)\)\. One uses code words and symbols in place of endings\. The other uses Southern Quechua noun paradigms, scored with each suffix as its own token and in standard spelling\.

#### Code scheme\.

The six tested models keep the suffix\-scheme ordering and score better\.P⁡\(copy\)P\(\\mathrm\{copy\}\)is\.55\.55,\.38\.38,\.17\.17,\.06\.06,\.01\.01and\.00\.00for Pythia\-1\.4B, Pythia\-2\.8B, OLMo\-2\-7B, Qwen3\-1\.7B, Qwen2\.5\-7B and Qwen3\-8B \(Figure[12](https://arxiv.org/html/2610.19113#A6.F12)a\)\. Blocking here covers the closest example’s whole line, so its size is not comparable with suffix\-ending blocking\. In OLMo\-2\-7B the requested\-value effect first exceeds the closest\-example effect at 168B, inside the interval of the fitted four\-choice crossing, 143B \[115, 178\]\. The two events coincide at this resolution, whereas in the suffix scheme the blocking turn came first \(Figure[12](https://arxiv.org/html/2610.19113#A6.F12)b\)\.

#### Quechua construction\.

Southern Quechua nouns combine possessor and case endings\([Adelaar and Muysken, 2004](https://arxiv.org/html/2610.19113#bib.bib33);[Kirov et al\., 2018](https://arxiv.org/html/2610.19113#bib.bib29);[Batsuren et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib30)\)\. Of the 494 UniMorphquenoun lemmas we keep the 242 vowel\-final, lower\-case ASCII lemmas whose cells are exact concatenations of lemma, possessor and case \(consonant\-final lemmas would need epenthesis\), minus function words, loans and proper names\. Possessor endings are*\-y, \-yki, \-n, \-nchik, \-nku*, and case endings*\-man, \-manta, \-pi, \-wan, \-kama, \-paq, \-rayku*\. Excluding prefix\-related pairs \(*y*/*yki*,*n*/*nchik*,*n*/*nku*,*man*/*manta*\) leaves 7 possessor pairs and 20 case pairs\. Each prompt draws one pair from each pool, and the closest example shares the possessor\. Segmented forms look liketanta\-y\-pi, standard spelling liketantaypi\.

#### Tokenization and scores\.

With suffixes tokenized separately, Pythia hasP⁡\(copy\)=\.77P\(\\mathrm\{copy\}\)=\.77–\.84\.84and correct\-ranks\-first\.04\.04–\.06\.06\. Qwen3\-1\.7B, Qwen2\.5\-7B, Qwen3\-4B and Qwen3\-8B haveP⁡\(copy\)=\.00P\(\\mathrm\{copy\}\)=\.00–\.14\.14and correct\-ranks\-first\.83\.83–1\.001\.00, and OLMo\-2 lies between\. Blocking the requested value lowersmmby−1\.4\-1\.4to−8\.5\-8\.5nats, while blocking the closest ending raises it by\+\.8\+\.8to\+5\.3\+5\.3\. Standard spelling, where the tokenizer often fuses the stem\-final vowel with the possessor suffix, drops eight models to correct\-ranks\-first\.01\.01–\.10\.10\(Qwen3\-4B\.31\.31, Qwen3\-8B\.42\.42\) and raisesP⁡\(copy\)P\(\\mathrm\{copy\}\)to\.37\.37–\.93\.93, though requested\-value blocking still orders the models \(−1\.1\-1\.1to−6\.0\-6\.0nats\)\. Models do not consistently prefer the copy, which shares the possessor, over the shown form that shares the case, so Quechua shows a pull toward shown forms as such\.

Figure 12:Surface\-form and tokenization checks\. \(a\)P⁡\(copy\)P\(\\mathrm\{copy\}\)and correct\-ranks\-first for the suffix scheme, code scheme, segmented Quechua, and Quechua in standard spelling\. Six models have code results and ten have Quechua results\. Guides mark\.5\.5and\.25\.25\. The broad model ordering survives the code and segmented\-Quechua versions, while standard spelling reduces accuracy in every model\. \(b\) OLMo\-2\-7B in the code scheme: magnitudes of the changes inmmfrom blocking the closest example’s line or the requested value, withP⁡\(copy\)P\(\\mathrm\{copy\}\)on the other axis\. The blocking turn at 168B overlaps the fitted accuracy crossing near 143B \(Table[1](https://arxiv.org/html/2610.19113#A2.T1)\)\.
#### Contamination check\.

Four\-choice accuracy is\.21\.21–\.31\.31with only the Quechua header and query, and\.22\.22–\.29\.29with demonstrations whose possessors and cases differ from the query’s\. All Wilson intervals contain\.25\.25except OLMo\-2\-13B’s header\-only\.31\.31\[\.26,\.37\.26,\.37\], so we show that model’s Quechua scores but exclude them from the ordering\. No model generates a correct form on any of the 320 header\-only items\. Training results on the code and Quechua versions are in Appendix[M](https://arxiv.org/html/2610.19113#A13), adapter results in Appendix[N](https://arxiv.org/html/2610.19113#A14)\.

## Appendix GActivation patching

Under the patch of Section[3](https://arxiv.org/html/2610.19113#S3), never\-shown shares in the low\-copy group rise asP⁡\(copy\)P\(\\mathrm\{copy\}\)falls:\.26\.26for OLMo\-2\-7B,\.33\.33for OLMo\-2\-13B,\.39\.39for Qwen3\-1\.7B,\.65\.65for Qwen2\.5\-7B and\.91\.91for Qwen3\-8B\. Shown shares are\.85\.85–1\.001\.00\. Pythia’s shown share is\.59\.59–\.69\.69, but its never\-shown share never exceeds\.08\.08\(Figure[13](https://arxiv.org/html/2610.19113#A7.F13)\)\. Counting only the requested ending, regardless of stem or other ending, Pythia’s never\-shown shares rise to\.11\.11,\.31\.31and\.19\.19while the low\-copy models move by at most\.03\.03\(supplement table\)\. High\-copy models can produce the requested ending without building the full form\.

![Refer to caption](https://arxiv.org/html/2610.19113v1/fig_patch.png)Figure 13:Activation patching after layer 4 at the requested\-value word\. The two series give the shares producing the complete form named by the replacement value when that target is never shown or already shown\. Models are grouped into high\-copy and low\-copy and ordered by their never\-shown share\. Both conditions start from a shown base form\.#### Why this position and layer?

Distributed alignment search\([Wu et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib32)\)on the final Pythia\-1\.4B fits an 8\-dimensional rotated subspace at the requested\-value word, the last stem token and “=”\. Only the requested\-value word responds, most strongly after layer 4, where the greedy ending changes on most items \(an approximation\) for the first feature and about half for the second\. Transfer is\.97\.97on the three fitted word pairs but\.04\.04on the held\-out pair\. The layer\-4 success rate rises from\.28\.28at step 2k \(about 4B tokens\) to\.95\.95at step 143k \(300B\)\. Layer 4, the Pythia\-1\.4B optimum, is used as the shared early layer for every model, although per\-model sweeps peak at layers 0, 1, 9, 11 and 14 \(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\.

Both conditions in Figure[13](https://arxiv.org/html/2610.19113#A7.F13)start from a shown base form, and the never\-shown base\-form condition was not run\. Position\-by\-position patching retains most of the change inmmthrough the first\.33\.33of layers in Pythia\-1\.4B and\.53\.53–\.64\.64in low\-copy models\. This depth measurement replaces a planned between\-group comparison at the requested\-value word that could not be evaluated as specified \(Appendix[P](https://arxiv.org/html/2610.19113#A16)\)\.

## Appendix HStage\-2 OLMo\-2\-7B models

The released OLMo\-2\-7B in Table[2](https://arxiv.org/html/2610.19113#A3.T2)ranks the correct form first on \.69 of prompts and writes it on \.30, above the \.53 and \.17 at 3\.9T of the last stage\-1 checkpoint, which ends the trajectory of Figure[6](https://arxiv.org/html/2610.19113#S5.F6)\. The release is a parameter average of three stage\-2 runs on different data mixtures\([Team OLMo et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib10)\)\. In the one run with public intermediate checkpoints, correct\-ranks\-first rises from\.53\.53to\.64\.64and exact generation from\.17\.17to\.22\.22within 5B additional tokens\. Over 9B–50B, correct\-ranks\-first stays mostly at\.63\.63–\.68\.68, dipping to\.55\.55at 17B, and exact generation ranges over\.15\.15–\.27\.27\. A gain is therefore already present during stage\-2 training, before averaging\. The comparison cannot separate further training from the change of data mixture\.

## Appendix IFine\-tuning a copying model

We ask whether direct fine\-tuning of Pythia\-1\.4B reproduces the pretraining order,DDchanging sign before accuracy rises, and whether the all\-shown result of Section[6](https://arxiv.org/html/2610.19113#S6)also holds under fine\-tuning\.

#### Design\.

We fully fine\-tune released Pythia\-1\.4B for 2,000 steps \(learning rate10−510^\{\-5\}, warm\-up, cosine decay, batch size 32\)\. Training uses its own prompt template, evaluation the standard one\. Two seeds target the correct held\-out form\. A third run queries only shown combinations, mirroring the all\-shown condition of Section[6](https://arxiv.org/html/2610.19113#S6)\. We predicted that the bootstrap interval forDDwould fall below zero at least 250 steps before the Wilson lower bound for four\-choice accuracy exceeded\.25\.25, in both target runs, and that neither event would occur in the all\-shown run\.

#### Outcome\.

Both target runs cross both thresholds at the first evaluation, step 250 \(supplement figure\)\. On a 10\-step grid, accuracy clears chance at step 10 or 20 and passes\.9\.9by step 50, whileDDturns negative at step 20 in both runs\. The closest\-ending effect falls from5\.35\.3to about33nats and then fluctuates between\+2\.3\+2\.3and\+7\.6\+7\.6, while the requested\-value effect grows from22to more than2020\. The all\-shown run also learns the held\-out form\. At step 250 it hasP⁡\(copy\)=\.83P\(\\mathrm\{copy\}\)=\.83, a closest\-ending effect near3131nats andD=\+21D=\+21\. By step 500,DDis negative and correct\-ranks\-first is\.50\.50, rising to\.70\.70–\.78\.78from step 1,000 with closest\-ending and requested\-value effects near1010and3030nats\. Neither part of the predicted pretraining order is reproduced\. The requested\-value effect grows while the closest\-ending effect is still present, instead of after it disappears\.

## Appendix JRe\-test on new items

We repeat the tests of Sections[4](https://arxiv.org/html/2610.19113#S4)and[5](https://arxiv.org/html/2610.19113#S5)on 320 new prompts per model, generated after those analyses were fixed \(Table[7](https://arxiv.org/html/2610.19113#A10.T7)\)\. The re\-test covers new prompts from the same generator, not independently trained models\.

#### Criteria\.

P⁡\(copy\)P\(\\mathrm\{copy\}\)must be at least\.70\.70in the high\-copy group and at most\.30\.30in the low\-copy group,DDmust take the predicted sign in each group, and the unrelated\-line 95% interval must lie within±\.3\\pm\.3nats\. OLMo\-2\-7B’s sign change must fall in 336B–630B and its fitted four\-choice accuracy must cross chance afterwards\. OLMo\-2\-1B and Qwen3\-0\.6B have no directional prediction, leaving nine in the sign test\.

#### Results\.

All probability thresholds and all nine signs hold \(Table[7](https://arxiv.org/html/2610.19113#A10.T7)\)\. The 336B–630B interval recurs in\.999\.999of joint item resamples, and the 1\.79T accuracy crossing follows the sign change in every resample\. The unrelated\-line interval fails for Pythia\-1\.4B and Pythia\-12B in bfloat16, with means of only\+\.24\+\.24and\+\.17\+\.17, and passes in the float32 re\-scores \(Appendix[D](https://arxiv.org/html/2610.19113#A4)\)\. The six trajectory outcomes of Appendix[E](https://arxiv.org/html/2610.19113#A5)are unchanged\. OLMo\-2\-13B still shows three sign changes, and the low\-accuracy OLMo\-2\-1B, OLMoE and standard Pythia runs never turn negative\.

ModelP⁡\(copy\)P\(\\mathrm\{copy\}\)\[95% interval\]fullpromptclosestendingrequestedvalueothervalueunrelatedlinesDD\[95% interval\]P⁡\(copy\)P\(\\mathrm\{copy\}\)predicted side≥\.70/≤\.30\\geq\.70/\\leq\.30DDsignas predictedunrelated 95%interval within±\.3\\pm\.3natsPythia 1\.4B\.81 \[\.77, \.83\]−2\.73\-2\.73\+5\.33\+5\.33−2\.06\-2\.06\+1\.47\+1\.47\+0\.24\+0\.24\+3\.27\+3\.27\[\+2\.86,\+3\.68\]\[\+2\.86,\+3\.68\]✓✓×\\timesPythia 2\.8B\.77 \[\.74, \.79\]−2\.08\-2\.08\+4\.63\+4\.63−2\.26\-2\.26\+1\.05\+1\.05\+0\.16\+0\.16\+2\.38\+2\.38\[\+2\.10,\+2\.64\]\[\+2\.10,\+2\.64\]✓✓✓Pythia 12B\.85 \[\.82, \.87\]−3\.20\-3\.20\+5\.35\+5\.35−1\.63\-1\.63\+1\.83\+1\.83\+0\.17\+0\.17\+3\.72\+3\.72\[\+3\.32,\+4\.08\]\[\+3\.32,\+4\.08\]✓✓×\\timesOLMo\-2 1B\.57 \[\.53, \.60\]−0\.43\-0\.43\+2\.80\+2\.80−3\.40\-3\.40\+1\.00\+1\.00\+0\.11\+0\.11−0\.60\-0\.60\[−0\.98,−0\.23\]\[\-0\.98,\-0\.23\]–––Qwen3 0\.6B\.53 \[\.50, \.56\]−0\.15\-0\.15\+2\.88\+2\.88−4\.39\-4\.39\+0\.78\+0\.78\+0\.04\+0\.04−1\.52\-1\.52\[−1\.85,−1\.17\]\[\-1\.85,\-1\.17\]–––OLMo\-2 7B\.24 \[\.21, \.27\]\+2\.54\+2\.54\+2\.08\+2\.08−5\.54\-5\.54−0\.77\-0\.77\+0\.10\+0\.10−3\.46\-3\.46\[−3\.93,−3\.02\]\[\-3\.93,\-3\.02\]✓✓✓OLMo\-2 13B\.16 \[\.13, \.19\]\+3\.74\+3\.74\+1\.55\+1\.55−7\.07\-7\.07−1\.65\-1\.65−0\.13\-0\.13−5\.52\-5\.52\[−6\.09,−5\.01\]\[\-6\.09,\-5\.01\]✓✓✓Qwen3 1\.7B\.10 \[\.08, \.12\]\+3\.97\+3\.97\+1\.83\+1\.83−7\.07\-7\.07−0\.15\-0\.15−0\.06\-0\.06−5\.25\-5\.25\[−5\.65,−4\.81\]\[\-5\.65,\-4\.81\]✓✓✓Gemma\-3 4B\.11 \[\.09, \.13\]\+4\.17\+4\.17\+2\.64\+2\.64−7\.10\-7\.10−1\.54\-1\.54−0\.08\-0\.08−4\.46\-4\.46\[−4\.90,−4\.04\]\[\-4\.90,\-4\.04\]✓✓✓Qwen2\.5 7B\.02 \[\.01, \.02\]\+5\.75\+5\.75\+0\.98\+0\.98−7\.99\-7\.99−1\.74\-1\.74−0\.04\-0\.04−7\.02\-7\.02\[−7\.36,−6\.64\]\[\-7\.36,\-6\.64\]✓✓✓Qwen3 8B\.00 \[\.00, \.00\]\+8\.89\+8\.89\+0\.87\+0\.87−10\.41\-10\.41−2\.96\-2\.96\+0\.05\+0\.05−9\.54\-9\.54\[−9\.92,−9\.16\]\[\-9\.92,\-9\.16\]✓✓✓Table 7:Re\-test on item sets 2 and 3 \(320 items per model, bfloat16\)\.P⁡\(copy\)P\(\\mathrm\{copy\}\)andDDhave 95% item\-bootstrap intervals, resampling within each set\. Blocking columns give changes inmm\(nats\);DDis the closest\-ending change minus the magnitude of the requested\-value change\. The last columns test the predicted probability side, sign ofDD, and unrelated\-line interval\. The two intermediate models have no directional prediction\. The body quotes float32 re\-scores of these same items: PythiaP⁡\(copy\)P\(\\mathrm\{copy\}\)is lower by up to\.06\.06andDDby up to1\.81\.8nats\.

## Appendix KSelected attention heads

#### Head contributions across training\.

Across OLMo\-2\-7B training \(Figure[14](https://arxiv.org/html/2610.19113#A11.F14)\), the copy\-favoring set’s direct contribution peaks at\.71\.71nats at 9B, falls below\.10\.10by 336B, and is about\.02\.02–\.05\.05at most later checkpoints\. The correct\-favoring set stays near zero until 336B \(dipping to−\.06\-\.06at 84B–168B\), reaches\.30\.30at 630B, dips to\.14\.14at 1049B, and exceeds\.6\.6only after 2\.5T \(\.83\.83at 3\.57T\)\. Heads are re\-selected at each checkpoint\. Swapping the selection and evaluation item sets moves the crossing from 630B to 1\.68T \(Figure[14](https://arxiv.org/html/2610.19113#A11.F14)\), so this timing is less stable than the whole\-model blocking interval\.

![Refer to caption](https://arxiv.org/html/2610.19113v1/fig_heads_traj.png)Figure 14:Direct contributions toward each set’s own answer, in nats, across OLMo\-2\-7B training, for the five heads selected for the copied answer and the five for the correct answer\. Swapping selection and evaluation prompt sets moves the crossing to 1\.68T\. The star marks the crossing, inside the 336B–630B interval where whole\-modelDDchanges sign\.
#### Stability and contribution\.

Selected twice, once on each item set, correct\-answer heads overlap \(Jaccard\) at\.67\.67–11in five of six low\-copy models and\.43\.43in OLMo\-2\-13B\. Copy\-answer heads overlap at\.67\.67except in OLMo\-2\-13B \(\.43\.43\)\. Pythia overlaps are\.25\.25to\.67\.67\. In every low\-copy model, direct logit attribution\([Wang et al\., 2023](https://arxiv.org/html/2610.19113#bib.bib51)\)credits the selected heads with more of the correct\-answer contribution than the multilayer\-perceptron blocks, and with\.14\.14to\.64\.64of all head contributions toward that answer\. Only Gemma\-3\-4B reaches half for both sets \(\.64\.64and\.56\.56\), and no model’s five\-head sets explain half the matching token\-blocking effect in both selection directions \(Section[4](https://arxiv.org/html/2610.19113#S4), Figure[5](https://arxiv.org/html/2610.19113#S4.F5)b\)\.

#### Silencing selected heads\.

Figure[5](https://arxiv.org/html/2610.19113#S4.F5)b averages the two directions\. Silencing the copy\-favoring sets raisesmmby\.7\.7–2\.02\.0nats in Pythia\-1\.4B, Qwen3\-1\.7B and the OLMo\-2\-7B checkpoints from 168B to 1049B, but only\.06\.06in Qwen3\-8B, within its random\-set range\. Silencing the correct\-favoring sets lowersmmby\.4\.4–1\.21\.2nats\. The copy\-head effect is as large at 168B as at 1049B, so these heads do not fall silent when control switches\. The threshold is twice the 95th percentile of twenty layer\-matched random sets in each direction\. Copy\-favoring sets clear it both ways in Qwen3\-1\.7B and OLMo\-2\-7B at 168B–1049B, but final OLMo\-2\-7B and Pythia\-1\.4B clear it one way only, which the averaged points hide\. Correct\-favoring sets clear it both ways in Pythia\-1\.4B and OLMo\-2\-7B and one way in the Qwen3 models\.

#### Induction\-head control\.

A separate, task\-independent control uses Pythia’s 16 heads with the highest induction score on a random 40\-token repeat\([Olsson et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib6)\)\. Replacing their answer\-position outputs with those from other prompts raises correct\-next\-ending accuracy, given correct preceding tokens, from\.02\.02to\.36\.36in Pythia\-1\.4B at step 4k \(\.39\.39and\.33\.33on the two item sets; random controls\.02\.02–\.03\.03\) and from\.18\.18to\.49\.49in final Pythia\-2\.8B\. Mean replacement gives\.44\.44in Pythia\-2\.8B\. In free generation, shown\-combination accuracy falls from\.66\.66to\.26\.26and\.81\.81to\.56\.56, so copying weakens, but held\-out exact generation rises only from\.03\.03to\.07\.07and\.05\.05to\.11\.11\.

## Appendix LContinued pretraining with synthetic documents and attention penalties \(360M\)

Five conditions continue pretraining SmolLM2\-360M for about 1B tokens, two runs each: original documents, two synthetic\-document variants and two attention penalties \(Section 6\.1; Figure[15](https://arxiv.org/html/2610.19113#A12.F15)\)\.

#### Design\.

All conditions share the starting checkpoint, compute budget and data order\.

- •Base:SmolLM2\-360Matstep\-1600000\(2\.52 trillion tokens\), continued on the public proxy corpussmollm\-corpus\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib66)\)for 636 steps, about 1 billion tokens\.
- •Synthetic documents: every 33rd packed sequence \(3 %, about 30 million tokens\) is replaced by documents that define endings\. One\-unshown documents leave one combination unshown and give its answer\. All\-shown documents show every requested combination on another stem\. Training stems, endings and templates are disjoint from the evaluation inventories, and neither condition contains code systems\.
- •Attention penalties: original documents plus a penalty on attention from the answer position to the closest ending, on either the ten heads with the largest direct contribution to the copied answer in the base model or ten random heads in the same layers \(Appendix[N](https://arxiv.org/html/2610.19113#A14)\)\.
- •Measures:P⁡\(copy\)P\(\\mathrm\{copy\}\), correct\-ranks\-first, exact generation andDD, code and Quechua scores, three external evaluations defined in the supplement, FineWeb\-Edu and Pile validation losses, and the penalized attention\.
- •Crossings: a condition’sDDturns negative at the first checkpoint with meanD<0D<0andP⁡\(copy\)≤\.55P\(\\mathrm\{copy\}\)\\leq\.55, and it reaches the accuracy criterion at the first with correct\-ranks\-first at least\.50\.50and exact generation at least\.15\.15\.

#### Comparisons\.

An intervention counts as earlier only if it crosses before its control, or crosses when the control never does\. Transfer to code and Quechua requires a gain of at least\.05\.05on both measures of both tasks in both runs, each with a positive 95% paired item\-bootstrap interval, and both validation losses within\+\.02\+\.02nats of original documents\.

#### Outcome\.

Both synthetic conditions cross both thresholds at step 32, about 50M tokens, so they tie and neither counts as earlier \(Figure[15](https://arxiv.org/html/2610.19113#A12.F15)\)\. At the end the all\-shown runs haveP⁡\(copy\)P\(\\mathrm\{copy\}\)of\.27/\.19\.27/\.19and exact generation of\.65/\.77\.65/\.77\. The one\-unshown runs have\.27/\.13\.27/\.13and\.71/\.84\.71/\.84\. The one\-unshown condition meets the transfer and loss criteria against original documents but not against all\-shown documents\. The selected\-head penalty drives the penalized attention from\.310/\.341\.310/\.341to\.008/\.006\.008/\.006, yetP⁡\(copy\)P\(\\mathrm\{copy\}\)stays at\.56/\.57\.56/\.57,DDat\+1\.01/\+1\.60\+1\.01/\+1\.60, and exact generation at zero\. Original documents and both penalty conditions reach neither threshold within 1B tokens, and the selected\-head penalty does not reach the transfer threshold\. Practicing an unshown combination is therefore not isolated as the cause, and reducing the selected heads’ attention to the closest ending does not create composing\.

Figure 15:SmolLM2\-360M after about 1B additional tokens\. Conditions: original documents, one\-unshown and all\-shown synthetic documents, and penalties on selected or random heads, two runs per condition as joined markers\. \(a\)P⁡\(copy\)P\(\\mathrm\{copy\}\), \(b\) exact generation and correct\-ranks\-first, \(c\) code and Quechua exact generation, \(d\) selected\-head and random\-head attention to the closest ending, and \(e\) FineWeb\-Edu and Pile validation loss\. Both synthetic conditions crossD<0D<0and the accuracy criterion at step 32 \(about 50M tokens\)\. The other conditions never cross within 1B\.

## Appendix MContinued pretraining with synthetic documents and mined passages \(1\.7B\)

Five conditions continue pretraining SmolLM2\-1\.7B for 1B tokens: plain continuation, 3% synthetic documents, and three mined\-passage variants \(Sections 6\.1–6\.2; Figures[16](https://arxiv.org/html/2610.19113#A13.F16)and[7](https://arxiv.org/html/2610.19113#S6.F7)\)\.

#### Design\.

All conditions share the base checkpoint and training schedule\.

- •Base:SmolLM2\-1\.7B\-intermediate\-checkpointsat 2\.097 trillion tokens\([Ben Allal et al\., 2025](https://arxiv.org/html/2610.19113#bib.bib66)\)\.
- •Schedule: 477 steps of 2\.097 million tokens \(1B total\) at learning rate5×10−45\\times 10^\{\-4\}, identical rows across conditions, every 33rd row \(3\.03%3\.03\\,\\%\) reserved for replacement, and 21 checkpoints\.
- •Conditions: original documents, the synthetic documents of Appendix[L](https://arxiv.org/html/2610.19113#A12), mined passages that define and later use a term, mined passages whose term is absent from a fixed reference vocabulary, and the same passages with a dictionary\-format prefix\. The all\-shown control was not repeated at this scale for lack of compute \(Appendix[P](https://arxiv.org/html/2610.19113#A16)\)\.
- •Measures: the diagnostic \(n=160n=160\) with its code and Quechua versions, five external tasks \(n=300n=300for the first,n=200n=200for the others\), and FineWeb\-Edu and Pile loss\([Penedo et al\., 2024](https://arxiv.org/html/2610.19113#bib.bib77);[Gao et al\., 2020](https://arxiv.org/html/2610.19113#bib.bib78)\)\. The external tasks are in\-context SCAN \(add\-primitive\-jump split\) and a 14\-example SCAN\-style task\([Lake and Baroni, 2018](https://arxiv.org/html/2610.19113#bib.bib19)\), in\-context ElementalTask\([Liu et al\., 2026](https://arxiv.org/html/2610.19113#bib.bib20)\), nested use of two defined operators, and an unshown word\-to\-action mapping through word\-to\-category and category\-to\-action definitions\. None passes the pre\-set qualification threshold\.
- •Crossings: a condition’sDDturns negative whenD<0D<0withP⁡\(copy\)≤\.55P\(\\mathrm\{copy\}\)\\leq\.55, and it reaches the accuracy criterion when correct\-ranks\-first is at least\.50\.50and exact generation at least\.15\.15\.

The fixed comparisons require each intervention to cross while original documents do not and to keep both validation losses within\+\.02\+\.02nats\. Mined passages must cross no later than synthetic documents\. External\-task changes are evaluated with Holm\-adjusted item\-bootstrap intervals but cannot establish qualified transfer\.

#### Outcome\.

After 1B tokens, synthetic\-document training hasP⁡\(copy\)=\.006P\(\\mathrm\{copy\}\)=\.006and exact generation\.994\.994, while plain and mined\-passage runs keepP⁡\(copy\)P\(\\mathrm\{copy\}\)near\.84\.84and exact generation at\.013\.013\(Figure[16](https://arxiv.org/html/2610.19113#A13.F16)\)\. No external task changes by more than\.03\.03under any condition\. Synthetic documents cross both thresholds at the first checkpoint, 50\.3 million tokens\. The other four conditions never cross within 1B\.111The synthetic\-document comparison meets its criterion only after a Quechua scoring fix\. The first analysis read a missing correctness field and scored every Quechua item 0 \(Appendix[P](https://arxiv.org/html/2610.19113#A16)\)\.FineWeb\-Edu loss changes by at most\+\.0005\+\.0005nats, and Pile changes range from−\.0110\-\.0110to\+\.0003\+\.0003\. The comparisons are between whole training conditions and do not isolate which property of the documents matters\.

Figure 16:SmolLM2\-1\.7B after 1B additional tokens\. Series are original documents, 3% synthetic documents, mined define\-and\-use passages, mined passages with the term absent from the reference vocabulary, and mined passages with a dictionary prefix\. \(a\)P⁡\(copy\)P\(\\mathrm\{copy\}\), correct\-ranks\-first, exact generation, and code and Quechua exact generation\. \(b\) Exact\-match accuracy on SCAN, ElementalTask, 14\-example SCAN, two operators, and word\-to\-category\-to\-action\. No external\-task change is significant\. \(c\) FinalDD\. \(d\) Validation losses\.

## Appendix NLayer sweep, head penalties, adapters, and exercises

Full tables are in the supplement\. Figure[17](https://arxiv.org/html/2610.19113#A14.F17)and Table[8](https://arxiv.org/html/2610.19113#A14.T8)cover the exercises\.

Figure 17:Exercise\-training changes against original\-text controls, in percentage points\. The series are two SmolLM2\-1\.7B exercise fine\-tunes, a SmolLM3\-3B exercise fine\-tune, and a SmolLM2\-1\.7B 15% exercise mixture\. Rows separate made\-up or exchanged labels on unseen datasets; prompt\-defined labels and relations in seen formats; and farther tasks, including arithmetic, operators, List Functions, Unnatural In\-Context Learning, SCAN, ElementalTask, bAbI and BIG\-Bench Hard\. Filled markers are significant after correction for multiple comparisons, hollow ones are not; the band marks the±1\.5\\pm 1\.5\-point threshold\. Grey DBpedia\-7 results are excluded from 3B testing by the base\-only qualification rule\. The mixture control matches total tokens, not natural\-text amount or repetition \(Appendix[P](https://arxiv.org/html/2610.19113#A16)\)\.Table 8:One real example per modification type: a passage from a real document \(left\) and the definition\-use exercise a language model built from it \(right; definitions, then the query and its answer\), each verified by a second model before training\.After Holm correction, a group of evaluations improves if some score rises and none falls, has a cost if any falls, and is otherwise unchanged\. Controls use original text on the same schedules\.

#### Layer sweep\.

Activation\-patching effects peak at layers 0/1 in Pythia\-1\.4B, 9/0 in OLMo\-2\-7B, 0/1 in Qwen3\-1\.7B, 1/1 in Qwen2\.5\-7B, and 14/11 in Gemma\-3\-4B \(each pair selected on one item set and evaluated on the other\)\. The body’s layer 4 is a shared early layer from Pythia’s alignment search, not a common peak \(Appendix[G](https://arxiv.org/html/2610.19113#A7)\)\.

#### Head outputs\.

Neither multiplying selected\-head outputs in thirteen models \(against 19 layer\-matched random head sets\) nor fine\-tuning Pythia\-1\.4B with head\-derived penalties in five conditions beats its random\-head or plain fine\-tuning control\. At SmolLM2\-360M, an attention penalty cuts selected\-head attention from\.310/\.341\.310/\.341to\.008/\.006\.008/\.006whileDDstays positive at\+1\.01/\+1\.60\+1\.01/\+1\.60\(Appendix[L](https://arxiv.org/html/2610.19113#A12)\)\. Transfer requires a gain of at least\.05\.05with a paired interval above zero on both measures of both tasks in both runs, which lower attention alone does not give\.

#### Adapters\.

Low\-rank adapters\([Hu et al\., 2022](https://arxiv.org/html/2610.19113#bib.bib70)\)train on 20M synthetic and 20M original\-text tokens for five released 7–8B bases, with held\-out or all\-shown documents\. Base exact generation is\.04\.04\(RedPajama\),\.29\.29\(OLMo\-2\),\.53\.53\(Comma\),\.69\.69\(Qwen2\.5\) and\.94\.94\(Llama\-3\.1\)\. Adapters reach\.91\.91–1\.001\.00, yet none meets the joint transfer criterion on the three external tasks \(Figure[8](https://arxiv.org/html/2610.19113#S6.F8)\)\. Only all\-shown Llama\-3\.1 gains on ElementalTask \(\+\.12\+\.12\), with a SCAN loss \(−\.07\-\.07\)\. OLMo\-2 loses\.17\.17–\.22\.22on SCAN and its Pile loss rises\+\.035\+\.035nats\.

#### Attention\.

Answer\-position attention to the defining ending, relative to the closest ending, separates the model groups with area under the receiver\-operating\-characteristic curve\.814\.814\[\.59,1\.00\.59,1\.00\], below the\.90\.90threshold\. None of the three attention predictions is met, and no trajectory shows the predicted increase\. The measure is correlational\.

#### Benchmarks and label flips\.

Synthetic documents change none of ten standard benchmarks at 1\.7B\. At 360M, MMLU loses\.4\.4points\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.19113#bib.bib72)\), LAMBADA gains\.6\.6\([Paperno et al\., 2016](https://arxiv.org/html/2610.19113#bib.bib73)\), and the other eight are unchanged\. With label meanings exchanged, 1\.7B loses3\.23\.2points on SST\-2\([Socher et al\., 2013](https://arxiv.org/html/2610.19113#bib.bib76)\)and1\.51\.5on AG News\([Zhang et al\., 2015](https://arxiv.org/html/2610.19113#bib.bib65)\)\. At 360M, SST\-2 loses2\.12\.1, AG News gains1\.21\.2, and two others are unchanged\.

#### Rewrites\.

Every rewritten\-document run stops at its first scheduled evaluation\. The 1\.7B run stops at 120 steps \(250M tokens\) when bAbI\-20 falls2\.82\.8points \(Holmp=\.04p=\.04\)\([Weston et al\., 2015](https://arxiv.org/html/2610.19113#bib.bib74)\), and an extension to 360 steps stops again when List Functions falls2\.42\.4points\([Srivastava and others, 2023](https://arxiv.org/html/2610.19113#bib.bib90)\)\. Neither shows a full\-window gain over original\-document training\.

#### Exercises\.

Figure[17](https://arxiv.org/html/2610.19113#A14.F17)shows exercise fine\-tunes and a 15% mixture against original\-text controls\. At 1\.7B, made\-up labels on unseen datasets gain17\.617\.6–20\.420\.4points on TREC\-6,17\.217\.2–19\.019\.0on DBpedia\-7, and3\.73\.7–6\.36\.3on Yahoo\-5\([Li and Roth, 2002](https://arxiv.org/html/2610.19113#bib.bib64);[Zhang et al\., 2015](https://arxiv.org/html/2610.19113#bib.bib65)\), and exchanged labels gain3\.63\.6–13\.613\.6\. At 3B, TREC\-6 and Yahoo\-5 gain14\.414\.4and6\.16\.1points and no qualified unseen\-label evaluation loses\. Seen formats show gains \(SST\-2 exchanged labels\+23\.4\+23\.4, Subj\+17\.4\+17\.4in the mixture\) and losses \(bAbI\-1−5\.3\-5\.3, CLUTRR−2\.7\-2\.7\)\([Weston et al\., 2015](https://arxiv.org/html/2610.19113#bib.bib74);[Sinha et al\., 2019](https://arxiv.org/html/2610.19113#bib.bib75)\), as do farther tasks \(List Functions\+3\.3\+3\.3–\+3\.6\+3\.6at 1\.7B, bAbI\-20−3\.6\-3\.6at 1\.7B and−6\.3\-6\.3at 3B\)\([Srivastava and others, 2023](https://arxiv.org/html/2610.19113#bib.bib90)\)\. The morphology diagnostic is unchanged\. The exercises practice prompt\-defined labels\([Wei et al\., 2023a](https://arxiv.org/html/2610.19113#bib.bib63)\)\(Table[8](https://arxiv.org/html/2610.19113#A14.T8)\)\. Mined and filtered passages are in Appendix[M](https://arxiv.org/html/2610.19113#A13)\.

## Appendix ORelease

We release the code, per\-item results for every table and figure, the analysis plans, and the full appendix as a supplement PDF \(the ‘suppl\.’ tables cited here\)\. Fine\-tuned weights are regenerable from pinned base revisions and seeds and are not released\. Experiments used about 320 GPU\-hours on single H100 80GB GPUs\.

## Appendix PChanges to the analysis plans

Three choices made after the plans were fixed change a reported value or verdict\. The blocking results are re\-scored \(Appendix[D](https://arxiv.org/html/2610.19113#A4)\)\. A corrected Quechua scoring line moves the synthetic\-document comparison from fail to pass \(Appendix[M](https://arxiv.org/html/2610.19113#A13)\)\. The attention\-matched control spans are split post hoc\. No post hoc choice or descriptive result turns an unmet criterion into a met one\. The supplement lists every departure\.