In-Context Binding Capacity in Language Models

# In-Context Binding Capacity in Language Models
Source: [https://arxiv.org/html/2609.30634](https://arxiv.org/html/2609.30634)
Samrath ChadhaAffiliation:Efficient Computation Inc\.Email:[samrath@perseus\.so](mailto:)

###### Abstract

How many assignments can a language model recall before it loses track of which value belongs to which entity? We measure this limit using continuous recall curves for 12 models at or below 3B parameters and a threshold sweep over 30 open models up to 12B\. On the continuous curves, the load at which recall falls halfway to chance followsK50=c​NαK\_\{50\}=cN^\{\\alpha\}, withα=0\.820\\alpha=0\.820andR2=0\.73R^\{2\}=0\.73\. The broader sweep shows an8×8\\timesrange associated with pretraining recipe, although the continuous curves show no detectable recipe effect after controlling for scale, with few modern models in the fit\. We derive why interference can lower measured capacity by reducing single\-binding recall even when the load\-dependent recall profile is unchanged\. Direct task training also exceeds the extrapolated zero\-shot law, but different measurement criteria prevent interpreting that comparison as a capacity gain\. Its formation times follow a power\-law form in two independent codebases, conditional on runs that succeed\. Together, these results characterize capacity at the model’s query interface\. Bounds on joint recall and a decomposition of policy errors connect this measurement to working memory and instruction following, without treating recall as a measure of alignment\. The controlled task also provides a baseline for testing whether binding limits constrain world\-state tracking; the present experiments do not measure state updates or downstream transfer\.

## 1Introduction

A language model that reads a configuration file must remember which value was assigned to which variable, and remember it for every variable at once\. This is an example of in\-context*binding*\. Like human working memory, it has a capacity: past some number of simultaneous bindings, recall collapses\.

The same problem arises when an assistant tracks a user’s constraints across turns, a program’s variables across execution, or objects as actions change their state\. These tasks also require updating facts and using the current state to answer or act\. Our task isolates the static retrieval step: it asks whether the model returns the value assigned to the queried entity\. Measuring that step can make later tests of state tracking more precise, but it does not establish that a model maintains a world model\.

Parameter count is a natural capacity predictor\. On continuous recall\-versus\-KKcurves at≤\\leq3B, capacity scales with parameters asK50=c​NαK\_\{50\}=c\\,N^\{\\alpha\}, withα=0\.820\\alpha=0\.820\. Yet on the instrument that spans the largest models, the Pythia ladder holdsk∗∈\[3,6\]k^\{\\ast\}\\in\[3,6\]from 410M to 12B, a29×29\\timesincrease in parameters, while the 3B Qwen2\.5 reaches the measurement ceiling\. This contrast is associated with recipe\. Both instruments measure capacity, but over different model ranges and at different resolution \(Section[2](https://arxiv.org/html/2609.30634#S2)\)\.

Binding capacity separates into what pretraining delivers for zero\-shot use, how that use withstands interference, and what direct task training can build\. Our capacity fit describes a zero\-shot operating regime, whereas the scaling laws of[Kaplan et al\. \(2020\)](https://arxiv.org/html/2609.30634#bib.bib11)and[Hoffmann et al\. \(2022\)](https://arxiv.org/html/2609.30634#bib.bib8)describe language\-model loss\. Direct task training provides a separate measure of learnability\. Section[5](https://arxiv.org/html/2609.30634#S5)reports a factor of2626relative to an extrapolation of the zero\-shot law, with different estimands and a prediction outside the task’s load domain\. The independent re\-measurement gives a post\-hoc zero\-shot scaling law on models at or below 3B, and the broad sweep shows recipe\-associated differences across model ladders\. The geometric comparisons do not establish a mechanism: recipe contrasts and trial\-level error prediction answer different questions \(Section[7](https://arxiv.org/html/2609.30634#S7)\)\.

The contribution is a measured capacity law together with conditions for interpreting it\. The threshold analysis shows when an exponent transfers to another recall requirement and when model order can change \(Section[3\.5](https://arxiv.org/html/2609.30634#S3.SS5)\)\. The interference identity separates changes in single\-binding reliability from changes in the remaining load profile \(Section[6\.1](https://arxiv.org/html/2609.30634#S6.SS1)\)\. These results motivate comparisons at a stated absolute recall requirement \(Section[8](https://arxiv.org/html/2609.30634#S8)\), while the geometric and formation analyses delimit the mechanistic and training claims that the measurements support\. Section[9](https://arxiv.org/html/2609.30634#S9)relates these findings to working\-memory tasks, sets out a test of world\-state tracking, and derives the additional assumptions needed to connect recall reliability to policy adherence\.

## 2Measuring capacity

#### Task\.

Each trial presentsKKbindings “eie\_\{i\}:oio\_\{i\}” \(entityeie\_\{i\}, single\-token obligationoio\_\{i\}, from disjoint pools\), then an interference block ofDDdistractor tokens, then the query “The task foreje\_\{j\}is:” for a randomjj\. The model is correct iff its highest\-logit obligation token isojo\_\{j\}\. VaryingKKprobes*capacity*; varyingDDprobes*interference\-robustness*\. Chance over the obligation pool is1/\|𝒪\|=0\.0401/\|\\mathcal\{O\}\|=0\.040\. When theKKobligations are distinct, choosing uniformly among those present answers the query correctly with probability1/K1/K\. BecauseKKvaries along the capacity axis, the capacity statistics below are computed against1/\|𝒪\|1/\|\\mathcal\{O\}\|\.

e1:o1e\_\{1\}\{:\}\\,o\_\{1\}e2:o2e\_\{2\}\{:\}\\,o\_\{2\}⋯\\cdotseK:oKe\_\{K\}\{:\}\\,o\_\{K\}DDdistractor tokens“task foreje\_\{j\}is:”capacity axis: varyKKrobustness axis: varyDDcorrect iff the toplogit isojo\_\{j\}KKrecallmidpointk∗,K50k^\{\\ast\},\\,K\_\{50\}Figure 1:The task and its two axes\.Each trial presentsKKentity–obligation bindings, aDD\-token interference block, and a recall query; the model is correct iff its highest\-logit obligation token matches the queried binding\. VaryingKKprobes capacity, varyingDDprobes interference\-robustness\. Right: both capacity statistics are midpoint crossings of the recall curve between the model’s ownK=1K\{=\}1ceiling and chance:k∗k^\{\\ast\}on the measurement grid andK50K\_\{50\}under a logistic fit to the continuous curves \(Section[2](https://arxiv.org/html/2609.30634#S2)\)\.
#### The capacity statistics, and their pitfall\.

k∗k^\{\\ast\}is the smallestKKat which recall falls below the midpoint between theK=1K=1ceiling and chance\. It is a threshold crossing of a smooth curve, and thresholding can manufacture discontinuities where the underlying quantity varies continuously\([Schaeffer et al\., 2023](https://arxiv.org/html/2609.30634#bib.bib20)\)\. We therefore treatk∗k^\{\\ast\}as a*summary statistic*and not as evidence of a sharp transition\. The released table containsk∗k^\{\\ast\}and the geometry but not the full recall\-versus\-KKcurves, so we cannot rule out the thresholding artifact from it directly\. Instead, we exclude any model whoseK=1K=1ceiling is near chance \(its threshold is noise\), report the median difference rather than the range, and cluster on families\. Section[3](https://arxiv.org/html/2609.30634#S3)releases the curves for twelve of the models\.K50K\_\{50\}is the continuous analogue ofk∗k^\{\\ast\}on those released curves: the load at which retained performancer⁡\(K\)r\(K\)\(recall normalised between the model’s ownK=1K\{=\}1ceiling and chance\) crosses one half under a logistic fit inlog⁡K\\log K\. We report right\-censoring rather than clipping it, and validate the estimator on synthetic curves with known answers before touching data\.

#### Broad\-sweep exclusions\.

*\(i\)*k∗k^\{\\ast\}is defined as a threshold crossing relative to a model’s ownK=1K=1ceiling\. If that ceiling is itself at chance, the threshold is noise\. One model, Pythia\-70M, has aK=1K=1accuracy of0\.0700\.070against a chance of0\.0400\.040: it cannot hold a*single*binding\. This model is excluded, leavingn=29n=29, as it is the source of the12×12\\timesrange in the raw table; without it, the range is33to2424, i\.e\.8×8\\times\.

*\(ii\)*Three models sit atk∗=24k^\{\\ast\}=24, the ceiling imposed by the entity pool, and are*right\-censored*: OLMo\-1\-7B, Qwen2\.5\-3B, and, informatively, the old\-recipe OPT\-6\.7B\. The low end of the range is measured\.

#### Models\.

The released capacity table covers 30 open models from ten families\.*Old*\(pre\-2023\)*recipe*\(n=18n=18\): GPT\-2 \(M/L/XL\)\([Radford et al\., 2019](https://arxiv.org/html/2609.30634#bib.bib19)\), GPT\-Neo \(125M, 1\.3B\), OPT \(350M, 1\.3B, 2\.7B, 6\.7B\), BLOOM \(560M, 1\.7B\), Cerebras\-1\.3B, and the Pythia ladder \(70M–12B\)\([Biderman et al\., 2023](https://arxiv.org/html/2609.30634#bib.bib2)\)\.*Modern*\(n=12n=12\): OLMo\-1\-7B, OLMo\-2\-7B with its SFT/DPO/Instruct ladder, OLMoE, Qwen2\.5 \(0\.5B–7B\), Mistral\-7B, DeepSeek\-Coder\-6\.7B\. All measurements are on frozen models\. The continuous re\-measurement of Section[3](https://arxiv.org/html/2609.30634#S3)covers twelve models at or below 3B: seven from this table \(OPT\-350M/1\.3B, Pythia\-410M/1\.4B/2\.8B, Qwen2\.5\-0\.5B/1\.5B\) together with OPT\-125M, Pythia\-160M, Pythia\-1B, SmolLM2\-360M and OLMo\-2\-1B\. The uncensored fit spans four families\. The interference sweep of Section[6](https://arxiv.org/html/2609.30634#S6)covers the eight models with committedDD\-sweep curves\.

## 3A zero\-shot scaling law for capacity

### 3\.1The continuous instrument

A re\-measurement with committed code re\-runs the models on one instrument and releases the full recall\-versus\-KKcurves\. A first wave found the GPT\-2 family and Pythia\-70M below theK=1K\{=\}1exclusion floor; a second ran the twelve models above that floor on aKKgrid to 64 with six seeds per cell\. The GPT\-2 result disagrees with the original instrument, where those models hold a measurablek∗k^\{\\ast\}; we report it as an instrument difference\. Models censored at the first wave’sK=24K\{=\}24grid resolve to finite values on the full grid \(OPT\-1\.3B to38\.138\.1\)\.

### 3\.2TheK50K\_\{50\}power law

Over these curves, capacity fits a power law,K50=c​NαK\_\{50\}=c\\,N^\{\\alpha\}withα=0\.820\\alpha=0\.820\(R2=0\.73R^\{2\}=0\.73\), family\-clustered CI\[0\.674,1\.251\]\[0\.674,1\.251\], leave\-one\-family\-out range\[0\.76,1\.09\]\[0\.76,1\.09\]\(Figure[2](https://arxiv.org/html/2609.30634#S3.F2)\)\. The interval contains proportionality and excludes zero, and removing any single model movesα\\alphaby at most0\.110\.11\. The fit uses the1111of twelve models with an uncensoredK50K\_\{50\}\(OLMo\-2\-1B is right\-censored at the grid maximum\)\. The uncensored models span four families, so the clustered interval and the leave\-one\-family\-out range rest on four clusters\.

### 3\.3Recipe after controlling for scale

At≤\\leq3B parameters, the recipe split of Section[4](https://arxiv.org/html/2609.30634#S4)does not survive controlling for scale\. Regressinglog⁡K50\\log K\_\{50\}on recipe andlog\\logparameters over the re\-measured curves gives a recipe coefficient of\+0\.123\+0\.123\(family\-clustered CI\[−0\.521,0\.363\]\[\-0\.521,0\.363\], spanning zero\) and a scale coefficient of\+0\.818\+0\.818\(CI\[0\.697,1\.206\]\[0\.697,1\.206\], excluding it\)\. The regression usesn=11n=11models across four families, of which three are modern \(Qwen2\.5\-0\.5B, Qwen2\.5\-1\.5B, SmolLM2\-360M\); the null is correspondingly weak\. The threshold sweep reaches 12B, whereas this continuous re\-measurement stops at 3B; continuous curves above that would show whether the recipe span reappears\.

Figure 2:TheK50K\_\{50\}scaling law and a comparison outside its fitted range\.Circles:K50K\_\{50\}for the1111uncensored re\-measured models; the line is the fittedK50=c​N0\.820K\_\{50\}=c\\,N^\{0\.820\}\. Star: a 4\.98M\-parameter transformer trained directly on the task, at26×26\\timesthe law’s prediction for its size \(dotted riser\)\. The two estimands differ \(largest learnableKKversus zero\-shotK50K\_\{50\}\)\. The extrapolated prediction falls below one binding, so the ratio is an extrapolation discrepancy rather than a measured capacity gain\.The line in Figure[2](https://arxiv.org/html/2609.30634#S3.F2)describes how the midpoint of a zero\-shot recall curve varies with parameter count in this cohort\. It does not imply better recall at every load: two models can differ in their single\-binding ceiling or in how sharply recall falls\. Those differences matter for a fixed absolute\-recall requirement \(Section[8](https://arxiv.org/html/2609.30634#S8)\), while slope differences can change a crossing at a stricter relative criterion \(Section[3\.5](https://arxiv.org/html/2609.30634#S3.SS5)\)\. The broader threshold sweep includes larger models, but its load grid and right\-censored values cannot resolve a continuous size law in that range\. Its recipe contrast and the fitted line therefore describe different comparisons, not two estimates of one effect\.

### 3\.4Sensitivity to the summary statistic

The following checks reuse the existing corpus and are exploratory: none was predicted before inspection of the main results\. They test whether the scale association depends on one summary, without treating a different summary as a replication of the same exponent\.

#### The scale association persists in a threshold\-free summary\.

K50K\_\{50\}is a threshold crossing, and one of twelve models is right\-censored and excluded from the fit\. Both choices can be removed at once\. The area under the retained\-performance curve \(AUC\) needs no threshold and is defined for every model, censored or not\. Regressinglog\\logAUC onlog\\logparameters gives an exponent of\+0\.329\+0\.329withR2=0\.799R^\{2\}=0\.799over all 12 models, family\-clustered CI\[\+0\.286,\+0\.480\]\[\+0\.286,\+0\.480\]; restricting to the 11 uncensored models gives\+0\.304\+0\.304withR2=0\.821R^\{2\}=0\.821\. Both fits have a higher in\-sampleR2R^\{2\}than theK50K\_\{50\}law’sR2=0\.73R^\{2\}=0\.73, although the dependent variables differ, so this does not rank their predictive validity\. The exponents are not comparable in magnitude toα=0\.820\\alpha=0\.820: AUC integrates performance over a finite load range, whereasK50K\_\{50\}locates a crossing\. What carries over is the positive association with scale and its interval excluding zero, with no threshold applied and no model discarded\. AUC can nevertheless saturate when the curve moves beyond the measured range; it does not identify unobserved capacity\.

#### Single\-binding recall is usually high\.

A model with a lowK=1K\{=\}1ceiling could show a lowK50K\_\{50\}for reasons unrelated to capacity\. 11 of 12 models hold a ceiling between0\.9180\.918and0\.990\.99; the exception is Pythia\-160M at0\.5890\.589\. Ceiling is only weakly related to scale \(R2=0\.259R^\{2\}=0\.259\), but this summary alone does not identify how ceiling uncertainty affects the fitted midpoints\.

#### The fitted collapse is gradual, and its steepness covaries with capacity\.

Fitting a logistic inlog⁡K\\log Kgives a slope per model, and those slopes span0\.4990\.499to1\.8011\.801over 11 models\. A finite\-slope logistic represents a smooth transition by construction; fitting it does not independently exclude a sharper underlying change\. The released curves permit that question to be examined beyond the thresholded table\([Schaeffer et al\., 2023](https://arxiv.org/html/2609.30634#bib.bib20)\)\. We looked for a second scaling law in the slope and did not find one: slope is strongly coupled toK50K\_\{50\}\(r=−0\.837r=\-0\.837in logs\), and onceK50K\_\{50\}is known, parameters add almost nothing \(partialr=−0\.169r=\-0\.169\)\. We report the range and not a law\.

### 3\.5When a scaling exponent transfers across thresholds

A scaling law at the midpoint leaves open whether size helps equally at a stricter recall requirement\. For positive loadKK, midpointK50,iK\_\{50,i\}, and parameter countNiN\_\{i\}, let the fitted retained\-performance curve for modeliibe

r^i​\(K\)=11\+exp⁡\{βi​\(log⁡K−log⁡K50,i\)\},βi\>0\.\\widehat\{r\}\_\{i\}\(K\)=\\frac\{1\}\{1\+\\exp\\\{\\beta\_\{i\}\(\\log K\-\\log K\_\{50,i\}\)\\\}\},\\qquad\\beta\_\{i\}\>0\.\(1\)For a retained\-performance criterionq∈\(0,1\)q\\in\(0,1\), solvingr^i​\(Kq,i\)=q\\widehat\{r\}\_\{i\}\(K\_\{q,i\}\)=qgives

log⁡Kq,i=log⁡K50,i\+1βi​log⁡1−qq\.\\log K\_\{q,i\}=\\log K\_\{50,i\}\+\\frac\{1\}\{\\beta\_\{i\}\}\\log\\frac\{1\-q\}\{q\}\.\(2\)Thus a common slopeβi=β\\beta\_\{i\}=\\betaand an exact midpoint lawK50,i=c​NiαK\_\{50,i\}=cN\_\{i\}^\{\\alpha\}implyKq,i=c​\[\(1−q\)/q\]1/β​NiαK\_\{q,i\}=c\[\(1\-q\)/q\]^\{1/\\beta\}N\_\{i\}^\{\\alpha\}: changing the criterion changes the coefficient but preserves the exponent\. This is a sufficient condition, not an assumption established by our fits\.

More generally, letxi=log⁡Nix\_\{i\}=\\log N\_\{i\}and fit ordinary least squares with an intercept over the*same*uncensored models\. Taking covariance and variance over that model set, linearity of the regression slope in its response gives

α^q=α^50\+log⁡1−qq​Cov⁡\(xi,βi−1\)Var⁡\(xi\),Var⁡\(xi\)\>0\.\\widehat\{\\alpha\}\_\{q\}=\\widehat\{\\alpha\}\_\{50\}\+\\log\\frac\{1\-q\}\{q\}\\,\\frac\{\\operatorname\{Cov\}\(x\_\{i\},\\beta\_\{i\}^\{\-1\}\)\}\{\\operatorname\{Var\}\(x\_\{i\}\)\},\\qquad\\operatorname\{Var\}\(x\_\{i\}\)\>0\.\(3\)This follows by substituting Equation \([2](https://arxiv.org/html/2609.30634#S3.E2)\) intoα^q=Cov⁡\(xi,log⁡Kq,i\)/Var⁡\(xi\)\\widehat\{\\alpha\}\_\{q\}=\\operatorname\{Cov\}\(x\_\{i\},\\log K\_\{q,i\}\)/\\operatorname\{Var\}\(x\_\{i\}\)\. The midpoint exponent therefore need not describe high\-reliability recall\. Slope variation can change the fitted exponent when inverse slope covaries with size\. The correction vanishes atq=1/2q=1/2, and also vanishes at every criterion if that covariance is zero\. Changing the model set because a different threshold becomes censored adds a separate selection effect\.

These identities apply to the fitted logistic, which does not pass throughr^i​\(1\)=1\\widehat\{r\}\_\{i\}\(1\)=1\. They must not be substituted for the exactly anchored normalisation in the empirical definition ofr⁡\(K\)r\(K\)in Section[2](https://arxiv.org/html/2609.30634#S2)\. The existing slope–capacity association motivates checking transfer across criteria; it does not determine the covariance in Equation \([3](https://arxiv.org/html/2609.30634#S3.E3)\) or provide a measured correction toα\\alpha\.

#### Model order can depend on the required recall\.

For modelsiiandjj, letδi​j=log⁡K50,i−log⁡K50,j\\delta\_\{ij\}=\\log K\_\{50,i\}\-\\log K\_\{50,j\}andhi​j=βi−1−βj−1h\_\{ij\}=\\beta\_\{i\}^\{\-1\}\-\\beta\_\{j\}^\{\-1\}\. Subtracting Equation \([2](https://arxiv.org/html/2609.30634#S3.E2)\) gives

log⁡Kq,iKq,j=δi​j\+hi​j​log⁡1−qq\.\\log\\frac\{K\_\{q,i\}\}\{K\_\{q,j\}\}=\\delta\_\{ij\}\+h\_\{ij\}\\log\\frac\{1\-q\}\{q\}\.\(4\)Whenhi​j≠0h\_\{ij\}\\neq 0, the formal order changes atq∗=\[1\+exp\(−δi​j/hi​j\)\]−1q^\{\*\}=\[1\+\\exp\(\-\\delta\_\{ij\}/h\_\{ij\}\)\]^\{\-1\}\. Equal slopes preserve the order at every threshold; equal slopes and midpoints give identical fitted curves\. With unequal slopes, a reversal is relevant to the measurements only if its crossing load lies inside both measured domains\. This is a conditional consequence of the fitted family, not an observed reversal in our models\. It shows why ranking models by their midpoint need not rank them by high\-reliability retained performance\.

## 4The threshold sweep to 12B

Figure[3](https://arxiv.org/html/2609.30634#S4.F3)showsk∗k^\{\\ast\}across the 29 measurable models\. It spans33to2424\. Old recipes have median33\(n=17n=17\), compared with1616for modern recipes \(n=12n=12\)\. Clustering the 30 models into ten families, the median difference has 95% CI\[\+10,\+13\]\[\+10,\\,\+13\], and the family\-level test givesp=0\.012p=0\.012\. The model\-level Mann–Whitney test, which treats siblings as independent draws, givesp=4\.8×10−4p=4\.8\\times 10^\{\-4\}\.

### 4\.1Recipe and scale

The Pythia ladder holdsk∗=3,4,6,4,4k^\{\\ast\}=3,4,6,4,4at 410M, 1\.4B, 2\.8B, 6\.9B and 12B: flat across a29×29\\timesrange of parameters\. Within a*modern*recipe, scale does help: Qwen2\.5 goes3→10→24→163\\to 10\\to 24\\to 16at 0\.5B, 1\.5B, 3B and 7B\. These ladders show a recipe\-associated contrast between scale trends, rather than an identified causal interaction\. The Qwen ladder is*not monotone*: the 3B model reaches the censored ceiling and the 7B model sits below it\. Alignment leaves this threshold unchanged: the OLMo\-2 base/SFT/DPO/Instruct ladder holdsk∗=16k^\{\\ast\}=16at every stage\.[Kim et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib12)found no consistent alignment benefit for entity tracking; this ladder shows no change detectable by the capacity threshold\. Identical grid crossings do not establish identical recall curves\.

### 4\.2Overlap between recipes

Modern recipes do not uniformly dominate\. OLMoE, a modern model, sits atk∗=6k^\{\\ast\}=6; OPT\-1\.3B, an old one, reaches1010; and OPT\-6\.7B, also old, reaches the censored ceiling at2424\. Both classes span the full33–2424range\. The recipe effect is a difference in medians \(33versus1616\) that survives family clustering\.

Figure 3:Binding capacityk∗k^\{\\ast\}clusters by pretraining recipe\.Mediank∗k^\{\\ast\}is33for old recipes and1616for modern ones \(family\-clustered95%95\\%CI of the difference\[\+10,\+13\]\[\+10,\\,\+13\]; family\-levelp=0\.012p=0\.012\)\. The classes*overlap*: OLMoE \(modern\) sits at66and OPT\-6\.7B \(old\) reaches the censored ceiling\. Arrows mark models right\-censored at the entity\-pool ceiling \(k∗=24k^\{\\ast\}=24\)\. Pythia\-70M is excluded due to itsK=1K=1accuracy being at chance\.
### 4\.3Sensitivity to the disputed family

This exclusion check is exploratory and uses the existing threshold table\.

#### The recipe split does not rest on the disputed family\.

The continuous instrument places the GPT\-2 family below theK=1K\{=\}1exclusion floor \(Section[3](https://arxiv.org/html/2609.30634#S3)\), while the threshold sweep assigns all 3 GPT\-2 modelsk∗=3k^\{\\ast\}=3, the old\-recipe median\. Dropping them moves the old\-recipe median from33to44and the median difference from\+13\+13to\+12\+12\. The split survives the instrument disagreement\.

## 5The boundary: direct task training

### 5\.1Direct task training

The task\-trained comparison tests the scope of the power law\. A 4\.98M\-parameter transformer trained directly on the task holds an interpolated3\.73\.7bindings where the law predicts0\.140\.14, a factor of2626, read over 400 from\-scratch runs in that cell from a clean corpus of 423\. The estimands differ: one interpolates the fraction of training runs that reach a formation criterion, while the other measures a midpoint of zero\-shot recall\. The denominator is also extrapolated below the measured model\-size range and below one binding\. Since normalisation setsr⁡\(1\)=1r\(1\)=1, a continuous decreasing recall curve can reach its midpoint only atK\>1K\>1\. Thus a power law withc\>0c\>0andα\>0\\alpha\>0cannot extend to arbitrarily small models:c​Nα<1cN^\{\\alpha\}<1leaves the domain of the capacity definition\. The factor above is an extrapolation discrepancy, not a measured training gain or a bound on unused architectural capacity\. The task\-trained result establishes learnability in its own setting; it does not calibrate the zero\-shot law outside its observed range\.

### 5\.2The formation\-time law

Direct task training has its own price inKK\. For small transformers \(1\.3M–5M parameters\) trained from scratch on the same task, median time to first formation grows as a power ofKK\. One implementation givest≈766​K2\.58t\\approx 766\\,K^\{2\.58\}\(R2=0\.95R^\{2\}=0\.95\), and the independent second fit ist≈667​K2\.12t\\approx 667\\,K^\{2\.12\}\(R2=0\.97R^\{2\}=0\.97\)\. We fixed the first fit before examining the second corpus\. The functional form replicates, but the exponent does not resolve\. The second corpus’sK=8K\{=\}8cell holds two runs, and removing it moves the exponent to 1\.7\. Both fits condition on formation within the training budgets; they do not estimate unconditional formation times\. Seed matters far beyond chance: across 24 seeds the variance of emergence rate is9\.05×9\.05\\timesthe binomial expectation, so per\-seed comparisons must be paired\. Figure[4](https://arxiv.org/html/2609.30634#S5.F4)shows both fits; their conditional medians imply a factor of 4–6 in training steps when load doubles\. Conditioning on success can alter the fitted exponent in either direction \(Section[5\.3](https://arxiv.org/html/2609.30634#S5.SS3)\)\.

Figure 4:The cost\-of\-load law, in two codebases\.Median steps to formation against binding load, both axes logarithmic, for two independent implementations; faint lines are the committed power\-law fits \(t≈766​K2\.58t\\approx 766\\,K^\{2\.58\}andt≈667​K2\.12t\\approx 667\\,K^\{2\.12\}\)\. The second corpus was analysed after the first fit was fixed\. The medians exclude runs that did not form within budget\. They describe successful runs, not an unconditional formation\-time law; censoring does not provide a lower bound on the exponent\.The two scaling plots also have different response variables\. Figure[2](https://arxiv.org/html/2609.30634#S3.F2)relates parameter count to zero\-shot recall at inference, whereas Figure[4](https://arxiv.org/html/2609.30634#S5.F4)relates binding load to training steps among runs that formed\. Neither plot maps an additional training step to a change in a pretrained model’sK50K\_\{50\}\. Combining their fitted exponents would therefore not estimate the training cost of raising zero\-shot capacity\.

### 5\.3What conditioning on formation changes

LetTKT\_\{K\}be the random time to first formation at loadKK, allowingTK=∞T\_\{K\}=\\inftyfor runs that never form\. WriteFK​\(t\)=Pr⁡\(TK≤t\)F\_\{K\}\(t\)=\\Pr\(T\_\{K\}\\leq t\)\. With a common observation budgetτ\\tauwithin a load cell andFK​\(τ\)\>0F\_\{K\}\(\\tau\)\>0, the population distribution among completed runs satisfies

Pr⁡\(TK≤t∣TK≤τ\)=FK​\(t\)FK​\(τ\),t≤τ\.\\Pr\(T\_\{K\}\\leq t\\mid T\_\{K\}\\leq\\tau\)=\\frac\{F\_\{K\}\(t\)\}\{F\_\{K\}\(\\tau\)\},\\qquad t\\leq\\tau\.\(5\)DefineFK−1​\(v\)=inf\{t:FK​\(t\)≥v\}F\_\{K\}^\{\-1\}\(v\)=\\inf\\\{t:F\_\{K\}\(t\)\\geq v\\\}\. The lower median among completed runs is consequentlymcompleted​\(K\)=FK−1​\(FK​\(τ\)/2\)m\_\{\\mathrm\{completed\}\}\(K\)=F\_\{K\}^\{\-1\}\(F\_\{K\}\(\\tau\)/2\), which is no larger than the unconditional medianmfull​\(K\)=FK−1​\(1/2\)m\_\{\\mathrm\{full\}\}\(K\)=F\_\{K\}^\{\-1\}\(1/2\)\. IfFK​\(τ\)<1/2F\_\{K\}\(\\tau\)<1/2, the latter lies beyond the budget and may be infinite\. These are population statements under the stated budget assumption, not confidence bounds for the reported sample medians\.

Even when both medians are positive and finite, the ordering does not bound their scaling exponents\. Writemcompleted​\(K\)=w⁡\(K\)​mfull​\(K\)m\_\{\\mathrm\{completed\}\}\(K\)=w\(K\)m\_\{\\mathrm\{full\}\}\(K\)with0<w⁡\(K\)≤10<w\(K\)\\leq 1\. For differentiable curves their log–log slopes differ byd​log⁡w/d​log⁡Kd\\log w/d\\log K, whose sign is unrestricted byw≤1w\\leq 1\. The two implementations therefore replicate a power\-law description among successful runs\. Establishing an unconditional formation\-time law would also require the completion probabilities and censoring budgets, including the runs that did not form\.

## 6Interference costs three separate things

Raw capacity and interference\-robustness are distinct axes, and the distinction is sharper than a single retained\-capacity number suggests\. A distractor block can damage a recall curve in three ways: it can lower the ceiling the curve starts from, move the load at which the curve crosses its midpoint, and change how steeply the curve falls\. On our curves the three behave differently\.

#### The ceiling falls, and its association with capacity is inconclusive\.

Interference damages recall atK=1K=1, before any capacity question arises\. Across the 12 continuous\-curve models the256256\-token block costs a median0\.1370\.137of theK=1K\{=\}1ceiling, from0\.0160\.016\(Qwen2\.5\-1\.5B\) to0\.5380\.538\(Pythia\-410M\)\. The association with capacity is inconclusive \(Spearmanρ=−0\.473\\rho=\-0\.473,p=0\.142p=0\.142\)\. These curves do not establish either protection from greater capacity or independence between capacity and ceiling loss\.

#### The midpoint shifts, and the size of the shift depends on the normalisation\.

Measured against the uninterfered ceiling, the median retained fraction is0\.5350\.535, with a reduction in1010of1111models: interference roughly halves effective capacity\. Measured against each curve’s own ceiling, the median retained fraction is0\.9630\.963with a reduction in55of99models, so the median is close to unchanged\. We report both, because the disagreement is itself the finding rather than a choice between estimators\. The two differ by exactly the ceiling term above: own\-ceiling normalisation divides each curve by a ceiling that interference has already lowered, which removes the damage from the measurement\. Pythia\-410M is the clearest case, losing0\.5380\.538of its single\-binding ceiling before the capacity axis is consulted at all\. We therefore take the uninterfered ceiling as the reference, and report the own\-ceiling figure so that the choice is visible\. The fixed\-reference association between capacity and retained capacity is also inconclusive \(Spearmanρ=\+0\.20\\rho=\+0\.20,p=0\.56p=0\.56,n=11n=11\)\.

#### The slopes show no consistent direction of change\.

The logistic slope atD=256D=256divided by the slope atD=0D=0has median1\.0081\.008over the 9 models with an uncensored fit at both loads, ranging0\.850\.85to1\.381\.38, with 5 of 9 steeper\. These summaries do not establish equivalence of curve shape\.

#### TheDD\-sweep at fixedKK\.

We sweepDDatK=8K=8for the eight models with committed curves \(Figure[5](https://arxiv.org/html/2609.30634#S6.F5)\)\. AtD=0D=0, recall spans0\.270\.27to0\.960\.96with median0\.540\.54\. The 4 models withk∗k^\{\\ast\}below88start at0\.270\.27–0\.380\.38, while the four models withk∗≥16k^\{\\ast\}\\geq 16start at0\.700\.70–0\.960\.96\. The 0\.5B entry is the Instruct variant of a base model withk∗=3k^\{\\ast\}=3\. ByD=256D=256every model sits at0\.100\.10–0\.310\.31\(median0\.200\.20\), and the collapse*rates*do not follow the capacity ranking\. OurDD\-sweep parallels the interference\-induced retrieval decline of[Wang and Sun \(2025\)](https://arxiv.org/html/2609.30634#bib.bib24), under a different manipulation: they overwrite earlier values for the same keys, whereas our block is unrelated filler\.

Figure 5:Capacity≠\\neqrobustness\.Recall vs\. interference loadDDat fixedK=8K=8for the eight models with committed curves \(teal: modern recipe; clay: pre\-2023\)\. Dashed curves mark the 4 models whosek∗k^\{\\ast\}is belowK=8K=8; they start at0\.270\.27–0\.380\.38before any distractor arrives\. Collapse rates do not follow the capacity ranking, so robustness is a second axis, measured separately\.
### 6\.1What the two normalisations identify

The disagreement between normalisations has an exact interpretation before curve fitting\. WritepD​\(K\)p\_\{D\}\(K\)for recall at loadKKand interferenceDD,bbfor the common chance level, andCD=pD​\(1\)C\_\{D\}=p\_\{D\}\(1\)for single\-binding recall\. SupposeC0\>bC\_\{0\}\>bandCD\>bC\_\{D\}\>b\. Define

rD​\(K\)=pD​\(K\)−bCD−b,gD=CD−bC0−b,r~D​\(K\)=pD​\(K\)−bC0−b=gD​rD​\(K\)\.r\_\{D\}\(K\)=\\frac\{p\_\{D\}\(K\)\-b\}\{C\_\{D\}\-b\},\\qquad g\_\{D\}=\\frac\{C\_\{D\}\-b\}\{C\_\{0\}\-b\},\\qquad\\widetilde\{r\}\_\{D\}\(K\)=\\frac\{p\_\{D\}\(K\)\-b\}\{C\_\{0\}\-b\}=g\_\{D\}\\,r\_\{D\}\(K\)\.\(6\)HeregDg\_\{D\}measures retained single\-binding performance above chance, whilerDr\_\{D\}measures the remaining load dependence relative to that performance\. The last equality follows by cancellingCD−bC\_\{D\}\-b\. It is an identity for normalised recall, not for the ratio of fitted capacity estimates\.

#### Midpoint consequence\.

Treat load as continuous for the crossing definition\. SupposerDr\_\{D\}is continuous and strictly decreasing onK≥1K\\geq 1, and both crossings exist\. For1/2<gD≤11/2<g\_\{D\}\\leq 1, their locations satisfy

K50own​\(D\)=rD−1​\(1/2\),K50fixed​\(D\)=rD−1​\(12​gD\)≤K50own​\(D\)\.K\_\{50\}^\{\\mathrm\{own\}\}\(D\)=r\_\{D\}^\{\-1\}\(1/2\),\\qquad K\_\{50\}^\{\\mathrm\{fixed\}\}\(D\)=r\_\{D\}^\{\-1\}\\\!\\left\(\\frac\{1\}\{2g\_\{D\}\}\\right\)\\leq K\_\{50\}^\{\\mathrm\{own\}\}\(D\)\.\(7\)The inequality follows because the inverse of a decreasing function is decreasing\. IfgD<1/2g\_\{D\}<1/2, the fixed reference criterion already fails atK=1K=1; equality holds there whengD=1/2g\_\{D\}=1/2\. These are equality crossings of a continuous curve, distinct from the first strictly\-below grid crossing used fork∗k^\{\\ast\}\.

This gives a concrete alternative to interpreting every capacity reduction as a loss of binding storage\. If interference lowersgDg\_\{D\}within\(1/2,1\)\(1/2,1\)while leavingrD=r0r\_\{D\}=r\_\{0\}, the own\-ceiling midpoint stays fixed but the fixed\-ceiling midpoint falls\. The measurements above are compatible with this possibility; they do not establish that it is the sole cause\. Conversely, matching own\-ceiling midpoints alone does not establishrD=r0r\_\{D\}=r\_\{0\}\. The full profiles must agree\. Reporting the ceiling, midpoint, and slope therefore separates observable changes without assigning them to distinct neural mechanisms\. Section[3\.5](https://arxiv.org/html/2609.30634#S3.SS5)gives the limits on threshold transfer; Section[7\.1](https://arxiv.org/html/2609.30634#S7.SS1)treats the mechanistic interpretation\.

## 7What geometry can explain

The recall curves establish a behavioral limit at the query interface\. Explaining that limit requires a different kind of evidence\. We examine the existing geometric comparisons and the trial\-level error analysis separately, because a cross\-model association and an explanation of individual retrieval failures are different claims\.

The released geometric quantities are an effective dimensionddecld\_\{\\mathrm\{decl\}\}\(a participation ratio\) and the mean off\-diagonal squared cosine between entity vectors\. Their provenance limits the comparison: the plotting code reads a committed table, but the measurement script was not preserved, so we cannot independently verify the token positions used for that table\. Appendix[A](https://arxiv.org/html/2609.30634#A1)reports the position and activation\-scale artifacts in the available measurements\. We retain this limitation beside the results; these comparisons do not identify binding storage\.

### 7\.1Recall alone does not identify storage failure

Even a complete recall curve need not identify where retrieval fails\. Consider a latent model in which the queried value is retained with probabilitys⁡\(K,D\)s\(K,D\)and is accessed correctly, conditional on retention, with probabilitya⁡\(K,D\)a\(K,D\)\. If either stage fails, the system guesses uniformly from the obligation pool\. Its recall is

pD​\(K\)=b\+\(1−b\)​s​\(K,D\)​a​\(K,D\),0≤b<1\.p\_\{D\}\(K\)=b\+\(1\-b\)s\(K,D\)a\(K,D\),\\qquad 0\\leq b<1\.\(8\)For any curve withb≤pD​\(K\)≤1b\\leq p\_\{D\}\(K\)\\leq 1, setu⁡\(K,D\)=\[pD​\(K\)−b\]/\(1−b\)u\(K,D\)=\[p\_\{D\}\(K\)\-b\]/\(1\-b\)\. Both\(s,a\)=\(u,1\)\(s,a\)=\(u,1\)and\(s,a\)=\(1,u\)\(s,a\)=\(1,u\)generate exactly that curve\. WhereverpD​\(K\)<1p\_\{D\}\(K\)<1, these describe different allocations of failure between retention and access\. Hence recall does not identify those factors without additional constraints\. The construction does not assume that transformers use these two stages; it demonstrates ambiguity in interpreting behaviour alone\. Activation interventions can provide additional evidence, as in[Feng and Steinhardt \(2024\)](https://arxiv.org/html/2609.30634#bib.bib4);[Gur\-Arieh et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib6)\. The present cross\-model geometry and recall curves do not supply that identification\.

### 7\.2Recipe contrasts and capacity prediction are different tests

Capacity has two geometric correlates at the model level \(Figure[6](https://arxiv.org/html/2609.30634#S7.F6)\)\. Across the 29 models,k∗k^\{\\ast\}rises withddecld\_\{\\mathrm\{decl\}\}\(Pearsonr=\+0\.64r=\+0\.64, Spearmanρ=\+0\.77\\rho=\+0\.77; partial\+0\.54\+0\.54controlling for interference\) and falls with pairwise interference \(Pearsonr=−0\.52r=\-0\.52; partial−0\.34\-0\.34\)\. Modern recipes show a higher packing efficiencyk∗/ddeclk^\{\\ast\}/d\_\{\\mathrm\{decl\}\}\(median0\.4620\.462vs\.0\.1340\.134\) at lower interference \(0\.1830\.183vs\.0\.3880\.388\), and every one of these comparisons is significant at the model level\. A simple account is that older recipes place entity codes on more overlapping directions, while modern recipes spread them\. This would connect binding capacity to the superposition regime of[Liu et al\. \(2025\)](https://arxiv.org/html/2609.30634#bib.bib15), where loss scales inversely with model dimension under strong superposition and weight decay controls superposition in their toy model\. It would also sit alongside[Huang et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib10), who attribute larger models’ advantage to reduced gradient interference on rare tasks\.

#### Thirty models are not thirty independent samples\.

They are*ten families*, and Pythia and OLMo contribute six apiece\. A model\-level test treats siblings as independent draws\. Under a bootstrap clustered on families, and under a family\-level test, the recipe contrasts have different uncertainty \(Table[1](https://arxiv.org/html/2609.30634#S7.T1)\)\.

Table 1:Family\-clustered comparisons\. Medians by recipe with the family\-clustered bootstrap interval and the family\-levelpp\-value for each quantity\. Bold highlights capacity and its derived ratio\. These are recipe contrasts, not tests of within\-family capacity prediction\.The clustered interval forddecld\_\{\\mathrm\{decl\}\}includes zero and its family\-level test is not significant\. For interference, the interval includes zero while the family\-level test gives a different summary of uncertainty\. These procedures test differences between recipes; neither estimates whether geometry predicts capacity within a family\. A null recipe contrast would not exclude such a relation\. Packing efficiency containsk∗k^\{\\ast\}in its numerator and therefore cannot serve as an independent predictor of that same outcome\.

The capacity contrast remains the result of Section[4](https://arxiv.org/html/2609.30634#S4)\. Establishing a geometric account would require family\-adjusted prediction using geometry measured independently of the capacity target\. The present table does not supply that test\. We report the model\-level patterns alongside the clustered recipe comparisons rather than infer a mechanism from either\. With ten families \(six old, four modern\), family\-level inference also has fewer independent units than the model count suggests\.

Figure 6:How a model spends directions on the entities it must keep distinct, at the model level\.\(a\)Packing efficiencyk∗/ddeclk^\{\\ast\}/d\_\{\\mathrm\{decl\}\}of the declaration\-site*entity*subspace and\(b\)pairwise interference separate the recipes at the model level \(each point is a model; black dash: median\)\.\(c\)k∗k^\{\\ast\}againstddecld\_\{\\mathrm\{decl\}\}over the 29 measurable models, colored by interference\. Both group comparisons are significant at the model level, while the family\-clustered recipe comparisons are reported in Table[1](https://arxiv.org/html/2609.30634#S7.T1)\. These panels do not establish within\-family prediction or a retrieval mechanism\.

### 7\.3Cross\-model capacity and within\-model retrieval separate

Capacity across models and retrieval within a model sit at different explanatory levels, and the geometry above speaks only to the first\. The packing story could be read as a competitor to the positional\-mechanism account of[Gur\-Arieh et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib6), who attribute multi\-binding retrieval failure to a positional circuit that grows noisy as bindings accumulate\. We ran the comparison on our own trials, regressing per\-trial error on serial position and on the pairwise overlap of the declaration\-site entity vectors\.The geometry loses\.Once position is controlled, subspace overlap adds between−0\.0005\-0\.0005and\+0\.0048\+0\.0048AUROC on all seven models tested; position alone reaches0\.5930\.593–0\.8430\.843\.

Neither our geometry nor a bare positional pointer predicts*which*distractor is emitted in our setting: the emitted obligation is the maximum\-overlap competitor on0\.120\.12–0\.190\.19of failures against a0\.200\.20chance rate, and a positional neighbour at or below chance\. We note that[Gur\-Arieh et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib6)do predict the emitted token, with a causal model that combines the positional mechanism with a*lexical*and a*reflexive*one and reaches95%95\\%agreement with next\-token distributions; our head\-to\-head pits only their positional term against our overlap term, and it is the positional term alone that wins\.

The ratiok∗/ddeclk^\{\\ast\}/d\_\{\\mathrm\{decl\}\}is a cross\-model description that includes the capacity target\. The trial\-level comparison asks a different question: whether overlap improves prediction beyond position\. Its reported increments do not establish a useful improvement\.

One caveat on the comparison itself: our error curve is a primacy curve rather than the U\-shape of[Gur\-Arieh et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib6), and includes a256256\-token interference block between the declarations and query\. The block is a possible explanation for the missing recency effect, but this comparison does not isolate its cause\. The two settings are not directly comparable\.

### 7\.4An earlier entity state can still serve as an address

An entity token before its assigned value cannot directly represent a change confined to that later value\. This constraint concerns the information available at a site, not every possible role the site can play in a distributed computation\.

This causal\-order argument has a narrower implication than excluding the site from binding\. Under deterministic causal attention, the state at tokeniiis a function of the tokenised prefixx≤ix\_\{\\leq i\}\. Replacing the obligation in the suffix while keeping that prefix, positions, and mask fixed leaves the entity state unchanged\. It cannot directly encode the newly supplied value, but it can encode an address that a later state associates with that value\. Such a distributed role is consistent with the shared binding IDs of[Feng and Steinhardt \(2024\)](https://arxiv.org/html/2609.30634#bib.bib4)\. Failure to decode the obligation from the entity state therefore does not rule out a causal role in retrieval\. It rules out using that decoder as evidence that the assigned value is already represented there\.

## 8Using capacity comparisons

A model selected for a binding task must meet a recall requirement at the relevant interference load\. An own\-ceiling midpoint instead describes how much performance remains relative to that model’s single\-binding baseline\. Section[3\.5](https://arxiv.org/html/2609.30634#S3.SS5)shows that even rankings under relative criteria need not transfer across thresholds\. Absolute recall imposes another condition\.

### 8\.1Absolute recall defines the feasible loads

Letπ∈\(b,1\)\\pi\\in\(b,1\)be a required recall probability and assumeCD\>bC\_\{D\}\>b\. The empirical normalisation in Equation \([6](https://arxiv.org/html/2609.30634#S6.E6)\) gives the exact equivalence

pD\(K\)≥π⟺rD\(K\)≥π−bCD−b\.p\_\{D\}\(K\)\\geq\\pi\\quad\\Longleftrightarrow\\quad r\_\{D\}\(K\)\\geq\\frac\{\\pi\-b\}\{C\_\{D\}\-b\}\.\(9\)Thus a common absolute requirement generally corresponds to a different retained\-performance criterion for each model and interference condition\. If recall is nonincreasing forK≥1K\\geq 1andCD<πC\_\{D\}<\\pi, no load meets the requirement\. IfCD=πC\_\{D\}=\\pi, load one meets it, as can a plateau; ifCD\>πC\_\{D\}\>\\pi, the criterion lies between zero and one, but its crossing may remain outside the measured range\. These statements concern the underlying curves\. Finite\-sample estimates require uncertainty on both the ceiling and the recall probabilities\.

For a measured set of load–interference pairs𝒢\\mathcal\{G\}, report the feasible set\{\(K,D\)∈𝒢:pD​\(K\)≥π\}\\\{\(K,D\)\\in\\mathcal\{G\}:p\_\{D\}\(K\)\\geq\\pi\\\}with its uncertainty\. This preserves the task requirement without converting unmeasured loads into fitted capacity claims\. It complements the midpoint, which remains useful for comparing the shape of the decline relative to baseline performance\.

### 8\.2Analyses that distinguish the remaining explanations

The equations motivate tests that can reuse saved trial outputs\. Threshold transfer should compare crossings at fixed recall criteria on the same model set, then report the additional selection caused by censoring\. Direct empirical crossings provide a check on logistic extrapolation\. A reversal supported within the measured range would reject a threshold\-independent model ranking; a reversal outside that range would remain a model prediction\.

The interference comparison admits a more specific test than a retained\-capacity ratio\. Under a ceiling\-only account, the own\-ceiling profile is unchanged, so the interfered recall predicted from the baseline profile isb\+\(CD−b\)​r0​\(K\)b\+\(C\_\{D\}\-b\)r\_\{0\}\(K\)\. Systematic residuals across load would reject this account\. Ceiling estimation and profile evaluation should use separate trials or propagate their shared uncertainty\. Agreement within a predeclared tolerance would support this restricted account on the measured grid, without identifying a neural mechanism\.

The geometric comparison should ask whether independently measured geometry improves prediction beyond model size and family, with validation that holds out families\. The trial\-level analogue compares position alone with position plus overlap on held\-out contexts\. A positive result at one level would not establish the other\. Formation\-time analyses should retain unsuccessful runs and their observation budgets, reporting completion probability alongside time among completed runs\. These are proposed analyses, not additional results of this paper\.

## 9Working memory, interpretability, and alignment

Binding capacity matters when correct behavior depends on retaining which fact or rule belongs to which entity\. The connection to working memory is operational: our task measures temporary access to relations supplied in context\. The connection to alignment requires another step, from recovering a relation to using it in an action\. The following deductions make that step explicit; they do not report an alignment evaluation\.

### 9\.1Working memory is a task\-dependent ability

An input window specifies how much text can be supplied\. Our capacity statistic specifies how many bindings can be recalled at a chosen criterion\. The two quantities answer different questions\.[Liu et al\. \(2023\)](https://arxiv.org/html/2609.30634#bib.bib16)show that the position of relevant information affects long\-context retrieval, while[Wang and Sun \(2025\)](https://arxiv.org/html/2609.30634#bib.bib24)study interference from overwritten values\. Our unrelated filler manipulation does not measure the same form of updating\. Working memory therefore provides a useful family of tasks, rather than a single capacity shared by these measurements\.

IncreasingKKin our task also changes the declaration sequence\. The present size law does not isolate binding count from every change in token length or query distance\. A matched\-length comparison would vary the number of distinct relations while holding total tokens and the queried entity’s position fixed\. A separate manipulation would change whether a value is retained, overwritten, or composed with another value\. Such tests would determine which operations the capacity statistic predicts\. They would not establish a human\-like memory architecture merely because a behavioral curve resembles a working\-memory limit\.

### 9\.2From binding to world\-state tracking

To test whether binding supports a model’s approximation of a world state, one must ask more than whether it retrieves a declared fact\. A stronger test would require the model to revise an entity’s properties after an event and use the revised state to predict or choose an action\. Our trials contain neither state changes nor action outcomes: each entity receives one obligation, and the model answers one query\. Controlled game sequences have supported tests of internal board\-state representations and their causal use\([Li et al\., 2023](https://arxiv.org/html/2609.30634#bib.bib14)\); state\-maintaining prompts have been used to track changes during embodied planning\([Yoneda et al\., 2024](https://arxiv.org/html/2609.30634#bib.bib28)\)\. These studies motivate the next operations to measure, not an inference that our capacity curve measures world\-model quality\.

Othello makes the comparison precise\. A board square has a changing value, while a legal move depends on the configuration of several squares\. The analogy is entity\-specific state, not a numerical equivalence between our loadKKand board size\. Equation \([8](https://arxiv.org/html/2609.30634#S7.E8)\) shows that the same recall curve can arise when a value is unavailable or when it is retained but inaccessible at the query\. Thus poor recall does not show that the state is absent internally\. Equation \([10](https://arxiv.org/html/2609.30634#S9.E10)\) shows that random single\-query accuracy does not determine the chance of retrieving every property needed for one decision\. Thus good midpoint capacity does not establish a coherent multi\-entity state\. Even perfect static lookup could solve every present trial without learning a state\-transition rule\.

Othello\-GPT goes beyond output accuracy by probing board state and intervening on the decoded representation before measuring changed move predictions\([Li et al\., 2023](https://arxiv.org/html/2609.30634#bib.bib14)\)\. In our setting, the declaration\-site entity vector precedes the assigned value \(Section[7\.4](https://arxiv.org/html/2609.30634#S7.SS4)\); a probe of the current value should instead use a later site that can access it\. An analogous binding intervention would change one entity’s represented value, predict a selective answer change, and check that other bindings remain unchanged\. Even that result would concern static retrieval until state updates and transitions are tested\.

A direct extension would compare static recall, overwritten values, and action\-conditioned state transitions on held\-out entity and value assignments\. It would score current\-state retrieval and the resulting prediction or action separately\. Matched\-length and matched\-position controls are needed before attributing differences to binding load rather than extra tokens or query distance\. One can then ask whether an absolute\-recall criterion at a statedKKandDDpredicts performance on the later operations\. A correlation across model families would remain confounded by model size and training recipe; a controlled training comparison would give a clearer test of transfer\.

The transfer question has two parts\. Synthetic numerical key–value training improved multi\-document question answering in the models tested by[Xiong et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib26)\. In contrast,[Yen et al\. \(2025\)](https://arxiv.org/html/2609.30634#bib.bib23)find that synthetic needle\-in\-a\-haystack scores do not reliably predict performance across downstream long\-context tasks\. These findings test different claims: an effect of training in one setting and the predictive value of a benchmark across settings\. Neither establishes that our measuredK50K\_\{50\}predicts a downstream task or that training to raise it improves that task\. The matched base\-versus\-code\-trained comparisons of[Kim et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib12)further show that training data can change entity\-tracking performance, without identifying binding capacity as the transferred ability\.

Training to raise the relativeK50K\_\{50\}is not itself the target\. BecauseK50K\_\{50\}is defined against each model’s own single\-binding ceiling, an increase inK50K\_\{50\}need not improve absolute recall at the load and interference level a task requires \(Section[8](https://arxiv.org/html/2609.30634#S8)\)\. A predictive test should ask whether absolute recall at fixed load and interference predicts held\-out task performance beyond a size\-only baseline, reporting within\-family and held\-out\-family results separately\. A binding curriculum should be compared with equal\-token, equal\-compute training controls, using held\-out bindings and downstream tasks to assess gains and losses\. Relevant endpoints include dialogue state tracking, program execution, multi\-document question answering, and state\-dependent planning\. General language performance and policy outcomes need separate measurement rather than inference from recall\.

### 9\.3Single\-query recall does not determine joint reliability

Consider an episode that requires correct retrieval ofm≥1m\\geq 1bindings\. LetRjR\_\{j\}be the event that bindingjjis retrieved correctly and letpj=Pr⁡\(Rj\)p\_\{j\}=\\Pr\(R\_\{j\}\)\. The events must refer to the same episode distribution and a specified retrieval protocol\. ForR=⋂j=1mRjR=\\bigcap\_\{j=1\}^\{m\}R\_\{j\},

max⁡\{0,1−∑j=1m\(1−pj\)\}≤Pr⁡\(R\)≤minj⁡pj\.\\max\\\!\\left\\\{0,1\-\\sum\_\{j=1\}^\{m\}\(1\-p\_\{j\}\)\\right\\\}\\leq\\Pr\(R\)\\leq\\min\_\{j\}p\_\{j\}\.\(10\)The lower bound follows by applying the union bound to the failure eventsRjcR\_\{j\}^\{c\}; the upper bound follows fromR⊆RjR\\subseteq R\_\{j\}\. No independence assumption is needed\. A uniform query, chosen independently of episode outcomes, measures the mean marginal recallp¯=m−1​∑jpj\\bar\{p\}=m^\{\-1\}\\sum\_\{j\}p\_\{j\}\. The lower bound is thenmax⁡\{0,1−m⁡\(1−p¯\)\}\\max\\\{0,1\-m\(1\-\\bar\{p\}\)\\\}\. The same average can accompany different dependencies among errors and therefore different probabilities of retrieving every binding\. Independent events with equal marginals would givePr⁡\(R\)=p¯m\\Pr\(R\)=\\bar\{p\}^\{\\,m\}; these assumptions are not established here\.

For a tolerated joint failure probabilityϵ∈\(0,1\)\\epsilon\\in\(0,1\), a sufficient condition is∑j\(1−pj\)≤ϵ\\sum\_\{j\}\(1\-p\_\{j\}\)\\leq\\epsilon\. In particular, marginal guaranteespj≥1−ϵ/mp\_\{j\}\\geq 1\-\\epsilon/mfor every required binding suffice\. This is a distribution\-free sufficient condition, not a necessary one or a guarantee estimated from our current trials\.

This distinction matters when several recovered constraints jointly determine an action\. A midpoint summary does not establish reliable joint retrieval, and a weak lower bound does not prove that joint retrieval is poor\. The paper’s random\-query trials do not directly measure success on a multi\-action episode\. Estimating that quantity requires a protocol that defines all required retrievals jointly, including any effects of earlier generated answers\.

### 9\.4Correct retrieval and policy adherence are separate outcomes

As a controlled example, an instruction\-following task can assign each resource an authorized action and a source of authority\. A model can retrieve the action correctly while applying the wrong authority, or retrieve the wrong action despite respecting the source hierarchy\.[Wallace et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib22)study instruction priority explicitly\. Our entity–obligation task has neither competing authority levels nor a policy\-adherence endpoint, so it cannot measure that ability\. This distinction also prevents interpreting the unchanged post\-training capacity threshold in Section[4](https://arxiv.org/html/2609.30634#S4)as evidence that alignment training has no effect\.

LetVVdenote a violation in a specified policy task and letRRdenote correct retrieval of all bindings required by that task\. Writer=Pr⁡\(R\)r=\\Pr\(R\),e1=Pr⁡\(V∣R\)e\_\{1\}=\\Pr\(V\\mid R\), ande0=Pr⁡\(V∣Rc\)e\_\{0\}=\\Pr\(V\\mid R^\{c\}\), with0<r<10<r<1\. The law of total probability gives

Pr⁡\(V\)=r​e1\+\(1−r\)​e0\.\\Pr\(V\)=re\_\{1\}\+\(1\-r\)e\_\{0\}\.\(11\)Thus retrieval reliability alone does not identify the violation rate\. In a comparison that holds both conditional violation rates fixed, a change fromrrtor′r^\{\\prime\}givesPr′⁡\(V\)−Pr⁡\(V\)=\(r′−r\)​\(e1−e0\)\\Pr^\{\\prime\}\(V\)\-\\Pr\(V\)=\(r^\{\\prime\}\-r\)\(e\_\{1\}\-e\_\{0\}\)\. Better retrieval reduces violations under this assumption only whene1<e0e\_\{1\}<e\_\{0\}\. A retrieval intervention can also change those conditional rates, and conditioning on naturally successful retrieval can select easier episodes\. The identity is therefore an accounting relation, not a causal mediation result\. Atr=0r=0orr=1r=1, use the unconditional joint probabilities because one conditional rate is undefined\.

A testable extension would randomize a retrieval aid against a matched control on the same synthetic policy tasks\. It would score retrieved bindings and final actions separately, and include permitted requests to detect indiscriminate refusal\. If the aid improves retrieval without reducing violations, the proposed benefit to adherence fails for that intervention\. Even a positive randomized effect would establish the effect of the aid; attributing it specifically to retrieval would require further mediation assumptions\. This is a bounded connection to alignment research, not a claim that greater binding capacity makes a model aligned\.

### 9\.5Interpretability must distinguish representation from use

Section[7\.1](https://arxiv.org/html/2609.30634#S7.SS1)shows why a recall curve cannot distinguish retention from access\. Section[7\.4](https://arxiv.org/html/2609.30634#S7.SS4)shows why the absence of a newly assigned value at an earlier entity token does not exclude an address role\. Together, they constrain what an interpretation experiment must establish\. A decoder tests whether its chosen representation exposes a label; it does not by itself test whether the model uses that information to choose its answer\.

The intervention designs of[Li et al\. \(2023\)](https://arxiv.org/html/2609.30634#bib.bib14);[Feng and Steinhardt \(2024\)](https://arxiv.org/html/2609.30634#bib.bib4);[Gur\-Arieh et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib6)provide examples of testing use through targeted changes in activations\. For our setting, a proposed test would swap an entity address or a later value representation while preserving the task’s other bindings\. It should predict which answer changes, not merely whether accuracy falls\. Unrelated donors and matched perturbations would test whether the change reflects binding content rather than generic disruption\. Such an intervention would connect the capacity curve to a candidate mechanism; the present geometry correlations and probes do not supply that connection\.

## 10Related work

[Kim et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib12)show with matched base and code\-trained pairs that code pretraining drives entity tracking, while alignment adds no consistent benefit\. We define the capacity thresholdk∗k^\{\\ast\}and sweep it across3030models, which exposes a flat Pythia ladder beside an8×8\\timesrecipe span\.[Feng and Steinhardt \(2024\)](https://arxiv.org/html/2609.30634#bib.bib4)identify a binding\-ID mechanism whose fidelity improves with scale\. Mechanism fidelity and capacity ceiling may vary separately, a distinction that calls for a direct dissociation experiment\. Architecture also changes entity\-tracking performance in closely matched transformer and hybrid models\([Li and Merrill, 2026](https://arxiv.org/html/2609.30634#bib.bib13)\), so our use of*recipe*includes architecture\. Across model families, the same composed task can rely on different circuits\([Xu, 2026](https://arxiv.org/html/2609.30634#bib.bib27)\), which motivates separating cross\-model geometry from trial\-level mechanism tests \(Section[7](https://arxiv.org/html/2609.30634#S7)\)\. On the scaling side,[Kaplan et al\. \(2020\)](https://arxiv.org/html/2609.30634#bib.bib11)and[Hoffmann et al\. \(2022\)](https://arxiv.org/html/2609.30634#bib.bib8)fit loss against scale; theK50K\_\{50\}law prices a specific in\-context capacity instead, and its regime boundary shows what such zero\-shot laws are laws of\.[Muennighoff et al\. \(2023\)](https://arxiv.org/html/2609.30634#bib.bib17)extend the loss accounting to data\-constrained training, pricing the decaying value of a repeated token; our boundary prices a named capability rather than loss\.[Allen\-Zhu and Li \(2024\)](https://arxiv.org/html/2609.30634#bib.bib1)price parametric knowledge in bits per parameter; in\-context binding state is the complementary quantity, held in activations rather than weights\. Related mechanisms, induction heads and associative memories, are described in[Olsson et al\. \(2022\)](https://arxiv.org/html/2609.30634#bib.bib18)and[Bietti et al\. \(2023\)](https://arxiv.org/html/2609.30634#bib.bib3)\.

Cognitive framings of the same limit measure it differently\.[Gong et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib7)adapt thenn\-back task to assess verbal and spatial working memory;[Xiong et al\. \(2026\)](https://arxiv.org/html/2609.30634#bib.bib25)report load\-dependent decline and human\-like interference signatures across pretrained models\. Our analysis instead estimates a load\-capacity scaling law across model sizes\. On the applied side,[Hsieh et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib9)include multi\-key retrieval against hard distractors and a variable\-tracking task in a long\-context benchmark\. Our task varies binding count and distractor length on separate axes and fits capacity across models; it does not fully separate binding count from total token length\.

Work on world\-state representations asks whether a model encodes and uses a changing state\.[Li et al\. \(2023\)](https://arxiv.org/html/2609.30634#bib.bib14)use a controlled board game and activation interventions, while[Yoneda et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib28)maintain an explicit state estimate for planning\. Our static binding task isolates an earlier operation and does not test those claims\.[Xiong et al\. \(2024\)](https://arxiv.org/html/2609.30634#bib.bib26)report transfer from synthetic retrieval training to multi\-document question answering, which motivates but does not settle the transfer question for our capacity measure\.[Yen et al\. \(2025\)](https://arxiv.org/html/2609.30634#bib.bib23)find that synthetic needle retrieval scores do not reliably predict downstream long\-context performance\. Whether our load\-dependent measure predicts performance beyond model size remains a separate empirical question\.

## 11Discussion and conclusion

#### Limitations\.

The recipe split is observational\. The 29 measurable models differ in data, tokenizer, optimizer, architecture, and era\. Our comparison identifies a recipe\-level regularity without causal attribution\. Thek∗k^\{\\ast\}statistic is also right\-censored at2424by the entity pool, making the low end of the range more informative than the ceiling\.

The continuous curves give a different view at or below 3B: recipe adds no detectable effect after scale is controlled\. Curves above 3B are needed to determine whether the broad recipe split returns at larger scale\. TheK50K\_\{50\}scaling fit was found post hoc and awaits pre\-registered replication\. The synthetic, single\-token task provides direct control over load and interference\. The resulting laws apply to this task, these model ranges, and these estimators\. The geometric quantities in Section[7](https://arxiv.org/html/2609.30634#S7)describe cross\-model correlates and do not identify a trial\-level mechanism\. The sensitivity checks in Sections[3\.4](https://arxiv.org/html/2609.30634#S3.SS4)and[4\.3](https://arxiv.org/html/2609.30634#S4.SS3), and the interference decomposition in Section[6](https://arxiv.org/html/2609.30634#S6), include exploratory re\-analyses of the existing corpus\. These analyses were not predicted in advance\. The threshold and identification results are conditional deductions, and the tests in Section[8](https://arxiv.org/html/2609.30634#S8)remain proposals\.

#### Conclusion\.

The broad sweep and continuous curves measure different parts of in\-context binding capacity\. On continuous curves at or below 3B,K50K\_\{50\}follows a size law and the recipe coefficient spans zero after scale is controlled; across models to 12B,k∗k^\{\\ast\}shows recipe\-associated differences across scale\.

Direct task training exceeds an extrapolation of the zero\-shot law by26×26\\times, under different measurement criteria\. Interference robustness varies on another axis\. The normalisation identity explains why a loss at the single\-binding ceiling can appear as a capacity loss, and the threshold analysis specifies when a scaling exponent transfers to other recall criteria\. Capacity numbers should therefore state the task, instrument, training regime, and interference load that produced them\. These qualifications turn the capacity curve into a description of usable recall while leaving the storage mechanism open\.

## Acknowledgements

We would like to thank Kevin Li for his help with proofreading, for helping us to better write the paper for readability, and for the suggestions that led to this paper\.

## References

- Allen\-Zhu and Li \[2024\]Z\. Allen\-Zhu and Y\. Li\. Physics of language models: part 3\.3, knowledge capacity scaling laws\. 2024\. arXiv:2404\.05405\.
- Biderman et al\. \[2023\]S\. Biderman, H\. Schoelkopf, Q\. Anthony, et al\. Pythia: a suite for analyzing large language models across training and scaling\.*ICML*, 2023\. arXiv:2304\.01373\.
- Bietti et al\. \[2023\]A\. Bietti, V\. Cabannes, D\. Bouchacourt, H\. Jegou, and L\. Bottou\. Birth of a transformer: a memory viewpoint\.*NeurIPS*, 2023\. arXiv:2306\.00802\.
- Feng and Steinhardt \[2024\]J\. Feng and J\. Steinhardt\. How do language models bind entities in context?*ICLR*, 2024\. arXiv:2310\.17191\.
- Gu et al\. \[2025\]X\. Gu, T\. Pang, C\. Du, Q\. Liu, F\. Zhang, C\. Du, Y\. Wang, and M\. Lin\. When attention sink emerges in language models: an empirical view\.*ICLR*, 2025\. arXiv:2410\.10781\.
- Gur\-Arieh et al\. \[2026\]Y\. Gur\-Arieh, M\. Geva, and A\. Geiger\. Mixing mechanisms: how language models retrieve bound entities in\-context\.*ICLR*, 2026\. arXiv:2510\.06182\.
- Gong et al\. \[2024\]D\. Gong, X\. Wan, and D\. Wang\. Working memory capacity of ChatGPT: an empirical study\.*AAAI*, 2024\. arXiv:2305\.03731\.
- Hoffmann et al\. \[2022\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, et al\. Training compute\-optimal large language models\. 2022\. arXiv:2203\.15556\.
- Hsieh et al\. \[2024\]C\.\-P\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg\. RULER: what’s the real context size of your long\-context language models?*COLM*, 2024\. arXiv:2404\.06654\.
- Huang et al\. \[2026\]J\. Huang, D\. Wurgaft, R\. Bansal, L\. Ruis, N\. Saphra, D\. Alvarez\-Melis, A\. K\. Lampinen, C\. Potts, and E\. S\. Lubana\. Why larger models learn more: effects of capacity, interference, and rare\-task retention\. 2026\. arXiv:2605\.29548\.
- Kaplan et al\. \[2020\]J\. Kaplan, S\. McCandlish, T\. Henighan, et al\. Scaling laws for neural language models\. 2020\. arXiv:2001\.08361\.
- Kim et al\. \[2024\]N\. Kim, S\. Schuster, and S\. Toshniwal\. Code pretraining improves entity tracking abilities of language models\. 2024\. arXiv:2405\.21068\.
- Li and Merrill \[2026\]Y\. Li and W\. Merrill\. Comparing transformers and hybrid models at the token level\. 2026\. arXiv:2606\.20936\.
- Li et al\. \[2023\]K\. Li, A\. K\. Hopkins, D\. Bau, F\. Viégas, H\. Pfister, and M\. Wattenberg\. Emergent world representations: exploring a sequence model trained on a synthetic task\.*ICLR*, 2023\. arXiv:2210\.13382\.
- Liu et al\. \[2025\]Y\. Liu, Z\. Liu, and J\. Gore\. Superposition yields robust neural scaling\.*NeurIPS*, 2025\. arXiv:2505\.10465\.
- Liu et al\. \[2023\]N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang\. Lost in the middle: how language models use long contexts\. 2023\. arXiv:2307\.03172\.
- Muennighoff et al\. \[2023\]N\. Muennighoff, A\. M\. Rush, B\. Barak, et al\. Scaling data\-constrained language models\. 2023\. arXiv:2305\.16264\.
- Olsson et al\. \[2022\]C\. Olsson, N\. Elhage, N\. Nanda, et al\. In\-context learning and induction heads\. 2022\. arXiv:2209\.11895\.
- Radford et al\. \[2019\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\. Language models are unsupervised multitask learners\. OpenAI technical report, 2019\.
- Schaeffer et al\. \[2023\]R\. Schaeffer, B\. Miranda, and S\. Koyejo\. Are emergent abilities of large language models a mirage?*NeurIPS*, 2023\. arXiv:2304\.15004\.
- Sun et al\. \[2024\]M\. Sun, X\. Chen, J\. Z\. Kolter, and Z\. Liu\. Massive activations in large language models\.*COLM*, 2024\. arXiv:2402\.17762\.
- Wallace et al\. \[2024\]E\. Wallace, K\. Xiao, R\. Leike, L\. Weng, J\. Heidecke, and A\. Beutel\. The instruction hierarchy: training LLMs to prioritize privileged instructions\. 2024\. arXiv:2404\.13208\.
- Yen et al\. \[2025\]H\. Yen, T\. Gao, M\. Hou, K\. Ding, D\. Fleischer, P\. Izsak, M\. Wasserblat, and D\. Chen\. HELMET: how to evaluate long\-context language models effectively and thoroughly\.*ICLR*, 2025\. arXiv:2410\.02694\.
- Wang and Sun \[2025\]C\. Wang and J\. V\. Sun\. Unable to forget: proactive interference reveals working memory limits in LLMs beyond context length\. 2025\. arXiv:2506\.08184\.
- Xiong et al\. \[2026\]H\.\-D\. Xiong, L\. Ji\-An, J\. Huang, R\. C\. Wilson, K\. Lee, and X\.\-X\. Wei\. In\-context superposition: human\-like working memory interference in large language models\. 2026\. arXiv:2604\.09670\.
- Xiong et al\. \[2024\]Z\. Xiong, V\. Papageorgiou, K\. Lee, and D\. Papailiopoulos\. From artificial needles to real haystacks: improving retrieval capabilities in LLMs by finetuning on synthetic data\. 2024\. arXiv:2406\.19292\.
- Xu \[2026\]Y\. Xu\. Pattern selectivity is not task\-causal structure: a cross\-architecture mechanistic study of composed\-task circuits in 1B\-class language models\. 2026\. arXiv:2606\.05378\.
- Yoneda et al\. \[2024\]T\. Yoneda, J\. Fang, P\. Li, et al\. Statler: state\-maintaining language models for embodied reasoning\.*ICRA*, 2024\. arXiv:2306\.17840\.

## Appendix AMeasurement diagnostics for declaration\-site geometry

This appendix reports measurement diagnostics for the declaration\-site geometric quantities\. Section[7](https://arxiv.org/html/2609.30634#S7)presents the recipe comparisons and trial\-level prediction tests\.

#### The geometric quantities, and what they measure\.

At the declaration site we take one residual vector per declared entity and computeddecld\_\{\\mathrm\{decl\}\}, the participation ratio of those vectors \(their effective dimension\), and the mean off\-diagonal squared cosine among them \(pairwise*interference*\)\.

The natural name for these quantities, a binding geometry, overstates them\. Each declaration readsentity: obligation\., so the entity token*precedes*its obligation, and attention is causal\. The hidden state at an entity token cannot directly encode a newly assigned obligation that appears later\. Section[7\.4](https://arxiv.org/html/2609.30634#S7.SS4)distinguishes this constraint from a possible address role\. With the per\-slot mean removed, the entity decodes at1\.0001\.000from the entity\-token residual\. Its bound obligation decodes at0\.0170\.017–0\.0540\.054against a chance of0\.0400\.040\(Pythia\-1\.4B/6\.9B, Qwen2\.5\-7B, OLMo\-2\-7B\)\. The quantitiesddecld\_\{\\mathrm\{decl\}\}and interference therefore describe how many directions the declaration\-site entity code uses and how much those directions overlap\. Capacity can depend on this geometry, but the geometry does not measure the binding itself\.

### A\.1Measurement artifacts at the declaration site

Each of these artifacts produces a wrong but plausible\-looking number, and each is easy to walk into\.

#### 1\. Position dominates the raw vectors\.

Entityiisits in slotii, so declaration\-site hidden states are overwhelmingly positional: the per\-slot mean carries98\.198\.1–99\.9%99\.9\\%of their variance on 7 of the 8 models with committed geometry; the exception is OLMo\-2, at23%23\\%\. A participation ratio taken on the raw vectors reports an effective dimension of at most1\.031\.03and a pairwise squared cosine of0\.840\.84–1\.001\.00: it is describing*position*\. Estimating the positional component as the per\-slot mean across trials and measuring on the residual gives an interference of0\.0060\.006–0\.0090\.009on all 8 models: the entity code is close to orthogonal to the positional one\.

#### 2\. Massive activations dominate the covariance\.

Prior work identifies massive activations and their relation to attention sinks\[[Sun et al\., 2024](https://arxiv.org/html/2609.30634#bib.bib21),[Gu et al\., 2025](https://arxiv.org/html/2609.30634#bib.bib5)\]\. In our measurements, after position is removed, a*covariance*participation ratio is dominated by these few activation directions\. It reports an effective dimension of1\.31\.3for Qwen2\.5\-7B, a model that carries54%54\\%of its residual variance in a single coordinate\. Standardizing per coordinate gives35\.335\.3, and the dimension then scales sensibly with model size within the family \(12\.012\.0,35\.335\.3,81\.181\.1for1\.51\.5B,77B,1414B\)\.

Both artifacts are*installed by pretraining*\. Across Pythia\-1\.4B checkpoints the positional variance fraction rises from0\.150\.15to0\.990\.99and the top coordinate’s share from0\.0010\.001to0\.170\.17, so a measurement that behaves well on an early checkpoint can invert on a converged one\. Under the corrected estimator, the declaration\-site subspace expands and then plateaus\. The covariance ratio instead suggests compression\.

#### 3\. The site precedes the assigned value\.

The declaration\-site vector is taken at the entity token, which causally precedes its obligation, and the probes above decode the entity at1\.0001\.000and the obligation at chance\. Thusddecld\_\{\\mathrm\{decl\}\}is the effective dimension of the entity code, and packing efficiency measures how the model spends directions on entities that it must hold apart\.

No downstream figure exposes this\. The artifact is visible only in the projection of the declaration\-site residuals onto the obligation classes, which gives no separation in any frame, including a supervised one\. A further provenance caveat: the plotting generator for Figure[6](https://arxiv.org/html/2609.30634#S7.F6)reads a committed table, but the*measurement*script that produced theddecld\_\{\\mathrm\{decl\}\}values was not preserved, so we cannot independently verify the token position at which those particular numbers were taken\. Re\-deriving them at a position that*can*carry a binding is the correct next measurement: the query site, where on 5 models the retrieved obligation decodes at0\.510\.51–0\.860\.86against a chance of0\.0400\.040\.