The information geometry of large language models is shared, learned, and controllable

# The information geometry of large language models is shared, learned, and controllable
Source: [https://arxiv.org/html/2609.11063](https://arxiv.org/html/2609.11063)
\\SetLinkTargetFilter

main\-\#1

Dario PicozziAffiliation:Department of Physics and Astronomy, University College London \(UCL\),Affiliation:Gower Street, London WC1E 6BT, United KingdomAffiliation:London Centre for Nanotechnology, 19 Gordon St, London WC1H 0AH, United KingdomEmail:[picozzi\.dario@gmail\.com](mailto:)

###### Abstract

Large language models learn similar behaviours, yet it remains unclear what structure they share or how to change one behaviour without disturbing others\. The Fisher–Rao geometry of next\-token probabilities connects these questions: behaviour determines this geometry up to output\-preserving symmetries, whereas activation geometry depends on coordinates\. Across transformer, state\-space and recurrent models, output geometries agree more strongly than activation geometries, and shared geometry supports semantic\-category transfer\. Agreement with human word choices increases with predictive accuracy, scale and training, and improves further after model\-only calibration\. Token probabilities and read\-out geometry jointly predict the spectrum and its effective dimension\. Controlled language assignments show that geometry follows the language law across architectures\. Pretraining corpus statistics predict held\-out fact acquisition without recalibration, while randomised experiments show that deeper evidence substantially delays acquisition across every tested architecture and evidence construction\. Finally, the geometry prescribes minimum\-disturbance local interventions, predicts their relative cost, and supports reusable control: updates learned on donor prompts transfer to unseen prompts while better preserving behaviour on reference prompts than Euclidean control\. The same geometric correction improves steering, editing, attribution, dictionary learning and fine\-tuning\.

A language model’s predictive behaviour does not privilege any coordinate system for its internal activations: the same input–output function can be realised in any coordinate system related by an invertible linear transformation\. Euclidean distances between activations are not invariant under these reparameterisations, yet Euclidean inner products underlie common measures of activation distance, feature importance, steering and fine\-tuning regularisation\. On the other hand, a model’s next\-token probability distribution can retain substantial information about its internal state and input: decoder\-only Transformer hidden\-state mappings are almost surely injective under mild conditions[1](https://arxiv.org/html/2609.11063#bib.bib1), and much of an input prompt can be reconstructed from the next\-token distribution alone[2](https://arxiv.org/html/2609.11063#bib.bib2),[3](https://arxiv.org/html/2609.11063#bib.bib3)\. A Riemannian metric assigns local lengths and angles to infinitesimal changes on a space\. Chentsov’s theorem shows that invariance under transformations preserving statistical information gives the Fisher–Rao metric as the unique canonical metric on the space of probability distributions, up to an overall scale[1](https://arxiv.org/html/2609.11063#as1_bib.bib1),[5](https://arxiv.org/html/2609.11063#bib.bib5),[6](https://arxiv.org/html/2609.11063#bib.bib6)\. Behaviour space therefore has a privileged geometry even when representation space does not\.

Pulling the output Fisher metric back through the network measures an activation change by the local output change it produces\. The corresponding damped natural gradient achieves a specified objective change at minimum regularised local output cost[15](https://arxiv.org/html/2609.11063#as1_bib.bib15),[8](https://arxiv.org/html/2609.11063#bib.bib8)\.

The same geometry supports comparison across models: for common contexts, each model gives a matrix of pairwise Fisher–Rao distances between its next\-token distributions\. Comparing these matrices requires no shared vocabulary, architecture or activation coordinates\. This recasts the convergence phenomenon behind the Platonic Representation Hypothesis[9](https://arxiv.org/html/2609.11063#bib.bib9)in an output geometry where, under the stated conditions, convergence follows from predictive fit rather than from a chosen activation\-similarity measure[10](https://arxiv.org/html/2609.11063#bib.bib10),[11](https://arxiv.org/html/2609.11063#bib.bib11)\. Activation geometry has no comparable standing: the same behaviour admits invertible hidden\-state reparameterisations with different Euclidean activation geometries, leaving activation\-space convergence as an empirical property of training\.

Closed\-form natural\-gradient steps grounded in Fisher information geometry have recently been validated for activation steering[6](https://arxiv.org/html/2609.11063#as1_bib.bib6),[7](https://arxiv.org/html/2609.11063#as1_bib.bib7), with FishBack extending to 8\-billion\-parameter models\. Nonlinear activation\-space steering has also been explored[14](https://arxiv.org/html/2609.11063#bib.bib14); representation\-level steering baselines are established practice[13](https://arxiv.org/html/2609.11063#as1_bib.bib13),[16](https://arxiv.org/html/2609.11063#bib.bib16),[17](https://arxiv.org/html/2609.11063#bib.bib17),[18](https://arxiv.org/html/2609.11063#bib.bib18); and identifiable properties of next\-token predictors have been characterised theoretically[19](https://arxiv.org/html/2609.11063#bib.bib19)\. Analyses of approximate agreement distinguish predictive closeness from representational similarity, including limitations of KL\-based guarantees and sufficient conditions based on logit distance[20](https://arxiv.org/html/2609.11063#bib.bib20),[21](https://arxiv.org/html/2609.11063#bib.bib21)\. Approximate identifiability has also been studied for representations with nonlinear decoders[22](https://arxiv.org/html/2609.11063#bib.bib22)\.

Related work has linked next\-token statistics to representation structure[23](https://arxiv.org/html/2609.11063#bib.bib23),[24](https://arxiv.org/html/2609.11063#bib.bib24), identified statistical symmetries shaping geometric structure[25](https://arxiv.org/html/2609.11063#bib.bib25), and studied transient semantic organization during training[26](https://arxiv.org/html/2609.11063#bib.bib26)\. Corpus correlations have also been linked to hierarchical language acquisition and data\-limited loss scaling[27](https://arxiv.org/html/2609.11063#bib.bib27),[28](https://arxiv.org/html/2609.11063#bib.bib28)\. Other studies document cross\-model convergence and cross\-architecture steering transfer in internal representations[29](https://arxiv.org/html/2609.11063#bib.bib29),[30](https://arxiv.org/html/2609.11063#bib.bib30)\. Here we connect these strands into a framework defined by the geometry of model outputs: predictive behaviour identifies the geometry, language statistics predict its structure and acquisition, and its measured anisotropy predicts intervention cost across architectures and operations\.

Predictive behaviour determines the canonical output geometry up to symmetries that preserve outputs, with explicit bounds on recovery error, whereas activation geometry remains coordinate\-dependent under invertible reparameterisation\. Across ten independently trained models spanning transformer, state\-space and recurrent architectures, the identified relational geometries have a mean rank agreement of 0\.88, compared with 0\.62 for mid\-layer activation geometries; their shared component supports semantic\-category transfer\. Human completion geometry becomes closer with predictive fit, and a relation fitted on one model family predicts agreement in others\. Controlled language assignments causally determine the learned geometry across architectures\. Token probabilities and learned read\-out structure explain complementary aspects of its spectrum, with effective dimension predicted without fitted parameters\. At fixed relative damping and solver tolerance, the damped natural\-gradient solve has a worst\-case conjugate\-gradient iteration bound independent of model width\. Corpus statistics also predict when behaviours are acquired: in a randomised synthetic\-language test, assigning otherwise identical facts to deeper statistical evidence delays early acquisition more than fourfold\. The geometry also prescribes minimum\-disturbance interventions\. A measured anisotropy ratio, which quantifies how unevenly different activation directions affect the output, predicts their advantage over Euclidean control without fitted parameters\. Averaging the metric over reference prompts produces reusable updates that transfer to unseen prompts while reducing reference\-sequence change\. Applying the same correction improves steering, knowledge editing, feature attribution, dictionary learning and fine\-tuning, with up to two orders of magnitude less off\-target change\.

### Results

#### Predictive behaviour fixes a canonical, identifiable output geometry

ab

Figure 1:Behaviour identifies the resolved read\-out, and geometry predicts local output change\.a, Behavioural identification: held\-out predictive loss after re\-optimising nuisance parameters grows linearly with the squared subspace distance between a perturbed read\-out subspace and the resolved subspace \(Pythia 70M–1\.4B\)\. Markers are the 40 model–condition–angle means and bars span the two independently calibrated held\-out folds \(smaller than the markers\); paths connect angles within each model and condition\. Green, read\-out deviation; grey, matched\-random control; circles, 70M; squares, 160M; triangles, 410M; diamonds, 1\.4B\. The local fits haveR2≥0\.9999R^\{2\}\\geq 0\.9999on all eight paths, and matched\-random control directions have systematically smaller loss–distance slopes \(slope ratios1\.141\.14–1\.271\.27\)\. The reference subspace is estimated from a stronger model \(Supplementary Note\)\.b, Predicted and measured output change under small activation perturbations\. Each point is one model–depth–prompt combination, giving 99 measurements across 11 models, six families and quarter\-, half\- and three\-quarter\-depth injection\. The prediction12​δ​h⊤​G​δ​h\\tfrac\{1\}\{2\}\\delta h^\{\\top\}G\\delta huses the metric before the intervention, without a fitted scale\. Points show the coarsest resolved local steps; colour identifies the model and the grey line marks equality\. The measured\-to\-predicted KL ratio has median1\.0001\.000and range0\.9480\.948–1\.2241\.224\(Methods; Supplementary Note\)\.A language model’s output is a probability distribution over its vocabulary\. The Fisher–Rao metric measures local separation between such distributions and is unique up to scale by its invariance under sufficient\-statistic transformations[1](https://arxiv.org/html/2609.11063#as1_bib.bib1),[5](https://arxiv.org/html/2609.11063#bib.bib5),[6](https://arxiv.org/html/2609.11063#bib.bib6)\. Lethhbe the activation at the chosen intervention layer andhL​\(h\)h\_\{L\}\(h\)the final hidden state it produces\. Writingph=softmax⁡\(WU​hL​\(h\)\+bU\)p\_\{h\}=\\mathrm\{softmax\}\(W\_\{U\}h\_\{L\}\(h\)\+b\_\{U\}\)for the output law, with unembedding rowswaw\_\{a\}andJh=∂hL/∂hJ\_\{h\}=\\partial h\_\{L\}/\\partial h, the output FisherHh=Covph​\(w\)H\_\{h\}=\\mathrm\{Cov\}\_\{p\_\{h\}\}\(w\)defines the \(possibly degenerate\) pullback Fisher metricGh=Jh⊤​Hh​JhG\_\{h\}=J\_\{h\}^\{\\top\}H\_\{h\}J\_\{h\}at the intervention layer\. For a small intervention,

KL\(ph∥ph\+δ​h\)=12δh⊤Ghδh\+O\(∥δh∥3\)\.\\mathrm\{KL\}\\\!\\left\(p\_\{h\}\\,\\\|\\,p\_\{h\+\\delta h\}\\right\)=\\tfrac\{1\}\{2\}\\,\\delta h^\{\\top\}G\_\{h\}\\,\\delta h\+O\\\!\\left\(\\lVert\\delta h\\rVert^\{3\}\\right\)\.\(1\)The location subscript is suppressed below\. The quadratic form measures local output change and matches the realised divergence locally across 99 model–depth–objective cells spanning 11 models, six families and 125M–1\.5B parameters \(median ratio1\.0001\.000; Fig\.[1](https://arxiv.org/html/2609.11063#Sx1.F1)b\), so geometric length predicts behavioural change\. For an objective gradientq=∇hϕq=\\nabla\_\{h\}\\phi, the damped natural\-gradient stepδ​h∝\(G\+α​R\)−1​q\\delta h\\propto\(G\+\\alpha R\)^\{\-1\}q[15](https://arxiv.org/html/2609.11063#as1_bib.bib15),[8](https://arxiv.org/html/2609.11063#bib.bib8)is coordinate\-covariant: transporting it across a seeded family of activation reparameterisations reproduces the native direction to<10−12<10^\{\-12\}, whereas resettingRRto the identity changes it by order one \(Extended Data Fig\.[1](https://arxiv.org/html/2609.11063#Sx9.F1)a; Supplementary Note\)\. The connection to training is exact: excess population cross\-entropy equals the Kullback–Leibler divergence to a model\-induced Gibbs law whose conditional curvature is this output Fisher metric\. Next\-token training therefore induces the geometry \(Supplementary Note\)\.

This geometry is also identifiable from behaviour alone: predictive behaviour singles out the language\-defined read\-out subspace at matched rank rather than a choice of hidden coordinates\. LetTTdenote this resolved subspace,SSa candidate subspace,d⁡\(S,T\)d\(S,T\)their chordal distance \(a standard measure of subspace separation\), and𝒦⁡\(S\)\\mathcal\{K\}\(S\)the profiled predictive risk after nuisance parameters have been optimised out\. Under the stated finite\-support and inverse\-chart conditions, behavioural identification obeys the global quadratic margin

𝒦⁡\(S\)≥c​d2​\(S,T\)\.\\mathcal\{K\}\(S\)\\;\\geq\\;c\\,d^\{2\}\(S,T\)\.\(2\)Herec\>0c\>0is a certified lower bound on predictive cost per unit squared subspace displacement, determined by the reference law and context weights \(Supplementary Note\)\. Zero profiled risk therefore identifies the resolved subspace exactly at matched rank\. More generally, finite\-rank approximation and excess prediction error bound recovery at a square\-root rate, so two accurate rank\-matched models of the same language must converge on the same subspace \(Supplementary Note\)\. Held\-out profiled loss grows quadratically with subspace distance across four model sizes \(R2≥0\.9999R^\{2\}\\geq 0\.9999on every path\), with steeper growth for actual read\-out deviations than for matched\-random controls \(Fig\.[1](https://arxiv.org/html/2609.11063#Sx1.F1)a\)\. The language law determines the learned geometry in a controlled intervention\. Transformer, gated\-recurrent and diagonal\-recurrent models at two capacities were trained on each of eight pairs of synthetic languages with known conditional laws, identical token frequencies and conditional entropy\. At matched predictive accuracy, changing the assigned law recovers more than 99% of the imposed squared geometric separation, whereas changing architecture within a law leaves the geometry nearly unchanged\. The assigned law is also a closer geometric match than its counterfactual in all eight pairs \(Supplementary Note\)\. By contrast, an invertible reparameterisation of a hidden layer, compensated downstream, leaves every conditional law unchanged: behaviour identifies the resolved output geometry while activation geometry retains an exactGL⁡\(d\)\\mathrm\{GL\}\(d\)gauge freedom, the freedom to apply any invertible linear change of activation coordinates \(Supplementary Note\)\. The resolved output geometry is therefore invariant under behaviour\-preserving reparameterisation\.

#### Independently trained models share the geometry

On a common outcome space, the convergence result in Supplementary Notebounds relational\-geometry differences by predictive error\. Native\-vocabulary agreement across tokenizers is tested empirically here\. For a common battery of contexts, each model defines a matrix of Fisher–Rao distances among its next\-token distributions, and models are compared by the Spearman correlation between the matrix upper triangles\. This asks whether they order the same context pairs from near to far and needs no shared tokenizer, architecture or activation alignment \(Methods\)\. Across ten models spanning transformer, state\-space and recurrent architectures, seven training pipelines, four tokenizers and 70 million to 7 billion parameters, the mean rank agreement on natural text is 0\.88, against 0\.62 for mid\-layer and 0\.61 for last\-layer activation geometries, a gap whose bootstrap interval excludes zero \(Fig\.[2](https://arxiv.org/html/2609.11063#Sx1.F2)a\)\. Under random anisotropic reparameterisations that leave behaviour unchanged, activation\-based measures degrade steadily while output\-geometry agreement is constant to machine precision \(Extended Data Fig\.[1](https://arxiv.org/html/2609.11063#Sx9.F1)c\), and partialling out orthographic surface geometry or the strongest corpusnn\-gram predictor leaves at least 98% of the agreement in place \(Supplementary Note\)\. As a tokenizer\-independent check, each next\-token law was mapped to the distribution of the first byte of the remaining text\. In this common outcome space, cross\-tokenizer agreement is 0\.91 and coarse\-graining contracts the divergence on every pair, so the sharing is not an artefact of token\-level bookkeeping \(Extended Data Fig\.[1](https://arxiv.org/html/2609.11063#Sx9.F1)b; Supplementary Note\)\.

The agreement can also be accounted for quantitatively\. Each distance\-matrix entry changes at most in proportion to the root\-probability distance∥p−q∥2\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}, so excess risk forces relational convergence at a proved rate \(Supplementary Note\)\. On same\-tokenizer pairs the resulting data\-dependent certificate gives lower bounds of0\.610\.61–0\.670\.67against observed0\.760\.76–0\.830\.83: the models disagree substantially per context \(median root\-probability distance0\.300\.30–0\.710\.71\), but the disagreement directions are nearly orthogonal to the relational structure \(coherence≈0\.03\\approx 0\.03; Supplementary Table\)\. With a fixed reference geometry, agreement decomposes into six variance and covariance terms: one reference, two residual, two reference–residual and one residual cross\-term\. This identity reconstructs every measured configuration to numerical precision \(112 model\-pair configurations\)\. Fitted on one half of the contexts, its exchangeable restriction predicts held\-out agreement with pooled median absolute error0\.0090\.009across the 96 defined cells\. From step 256, predictions are defined for every cell and checkpoint\-median errors are below0\.0140\.014\(Fig\.[2](https://arxiv.org/html/2609.11063#Sx1.F2)b; Supplementary Note\)\.

The shared component carries semantic alignment and category transfer\. Averaging rank\-transformed distance matrices gives a cross\-model consensus, and each model’s deviation from it is its residual\. Consensus geometry aligns with an independent sentence\-encoder semantic geometry on every battery; the alignment rises to 0\.44 when the next token answers a factual relation and survives a surface control at\+0\.23\+0\.23, whereas residual alignment is indistinguishable from zero on the natural and templated batteries and small \(\+0\.05\+0\.05\) on the semantic battery \(Supplementary Note\)[3](https://arxiv.org/html/2609.11063#as1_bib.bib3)\. Separately, the mean cosine\-distance geometry of last\-token, last\-hidden\-state representations from two language models aligns with the representational geometry of a vision model trained without language \(rank agreement 0\.39, permutationP<0\.001P<0\.001\), with 84% of the association retained after controlling for textual co\-occurrence[17](https://arxiv.org/html/2609.11063#as1_bib.bib17)\. A linear semantic probe trained on one model transfers across models with eight\-way accuracy of 0\.66 \(0\.700\.70through the consensus geometry alone\), close to the within\-model 0\.72 and far above the 0\.125 chance level, whereas transfer through the residual is below chance \(Supplementary Notesand\)\. What is shared beyond the distributions themselves is narrow: against a displacement\-magnitude\-matched null, residual eigenvector sharing between families is about 9% of the raw overlap and is concentrated in the leading modes \(Extended Data Fig\.[2](https://arxiv.org/html/2609.11063#Sx9.F2)\); the residual is also graded by training lineage \(Supplementary Note\)\.

The comparison extends to human predictions\. On 512 sentence contexts with human completion norms[20](https://arxiv.org/html/2609.11063#as1_bib.bib20), semantic distributions estimated from native model samples become closer to human completions with model scale: mean squared distance falls from about0\.340\.34at 70M to0\.300\.30at 2\.8B parameters \(Fig\.[2](https://arxiv.org/html/2609.11063#Sx1.F2)c\)\. Reliability\-corrected alignment of the relational semantic geometry is about0\.820\.82–0\.850\.85across the six sizes \(Methods\)\. Predictive fit also accounts for human alignment across architectures and training\. On a separate 384\-context battery, with model laws conditioned on human\-observed completion events, a risk\-to\-alignment relation fitted on Pythia sizes and checkpoints predicts OLMo checkpoints and five external models without refitting, with correlation0\.760\.76and root\-mean\-square error0\.0290\.029\(Fig\.[2](https://arxiv.org/html/2609.11063#Sx1.F2)d\)\.

A separate corpus of 1,726 human next\-word predictions provides a direct test in the output probability geometry[22](https://arxiv.org/html/2609.11063#as1_bib.bib22)\. After mapping model and human laws to the same 65 outcomes, model\-derived leading directions recover human directional structure above a random subspace control\. Simultaneous lower bounds are positive in six of seven model families, with alignment retained across disjoint participant groups \(Supplementary Note\)\. These comparisons connect shared model geometry to human predictive structure\.

The native scale trend replicated on independent data from 640 word positions in five previously unused narratives \(64,000 responses\)[23](https://arxiv.org/html/2609.11063#as1_bib.bib23)\. Across six Pythia sizes, semantic distance fell with log parameter count \(slope−0\.00872\-0\.00872, 95% interval−0\.01202\-0\.01202to−0\.00556\-0\.00556\), and it also fell from the initial to final checkpoint in both Pythia and OLMo \(Extended Data Fig\.[2](https://arxiv.org/html/2609.11063#Sx9.F2)b,c\)\. A calibration learned only from model probability laws, and fixed before the new human responses, improved expected human log score for the complete 65\-outcome coarse distribution in six of seven model families \(gains0\.01960\.0196–0\.05070\.0507nats; Qwen2,−0\.0030\-0\.0030nats; Extended Data Fig\.[2](https://arxiv.org/html/2609.11063#Sx9.F2)d\)\. Shared model\-law structure therefore supports out\-of\-corpus prediction as well as directional alignment\.

abcd

Figure 2:Predictive geometry is shared across models and approaches human completion structure\.a, Output Fisher–Rao agreement against activation\-space agreement on natural text for all 29 cross\-tokenizer, cross\-family dyads among ten models \(70M–7B; transformer, state\-space and recurrent\)\. Each dyad contributes a blue mid\-layer circle and vermilion last\-layer square joined by a horizontal line; the grey line is equality\. The output relation is stronger than either activation relation for every displayed mark\.b, Absolute held\-out error of the exchangeable six\-scalar prediction across ten public training checkpoints\. Circles show all 96 defined predictions; colours distinguish individual comparisons and lines follow them across training\. Six predictions are defined per checkpoint at steps 1–64 and twelve from step 256; estimator definitions and early\-checkpoint coverage are given in Supplementary Note\.c, Native model samples approach human completions with scale\. Points show all 96 context\-fold means, 16 at each Pythia size; colours identify models in panels c and d\. The ordinate is squared distance between semantic kernel means, cross\-fitted over participant and model\-sample halves \(512 contexts, 32 samples per context; eight\-token generation limit\)\. No human\-answer restriction is applied to sampling\.d, A relation between conditional prediction risk and human semantic alignment fitted on Pythia is applied without refitting to five OLMo checkpoints and five external models\. Points show all 160 model–context\-fold comparisons; checkpoints and folds are repeated measurements\. Alignment is reliability\-corrected kernel CKA on a separate 384\-context battery, using model laws conditional on the observed completion events\. The grey line is equality; correlation0\.7580\.758, root\-mean\-square error0\.02950\.0295\.
#### The geometry inherits the statistics of language

In the standardised centred read\-out representation, output\-Fisher eigenvalues rank directions by local output effect, and damping defines the effective dimension, a soft count of resolved modes\. Letuau\_\{a\}be tokenaa’s centred, orthonormalised read\-out row after removing common\-logit shifts \(Methods\), withu¯=∑apa​ua\\bar\{u\}=\\sum\_\{a\}p\_\{a\}u\_\{a\}\. The profileqa=pa​∥ua−u¯∥2q\_\{a\}=p\_\{a\}\\lVert u\_\{a\}\-\\bar\{u\}\\rVert^\{2\}weights probability by squared distance from this centre;q\(k\)q\_\{\(k\)\}lists these values in descending order\. Settingλ^k=q\(k\)\\widehat\{\\lambda\}\_\{k\}=q\_\{\(k\)\}gives a prediction of effective dimension without fitted parameters:

N^eff​\(α\)=∑k=1rq\(k\)q\(k\)\+α\.\\widehat\{N\}\_\{\\mathrm\{eff\}\}\(\\alpha\)\\;=\\;\\sum\_\{k=1\}^\{r\}\\frac\{q\_\{\(k\)\}\}\{q\_\{\(k\)\}\+\\alpha\}\.\(3\)Hererris the read\-out rank, and each term contributes between zero and one according to whether its mode is resolved at scaleα\\alpha\. Replacingq\(k\)q\_\{\(k\)\}with the measured eigenvalues gives the identityNeff​\(α\)=∑kλk/\(λk\+α\)N\_\{\\mathrm\{eff\}\}\(\\alpha\)=\\sum\_\{k\}\\lambda\_\{k\}/\(\\lambda\_\{k\}\+\\alpha\)\. An inheritance theorem bounds each leading measured eigenvalue above and below by the corresponding profile value, up to explicit frame and tail constants\. A more robust fractional\-frame form requires only that a fixed fraction of the leading direction\-Gram modes remain above a declared threshold, yielding a rank\-shifted lower bound and a direction\-free tail upper bound \(Methods; Supplementary Note\)\. Output\-Fisher eigenvalues fall approximately inversely with rank,λi∼i−β\\lambda\_\{i\}\\sim i^\{\-\\beta\}withβ≈1\\beta\\approx 1, over the rank\-22–8080core window from 70M to 6\.9B parameters and across families and vocabularies \(Supplementary Note\)\. In the matched external\-family panel, rank\-88–3232exponents remain of order one across all nine models from 125M to 1\.5B parameters \(Supplementary Note\), while the fitted exponent tracks that of the model’s own weighted token profile \(median deviation0\.040\.04,r=0\.91r=0\.91; Supplementary Note\)[36](https://arxiv.org/html/2609.11063#bib.bib36),[37](https://arxiv.org/html/2609.11063#bib.bib37)\. In synthetic controls, weakly aligned token directions inherit the profile exponent, whereas strongly aligned low\-rank token directions decouple from it\. On the measured battery, high\-probability outcomes require singleton or near\-singleton resolution, whereas the remaining probability mass forms interchangeable clusters with exact information budgets \(Supplementary Note\)\.

The weighted profile also predicts held\-out spectra without an eigendecomposition\. On the held\-out battery in all five external families, the profile prediction reproduces the spectral profile and effective\-dimension curve and outperforms a same\-trace flat spectrum and its effective\-dimension curve, as well as a rank\-shuffled profile, in every family\. It was fixed before exact spectra were revealed for a disjoint 64\-context battery in each of nine models from five external families—BLOOM, GPT\-Neo, Mamba, Qwen2 and RWKV\. Over ranks 8–32 it predicts the held\-out eigenvalues with family\-median errors of0\.06170\.0617–0\.08840\.0884decades, typical multiplicative deviations of1\.151\.15–1\.231\.23\-fold, and spectral\-exponent errors of0\.06230\.0623–0\.08560\.0856\. Applying the resolution transform directly to the same fixed profile predicts the complete normalisedNeffN\_\{\\mathrm\{eff\}\}curve with family\-median RMSEs of0\.02700\.0270–0\.05210\.0521\(Fig\.[3](https://arxiv.org/html/2609.11063#Sx1.F3)a,b; Supplementary Tableand Supplementary Fig\.\)\. On matched controls, spectral error is4\.894\.89–9\.229\.22\-fold lower than for a same\-trace flat spectrum and13\.5513\.55–25\.6325\.63\-fold lower than after shuffling profile ranks; effective\-dimension error is6\.446\.44–7\.887\.88\-fold lower than for the effective\-dimension curve induced by the same\-trace flat spectrum\. Exact inheritance certificates apply in98\.498\.4–100%100\\%of external\-family cells \(Methods; Supplementary Note\)\.

Probability concentration and read\-out geometry make distinct contributions to the spectrum\. In a matched comparison of 14 models from seven families on 192 fresh contexts, a probability\-only profile predicts effective dimension more accurately than the weighted profile in every family \(Fig\.[3](https://arxiv.org/html/2609.11063#Sx1.F3)c\)\. The weighted profile, which includes the learned read\-out directions, predicts the spectral exponent more accurately in every family \(Fig\.[3](https://arxiv.org/html/2609.11063#Sx1.F3)d\)\. Thus the concentration of probability captures the number of resolved modes, while read\-out structure improves prediction of how their strengths decay\. This comparison uses a common outcome representation; the native\-vocabulary prediction in panels a and b provides the complementary across\-vocabulary test \(Methods; Supplementary Note\)\.

Across the external families, the token profile alone predicts effective dimension without fitted parameters\. In a separate finite\-size confirmation, a bounded effective\-dimension curve fitted at four anchor dampings was evaluated on 240 interleaved held\-out values and outperformed an equal\-parameter unbroken power curve on every one of 80 curves \(Extended Data Fig\.[3](https://arxiv.org/html/2609.11063#Sx9.F3)c,d\)\. Median held\-out error was0\.004930\.00493for the bounded curve and0\.08140\.0814for the power curve\. The fixed\-relative\-damping conjugate\-gradient bound is documented in Supplementary Note\.

a![Refer to caption](https://arxiv.org/html/2609.11063v1/fig3a_profile_spectrum.png)b![Refer to caption](https://arxiv.org/html/2609.11063v1/fig3c_profile_response.png)

cd

Figure 3:Language statistics shape the spectrum and its effective dimension\.a, Profile\-predicted against measured spectral exponent over ranks 8–32 for all 576 held\-out model–context cells \(nine models from five external families, 64 contexts per model\)\. Small circles show every cell and the grey line is exact prediction\. Colours identify models throughout the figure\. The Pearson and Spearman correlations are 0\.968 and 0\.955; family\-median absolute exponent errors are0\.06230\.0623–0\.08560\.0856\. On a matched 16\-context\-per\-model control battery, spectral errors are4\.894\.89–9\.229\.22\-fold lower than for a same\-trace flat spectrum and13\.5513\.55–25\.6325\.63\-fold lower than after shuffling profile ranks\.b, Prediction of effective dimension without fitted parameters, the number of modes resolved at a given damping, from token statistics\. Small circles show all 5,760 model–context–resolution values from the same 576 cells, coloured by model; the grey diagonal is exact prediction\. No measured eigenvalue or fitted parameter enters the prediction\. Family\-median effective\-dimension RMSEs are0\.02700\.0270–0\.05210\.0521,6\.446\.44–7\.887\.88\-fold lower than for the same\-trace flat\-spectrum prediction\.c, Effective\-dimension prediction in a matched 14\-model, seven\-family comparison on 192 fresh contexts per model\. All 2,688 points compare the weighted\-profile and probability\-only prediction errors on the same context, measured as effective\-dimension RMSE across damping levels\. The probability\-only profileλ^k=\(r/V\)​p\(k\)\\widehat\{\\lambda\}\_\{k\}=\(r/V\)p\_\{\(k\)\}has lower family\-median error in all seven families\.d, All 672 paired spectral\-exponent errors on the 48\-context exact\-spectrum subset per model; the weighted profile has lower family\-median error in all seven families\. Both panels use logarithmic axes, a common 27,450\-outcome representation and the same standardised read\-out; the grey diagonal marks equal error\. Inference gives equal weight to families\. Exact one\-sided family sign\-flipP=1/128P=1/128for each contrast \(Supplementary Note\)\.
#### Corpus n\-gram statistics predict acquisition, and evidence depth shifts its timing

Corpusnn\-gram margins separately predict acquisition timing \(Fig\.[4](https://arxiv.org/html/2609.11063#Sx1.F4)\)\. We measure acquisition time on a logarithmic scale,τ=log2⁡t\\tau=\\log\_\{2\}t, wherettis the first step after which the designated answer’s log\-probability advantage over its paired alternative remains above a fixed threshold\.

Measured before training at unigram, bigram and trigram levels, these margins predict, without refitting, the complete trajectories of fresh facts disjoint from calibration at all three tested model sizes: trajectoryR2=0\.775R^\{2\}=0\.775–0\.7920\.792, acquisition\-status agreement0\.7920\.792–0\.8540\.854, and continuous timing error of0\.770\.77–0\.960\.96log2\\log\_\{2\}training\-step units, outperforming constant acquisition\-time and label\-permutation baselines \(Fig\.[4](https://arxiv.org/html/2609.11063#Sx1.F4)a\)\. Evidence depth causally delays acquisition\. In a controlled synthetic language in which otherwise identical facts are randomly assigned to shallow or deep evidence constructions with paired initial weights and batch streams, no statistic below the assigned depth distinguishes the two answers; the complete deciding margin appears at and above that depth\. Across the analysed 10th–20th percentiles, acquisition\-time quantiles under deep evidence were approximately4\.34\.3times the corresponding quantiles under shallow evidence, a near\-uniform shift of2\.102\.10inlog2\\log\_\{2\}training steps\. This shift was stable under leave\-one\-seed\-out analysis \(Extended Data Fig\.[4](https://arxiv.org/html/2609.11063#Sx9.F4)a; Supplementary Note\)\. The paired randomisation identifies the causal effect\. Depth therefore delays persistent preference for the designated answer, but not the earlier onset of unsigned movement along the same target–alternative margin direction \(Supplementary Note\)\. Conditional analyses of acquisition\-time variation and of smooth geometric motion are given in Supplementary Notesand, respectively\. The causal direction generalises beyond that construction\. In a fully crossed randomised study of GPT\-2\-, GPT\-NeoX\- and Llama\-style decoders and two independent controlled languages, placing otherwise matched evidence in the deeper construction delayed persistent acquisition in every cell by0\.810\.81–1\.511\.51log2\\log\_\{2\}training\-step units, equivalent to1\.751\.75–2\.862\.86\-fold more training steps\. All six 95% intervals excluded zero, whereas all 12 fixed\-treatment and label\-randomisation control intervals contained zero \(Extended Data Fig\.[4](https://arxiv.org/html/2609.11063#Sx9.F4)b,c; Supplementary Note\)\. Evidence depth therefore delays acquisition across architectures; evidence construction modulates the magnitude on average\.

The cumulative number of acquired facts is shown across Pythia 70M, 160M and 410M against both training\-step and held\-out\-loss coordinates \(Fig\.[4](https://arxiv.org/html/2609.11063#Sx1.F4)b,c\)\. In the quantified 70M\-to\-160M comparison, aligning the cumulative curves by held\-out loss gives a median absolute error of 2\.1 facts out of 300, about three times lower than alignment by training step\.

The n\-gram predictor was independently applied without refitting to the fresh 300\-fact battery across 9 seeds and 20 checkpoints at 70M, 160M and 410M \(Supplementary Note\)\.

a![Refer to caption](https://arxiv.org/html/2609.11063v1/edacq_a.png)

bc

Figure 4:Corpus statistics predict acquisition across model sizes\.a, Ex\-ante corpus n\-gram margins predict subsequent margins without refitting for fresh facts disjoint from calibration at Pythia 70M, 160M and 410M\. Points are all 9,840 fact–checkpoint means over 15 checkpoints: 5,340 from 178 defined facts at 70M and 160M \(ten runs\) and 4,500 from 300 facts at 410M \(nine runs\); the shared model\-colour key below the panels distinguishes Pythia 70M, 160M and 410M; the grey diagonal is exact prediction\. TrajectoryR2R^\{2\}is0\.7900\.790,0\.7920\.792and0\.7750\.775, persistent\-status agreement is0\.7920\.792,0\.8540\.854and0\.8070\.807, and continuous timing error is0\.770\.77,0\.950\.95and0\.960\.96log2\\log\_\{2\}training\-step units, respectively\. All three timing errors outperform their constant acquisition\-time and label\-permutation baselines\.b, Cumulative acquisition curves for the same 300 facts against training step at 70M, 160M and 410M\. Thin curves are individual seeds and heavy curves with circular markers are medians over nine seeds for each model size, using the shared model colours\.c, The same curves against held\-out loss\. In the descriptive 70M\-to\-160M comparison, alignment by held\-out loss has a median absolute error of 2\.1 facts\.
#### The geometry determines minimum\-disturbance interventions and their relative cost

The same geometry determines how to change behaviour at minimum cost, and quantifies in advance the penalty for ignoring it\. LetA=G\+α​R≻0A=G\+\\alpha R\\succ 0, with dampingα\\alphaand reference metricRR\. For objective gradientq≠0q\\neq 0and target changeq⊤​δ=bq^\{\\top\}\\delta=b, the unique minimum\-disturbance step isδ⋆=b​A−1​q/\(q⊤​A−1​q\)\\delta^\{\\star\}=bA^\{\-1\}q/\(q^\{\\top\}A^\{\-1\}q\), attaining the minimum regularised costCA,min​\(b\)=12​δ⋆⁣⊤​A​δ⋆=b2/\(2​q⊤​A−1​q\)C\_\{A,\\min\}\(b\)=\\tfrac\{1\}\{2\}\\delta^\{\\star\\top\}A\\delta^\{\\star\}=b^\{2\}/\(2q^\{\\top\}A^\{\-1\}q\)\. Every other feasible step incurs an excess regularised cost equal to half its squaredAA\-distance from this optimum \(Supplementary Note\)\. To predict the realised output disturbance at finite damping, define theGG\-cost per unit squared objective change asΠG​\(u\)=\(u⊤​G​u\)/\(q⊤​u\)2\\Pi\_\{G\}\(u\)=\(u^\{\\top\}Gu\)/\(q^\{\\top\}u\)^\{2\}; smaller values mean less predicted local output change for the same target effect\. For the coordinate\-Euclidean comparator used here \(R=IR=I\), the prediction without fitted parameters is

Rpred​\(q\)=ΠG​\(q\)ΠG​\(A−1​q\)\.R\_\{\\mathrm\{pred\}\}\(q\)=\\frac\{\\Pi\_\{G\}\(q\)\}\{\\Pi\_\{G\}\(A^\{\-1\}q\)\}\.\(4\)This finite\-damping cost ratio was fixed before intervention; the idealised ratio in the full regularised metric is analysed in Supplementary Note\.

The cost ratio is predicted without fitted parameters\. Across 11 models from six architecture families, three steering objectives and three relative layer depths, the finite\-damping ratio computed before intervention tracks 3,515 usable matched\-response measurements from 1,188 cells \(98\.6% coverage; median measured\-to\-predicted ratio0\.9670\.967, 95% interval0\.9570\.957–0\.9750\.975; log–log slope0\.9820\.982, 95% interval0\.9660\.966–0\.9970\.997; Fig\.[5](https://arxiv.org/html/2609.11063#Sx1.F5)a\)\. The advantage persists along finite intervention paths\. Recomputing both Fisher and Euclidean directions after each accepted step reaches four matched objective changes across 216 prompts and 18 model–objective cells\. The Fisher paths accumulate about1111–127127times less local KL than their Euclidean counterparts, with positive paired intervals in every cell and target \(Fig\.[5](https://arxiv.org/html/2609.11063#Sx1.F5)b; Supplementary Note\)\.

The metric also supports interventions that can be reused across prompts\. A shared activation update is learned from four donor prompts per objective, using the Fisher metric averaged over four reference prompts to limit disturbance\. The resulting update is then applied to eight unseen target prompts without computing target gradients\. Across BLOOM\-560M, Pythia\-410M, Pythia\-1\.4B and Qwen2\-1\.5B, it transfers a positive mean truth\-preference or anti\-sycophancy change in all 24 model–objective–target settings \(Fig\.[5](https://arxiv.org/html/2609.11063#Sx1.F5)d\)\. At matched donor changes, the reference metric reduces sequence KL on reference continuations by about33–66times relative to Euclidean control and by at least22times relative to a donor\-only Fisher metric \(Fig\.[5](https://arxiv.org/html/2609.11063#Sx1.F5)c\)\. A joint solve also transfers both objectives with lower reference\-sequence cost in all four models \(Supplementary Note\)\.

Prompt\-specific control extends to instruction\-tuned models: at equal aggregate mean objective change, natural\-gradient frontiers have88–7474times lower off\-target divergence for anti\-sycophancy,3030–190190times for capital\-city truth preferences and44–550550times for style \(Supplementary Note; Extended Data Fig\.[5](https://arxiv.org/html/2609.11063#Sx9.F5)a\)\. Several objectives combine through their weighted gradient sum, with lower off\-target cost across the measured composition frontier \(Extended Data Fig\.[5](https://arxiv.org/html/2609.11063#Sx9.F5)b\)\.

ab

cd

Figure 5:The geometry predicts intervention cost and supports reusable control\.a, At matched behaviour change, the measured ratio of off\-target output divergence for Euclidean versus damped natural\-gradient updates is compared with the finite\-damping predictionRpredR\_\{\\mathrm\{pred\}\}, which has no fitted parameters and is computed before intervention\. The axes are logarithmic and the grey identity line denotes perfect agreement\. Small circles show all 3,515 usable matched\-response observations \(of 3,564\) from 1,188 model–objective–layer–context cells: 11 models, six families, three objectives, three relative layer depths and 12 contexts per objective, each measured at three intervention magnitudes \(98\.6% coverage; median measured\-to\-predicted ratio0\.9670\.967, 95% interval0\.9570\.957–0\.9750\.975; log–log slope0\.9820\.982, 95% interval0\.9660\.966–0\.9970\.997\)\.b, Finite paths with both methods relinearised after every accepted step\. All 864 points compare Euclidean and Fisher cumulative local KL at matched target changes, from 216 prompts, 16 models and 18 model–objective cells\. The metric sums KL between successive accepted outputs\. The 72 cell–target geometric\-mean cost ratios span11\.511\.5–127\.1127\.1, with every paired 95% interval above one\. Colours identify models in panels a and b\.c, Reusable control on four base models from three families\. All 192 points compare reference\-Fisher sequence KL with Euclidean \(vermilion\) or donor\-only Fisher \(grey\) sequence KL for the same reference continuation and matched donor change, across two objectives and three target changes\. Each setting contributes four reference continuations per comparator\. Reference sequences are teacher\-forced continuations of the four prompts used to construct the reference metric\. Ratios of mean Euclidean/reference\-Fisher sequence KL span2\.902\.90–6\.296\.29\.d, The same updates change unseen target prompts without target gradients\. Each point shows mean sequence KL over four reference continuations and mean target log\-odds change \(nats\) over eight unseen prompts, for one model, objective, donor change and method \(48 points\)\. Green, natural gradient using the reference metric; vermilion, Euclidean\. Grey segments join matched donor changes\. All 24 natural\-gradient mean target changes are positive\. Grey diagonals in panels b and c mark equality\.
#### One geometric correction spans model operations

A single correction, replacing the Euclidean inner product with the pullback metric of the relevant output map, improves every matched endpoint in Table[1](https://arxiv.org/html/2609.11063#Sx1.T1)\. Its prediction without fitted parameters generalises across the 11\-model, six\-family panel in Fig\.[5](https://arxiv.org/html/2609.11063#Sx1.F5)a, and the reference metric supports reusable control across unseen prompts\. Applied to a single fact, the steer becomes a training\-free editor: on 30 held\-out CounterFact records \(GPT\-2\), with each edit solved once and reused through a shared prompt\-selection rule, the pullback\-metric edit had10\.5710\.57\-fold lower geometric\-mean other\-token divergence than its matched Euclidean counterpart \(95% interval8\.528\.52–13\.3713\.37\); both operators attained the fixed\+5\+5log\-odds calibration shift on all 30 edits \(Extended Data Fig\.[5](https://arxiv.org/html/2609.11063#Sx9.F5)c\)\. Separate GPT\-2\-XL evaluations cover 100 edits under standard metrics and 300 under a stricter probability\-margin criterion \(Supplementary Table; Supplementary Note\)\. For interpretability, Fisher\-weighted output sensitivityv⊤​G​vv^\{\\top\}Gvmeasures the output change caused by a feature direction; activation magnitude measures how strongly it activates\. Across 303 zero ablations of sparse\-autoencoder \(SAE\) features at Pythia\-410M, each setting one active feature to zero[14](https://arxiv.org/html/2609.11063#as1_bib.bib14),[39](https://arxiv.org/html/2609.11063#bib.bib39),[40](https://arxiv.org/html/2609.11063#bib.bib40), the intervention\-specific Fisher cost rank\-correlated with the exact divergence atρ=0\.997\\rho=0\.997, against0\.8960\.896for activation magnitude\. The objective\-effect\-to\-Fisher\-cost ratio, the squared linear objective effect per unit Fisher cost, predicted the selected\-token probability\-change fraction at0\.9350\.935against0\.5060\.506for attribution patching[41](https://arxiv.org/html/2609.11063#bib.bib41), which remains the better predictor of raw effect magnitude \(Extended Data Fig\.[6](https://arxiv.org/html/2609.11063#Sx9.F6); Supplementary Note\)\.

The correction carries over to training\. The code\-Gram metric, the second moment of sparse feature activations, increases Hungarian\-matched mean absolute decoder–ground\-truth cosine from0\.1670\.167to0\.4800\.480\(\+0\.313\+0\.313\) when used to precondition a sparse\-autoencoder decoder at 410M with planted\-signal/background\-RMS ratioβ=0\.5\\beta=0\.5\(Supplementary Note\)\. At the same scale, natural\-gradient low\-rank adaptation produces6060–250250\-fold less off\-target change than Adam at matched on\-target change\. In a separate held\-out comparison at 70M, it improves the preservation\-KL frontier1\.571\.57\-fold relative to tuned AdamW with explicit KL regularisation \(Extended Data Fig\.[7](https://arxiv.org/html/2609.11063#Sx9.F7); Supplementary Note\)\.

The size of the available correction is itself set by the geometry\. The advantage is largest at early\-to\-mid injection layers, where the metric pulled back through the rest of the network is most anisotropic, and shrinks toward the near\-linear read\-out \(Extended Data Fig\.[8](https://arxiv.org/html/2609.11063#Sx9.F8)a\)\. It vanishes in the isotropic limit, where the natural\-gradient and Euclidean steps coincide\. Across the same 11\-model panel, every measured\-to\-predicted path decreases over the three declared intervention fractions; the model\-balanced geometric means are0\.9180\.918,0\.8600\.860and0\.7550\.755\(Extended Data Fig\.[8](https://arxiv.org/html/2609.11063#Sx9.F8)b\)\. This quantifies attenuation of the local prediction as intervention magnitude grows; Supplementary Noteanalyses the finite\-magnitude correction to cubic order\. Output sensitivity, residual cross\-model sharing and answer\-associated probability motion concentrate in the same leading spectral modes \(Extended Data Fig\.[9](https://arxiv.org/html/2609.11063#Sx9.F9)\), and cyclic concepts trace nearly closed rotations, whereas analogy relations instead share a probability\-displacement direction \(Extended Data Fig\.[10](https://arxiv.org/html/2609.11063#Sx9.F10)\)\.

Table 1:One correction spans the evaluated operations\.The result column reports the strongest reported matched comparison for each operation; the final column gives its evidence base\. Intervals are 95% bootstrap intervals\. AUC, area under the curve; KL, Kullback–Leibler; LoRA, low\-rank adaptation\. Sources and protocols are given in Methods\.

### Discussion

Next\-token training induces a canonical output geometry\. Predictive behaviour identifies the language\-determined read\-out subspace up to transformations that leave outputs unchanged; this predicts convergence of independently trained models estimating the same language\. Controlled language assignment supports this origin: changing the law changes the learned geometry, while different architectures fitted to the same law recover nearly identical geometry\. Activation\-space convergence measures, by contrast, must tolerate arbitrary reparameterisations\. Among those evaluated here, agreement is higher in output geometry and reparameterisation\-invariant; it remains high after permutation\-null calibration for scale\-dependent baselines[19](https://arxiv.org/html/2609.11063#as1_bib.bib19)\.

Their measured agreement is reconstructed exactly from six variance and covariance terms, and its shared component carries the measured semantic alignment\. A semantic\-category probe transfers across models, and the comparison extends to human predictions\. Models with lower prediction error agree more closely with human word choices across families\. On independent data, agreement also increases with scale and training, while a calibration learned only from model laws improves predictions across the full coarse distribution of human word choices\. Direct probability\-space comparisons on a separate human corpus also recover leading directional structure\. These completion\-distribution comparisons complement work on the finite resolution of human cloze estimates and the semantic distinctions captured by model probabilities[43](https://arxiv.org/html/2609.11063#bib.bib43)\.

Token statistics predict the spectrum and its effective dimension, the number of modes resolved at a given damping, without fitted parameters\. The matched mechanism comparison separates two contributions: probability concentration captures effective dimension, while the weighted read\-out profile improves prediction of spectral shape across all seven families\. At fixed relative damping and solver tolerance, the damped natural\-gradient solve has a proved worst\-case conjugate\-gradient iteration bound independent of model width; atc=10−2c=10^\{\-2\}and Euclidean relative residual10−610^\{\-6\}, the exact\-arithmetic bound is 85 iterations\.

Separately, corpus n\-gram evidence measured before training predicts individual\-fact acquisition with median absolute errors of0\.770\.77–0\.960\.96on thelog2\\log\_\{2\}training\-step scale at all three sizes\. In the controlled languages, randomised evidence depth causally delays acquisition\.

The same geometry prescribes minimum\-disturbance control whose advantage is predicted in advance by a measured anisotropy ratio\. This advantage persists along relinearised finite paths and extends to updates reused across prompts\. Its recurrence across comparison, steering, editing, feature analysis and training shows that the correction is not confined to a single evaluated operation\. Supplementary Tablemaps these claims to their analytical, observational and interventional evidence\.

Alignment systems face two separate questions: what behaviour should change, and how to make that change without unnecessarily disturbing the model elsewhere\. Human preferences, constitutions, reward or cost models and other forms of oversight must answer the first question\. The output geometry addresses the second\. Given a preference covector and a desired first\-order change, the regularised Fisher pullback gives the activation change with minimum local output disturbance\. It can therefore turn an externally supplied preference into an intervention, complementing existing representation\-level methods for inference\-time alignment[18](https://arxiv.org/html/2609.11063#bib.bib18),[13](https://arxiv.org/html/2609.11063#as1_bib.bib13),[16](https://arxiv.org/html/2609.11063#bib.bib16)\. On instruction\-tuned models, the construction improves the aggregate trade\-off between objective change and off\-target change for anti\-sycophancy, capital\-city truth preferences and writing style\. Averaging the metric over a reference prompt set extends this construction to reusable control\. Updates learned from four donor prompts transfer to eight unseen prompts per objective in four base models, while reducing reference\-sequence change by about three\- to sixfold relative to Euclidean control at matched donor effects\. The solve is performed on donors and reference prompts; the resulting activation update is applied to each target without computing its gradient\. Several objectives also combine in one solve, allowing their balance to be set through externally supplied weights\.

The same idea may be useful during post\-training\. Rather than keeping an update close to the original model in parameter or activation space, the reference\-set construction could be applied in parameter space to limit changes to the model’s predictions on protected prompts\. This would give preference fine\-tuning and model customisation a behaviour\-level trust region\. The held\-out LoRA comparison provides an initial local\-preservation result in this direction, although it does not establish safety\-specific retention or broader capability preservation\.

Mechanistic interpretability has a related measurement problem\. A feature can activate strongly or admit a clear verbal description without playing an important role in the model’s output\. Conversely, an intervention can change a chosen target only by disrupting many other predictions\. Fisher\-weighted output sensitivity measures the predicted local output consequence of a specified feature intervention, while the objective\-effect\-to\-Fisher\-cost ratio measures its squared linear effect on a chosen objective per unit Fisher cost\. In the sparse\-feature experiments, these quantities accurately predicted both the output cost of removing a feature and the selectivity of its effect\. They could therefore help prioritise candidate features and circuits in audits of sycophancy, deception, refusal and other safety\-relevant behaviours, and help choose interventions with a limited behavioural footprint\. End\-to\-end sparse dictionary learning already prioritises functional importance by minimising the KL divergence between original and reconstructed model outputs[44](https://arxiv.org/html/2609.11063#bib.bib44)\. The present sensitivity measurements motivate investigating local geometric weighting for sparse dictionaries and replacement models, so that limited explanatory capacity is spent on distinctions that matter to the model’s behaviour\. These uses would complement semantic labelling and causal attribution: the metric measures the functional consequence of a proposed feature, but does not by itself explain what that feature represents\.

The comparison results suggest a further use in model auditing\. Because relational output geometry does not require corresponding hidden coordinates, architectures or tokenizers, a fixed battery of prompts and anchors could be used to compare successive checkpoints, post\-training variants and independently developed models\. The consensus–residual decomposition could then separate widely shared structure from model\-specific deviations\. The acquisition results could likewise help place costly evaluations near the predicted learning transitions of target facts or associations, while routed geometric edits could provide modular factual corrections with a controlled output footprint\. Together, these applications suggest a role for output geometry throughout the model lifecycle: anticipating behavioural change during training, auditing it across models, interpreting its internal causes and controlling it when intervention is needed\.

The same information\-geometric principles have a direct analogue in quantum computing: on pure quantum states, the Fubini–Study metric is, up to a constant, the Fisher–Rao metric naturally induced by optimal quantum measurements[45](https://arxiv.org/html/2609.11063#bib.bib45)\. Its pullback underlies quantum natural\-gradient optimisation and variational dynamics, including in the author’s recent work[46](https://arxiv.org/html/2609.11063#bib.bib46)\.

Across the models and operations tested here, predictive behaviour identifies a canonical output geometry shared across models and related to human predictions\. Token statistics shape its resolved structure, corpus n\-gram evidence predicts when individual facts are acquired, and the metric supplies calibrated control that can be reused across prompts\.

### Methods

#### Models, datasets and geometric definitions

###### Models and data\.

Pretrained\-model analyses use public checkpoints; causal language\-assignment and acquisition experiments train controlled synthetic systems\. The language\-model ladder is the Pythia suite[24](https://arxiv.org/html/2609.11063#as1_bib.bib24)\(70M–6\.9B\)\. Alignment\-objective experiments use Qwen2\-1\.5B\-Instruct[26](https://arxiv.org/html/2609.11063#as1_bib.bib26), replicated on Qwen1\.5\-1\.8B\-Chat[27](https://arxiv.org/html/2609.11063#as1_bib.bib27), and knowledge editing uses GPT\-2 and GPT\-2\-XL[25](https://arxiv.org/html/2609.11063#as1_bib.bib25)\. The spectral analysis of the steering dimension additionally spans GPT\-2, GPT\-2\-XL, Qwen2\-1\.5B and BLOOM\-1b1[32](https://arxiv.org/html/2609.11063#as1_bib.bib32), whose vocabularies reach∼\\sim250k tokens\. The cross\-model and dense\-response batteries additionally include GPT\-Neo, Mistral\-7B, Mamba, RWKV and StarCoder2 checkpoints[28](https://arxiv.org/html/2609.11063#as1_bib.bib28),[29](https://arxiv.org/html/2609.11063#as1_bib.bib29),[30](https://arxiv.org/html/2609.11063#as1_bib.bib30),[31](https://arxiv.org/html/2609.11063#as1_bib.bib31),[33](https://arxiv.org/html/2609.11063#as1_bib.bib33)\. Unless stated otherwise, interventions act on the residual stream at a middle layer,L=round⁡\(0\.45×depth\)L=\\mathrm\{round\}\(0\.45\\times\\text\{depth\}\)\(Qwen2\-1\.5B\-Instruct uses layer 14\); the feature\-importance analyses use layer 12 of Pythia\-410M\.

Layer\-dependence is reported in Extended Data\. Objectives and concepts are: morphological and sentiment \(SST\-2[57](https://arxiv.org/html/2609.11063#bib.bib57)\) contrasts; factual capital\-city true/false preference pairs; a cyclic registry \(weekdays, months\); sycophancy, truthfulness \(TruthfulQA[58](https://arxiv.org/html/2609.11063#bib.bib58)and capital\-city true/false pairs\) and writing style in experiments on instruction\-tuned models; and CounterFact[9](https://arxiv.org/html/2609.11063#as1_bib.bib9)for knowledge editing\. The feature\-importance analyses use a trained Top\-kksparse autoencoder[14](https://arxiv.org/html/2609.11063#as1_bib.bib14)\(tim\-lawson/sae\-pythia\-410m\-deduped\-x64\-k32\-layers\-12; residual\-stream Top\-kk,k=32k\{=\}32, layer 12\), which retains thekklargest feature activations per token\.

###### Output metric and pullback\.

Write the next\-token distribution asp=softmax⁡\(WU​hL\+bU\)p=\\mathrm\{softmax\}\(W\_\{U\}h\_\{L\}\+b\_\{U\}\), wherehL∈ℝdh\_\{L\}\\in\\mathbb\{R\}^\{d\}is the final pre\-unembedding activation andWU∈ℝ\|V\|×dW\_\{U\}\\in\\mathbb\{R\}^\{\|V\|\\times d\}is the unembedding over a vocabulary of size\|V\|\|V\|\. Denote the rows ofWUW\_\{U\}bywaw\_\{a\}\. Pulling the Fisher–Rao metric ofppback tohLh\_\{L\}through the softmax gives the*output Fisher*H=Covp​\(w\)=WU⊤​\(diag⁡\(p\)−p​p⊤\)​WUH=\\mathrm\{Cov\}\_\{p\}\(w\)=W\_\{U\}^\{\\top\}\(\\mathrm\{diag\}\(p\)\-pp^\{\\top\}\)\\,W\_\{U\}\. For an intervention on an activationhhat layerℓ\\ell, letJ=∂hL/∂hJ=\\partial h\_\{L\}/\\partial hbe the \(fixed\-context\) Jacobian of the network fromℓ\\ellto the read\-out; the \(possibly degenerate\) pullback Fisher metric isG=J⊤​H​JG=J^\{\\top\}HJ, reducing toHHat the final layer\.GGis the second\-order model of output change,KL\(p∥pδ​h\)=12δh⊤Gδh\+O\(∥δh∥3\)\\mathrm\{KL\}\(p\\\|p\_\{\\delta h\}\)=\\tfrac\{1\}\{2\}\\delta h^\{\\top\}G\\,\\delta h\+O\(\\\|\\delta h\\\|^\{3\}\)\.

For an objective covectorq=∇hϕq=\\nabla\_\{h\}\\phi, the regularised natural\-gradient step isδ​h∝\(G\+α​R\)−1​q\\delta h\\propto\(G\+\\alpha R\)^\{\-1\}q, with positive reference metricRR\.

Underh↦A​hh\\mapsto Ah,G↦A−⁣⊤​G​A−1G\\mapsto A^\{\-\\top\}GA^\{\-1\}\. The same damped problem is coordinate\-covariant whenR↦A−⁣⊤​R​A−1R\\mapsto A^\{\-\\top\}RA^\{\-1\}\. Native\-coordinate experiments useR=IR=I; canonicity, coordinate covariance and the numerical coordinate test are detailed in Supplementary Note\.

#### Behavioural identification and causal language assignment

###### Profiled\-loss experiment\.

The identification test uses a rank\-32 resolved reference family estimated from Pythia\-6\.9B on 500 calibration contexts\. For Pythia 70M, 160M, 410M and 1\.4B, candidate subspaces follow paths towards each model’s read\-out and random controls preserving the same principal angles\. At five positions along each path, the intercept and context coordinates are jointly optimised on each of two disjoint 100\-context calibration folds\. Each intercept is then fixed and only coordinates are optimised on 100 held\-out contexts, disjoint from reference construction\. The mean of the two held\-out profiled KL losses is compared with squared chordal distance; Figure[1](https://arxiv.org/html/2609.11063#Sx1.F1)a displays both fold estimates\. Profiling uses float64 and explicit stationarity tolerances \(Supplementary Note\)\.

###### Causal language assignment\.

Three model architectures \(transformer, gated recurrence and diagonal recurrence\), each at two capacities, were trained on paired synthetic languages with 64 contexts and 64 outcomes\. Training schedules were selected using two separate pilot language pairs and conditional prediction risk, then applied to all eight test pairs\. The risk requirement was excess cross\-entropy belowΔ2/32\\Delta^\{2\}/32, whereΔ2\\Delta^\{2\}is the mean squared separation between the two known language geometries\. Geometry was represented by scaled root\-probability chord distances among contexts\. The same architecture was compared across laws, and different architectures were compared within a law at the same capacity\. Both differences were normalised byΔ2\\Delta^\{2\}\. Inference resampled whole language pairs, retaining all architecture and capacity comparisons within each pair \(Supplementary Note\)\.

#### Cross\-model geometry and human completions

###### Relational comparison and controls\.

For a common context battery, each model defines a matrix of Fisher–Rao distancesdi​j=2​arccos​∑api​\(a\)​pj​\(a\)d\_\{ij\}=2\\arccos\\sum\_\{a\}\\sqrt\{p\_\{i\}\(a\)\\,p\_\{j\}\(a\)\}between its next\-token distributions\. Two models are compared by the Spearman correlation of the upper triangles of their matrices \(rank agreement\)\. This is an instance of representational similarity analysis[60](https://arxiv.org/html/2609.11063#bib.bib60)that requires no shared vocabulary or architecture\. Activation geometries are compared identically using cosine distances of mid\- or last\-layer states\. Three batteries are used: 200 natural sentence prefixes drawn from the held\-out WikiText test split[16](https://arxiv.org/html/2609.11063#as1_bib.bib16), 160 templated contexts spanning eight topics, and 120 factual\-relation prompts whose next token is the answer \(the natural battery is primary\)\.

The stability analysis uses the per\-context Hellinger bound and the excess\-risk rank\-convergence theorem \(Supplementary Note\)\. It is carried out on the Bhattacharyya affinity matricesCi​j=∑api​\(a\)​pj​\(a\)C\_\{ij\}=\\sum\_\{a\}\\sqrt\{p\_\{i\}\(a\)\\,p\_\{j\}\(a\)\}, whose entries determine the corresponding distance\-matrix ranks\. The realised\-perturbation certificate uses ten same\-tokenizer pairs so that per\-context divergences are defined on a shared outcome space, with vocabulary positions beyond the common range merged into a single bucket\. The population excess risks to the unknown training conditional law are not estimated\. The common\-byte comparison below provides a cross\-tokenizer extension; native\-vocabulary agreement is reported separately \(Supplementary Note\)\.

Surface and predictability controls partial out, respectively, a characternn\-gram geometry of the contexts and the geometry of a corpusnn\-gram predictor with backoff; the tokenizer control compares same\-tokenizer with different\-tokenizer model pairs; the corpus control contrasts reference models that share a target pair’s training corpus with reference models that do not, at matched scale and tokenizer\.

In the reparameterisation experiment, each model’s hidden states are transformed by an independent random invertible map of condition numberκ\\kappabefore activation\-based similarity is computed\. Output\-geometry agreement is unchanged because the model’s input–output function is fixed, verified numerically\.

Reference mediation conditions the agreement of every target\-model pair on a reference model’s geometry, sweeping references from 70M to 7B parameters\. Unembedding geometries are cosine similarities of unembedding rows over a vocabulary of words that are single tokens in every model, after subtracting the vocabulary mean[62](https://arxiv.org/html/2609.11063#bib.bib62); covariance\-based quantities \(the output Fisher and its eigenvectors\) are invariant to that correction and are used uncorrected\.

###### Common\-byte comparison\.

Each tokenizer’s next\-token law is mapped to the distribution of the first byte of the remaining text, with byte classes providing a second, coarser common outcome space\. Byte\-level round trips and non\-decreasing Bhattacharyya affinity under both coarse\-grainings are checked over all 28 model pairs \(Supplementary Note\)\.

###### Prediction of cross\-model agreement\.

The six\-scalar accounting uses a fixed leave\-pair\-out reference over the context battery; all variances and covariances are population \(ddof=0=0\) statistics over a declared pair population, strict upper triangle for the relational accounting and all ordered pairs including the diagonal where the theory’s independent\-draw convention requires it\. Risk floors and chord\-based checks apply only where the reference is a genuine conditional law on the same outcome experiment\. The cross\-fitted prediction pools fluctuation energy and alignment on a training half of the contexts, estimates the shared\-fluctuation fraction for each target pair from the other pairs, and predicts the held\-out\-half agreement with no test\-half error statistic entering any estimate\. Predictions cover six ordered pairs of four independently trained runs in both directions at ten public checkpoints \(120 predictions\)\. The estimator is undefined where its variance denominator is non\-positive, which occurs in 24 early\-checkpoint cells \(steps≤64\\leq 64\), leaving 96 defined predictions \(Supplementary Note\)\.

###### Semantic geometry and transfer\.

The consensus geometry is the mean of the rank\-transformed distance matrices across models; a model’s residual is its rank\-transformed matrix with the consensus partialled out\. The external semantic geometry is the mean\-centred cosine geometry of the fixedall\-MiniLM\-L6\-v2Sentence\-BERT encoder[3](https://arxiv.org/html/2609.11063#as1_bib.bib3)over the same contexts\. Transfer probes represent each prompt by its Fisher–Rao distances to a fixed anchor set \(relative representations[11](https://arxiv.org/html/2609.11063#bib.bib11)\),zz\-scored per model; a multinomial logistic probe is trained on one model’s representation and evaluated on held\-out prompts through another’s, with five\-fold cross\-validation\.

The cross\-modal control compares the mean cosine\-distance matrix of last\-token, last\-hidden\-state concept representations from Pythia\-1\.4B and GPT\-2 Large with a concept dissimilarity matrix formed by averaging DINOv2[17](https://arxiv.org/html/2609.11063#as1_bib.bib17)features over CIFAR\-100[18](https://arxiv.org/html/2609.11063#as1_bib.bib18)images\. This is an activation\-space representational analysis, separate from the output Fisher–Rao analyses\.

###### Eigenvector sharing\.

The eigenstructure analyses band the output\-Fisher spectrum by eigenvalue rank; each band reports the eigenvalue mass, the sharing of eigenvector structure across model pairs measured as the subspace overlap of eigenvector\-induced probability displacements over the common vocabulary, and the fraction of the band’s probability motion on the highest\-probability tokens\. Sharing is always reported against a displacement\-magnitude\-matched null that preserves each word’s displacement magnitude while randomising directions\. Most raw eigenvector overlap reflects similar token\-probability profiles; subtracting the null isolates sharing beyond this displacement\-magnitude effect\.

###### Native human\-completion comparison\.

Human sentence\-final completions were drawn from the norms of Peelle et al\.[20](https://arxiv.org/html/2609.11063#as1_bib.bib20)\. Native sampling used 512 contexts, six Pythia sizes from 70M to 2\.8B, and 32 categorical samples per context at temperature one, without top\-kkor nucleus filtering\. Generation stopped at eight new tokens; a fixed parser extracted the completion through the first punctuation, newline or end\-of\-sequence boundary\. Empty and length\-limited samples were retained\. Each completion was embedded with the fixedall\-MiniLM\-L6\-v2encoder and mapped through a multiscale Gaussian kernel approximation with three bandwidths and 768 random Fourier features\. Model and human sample means were compared by squared feature distance and centred kernel alignment \(CKA\)\. Human participants and model samples were split into independent halves; cross\-pair values were averaged and CKA was corrected using human and model split\-half reliability\. Intervals and log\-parameter slopes used 4,000 paired bootstrap draws over 16 disjoint context folds\.

###### Prediction of human alignment\.

The predictive\-fit comparison used a separate 384\-context subset, the same semantic features, six final Pythia sizes, five Pythia\-160M checkpoints, five OLMo\-2\-1B checkpoints[21](https://arxiv.org/html/2609.11063#as1_bib.bib21), and five external models\. Here each model law was conditional on complete human\-observed completion\-plus\-end\-of\-sequence events, retaining tokenizer collisions\. Conditional root\-probability risk predicted reliability\-corrected human alignment through a linear relation fitted only to the Pythia sizes and trajectory, separately within each context fold\. This relation was applied to OLMo and external models without refitting\.

###### Human probability directions\.

A separate next\-word corpus[22](https://arxiv.org/html/2609.11063#as1_bib.bib22)provided 1,726 contexts in 205 sentences for comparison of leading log\-probability directions across seven families\. Model and human laws were mapped to 64 fixed semantic bins plus a residual outcome\. Whole\-sentence splits separated human reference, model direction estimation, calibration and evaluation; participant splits assessed respondent generalisation \(Supplementary Note\)\.

###### Independent human\-completion replication\.

The DERCo resource[23](https://arxiv.org/html/2609.11063#as1_bib.bib23)supplied word\-prediction responses for five narratives\. Before viewing response values, 128 positions per narrative were selected from stimulus metadata by a fixed hash, giving 640 contexts, 64,000 responses and 499 participant identities\. Models received the preceding ten displayed words\. For six final Pythia sizes and five ordered checkpoints each from Pythia\-160M and OLMo\-2\-1B, 32 native continuations per context were divided between two independently seeded halves\. The first lexical word was embedded with the same fixed semantic encoder and kernel map as above\. Squared distances averaged the four pairings of participant and model\-sample halves, with equal weight for contexts and stories\. The 512 bootstrap draws resampled participants globally, sentences within each fixed story and native sequences; intervals are conditional on the five stories\.

###### Model\-only calibration on independent human data\.

The same contexts were mapped to 64 fixed semantic bins plus a residual outcome\. For each of seven model families, a full 65\-dimensional affine ridge map in centred log\-probability coordinates was trained to predict leave\-tokenizer\-cluster\-out source\-model laws from candidate\-model laws on the earlier corpus\. No human response entered the fit or model selection\. Four source\-only rotations were averaged equally, and the complete recipe was fixed before the DERCo responses were examined\. The primary comparison was the change from the native model law in expected multinomial log score against unsmoothed human counts, weighting contexts equally within stories and the five stories equally\. These outcomes test the complete coarse distribution, not recovery of individual word probabilities\.

#### Spectral measurements and effective\-dimension prediction

###### Spectrum estimation and inheritance\.

Output\-Fisher spectra are computed by exact diagonalisation on a renormalised top\-512512token support\. The target support is the smallest token set containing0\.990\.99of the probability mass\. No evaluated context reaches this target within 512 tokens, and94%94\\%still do not reach it within 1024 tokens; the retained probability mass at 512 tokens averages0\.900\.90\. A full\-vocabulary calculation reproduces the rank\-22–8080core fit \(Supplementary Note\)\.

Power\-law exponentsβ\\betaandR2R^\{2\}are fitted as log–log slopes over ranks22–8080, and high\-sensitivity mode counts are\#\{λi\>cλmax\}\\\#\\\{\\lambda\_\{i\}\>c\\,\\lambda\_\{\\max\}\\\}atc=10−1,10−2,10−3c=10^\{\-1\},10^\{\-2\},10^\{\-3\}\. The empirical inheritance test compares, per context, the sorted eigenvalues ofHHwith the sorted values ofpa​∥wa∥2p\_\{a\}\\lVert w\_\{a\}\\rVert^\{2\}\(fitted on the same rank window\), and the Zipf exponentssof the sorted profilepp; this is the uncentred empirical exponent test\. The theorem\-level analysis uses the exact centred profileqa=pa​∥wa−∑bpb​wb∥2q\_\{a\}=p\_\{a\}\\lVert w\_\{a\}\-\\sum\_\{b\}p\_\{b\}w\_\{b\}\\rVert^\{2\}over the full vocabulary, measures its frame and tail constants, checks the conditional eigenvalue bounds, and reports the realisedλk/q\(k\)\\lambda\_\{k\}/q\_\{\(k\)\}band\. Its profile\-onlykprof∗k^\{\*\}\_\{\\mathrm\{prof\}\}is evaluated as an empirical crossover predictor, separately from the conditional trace\-floor theorem\. Eigenvector inheritance is scored as the alignment between theii\-th eigenvector and the unit read\-out direction of the token with theii\-th largestpa​∥wa∥2p\_\{a\}\\lVert w\_\{a\}\\rVert^\{2\}\. Synthetic read\-out controls and spectral model\-selection/coverage checks are detailed in Supplementary Note; exponent claims concern the declared core windows\.

###### Standardised read\-out representation\.

We use the standardised centred read\-out representationU=orth⁡\(P0​WU\)U=\\operatorname\{orth\}\(P\_\{0\}W\_\{U\}\), an orthonormal basis for the centred read\-out subspace, whereP0=I−\|V\|−1​𝟏𝟏⊤P\_\{0\}=I\-\|V\|^\{\-1\}\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}removes the output\-softmax gauge\. Writinguau\_\{a\}for rowaaofUUandu¯=∑apa​ua\\bar\{u\}=\\sum\_\{a\}p\_\{a\}u\_\{a\}, each model–context cell has centred weighted profileqa=pa​∥ua−u¯∥2q\_\{a\}=p\_\{a\}\\lVert u\_\{a\}\-\\bar\{u\}\\rVert^\{2\}\. The exact trace identity between the per\-context output Fisher and its weighted profile is checked in every context\. Frame certificates are evaluated atb=1/2b=1/2,k=64k=64on the fixed 56\-curve battery, in both the standardised basis and native read\-out coordinates\.

###### External\-family prediction\.

The profile\-to\-spectrum\-to\-effective\-dimension test uses design choices fixed with Pythia and GPT\-2, then evaluates a text\-disjoint 64\-context battery on BLOOM\-560M/1\.1B, GPT\-Neo\-125M/1\.3B, Mamba\-130M/1\.4B, Qwen2\-0\.5B/1\.5B and RWKV\-1\.5B\. A prior compatibility context is text\-disjoint from the scored battery; no previously probed model–context cell enters the evaluation\.

The spectrum predictor fixed before evaluation isλ^k=q\(k\)\\widehat\{\\lambda\}\_\{k\}=q\_\{\(k\)\}for1≤k≤r1\\leq k\\leq r, and Eq\. \([3](https://arxiv.org/html/2609.11063#Sx1.E3)\) applies the effective\-dimension transform to these predicted eigenvalues\. The measured quantity is computed separately from the revealed spectrum asNeff​\(α\)=∑kλk/\(λk\+α\)N\_\{\\mathrm\{eff\}\}\(\\alpha\)=\\sum\_\{k\}\\lambda\_\{k\}/\(\\lambda\_\{k\}\+\\alpha\)\. Spectral error is the median of\|log10⁡\(λk/q\(k\)\)\|\|\\log\_\{10\}\(\\lambda\_\{k\}/q\_\{\(k\)\}\)\|over8≤k≤328\\leq k\\leq 32, exponent error is\|βλ−βq\|\|\\beta\_\{\\lambda\}\-\\beta\_\{q\}\|over the same ranks, and effective\-dimension error is the RMSE over ten fixed relative dampings between measuredNeff/rN\_\{\\mathrm\{eff\}\}/rand the value obtained by applying the same transform to the leadingrrprofile values\. Family summaries are medians over the complete model\-by\-context grid; 95% intervals use 2,000 crossed\-bootstrap draws that resample model and context axes independently\. A separately fixed 16\-context\-per\-model battery compares paired errors with a same\-trace flat spectrum and its effective\-dimension curve and, for the spectrum, a random permutation of profile ranks\. Exact eigenvalues are revealed only to score the fixed predictions: the profile prediction itself uses neither eigendecomposition nor a fitted parameter\.

###### Probability and read\-out contributions\.

The mechanism comparison used two model sizes from each of seven families: Pythia, GPT\-2, GPT\-Neo, Qwen2, BLOOM, Mamba and RWKV\. The 192\-context battery was balanced across Wikipedia, AG News and LAMBADA\. A common set of 27,450 decoded singleton outcomes defined conditional model laws, with read\-out rows centred and standardised as above\. The weighted profile was compared with the probability\-only profile\(r/V\)​p\(k\)\(r/V\)p\_\{\(k\)\}, using the same rankrr, vocabulary sizeVVand damping values\. Effective\-dimension errors were computed across ten damping values on all contexts; spectral\-exponent errors used exact spectra on 48 contexts per model over ranks 8–32\. Errors were summarised by their median within each family\. The two contrasts were tested by enumerating all272^\{7\}sign patterns of the family log\-error ratios \(Supplementary Note\)\.

###### Finite\-size validation\.

A separate test fits the bounded curveNeff/d=\[1\+\(c/c⋆\)γ\]−1N\_\{\\mathrm\{eff\}\}/d=\[1\+\(c/c^\{\\star\}\)^\{\\gamma\}\]^\{\-1\}and an equal\-parameter unbroken power curve to four anchor dampings, then scores three interleaved held\-out dampings on each of 80 curves from ten models in seven families\. Median curve\-level error ratios use 5,000 crossed family–model–prompt bootstrap draws; the exact grids and sampling scheme are given in Supplementary Note\.

#### Acquisition forecasting and causal evidence depth

###### Corpus\-based forecast\.

Acquisition time is measured on a logarithmic scale,τ=log2⁡t\\tau=\\log\_\{2\}t, wherettis the training step at which a fact’s margin first crosses a fixed threshold and persists\. Ex\-ante corpusnn\-gram margins are computed at three statistical levels, fixed before any model evaluation; a model\-size\-specificnn\-gram predictor is fitted once on the discovery battery and applied without refitting to two fresh batteries whose prefixes never occur in the discovery battery: 180 facts at Pythia 70M and 160M, and 300 facts at Pythia 410M \(timing scored inlog2\\log\_\{2\}training\-step units against constant acquisition\-time and label\-permutation baselines\)\.

###### Randomised evidence depth\.

The causal depth experiment runs in a randomised controlled synthetic language: otherwise identical facts are assigned by a cyclic5×55\\times 5Latin\-square design to five evidence\-depth rungs across paired arms with byte\-identical initial weights and batch streams \(12 fresh seeds, 45 quintets, 225 distinct facts, a 40,000\-step horizon on a 145\-point logarithmic grid\)\. The primary analysis estimates the translation of the fixed lower\-tail persistent\-acquisition quantile band \(p=0\.10p=0\.10–0\.200\.20\) between the extreme rungs, with crossed seed\-by\-quintet bootstrap intervals, leave\-one\-seed\-out ranges and an alternative random\-number stream as robustness; the shallow adjacent contrast is dilation\-only, and per\-fact additive readings are first\-order descriptions with a measurednn\-gram\-margin\-by\-depth interaction\.

###### Architecture and evidence\-construction generalisation\.

A separate fully crossed randomisation tests whether the causal direction generalises across architecture and evidence construction\. Compact four\-layer, width\-256 GPT\-2\-, GPT\-NeoX\- and Llama\-style decoders are each trained on distant copy\-back\-reference and four\-cue\-parity controlled languages\. Within each architecture, language and seed, a crossover exchanges shallow and deep assignment within 384 matched reciprocal\-fact blocks while holding initial parameters, minibatch streams, exposure counts, sequence length and the global shallow:deep mixture fixed \(eight seeds per cell; 96 runs\)\. The response is the mean paired deep\-minus\-shallow persistent\-acquisition delay at the 0\.75\-nat margin threshold on a 100\-point grid through step 44,800\. Confidence intervals use 10,000 crossed bootstrap draws over seed and matched block\. Two control families preserve the nuisance structure: 16 fixed\-treatment blocks per cell and randomisation of deep/shallow labels within the matched blocks\.

###### Cumulative acquisition and observational composition\.

The cumulative acquisition curves use the 300\-fact battery at all three sizes, with nine seeds and 20 checkpoints per size; their held\-out loss coordinate is mean token negative log\-likelihood on the same fixed 50,000\-token Wikitext validation prefix\. The real\-model observational composition uses a fresh 300\-fact battery over two model sizes, nine initialisation seeds and twenty public checkpoints per size \(360 response cells\) with four measured components: thenn\-gram predictor applied without refitting, the observational level schedule, cumulative\-curve alignment by held\-out loss and a between\-seed variability threshold\. The loss\-versus\-step comparison is descriptive; causal effects are tested by the randomised experiments above\. Numerical resolution was measured before held\-out evaluation and is reported in Supplementary Note\.

#### Intervention computation, calibration and reusable control

###### Matrix\-free solves and damping\.

Intervention solves applyGGwithout formingJJorGG, using two operators\.

The low\-rank apply restricts the law to the top\-\|S\|\|S\|tokens by mass and renormalises their probabilities, givingH≈B​B⊤H\\approx BB^\{\\top\}withB=WU\[S,:\]⊤\(diag\(pS\)−pSpS⊤\)1/2B=W\_\{U\}\[S,:\]^\{\\top\}\\,\(\\mathrm\{diag\}\(p\_\{S\}\)\-p\_\{S\}p\_\{S\}^\{\\top\}\)^\{1/2\}, andM=J⊤​BM=J^\{\\top\}Bobtained by\|S\|\|S\|vector–Jacobian products; the damped solve\(G\+α​I\)−1​q\(G\+\\alpha I\)^\{\-1\}qthen follows from a Woodbury identity requiring only an\|S\|×\|S\|\|S\|\\times\|S\|system in the nativeR=IR=Iimplementation\.

For the exact apply, a Jacobian–vector product computesJ​vJv, the exact full\-vocabularyH⁡\(J​v\)=WU⊤​\(diag⁡\(p\)−p​p⊤\)​WU​\(J​v\)H\(Jv\)=W\_\{U\}^\{\\top\}\(\\mathrm\{diag\}\(p\)\-pp^\{\\top\}\)W\_\{U\}\(Jv\)is applied in closed form, and a vector–Jacobian product returnsJ⊤​\(⋅\)J^\{\\top\}\(\\cdot\), givingG​vGvin two passes using standard matrix\-free automatic differentiation[65](https://arxiv.org/html/2609.11063#bib.bib65)\. The native\-reference solve uses conjugate gradients \(CG\) onG\+α​IG\+\\alpha I, while a generalRRis handled by whitening or preconditioned generalised CG \(Supplementary Note\)\. Implicit pullback\-Fisher products and conjugate\-gradient solves also appear in distribution\-space attribution[66](https://arxiv.org/html/2609.11063#bib.bib66)\.

For a reference metricRR, damping is set asα=c​λmax​\(G,R\)\\alpha=c\\,\\lambda\_\{\\max\}\(G,R\), where the largest generalised eigenvalue is estimated by power iteration in whitened coordinates andccis typically10−210^\{\-2\}\. The native\-reference analyses useR=IR=Iand therefore reduce toα=c​λmax​\(G\)\\alpha=c\\lambda\_\{\\max\}\(G\)\. The operator can be exactly singular, numerically rank\-deficient, or merely ill\-conditioned; these cases are distinguished from deliberate low\-rank output truncation\. When needed, the solve is restricted to the numerically resolved eigenspace ofGG\. Iterations to a relative residual of10−610^\{\-6\}quantify fixed\-resolution solver convergence\.

###### Objectives and matched\-response calibration\.

For a contrastive pair\(yw,yl\)\(y\_\{w\},y\_\{l\}\)the objective is the summed log\-oddsϕ⁡\(h\)=∑t\[log⁡p⁡\(yw\(t\)∣x,yw<t\)−log⁡p⁡\(yl\(t\)∣x,yl<t\)\]\\phi\(h\)=\\sum\_\{t\}\[\\log p\(y\_\{w\}^\{\(t\)\}\\\!\\mid\\\!x,y\_\{w\}^\{<t\}\)\-\\log p\(y\_\{l\}^\{\(t\)\}\\\!\\mid\\\!x,y\_\{l\}^\{<t\}\)\]over the continuation tokens, with covectorq=∇hϕq=\\nabla\_\{h\}\\phi\. The objective spans the full continuation\. Its relation to direct preference optimisation \(DPO\)[5](https://arxiv.org/html/2609.11063#as1_bib.bib5), which updates parameters rather than activations, is given in Supplementary Note\.

Matched\-response comparisons use the same realised objective changeΔ​ϕ\\Delta\\phi[6](https://arxiv.org/html/2609.11063#as1_bib.bib6),[7](https://arxiv.org/html/2609.11063#as1_bib.bib7)\. Each method’s step scale is binary\-searched, with interpolation between bracketing steps where necessary; all other endpoints are evaluated at that matched change\. The instruction\-model frontiers instead evaluate each prompt\-specific direction on a method\-specific strength grid shared across the 12 prompts, average behaviour and off\-target KL at each grid point, and compare interpolated frontiers at equal aggregate mean behaviour change\. The amortised contrastive\-activation\-addition baseline uses the mean covector direction over the contrast set and the same frontier convention\. The six\-size Pythia factual sweep reports its three requested shifts as nominal targets\.

###### Disturbance and paired steering comparisons\.

Off\-target change is measured as off\-target KL: the KL of the output change evaluated over the complement of the objectiveϕ\\phi’s support \(off\_target\_kl\_set\)\. For scalar objectives the analyses additionally use the Fisher\-orthogonal decompositionKL\(p0∥ps\)−Δϕ2/\(2Varp0ϕ\)\\mathrm\{KL\}\(p\_\{0\}\\\|p\_\{s\}\)\-\\Delta\\phi^\{2\}/\(2\\,\\mathrm\{Var\}\_\{p\_\{0\}\}\\phi\), withpsp\_\{s\}the distribution after a steer of magnitudess, which isolates the disturbance orthogonal to the intended change \(clamped at zero, as the second\-order decomposition can go slightly negative at large steps\)\. Capability retention on the frontier for instruction\-tuned models is off\-target KL on held\-out neutral prompts that are semantically disjoint from the steered attribute\. Win rates are computed per prompt at the applicable comparison setting\.

In the 12\-pair steering comparisons, advantages are disturbance ratios and win rates are the fraction of pairs with strictly lower natural\-gradient disturbance at the comparison setting\. Paired bootstrap draws resample the 12 contrastive pairs in log\-ratio space\. The same pairs are used across width \(410M versus 1\.4B, both 24 layers\) and depth \(1B at 16 layers versus 1\.4B at 24 layers\); nominal\-target sweep summaries use the two arms at each requested target \(Supplementary Note\)\.

###### Prediction of relative intervention cost\.

The finite\-damping cost ratioRpred=ΠG​\(q\)/ΠG​\(A−1​q\)R\_\{\\mathrm\{pred\}\}=\\Pi\_\{G\}\(q\)/\\Pi\_\{G\}\(A^\{\-1\}q\), withΠG​\(u\)=\(u⊤​G​u\)/\(q⊤​u\)2\\Pi\_\{G\}\(u\)=\(u^\{\\top\}Gu\)/\(q^\{\\top\}u\)^\{2\}andA=G\+α​IA=G\+\\alpha I, is computed from predictor state fixed before any response is measured\. The held\-out evaluation spans 11 models from six families, three objectives, three relative layer depths and 12 held\-out contexts per objective \(1,188 cells\), with three primary matched\-effect magnitudes per cell\. Damped natural\-gradient and Euclidean interventions are compared by their realised output KL at matched achieved objective change via binary\-searched step scales\. Of 3,564 primary measurements, 3,515 meet the fixed reachability, numerical\-floor and effect\-matching criteria\. Evaluation reports the median measured\-to\-predicted ratio and log–log slope with 10,000 prompt\-template\-cluster bootstrap draws, improvement over an oracle constant and a stratified permutation test\. To resolve intervention\-magnitude dependence, the three fractions are summarised within each model as geometric means of the measured\-to\-predicted ratio\. Paired layer intervals resample matched objective–context–magnitude tuples within model while preserving the available layer tuple, and model\-balanced pooled intervals resample models\. Numerical resolution is checked with an fp64 deep ladder, detailed in Supplementary Note\.

###### Finite intervention paths\.

Finite paths recomputed both Fisher and Euclidean directions after every accepted step, using a prompt\-KL trust region of0\.020\.02nats\. Scale was chosen by expansion and bisection to reach cumulative objective changes of0\.050\.05,0\.10\.1,0\.20\.2and0\.40\.4nats within 1%\. Cumulative local KL sums the divergence between successive accepted output laws; endpoint KL is measured separately\. The panel contains 14 base models and two instruction\-tuned models, giving 18 model–objective cells with 12 prompts each\. Paired log\-cost intervals used 4,000 prompt bootstrap draws per cell\.

###### Reusable control\.

Reusable control used a common residual\-stream activation update near 45% of network depth at the final prompt position, estimated from four donor prompts per objective\. Three directions were compared: the Euclidean gradient, the gradient preconditioned by donor\-averaged Fisher, and the gradient preconditioned by Fisher averaged over four reference prompts\. All three methods were relinearised along their paths and matched to donor log\-odds changes of0\.050\.05,0\.10\.1and0\.20\.2nats\. Reference\-Fisher damping was0\.030\.03times the estimated largest eigenvalue\. The resulting update was applied unchanged to eight unseen target prompts per objective without target gradients\. Cost was the sum of teacher\-forced next\-token KL along each reference continuation, averaged over the four reference sequences; their prompt\-end metrics were used during optimisation\. A two\-objective solve used the same reference metric and requested0\.050\.05nats for each donor objective \(Supplementary Note\)\.

###### Layer and intervention\-magnitude dependence\.

Layer dependence is measured at layers4,8,12,204,8,12,20of Pythia\-410M using advance\-one\-step steering at matched objective change\. Conditioning is reported in native coordinates and as generalised anisotropy relative toRR\. The equality condition at the intervention layer isG∝RG\\propto R\(or, for a particular objective, support ofqqin one generalised eigenspace\), in which case natural and reference\-gradient directions coincide\. A condition onHHalone is insufficient unlessJJis also a scaled isometry\.

The cubic KL correction combines the pulled\-back Amari–Chentsov term with a network\-Hessian term; the former alone is complete only at an affine read\-out\. Nested automatic differentiation computes both terms and directional finite differences check their sum\. Erosion prediction is analysed on moderate\-intervention crossings along monotone scan segments, controlling for the gentle\-regime advantage\. Scan details and regime\-dependent results are in Supplementary Note\.

#### Editing, attribution, dictionary learning and fine\-tuning

###### Knowledge editing\.

Fisher information has also been used to select model components for knowledge editing[68](https://arxiv.org/html/2609.11063#bib.bib68); the comparisons here concern the intervention and its output disturbance\. For 30 held\-out CounterFact edits on GPT\-2, Fisher and Euclidean displacements are each computed once on the canonical prompt, calibrated to the same contrastive target shift and reused unchanged\. Both operators share a gating rule evaluated once per query\. The primary contrast is the paired ratio of other\-token KL after both reach the target; intervals bootstrap edits\.

A separate GPT\-2\-XL benchmark applies query\-time Fisher steps behind a normalised subject\-string rule\. It uses EasyEdit metrics on 100 edits and a stricter probability\-margin protocol on 300 edits\. Different access and amortisation conditions make these descriptive comparisons with persisted editors \(Supplementary Note\)\.

###### Sparse\-feature sensitivity and attribution\.

For a sparse\-autoencoder feature with decoder directionvv, the Fisher\-weighted output sensitivity isv⊤​G​vv^\{\\top\}Gv, the second\-order KL coefficient per squared unit displacement along the feature\. For zero ablation of an active sparse\-autoencoder \(SAE\) feature, with displacementΔ​h\\Delta h, the corresponding predicted cost is12​Δ​h⊤​G​Δ​h\\tfrac\{1\}\{2\}\\Delta h^\{\\top\}G\\Delta h\. The held\-out intervention study sets the active feature to zero before decoding and measures the resulting full\-vocabulary KL divergence\. Separate discovery contexts are used to select features and the token IDs and signs defining the linear objective; all reported causal outcomes use 18 held\-out evaluation\-context clusters\. The selected\-token probability\-change fraction is the fraction of the full\-vocabulary absolute probability change on the discovery\-selected tokens, and sign agreement measures whether their nonlinear probability changes have the discovery\-defined signs\. The objective\-effect\-to\-Fisher\-cost ratio is the squared linear objective effect divided byΔ​h⊤​G​Δ​h\\Delta h^\{\\top\}G\\Delta h; attribution\-patching magnitude is the absolute activation–gradient product\. The minimal selective basis is grown by orthogonal matching pursuit in the Fisher metric on the top eigen\-directions ofGG, and compared with greedy per\-feature selection at equal basis size\.

###### Interpretable partitions\.

For interpretable token partitions, probability\-weighted group means define a mass\-preserving Fisher coarse\-graining\. Equal\-objective\-value groups preserve information about a scalar tilt; read\-out clustering refines the approximation to the full update\. Fidelity and the interpretation–computation tradeoff are evaluated against the ungrouped solve \(Supplementary Note\)\.

###### Dictionary learning\.

Top\-kksparse autoencoders are trained and precondition the*decoder*update by the natural gradient in the code geometry:Δ​D∝\(𝔼⁡\[a​a⊤\]\+ε​I\)−1​∇Dℒ\\Delta D\\propto\(\\mathbb\{E\}\[aa^\{\\top\}\]\+\\varepsilon I\)^\{\-1\}\\nabla\_\{D\}\\mathcal\{L\}, whereaaare the codes\. The preconditioner is applied matrix\-free per batch by a Woodbury solve, never forming the code\-Gram matrix\. Real residual activations are heavy\-tailed; in the reported configuration, per\-token norms are winsorised at approximately the 95th percentile before training\.

Comparisons use three paired seeds\. Recovery, reconstruction variance explained, feature use and activation frequency are reported together\.

Disentangling is evaluated against planted ground\-truth features by the Hungarian\-matched mean absolute cosine between learned decoder atoms and ground\-truth features\. Scale comparisons hold the*per\-feature*training budget fixed \(Supplementary Note\)\.

###### Low\-rank adaptation\.

For low\-rank adaptation \(LoRA[69](https://arxiv.org/html/2609.11063#bib.bib69)\) the preconditioner is the generalised Gauss–Newton/Fisher of the output loss aggregated over examples,F=∑xJx⊤​Hx​JxF=\\sum\_\{x\}J\_\{x\}^\{\\top\}H\_\{x\}J\_\{x\}, applied to both theAAandBBadapter factors\. Exact matrix\-free natural\-gradient LoRA is compared against a Kronecker\-factored approximate\-curvature \(K\-FAC\) method[70](https://arxiv.org/html/2609.11063#bib.bib70)and against the coarse\-grained metric \(Supplementary Note\)\. Off\-target preservation is measured on held\-out prompts disjoint from the adaptation set\. The held\-out comparison uses rank\-4 adapters on Pythia\-70M, three matched target gains, eight outer seeds and 16 held\-out preservation prompts\. Its primary endpoint is the area under the preservation\-KL frontier\. Exact natural\-gradient LoRA is compared with a tuned AdamW baseline that includes an explicit KL penalty\. Task transfer and capability retention are measured separately; these endpoints did not support a broader conclusion, so the reported result is restricted to local preservation\.

###### Cyclic concepts and relations\.

For a cyclic registry \(weekdays, months\), the advance\-one\-step map is fitted as the metric isometry \(rotation\) in the Fourier\-1 phase representation that best carries each member’s distribution to the next\. The rotation angle is compared with the ideal2​π/n2\\pi/n\. The off\-rotation component is the metric\-isometry defect reported in the double dissociation, and the closure defect is the Fisher–Rao distance between the start point and the composition ofnnsteps\.

Relations \(capital\-of, gender, tense, comparative\) are scored by probability\-displacement alignment, the mean cosine alignment of the probability\-displacement directions across instances, reported against a permuted\-geometry control that preserves marginals \(Supplementary Note\)\.

#### Statistical analysis and reproducibility

###### Statistical analysis\.

Confidence intervals are 95% unless stated otherwise\. Resampling follows each experiment’s replication unit and preserves pairing between methods: contexts or context folds for geometry, whole language pairs for causal assignment, crossed seeds and facts or matched blocks for acquisition, and prompts or edits for interventions\. Model/family resampling and analysis\-specific exclusions are stated with the corresponding protocols\. Cross\-model agreement, consensus semantic alignment, corpus mediation and eigenvector\-sharing intervals use 5,000 full\-size context draws with replacement, paired across fixed model rosters and contrasts; distance\-matrix analyses average tied ranks and exclude comparisons between repeated copies of the same original context\. Separately labelled 80%\-subsample ranges describe stability under context removal \(Supplementary Note\)\. Probe accuracies average folds and ordered model pairs\.

###### Compute and reproducibility\.

Model execution and intervention solves use float32, with float64 exceptions specified in the individual protocols\. Most GPU\-resident model\-forward runs use one NVIDIA RTX 5000 Ada \(32 GB\), with additional runs on NVIDIA A100 GPUs and CPU execution for dense statistical analyses\. Full GPU\-resident pullback solves fit through 2\.8B parameters; at 6\.9B the evidence comprises graph\-free spectra and CPU\-based pullback calibration/offload analyses, detailed in Supplementary Note\. For the reported computational analyses, applicable random seeds, model revisions, prompts and objective\-set definitions were recorded and are available as described in the Data and Code availability statements \(Supplementary Notesand\)\.

###### Use of generative AI\.

Generative AI tools assisted with research and manuscript preparation\. The author independently verified the outputs, made the substantive decisions and takes full responsibility for the work\.

### Data availability

The public pretrained checkpoints and benchmark datasets used in the reported analyses include the Pythia suite[24](https://arxiv.org/html/2609.11063#as1_bib.bib24), GPT\-2[25](https://arxiv.org/html/2609.11063#as1_bib.bib25), GPT\-Neo[28](https://arxiv.org/html/2609.11063#as1_bib.bib28), Qwen2[26](https://arxiv.org/html/2609.11063#as1_bib.bib26), Qwen1\.5[27](https://arxiv.org/html/2609.11063#as1_bib.bib27), Mistral\-7B[29](https://arxiv.org/html/2609.11063#as1_bib.bib29), Mamba[30](https://arxiv.org/html/2609.11063#as1_bib.bib30), RWKV[31](https://arxiv.org/html/2609.11063#as1_bib.bib31), StarCoder2[33](https://arxiv.org/html/2609.11063#as1_bib.bib33), BLOOM[32](https://arxiv.org/html/2609.11063#as1_bib.bib32), OLMo[21](https://arxiv.org/html/2609.11063#as1_bib.bib21), Sentence\-BERT[3](https://arxiv.org/html/2609.11063#as1_bib.bib3)and DINOv2[17](https://arxiv.org/html/2609.11063#as1_bib.bib17); and WikiText[16](https://arxiv.org/html/2609.11063#as1_bib.bib16), LAMBADA[4](https://arxiv.org/html/2609.11063#as1_bib.bib4), SST\-2[57](https://arxiv.org/html/2609.11063#bib.bib57), TruthfulQA[58](https://arxiv.org/html/2609.11063#bib.bib58), CounterFact[9](https://arxiv.org/html/2609.11063#as1_bib.bib9), CIFAR\-100[18](https://arxiv.org/html/2609.11063#as1_bib.bib18), and the human prediction datasets of Peelle et al\.[20](https://arxiv.org/html/2609.11063#as1_bib.bib20), de Varda et al\.[22](https://arxiv.org/html/2609.11063#as1_bib.bib22)and DERCo[23](https://arxiv.org/html/2609.11063#as1_bib.bib23)\. Derived data supporting the findings are available from the corresponding author on reasonable request\.

### Code availability

The matrix\-free implementation and analysis scripts supporting the reported computational results are available from the corresponding author on reasonable request\.

### Acknowledgements

D\.P\. acknowledges support from the Engineering and Physical Sciences Research Council \(EPSRC\) under grant numbers EP/T517793/1, EP/S021582/1, and EP/W524335/1\.

### Author contributions

D\.P\. conceived the project, performed all analytical derivations and numerical experiments, and wrote the manuscript\.

### Competing interests

The author declares no competing interests\.

### References

- 1Giorgos Nikolaou, Tommaso Mencattini, Donato Crisostomi, Andrea Santilli, Yannis Panagakis, and Emanuele Rodolà\.Language models are injective and hence invertible, 2025\.arXiv:2510\.15511\.
- 2John X\. Morris, Wenting Zhao, Justin T\. Chiu, Vitaly Shmatikov, and Alexander M\. Rush\.Language model inversion\.In*International Conference on Learning Representations*, 2024\.arXiv:2311\.13647\.
- 3Murtaza Nazir, Matthew Finlayson, John X\. Morris, Xiang Ren, and Swabha Swayamdipta\.Better language model inversion by compactly representing next\-token distributions, 2025\.arXiv:2506\.17090\.
- 4N\. N\. Chentsov\.*Statistical Decision Rules and Optimal Inference*, volume 53 of*Translations of Mathematical Monographs*\.American Mathematical Society, 1982\.
- 5C\. Radhakrishna Rao\.Information and the accuracy attainable in the estimation of statistical parameters\.*Bulletin of the Calcutta Mathematical Society*, 37:81–91, 1945\.
- 6Shun\-ichi Amari\.*Information Geometry and Its Applications*\.Springer, 2016\.
- 7Shun\-ichi Amari\.Natural gradient works efficiently in learning\.*Neural Computation*, 10\(2\):251–276, 1998\.
- 8James Martens\.New insights and perspectives on the natural gradient method\.*Journal of Machine Learning Research*, 21\(146\):1–76, 2020\.
- 9Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola\.Position: The platonic representation hypothesis\.In*Proceedings of the 41st International Conference on Machine Learning*, 2024\.arXiv:2405\.07987\.
- 10Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton\.Similarity of neural network representations revisited\.In*Proceedings of the 36th International Conference on Machine Learning*, 2019\.
- 11Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà\.Relative representations enable zero\-shot latent space communication\.In*International Conference on Learning Representations*, 2023\.arXiv:2209\.15430\.
- 12Sihan Wang, Jiayi Zhao, Qingyan Cao, Hongbo Yao, and Lin Shu\.FishBack: Pullback Fisher geometry for optimal activation steering in transformers, 2026\.arXiv:2605\.17231\.
- 13Kiho Park, Todd Nief, Yo Joong Choe, and Victor Veitch\.The information geometry of softmax: Probing and steering\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.arXiv:2602\.15293\.
- 14Shivam Raval, Hae Jin Song, Linlin Wu, Abir Harrasse, Jeff M\. Phillips, Fazl Barez, and Amirali Abdullah\.Curveball steering: The right direction to steer isn’t always linear, 2026\.arXiv:2603\.09313\.
- 15Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner\.Steering Llama 2 via contrastive activation addition\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15504–15522\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.acl\-long\.828\.
- 16Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann\-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J\. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J\. Zico Kolter, and Dan Hendrycks\.Representation engineering: A top\-down approach to AI transparency, 2023\.arXiv:2310\.01405\.
- 17Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J\. Vazquez, Ulisse Mini, and Monte MacDiarmid\.Steering language models with activation engineering, 2024\.arXiv:2308\.10248\.
- 18Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg\.Inference\-time intervention: Eliciting truthful answers from a language model\.In*Advances in Neural Information Processing Systems*, volume 36, pages 41451–41530\. Curran Associates, Inc\., 2023\.
- 19Emanuele Marconato, Sébastien Lachapelle, Sebastian Weichwald, and Luigi Gresele\.All or none: Identifiable linear properties of next\-token predictors in language modeling\.In*Proceedings of the 28th International Conference on Artificial Intelligence and Statistics*, volume 258 of*Proceedings of Machine Learning Research*, pages 4123–4131\. PMLR, 2025\.URL[https://proceedings\.mlr\.press/v258/marconato25a\.html](https://proceedings.mlr.press/v258/marconato25a.html)\.
- 20Beatrix M\. G\. Nielsen, Emanuele Marconato, Andrea Dittadi, and Luigi Gresele\.When does closeness in distribution imply representational similarity? an identifiability perspective\.In*Advances in Neural Information Processing Systems*, volume 38, 2025\.URL[https://arxiv\.org/abs/2506\.03784](https://arxiv.org/abs/2506.03784)\.arXiv:2506\.03784\.
- 21Beatrix M\. G\. Nielsen, Emanuele Marconato, Luigi Gresele, Andrea Dittadi, and Simon Buchholz\.Logit distance bounds representational similarity\.In*Proceedings of the 43rd International Conference on Machine Learning*, 2026\.URL[https://arxiv\.org/abs/2602\.15438](https://arxiv.org/abs/2602.15438)\.arXiv:2602\.15438\.
- 22Walter Nelson, Marco Fumero, Theofanis Karaletsos, and Francesco Locatello\.Statistical and structural identifiability in representation learning\.In*International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=Wa3cfE3Iay](https://openreview.net/forum?id=Wa3cfE3Iay)\.
- 23Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis\.Implicit geometry of next\-token prediction: From language sparsity patterns to model representations, 2024\.URL[https://arxiv\.org/abs/2408\.15417](https://arxiv.org/abs/2408.15417)\.arXiv:2408\.15417\.
- 24Yize Zhao and Christos Thrampoulidis\.On the geometry of semantics in next\-token prediction, 2025\.URL[https://arxiv\.org/abs/2505\.08348](https://arxiv.org/abs/2505.08348)\.arXiv:2505\.08348\.
- 25Dhruva Karkada, Daniel J\. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri\.Symmetry in language statistics shapes the geometry of model representations\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.arXiv:2602\.15029\.
- 26Yize Zhao, Isabel Papadimitriou, and Christos Thrampoulidis\.Structure before collapse: Transient semantic geometry in next\-token prediction, 2026\.URL[https://arxiv\.org/abs/2606\.26749](https://arxiv.org/abs/2606.26749)\.arXiv:2606\.26749\.
- 27Francesco Cagnetta and Matthieu Wyart\.Towards a theory of how the structure of language is acquired by deep neural networks\.In*Advances in Neural Information Processing Systems*, volume 37, pages 83119–83163, 2024\.URL[https://arxiv\.org/abs/2406\.00048](https://arxiv.org/abs/2406.00048)\.arXiv:2406\.00048\.
- 28Francesco Cagnetta, Allan Raventós, Surya Ganguli, and Matthieu Wyart\.Deriving neural scaling laws from the statistics of natural language\.In*Proceedings of the 43rd International Conference on Machine Learning*, 2026\.URL[https://arxiv\.org/abs/2602\.07488](https://arxiv.org/abs/2602.07488)\.arXiv:2602\.07488\.
- 29JaeSeong Kim and Suan Lee\.Same benchmark, same subspace: Task\-selective convergence in LLM representations\.In*Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence*, volume 337 of*Proceedings of Machine Learning Research*, pages 3101–3116\. PMLR, 2026\.URL[https://proceedings\.mlr\.press/v337/kim26g\.html](https://proceedings.mlr.press/v337/kim26g.html)\.
- 30Ayushi Agarwal\.Cross\-architecture steering transfer in language models: A systematic empirical study, 2026\.URL[https://arxiv\.org/abs/2608\.05164](https://arxiv.org/abs/2608.05164)\.arXiv:2608\.05164\.
- 31Nils Reimers and Iryna Gurevych\.Sentence\-BERT: Sentence embeddings using siamese BERT\-networks\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing*, pages 3982–3992\. Association for Computational Linguistics, 2019\.doi:10\.18653/v1/D19\-1410\.
- 32Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V\. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El\-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po\-Yao Huang, Shang\-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski\.DINOv2: Learning robust visual features without supervision\.*Transactions on Machine Learning Research*, 2024\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=a68SUt6zFt](https://openreview.net/forum?id=a68SUt6zFt)\.arXiv:2304\.07193\.
- 33J\. E\. Peelle, R\. L\. Miller, C\. S\. Rogers, B\. Spehar, M\. S\. Sommers, and K\. J\. Van Engen\.Completion norms for 3085 English sentence contexts\.*Behavior Research Methods*, 52:1795–1799, 2020\.doi:10\.3758/s13428\-020\-01351\-1\.
- 34Andrea Gregor de Varda, Marco Marelli, and Simona Amenta\.Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data\.*Behavior Research Methods*, 56:5190–5213, 2024\.doi:10\.3758/s13428\-023\-02261\-8\.
- 35Boi Mai Quach, Cathal Gurrin, and Graham Healy\.DERCo: A dataset for human behaviour in reading comprehension using EEG\.*Scientific Data*, 11:1104, 2024\.doi:10\.1038/s41597\-024\-03915\-8\.
- 36Steven T\. Piantadosi\.Zipf’s word frequency law in natural language: a critical review and future directions\.*Psychonomic Bulletin & Review*, 21\(5\):1112–1130, 2014\.doi:10\.3758/s13423\-014\-0585\-6\.
- 37George Kingsley Zipf\.*Human Behavior and the Principle of Least Effort*\.Addison\-Wesley, Cambridge, MA, 1949\.
- 38Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu\.Scaling and evaluating sparse autoencoders, 2024\.arXiv:2406\.04093\.
- 39Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey\.Sparse autoencoders find highly interpretable features in language models\.*arXiv preprint arXiv:2309\.08600*, 2023\.
- 40Trenton Bricken, Adly Templeton, Joshua Batson, Tom Henighan, et al\.Towards monosemanticity: Decomposing language models with dictionary learning\.*Transformer Circuits Thread*, 2023\.
- 41Aaquib Syed, Can Rager, and Arthur Conmy\.Attribution patching outperforms automated circuit discovery, 2023\.arXiv:2310\.10348\.
- 42Fabian Gröger, Shuo Wen, and Maria Brbić\.Revisiting the platonic representation hypothesis: An aristotelian view\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.arXiv:2602\.14486\.
- 43Sathvik Nair and Byung\-Doh Oh\.Clozing the gap: Exploring why language model surprisal outperforms cloze surprisal\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 33193–33210\. Association for Computational Linguistics, 2026\.doi:10\.18653/v1/2026\.acl\-long\.1534\.URL[https://aclanthology\.org/2026\.acl\-long\.1534/](https://aclanthology.org/2026.acl-long.1534/)\.
- 44Dan Braun, Jordan Taylor, Nicholas Goldowsky\-Dill, and Lee Sharkey\.Identifying functionally important features with end\-to\-end sparse dictionary learning\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.URL[https://arxiv\.org/abs/2405\.12241](https://arxiv.org/abs/2405.12241)\.arXiv:2405\.12241\.
- 45Samuel L\. Braunstein and Carlton M\. Caves\.Statistical distance and the geometry of quantum states\.*Physical Review Letters*, 72\(22\):3439–3443, 1994\.doi:10\.1103/PhysRevLett\.72\.3439\.
- 46Dario Picozzi\.A hardware\-efficient variational ansatz with an exact diagonal metric for real\- and imaginary\-time evolution and Haar sampling, 2026\.arXiv:2607\.07942\.
- 47Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, 2023\.
- 48An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan\.Qwen2 technical report, 2024\.arXiv:2407\.10671\.
- 49Qwen Team\.Introducing Qwen1\.5\.Qwen blog, February 2024\.URL[https://qwenlm\.github\.io/blog/qwen1\.5/](https://qwenlm.github.io/blog/qwen1.5/)\.
- 50Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.Technical report, OpenAI, 2019\.
- 51BigScience Workshop\.BLOOM: A 176b\-parameter open\-access multilingual language model\.*Journal of Machine Learning Research*, 25\(422\):1–74, 2024\.
- 52Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman\.GPT\-Neo: Large scale autoregressive language modeling with mesh\-TensorFlow, 2021\.
- 53Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\.Mistral 7b, 2023\.arXiv:2310\.06825\.
- 54Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2)\.
- 55Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui\-Jie Zhu\.RWKV: Reinventing RNNs for the transformer era\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 14048–14077\. Association for Computational Linguistics, 2023\.doi:10\.18653/v1/2023\.findings\-emnlp\.936\.
- 56Anton Lozhkov, Raymond Li, Loubna Ben Allal, et al\.StarCoder 2 and The Stack v2: The next generation, 2024\.arXiv:2402\.19173\.
- 57Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D\. Manning, Andrew Y\. Ng, and Christopher Potts\.Recursive deep models for semantic compositionality over a sentiment treebank\.In*Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing*, pages 1631–1642\. Association for Computational Linguistics, 2013\.
- 58Stephanie Lin, Jacob Hilton, and Owain Evans\.TruthfulQA: Measuring how models mimic human falsehoods\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2022\.
- 59Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\.Locating and editing factual associations in GPT\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- 60Nikolaus Kriegeskorte, Marieke Mur, and Peter A\. Bandettini\.Representational similarity analysis—connecting the branches of systems neuroscience\.*Frontiers in Systems Neuroscience*, 2:4, 2008\.doi:10\.3389/neuro\.06\.004\.2008\.
- 61Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- 62Jiaqi Mu and Pramod Viswanath\.All\-but\-the\-top: simple and effective postprocessing for word representations\.In*International Conference on Learning Representations*, 2018\.
- 63Alex Krizhevsky\.Learning multiple layers of features from tiny images\.Technical report, University of Toronto, 2009\.
- 64Allen Institute for AI\.OLMo 2 1B: model and training checkpoints\.[https://huggingface\.co/allenai/OLMo\-2\-0425\-1B](https://huggingface.co/allenai/OLMo-2-0425-1B), 2025\.
- 65Barak A\. Pearlmutter\.Fast exact multiplication by the Hessian\.*Neural Computation*, 6\(1\):147–160, 1994\.doi:10\.1162/neco\.1994\.6\.1\.147\.
- 66Gabriele Martino and Sebastian Tschiatschek\.FRInGe: Distribution\-space integrated gradients with Fisher–Rao geometry, 2026\.URL[https://arxiv\.org/abs/2605\.06404](https://arxiv.org/abs/2605.06404)\.arXiv:2605\.06404\.
- 67Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- 68Chenghao Xu, Jiexi Yan, Guangtao Lyu, Qi Liu, Muli Yang, and Cheng Deng\.Fisher\-driven adaptive locating for knowledge editing in large language models\.In*Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 20902–20913\. Association for Computational Linguistics, 2026\.doi:10\.18653/v1/2026\.acl\-long\.957\.URL[https://aclanthology\.org/2026\.acl\-long\.957/](https://aclanthology.org/2026.acl-long.957/)\.
- 69Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.
- 70James Martens and Roger Grosse\.Optimizing neural networks with kronecker\-factored approximate curvature\.In*Proceedings of the 32nd International Conference on Machine Learning \(ICML\)*, 2015\.
- 71Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc\-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández\.The LAMBADA dataset: Word prediction requiring a broad discourse context\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics*, pages 1525–1534\. Association for Computational Linguistics, 2016\.doi:10\.18653/v1/P16\-1144\.

### Extended Data

a

bc

Figure 1:Extended Data Fig\. 1\|\|Coordinate covariance and common\-outcome controls\.a, Coordinate covariance at finite damping\. For 12 prompts under one seeded symmetric positive\-definite gauge family withκ⁡\(A\)=1,3,10,30,100\\kappa\(A\)=1,3,10,30,100, transporting the reference asR′=A−⁣⊤​R​A−1R^\{\\prime\}=A^\{\-\\top\}RA^\{\-1\}reproduces the native damped direction with mapped relative error at most7\.4×10−137\.4\\times 10^\{\-13\}\(green circles\)\. Installing a fresh identity instead changes the regularised problem: over non\-identity gauges, the mapped\-direction error has median 1\.23 and minimum 0\.54 \(vermilion squares\)\. Points, prompts; connected symbols, medians; error bars, 2\.5th–97\.5th percentiles\. Pythia\-410M, final pre\-unembedding activation, top\-512 output\-Fisher support andα=0\.01​λmax\\alpha=0\.01\\lambda\_\{\\max\}\.b, Pairwise output\-geometry agreement before and after mapping each next\-token law to its first\-byte distribution \(21 dyads among eight models\)\. Aggregate agreement is0\.8990\.899at token level and0\.9100\.910after the push\-forward; rank agreement between dyad\-level values is0\.5510\.551\. The grey line marks equality\.c, Hidden\-coordinate reparameterisation at nine anisotropy strengths and three seeds\. Each point aggregates the same 21 model pairs; faint paths show seeds and heavy paths their means\. Output Fisher–Rao agreement remains constant, while activation CKA, mutual nearest\-neighbour overlap and cosine RSA degrade\. Small horizontal offsets separate seeds\.ab

cd

Figure 2:Extended Data Fig\. 2\|\|Shared output structure beyond displacement magnitude and on independent human data\.a, The 24 displayed estimates are six fixed eigenvalue\-rank bands crossed with four summaries: raw subspace overlap \(grey circles\), a displacement\-magnitude\-matched null that preserves each word’s displacement magnitude \(amber squares\), a shuffled\-word null \(magenta triangles\) and the residual above the displacement\-magnitude\-matched null \(blue diamonds\)\. Opaque markers have black borders; lines connect like summaries across bands\. Error bars on the residual span the 2\.5th–97\.5th percentiles across 400 random 80%\-context subsamples\. Raw overlap is nearly flat, but the residual is concentrated in the leading modes and falls from0\.0800\.080in ranks 0–5 to0\.0130\.013in ranks 90–120\. Thus about 89% of the leading\-band raw overlap is explained by shared displacement magnitudes, while the smaller residual carries the rank\-dependent cross\-family structure\.b, Native model samples on 640 word positions from five previously unused narratives\. Points are the equal\-story semantic distances for six final Pythia sizes; bars show conditional 95% bootstrap intervals and the grey line is the registered fit against log parameter count \(slope−0\.00872\-0\.00872, interval−0\.01202\-0\.01202to−0\.00556\-0\.00556\)\.c, The same distance at five ordered checkpoints from Pythia\-160M and OLMo\-2\-1B\. Lines show point estimates and shaded regions the conditional 95% intervals\. Final\-minus\-initial changes are−0\.12080\-0\.12080and−0\.10962\-0\.10962, respectively\.d, Expected human log\-score change after applying the model\-only source\-trained full 65\-outcome map instead of each model’s native law\. Small circles show the five fixed\-story differences and diamonds their equal\-story means; positive values favour calibration\. Six of seven family means are positive \(0\.01960\.0196–0\.05070\.0507nats\); Qwen2 is−0\.0030\-0\.0030nats\. Panels b–d are conditional on the five observed stories and make no population\-of\-stories claim\.ab

cd

Figure 3:Extended Data Fig\. 3\|\|Damping reveals bounded effective dimension\.a, Predicted versus measured stiff\-mode counts for 24 model–context spectra at relative damping10−110^\{\-1\},10−210^\{\-2\}and10−310^\{\-3\}\(72 points\)\. Predictions are\(1/c\)1/β\(1/c\)^\{1/\\beta\}from the power\-law exponent fitted over ranks 2–80; colours identify the eight models, and the grey line denotes equality\.b, All 72 resolved\-mode counts, joined within each of the 24 spectra; colours identify models and black diamonds show the cross\-spectrum median\. The count grows as damping is lowered\. The native\-reference exponent is empirically stable across the tested models but is not an invariant of the metric alone\. In held\-out estimator validation, nominal 95% interval coverage is0\.900\.90for continuous effective dimensions and0\.850\.85for discrete threshold counts; primary dimension estimates therefore use the continuous form \(Methods\)\.c, Finite\-size effective\-dimension confirmation on 80 model–prompt curves spanning ten models and seven families\. For each curve, a bounded effective\-dimension curve was fitted at four fixed anchor dampings; points are all 240 observations at three interleaved, held\-out dampings after rescaling by the fitted crossoverα∗\\alpha^\{\*\}\. The black line is the bounded effective\-dimension curve at the median fitted exponent\.d, Curve\-level held\-out root\-mean\-square error for an equal\-parameter unbroken power law \(abscissa\) and the bounded effective\-dimension curve \(ordinate\)\. Every one of 80 points lies below equality\. Median errors are0\.08140\.0814and0\.004930\.00493, respectively; the median bounded\-to\-power error ratio is0\.07490\.0749\(95% crossed\-bootstrap interval0\.04280\.0428–0\.12340\.1234\)\.a

bc

Figure 4:Extended Data Fig\. 4\|\|Evidence depth delays acquisition across architectures and evidence constructions\.a, Randomised evidence depth delays persistent acquisition\. Curves follow the 10th–20th acquisition\-time percentiles for each of 12 training seeds of the same transformer, pooling over the same facts and probes within each seed\. Black curves show individual seeds; blue, pink and green shading to their right marks acquisition levels already reached under shallow, intermediate and deep evidence, respectively\. The pooled deepest\-versus\-shallowest shift isΔ=2\.10\\Delta=2\.10log2\\log\_\{2\}training\-step units \(crossed seed\-by\-quintet bootstrap 95% interval1\.791\.79–2\.402\.40\), equivalent to4\.3×4\.3\\timesmore training steps; the deepest\-versus\-intermediate contrast is smaller\.b, Mean paired delay in persistent acquisition for evidence assigned to the deep construction relative to otherwise matched evidence in the shallow construction, crossing GPT\-2\-, GPT\-NeoX\- and Llama\-style decoders with copy\-back\-reference and four\-cue\-parity controlled languages\. Blue diamonds denote copy\-back\-reference cells and amber diamonds four\-cue\-parity cells; horizontal lines are 95% crossed seed\-by\-matched\-block bootstrap intervals around the cell means over eight seeds and 384 matched fact blocks per cell\. All six intervals and all 48 seed means are positive; effects span0\.8100\.810–1\.5141\.514log2\\log\_\{2\}training\-step units \(1\.751\.75–2\.862\.86\-fold in training steps\)\. The equal\-family mean parity\-minus\-back\-reference contrast is0\.4140\.414log2\\log\_\{2\}training\-step units \(95% interval0\.2910\.291–0\.5360\.536\)\.c, Corresponding fixed\-treatment \(blue circles\) and label\-randomisation \(vermilion squares\) controls\. Every one of the twelve 95% intervals contains zero\. The grey vertical line in panel c marks zero delay; delays are inlog2\\log\_\{2\}training\-step units\.a

bc

Figure 5:Extended Data Fig\. 5\|\|Additional intervention trade\-off curves, composition and fixed activation editing\.a, The diffuse style objective: behaviour change against off\-target change on neutral prompts\. The complete 19\-point operating sweep contains seven Fisher, six Euclidean and six contrastive\-activation\-addition \(CAA\) settings\.b, Composition: sycophancy and style steered simultaneously by one solve on the weighted covector sum; varying the weight changes the balance between objectives, whereas summed CAA vectors change sycophancy in the opposite direction\. The 20 distinct operating points produce 34 displayed objective outcomes; all markers are opaque with black borders\.c, Matched fixed activation edits on 30 held\-out CounterFact records \(GPT\-2\)\. Each point compares Fisher other\-token Kullback–Leibler \(KL\) divergence \(abscissa\) with Euclidean other\-token KL divergence \(ordinate\) after both operators attained the same fixed\+5\+5log\-odds shift; axes are logarithmic and the grey line denotes equality\. Euclidean cost is higher for all 30 edits, with a geometric\-mean Euclidean\-to\-Fisher ratio of10\.5710\.57\(95% interval8\.528\.52–13\.3713\.37\)\.a![Refer to caption](https://arxiv.org/html/2609.11063v1/ed6_metric_cost.png)

b![Refer to caption](https://arxiv.org/html/2609.11063v1/ed6_ap_specificity.png)c![Refer to caption](https://arxiv.org/html/2609.11063v1/ed6_fisher_specificity.png)

Figure 6:Extended Data Fig\. 6\|\|Output geometry predicts the causal cost and selectivity of sparse features\.Pythia\-410M, layer 12, Top\-kksparse autoencoder\. Points in every panel are the same 303 unique zero ablations of active sparse\-autoencoder \(SAE\) features from 57 evaluation\-active features across 18 held\-out context clusters; feature and objective selection used 18 disjoint discovery contexts\. Opaque green points inaandcmark Fisher quantities, while blue points inbmark attribution patching; every point has a black border\.a, The Fisher prediction12​Δ​h⊤​G​Δ​h\\tfrac\{1\}\{2\}\\Delta h^\{\\top\}G\\Delta hagainst exact full\-vocabularyKL\(p0∥pzero\)\\mathrm\{KL\}\(p\_\{0\}\\\|p\_\{\\mathrm\{zero\}\}\)\(Spearmanρ=0\.997\\rho=0\.997\); grey line, equality\.b, Attribution\-patching magnitude against the selected\-token probability\-change fraction, the fraction of full\-vocabulary absolute probability change on discovery\-selected tokens \(ρ=0\.506\\rho=0\.506\)\.c, The objective\-effect\-to\-Fisher\-cost ratio against the same endpoint \(ρ=0\.935\\rho=0\.935\); its correlation exceeds that of attribution patching by0\.4290\.429\(95% context\-cluster bootstrap interval0\.2960\.296–0\.5530\.553\)\. The comparison concerns concentration on selected tokens; attribution\-patching magnitude remains the stronger predictor of absolute effect magnitude\.Figure 7:Extended Data Fig\. 7\|\|Natural\-gradient low\-rank adaptation lowers held\-out preservation cost at matched training gain\.Exact natural\-gradient LoRA is compared with tuned AdamW with explicit KL regularisation using rank\-4 adapters on Pythia\-70M\. Points are all 48 method–target–seed cells from eight paired outer seeds, three fixed matched training\-gain targets and two methods; each cell averages 16 held\-out preservation prompts\. Vermilion squares denote AdamW\+KL and green circles exact natural gradient\. Pale lines join paired methods within seed and target, horizontal offsets are fixed display jitter, and the larger connected symbols are method means\. The ordinate is logarithmic\. Every method reached every target\. Across this frontier, AdamW\+KL incurred1\.57×1\.57\\timesthe preservation\-KL AUC of exact natural gradient \(95% paired hierarchical bootstrap interval1\.141\.14–1\.981\.98\)\. The experiment did not support a broader task\-transfer or capability\-retention conclusion\.ab

Figure 8:Extended Data Fig\. 8\|\|Layer and intervention magnitude delimit the geometric correction\.a, Off\-target advantage by injection layer for 3,515 usable matched\-response observations from 1,188 model–layer–objective–context cells across 11 models and six families, summarized as 33 within\-model estimates\. Coloured circles are geometric means and vertical bars are 95% paired objective–context–magnitude bootstrap intervals that preserve the available layer tuple; lines connect the three depths within model\. Black diamonds are model\-balanced geometric means with 95% model\-bootstrap intervals\. The pooled means fall from11\.7911\.79to11\.0811\.08to6\.626\.62at quarter, half and three\-quarter depth, respectively\. Every model mean is lower at three\-quarter than at quarter depth; nine of eleven paths decrease at both adjacent steps, with one intermediate\-depth increase for Mamba 130M and Qwen2 1\.5B\.b, Measured\-to\-predicted off\-target advantage at intervention fractionsΔ​C/C=0\.01,0\.03,0\.10\\Delta C/C=0\.01,0\.03,0\.10\. Coloured circles are all 33 within\-model geometric means \(11 models crossed with three fractions\), with 95% paired\-cell bootstrap intervals; lines connect fractions within model\. Black diamonds are model\-balanced geometric means with model\-bootstrap intervals, and the grey line denotes exact calibration\. All 11 model paths decrease strictly\. The model\-balanced means are0\.9180\.918,0\.8600\.860and0\.7550\.755, quantifying the expected attenuation of a local prediction as intervention magnitude grows\.ab

Figure 9:Extended Data Fig\. 9\|\|Output sensitivity, cross\-model sharing and answer motion concentrate in the same native\-reference modes\.Each panel shows all 21 summaries \(seven predefined eigenvalue\-rank bands×\\timesthree quantities\) from six models: the band’s eigenvalue\-mass share, cross\-model eigenvector sharing against the displacement\-magnitude\-matched null and probability motion on the highest\-probability tokens\. To compare their decay directly, each quantity and its 95% context\-bootstrap interval are divided by that quantity’s leading\-band point estimate\. On natural text the top five modes carry 55% of the eigenvalue mass, the largest residual sharing \(\+0\.069\+0\.069, 95% interval0\.0620\.062–0\.0770\.077\) and the largest answer motion \(0\.590\.59\), all three decaying together to the last band\.a, Natural\-text battery\.b, Templated battery\.#### Extended Data note: cyclic concepts and two geometries of behaviour

ab

c

Figure 10:Extended Data Fig\. 10\|\|Two geometries of behaviour: cyclic reweighting and relation\-induced probability displacement\(Pythia\-410M\)\.a,b, Walking the ring by steering month→\\tomonth \(a\) or day→\\today \(b\) around the full cycle at matched concept progress: the plane is the concept’s phase circle in output space and height is accumulated off\-target change \(bits\)\. Both paths traverse the same ordered concept cycle; the reference phase sectors are separated, by construction, by2​π/122\\pi/12\(or2​π/72\\pi/7\)\. After one circuit the endpoint returns towards the starting sector with a method\-dependent residual displacement, while the natural\-gradient path accumulates less off\-target change\.c, The dissociation from analogy relations\. Paired circles and squares show probability\-displacement alignment and its permuted control for all twelve concept sets across three categories \(24 displayed values\); thin grey lines join each concept’s pair and short horizontal bars show category means\. Relations move mass along a shared probability\-displacement direction \(mean alignment0\.490\.49; permuted control0\.330\.33\), while cyclic concepts show none \(0\.000\.00\) and are near\-isometries of the metric\. The shared colour key applies to both trajectory panels\.Beyond local correction of the preceding operations, the output metric also reveals global structure in selected concept sets\. In the Fourier\-1 phase coordinates used in Extended Data Fig\.[10](https://arxiv.org/html/2609.11063#Sx9.F10)a,b, the days of the week and months of the year occupy ordered, equally spaced reference sectors\. Steering once around either full cycle returns the concept phase towards its starting sector, with the displayed method\-dependent residual displacement\. At each transition, the natural\-gradient step disturbs unrelated tokens less than the Euclidean step\.

Analogy relations behave differently\. Pairs such as capital\-of, gender and tense do not reweight a shared set of tokens; they move probability mass between disjoint sets \(Paris to Tokyo,*he*to*she*\)\. The two classes separate as a double dissociation \(Extended Data Fig\.[10](https://arxiv.org/html/2609.11063#Sx9.F10)c\): across twelve concepts, relations move mass along a shared probability\-displacement direction \(high probability\-displacement alignment,0\.490\.49against0\.330\.33for a permuted\-geometry control, essentially zero for cyclic concepts\), while cyclic concepts are near\-isometries of the metric, with a low isometry defect \(0\.290\.29against1\.021\.02for relations\); ordinal concepts fall between\. The pullback metric characterises probability reweighting, whereas movement between disjoint token sets additionally requires a ground metric on tokens; the corresponding structure in embedding space is analysed in a related symmetry study[25](https://arxiv.org/html/2609.11063#bib.bib25)\.

### Supplementary Information The information geometry of large language models is shared, learned, and controllable

\\SetLinkTargetFilter

si\-\#1

Contents

###### Organisation\.

The Supplementary Information follows the paper’s argument\. Part I defines the canonical output geometry, derives its origin in cross\-entropy training and establishes what predictive behaviour identifies\. Part II proves and measures the geometry shared across independently trained models\. Part III shows how language statistics and the location of evidence shape the geometry and its acquisition\. Part IV derives minimal\-disturbance control and gives the computational and operational results\. Part V records the experimental designs, controls and reproducibility information\. Original numbered mathematical results are accompanied by proofs; the classical Chentsov theorem is given as an attributed proof sketch with references to complete proofs\.

## Part I: Origin and identification

### 1The canonical output metric and its pullback

This section supports the main text’s definition of the metric and the instrument: it states canonicity and the exact chart\-covariance of the pullback solve\.

#### 1\.1Canonicity of the Fisher–Rao metric \(Chentsov’s theorem\)

LetΔn−1\\Delta\_\{n\-1\}denote the open simplex of probability distributions onnnoutcomes\. A*congruent Markov embedding*T:Δn−1→Δm−1T:\\Delta\_\{n\-1\}\\to\\Delta\_\{m\-1\},m≥nm\\geq n, is the linear map induced by a stochastic matrix that splits each outcomeaainto sub\-outcomes with fixed conditional probabilities and admits a stochastic left inverse that recoversaa\. Such an embedding therefore preserves statistical information; the corresponding left inverse is a sufficient statistic \(coarse\-graining\) on the embedded family\.

###### Theorem 1\(Chentsov\)\.

Up to a positive multiplicative constant, the Fisher–Rao metricgp​\(u,v\)=∑aua​va/pag\_\{p\}\(u,v\)=\\sum\_\{a\}u\_\{a\}v\_\{a\}/p\_\{a\}is the unique family of Riemannian metrics on the simplices\{Δn−1\}n≥2\\\{\\Delta\_\{n\-1\}\\\}\_\{n\\geq 2\}invariant under every congruent Markov embedding\.

###### Proof sketch\.

Invariance under outcome permutations forces the metric at the barycentrep=\(1/n,…,1/n\)p=\(1/n,\\dots,1/n\)into the formαn​∑aua​va\+βn​\(∑aua\)​\(∑ava\)\\alpha\_\{n\}\\sum\_\{a\}u\_\{a\}v\_\{a\}\+\\beta\_\{n\}\(\\sum\_\{a\}u\_\{a\}\)\(\\sum\_\{a\}v\_\{a\}\); on the tangent space of the simplex, where∑aua=0\\sum\_\{a\}u\_\{a\}=0, only theαn\\alpha\_\{n\}term survives\. Invariance under the equal\-split morphismsΔn−1→Δk​n−1\\Delta\_\{n\-1\}\\to\\Delta\_\{kn\-1\}ties the constantsαn\\alpha\_\{n\}together acrossnnand propagates the barycentric form to every rational point, yieldinggp​\(u,v\)=c​∑aua​va/pag\_\{p\}\(u,v\)=c\\sum\_\{a\}u\_\{a\}v\_\{a\}/p\_\{a\}there; continuity extends this to all ofΔn−1\\Delta\_\{n\-1\}, and any invariant metric must agree with the construction, giving uniqueness up to the scalec\>0c\>0\. A complete proof is given by Chentsov\[[1](https://arxiv.org/html/2609.11063#as1_bib.bib1)\]; a modern treatment is given by Ay, Jost, Lê and Schwachhöfer\[[2](https://arxiv.org/html/2609.11063#as1_bib.bib2)\]\. ∎

Chentsov uniqueness is up to scale and applies within the class of sufficient\-statistic–invariant metrics; the matching unique invariant*3\-tensor*is the Amari–Chentsov cubic used in Supplementary Note[10\.3](https://arxiv.org/html/2609.11063#as1_S10.SS3)\. The metric statement is infinitesimal: Kullback–Leibler \(KL\) divergence satisfiesKL\(p∥p′\)=12dFR\(p,p′\)2\+O\(d3\)\\mathrm\{KL\}\(p\\\|p^\{\\prime\}\)=\\tfrac\{1\}\{2\}d\_\{\\mathrm\{FR\}\}\(p,p^\{\\prime\}\)^\{2\}\+O\(d^\{3\}\), wheredFRd\_\{\\mathrm\{FR\}\}is Fisher–Rao distance\. Finite steps may instead follow a metric geodesic or exponential tilt\. Coordinate covariance is a separate property: every Riemannian metric transforms tensorially under a smooth reparametrisation, whereas Chentsov’s theorem is what singles out Fisher–Rao, up to scale, after requiring invariance under the statistical transformations above\.

#### 1\.2The pullback metric and reparametrisation invariance

The possibly degenerate pullback formG=J⊤​H​JG=J^\{\\top\}HJneed not preserve the rank or spectrum ofHH\. Under an invertible activation reparametrisation, however,GGitself transforms by congruence, which preserves its rank and inertia across charts while changing its ordinary coordinate spectrum\. The null space ofGGis the set of activation directions the read\-out maps to no change inpp, which are invisible in the output by construction\. Under a change of activation charth↦A​hh\\mapsto Ahthe metric transforms as a tensor,G↦A−⁣⊤​G​A−1G\\mapsto A^\{\-\\top\}GA^\{\-1\}\. The intrinsic natural gradient is the unique class on the quotient of activation tangents by the read\-out kernel; Euclidean Moore–Penrose inversion merely selects a coordinate\-dependent representative of that class\. For finite damping the solved problem isδ​h=\(G\+α​R\)−1​q\\delta h=\(G\+\\alpha R\)^\{\-1\}qwith a declared reference metricRR\. IfR↦A−⁣⊤​R​A−1R\\mapsto A^\{\-\\top\}RA^\{\-1\}, thenδ​h↦A​δ​h\\delta h\\mapsto A\\delta hexactly; replacing it by a fresh identity after the coordinate change does not preserve the problem\. The current native\-coordinate implementation usesR=IR=I\. Thus the quotient result is canonical, while the finite damping/reference is an explicit modelling choice, tested in the numerical coordinate analysis below\.

The end\-to\-end reparameterisation test evaluates this distinction at the final intervention site\. For each of 12 prompts in Pythia\-410M, a shared random orthogonal basis is drawn and transformations applied with condition numberκ⁡\(A\)=1,3,10,30,100\\kappa\(A\)=1,3,10,30,100\. Solving withR′=A−⁣⊤​I​A−1R^\{\\prime\}=A^\{\-\\top\}IA^\{\-1\}and mapping the result back reproduces the native damped direction with maximum relative error7\.4×10−137\.4\\times 10^\{\-13\}; the generalised spectrum of\(G′,R′\)\(G^\{\\prime\},R^\{\\prime\}\)reproduces that of\(G,I\)\(G,I\)to3\.9×10−143\.9\\times 10^\{\-14\}\. In contrast, setting a freshIIin the transformed chart changes the direction by a median relative error of 1\.23 over non\-identity transformations \(minimum 0\.54\)\. Thus, resetting the reference metric to the identity after a coordinate transformation is not invariant, whereas transporting the native identity is covariant to numerical precision\. The transported native identity provides a coordinate\-consistent reference, but is not a canonical choice; aggregated output\-Fisher and data\-Mahalanobis references are evaluated separately\.

### 2Cross\-entropy induces and determines the output geometry

This section gives the training origin of the object defined above\. At every affine\-softmax checkpoint, fitting an output distribution by cross\-entropy induces a Gibbs joint law in which the read\-out rows are natural parameters and the output Fisher is one of two paired curvatures\. Excess predictive risk then controls the canonical observables used throughout the paper\.

LetPPbe the population law of a learned hidden stateh∈ℝdh\\in\\mathbb\{R\}^\{d\}and next tokenY∈\{1,…,n\}Y\\in\\\{1,\\ldots,n\\\}\. Once a checkpoint is fixed, write

q⁡\(a∣h\)=exp⁡\(wa⊤​h\+ba\)Z⁡\(h\),Z⁡\(h\)=∑j=1nexp⁡\(wj⊤​h\+bj\)\.q\(a\\mid h\)=\\frac\{\\exp\(w\_\{a\}^\{\\top\}h\+b\_\{a\}\)\}\{Z\(h\)\},\\qquad Z\(h\)=\\sum\_\{j=1\}^\{n\}\\exp\(w\_\{j\}^\{\\top\}h\+b\_\{j\}\)\.\(1\)No assumption about the optimiser is needed; the distributional regularity required for derivatives of the hidden\-side log\-partition is stated below\.

###### Theorem 2\(excess\-risk Gibbs duality\)\.

Assume the population cross\-entropy ofqqis finite and define the model\-induced joint lawQ⁡\(d​h,a\)=PH​\(d​h\)​q​\(a∣h\)Q\(dh,a\)=P\_\{H\}\(dh\)q\(a\\mid h\)\. Then:

Risk\.The excess population cross\-entropy over the Bayes predictor is exactly

ℒ\(q\)−HP\(Y∣H\)=KL\(P∥Q\);\\mathcal\{L\}\(q\)\-H\_\{P\}\(Y\\mid H\)=\\KL\(P\\\|Q\);\(2\)
Gibbs family\.Withd​ν​\(h\)=d​PH​\(h\)/Z⁡\(h\)d\\nu\(h\)=dP\_\{H\}\(h\)/Z\(h\), the joint and conditional laws are

Q⁡\(d​h,a\)\\displaystyle Q\(dh,a\)=exp⁡\(wa⊤​h\+ba\)​d​ν​\(h\),\\displaystyle=\\exp\(w\_\{a\}^\{\\top\}h\+b\_\{a\}\)d\\nu\(h\),\(3\)Q⁡\(d​h∣a\)\\displaystyle Q\(dh\\mid a\)=exp\{wa⊤h−A\(wa\)\}dν\(h\),A\(w\)=log∫ew⊤​hdν\(h\);\\displaystyle=\\exp\\\{w\_\{a\}^\{\\top\}h\-A\(w\_\{a\}\)\\\}d\\nu\(h\),\\qquad A\(w\)=\\log\\int e^\{w^\{\\top\}h\}d\\nu\(h\);\(4\)thus all token\-conditioned laws form one natural exponential family whose natural parameters are the read\-out rows;

Paired curvatures\.WritingB\(h\)=log∑aexp\(wa⊤h\+ba\)B\(h\)=\\log\\sum\_\{a\}\\exp\(w\_\{a\}^\{\\top\}h\+b\_\{a\}\), and assuming thatAAis twice differentiable in a neighbourhood of eachwaw\_\{a\}for which the hidden\-side moments are used,

∇2B​\(h\)\\displaystyle\\nabla^\{2\}B\(h\)=CovQ⁡\(wY∣h\),\\displaystyle=\\Cov\_\{Q\}\(w\_\{Y\}\\mid h\),\(5\)∇2A​\(wa\)\\displaystyle\\nabla^\{2\}A\(w\_\{a\}\)=CovQ⁡\(h∣a\),\\displaystyle=\\Cov\_\{Q\}\(h\\mid a\),\(6\)and the first Hessian is exactly the output Fisher metric;

Classwise control\.IfπaP=P⁡\(Y=a\)\\pi\_\{a\}^\{P\}=P\(Y=a\),Pa=P\(H∈⋅∣Y=a\)P\_\{a\}=P\(H\\in\\cdot\\mid Y=a\)for classes withπaP\>0\\pi\_\{a\}^\{P\}\>0, andQaQ\_\{a\}is defined analogously, then

KL\(P∥Q\)=KL\(πP∥πQ\)\+∑a:πaP\>0πaPKL\(Pa∥Qa\)\.\\KL\(P\\\|Q\)=\\KL\(\\pi^\{P\}\\\|\\pi^\{Q\}\)\+\\sum\_\{a:\\pi\_\{a\}^\{P\}\>0\}\\pi\_\{a\}^\{P\}\\KL\(P\_\{a\}\\\|Q\_\{a\}\)\.\(7\)Consequently, if the excess risk isε\\varepsilonand∥h∥≤R0\\lVert h\\rVert\\leq R\_\{0\}almost surely,

∑a:πaP\>0πaPKL\(Pa∥Qa\)\\displaystyle\\sum\_\{a:\\pi\_\{a\}^\{P\}\>0\}\\pi\_\{a\}^\{P\}\\KL\(P\_\{a\}\\\|Q\_\{a\}\)≤ε,\\displaystyle\\leq\\varepsilon,\(8\)∑a:πaP\>0πaP∥𝔼P\[h∣a\]−∇A\(wa\)∥\\displaystyle\\sum\_\{a:\\pi\_\{a\}^\{P\}\>0\}\\pi\_\{a\}^\{P\}\\lVert\\mathbb\{E\}\_\{P\}\[h\\mid a\]\-\\nabla A\(w\_\{a\}\)\\rVert≤R0​2​ε\.\\displaystyle\\leq R\_\{0\}\\sqrt\{2\\varepsilon\}\.\(9\)

###### Proof\.

BecausePPandQQhave the same hidden\-state marginal,

KL\(P∥Q\)=𝔼PHKL\{P\(Y∣H\)∥q\(Y∣H\)\}=𝔼P\[−logq\(Y∣H\)\]−HP\(Y∣H\),\\KL\(P\\\|Q\)=\\mathbb\{E\}\_\{P\_\{H\}\}\\KL\\\{P\(Y\\mid H\)\\\|q\(Y\\mid H\)\\\}=\\mathbb\{E\}\_\{P\}\[\-\\log q\(Y\\mid H\)\]\-H\_\{P\}\(Y\\mid H\),which proves Eq\. \([2](https://arxiv.org/html/2609.11063#as1_S2.E2)\)\. Substitution of Eq\. \([1](https://arxiv.org/html/2609.11063#as1_S2.E1)\) in the definition ofQQgives Eq\. \([3](https://arxiv.org/html/2609.11063#as1_S2.E3)\)\. Integrating itsaath slice yieldsπaQ=exp⁡\{ba\+A⁡\(wa\)\}\\pi\_\{a\}^\{Q\}=\\exp\\\{b\_\{a\}\+A\(w\_\{a\}\)\\\}, and division by this marginal gives Eq\. \([4](https://arxiv.org/html/2609.11063#as1_S2.E4)\)\. Differentiation of the two log\-partition functions gives their means and covariances, proving Eqs\. \([5](https://arxiv.org/html/2609.11063#as1_S2.E5)\) and \([6](https://arxiv.org/html/2609.11063#as1_S2.E6)\)\. Equation \([7](https://arxiv.org/html/2609.11063#as1_S2.E7)\) is the chain rule for relative entropy after factorising both laws byYY\.

Dropping the non\-negative marginal divergence in Eq\. \([7](https://arxiv.org/html/2609.11063#as1_S2.E7)\) proves Eq\. \([8](https://arxiv.org/html/2609.11063#as1_S2.E8)\)\. For every class withπaP\>0\\pi\_\{a\}^\{P\}\>0, Pinsker’s inequality and the boundedness ofhhgive

∥𝔼Pa​h−𝔼Qa​h∥≤2​R0​∥Pa−Qa∥TV≤R0​2KL\(Pa∥Qa\)\.\\lVert\\mathbb\{E\}\_\{P\_\{a\}\}h\-\\mathbb\{E\}\_\{Q\_\{a\}\}h\\rVert\\leq 2R\_\{0\}\\lVert P\_\{a\}\-Q\_\{a\}\\rVert\_\{\\rm TV\}\\leq R\_\{0\}\\sqrt\{2\\KL\(P\_\{a\}\\\|Q\_\{a\}\)\}\.Average with weightsπaP\\pi\_\{a\}^\{P\}, apply Cauchy–Schwarz and use Eq\. \([8](https://arxiv.org/html/2609.11063#as1_S2.E8)\)\. This proves Eq\. \([9](https://arxiv.org/html/2609.11063#as1_S2.E9)\)\. ∎

The theorem explains why the output Fisher used in this work is tied to the training objective: it is the conditional curvature of the Gibbs law induced by the affine\-softmax read\-out\. The class\-conditional hidden covariance is the opposite conditional curvature of the same joint law\. The identity is exact at a checkpoint; it does not require the representation itself to have been trained by a particular optimiser\.

For a lawpp, writep↓p^\{\\downarrow\}for its decreasing rearrangement and define the smooth response at resolutionα\>0\\alpha\>0by

dfp⁡\(α\)=∑kpk↓pk↓\+α\.\\dfop\_\{p\}\(\\alpha\)=\\sum\_\{k\}\\frac\{p\_\{k\}^\{\\downarrow\}\}\{p\_\{k\}^\{\\downarrow\}\+\\alpha\}\.For read\-out rowsW=\(wa\)W=\(w\_\{a\}\)letHp,W=Covp⁡\(w\)H\_\{p,W\}=\\Cov\_\{p\}\(w\)and letRW=infgmaxa⁡∥wa−g∥R\_\{W\}=\\inf\_\{g\}\\max\_\{a\}\\lVert w\_\{a\}\-g\\rVertbe their gauge\-optimal radius\.

###### Theorem 3\(determination by predictive risk\)\.

Letppandqqbe laws on the same outcome space and putρ=∥p−q∥2\\rho=\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}\. Then

∥p↓−q↓∥1\\displaystyle\\lVert p^\{\\downarrow\}\-q^\{\\downarrow\}\\rVert\_\{1\}≤2​ρ,\\displaystyle\\leq 2\\rho,\(10\)\|dfp⁡\(α\)−dfq⁡\(α\)\|\\displaystyle\|\\dfop\_\{p\}\(\\alpha\)\-\\dfop\_\{q\}\(\\alpha\)\|≤2​ρα,\\displaystyle\\leq\\frac\{2\\rho\}\{\\alpha\},\(11\)∥Hp,W−Hq,W∥op\\displaystyle\\lVert H\_\{p,W\}\-H\_\{q,W\}\\rVert\_\{\\rm op\}≤6​RW2​ρ\.\\displaystyle\\leq 6R\_\{W\}^\{2\}\\rho\.\(12\)Hence every eigenvalue of the fixed\-read\-out metric changes by at most6​RW2​ρ6R\_\{W\}^\{2\}\\rho\. Moreoverρ2≤KL\(p∥q\)\\rho^\{2\}\\leq\\KL\(p\\\|q\)\. If two models have excess risksεm\\varepsilon\_\{m\}andεm′\\varepsilon\_\{m^\{\\prime\}\}to a common conditional reference law, their battery\-averaged profile, response and fixed\-read\-out metric discrepancies are therefore bounded at rateO⁡\(εm\+εm′\)O\(\\sqrt\{\\varepsilon\_\{m\}\}\+\\sqrt\{\\varepsilon\_\{m^\{\\prime\}\}\}\)\. The relational counterpart is proved in Theorem[6](https://arxiv.org/html/2609.11063#as1_Thmtheorem6)\.

###### Proof\.

For scalarsx≥x′x\\geq x^\{\\prime\}andy≥y′y\\geq y^\{\\prime\},\|x−y\|\+\|x′−y′\|≤\|x−y′\|\+\|x′−y\|\|x\-y\|\+\|x^\{\\prime\}\-y^\{\\prime\}\|\\leq\|x\-y^\{\\prime\}\|\+\|x^\{\\prime\}\-y\|\. Repeatedly uncrossing a matching shows that sorted matching minimises total absolute deviation, so∥p↓−q↓∥1≤∥p−q∥1\\lVert p^\{\\downarrow\}\-q^\{\\downarrow\}\\rVert\_\{1\}\\leq\\lVert p\-q\\rVert\_\{1\}\. Cauchy–Schwarz gives

∥p−q∥1=∑a\|pa−qa\|​\(pa\+qa\)≤2​ρ,\\lVert p\-q\\rVert\_\{1\}=\\sum\_\{a\}\|\\sqrt\{p\_\{a\}\}\-\\sqrt\{q\_\{a\}\}\|\(\\sqrt\{p\_\{a\}\}\+\\sqrt\{q\_\{a\}\}\)\\leq 2\\rho,proving Eq\. \([10](https://arxiv.org/html/2609.11063#as1_S2.E10)\)\. Sincet↦t/\(t\+α\)t\\mapsto t/\(t\+\\alpha\)is1/α1/\\alpha\-Lipschitz on\[0,∞\)\[0,\\infty\), Eq\. \([11](https://arxiv.org/html/2609.11063#as1_S2.E11)\) follows\.

Shift all read\-out rows by the gauge minimiser, which leaves the covariance unchanged, so that∥wa∥≤RW\\lVert w\_\{a\}\\rVert\\leq R\_\{W\}\. Putcp=∑apa​wac\_\{p\}=\\sum\_\{a\}p\_\{a\}w\_\{a\}\. For any unit vectoruu,

\|u⊤​\(Hp,W−Hq,W\)​u\|\\displaystyle\|u^\{\\top\}\(H\_\{p,W\}\-H\_\{q,W\}\)u\|≤∑a\|pa−qa\|​⟨wa,u⟩2\+\|⟨cp,u⟩2−⟨cq,u⟩2\|\\displaystyle\\leq\\sum\_\{a\}\|p\_\{a\}\-q\_\{a\}\|\\langle w\_\{a\},u\\rangle^\{2\}\+\|\\langle c\_\{p\},u\\rangle^\{2\}\-\\langle c\_\{q\},u\\rangle^\{2\}\|≤2​RW2​ρ\+4​RW2​ρ\.\\displaystyle\\leq 2R\_\{W\}^\{2\}\\rho\+4R\_\{W\}^\{2\}\\rho\.Taking the supremum proves Eq\. \([12](https://arxiv.org/html/2609.11063#as1_S2.E12)\); Weyl’s inequality gives the eigenvalue statement\. Finally, withBC⁡\(p,q\)=∑apa​qa\\mathrm\{BC\}\(p,q\)=\\sum\_\{a\}\\sqrt\{p\_\{a\}q\_\{a\}\},

KL\(p∥q\)=−2∑apalogqa/pa≥2\{1−BC\(p,q\)\}=ρ2,\\KL\(p\\\|q\)=\-2\\sum\_\{a\}p\_\{a\}\\log\\sqrt\{q\_\{a\}/p\_\{a\}\}\\geq 2\\\{1\-\\mathrm\{BC\}\(p,q\)\\\}=\\rho^\{2\},usinglog⁡t≤t−1\\log t\\leq t\-1\. Jensen’s inequality and the triangle inequality through the common reference law give the battery\-averaged conclusion\. ∎

### 3Identification of the read\-out geometry

This section supports the main text’s identifiability results: local identification with quantitative curvature, global identification with explicit constants, and the non\-identifiability of activation geometry\.

Fix a strictly positive reference lawρ\\rhoon a common outcome space of sizeVVand define the gauge\-free weighted natural\-coordinate space

Dρ=diag⁡\(ρ\),ℋρ=\{u∈ℝV:ρ⊤​u=0\}\.D\_\{\\rho\}=\\operatorname\{diag\}\(\\rho\),\\qquad\\mathcal\{H\}\_\{\\rho\}=\\\{u\\in\\mathbb\{R\}^\{V\}:\\sqrt\{\\rho\}^\{\\,\\top\}u=0\\\}\.Forθ∈ℋρ\\theta\\in\\mathcal\{H\}\_\{\\rho\}, let

pθ=softmax\(Dρ−1/2θ\),Hθ=Dρ−1/2\{diag\(pθ\)−pθpθ⊤\}Dρ−1/2\.p\_\{\\theta\}=\\operatorname\{softmax\}\(D\_\{\\rho\}^\{\-1/2\}\\theta\),\\qquad H\_\{\\theta\}=D\_\{\\rho\}^\{\-1/2\}\\\{\\operatorname\{diag\}\(p\_\{\\theta\}\)\-p\_\{\\theta\}p\_\{\\theta\}^\{\\top\}\\\}D\_\{\\rho\}^\{\-1/2\}\.The Fisher operatorHθH\_\{\\theta\}is positive definite onℋρ\\mathcal\{H\}\_\{\\rho\}\. LetXXhave finite support with strictly positive weights, and suppose the target conditional family has the resolved representation

πX=pθX,θX=β\+U​zX,U⊤​U=Ir,𝔼​zX=0,Σz=Cov⁡\(zX\)≻0\.\\pi\_\{X\}=p\_\{\\theta\_\{X\}\},\\qquad\\theta\_\{X\}=\\beta\+Uz\_\{X\},\\qquad U^\{\\top\}U=I\_\{r\},\\qquad\\mathbb\{E\}z\_\{X\}=0,\\qquad\\Sigma\_\{z\}=\\Cov\(z\_\{X\}\)\\succ 0\.\(13\)WriteT=col⁡\(U\)T=\\operatorname\{col\}\(U\)\. For add\-dimensional candidate subspaceS⊂ℋρS\\subset\\mathcal\{H\}\_\{\\rho\},d≥rd\\geq r, define the fully profiled risk

𝒦D,d\(S\)=infb∈ℋρinff:X→S𝔼XD\(πX,pb\+f⁡\(X\)\),D∈\{KL,h2\},\\mathcal\{K\}\_\{D,d\}\(S\)=\\inf\_\{b\\in\\mathcal\{H\}\_\{\\rho\}\}\\inf\_\{f:X\\to S\}\\mathbb\{E\}\_\{X\}D\(\\pi\_\{X\},p\_\{b\+f\(X\)\}\),\\qquad D\\in\\\{\\KL,h^\{2\}\\\},\(14\)whereh⁡\(p,q\)=∥p−q∥2h\(p,q\)=\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}\. The zero\-risk manifold expected for an overcomplete read\-out is

ℳd=\{S∈Gr⁡\(d,ℋρ\):T⊂S\},Δd​\(S,T\)=∥PS⟂​PT∥F\.\\mathcal\{M\}\_\{d\}=\\\{S\\in\\operatorname\{Gr\}\(d,\\mathcal\{H\}\_\{\\rho\}\):T\\subset S\\\},\\qquad\\Delta\_\{d\}\(S,T\)=\\lVert P\_\{S^\{\\perp\}\}P\_\{T\}\\rVert\_\{\\rm F\}\.When differentiating the profile, fix the unique moving gaugeb∈S⟂∩ℋρb\\in S^\{\\perp\}\\cap\\mathcal\{H\}\_\{\\rho\}\.

###### Proof map\.

The local theorem first profiles the intercept and context coordinates and identifies the Fisher Schur complement that supplies curvature normal to the containment fibre\. The inverse\-softmax lemma then separates uniformly near candidates from candidates with a fixed Hellinger penalty\. The global theorem combines those two branches, after which the recovery and capture corollaries translate predictive error into subspace and partition guarantees\.

###### Theorem 4\(local identification by profiled Fisher curvature\)\.

FixS0∈ℳdS\_\{0\}\\in\\mathcal\{M\}\_\{d\}with orthonormal basisV=\(U,E\)V=\(U,E\)\. A Grassmann tangent isB=\(BT,BE\)B=\(B\_\{T\},B\_\{E\}\)with columns inS0⟂∩ℋρS\_\{0\}^\{\\perp\}\\cap\\mathcal\{H\}\_\{\\rho\}, and the orthonormal retraction isVB\(t\)=\(V\+tB\)\(I\+t2B⊤B\)−1/2V\_\{B\}\(t\)=\(V\+tB\)\(I\+t^\{2\}B^\{\\top\}B\)^\{\-1/2\}\. Define

RX,S0\\displaystyle R\_\{X,S\_\{0\}\}=HθX−HθX​V​\(V⊤​HθX​V\)−1​V⊤​HθX,\\displaystyle=H\_\{\\theta\_\{X\}\}\-H\_\{\\theta\_\{X\}\}V\(V^\{\\top\}H\_\{\\theta\_\{X\}\}V\)^\{\-1\}V^\{\\top\}H\_\{\\theta\_\{X\}\},\(15\)QS0​\(BT\)\\displaystyle Q\_\{S\_\{0\}\}\(B\_\{T\}\)=infa∈S0⟂∩ℋρ𝔼​\(BT​zX−a\)⊤​RX,S0​\(BT​zX−a\)\.\\displaystyle=\\inf\_\{a\\in S\_\{0\}^\{\\perp\}\\cap\\mathcal\{H\}\_\{\\rho\}\}\\mathbb\{E\}\(B\_\{T\}z\_\{X\}\-a\)^\{\\top\}R\_\{X,S\_\{0\}\}\(B\_\{T\}z\_\{X\}\-a\)\.\(16\)Then

𝒦KL,d​\(col⁡VB​\(t\)\)\\displaystyle\\mathcal\{K\}\_\{\\KL,d\}\(\\operatorname\{col\}V\_\{B\}\(t\)\)=12​t2​QS0​\(BT\)\+o⁡\(t2\),\\displaystyle=\\tfrac\{1\}\{2\}t^\{2\}Q\_\{S\_\{0\}\}\(B\_\{T\}\)\+o\(t^\{2\}\),\(17\)𝒦h2,d​\(col⁡VB​\(t\)\)\\displaystyle\\mathcal\{K\}\_\{h^\{2\},d\}\(\\operatorname\{col\}V\_\{B\}\(t\)\)=14​t2​QS0​\(BT\)\+o⁡\(t2\)\.\\displaystyle=\\tfrac\{1\}\{4\}t^\{2\}Q\_\{S\_\{0\}\}\(B\_\{T\}\)\+o\(t^\{2\}\)\.\(18\)IfmH=minX⁡λmin​\(HθX\|ℋρ\)m\_\{H\}=\\min\_\{X\}\\lambda\_\{\\min\}\(H\_\{\\theta\_\{X\}\}\|\_\{\\mathcal\{H\}\_\{\\rho\}\}\)andmz=λmin​\(Σz\)m\_\{z\}=\\lambda\_\{\\min\}\(\\Sigma\_\{z\}\), then

QS0​\(BT\)≥mH​mz​∥BT∥F2\.Q\_\{S\_\{0\}\}\(B\_\{T\}\)\\geq m\_\{H\}m\_\{z\}\\lVert B\_\{T\}\\rVert\_\{\\rm F\}^\{2\}\.\(19\)Thus the Hessian kernel is exactly the tangent space ofℳd\\mathcal\{M\}\_\{d\}and every normal direction has strictly positive curvature\.

###### Proof\.

AtS0S\_\{0\}, choose the identifiable interceptb0=PS0⟂​βb\_\{0\}=P\_\{S\_\{0\}^\{\\perp\}\}\\beta\. Rotating the basis produces a common normal termB​V⊤​βBV^\{\\top\}\\beta, which is cancelled exactly by the freely optimised intercept\. The remaining first\-order candidate\-minus\-target displacement isBT​zX−a\+V​cXB\_\{T\}z\_\{X\}\-a\+Vc\_\{X\}, whereaais the normal change of the intercept andcXc\_\{X\}is the freely optimised context coordinate\. The local expansions are

KL⁡\(pθ,pθ\+u\)=12​u⊤​Hθ​u\+o⁡\(∥u∥2\),h2​\(pθ,pθ\+u\)=14​u⊤​Hθ​u\+o⁡\(∥u∥2\)\.\\KL\(p\_\{\\theta\},p\_\{\\theta\+u\}\)=\\tfrac\{1\}\{2\}u^\{\\top\}H\_\{\\theta\}u\+o\(\\lVert u\\rVert^\{2\}\),\\qquad h^\{2\}\(p\_\{\\theta\},p\_\{\\theta\+u\}\)=\\tfrac\{1\}\{4\}u^\{\\top\}H\_\{\\theta\}u\+o\(\\lVert u\\rVert^\{2\}\)\.Minimising first overcXc\_\{X\}gives the Fisher Schur complement Eq\. \([15](https://arxiv.org/html/2609.11063#as1_S3.E15)\); minimising overaagives Eq\. \([16](https://arxiv.org/html/2609.11063#as1_S3.E16)\)\. After the moving gauge is fixed, the nuisance Hessian is positive definite, so the implicit\-function theorem gives a unique smooth local stationary branch with the displayed quadratic expansions\.

It remains to identify this local branch with the fully profiled infimum\. Its risk isO⁡\(t2\)O\(t^\{2\}\)\. For any sequencetn→0t\_\{n\}\\to 0, a corresponding sequence of near\-minimisers has mean squared\-Hellinger error tending to zero, usingh2≤KLh^\{2\}\\leq\\KLin the KL case\. Finite context support with positive weights makes this convergence contextwise\. Every target probability is positive, so the candidate probabilities eventually lie in a common compact subset of the open simplex\. The gauge\-fixed inverse\-softmax map is bounded there, as are the moving\-gauge intercepts and context coordinates\. Every subsequential limit is therefore an exact representation atS0S\_\{0\}and lies in the implicit\-function neighbourhood\. The same compactness gives attainment for all sufficiently smalltt; uniqueness then makes every global minimiser the local branch\. This proves Eqs\. \([17](https://arxiv.org/html/2609.11063#as1_S3.E17)\) and \([18](https://arxiv.org/html/2609.11063#as1_S3.E18)\)\.

Forv∈S0⟂∩ℋρv\\in S\_\{0\}^\{\\perp\}\\cap\\mathcal\{H\}\_\{\\rho\},

v⊤​RX,S0​v=minc⁡\(v\+V​c\)⊤​HθX​\(v\+V​c\)≥mH​minc​∥v\+V​c∥2=mH​∥v∥2\.v^\{\\top\}R\_\{X,S\_\{0\}\}v=\\min\_\{c\}\(v\+Vc\)^\{\\top\}H\_\{\\theta\_\{X\}\}\(v\+Vc\)\\geq m\_\{H\}\\min\_\{c\}\\lVert v\+Vc\\rVert^\{2\}=m\_\{H\}\\lVert v\\rVert^\{2\}\.Therefore

QS0​\(BT\)≥mH​infa𝔼​∥BT​zX−a∥2=mH​𝔼​∥BT​zX∥2≥mH​mz​∥BT∥F2,Q\_\{S\_\{0\}\}\(B\_\{T\}\)\\geq m\_\{H\}\\inf\_\{a\}\\mathbb\{E\}\\lVert B\_\{T\}z\_\{X\}\-a\\rVert^\{2\}=m\_\{H\}\\mathbb\{E\}\\lVert B\_\{T\}z\_\{X\}\\rVert^\{2\}\\geq m\_\{H\}m\_\{z\}\\lVert B\_\{T\}\\rVert\_\{\\rm F\}^\{2\},where centring makesa=0a=0optimal in the Euclidean lower bound\. The form vanishes exactly whenBT=0B\_\{T\}=0, which is precisely the tangent condition for retainingT⊂ST\\subset S\. ∎

###### Lemma 1\(inverse\-softmax dichotomy\)\.

Letp=pθp=p\_\{\\theta\}andq=pηq=p\_\{\\eta\}forθ,η∈ℋρ\\theta,\\eta\\in\\mathcal\{H\}\_\{\\rho\}, and supposemini⁡pi≥p∗\>0\\min\_\{i\}p\_\{i\}\\geq p\_\{\*\}\>0\. Then either

h2​\(p,q\)≥p∗264​∥θ−η∥2,h^\{2\}\(p,q\)\\geq\\frac\{p\_\{\*\}^\{2\}\}\{64\}\\lVert\\theta\-\\eta\\rVert^\{2\},\(20\)or

h2​\(p,q\)\>p∗4\.h^\{2\}\(p,q\)\>\\frac\{p\_\{\*\}\}\{4\}\.\(21\)The first branch holds wheneverh⁡\(p,q\)≤p∗/2h\(p,q\)\\leq\\sqrt\{p\_\{\*\}\}/2\.

###### Proof\.

The gauge\-fixed inverse\-softmax chart isθ=Pℋρ​Dρ1/2​log⁡p\\theta=P\_\{\\mathcal\{H\}\_\{\\rho\}\}D\_\{\\rho\}^\{1/2\}\\log pand similarly forη\\eta\. Ifh⁡\(p,q\)≤p∗/2h\(p,q\)\\leq\\sqrt\{p\_\{\*\}\}/2, then coordinatewiseqi≥p∗/4q\_\{i\}\\geq p\_\{\*\}/4\. Contractivity of orthogonal projection, the mean\-value theorem on\[p∗/4,1\]\[p\_\{\*\}/4,1\]and\|pi−qi\|≤2​\|pi−qi\|\|p\_\{i\}\-q\_\{i\}\|\\leq 2\|\\sqrt\{p\_\{i\}\}\-\\sqrt\{q\_\{i\}\}\|give

∥θ−η∥≤∥log⁡p−log⁡q∥2≤4p∗​∥p−q∥2≤8p∗​h​\(p,q\)\.\\lVert\\theta\-\\eta\\rVert\\leq\\lVert\\log p\-\\log q\\rVert\_\{2\}\\leq\\frac\{4\}\{p\_\{\*\}\}\\lVert p\-q\\rVert\_\{2\}\\leq\\frac\{8\}\{p\_\{\*\}\}h\(p,q\)\.Squaring proves Eq\. \([20](https://arxiv.org/html/2609.11063#as1_S3.E20)\)\. Otherwiseh2​\(p,q\)\>p∗/4h^\{2\}\(p,q\)\>p\_\{\*\}/4, which is Eq\. \([21](https://arxiv.org/html/2609.11063#as1_S3.E21)\)\. ∎

###### Theorem 5\(global identification and the overcomplete fibre\)\.

AssumeXXhas finite support, put

p∗=minX,iπX\(i\)\>0,w∗=minx:PX​\(x\)\>0PX\(x\)\>0,p\_\{\*\}=\\min\_\{X,i\}\\pi\_\{X\}\(i\)\>0,\\qquad w\_\{\*\}=\\min\_\{x:P\_\{X\}\(x\)\>0\}P\_\{X\}\(x\)\>0,and retain Eq\. \([13](https://arxiv.org/html/2609.11063#as1_S3.E13)\)\. For eitherD=KLD=\\KLorD=h2D=h^\{2\},

𝒦D,d​\(S\)≥cD,d​Δd​\(S,T\)2,cD,d:=min⁡\{p∗2​mz64,w∗​p∗4​r\}\>0\.\\mathcal\{K\}\_\{D,d\}\(S\)\\geq c\_\{D,d\}\\Delta\_\{d\}\(S,T\)^\{2\},\\qquad c\_\{D,d\}:=\\min\\left\\\{\\frac\{p\_\{\*\}^\{2\}m\_\{z\}\}\{64\},\\frac\{w\_\{\*\}p\_\{\*\}\}\{4r\}\\right\\\}\>0\.\(22\)Moreover𝒦D,d​\(S\)=0\\mathcal\{K\}\_\{D,d\}\(S\)=0if and only ifT⊂ST\\subset S\. Hence a rank\-matched read\-out identifies the unique subspaceTT, while an overcomplete read\-out is identified exactly up to the containment fibreℳd\\mathcal\{M\}\_\{d\}\.

###### Proof\.

IfT⊂ST\\subset S, chooseb=βb=\\betaandf⁡\(X\)=U​zXf\(X\)=Uz\_\{X\}in Eq\. \([14](https://arxiv.org/html/2609.11063#as1_S3.E14)\)\. Conversely consider any candidate natural parametersηX=b\+f⁡\(X\)\\eta\_\{X\}=b\+f\(X\)withf⁡\(X\)∈Sf\(X\)\\in S\. SinceKL≥h2\\KL\\geq h^\{2\}, it is enough to lower\-bound the Hellinger risk\.

If the near branch of Lemma[1](https://arxiv.org/html/2609.11063#as1_Thmlemma1)holds at every context, orthogonal projection,𝔼​zX=0\\mathbb\{E\}z\_\{X\}=0andΣz⪰mz​I\\Sigma\_\{z\}\\succeq m\_\{z\}Igive

𝔼​∥θX−ηX∥2\\displaystyle\\mathbb\{E\}\\lVert\\theta\_\{X\}\-\\eta\_\{X\}\\rVert^\{2\}≥𝔼​∥PS⟂​\(β\+U​zX−b\)∥2\\displaystyle\\geq\\mathbb\{E\}\\lVert P\_\{S^\{\\perp\}\}\(\\beta\+Uz\_\{X\}\-b\)\\rVert^\{2\}≥mz​∥PS⟂​U∥F2=mz​Δd​\(S,T\)2\.\\displaystyle\\geq m\_\{z\}\\lVert P\_\{S^\{\\perp\}\}U\\rVert\_\{\\rm F\}^\{2\}=m\_\{z\}\\Delta\_\{d\}\(S,T\)^\{2\}\.The candidate risk is therefore at leastp∗2​mz​Δd​\(S,T\)2/64p\_\{\*\}^\{2\}m\_\{z\}\\Delta\_\{d\}\(S,T\)^\{2\}/64\. If the far branch holds at any context, that context has weight at leastw∗w\_\{\*\}, so the candidate risk exceedsw∗​p∗/4w\_\{\*\}p\_\{\*\}/4\. SinceΔd​\(S,T\)2≤r\\Delta\_\{d\}\(S,T\)^\{2\}\\leq r, this is at leastw∗​p∗​Δd​\(S,T\)2/\(4​r\)w\_\{\*\}p\_\{\*\}\\Delta\_\{d\}\(S,T\)^\{2\}/\(4r\)\. Taking the smaller branch constant and then the profile infimum proves Eq\. \([22](https://arxiv.org/html/2609.11063#as1_S3.E22)\) and the stated zero set\. ∎

###### Corollary 1\(quantitative approximate recovery\)\.

Letqqbe the language law,qrq\_\{r\}a rank\-rrlanguage core, andpmp\_\{m\}a model law\. WriteRHR\_\{H\}for expected squared root\-probability distance andΔd=∥PSm⟂​PTr∥F\\Delta\_\{d\}=\\lVert P\_\{S\_\{m\}^\{\\perp\}\}P\_\{T\_\{r\}\}\\rVert\_\{F\}\. Suppose

RH\(q,qr\)≤ζr,𝔼xKL\(qx∥pm,x\)≤εm,ch2,rΔd2≤RH\(qr,pm\),ch2,r\>0\.R\_\{H\}\(q,q\_\{r\}\)\\leq\\zeta\_\{r\},\\qquad\\mathbb\{E\}\_\{x\}\\mathrm\{KL\}\(q\_\{x\}\\\|p\_\{m,x\}\)\\leq\\varepsilon\_\{m\},\\qquad c\_\{h^\{2\},r\}\\Delta\_\{d\}^\{2\}\\leq R\_\{H\}\(q\_\{r\},p\_\{m\}\),\\qquad c\_\{h^\{2\},r\}\>0\.Then

Δd≤ζr\+εmch2,r\.\\Delta\_\{d\}\\leq\\frac\{\\sqrt\{\\zeta\_\{r\}\}\+\\sqrt\{\\varepsilon\_\{m\}\}\}\{\\sqrt\{c\_\{h^\{2\},r\}\}\}\.\(23\)For the finite\-support global theorem one may takech2,r=min⁡\{p∗2​mz/64,w∗​p∗/\(4​r\)\}c\_\{h^\{2\},r\}=\\min\\\{p\_\{\*\}^\{2\}m\_\{z\}/64,\\,w\_\{\*\}p\_\{\*\}/\(4r\)\\\}under its stated inverse\-chart, context\-floor and covariance hypotheses\.

###### Proof\.

Minkowski’s inequality in the joint context–outcome root space and the Hellinger–KL inequality give

RH​\(qr,pm\)≤RH​\(qr,q\)\+RH​\(q,pm\)≤ζr\+εm\.\\sqrt\{R\_\{H\}\(q\_\{r\},p\_\{m\}\)\}\\leq\\sqrt\{R\_\{H\}\(q\_\{r\},q\)\}\+\\sqrt\{R\_\{H\}\(q,p\_\{m\}\)\}\\leq\\sqrt\{\\zeta\_\{r\}\}\+\\sqrt\{\\varepsilon\_\{m\}\}\.The margin premise and nonnegativity givech2,r​Δd≤RH​\(qr,pm\)\\sqrt\{c\_\{h^\{2\},r\}\}\\,\\Delta\_\{d\}\\leq\\sqrt\{R\_\{H\}\(q\_\{r\},p\_\{m\}\)\}\. Division by the positive square root proves Eq\. \([23](https://arxiv.org/html/2609.11063#as1_S3.E23)\)\. The displayed finite constant is exactly the near/far global margin in Theorem[5](https://arxiv.org/html/2609.11063#as1_Thmtheorem5); it is not introduced by this final scalar step\. ∎

###### Corollary 2\(capture inherited at the recovery rate\)\.

Let the rank\-rrlanguage coreTTand a rank\-rrmodel coreSmS\_\{m\}be at chordal distance at mostBmB\_\{m\}\. For a fixed partition projector0⪯ΠG⪯I0\\preceq\\Pi\_\{G\}\\preceq I, writeqG​\(S\)=tr⁡\(ΠG​PS\)/rq\_\{G\}\(S\)=\\operatorname\{tr\}\(\\Pi\_\{G\}P\_\{S\}\)/rfor mean capture andqmin,G​\(S\)q\_\{\\min,G\}\(S\)for the smallest eigenvalue of its compression toSS\. Then

qG​\(Sm\)≥qG​\(T\)−Bmr,qmin,G​\(Sm\)≥qmin,G​\(T\)−min⁡\{1,2​Bm\}\.q\_\{G\}\(S\_\{m\}\)\\geq q\_\{G\}\(T\)\-\\frac\{B\_\{m\}\}\{\\sqrt\{r\}\},\\qquad q\_\{\\min,G\}\(S\_\{m\}\)\\geq q\_\{\\min,G\}\(T\)\-\\min\\\{1,2B\_\{m\}\\\}\.\(24\)In particular Corollary[1](https://arxiv.org/html/2609.11063#as1_Thmcorollary1)suppliesBm=\(ζr\+εm\)/ch2,rB\_\{m\}=\(\\sqrt\{\\zeta\_\{r\}\}\+\\sqrt\{\\varepsilon\_\{m\}\}\)/\\sqrt\{c\_\{h^\{2\},r\}\}\.

###### Proof\.

Ifθ1,…,θr\\theta\_\{1\},\\ldots,\\theta\_\{r\}are the principal angles,PT−PSmP\_\{T\}\-P\_\{S\_\{m\}\}has nonzero eigenvalues±sin⁡θi\\pm\\sin\\theta\_\{i\}\. Positivity and contractivity ofΠG\\Pi\_\{G\}give

\|tr⁡\(ΠG​\(PT−PSm\)\)\|≤∑isin⁡θi≤r​\(∑isin2⁡θi\)1/2=r​dch​\(Sm,T\)\.\|\\operatorname\{tr\}\(\\Pi\_\{G\}\(P\_\{T\}\-P\_\{S\_\{m\}\}\)\)\|\\leq\\sum\_\{i\}\\sin\\theta\_\{i\}\\leq\\sqrt\{r\}\\left\(\\sum\_\{i\}\\sin^\{2\}\\theta\_\{i\}\\right\)^\{1/2\}=\\sqrt\{r\}\\,d\_\{\\mathrm\{ch\}\}\(S\_\{m\},T\)\.Division byrrproves the mean bound\. For worst capture,

P​ΠG​P−Q​ΠG​Q=\(P−Q\)​ΠG​P\+Q​ΠG​\(P−Q\),P\\Pi\_\{G\}P\-Q\\Pi\_\{G\}Q=\(P\-Q\)\\Pi\_\{G\}P\+Q\\Pi\_\{G\}\(P\-Q\),so the compression difference has operator norm at most2​∥P−Q∥op2\\lVert P\-Q\\rVert\_\{\\mathrm\{op\}\}\. Weyl’s inequality and∥P−Q∥op=maxi⁡sin⁡θi≤dch​\(Sm,T\)\\lVert P\-Q\\rVert\_\{\\mathrm\{op\}\}=\\max\_\{i\}\\sin\\theta\_\{i\}\\leq d\_\{\\mathrm\{ch\}\}\(S\_\{m\},T\)give the2​Bm2B\_\{m\}bound\. Both captures lie in\[0,1\]\[0,1\], which gives the independent cap by one\. ∎

The measured profiled\-curvature margin is positive in every evaluated configuration\. The separate fractional\-frame measurements concern spectral resolution rather than identification and are reported with the spectral measurements below\.

The contrast that completes the identification picture is that no analogous statement holds on the hidden face\.

###### Proposition 1\(activation geometry is not behaviourally identifiable\)\.

Let a network factor asf=g∘hf=g\\circ hthrough add\-dimensional hidden layer\. For everyA∈GL⁡\(d\)A\\in\\mathrm\{GL\}\(d\), the factorisationf=\(g∘A−1\)∘\(A​h\)f=\(g\\circ A^\{\-1\}\)\\circ\(Ah\)has identical conditional laws\. IfZZcontains sampled activations as rows, its Gram matrix becomesZ​A⊤​A​Z⊤ZA^\{\\top\}AZ^\{\\top\}; unlessAAis conformal on the sampled activation span \(or on the span of activation differences for distance geometry\), Euclidean and generally cosine relational geometry changes\. Therefore behaviour does not determine a unique Euclidean activation geometry, whereas the output geometry is a functional of the conditional laws\.

###### Proof\.

The two factorisations compute exactly the same function\. The Gram transformation is direct substitution\. Non\-conformal choices ofA⊤​AA^\{\\top\}Achange some inner products or squared distances whenever the sampled span has dimension at least two\. In the gauge test, re\-chartings with condition numbers up to10410^\{4\}leave output geometry constant to machine precision while the evaluated activation measures degrade\. ∎

###### Profiled\-loss experimental protocol\.

The rank\-32 reference is constructed from a Pythia\-6\.9B teacher: its marginal law, affine mean and leading weighted log\-probability modes are estimated on 500 calibration contexts\. The target probabilities are those of this resolved reference family\. Principal\-vector paths move from the reference subspace towards the read\-outs of Pythia 70M, 160M, 410M and 1\.4B; one random\-normal control per model preserves every principal angle\. Path fractions are00,0\.1250\.125,0\.250\.25,0\.50\.5and11\.

At each fraction, the affine intercept and context coordinates are jointly profiled on each of two disjoint 100\-context calibration folds, followed by alternating convex coordinate/intercept refinements\. Each fitted intercept is then frozen and only coordinates are optimised on 100 held\-out contexts, disjoint from both calibration folds and the 500\-context reference fit\. Optimisation uses float64 and L\-BFGS\. Maximum absolute and root\-mean\-square gradient tolerances are respectively10−310^\{\-3\}and10−510^\{\-5\}for the calibration intercept,5×10−45\\times 10^\{\-4\}and5×10−55\\times 10^\{\-5\}for calibration coordinates, and3×10−43\\times 10^\{\-4\}and3×10−53\\times 10^\{\-5\}for held\-out coordinates\. The two held\-out losses are averaged\. The local slope is fitted through the origin against squared chordal distance at fractions0\.1250\.125,0\.250\.25and0\.50\.5, after subtracting the zero\-angle loss\. Its interval uses 1,000 held\-out\-context bootstrap draws shared across path fractions; fold\-specific estimates are retained\. This tests the resolved reference family along the specified paths\.

#### 3\.1Causal language assignment

The controlled comparison crossed three architectures and two capacities with eight independent pairs of 64\-context, 64\-outcome synthetic languages\. Two separate pilot pairs selected the training schedule for each architecture–capacity cell using excess predictive risk alone\. The same selected schedule was applied to every test pair\. All 96 labelled test runs met the common requirementϵ<Δ2/32\\epsilon<\\Delta^\{2\}/32, whereΔ2\\Delta^\{2\}is the mean squared difference between the two true scaled chord\-distance matrices\. This defines a common accuracy requirement rather than exact equality of training loss\.

For a learned distance matrix, the assigned\-law advantage is its mean squared error relative to the counterfactual law minus that relative to the assigned law, divided byΔ2\\Delta^\{2\}\. The mean over the eight language pairs is0\.9978360\.997836\(95% language\-pair bootstrap interval0\.9977390\.997739–0\.9979330\.997933\)\. The normalised geometry change across laws exceeds the same\-law cross\-architecture change by0\.9956780\.995678\(0\.9954850\.995485–0\.9958710\.995871\)\. Both contrasts are positive in all eight independent pairs \(exact one\-sided signP=1/256P=1/256\)\. The controlled assignment isolates the effect of the language law in these synthetic systems\.

## Part II: Shared

### 4Stability and attenuation of cross\-model agreement

This section supports the main text’s convergence results: rank stability, the excess\-risk route to rank convergence, and the exact attenuation account of the measured agreement\.

#### 4\.1Stability of the shared output geometry

This subsection proves entrywise and rank stability on a shared outcome space, and then derives population rank convergence from vanishing excess risk under an explicit margin condition\. It also proves that behaviour does not identify a unique Euclidean activation geometry and gives a common\-outcome coarse\-graining extension for different tokenizers\.

###### Setting\.

A modelmmenters only through its conditional lawspm\(⋅\|x\)p\_\{m\}\(\\cdot\\,\|\\,x\)on the context battery; writeum​\(x\)=pm\(⋅\|x\)u\_\{m\}\(x\)=\\sqrt\{p\_\{m\}\(\\cdot\\,\|\\,x\)\}, a unit vector\. The relational geometry compared in the main text is the matrix of Fisher–Rao distancesDm​\(x,y\)=2​arccos⁡Cm​\(x,y\)D\_\{m\}\(x,y\)=2\\arccos C\_\{m\}\(x,y\), withCm​\(x,y\)=⟨um​\(x\),um​\(y\)⟩∈\[0,1\]C\_\{m\}\(x,y\)=\\langle u\_\{m\}\(x\),u\_\{m\}\(y\)\\rangle\\in\[0,1\]the Bhattacharyya affinity\. SinceDDis a strictly decreasing function ofCC, reversing both models’ entry rankings alike, every rank\-agreement statistic of the distance matrices coincides with that of the affinity matrices, so all statements are proved forCC\. Models with a shared tokenizer share the outcome space; vocabulary positions beyond the common range are merged into a single bucket, a deterministic coarse\-graining applied to each model \(for the smaller vocabulary the bucket is empty\)\.

###### Lemma 2\(determination and per\-context control\)\.

\(a\)*Determination\.*CmC\_\{m\}is a functional of the conditional laws alone\. \(b\)*Entrywise control\.*For two modelsm,m′m,m^\{\\prime\}on a shared outcome space, letwz=um′​\(z\)−um​\(z\)w\_\{z\}=u\_\{m^\{\\prime\}\}\(z\)\-u\_\{m\}\(z\)andρ⁡\(z\)=∥wz∥\\rho\(z\)=\\lVert w\_\{z\}\\rVert\(the root\-probability distance\)\. Then, exactly,

e⁡\(x,y\):=Cm′​\(x,y\)−Cm​\(x,y\)=⟨wx,um​\(y\)⟩\+⟨um′​\(x\),wy⟩,\|e⁡\(x,y\)\|≤ρ⁡\(x\)\+ρ⁡\(y\)\.e\(x,y\)\\;:=\\;C\_\{m^\{\\prime\}\}\(x,y\)\-C\_\{m\}\(x,y\)\\;=\\;\\langle w\_\{x\},\\,u\_\{m\}\(y\)\\rangle\+\\langle u\_\{m^\{\\prime\}\}\(x\),\\,w\_\{y\}\\rangle,\\qquad\\lvert e\(x,y\)\\rvert\\;\\leq\\;\\rho\(x\)\+\\rho\(y\)\.\(c\)*Risk control\.*ρ\(z\)2≤KL\(pm\(⋅\|z\)∥pm′\(⋅\|z\)\)\\rho\(z\)^\{2\}\\leq\\mathrm\{KL\}\\bigl\(p\_\{m\}\(\\cdot\|z\)\\,\\\|\\,p\_\{m^\{\\prime\}\}\(\\cdot\|z\)\\bigr\)and likewise with the arguments exchanged\. \(d\)*Reference triangle\.*For any reference lawq\(⋅\|z\)q\(\\cdot\\,\|\\,z\),ρ⁡\(z\)≤δm​\(z\)\+δm′​\(z\)\\rho\(z\)\\leq\\delta\_\{m\}\(z\)\+\\delta\_\{m^\{\\prime\}\}\(z\)withδm​\(z\)=∥um​\(z\)−q\(⋅\|z\)∥\\delta\_\{m\}\(z\)=\\lVert u\_\{m\}\(z\)\-\\sqrt\{q\(\\cdot\|z\)\}\\rVert\.

###### Proof\.

\(a\) is immediate\. \(b\): expand⟨um′​\(x\),um′​\(y\)⟩−⟨um​\(x\),um​\(y\)⟩\\langle u\_\{m^\{\\prime\}\}\(x\),u\_\{m^\{\\prime\}\}\(y\)\\rangle\-\\langle u\_\{m\}\(x\),u\_\{m\}\(y\)\\rangleand collect; the bound is Cauchy–Schwarz on each term with∥u∥=1\\lVert u\\rVert=1\. \(c\): withBC=⟨um,um′⟩\\mathrm\{BC\}=\\langle u\_\{m\},u\_\{m^\{\\prime\}\}\\rangle,ρ2=2​\(1−BC\)\\rho^\{2\}=2\(1\-\\mathrm\{BC\}\), andKL\(p∥p′\)=−2∑apalogpa′/pa≥−2∑apa\(pa′/pa−1\)=2\(1−BC\)=ρ2\\mathrm\{KL\}\(p\\,\\\|\\,p^\{\\prime\}\)=\-2\\sum\_\{a\}p\_\{a\}\\log\\sqrt\{p^\{\\prime\}\_\{a\}/p\_\{a\}\}\\geq\-2\\sum\_\{a\}p\_\{a\}\(\\sqrt\{p^\{\\prime\}\_\{a\}/p\_\{a\}\}\-1\)=2\(1\-\\mathrm\{BC\}\)=\\rho^\{2\}, usinglog⁡t≤t−1\\log t\\leq t\-1\. \(d\) is the norm triangle inequality\. ∎

###### Theorem 6\(excess risk forces relational convergence\)\.

LetAAandBBbe two models on a common outcome space and letqqbe a common reference conditional law\. For contextxx, putuM​\(x\)=pM\(⋅∣x\)u\_\{M\}\(x\)=\\sqrt\{p\_\{M\}\(\\cdot\\mid x\)\}and define the root\-probability distance matrixdM​\(x,y\)=∥uM​\(x\)−uM​\(y\)∥2d\_\{M\}\(x,y\)=\\lVert u\_\{M\}\(x\)\-u\_\{M\}\(y\)\\rVert\_\{2\}\. If

εM=𝔼xKL\{q\(⋅∣x\)∥pM\(⋅∣x\)\},\\varepsilon\_\{M\}=\\mathbb\{E\}\_\{x\}\\KL\\\{q\(\\cdot\\mid x\)\\\|p\_\{M\}\(\\cdot\\mid x\)\\\},then, for independent contextsX,YX,Ydrawn from the battery law,

𝔼X,Y​\{dA​\(X,Y\)−dB​\(X,Y\)\}2≤4​\(εA\+εB\)\.\\mathbb\{E\}\_\{X,Y\}\\\{d\_\{A\}\(X,Y\)\-d\_\{B\}\(X,Y\)\\\}^\{2\}\\leq 4\(\\varepsilon\_\{A\}\+\\varepsilon\_\{B\}\)\.\(25\)For any finite vectorisation of the distance matrices, letac,bca\_\{c\},b\_\{c\}denote their centred vectors and suppose both have non\-zero norm\. Then

1−corr⁡\(a,b\)≤∥ac−bc∥222​∥ac∥2​∥bc∥2\.1\-\\operatorname\{corr\}\(a,b\)\\leq\\frac\{\\lVert a\_\{c\}\-b\_\{c\}\\rVert\_\{2\}^\{2\}\}\{2\\lVert a\_\{c\}\\rVert\_\{2\}\\lVert b\_\{c\}\\rVert\_\{2\}\}\.\(26\)Thus excess\-risk minimisation forces the relational geometry to converge at root\-risk rate, with the realised spread of the battery appearing only as the conditioning factor in Eq\. \([26](https://arxiv.org/html/2609.11063#as1_S4.E26)\)\.

###### Proof\.

Putw⁡\(x\)=uA​\(x\)−uB​\(x\)w\(x\)=u\_\{A\}\(x\)\-u\_\{B\}\(x\)\. The reverse triangle inequality gives

\|dA​\(x,y\)−dB​\(x,y\)\|≤∥w⁡\(x\)−w⁡\(y\)∥\.\|d\_\{A\}\(x,y\)\-d\_\{B\}\(x,y\)\|\\leq\\lVert w\(x\)\-w\(y\)\\rVert\.Squaring and averaging over independent contexts gives an upper bound2​𝔼x​∥w⁡\(x\)∥2−2​∥𝔼x​w​\(x\)∥2≤2​𝔼x​∥w⁡\(x\)∥22\\mathbb\{E\}\_\{x\}\\lVert w\(x\)\\rVert^\{2\}\-2\\lVert\\mathbb\{E\}\_\{x\}w\(x\)\\rVert^\{2\}\\leq 2\\mathbb\{E\}\_\{x\}\\lVert w\(x\)\\rVert^\{2\}\. Passing throughq\\sqrt\{q\}and applying\(s\+t\)2≤2​s2\+2​t2\(s\+t\)^\{2\}\\leq 2s^\{2\}\+2t^\{2\}yields

𝔼x​∥uA−uB∥2≤2​𝔼x​∥uA−q∥2\+2​𝔼x​∥uB−q∥2≤2​\(εA\+εB\),\\mathbb\{E\}\_\{x\}\\lVert u\_\{A\}\-u\_\{B\}\\rVert^\{2\}\\leq 2\\mathbb\{E\}\_\{x\}\\lVert u\_\{A\}\-\\sqrt\{q\}\\rVert^\{2\}\+2\\mathbb\{E\}\_\{x\}\\lVert u\_\{B\}\-\\sqrt\{q\}\\rVert^\{2\}\\leq 2\(\\varepsilon\_\{A\}\+\\varepsilon\_\{B\}\),where the final inequality is the Hellinger–KL bound\. This proves Eq\. \([25](https://arxiv.org/html/2609.11063#as1_S4.E25)\)\.

For the correlation statement, expand

∥ac−bc∥2=∥ac∥2\+∥bc∥2−2​⟨ac,bc⟩\.\\lVert a\_\{c\}\-b\_\{c\}\\rVert^\{2\}=\\lVert a\_\{c\}\\rVert^\{2\}\+\\lVert b\_\{c\}\\rVert^\{2\}\-2\\langle a\_\{c\},b\_\{c\}\\rangle\.Because∥ac∥2\+∥bc∥2≥2​∥ac∥​∥bc∥\\lVert a\_\{c\}\\rVert^\{2\}\+\\lVert b\_\{c\}\\rVert^\{2\}\\geq 2\\lVert a\_\{c\}\\rVert\\lVert b\_\{c\}\\rVert,

∥ac−bc∥2≥2​∥ac∥​∥bc∥​\{1−corr⁡\(a,b\)\},\\lVert a\_\{c\}\-b\_\{c\}\\rVert^\{2\}\\geq 2\\lVert a\_\{c\}\\rVert\\lVert b\_\{c\}\\rVert\\\{1\-\\operatorname\{corr\}\(a,b\)\\\},which rearranges to Eq\. \([26](https://arxiv.org/html/2609.11063#as1_S4.E26)\)\. ∎

###### Theorem 7\(rank stability of the relational geometry\)\.

Let\(x,y\)\(x,y\)and\(u,v\)\(u,v\)be two entries of the affinity matrices, with marginμ=\|Cm​\(x,y\)−Cm​\(u,v\)\|\\mu=\\lvert C\_\{m\}\(x,y\)\-C\_\{m\}\(u,v\)\\rvert\. If the two models order these entries discordantly, then

μ≤\|e⁡\(x,y\)−e⁡\(u,v\)\|≤min⁡\{\|e⁡\(x,y\)\|\+\|e⁡\(u,v\)\|,ρ⁡\(x\)\+ρ⁡\(y\)\+ρ⁡\(u\)\+ρ⁡\(v\)\}\.\\mu\\;\\leq\\;\\lvert e\(x,y\)\-e\(u,v\)\\rvert\\;\\leq\\;\\min\\bigl\\\{\\,\\lvert e\(x,y\)\\rvert\+\\lvert e\(u,v\)\\rvert,\\;\\;\\rho\(x\)\+\\rho\(y\)\+\\rho\(u\)\+\\rho\(v\)\\,\\bigr\\\}\.Consequently the Kendall discordance rate, the probability of discordance over entry pairs drawn uniformly from the battery, is at mostPr\[μ≤\|Δe\|\]\\Pr\[\\mu\\leq\\lvert\\Delta e\\rvert\]\.

###### Proof\.

Discordance meansCm​\(x,y\)−Cm​\(u,v\)C\_\{m\}\(x,y\)\-C\_\{m\}\(u,v\)andCm′​\(x,y\)−Cm′​\(u,v\)C\_\{m^\{\\prime\}\}\(x,y\)\-C\_\{m^\{\\prime\}\}\(u,v\)have strictly opposite signs\. Their difference ise⁡\(x,y\)−e⁡\(u,v\)e\(x,y\)\-e\(u,v\), and a difference of two opposite\-signed quantities has absolute value at least each of them, so\|Δ​e\|≥μ\\lvert\\Delta e\\rvert\\geq\\mu\. The upper bounds are the triangle inequality and Lemma[2](https://arxiv.org/html/2609.11063#as1_Thmlemma2)\(b\)\. ∎

###### Theorem 8\(population excess risk forces rank convergence\)\.

Letq\(⋅\|z\)q\(\\cdot\|z\)be a reference conditional law on the shared outcome space and letZ∼νZ\\sim\\nube the context law\. Draw two compared entriesA=\(X,Y\)A=\(X,Y\)andB=\(U,V\)B=\(U,V\)by any scheme for which each ofX,Y,U,VX,Y,U,Vhas marginalν\\nu\. Define

Rm=𝔼Z∼νKL\(q\(⋅\|Z\)∥pm\(⋅\|Z\)\),Fq\(t\)=Pr\[\|Cq\(A\)−Cq\(B\)\|≤t\]\.R\_\{m\}=\\mathbb\{E\}\_\{Z\\sim\\nu\}\\mathrm\{KL\}\\bigl\(q\(\\cdot\|Z\)\\,\\\|\\,p\_\{m\}\(\\cdot\|Z\)\\bigr\),\\qquad F\_\{q\}\(t\)=\\Pr\\bigl\[\|C\_\{q\}\(A\)\-C\_\{q\}\(B\)\|\\leq t\\bigr\]\.If the compared rankings have no ties, then for everyt\>0t\>0,

1−τ⁡\(m,m′\)≤2​Fq​\(t\)\+32​\(Rm\+Rm′\)t2\.1\-\\tau\(m,m^\{\\prime\}\)\\;\\leq\\;2F\_\{q\}\(t\)\+\\frac\{32\(R\_\{m\}\+R\_\{m^\{\\prime\}\}\)\}\{t^\{2\}\}\.Consequently, ifFq​\(0\)=0F\_\{q\}\(0\)=0andRm\+Rm′→0R\_\{m\}\+R\_\{m^\{\\prime\}\}\\to 0, thenτ⁡\(m,m′\)→1\\tau\(m,m^\{\\prime\}\)\\to 1\. More quantitatively, ifFq​\(t\)≤L​tκF\_\{q\}\(t\)\\leq Lt^\{\\kappa\}near zero for someκ\>0\\kappa\>0, optimising inttgives1−τ=O⁡\(\(Rm\+Rm′\)κ/\(κ\+2\)\)1\-\\tau=O\(\(R\_\{m\}\+R\_\{m^\{\\prime\}\}\)^\{\\kappa/\(\\kappa\+2\)\}\)\.

###### Proof\.

Putδm​\(z\)=∥pm\(⋅\|z\)−q\(⋅\|z\)∥\\delta\_\{m\}\(z\)=\\lVert\\sqrt\{p\_\{m\}\(\\cdot\|z\)\}\-\\sqrt\{q\(\\cdot\|z\)\}\\rVert\. Lemma[2](https://arxiv.org/html/2609.11063#as1_Thmlemma2)\(c\) gives𝔼​δm​\(Z\)2≤Rm\\mathbb\{E\}\\delta\_\{m\}\(Z\)^\{2\}\\leq R\_\{m\}\. If modelmmreverses the reference ordering ofAAandBBwhile the reference margin exceedstt, then

t<δm​\(X\)\+δm​\(Y\)\+δm​\(U\)\+δm​\(V\)=:Sm\.t<\\delta\_\{m\}\(X\)\+\\delta\_\{m\}\(Y\)\+\\delta\_\{m\}\(U\)\+\\delta\_\{m\}\(V\)=:S\_\{m\}\.By Cauchy–Schwarz,Sm2≤4​\(δm​\(X\)2\+δm​\(Y\)2\+δm​\(U\)2\+δm​\(V\)2\)S\_\{m\}^\{2\}\\leq 4\(\\delta\_\{m\}\(X\)^\{2\}\+\\delta\_\{m\}\(Y\)^\{2\}\+\\delta\_\{m\}\(U\)^\{2\}\+\\delta\_\{m\}\(V\)^\{2\}\), so𝔼​Sm2≤16​Rm\\mathbb\{E\}S\_\{m\}^\{2\}\\leq 16R\_\{m\}and Markov’s inequality givesPr⁡\(Sm\>t\)≤16​Rm/t2\\Pr\(S\_\{m\}\>t\)\\leq 16R\_\{m\}/t^\{2\}\. Away from a reference tie, discordance betweenmmandm′m^\{\\prime\}implies that at least one model reverses the reference ordering\. Hence their discordance probability is at mostFq​\(t\)\+16​\(Rm\+Rm′\)/t2F\_\{q\}\(t\)\+16\(R\_\{m\}\+R\_\{m^\{\\prime\}\}\)/t^\{2\}\. Multiplying by two gives the Kendall bound\. IfFq​\(0\)=0F\_\{q\}\(0\)=0, chooset↓0t\\downarrow 0slowly enough that\(Rm\+Rm′\)/t2→0\(R\_\{m\}\+R\_\{m^\{\\prime\}\}\)/t^\{2\}\\to 0; the rate follows by balancing the two terms\. ∎

###### Corollary 3\(finite\-list Spearman convergence\)\.

For two tie\-free rankings of a finite list of lengthn≥2n\\geq 2, Spearman’s rank correlationρS\\rho\_\{S\}and Kendall’sτ\\tausatisfy

1−ρS≤3​\(1−τ\)\.1\-\\rho\_\{S\}\\leq 3\(1\-\\tau\)\.Thus the conclusion of Theorem[8](https://arxiv.org/html/2609.11063#as1_Thmtheorem8)also forcesρS→1\\rho\_\{S\}\\to 1\.

###### Proof\.

For a list of lengthnn, letKKbe the inversion count anddid\_\{i\}the rank displacements\. Since\|di\|≤n−1\|d\_\{i\}\|\\leq n\-1, the Spearman and Kendall formulae, together with the footrule bound∑i\|di\|≤2​K\\sum\_\{i\}\|d\_\{i\}\|\\leq 2K, give

1−ρS=6​∑idi2n⁡\(n2−1\)≤6​\(n−1\)​∑i\|di\|n⁡\(n2−1\)≤12​Kn⁡\(n\+1\)≤12​Kn⁡\(n−1\)=3​\(1−τ\)\.1\-\\rho\_\{S\}=\\frac\{6\\sum\_\{i\}d\_\{i\}^\{2\}\}\{n\(n^\{2\}\-1\)\}\\leq\\frac\{6\(n\-1\)\\sum\_\{i\}\|d\_\{i\}\|\}\{n\(n^\{2\}\-1\)\}\\leq\\frac\{12K\}\{n\(n\+1\)\}\\leq\\frac\{12K\}\{n\(n\-1\)\}=3\(1\-\\tau\)\.∎

###### Corollary 4\(a common outcome space by coarse\-graining\)\.

Models may have different outcome spacesΩm\\Omega\_\{m\}\. At each contextzz, letTm,zT\_\{m,z\}be a Markov kernel fromΩm\\Omega\_\{m\}to a common spaceΩ0\\Omega\_\{0\}, and suppose reference laws satisfyTm,zqm\(⋅\|z\)=q0\(⋅\|z\)T\_\{m,z\}q\_\{m\}\(\\cdot\|z\)=q\_\{0\}\(\\cdot\|z\)\. Then the preceding results apply top¯m=Tm,z​pm\\bar\{p\}\_\{m\}=T\_\{m,z\}p\_\{m\}onΩ0\\Omega\_\{0\}, with

KL\(q0∥p¯m\)≤KL\(qm∥pm\)\.\\mathrm\{KL\}\(q\_\{0\}\\\|\\bar\{p\}\_\{m\}\)\\leq\\mathrm\{KL\}\(q\_\{m\}\\\|p\_\{m\}\)\.For tokenizers with deterministic byte decoding, mapping every token to the first newly decoded byte after the context \(with an empty/special\-token bucket\) is one such common coarse outcome\. This proves cross\-tokenizer comparison after coarse\-graining to the common outcome space; native\-vocabulary agreement remains empirical\.

###### Proof\.

The Kullback–Leibler inequality is data processing underTm,zT\_\{m,z\}at each context; apply Theorem[8](https://arxiv.org/html/2609.11063#as1_Thmtheorem8)to the pushed\-forward laws\. ∎

#### 4\.2The resolution law

The population theorem above controls average disagreement\. The following result states exactly when that control becomes a resolved count or an identical discrete structure\. It also separates the hard threshold count from the smooth response used for effective dimension\.

For a nonempty finite score vectors=\(s1,…,sd\)s=\(s\_\{1\},\\ldots,s\_\{d\}\)defineℛs​\(α\)=\{i:si≥α\}\\mathcal\{R\}\_\{s\}\(\\alpha\)=\\\{i:s\_\{i\}\\geq\\alpha\\\}andDs​\(α\)=\|ℛs​\(α\)\|D\_\{s\}\(\\alpha\)=\|\\mathcal\{R\}\_\{s\}\(\\alpha\)\|\.

###### Theorem 9\(loss, resolution and margin\)\.

Let

δm\(x\)=∥pm\(⋅\|x\)−q\(⋅\|x\)∥2,ρm2=𝔼xδm\(x\)2,εm=𝔼xKL\(qx∥pm,x\)\.\\delta\_\{m\}\(x\)=\\lVert\\sqrt\{p\_\{m\}\(\\cdot\|x\)\}\-\\sqrt\{q\(\\cdot\|x\)\}\\rVert\_\{2\},\\qquad\\rho\_\{m\}^\{2\}=\\mathbb\{E\}\_\{x\}\\delta\_\{m\}\(x\)^\{2\},\\qquad\\varepsilon\_\{m\}=\\mathbb\{E\}\_\{x\}\\mathrm\{KL\}\(q\_\{x\}\\\|p\_\{m,x\}\)\.
Risk\.Then

ρm2≤εm\.\\rho\_\{m\}^\{2\}\\leq\\varepsilon\_\{m\}\.\(27\)
Hard counts and margins\.More generally, if finite score vectors satisfymaxi⁡\|sim−siq\|≤bm\\max\_\{i\}\|s\_\{i\}^\{m\}\-s\_\{i\}^\{q\}\|\\leq b\_\{m\}, then

Dsq​\(α\+bm\)≤Dsm​\(α\)≤Dsq​\(α−bm\)\.D\_\{s^\{q\}\}\(\\alpha\+b\_\{m\}\)\\leq D\_\{s^\{m\}\}\(\\alpha\)\\leq D\_\{s^\{q\}\}\(\\alpha\-b\_\{m\}\)\.\(28\)Ifγq​\(α\)=mini⁡\|siq−α\|\\gamma\_\{q\}\(\\alpha\)=\\min\_\{i\}\|s\_\{i\}^\{q\}\-\\alpha\|, then

γq​\(α\)\>max⁡\{bm,bm′\}⟹ℛsm​\(α\)=ℛsm′​\(α\)\.\\gamma\_\{q\}\(\\alpha\)\>\\max\\\{b\_\{m\},b\_\{m^\{\\prime\}\}\\\}\\quad\\Longrightarrow\\quad\\mathcal\{R\}\_\{s^\{m\}\}\(\\alpha\)=\\mathcal\{R\}\_\{s^\{m^\{\\prime\}\}\}\(\\alpha\)\.\(29\)
Sorted probability profile\.At a fixed context, take the scores to be the sorted probabilities and letDp​\(α\)=\#⁡\{k:p\(k\)≥α\}D\_\{p\}\(\\alpha\)=\\\#\\\{k:p\_\{\(k\)\}\\geq\\alpha\\\}\. Then

Dq​\(α\+2​δm​\(x\)\)≤Dpm​\(α\)≤Dq​\(α−2​δm​\(x\)\)\.D\_\{q\}\(\\alpha\+2\\delta\_\{m\}\(x\)\)\\leq D\_\{p\_\{m\}\}\(\\alpha\)\\leq D\_\{q\}\(\\alpha\-2\\delta\_\{m\}\(x\)\)\.\(30\)For everyη\>0\\eta\>0, Eq\. \([30](https://arxiv.org/html/2609.11063#as1_S4.E30)\) therefore holds outside a context set of probability at mostη\\etaafter replacingδm​\(x\)\\delta\_\{m\}\(x\)byεm/η\\sqrt\{\\varepsilon\_\{m\}/\\eta\}\.

Smooth response\.Finally, for the smooth probability response

Φp​\(α\)=∑apapa\+α,α\>0,\\Phi\_\{p\}\(\\alpha\)=\\sum\_\{a\}\\frac\{p\_\{a\}\}\{p\_\{a\}\+\\alpha\},\\qquad\\alpha\>0,one has

\|Φp​\(α\)−Φq​\(α\)\|\\displaystyle\|\\Phi\_\{p\}\(\\alpha\)\-\\Phi\_\{q\}\(\\alpha\)\|≤2α​∥p−q∥2,\\displaystyle\\leq\\frac\{2\}\{\\alpha\}\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\},\(31\)𝔼x​\|Φpm​\(x\)​\(α\)−Φpm′​\(x\)​\(α\)\|\\displaystyle\\mathbb\{E\}\_\{x\}\|\\Phi\_\{p\_\{m\}\(x\)\}\(\\alpha\)\-\\Phi\_\{p\_\{m^\{\\prime\}\}\(x\)\}\(\\alpha\)\|≤2​\(εm\+εm′\)α\.\\displaystyle\\leq\\frac\{2\(\\sqrt\{\\varepsilon\_\{m\}\}\+\\sqrt\{\\varepsilon\_\{m^\{\\prime\}\}\}\)\}\{\\alpha\}\.\(32\)

###### Proof\.

The coordinate Hellinger–KL inequality gives

δm\(x\)2≤KL\(qx∥pm,x\),𝔼xδm\(x\)2≤εm;\\delta\_\{m\}\(x\)^\{2\}\\leq\\mathrm\{KL\}\(q\_\{x\}\\\|p\_\{m,x\}\),\\qquad\\mathbb\{E\}\_\{x\}\\delta\_\{m\}\(x\)^\{2\}\\leq\\varepsilon\_\{m\};the second inequality proves Eq\. \([27](https://arxiv.org/html/2609.11063#as1_S4.E27)\)\.

Ifsiq≥α\+bms\_\{i\}^\{q\}\\geq\\alpha\+b\_\{m\}, thensim≥αs\_\{i\}^\{m\}\\geq\\alpha, whilesim≥αs\_\{i\}^\{m\}\\geq\\alphaimpliessiq≥α−bms\_\{i\}^\{q\}\\geq\\alpha\-b\_\{m\}\. These two set inclusions prove Eq\. \([28](https://arxiv.org/html/2609.11063#as1_S4.E28)\)\. If\|siq−α\|\>bm\|s\_\{i\}^\{q\}\-\\alpha\|\>b\_\{m\}, the reference score lies either aboveα\+bm\\alpha\+b\_\{m\}or belowα−bm\\alpha\-b\_\{m\}, so modelmmhas the same threshold membership at coordinateii\. Applying this to both models proves Eq\. \([29](https://arxiv.org/html/2609.11063#as1_S4.E29)\)\.

Sorting is non\-expansive in the sup norm, hence

\|pm,\(k\)−q\(k\)\|≤δm​\(x\)\.\|\\sqrt\{p\_\{m,\(k\)\}\}\-\\sqrt\{q\_\{\(k\)\}\}\|\\leq\\delta\_\{m\}\(x\)\.Both roots are at most one, so

\|pm,\(k\)−q\(k\)\|≤2​δm​\(x\)\.\|p\_\{m,\(k\)\}\-q\_\{\(k\)\}\|\\leq 2\\delta\_\{m\}\(x\)\.Equation \([30](https://arxiv.org/html/2609.11063#as1_S4.E30)\) is now the generic count sandwich\. Markov’s inequality applied toδm2\\delta\_\{m\}^\{2\}givesPr\{δm\>εm/η\}≤η\\Pr\\\{\\delta\_\{m\}\>\\sqrt\{\\varepsilon\_\{m\}/\\eta\}\\\}\\leq\\eta, proving its population form\. This is why an RMS blur does not yield a uniform contextwise identity\.

Forx,y≥0x,y\\geq 0,

\|xx\+α−yy\+α\|≤\|x−y\|α\.\\left\|\\frac\{x\}\{x\+\\alpha\}\-\\frac\{y\}\{y\+\\alpha\}\\right\|\\leq\\frac\{\|x\-y\|\}\{\\alpha\}\.Also, by Cauchy–Schwarz,

∑a\|pa−qa\|=∑a\|pa−qa\|​\(pa\+qa\)≤2​∥p−q∥2\.\\sum\_\{a\}\|p\_\{a\}\-q\_\{a\}\|=\\sum\_\{a\}\|\\sqrt\{p\_\{a\}\}\-\\sqrt\{q\_\{a\}\}\|\(\\sqrt\{p\_\{a\}\}\+\\sqrt\{q\_\{a\}\}\)\\leq 2\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}\.This proves Eq\. \([31](https://arxiv.org/html/2609.11063#as1_S4.E31)\)\. Average, use𝔼​δm≤ρm≤εm\\mathbb\{E\}\\delta\_\{m\}\\leq\\rho\_\{m\}\\leq\\sqrt\{\\varepsilon\_\{m\}\}, and pass through the common reference law by the triangle inequality to obtain Eq\. \([32](https://arxiv.org/html/2609.11063#as1_S4.E32)\)\. ∎

###### Empirical evaluation\.

All quantities are computed from the cachedp\\sqrt\{p\}vectors of the natural battery \(200 contexts, full vocabulary, float64\) for ten same\-tokenizer pairs: the cross\-architecture trio Pythia\-1\.4B / Mamba\-1\.4B / RWKV\-1\.5B \(attention, state\-space, recurrent; one tokenizer, one corpus\), mixed\-scale pairs, the Pythia ladder against Pythia\-6\.9B, and GPT\-2\-large against GPT\-Neo\-1\.3B \(one tokenizer, two corpora\)\. Ranks use the affinity matrices; discordance is estimated on4×1054\\times 10^\{5\}sampled entry pairs; the identity of Lemma[2](https://arxiv.org/html/2609.11063#as1_Thmlemma2)\(b\) is checked to10−910^\{\-9\}; the boundρ2≤KL\\rho^\{2\}\\leq\\mathrm\{KL\}holds at every context of every pair \(medianρ/KL≈0\.70\\rho/\\sqrt\{\\mathrm\{KL\}\}\\approx 0\.70\); the discordance bounds hold on every pair\. Supplementary Table[1](https://arxiv.org/html/2609.11063#as1_S4.T1)summarises the realised stability and deterministic lower certificates\.

Supplementary Table 1:Held\-out stability certificates on the natural\-context battery\.The deterministic lower certificate is evaluated with realised perturbations; it is distinct from the worst\-case excess\-risk bound\.The deterministic lower certificate from Theorem[7](https://arxiv.org/html/2609.11063#as1_Thmtheorem7), evaluated a posteriori with the realised perturbations, sits0\.140\.14–0\.170\.17below the observedτ\\tauon every pair\. It is not the worst\-case excess\-risk bound of Theorem[8](https://arxiv.org/html/2609.11063#as1_Thmtheorem8); its residual slack reflects the certificate’s one\-sidedness \(a large perturbation difference is necessary for a flip but not sufficient\)\. A trajectory consistent with the loss linkage appears along the ladder: as the weaker Pythia model improves from 70M to 1\.4B, the median symmetrised per\-context KL to 6\.9B falls from1\.071\.07to0\.230\.23, the medianρ\\rhofalls from0\.710\.71to0\.350\.35in step withρ≤KL\\rho\\leq\\sqrt\{\\mathrm\{KL\}\}, and the observed agreement rises from0\.780\.78to0\.930\.93\(Spearman\)\. Routing the deterministic pairwise bound through the strongest common model as reference \(Lemma[2](https://arxiv.org/html/2609.11063#as1_Thmlemma2)\(d\)\) costs a measured factor of1\.81\.8–2\.02\.0\. The quantitative bound applies on a shared outcome space and hence directly to same\-tokenizer pairs\. Cross\-tokenizer native\-vocabulary agreement sits at the same level \(same\-tokenizer pairs agree only marginally more\) but remains empirical\. Corollary[4](https://arxiv.org/html/2609.11063#as1_Thmcorollary4)supplies a common first\-byte outcome: at this level the squared Hellinger distance does not increase on any of the 28 model pairs, different\-tokenizer agreement is0\.9100\.910\(80%\-subsample stability interval0\.8960\.896–0\.9220\.922\), and the token\-level anchor value is reproduced byte\-exactly\. The derived results comprise determination, Lipschitz and Kullback–Leibler controls, margin rank stability, population excess\-risk convergence, and non\-identifiability of a unique Euclidean activation geometry\. The battery supplies the empirical margin distribution and the coherence distribution of trained\-model disagreement\. A model pair that disagreed systematically along relational directions could agree far less at the same per\-context excess risk\.

#### 4\.3The attenuation identity and floors

The name is the classical one: attenuation is the weakening of an observed correlation by noise in the measured quantities, and the identity below is the exact form of that accounting for relational geometries\. Take a fixed reference relational mapSSover a context battery and, for each modelmm, write its relational map asCm=S\+emC\_\{m\}=S\+e\_\{m\}\. LetvS=Var⁡\(S\)v\_\{S\}=\\mathrm\{Var\}\(S\),vm=Var⁡\(em\)v\_\{m\}=\\mathrm\{Var\}\(e\_\{m\}\),cm=Cov⁡\(S,em\)c\_\{m\}=\\mathrm\{Cov\}\(S,e\_\{m\}\)andkm​n=Cov⁡\(em,en\)k\_\{mn\}=\\mathrm\{Cov\}\(e\_\{m\},e\_\{n\}\)over a declared population of context pairs\.

###### Theorem 10\(six\-scalar agreement identity\)\.

For arbitrary modelsm≠nm\\neq nwithVar⁡\(Cm\)\>0\\Var\(C\_\{m\}\)\>0andVar⁡\(Cn\)\>0\\Var\(C\_\{n\}\)\>0,

Corr⁡\(Cm,Cn\)=vS\+cm\+cn\+km​n\(vS\+vm\+2​cm\)​\(vS\+vn\+2​cn\),\\mathrm\{Corr\}\(C\_\{m\},C\_\{n\}\)=\\frac\{v\_\{S\}\+c\_\{m\}\+c\_\{n\}\+k\_\{mn\}\}\{\\sqrt\{\(v\_\{S\}\+v\_\{m\}\+2c\_\{m\}\)\(v\_\{S\}\+v\_\{n\}\+2c\_\{n\}\)\}\},with no exchangeability, homogeneity or distributional assumption\.

###### Proof\.

ExpandCov⁡\(S\+em,S\+en\)=vS\+cm\+cn\+km​n\\Cov\(S\+e\_\{m\},S\+e\_\{n\}\)=v\_\{S\}\+c\_\{m\}\+c\_\{n\}\+k\_\{mn\}andVar⁡\(S\+ej\)=vS\+vj\+2​cj\\Var\(S\+e\_\{j\}\)=v\_\{S\}\+v\_\{j\}\+2c\_\{j\}, then divide the covariance by the two standard deviations\. ∎

###### Corollary 5\(exchangeable runs\)\.

If the joint law ofSSand the complete vector of run errors is invariant under permutations of the run labels, thenvm=vev\_\{m\}=v\_\{e\},cm=cc\_\{m\}=candkm​n=γ​vek\_\{mn\}=\\gamma v\_\{e\}for everym≠nm\\neq n\. IfvS\+2​c\+ve\>0v\_\{S\}\+2c\+v\_\{e\}\>0, this gives

Corr⁡\(Cm,Cn\)=vS\+2​c\+γ​vevS\+2​c\+ve\.\\Corr\(C\_\{m\},C\_\{n\}\)=\\frac\{v\_\{S\}\+2c\+\\gamma v\_\{e\}\}\{v\_\{S\}\+2c\+v\_\{e\}\}\.\(33\)Its difference from the naive reliability ratio is exactly

Corr⁡\(Cm,Cn\)−vSvS\+ve=ve​\{γ⁡\(vS\+ve\)\+2​c\}\(vS\+ve\)​\(vS\+ve\+2​c\)\.\\Corr\(C\_\{m\},C\_\{n\}\)\-\\frac\{v\_\{S\}\}\{v\_\{S\}\+v\_\{e\}\}=\\frac\{v\_\{e\}\\\{\\gamma\(v\_\{S\}\+v\_\{e\}\)\+2c\\\}\}\{\(v\_\{S\}\+v\_\{e\}\)\(v\_\{S\}\+v\_\{e\}\+2c\)\}\.\(34\)

###### Proof\.

Exchangeability equates the corresponding second moments\. Substitution in Theorem[10](https://arxiv.org/html/2609.11063#as1_Thmtheorem10)proves Eq\. \([33](https://arxiv.org/html/2609.11063#as1_S4.E33)\); bringing the two fractions to a common denominator proves Eq\. \([34](https://arxiv.org/html/2609.11063#as1_S4.E34)\)\. ∎

###### Lemma 3\(root\-affinity risk lemma\)\.

If the reference is a genuine conditional law on the same outcome experiment, both marginals of the declared context\-pair law equal the context law, andδm​\(x\)\\delta\_\{m\}\(x\)is the root\-probability distance at contextxxwith riskεm=𝔼​δm2\\varepsilon\_\{m\}=\\mathbb\{E\}\\,\\delta\_\{m\}^\{2\}, then\|em\|≤δm​\(X\)\+δm​\(Y\)\|e\_\{m\}\|\\leq\\delta\_\{m\}\(X\)\+\\delta\_\{m\}\(Y\)entrywise,vm≤4​εmv\_\{m\}\\leq 4\\varepsilon\_\{m\},\|cm\|≤2​vS​εm\|c\_\{m\}\|\\leq 2\\sqrt\{v\_\{S\}\\varepsilon\_\{m\}\}and\|km​n\|≤4​εm​εn\|k\_\{mn\}\|\\leq 4\\sqrt\{\\varepsilon\_\{m\}\\varepsilon\_\{n\}\}\.

###### Proof\.

Lemma[2](https://arxiv.org/html/2609.11063#as1_Thmlemma2)gives\|em\|≤δm​\(X\)\+δm​\(Y\)\|e\_\{m\}\|\\leq\\delta\_\{m\}\(X\)\+\\delta\_\{m\}\(Y\)\. Therefore

𝔼​em2≤2​𝔼​δm​\(X\)2\+2​𝔼​δm​\(Y\)2=4​εm\.\\mathbb\{E\}e\_\{m\}^\{2\}\\leq 2\\mathbb\{E\}\\delta\_\{m\}\(X\)^\{2\}\+2\\mathbb\{E\}\\delta\_\{m\}\(Y\)^\{2\}=4\\varepsilon\_\{m\}\.Variance is bounded by the second moment\. The remaining inequalities are Cauchy–Schwarz applied toCov⁡\(S,em\)\\Cov\(S,e\_\{m\}\)andCov⁡\(em,en\)\\Cov\(e\_\{m\},e\_\{n\}\)\. ∎

###### Theorem 11\(attenuation floors\)\.

Writeu=vSu=\\sqrt\{v\_\{S\}\}andwj=2​εjw\_\{j\}=2\\sqrt\{\\varepsilon\_\{j\}\}\. Wheneveru2−u⁡\(wm\+wn\)−wm​wn\>0u^\{2\}\-u\(w\_\{m\}\+w\_\{n\}\)\-w\_\{m\}w\_\{n\}\>0,Corr⁡\(Cm,Cn\)≥\(u2−u⁡\(wm\+wn\)−wm​wn\)/\(\(u\+wm\)​\(u\+wn\)\)\\mathrm\{Corr\}\(C\_\{m\},C\_\{n\}\)\\geq\\bigl\(u^\{2\}\-u\(w\_\{m\}\+w\_\{n\}\)\-w\_\{m\}w\_\{n\}\\bigr\)/\\bigl\(\(u\+w\_\{m\}\)\(u\+w\_\{n\}\)\\bigr\); if additionallykm​n≥0k\_\{mn\}\\geq 0andu\>wm\+wnu\>w\_\{m\}\+w\_\{n\}, the numerator improves tou2−u⁡\(wm\+wn\)u^\{2\}\-u\(w\_\{m\}\+w\_\{n\}\)\. For conditionally independent, identically distributed training runs measured against an external reference, the run\-averaged cross covariance is automatically nonnegative; an individual realised pair may still be negative\.

###### Proof\.

Lemma[3](https://arxiv.org/html/2609.11063#as1_Thmlemma3)bounds the numerator of Theorem[10](https://arxiv.org/html/2609.11063#as1_Thmtheorem10)below byu2−u⁡\(wm\+wn\)−wm​wnu^\{2\}\-u\(w\_\{m\}\+w\_\{n\}\)\-w\_\{m\}w\_\{n\}, improving the final term to zero whenkm​n≥0k\_\{mn\}\\geq 0\. Its two variance factors obeyvS\+vj\+2​cj≤u\+wj\\sqrt\{v\_\{S\}\+v\_\{j\}\+2c\_\{j\}\}\\leq u\+w\_\{j\}\. When the numerator lower bound is positive, division proves the displayed floors\. For conditionally independent identically distributed runs, the law of total covariance gives

Cov⁡\(em,en\)\\displaystyle\\Cov\(e\_\{m\},e\_\{n\}\)=𝔼\{Cov\(em,en∣D,X,Y\)\}\\displaystyle=\\mathbb\{E\}\\\{\\Cov\(e\_\{m\},e\_\{n\}\\mid D,X,Y\)\\\}\+Cov⁡\{𝔼⁡\(em∣D,X,Y\),𝔼⁡\(en∣D,X,Y\)\}\\displaystyle\\quad\+\\Cov\\\{\\mathbb\{E\}\(e\_\{m\}\\mid D,X,Y\),\\mathbb\{E\}\(e\_\{n\}\\mid D,X,Y\)\\\}=Var⁡\{e¯​\(D,X,Y\)\}≥0\.\\displaystyle=\\Var\\\{\\bar\{e\}\(D,X,Y\)\\\}\\geq 0\.∎

###### Proposition 2\(the±𝟏/𝟐\\boldsymbol\{\\pm 1/2\}idiosyncrasy signature\)\.

Letrir\_\{i\}be model\-specific deviation vectors in an inner\-product space and putwi​j=ri−rjw\_\{ij\}=r\_\{i\}\-r\_\{j\}\. Then

⟨wi​j,wi​k⟩=12​\{∥wi​j∥2\+∥wi​k∥2−∥wj​k∥2\}\.\\langle w\_\{ij\},w\_\{ik\}\\rangle=\\tfrac\{1\}\{2\}\\\{\\lVert w\_\{ij\}\\rVert^\{2\}\+\\lVert w\_\{ik\}\\rVert^\{2\}\-\\lVert w\_\{jk\}\\rVert^\{2\}\\\}\.\(35\)If distinct deviations are mutually orthogonal and have the same positive norm, then

cos⁡\(wi​j,wi​k\)=12,cos⁡\(wi​j,wj​k\)=−12\.\\cos\(w\_\{ij\},w\_\{ik\}\)=\\tfrac\{1\}\{2\},\\qquad\\cos\(w\_\{ij\},w\_\{jk\}\)=\-\\tfrac\{1\}\{2\}\.\(36\)Thus the signs distinguish edges with a common terminal endpoint from successive oriented edges; a common disagreement axis would instead approach unit magnitude\.

###### Proof\.

Equation \([35](https://arxiv.org/html/2609.11063#as1_S4.E35)\) is the polarization identity applied towj​k=wi​k−wi​jw\_\{jk\}=w\_\{ik\}\-w\_\{ij\}\. Under mutual orthogonality and common squared normd2d^\{2\}, each difference has squared norm2​d22d^\{2\}and⟨ri−rj,ri−rk⟩=d2\\langle r\_\{i\}\-r\_\{j\},r\_\{i\}\-r\_\{k\}\\rangle=d^\{2\}, whereas⟨ri−rj,rj−rk⟩=−d2\\langle r\_\{i\}\-r\_\{j\},r\_\{j\}\-r\_\{k\}\\rangle=\-d^\{2\}\. Division by2​d22d^\{2\}proves Eq\. \([36](https://arxiv.org/html/2609.11063#as1_S4.E36)\)\. ∎

###### Proposition 3\(no hidden\-face floor\)\.

No analogous floor exists for hidden\-layer geometries: there are pairs of models with identical output behaviour, related by an exact output\-preserving symmetry, whose hidden relational geometries can have negative agreement \(an explicit integer construction attains Pearson agreement−0\.2053\-0\.2053and rank agreement exactly−4/17\-4/17at identical outputs\)\.

###### Proof\.

Take four hidden rows

H=\(0210−11−1−1−2−100\)H=\\begin\{pmatrix\}0&2&1\\\\ 0&\-1&1\\\\ \-1&\-1&\-2\\\\ \-1&0&0\\end\{pmatrix\}and chartsA=diag⁡\(1,3,1\)A=\\operatorname\{diag\}\(1,3,1\)andB=diag⁡\(1,1,3\)B=\\operatorname\{diag\}\(1,1,3\)\. The two batteries are related by the invertible changeH​B=\(H​A\)​diag⁡\(1,1/3,3\)HB=\(HA\)\\operatorname\{diag\}\(1,1/3,3\)\. A read\-out using only the first coordinate is identical and non\-constant in both charts\. Over the six unordered context pairs, direct inner products give

cA=\(−17370,−20518,0,7140,0,114\),c\_\{A\}=\\left\(\-\\frac\{17\}\{\\sqrt\{370\}\},\-\\frac\{20\}\{\\sqrt\{518\}\},0,\\frac\{7\}\{\\sqrt\{140\}\},0,\\frac\{1\}\{\\sqrt\{14\}\}\\right\),cB=\(7130,−20494,0,−17380,0,138\)\.c\_\{B\}=\\left\(\\frac\{7\}\{\\sqrt\{130\}\},\-\\frac\{20\}\{\\sqrt\{494\}\},0,\-\\frac\{17\}\{\\sqrt\{380\}\},0,\\frac\{1\}\{\\sqrt\{38\}\}\\right\)\.Their centred Pearson correlation is−0\.2053387​…\-0\.2053387\\ldots\. Their average\-rank vectors are\(1,2,3\.5,6,3\.5,5\)\(1,2,3\.5,6,3\.5,5\)and\(6,1,3\.5,2,3\.5,5\)\(6,1,3\.5,2,3\.5,5\); the centred vectors have squared norm1717and inner product−4\-4, so Spearman agreement is exactly−4/17\-4/17\. Since the output laws and their non\-zero relational signal are identical, no positive hidden\-geometry floor can be a function only of output risk and signal\. ∎

Empirically, the identity reconstructs every measured agreement on the eight\-model battery \(112 model\-pair configurations, covering output\-level and hidden\-layer geometries in raw and rank conventions, tolerance10−1010^\{\-10\}\), the theorem identities and inequalities hold in all forty per\-model evaluations, and all sixty realised cross covariances are nonnegative\. The floors are valid but vacuous at the measured model\-to\-reference distances: realised fluctuation variances are near0\.0050\.005against bounds near1\.31\.3\. The floor calculations and held\-out prediction study are documented in Supplementary Note[12](https://arxiv.org/html/2609.11063#as1_S12)\.

The held\-out prediction test measures the operational performance of the exchangeable formula\. All 120 full\-sample attempts and 120,000 substream attempts satisfy the numerical consistency checks\. The estimator is undefined in 24 of the 120 full\-sample cells \(transported denominator non\-positive at steps 1–64\), and the defined early\-checkpoint predictions are inaccurate \(median error0\.250\.25–0\.310\.31at steps 1–16 and0\.0560\.056at step 64\)\. From step 256 onward, where the estimator is defined and stable, the median absolute prediction error is0\.00360\.0036–0\.0140\.014; the pooled defined median is0\.009380\.00938and the substream 95th percentile is0\.014370\.01437\. The evidence therefore identifies the onset of an exchangeable predictive regime during training: the estimator is accurate after step 256 but unreliable and sometimes undefined before it\. The independently trained model deviations also show the signature of Proposition[2](https://arxiv.org/html/2609.11063#as1_Thmproposition2):\+0\.57\+0\.57and−0\.51\-0\.51on the cross\-architecture trio, and\+0\.487\+0\.487and−0\.487\-0\.487across PolyPythias seeds\.

### 5Shared geometry carries meaning and transfers

The shared component is the mean of the models’ rank\-transformed distance matrices; a model’s residual is its matrix after the shared component has been partialled out\. Against the external semantic geometry defined by theall\-MiniLM\-L6\-v2Sentence\-BERT encoder\[[3](https://arxiv.org/html/2609.11063#as1_bib.bib3)\], residual alignment is negligible on natural, templated and factual\-relation batteries \(−0\.001\-0\.001,\+0\.003\+0\.003and\+0\.051\+0\.051\), whereas shared\-component alignment is0\.0440\.044,0\.1020\.102and0\.4400\.440\. On the factual\-relation battery,\+0\.231\+0\.231\(95% interval\+0\.159\+0\.159to\+0\.316\+0\.316\) remains after partialling out surface form\. The shared component also contains structure beyond its alignment with the external meaning reference\.

Transfer is tested through relative representations\. Each prompt is represented by its Fisher–Rao distances to a fixed set of 40 anchors and a multinomial probe is trained on one model and evaluated on another\. Eight\-way accuracy is0\.7240\.724within model,0\.6590\.659across models and0\.7000\.700using the consensus alone, against chance0\.1250\.125\. The residual supports0\.4070\.407within its own model but0\.0910\.091across models\. The shared component is therefore the transferable one\.

A separate activation\-space analysis extends the evidence beyond textual co\-occurrence\. Over 100 concrete concepts, the mean cosine\-distance geometry of last\-token, last\-hidden\-state representations from Pythia\-1\.4B and GPT\-2 Large agrees with a vision\-only DINOv2 geometry at0\.3850\.385\(permutationP<0\.001P<0\.001\), and 84% of this association remains after controlling textual co\-occurrence\. The output\-geometric connection to control is supplied independently by the top spectral bands: the first five modes carry approximately half of the steering weight, and the first 40 carry approximately 78%; sharing and answer\-relevant motion decay over the same bands on both templated and natural batteries\. The geometry that transfers across models is therefore the geometry in which minimal\-disturbance control is concentrated\.

## Part III: Learned

### 6Spectral inheritance, cluster anatomy and multiscale response

This section supports the main text’s anatomy results: the spectrum tracks the weighted token profile, the resolved core splits into singleton tokens and interchangeable clusters, and the effective dimension counts spectrally resolved structure\.

#### 6\.1Spectral constraints and effective dimension

This subsection gives exact conditional bounds relating the sorted spectrum of the output Fisher metric to the sorted weighted token profile, an exact finite\-width majorization constraint, and the effective\-dimension transform of a power\-law spectrum\. The hypotheses of the conditional bounds are measurable constants of the read\-out, evaluated below on the real models\.

##### Exact finite spectral bounds

###### Setting\.

With read\-out rowswaw\_\{a\}and next\-token distributionpp, the output Fisher metric is thepp\-weighted covariance of the read\-out,

H=WU⊤​\(diag⁡\(p\)−p​p⊤\)​WU=∑apa​va​va⊤,va=wa−w¯,w¯=∑bpb​wb,H\\;=\\;W\_\{U\}^\{\\top\}\\bigl\(\\mathrm\{diag\}\(p\)\-pp^\{\\top\}\\bigr\)W\_\{U\}\\;=\\;\\sum\_\{a\}p\_\{a\}\\,v\_\{a\}v\_\{a\}^\{\\top\},\\qquad v\_\{a\}=w\_\{a\}\-\\bar\{w\},\\quad\\bar\{w\}=\\sum\_\{b\}p\_\{b\}w\_\{b\},\(37\)so the centring is intrinsic:HHnever sees thepp\-weighted mean direction, and the uncentred second moment differs fromHHonly by the rank\-one spikew¯​w¯⊤\\bar\{w\}\\bar\{w\}^\{\\top\}\. The natural variables are the weighted profileqa=pa​∥va∥2q\_\{a\}=p\_\{a\}\\lVert v\_\{a\}\\rVert^\{2\}and, whenva≠0v\_\{a\}\\neq 0, the centred unit directionv^a=va/∥va∥\\hat\{v\}\_\{a\}=v\_\{a\}/\\lVert v\_\{a\}\\rVert; forva=0v\_\{a\}=0, choose any unitv^a\\hat\{v\}\_\{a\}\. Relabel so thatq1≥q2≥⋯≥0q\_\{1\}\\geq q\_\{2\}\\geq\\dots\\geq 0; since the read\-out norms vary mildly across tokens, the sorted profile carries the Zipf profile ofpp\. Eigenvalues are listed in nonincreasing order\. Because thev^a\\hat\{v\}\_\{a\}are unit vectors,

tr⁡H=∑aqa=:Q\.\\operatorname\{tr\}H\\;=\\;\\sum\_\{a\}q\_\{a\}\\;=:\\;Q\.\(38\)FixK≤min⁡\{d,\|V\|\}K\\leq\\min\\\{d,\|V\|\\\}and letMK=\(⟨v^i,v^j⟩\)i,j≤KM\_\{K\}=\(\\langle\\hat\{v\}\_\{i\},\\hat\{v\}\_\{j\}\\rangle\)\_\{i,j\\leq K\}be the Gram matrix of the top\-KKdirections\. Three measurable constants control the spectrum: the frame boundsaK=λmin​\(MK\)a\_\{K\}=\\lambda\_\{\\min\}\(M\_\{K\}\)andAK=λmax​\(MK\)A\_\{K\}=\\lambda\_\{\\max\}\(M\_\{K\}\), and the tail spreadεK=λmax​\(∑k\>Kqk​v^k​v^k⊤\)\\varepsilon\_\{K\}=\\lambda\_\{\\max\}\(\\sum\_\{k\>K\}q\_\{k\}\\hat\{v\}\_\{k\}\\hat\{v\}\_\{k\}^\{\\top\}\)\. All three are computable from the model weights and one forward pass, with no randomness or independence hypothesis; for an exactly orthonormal top\-KKread\-out,aK=AK=1a\_\{K\}=A\_\{K\}=1\.

###### Theorem 12\(inheritance band\)\.

For everyk≤Kk\\leq K, the*frame lower bound*and*frame\-and\-tail upper bound*are

aK​qk≤λk​\(H\)≤AK​qk\+εK\.a\_\{K\}\\,q\_\{k\}\\leq\\lambda\_\{k\}\(H\)\\leq A\_\{K\}\\,q\_\{k\}\+\\varepsilon\_\{K\}\.

###### Proof\.

SplitH=HK\+H\>KH=H\_\{K\}\+H\_\{\>K\}into the top blockHK=∑i≤Kqi​v^i​v^i⊤H\_\{K\}=\\sum\_\{i\\leq K\}q\_\{i\}\\hat\{v\}\_\{i\}\\hat\{v\}\_\{i\}^\{\\top\}and the tail; both are positive semidefinite\.

*Lower bound\.*IfaK=0a\_\{K\}=0the bound is trivial, so assumeaK\>0a\_\{K\}\>0; thenv^1,…,v^K\\hat\{v\}\_\{1\},\\dots,\\hat\{v\}\_\{K\}are linearly independent\. Fixk≤Kk\\leq Kand letS=span⁡\{v^1,…,v^k\}S=\\mathrm\{span\}\\\{\\hat\{v\}\_\{1\},\\dots,\\hat\{v\}\_\{k\}\\\}, of dimensionkk\. LetGkG\_\{k\}be the Gram matrix of the firstkkdirections; as a principal submatrix ofMKM\_\{K\}its eigenvalues lie betweenaKa\_\{K\}andAKA\_\{K\}\. Any unitx∈Sx\\in Scan be writtenx=∑i≤kci​v^ix=\\sum\_\{i\\leq k\}c\_\{i\}\\hat\{v\}\_\{i\}, so that1=∥x∥2=c⊤​Gk​c1=\\lVert x\\rVert^\{2\}=c^\{\\top\}G\_\{k\}cand⟨v^j,x⟩=\(Gk​c\)j\\langle\\hat\{v\}\_\{j\},x\\rangle=\(G\_\{k\}c\)\_\{j\}forj≤kj\\leq k\. DiagonalisingGkG\_\{k\},

∑j≤k⟨v^j,x⟩2=c⊤​Gk2​c≥λmin​\(Gk\)​c⊤​Gk​c≥aK\.\\sum\_\{j\\leq k\}\\langle\\hat\{v\}\_\{j\},x\\rangle^\{2\}=c^\{\\top\}G\_\{k\}^\{2\}c\\geq\\lambda\_\{\\min\}\(G\_\{k\}\)\\,c^\{\\top\}G\_\{k\}c\\geq a\_\{K\}\.Dropping nonnegative terms and usingqj≥qkq\_\{j\}\\geq q\_\{k\}forj≤kj\\leq k,

x⊤​H​x≥x⊤​HK​x≥∑j≤kqj​⟨v^j,x⟩2≥qk​∑j≤k⟨v^j,x⟩2≥aK​qk,x^\{\\top\}Hx\\geq x^\{\\top\}H\_\{K\}x\\geq\\sum\_\{j\\leq k\}q\_\{j\}\\langle\\hat\{v\}\_\{j\},x\\rangle^\{2\}\\geq q\_\{k\}\\sum\_\{j\\leq k\}\\langle\\hat\{v\}\_\{j\},x\\rangle^\{2\}\\geq a\_\{K\}q\_\{k\},and the Courant–Fischer max–min characterisation givesλk​\(H\)≥minx∈S,∥x∥=1⁡x⊤​H​x≥aK​qk\\lambda\_\{k\}\(H\)\\geq\\min\_\{x\\in S,\\,\\lVert x\\rVert=1\}x^\{\\top\}Hx\\geq a\_\{K\}q\_\{k\}\.

*Upper bound\.*Weyl’s inequality givesλk​\(H\)≤λk​\(HK\)\+εK\\lambda\_\{k\}\(H\)\\leq\\lambda\_\{k\}\(H\_\{K\}\)\+\\varepsilon\_\{K\}\. Forλk​\(HK\)\\lambda\_\{k\}\(H\_\{K\}\), choose any\(k−1\)\(k\-1\)\-dimensional subspaceWWcontaining the firstk−1k\-1directions\. For a unitx⟂Wx\\perp Wthe termsj<kj<kvanish, so withUUtheK×dK\\times dmatrix with rowsv^j⊤\\hat\{v\}\_\{j\}^\{\\top\},

x⊤​HK​x=∑j=kKqj​⟨v^j,x⟩2≤qk​∑j≤K⟨v^j,x⟩2=qk​∥U​x∥2≤qk​λmax​\(U​U⊤\)=AK​qk,x^\{\\top\}H\_\{K\}x=\\sum\_\{j=k\}^\{K\}q\_\{j\}\\langle\\hat\{v\}\_\{j\},x\\rangle^\{2\}\\leq q\_\{k\}\\sum\_\{j\\leq K\}\\langle\\hat\{v\}\_\{j\},x\\rangle^\{2\}=q\_\{k\}\\lVert Ux\\rVert^\{2\}\\leq q\_\{k\}\\,\\lambda\_\{\\max\}\(UU^\{\\top\}\)=A\_\{K\}q\_\{k\},using thatU⊤​UU^\{\\top\}UandU​U⊤=MKUU^\{\\top\}=M\_\{K\}have the same nonzero eigenvalues\. The Courant–Fischer min–max characterisation givesλk​\(HK\)≤maxx⟂W,∥x∥=1⁡x⊤​HK​x≤AK​qk\\lambda\_\{k\}\(H\_\{K\}\)\\leq\\max\_\{x\\perp W,\\,\\lVert x\\rVert=1\}x^\{\\top\}H\_\{K\}x\\leq A\_\{K\}q\_\{k\}\. ∎

###### Theorem 13\(fractional\-frame inheritance\)\.

Forb\>0b\>0, let

rK​\(b\)=\#⁡\{j:λj​\(MK\)<b\}\.r\_\{K\}\(b\)=\\\#\\\{j:\\lambda\_\{j\}\(M\_\{K\}\)<b\\\}\.No randomness or independence assumption is required\. The following bounds hold\.

Lower bound\.IfK≤min⁡\{d,\|V\|\}K\\leq\\min\\\{d,\|V\|\\\}andk\+rK​\(b\)≤Kk\+r\_\{K\}\(b\)\\leq K, then

λk​\(H\)≥b​qk\+rK​\(b\)\.\\lambda\_\{k\}\(H\)\\geq b\\,q\_\{k\+r\_\{K\}\(b\)\}\.\(39\)
Direction\-free upper bound\.If0≤m<k≤d0\\leq m<k\\leq dandQ\>m=∑j\>mqjQ\_\{\>m\}=\\sum\_\{j\>m\}q\_\{j\}, then

λk​\(H\)≤Q\>mk−m\.\\lambda\_\{k\}\(H\)\\leq\\frac\{Q\_\{\>m\}\}\{k\-m\}\.\(40\)Thus the lower bound requires only that sufficiently many leading Gram modes remain above resolutionbb, while the upper bound does not depend on the read\-out directions\.

###### Proof\.

PutDK=diag⁡\(q1,…,qK\)D\_\{K\}=\\operatorname\{diag\}\(q\_\{1\},\\ldots,q\_\{K\}\)andSK=DK1/2​MK​DK1/2S\_\{K\}=D\_\{K\}^\{1/2\}M\_\{K\}D\_\{K\}^\{1/2\}\. Its eigenvalues are those ofHK=∑j≤Kqj​v^j​v^j⊤H\_\{K\}=\\sum\_\{j\\leq K\}q\_\{j\}\\hat\{v\}\_\{j\}\\hat\{v\}\_\{j\}^\{\\top\}, with zeros included, andH⪰HKH\\succeq H\_\{K\}\. LetP=\(b​I−MK\)\+P=\(bI\-M\_\{K\}\)\_\{\+\}\. ThenP⪰0P\\succeq 0has rankr=rK​\(b\)r=r\_\{K\}\(b\)and

SK\+DK1/2​P​DK1/2⪰b​DK\.S\_\{K\}\+D\_\{K\}^\{1/2\}PD\_\{K\}^\{1/2\}\\succeq bD\_\{K\}\.A positive rank\-rrupdate moves an eigenvalue upward by at mostrrpositions\. Hence rank interlacing and eigenvalue monotonicity give

λk​\(H\)≥λk​\(SK\)≥λk\+r​\(SK\+DK1/2​P​DK1/2\)≥b​qk\+r,\\lambda\_\{k\}\(H\)\\geq\\lambda\_\{k\}\(S\_\{K\}\)\\geq\\lambda\_\{k\+r\}\(S\_\{K\}\+D\_\{K\}^\{1/2\}PD\_\{K\}^\{1/2\}\)\\geq bq\_\{k\+r\},which proves Eq\. \([39](https://arxiv.org/html/2609.11063#as1_S6.E39)\)\.

For the upper bound, writeH=Am\+BmH=A\_\{m\}\+B\_\{m\}, whereAm=∑j≤mqj​v^j​v^j⊤A\_\{m\}=\\sum\_\{j\\leq m\}q\_\{j\}\\hat\{v\}\_\{j\}\\hat\{v\}\_\{j\}^\{\\top\}has rank at mostmmandBm=∑j\>mqj​v^j​v^j⊤⪰0B\_\{m\}=\\sum\_\{j\>m\}q\_\{j\}\\hat\{v\}\_\{j\}\\hat\{v\}\_\{j\}^\{\\top\}\\succeq 0\. Weyl’s rank inequality givesλk​\(H\)≤λk−m​\(Bm\)\\lambda\_\{k\}\(H\)\\leq\\lambda\_\{k\-m\}\(B\_\{m\}\)\. The\(k−m\)\(k\-m\)th eigenvalue of a positive matrix is at most its trace divided byk−mk\-m, andtr⁡Bm=Q\>m\\operatorname\{tr\}B\_\{m\}=Q\_\{\>m\}\. ∎

###### Corollary 6\(finite constant\-factor certificate\)\.

Fixkk\. Suppose that, for someK≤min⁡\{d,\|V\|\}K\\leq\\min\\\{d,\|V\|\\\},b\>0b\>0,α∈\(0,1\]\\alpha\\in\(0,1\]andcd,ct\>0c\_\{\\rm d\},c\_\{\\rm t\}\>0, at leastα​K\\alpha Keigenvalues ofMKM\_\{K\}are at leastbb,k≤α​Kk\\leq\\alpha K, andqK≥cd​qkq\_\{K\}\\geq c\_\{\\rm d\}q\_\{k\}\. If alsoQ\>m≤ct​\(k−m\)​qkQ\_\{\>m\}\\leq c\_\{\\rm t\}\(k\-m\)q\_\{k\}for somem<km<k, then

b​cd​qk≤λk​\(H\)≤ct​qk\.bc\_\{\\rm d\}q\_\{k\}\\leq\\lambda\_\{k\}\(H\)\\leq c\_\{\\rm t\}q\_\{k\}\.

###### Proof\.

The Gram\-mode premise givesrK​\(b\)≤K−α​K≤K−kr\_\{K\}\(b\)\\leq K\-\\alpha K\\leq K\-k, so Theorem[13](https://arxiv.org/html/2609.11063#as1_Thmtheorem13)yieldsλk​\(H\)≥b​qk\+rK​\(b\)≥b​qK\\lambda\_\{k\}\(H\)\\geq bq\_\{k\+r\_\{K\}\(b\)\}\\geq bq\_\{K\}\. Its direction\-free upper bound gives the other inequality\. ∎

###### Corollary 7\(upper\-outlier rank\-shift refinement\)\.

Fixbmax\>0b\_\{\\max\}\>0and letr↑r\_\{\\uparrow\}be the number of eigenvalues ofMKM\_\{K\}abovebmaxb\_\{\\max\}\. Forr↑<k≤Kr\_\{\\uparrow\}<k\\leq K,

λk​\(H\)≤bmax​qk−r↑\+εK\.\\lambda\_\{k\}\(H\)\\leq b\_\{\\max\}q\_\{k\-r\_\{\\uparrow\}\}\+\\varepsilon\_\{K\}\.Together with Theorem[13](https://arxiv.org/html/2609.11063#as1_Thmtheorem13)atb=bminb=b\_\{\\min\}, this gives

bmin​qk\+rK​\(bmin\)≤λk​\(H\)≤bmax​qk−r↑\+εKb\_\{\\min\}q\_\{k\+r\_\{K\}\(b\_\{\\min\}\)\}\\leq\\lambda\_\{k\}\(H\)\\leq b\_\{\\max\}q\_\{k\-r\_\{\\uparrow\}\}\+\\varepsilon\_\{K\}whenever both rank conditions hold\. On a power\-law profile a fixed index shift preserves the exponent forkkwell above the shift; exponent inheritance additionally requires control of the additive tail term relative toqkq\_\{k\}\.

###### Proof\.

WriteMK=B\+P↑M\_\{K\}=B\+P\_\{\\uparrow\}by clipping its eigenvalues from above atbmaxb\_\{\\max\}\. Then0⪯B⪯bmax​I0\\preceq B\\preceq b\_\{\\max\}IandP↑⪰0P\_\{\\uparrow\}\\succeq 0has rank at mostr↑r\_\{\\uparrow\}\. WithDK=diag⁡\(q1,…,qK\)D\_\{K\}=\\operatorname\{diag\}\(q\_\{1\},\\ldots,q\_\{K\}\), rank interlacing gives

λk​\(DK1/2​MK​DK1/2\)≤λk−r↑​\(DK1/2​B​DK1/2\)≤bmax​qk−r↑\.\\lambda\_\{k\}\(D\_\{K\}^\{1/2\}M\_\{K\}D\_\{K\}^\{1/2\}\)\\leq\\lambda\_\{k\-r\_\{\\uparrow\}\}\(D\_\{K\}^\{1/2\}BD\_\{K\}^\{1/2\}\)\\leq b\_\{\\max\}q\_\{k\-r\_\{\\uparrow\}\}\.The leading block has these eigenvalues, and adding the tail contributes at mostεK\\varepsilon\_\{K\}by Weyl’s inequality\. The lower clause is exactly Theorem[13](https://arxiv.org/html/2609.11063#as1_Thmtheorem13)\. ∎

###### Corollary 8\(cluster\-deflated inheritance\)\.

LetPPbe any orthogonal projector of rankr<dr<d, defineH⟂=P⟂​H​P⟂\|P⟂H\_\{\\perp\}=P^\{\\perp\}HP^\{\\perp\}\|\_\{P^\{\\perp\}\}, and putN⟂=\#⁡\{a:P⟂​va≠0\}N\_\{\\perp\}=\\\#\\\{a:P^\{\\perp\}v\_\{a\}\\neq 0\\\}\. After omitting zero projected vectors, sortqa⟂=pa​∥P⟂​va∥2q\_\{a\}^\{\\perp\}=p\_\{a\}\\lVert P^\{\\perp\}v\_\{a\}\\rVert^\{2\}and define the corresponding projected constantsaK⟂,AK⟂,εK⟂a\_\{K\}^\{\\perp\},A\_\{K\}^\{\\perp\},\\varepsilon\_\{K\}^\{\\perp\}\. IfK≤min⁡\{d−r,N⟂\}K\\leq\\min\\\{d\-r,N\_\{\\perp\}\\\},k≤Kk\\leq K, andμ1≥⋯≥μd−r\\mu\_\{1\}\\geq\\cdots\\geq\\mu\_\{d\-r\}are the eigenvalues ofH⟂H\_\{\\perp\}, then

aK⟂​qk⟂≤μk≤AK⟂​qk⟂\+εK⟂,λk​\(H\)≥μk≥λk\+r​\(H\)\.a\_\{K\}^\{\\perp\}q\_\{k\}^\{\\perp\}\\leq\\mu\_\{k\}\\leq A\_\{K\}^\{\\perp\}q\_\{k\}^\{\\perp\}\+\\varepsilon\_\{K\}^\{\\perp\},\\qquad\\lambda\_\{k\}\(H\)\\geq\\mu\_\{k\}\\geq\\lambda\_\{k\+r\}\(H\)\.Thus removing anrr\-dimensional cluster subspace gives a valid residual\-profile certificate and locates the associated original eigenvalues up to a rank shiftrr\.

###### Proof\.

Apply Theorem[12](https://arxiv.org/html/2609.11063#as1_Thmtheorem12)inP⟂P^\{\\perp\}, then use Poincaré separation for the codimension\-rrcompression\. ∎

##### Finite\-width and response consequences

###### Theorem 14\(trace floor\)\.

For everyk≤dk\\leq d,

λk​\(H\)≥Q−∑j<kλj​\(H\)d−k\+1≥Q≥k−\(AK−1\)​Q<k−\(k−1\)​εKd−k\+1\(k≤K\+1\),\\lambda\_\{k\}\(H\)\\;\\geq\\;\\frac\{Q\-\\sum\_\{j<k\}\\lambda\_\{j\}\(H\)\}\{d\-k\+1\}\\;\\geq\\;\\frac\{Q\_\{\\geq k\}\-\(A\_\{K\}\-1\)\\,Q\_\{<k\}\-\(k\-1\)\\,\\varepsilon\_\{K\}\}\{d\-k\+1\}\\qquad\(k\\leq K\+1\),whereQ<k=∑j<kqjQ\_\{<k\}=\\sum\_\{j<k\}q\_\{j\}andQ≥k=Q−Q<kQ\_\{\\geq k\}=Q\-Q\_\{<k\}\. The first lower bound is the average remaining*spectral*mass per remaining dimension\. The second is a profile\-based relaxation, informative only when its numerator is positive; it reduces approximately toQ≥k/\(d−k\+1\)Q\_\{\\geq k\}/\(d\-k\+1\)whenAK−1A\_\{K\}\-1andεK\\varepsilon\_\{K\}are sufficiently small\.

###### Proof\.

λk\\lambda\_\{k\}is the largest of thed−k\+1d\-k\+1trailing eigenvalues, hence at least their average, and their sum isQ−∑j<kλjQ\-\\sum\_\{j<k\}\\lambda\_\{j\}by \([38](https://arxiv.org/html/2609.11063#as1_S6.E38)\)\. The second inequality applies the upper bound of Theorem[12](https://arxiv.org/html/2609.11063#as1_Thmtheorem12)to eachλj\\lambda\_\{j\},j<kj<k\. ∎

###### Theorem 15\(finite\-width majorization\)\.

Assume the vocabulary has at leastddoutcomes\. Then

\(q1,…,q\|V\|\)≺\(λ1​\(H\),…,λd​\(H\),0,…,0\)\.\(q\_\{1\},\\ldots,q\_\{\|V\|\}\)\\;\\prec\\;\(\\lambda\_\{1\}\(H\),\\ldots,\\lambda\_\{d\}\(H\),0,\\ldots,0\)\.Equivalently, for everyk≤dk\\leq d,

∑j≤kqj≤∑j≤kλj​\(H\),∑j=k\+1dλj​\(H\)≤Q\>k,\\sum\_\{j\\leq k\}q\_\{j\}\\;\\leq\\;\\sum\_\{j\\leq k\}\\lambda\_\{j\}\(H\),\\qquad\\sum\_\{j=k\+1\}^\{d\}\\lambda\_\{j\}\(H\)\\;\\leq\\;Q\_\{\>k\},with equality of the total sums\. In particular,∑j=1d\(λj−qj\)=Q\>d\\sum\_\{j=1\}^\{d\}\(\\lambda\_\{j\}\-q\_\{j\}\)=Q\_\{\>d\}: finite width forces the profile mass beyond rankddinto the firstddspectral modes, but majorization does not identify the rank at which an individual eigenvalue leaves the profile\.

###### Proof\.

LetBBbe thed×\|V\|d\\times\|V\|matrix with columnsqa​v^a\\sqrt\{q\_\{a\}\}\\,\\hat\{v\}\_\{a\}\. ThenH=B​B⊤H=BB^\{\\top\}, while the Gram matrixB⊤​BB^\{\\top\}Bhas diagonal\(qa\)\(q\_\{a\}\)and eigenvalues\(λ1​\(H\),…,λd​\(H\),0,…,0\)\(\\lambda\_\{1\}\(H\),\\ldots,\\lambda\_\{d\}\(H\),0,\\ldots,0\)\. The Schur–Horn theorem gives the majorization statement\. Subtracting its partial\-sum inequalities from the common totalQQgives the trailing\-mass inequalities\. ∎

###### Profile crossover statistic\.

The quantity

kprof∗=min⁡\{k<d:qk≤Q\>k/\(d−k\)\}k^\{\*\}\_\{\\mathrm\{prof\}\}=\\min\\\{k<d:q\_\{k\}\\leq Q\_\{\>k\}/\(d\-k\)\\\}compares a profile entry with the mean profile mass available per remaining dimension\. It is a profile\-only, mean\-field predictor of the measured spectral crossover\. Theorems[14](https://arxiv.org/html/2609.11063#as1_Thmtheorem14)and[15](https://arxiv.org/html/2609.11063#as1_Thmtheorem15)supply separate conditional results and do not imply this predictor\. For a Zipf profileqk=Z−1​k−1q\_\{k\}=Z^\{\-1\}k^\{\-1\}with\|V\|≫d\|V\|\\gg d, its defining equation giveskprof∗​\(1\+ln⁡\(\|V\|/kprof∗\)\)≈dk^\{\*\}\_\{\\mathrm\{prof\}\}\(1\+\\ln\(\|V\|/k^\{\*\}\_\{\\mathrm\{prof\}\}\)\)\\approx d\. Numerically, doublingddat realistic\(\|V\|,d\)\(\|V\|,d\)multiplies this statistic by about2\.12\.1–2\.32\.3\(for\|V\|=5×104\|V\|=5\\times 10^\{4\},151151atd=1024d=1024and343343atd=2048d=2048\), corresponding to a log–log width slope about1\.11\.1–1\.21\.2\.

###### Corollary 9\(slope stability\)\.

Supposeλk​\(H\)=θk​qk\\lambda\_\{k\}\(H\)=\\theta\_\{k\}q\_\{k\}withθk∈\[a,b\]\\theta\_\{k\}\\in\[a,b\],0<a≤b0<a\\leq b, for allkkin a fit window𝒲\\mathcal\{W\}, and letβ\\betaands𝒲s\_\{\\mathcal\{W\}\}be the least\-squares slopes of−log⁡λk\-\\log\\lambda\_\{k\}and−log⁡qk\-\\log q\_\{k\}againstlog⁡k\\log kover𝒲\\mathcal\{W\}\. Then, withxk=log⁡k−log⁡k¯x\_\{k\}=\\log k\-\\overline\{\\log k\},

\|β−s𝒲\|≤log⁡\(b/a\)2​∑k\|xk\|∑kxk2\.\\lvert\\beta\-s\_\{\\mathcal\{W\}\}\\rvert\\;\\leq\\;\\frac\{\\log\(b/a\)\}\{2\}\\,\\frac\{\\sum\_\{k\}\\lvert x\_\{k\}\\rvert\}\{\\sum\_\{k\}x\_\{k\}^\{2\}\}\.

###### Proof\.

The least\-squares slope is linear in the ordinates and∑kxk=0\\sum\_\{k\}x\_\{k\}=0, so the deviation is−∑kxkηk/∑kxk2\-\\sum\_\{k\}x\_\{k\}\\eta\_\{k\}/\\sum\_\{k\}x\_\{k\}^\{2\}withηk=log⁡θk\\eta\_\{k\}=\\log\\theta\_\{k\}; centringη\\etain\[−12​log⁡\(b/a\),\+12​log⁡\(b/a\)\]\[\-\\tfrac\{1\}\{2\}\\log\(b/a\),\+\\tfrac\{1\}\{2\}\\log\(b/a\)\]and applying the triangle inequality gives the bound\. A rank\-uniform multiplicative band therefore perturbs the fitted spectral exponent by at mostO⁡\(log⁡\(b/a\)\)O\(\\log\(b/a\)\), and far less when theθk\\theta\_\{k\}are not adversarially ordered\. ∎

###### Corollary 10\(bounds for effective dimension\)\.

LetdfH​\(α\)=∑k≤dλk/\(λk\+α\)\\mathrm\{df\}\_\{H\}\(\\alpha\)=\\sum\_\{k\\leq d\}\\lambda\_\{k\}/\(\\lambda\_\{k\}\+\\alpha\)\(the effective dimension of the main text\) anddfq\(K\)​\(α\)=∑k≤Kqk/\(qk\+α\)\\mathrm\{df\}\_\{q\}^\{\(K\)\}\(\\alpha\)=\\sum\_\{k\\leq K\}q\_\{k\}/\(q\_\{k\}\+\\alpha\)\. IfaK\>0a\_\{K\}\>0then, for everyα\>0\\alpha\>0,

dfq\(K\)​\(αaK\)≤dfH​\(α\)≤dfq\(K\)​\(αAK\)\+d​λ¯λ¯\+α,λ¯=AK​qK\+εK\.\\mathrm\{df\}\_\{q\}^\{\(K\)\}\\\!\\Bigl\(\\tfrac\{\\alpha\}\{a\_\{K\}\}\\Bigr\)\\;\\leq\\;\\mathrm\{df\}\_\{H\}\(\\alpha\)\\;\\leq\\;\\mathrm\{df\}\_\{q\}^\{\(K\)\}\\\!\\Bigl\(\\tfrac\{\\alpha\}\{A\_\{K\}\}\\Bigr\)\+\\frac\{d\\,\\bar\{\\lambda\}\}\{\\bar\{\\lambda\}\+\\alpha\},\\qquad\\bar\{\\lambda\}=A\_\{K\}q\_\{K\}\+\\varepsilon\_\{K\}\.

###### Proof\.

fα​\(x\)=x/\(x\+α\)f\_\{\\alpha\}\(x\)=x/\(x\+\\alpha\)is increasing, subadditive onx≥0x\\geq 0and satisfiesfα​\(t​q\)=fα/t​\(q\)f\_\{\\alpha\}\(tq\)=f\_\{\\alpha/t\}\(q\)\. The lower bound drops the termsk\>Kk\>Kand appliesλk≥aK​qk\\lambda\_\{k\}\\geq a\_\{K\}q\_\{k\}\. For the upper bound,λk≤λ¯\\lambda\_\{k\}\\leq\\bar\{\\lambda\}for everyk≥Kk\\geq Kby Theorem[12](https://arxiv.org/html/2609.11063#as1_Thmtheorem12)and monotonicity; fork≤Kk\\leq K, subadditivity givesfα​\(λk\)≤fα​\(AK​qk\)\+fα​\(εK\)f\_\{\\alpha\}\(\\lambda\_\{k\}\)\\leq f\_\{\\alpha\}\(A\_\{K\}q\_\{k\}\)\+f\_\{\\alpha\}\(\\varepsilon\_\{K\}\), and the collected remainders total at mostd​fα​\(λ¯\)df\_\{\\alpha\}\(\\bar\{\\lambda\}\)sinceεK≤λ¯\\varepsilon\_\{K\}\\leq\\bar\{\\lambda\}\. ∎

If the additive remainderd​λ¯/\(λ¯\+α\)d\\bar\{\\lambda\}/\(\\bar\{\\lambda\}\+\\alpha\)is negligible relative to the profile term, these bounds identify the response up to the rescalingsα↦α/aK\\alpha\\mapsto\\alpha/a\_\{K\}andα↦α/AK\\alpha\\mapsto\\alpha/A\_\{K\}\. The conditionα≫λ¯\\alpha\\gg\\bar\{\\lambda\}alone is not sufficient whenddis large\.

###### Proposition 4\(power\-law effective\-dimension asymptotics\)\.

Let\(λk\)k≥1\(\\lambda\_\{k\}\)\_\{k\\geq 1\}be a nonincreasing infinite spectrum withλk∼C​k−s\\lambda\_\{k\}\\sim Ck^\{\-s\},s\>1s\>1, and truncate it afterddterms\. In an intermediate\-resolution limit in whichnα=\(C/α\)1/s→∞n\_\{\\alpha\}=\(C/\\alpha\)^\{1/s\}\\to\\inftyandd/nα→∞d/n\_\{\\alpha\}\\to\\infty,

dfH​\(α\)∼Is​\(Cα\)1/s,Is=∫0∞d​x1\+xs=πs​sin⁡\(π/s\)\.\\mathrm\{df\}\_\{H\}\(\\alpha\)\\;\\sim\\;I\_\{s\}\\left\(\\frac\{C\}\{\\alpha\}\\right\)^\{1/s\},\\qquad I\_\{s\}=\\int\_\{0\}^\{\\infty\}\\frac\{dx\}\{1\+x^\{s\}\}=\\frac\{\\pi\}\{s\\sin\(\\pi/s\)\}\.Thus a genuine power\-law spectral window gives response exponentγ=1/s\\gamma=1/son the corresponding resolution window\. Ats=1s=1the finite\-width response instead contains a logarithmic factor, of ordernα​log⁡\(1\+d/nα\)n\_\{\\alpha\}\\log\(1\+d/n\_\{\\alpha\}\), and at fixedddthe exact response tends torank⁡\(H\)\\operatorname\{rank\}\(H\)asα↓0\\alpha\\downarrow 0\.

###### Proof\.

For the exact lawλk=C​k−s\\lambda\_\{k\}=Ck^\{\-s\},

dfH​\(α\)nα=1nα​∑k=1d11\+\(k/nα\)s,\\frac\{\\mathrm\{df\}\_\{H\}\(\\alpha\)\}\{n\_\{\\alpha\}\}=\\frac\{1\}\{n\_\{\\alpha\}\}\\sum\_\{k=1\}^\{d\}\\frac\{1\}\{1\+\(k/n\_\{\\alpha\}\)^\{s\}\},which converges toIsI\_\{s\}by Riemann sums and the integrable tail fors\>1s\>1\. Sandwiching an asymptotic power law between\(1±η\)​C​k−s\(1\\pm\\eta\)Ck^\{\-s\}beyond a fixed rank, then takingη↓0\\eta\\downarrow 0, gives the stated result\. Thes=1s=1estimate follows by integrating\(1\+x/nα\)−1\(1\+x/n\_\{\\alpha\}\)^\{\-1\}, and the fixed\-ddlimit is termwise\. ∎

###### Spectral model selection and estimator validation\.

Five parametric spectral families \(single power law, broken power law, power law with exponential cutoff, stretched exponential, log\-quadratic\) are fitted to each spectrum and compared by the Bayesian information criterion \(BIC\), finite\-sample corrected Akaike information criterion \(AICc\) and blocked cross\-validation, and a fixed change\-point detector locates the core\-to\-tail crossover with hierarchical family–model–context bootstrap intervals\. A single global power law is preferred by BIC in under77percent of model–context pairs, and remains a minority choice under blocked cross\-validation; a crossover is detected in every pair; the core window retainsR2=0\.99R^\{2\}=0\.99and the median tail exponent beyond the crossover is1\.731\.73\. The main\-text power\-law results therefore apply to the rank\-22–8080core window\.

Estimator validity is checked against exactly computed spectra on held\-out model–context items\. The estimator and evaluation criteria were fixed before these exact spectra were revealed\. Empirical coverage of the nominal 95% percentile\-bootstrap intervals is0\.900\.90for continuous effective dimensions and0\.850\.85for discrete threshold counts, so the primary dimension statements use the continuous forms\.

###### Finite\-size prediction protocol\.

A separate finite\-size effective\-dimension confirmation uses 80 model–prompt curves from ten models in seven families\. For each curve, the bounded effective\-dimension curveNeff/d=\[1\+\(c/c⋆\)γ\]−1N\_\{\\mathrm\{eff\}\}/d=\[1\+\(c/c^\{\\star\}\)^\{\\gamma\}\]^\{\-1\}and an equal\-parameter unbroken power curve are fitted by nonlinear least squares at four fixed anchor dampingsc∈\{10−2,10−3,10−4,10−5\}c\\in\\\{10^\{\-2\},10^\{\-3\},10^\{\-4\},10^\{\-5\}\\\}, wherec=α/λmaxc=\\alpha/\\lambda\_\{\\max\}, and scored by root\-mean\-square error at the three interleaved held\-out dampingsc∈\{3×10−3,3×10−4,3×10−5\}c\\in\\\{3\\times 10^\{\-3\},3\\times 10^\{\-4\},3\\times 10^\{\-5\}\\\}\. This gives 240 held\-out effective\-dimension values\. The uncertainty interval for the median curve\-level error ratio uses 5,000 crossed draws that resample families, one model within each selected family and one common prompt\-index multiset across the sampled models\.

###### Synthetic inheritance controls\.

Synthetic controls draw512512read\-out directions ind=1024d=1024\(independent Gaussian, unit or log\-normal norms, and a coherent control dominated by a rank\-88component\), assign Zipf\(s\)\(s\)profiles withs∈\[0\.6,1\.5\]s\\in\[0\.6,1\.5\]under random token permutations, and fitβ^\\hat\{\\beta\}with the identical protocol \(three seeds\)\.

##### Evaluation on real models

Evaluation began with 64 held\-out effective\-dimension curves with vocabulary≤160\\leq 160k and width≤2048\\leq 2048\(8 prompts×\\times8 architectures: GPT\-2, GPT\-2\-XL, GPT\-Neo\-125M/1\.3B, Mamba\-130M/1\.4B, Qwen2\-0\.5B, RWKV\-1\.5B\)\. The stored next\-token distributions were independently reproduced within relative tolerance3×10−43\\times 10^\{\-4\}for 56 curves; the eight RWKV curves were excluded before spectrum analysis\. Constants and bounds were then evaluated in dense float64 on a 12\-point logarithmic grid inKK\. There are zero violations of Theorem[12](https://arxiv.org/html/2609.11063#as1_Thmtheorem12)across all included curves, grid values and ranks\. Over the exponent\-fit window \(ranks 8–64\) the ratioλk/qk\\lambda\_\{k\}/q\_\{k\}has per\-curve median0\.870\.87and median extremes\[0\.69,1\.13\]\[0\.69,1\.13\]; Corollary[9](https://arxiv.org/html/2609.11063#as1_Thmcorollary9)with this measured band and window constant1\.501\.50gives the worst\-case bound\|β−s\|≤0\.37\\lvert\\beta\-s\\rvert\\leq 0\.37, and the measured inheritance \(0\.040\.04median\) sits well inside it\. The profile crossover statistickprof∗k^\{\*\}\_\{\\mathrm\{prof\}\}matches the measured broken\-power crossover rank with median ratio0\.930\.93\(Spearman0\.480\.48,n=56n=56\); per\-model medians track the width empirically, fromkprof∗=172k^\{\*\}\_\{\\mathrm\{prof\}\}=172–229229\(measured264264–319319\) atd=768d=768through557557\(measured551551\) atd=1600d=1600to776776–852852\(measured597597–718718\) atd=2048d=2048\.

The fractional\-frame certificate of Theorem[13](https://arxiv.org/html/2609.11063#as1_Thmtheorem13), evaluated atk=64k=64andb=1/2b=1/2, holds in all 56 evaluated curves in both the standardised and native read\-out representations\. At the smallest admissible truncation in each representation, the median surviving Gram\-mode fractions are0\.7000\.700and0\.5720\.572, at medianK=112K=112andK=145K=145, respectively\. These different\-truncation statistics quantify spectral resolution rather than identification\.

##### Prediction on a held\-out battery across external model families

Design choices were fixed using Pythia and GPT\-2, before exact spectra were revealed for a text\-disjoint 64\-context battery in nine external\-family models: two BLOOM, two GPT\-Neo, two Mamba, two Qwen2 and one RWKV checkpoint\. The predictor uses the standardised centred read\-out representation\. Writinguau\_\{a\}for its rows andu¯=∑apa​ua\\bar\{u\}=\\sum\_\{a\}p\_\{a\}u\_\{a\}, each model–context cell has centred weighted profileqa=pa​∥ua−u¯∥2q\_\{a\}=p\_\{a\}\\lVert u\_\{a\}\-\\bar\{u\}\\rVert^\{2\}\. The construction setsλ^k=q\(k\)\\widehat\{\\lambda\}\_\{k\}=q\_\{\(k\)\}and applies the effective\-dimension transform to its leadingrrvalues, givingN^eff​\(α\)=∑k=1rq\(k\)/\(q\(k\)\+α\)\\widehat\{N\}\_\{\\mathrm\{eff\}\}\(\\alpha\)=\\sum\_\{k=1\}^\{r\}q\_\{\(k\)\}/\(q\_\{\(k\)\}\+\\alpha\)at ten fixed relative dampings\. The measured effective dimension is instead given by the identityNeff​\(α\)=∑kλk/\(λk\+α\)N\_\{\\mathrm\{eff\}\}\(\\alpha\)=\\sum\_\{k\}\\lambda\_\{k\}/\(\\lambda\_\{k\}\+\\alpha\)evaluated after the exact spectrum is revealed\. No eigenvalue or fitted parameter enters the prediction\.

Supplementary Table[2](https://arxiv.org/html/2609.11063#as1_S6.T2)reports family medians and crossed\-bootstrap 95% intervals over the full model\-by\-context grid\. The certificate column is the fraction of cells admitting a measured frame\-and\-tail inheritance certificate\. The final three columns are multiplicative reductions in paired error relative to a same\-trace flat spectrum, a random permutation of profile ranks, and an effective\-dimension curve induced by the same\-trace flat spectrum, respectively, on a separately fixed 16\-context\-per\-model battery\.

Supplementary Table 2:Profile prediction on a held\-out battery across external model families\.Errors and certificate rates are family summaries; the final three columns give paired error reductions relative to the indicated matched alternatives\.Across the full 1,664\-cell evaluation, the trace identity, profile reproduction, forward reproduction and majorisation checks have zero violations, as do all applicable inheritance bands\. Exact certificates are available in 98\.4–100% of each external family’s cells; where no admissible frame truncation exists, no band is asserted\. The profile prediction generalises on the held\-out battery across all five external families: it reproduces the spectral profile and effective\-dimension curve and outperforms the matched flat\-spectrum comparator, the rank\-shuffled comparator and the effective\-dimension curve induced by the same\-trace flat spectrum in every family\. The matched alternatives quantify the accuracy lost when spectral structure is removed\. Supplementary Figure[1](https://arxiv.org/html/2609.11063#as1_S6.F1)shows the full calibration and the resolution\-dependent residual structure\.

ab

Supplementary Figure 1:Full calibration and residual structure of the token\-profile effective\-dimension prediction\.a, All 5,760 model–context–resolution values from 576 model–context cells in the held\-out battery are summarized as 50 family\-by\-resolution points\. Opaque coloured circles with black borders compare family medians of the normalised effective dimension predicted from the sorted token profile with that measured from the exact spectrum; horizontal and vertical bars are the corresponding interquartile ranges, point–lines join the ten fixed resolution thresholds and the grey line is exact prediction\.b, Measured minus predicted normalised effective dimension as the relative resolution thresholdα/q\(1\)\\alpha/q\_\{\(1\)\}decreases from left to right\. Lines are family medians and ribbons are interquartile ranges\. The modest mid\-resolution overprediction is resolved again at the coarse and fine endpoints\.###### Measured\-spectrum effective\-dimension identity\.

Distribution reproduction was attempted for 80 held\-out effective\-dimension curves against the stored top\-512 probabilities at relative tolerance3×10−43\\times 10^\{\-4\}\. Seventy\-two curves from nine models were reproduced to that tolerance and comprise the measured\-spectrum effective\-dimension analysis; the eight RWKV curves were excluded\. For each included curve, the dense output Fisher was formed in float64 and eigendecomposed, and the effective dimension was evaluated at the stored relative dampings with no fitted parameters\. The median held\-out error is0\.002450\.00245, against0\.004730\.00473for the fitted bounded law on the same curves; the dense core exponent \(median1\.1551\.155\) confirms the previously measured1\.141\.14; and the fitted crossoverα∗\\alpha^\{\*\}coincides with the resolution at which the resolved count reaches half its ceiling \(Spearman1\.0001\.000, median ratio1\.061\.06\)\. The effective dimension is therefore exactly determined by the measured spectrum; the two\-parameter Hill curve accurately interpolates it\.

###### Origin of the tail steepening\.

A factorial attribution varies the profile \(realppagainst a pure power law at the matched core slope\) and the read\-out \(realWUW\_\{U\}against an independent Gaussian read\-out with matched column norms\) at three widths \(dd,d/2d/2,d/4d/4by column subsampling\), for models with vocabulary≤160\\leq 160k\. The steepening of the spectral tail beyond the core window survives the random read\-out with the real profile \(median\+0\.47\+0\.47\) and disappears for the pure power\-law profile with the random read\-out \(−0\.07\-0\.07\), ruling out a finite\-width random\-matrix edge; the real read\-out contributes a secondary component \(\+0\.27\+0\.27\)\. The crossover rank scales with width with a log–log slope of1\.11\.1–1\.31\.3in every cell: the width sets the position of the band edge, the profile’s high\-rank depletion sets the shape of the tail\.

#### 6\.2Separating probability concentration from read\-out structure

The matched mechanism panel compared the weighted profileqa=pa​∥ua−u¯∥2q\_\{a\}=p\_\{a\}\\lVert u\_\{a\}\-\\bar\{u\}\\rVert^\{2\}with the probability\-only profile\(r/V\)​pa\(r/V\)p\_\{a\}in a common 27,450\-outcome conditional representation\. Fourteen models covered two sizes each of Pythia, GPT\-2, GPT\-Neo, Qwen2, BLOOM, Mamba and RWKV\. The 192 fresh contexts per model comprised equal Wikipedia, AG News and LAMBADA subsets\. Absolute effective\-dimension error was the root\-mean\-square count error across ten dampings,α/\(tr⁡H/r\)∈\{30,10,3,1,0\.3,0\.1,0\.03,0\.01,0\.003,0\.0003\}\\alpha/\(\\operatorname\{tr\}H/r\)\\in\\\{30,10,3,1,0\.3,0\.1,0\.03,0\.01,0\.003,0\.0003\\\}\. Exact spectral\-exponent error was measured over ranks 8–32 on 48 contexts per model\. Within\-family medians were computed for each predictor before taking their log ratio\.

In family order BLOOM, GPT\-2, GPT\-Neo, Mamba, Pythia, Qwen2 and RWKV, the weighted/probability\-only log\-error ratios for effective dimension are0\.621490\.62149,0\.164560\.16456,0\.494570\.49457,0\.459380\.45938,0\.430750\.43075,0\.882030\.88203and0\.521990\.52199\. Their equal\-family mean is0\.510680\.51068, favouring the probability\-only profile\. For spectral shape, the corresponding ratios are−0\.49943\-0\.49943,−0\.40051\-0\.40051,−0\.25089\-0\.25089,−0\.21088\-0\.21088,−0\.31919\-0\.31919,−0\.67191\-0\.67191and−0\.29760\-0\.29760; their mean is−0\.37863\-0\.37863, favouring the weighted profile\. Enumerating all 128 family sign patterns gives one\-sidedP=1/128P=1/128for each contrast\. The common\-outcome comparison separates probability concentration from read\-out weighting; it complements the native\-vocabulary external\-family prediction above, whose estimator and outcome representation are retained in main Figurea,b\.

#### 6\.3Singleton tokens, interchangeable clusters and multiscale inheritance

LetΠL\\Pi\_\{L\}denote the conditional\-expectation projector of a partitionLLof outcomes in the weighted coordinate representation, and letU⊤​U=IrU^\{\\top\}U=I\_\{r\}be a resolved rank\-rrframe with explained energy fractionq⁡\(L\)=tr⁡\(U⊤​ΠL​U\)/rq\(L\)=\\operatorname\{tr\}\\bigl\(U^\{\\top\}\\Pi\_\{L\}U\\bigr\)/r\. Whenrank⁡\(ΠL​U\)=r\\operatorname\{rank\}\(\\Pi\_\{L\}U\)=r, writePT=U​U⊤P\_\{T\}=UU^\{\\top\}and letPCP\_\{C\}be the orthogonal projector ontocol⁡\(ΠL​U\)\\operatorname\{col\}\(\\Pi\_\{L\}U\)\.

###### Proposition 5\(clusters as conditional expectations\)\.

ΠL\\Pi\_\{L\}is the orthogonal projector onto partition\-measurable directions; maximising the explained fraction over partitions withKKcells is a weightedKK\-means problem on the frame rows\. Ifrank⁡\(ΠL​U\)=r\\operatorname\{rank\}\(\\Pi\_\{L\}U\)=r, then∥PC−PT∥F2=2​r​\{1−q⁡\(L\)\}\\lVert P\_\{C\}\-P\_\{T\}\\rVert\_\{F\}^\{2\}=2r\\\{1\-q\(L\)\\\}links the explained fraction to projector distance\.

###### Proof\.

Conditional expectation is self\-adjoint, idempotent and has as its image exactly the functions constant on partition cells, so it is the orthogonal projector onto that subspace\. Write the rows of the weighted frame asxa=Ua,:/ρax\_\{a\}=U\_\{a,:\}/\\sqrt\{\\rho\_\{a\}\}\. The isotropy identity∑aρa​xa​xa⊤=Ir\\sum\_\{a\}\\rho\_\{a\}x\_\{a\}x\_\{a\}^\{\\top\}=I\_\{r\}gives

r⁡\{1−q⁡\(L\)\}=∑C∈L∑a∈Cρa​∥xa−x¯C∥2,x¯C=∑a∈Cρa​xa∑a∈Cρa\.r\\\{1\-q\(L\)\\\}=\\sum\_\{C\\in L\}\\sum\_\{a\\in C\}\\rho\_\{a\}\\lVert x\_\{a\}\-\\bar\{x\}\_\{C\}\\rVert^\{2\},\\qquad\\bar\{x\}\_\{C\}=\\frac\{\\sum\_\{a\\in C\}\\rho\_\{a\}x\_\{a\}\}\{\\sum\_\{a\\in C\}\\rho\_\{a\}\}\.Thus maximising capture is exactly weightedKK\-means\. If the projected frame retains rankrr, the projector formula givestr⁡\(PC​PT\)=tr⁡\(U⊤​ΠL​U\)=r​q​\(L\)\\operatorname\{tr\}\(P\_\{C\}P\_\{T\}\)=\\operatorname\{tr\}\(U^\{\\top\}\\Pi\_\{L\}U\)=rq\(L\)\. The principal\-angle identity for two rank\-rrprojectors then gives∥PC−PT∥F2=2​r−2​tr⁡\(PC​PT\)=2​r​\{1−q⁡\(L\)\}\\lVert P\_\{C\}\-P\_\{T\}\\rVert\_\{\\rm F\}^\{2\}=2r\-2\\operatorname\{tr\}\(P\_\{C\}P\_\{T\}\)=2r\\\{1\-q\(L\)\\\}\. ∎

###### Theorem 16\(interchangeability from capture\)\.

Letqmin​\(L\)=λmin​\(U⊤​ΠL​U\)q\_\{\\min\}\(L\)=\\lambda\_\{\\min\}\(U^\{\\top\}\\Pi\_\{L\}U\)\. For any resolved vectory=U​zy=Uzand any within\-cluster redistributionddsatisfyingΠL​d=0\\Pi\_\{L\}d=0,

\|⟨d,y⟩\|≤∥d∥​1−qmin​\(L\)​∥y∥\.\|\\langle d,y\\rangle\|\\leq\\lVert d\\rVert\\,\\sqrt\{1\-q\_\{\\min\}\(L\)\}\\,\\lVert y\\rVert\.\(41\)Ifqmin​\(L\)=1q\_\{\\min\}\(L\)=1, every resolved observable is exactly invariant to redistribution within the cells ofLL\. The statement concerns the resolved geometry; the full output distribution can still distinguish tokens inside a cell\.

###### Proof\.

SinceΠL​d=0\\Pi\_\{L\}d=0andΠL\\Pi\_\{L\}is self\-adjoint,⟨d,y⟩=⟨d,\(I−ΠL\)​y⟩\\langle d,y\\rangle=\\langle d,\(I\-\\Pi\_\{L\}\)y\\rangle\. Cauchy–Schwarz gives the first factor\. Moreover

∥\(I−ΠL\)​U​z∥2=z⊤​\{I−U⊤​ΠL​U\}​z≤\{1−qmin​\(L\)\}​∥z∥2\.\\lVert\(I\-\\Pi\_\{L\}\)Uz\\rVert^\{2\}=z^\{\\top\}\\\{I\-U^\{\\top\}\\Pi\_\{L\}U\\\}z\\leq\\\{1\-q\_\{\\min\}\(L\)\\\}\\lVert z\\rVert^\{2\}\.BecauseUUis orthonormal,∥z∥=∥y∥\\lVert z\\rVert=\\lVert y\\rVert\. This proves Eq\. \([41](https://arxiv.org/html/2609.11063#as1_S6.E41)\); the exact case follows by settingqmin=1q\_\{\\min\}=1\. ∎

The empirical partition has two regimes\. On the fixed language\-mode battery, weightedKK\-means achieves mean capture0\.7180\.718and worst\-direction capture0\.2520\.252, both well above the capture levels specified before measurement\. Twenty\-six of 128 cells contain fewer than eight tokens, including six singleton cells\. Those cells contain probability mass0\.33150\.3315, and every such cell contains a token in the top 158 by reference probability\. The high\-probability head therefore requires singleton or near\-singleton resolution, while the remaining mass forms genuine interchangeable clusters\. Specificity, reference\-swap and outer\-half replications recover the same structure\.

Algebraically, letJexJ\_\{\\rm ex\}andJbkJ\_\{\\rm bk\}be the coordinate projectors onto the exceptional singleton block and clustered bulk, respectively\. The partition respects this split\. PutΠg=Jg​ΠL​Jg\\Pi\_\{g\}=J\_\{g\}\\Pi\_\{L\}J\_\{g\},Ag=U⊤​Πg​UA\_\{g\}=U^\{\\top\}\\Pi\_\{g\}UandEg=U⊤​\(Jg−Πg\)​UE\_\{g\}=U^\{\\top\}\(J\_\{g\}\-\\Pi\_\{g\}\)Uforg∈\{ex,bk\}g\\in\\\{\\mathrm\{ex\},\\mathrm\{bk\}\\\}\. These matrices are positive semidefinite and give the exact identity

Ir=Aex\+Abk\+Eex\+Ebk,I\_\{r\}=A\_\{\\rm ex\}\+A\_\{\\rm bk\}\+E\_\{\\rm ex\}\+E\_\{\\rm bk\},where the twoAAterms are captured information and the twoEEterms are unresolved residuals\. The exceptional contribution has rank at most the singleton budget\. This separates a probability\-declared head from a support\-rich bulk and prevents a high total capture from hiding poor bulk capture\.

Fresh\-context tests do not support a universal fixed partition: in a held\-out test of the candidate partitions, the proposed within\-cell attenuation failed in all six model\-halves, with within\-to\-cross median\-response ratios of1\.221\.22–3\.113\.11\. A separate cross\-model partition comparison finds bulk agreement1\.01\.0against a matched\-null median0\.9790\.979in both splits, which is descriptive evidence of shared partition structure\. The actual\-Fisher inheritance analysis recovers all 176 theorem certificates and the primary inheritance effect, while the stronger half\-sample directionality estimates do not support a directional effect in any arm\. Together these results establish the within\-battery singleton\-head/cluster\-bulk anatomy and the exact interchangeability theorem rather than a universal fixed partition across contexts\.

### 7Acquisition time on a logarithmic scale

This section gives the acquisition evidence reported here\. It distinguishes the statistical schedule measured on pretrained\-model checkpoint ladders from the causal depth intervention, then connects both to the geometric motion and margin results below\.

For a factff, lett⁡\(f\)t\(f\)be the first checkpoint after which its signed answer margin remains above a fixed threshold specified before model evaluation, and define its logarithmic acquisition time

τ⁡\(f\)=log2⁡\{t⁡\(f\)/t0\}\.\\tau\(f\)=\\log\_\{2\}\\\{t\(f\)/t\_\{0\}\\\}\.\(42\)The surviving depth effect is a translation in this logarithmic time coordinate, and alignment by held\-out loss across model sizes is more accurate than alignment by training step\.

#### 7\.1Corpusnn\-gram statistics predict acquisition before training

Across Pythia\-70M, Pythia\-160M and Pythia\-410M, the margin trajectory of a fact is predicted from its ex\-ante corpus margins at unigram, bigram and trigram resolution\. At 70M and 160M, the selected log\-linear model hands weight from coarse to fine statistics during training; at the final checkpoints the trigram and bigram terms dominate, with the trigram\-to\-unigram weight ratio ranging from2\.72\.7to4\.94\.9\. Without refitting on fresh facts with zero prefix overlap, the predictor explains0\.7750\.775–0\.7920\.792of margin\-trajectory variance, attains acquisition\-status agreement0\.7920\.792–0\.8540\.854and predicts continuous acquisition time with error0\.770\.77–0\.960\.96inτ\\tau\. The corresponding errors are1\.041\.04,2\.202\.20and0\.980\.98for the constant acquisition\-time baseline and2\.022\.02,2\.562\.56and2\.342\.34for the label\-permutation baseline\. Corpusnn\-gram margins therefore predict the acquisition schedule before training\.

#### 7\.2Evidence depth causes a multiplicative delay

The causal experiment assigns the same synthetic facts to five evidence depths while sharing initial weights and batch streams across arms\. Every statistic below the assigned depth has zero margin and the complete deciding margin appears at and above it\. For the fixed lower\-tail quantile bandp∈\[0\.10,0\.20\]p\\in\[0\.10,0\.20\],

Qp​\{τ∣R​4\}−Qp​\{τ∣R​0\}≃Δ,Δ=2\.103​\[1\.79,2\.40\]\.Q\_\{p\}\\\{\\tau\\mid R4\\\}\-Q\_\{p\}\\\{\\tau\\mid R0\\\}\\simeq\\Delta,\\qquad\\Delta=2\.103\\ \[1\.79,2\.40\]\.\(43\)Equivalently, deepest evidence delays acquisition by a factor2Δ≃4\.32^\{\\Delta\}\\simeq 4\.3in training steps on this band\. The fitted shift varies by only0\.1670\.167inτ\\tauacross the analysed quantiles, small relative toΔ=2\.103\\Delta=2\.103\. TheR​4−R​2R4\-R2difference inτ\\tauis1\.3651\.365\[1\.14,1\.59\]\[1\.14,1\.59\], approximately2\.62\.6\-fold in steps\. The near\-uniform translation is supported forR​4−R​0R4\-R0;R​4−R​2R4\-R2independently establishes the direction and magnitude\. The onset of unsigned target–alternative margin change remains stable: depth delays persistent signed commitment\. Random reassignment of the same facts across depth arms, with shared initial weights and batch streams, identifies evidence depth as the cause of the delay while holding fact composition fixed\.

#### 7\.3Generalisation across architectures and evidence constructions

A separate fully crossed crossover exchanges shallow and deep assignment within matched reciprocal\-fact blocks while holding initial parameters, minibatch streams, exposure counts, sequence lengths and the global shallow:deep mixture fixed\. The design crosses three compact four\-layer decoders—absolute\-position/GELU GPT\-2, rotary/GELU GPT\-NeoX and RoPE/RMSNorm/SwiGLU Llama—with distant copy\-back\-reference and four\-cue\-parity controlled languages\. It uses eight seeds and 384 matched reciprocal\-fact blocks per cell, for 96 runs, and evaluates the persistent 0\.75\-nat acquisition time on a 100\-point checkpoint grid through step 44,800\. The crossed bootstrap uses 10,000 resamples of seed and matched block\.

For copy\-back\-reference and four\-cue parity respectively, the mean paired differences inτ\\tauare1\.0171\.017\[0\.876,1\.161\]\[0\.876,1\.161\]and1\.2051\.205\[1\.060,1\.353\]\[1\.060,1\.353\]for GPT\-2,0\.9960\.996\[0\.849,1\.150\]\[0\.849,1\.150\]and1\.3451\.345\[1\.196,1\.497\]\[1\.196,1\.497\]for GPT\-NeoX, and0\.8100\.810\[0\.699,0\.929\]\[0\.699,0\.929\]and1\.5141\.514\[1\.365,1\.669\]\[1\.365,1\.669\]for Llama \(95% intervals; Extended Data Fig\.b,c\)\. All six intervals and all 48 seed means are positive\. All twelve fixed\-treatment and randomised\-label control intervals contain zero\. Averaging architectures equally, parity exceeds copy\-back\-reference by0\.4140\.414\[0\.291,0\.536\]\[0\.291,0\.536\]inτ\\tau; the corresponding within\-architecture contrasts are0\.1890\.189\[−0\.015,0\.384\]\[\-0\.015,0\.384\]for GPT\-2,0\.3490\.349\[0\.145,0\.552\]\[0\.145,0\.552\]for GPT\-NeoX and0\.7040\.704\[0\.500,0\.912\]\[0\.500,0\.912\]for Llama\. Evidence depth therefore delays acquisition across architectures, while evidence construction modulates that delay on average\. This mean paired difference in logarithmic acquisition time complements the lower\-tail translation above and leaves itsΔ=2\.103\\Delta=2\.103value unchanged\.

#### 7\.4Held\-out loss aligns cumulative acquisition curves across scale

Main\-text Fig\.b,c shows the cumulative number of persistently acquired facts across the 70M, 160M and 410M checkpoint ladders in both training\-step and held\-out\-loss coordinates\. In the quantified 70M\-to\-160M comparison, membership sign agreement is0\.920\.92–0\.930\.93, and the trajectory of thenn\-gram predictor aligns approximately three times more accurately by held\-out loss than by training step \(gap0\.1300\.130against0\.4130\.413\), a descriptive comparison\. On the 300\-fact real\-model battery, thenn\-gram predictor reproduces the trajectories independently at all three sizes without refitting; the observational level schedule, absolute cross\-size alignment error under held\-out loss and between\-seed variability threshold are also evaluated at 70M and 160M\. These real\-checkpoint results establish statistical composition but remain observational\. Causal depth identification and effects at matchednn\-gram evidence come from the randomised synthetic system, and the held\-out\-loss alignment advantage is descriptive\.

The resulting acquisition law is a composition of four measured objects:

τ⁡\(f\)≃τ0​\{deciding\-level n\-gram margin\}\+Δ⁡\{depth\},\\tau\(f\)\\simeq\\tau\_\{0\}\\\{\\text\{deciding\-level n\-gram margin\}\\\}\+\\Delta\\\{\\text\{depth\}\\\},\(44\)with the cumulative acquisition curve re\-indexed across scale by held\-out loss\. Equation \([43](https://arxiv.org/html/2609.11063#as1_S7.E43)\) summarises the near\-uniform measured shift over the analysed quantile band; Eq\. \([44](https://arxiv.org/html/2609.11063#as1_S7.E44)\) is the compact first\-order reading that connects the measured population\-level components\. Per\-fact additive fits reachR2R^\{2\}of0\.720\.72–0\.830\.83with a measurednn\-gram\-margin\-by\-depth interaction; the direct causal result is the near\-uniform lower\-tail quantile shift of Eq\. \([43](https://arxiv.org/html/2609.11063#as1_S7.E43)\)\.

### 8Projected\-margin dynamics and conditional acquisition time

The persistent acquisition time above is an empirical population summary\. This note supplies a model\-relative accounting framework for a fixed weighted margin projection and states the additional assumptions needed to infer first\-passage or logarithmic\-time relations from its trajectory\.

Letℱ\\mathcal\{F\}be a finite population of facts, with fixed weightswf≥0w\_\{f\}\\geq 0satisfying∑fwf=1\\sum\_\{f\}w\_\{f\}=1\. Letzf∈ℝRz\_\{f\}\\in\\mathbb\{R\}^\{R\}be fixed, response\-independent resolved evidence coordinates, and letmf​\(θ\)m\_\{f\}\(\\theta\)be the oriented answer margin, so larger values favour the declared answer\. For a parameter pathθ⁡\(t\)\\theta\(t\)define the weighted margin projection

βr​\(t\)=𝔼w​\[zf​r​mf​\(θ⁡\(t\)\)\]\.\\beta\_\{r\}\(t\)=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}m\_\{f\}\(\\theta\(t\)\)\]\.\(45\)The estimand fixes the fact population, weights and response\-independent coordinates and measures the trajectory of the declared weighted margin projection\.

#### 8\.1Exact continuous and finite\-step accounting

###### Theorem 17\(exact rate of change of the weighted margin projection\)\.

Supposeθ\\thetais differentiable atttand everymfm\_\{f\}is differentiable atθ⁡\(t\)\\theta\(t\), and putqf​\(t\)=∇θmf​\(θ⁡\(t\)\)q\_\{f\}\(t\)=\\nabla\_\{\\theta\}m\_\{f\}\(\\theta\(t\)\)andq¯r​\(t\)=𝔼w​\[zf​r​qf​\(t\)\]\\bar\{q\}\_\{r\}\(t\)=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}q\_\{f\}\(t\)\]\. Then:

Instantaneous rate\.

β˙r​\(t\)=𝔼w​\[zf​r​qf​\(t\)⊤​θ˙​\(t\)\]=q¯r​\(t\)⊤​θ˙​\(t\)\.\\dot\{\\beta\}\_\{r\}\(t\)=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}q\_\{f\}\(t\)^\{\\top\}\\dot\{\\theta\}\(t\)\]=\\bar\{q\}\_\{r\}\(t\)^\{\\top\}\\dot\{\\theta\}\(t\)\.\(46\)
Integrated change\.Ifθ\\thetais absolutely continuous on\[t0,t1\]\[t\_\{0\},t\_\{1\}\]and everymfm\_\{f\}isC1C^\{1\}on a neighbourhood of the parameter path, thenβr\\beta\_\{r\}is absolutely continuous, the instantaneous identity holds almost everywhere, and

βr​\(t1\)−βr​\(t0\)=∫t0t1q¯r​\(t\)⊤​θ˙​\(t\)​𝑑t\.\\beta\_\{r\}\(t\_\{1\}\)\-\\beta\_\{r\}\(t\_\{0\}\)=\\int\_\{t\_\{0\}\}^\{t\_\{1\}\}\\bar\{q\}\_\{r\}\(t\)^\{\\top\}\\dot\{\\theta\}\(t\)\\,\\mathrm\{d\}t\.\(47\)

###### Proof\.

The fact set is finite andzf,wfz\_\{f\},w\_\{f\}are fixed, so differentiation commutes with the weighted sum\. The chain rule givesm˙f=qf⊤​θ˙\\dot\{m\}\_\{f\}=q\_\{f\}^\{\\top\}\\dot\{\\theta\}; finite linearity gives Eq\. \([46](https://arxiv.org/html/2609.11063#as1_S8.E46)\)\. Under the stated regularity assumptions, the chain rule for absolutely continuous paths holds almost everywhere, and the fundamental theorem of calculus gives Eq\. \([47](https://arxiv.org/html/2609.11063#as1_S8.E47)\)\. ∎

The fixed\-coordinate assumption is essential\. Ifzf=zf​\(t\)z\_\{f\}=z\_\{f\}\(t\), the product rule adds𝔼w​\[z˙f​r​\(t\)​mf​\(t\)\]\\mathbb\{E\}\_\{w\}\[\\dot\{z\}\_\{fr\}\(t\)m\_\{f\}\(t\)\]to Eq\. \([46](https://arxiv.org/html/2609.11063#as1_S8.E46)\); a checkpoint\-dependent resolver can therefore create apparent movement of the projection even when the margins do not move\.

###### Theorem 18\(exact finite\-step path accounting\)\.

Letθ0,…,θK\\theta\_\{0\},\\ldots,\\theta\_\{K\}be any finite parameter path,Δ​θk=θk\+1−θk\\Delta\\theta\_\{k\}=\\theta\_\{k\+1\}\-\\theta\_\{k\}andΔ​mf​k=mf​\(θk\+1\)−mf​\(θk\)\\Delta m\_\{fk\}=m\_\{f\}\(\\theta\_\{k\+1\}\)\-m\_\{f\}\(\\theta\_\{k\}\)\. Define the finite\-step projected\-margin changeWr​k=𝔼w​\[zf​r​Δ​mf​k\]W\_\{rk\}=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}\\Delta m\_\{fk\}\]\. Then:

Telescope\.

βr​\(K\)−βr​\(0\)=∑k=0K−1Wr​k\.\\beta\_\{r\}\(K\)\-\\beta\_\{r\}\(0\)=\\sum\_\{k=0\}^\{K\-1\}W\_\{rk\}\.\(48\)
Arbitrary tangent\.For any supplied covectorsqf​kq\_\{fk\}, let

Rf​k=Δ​mf​k−qf​k⊤​Δ​θk,q¯r​k=𝔼w​\[zf​r​qf​k\],R¯r​k=𝔼w​\[zf​r​Rf​k\]\.R\_\{fk\}=\\Delta m\_\{fk\}\-q\_\{fk\}^\{\\top\}\\Delta\\theta\_\{k\},\\qquad\\bar\{q\}\_\{rk\}=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}q\_\{fk\}\],\\qquad\\bar\{R\}\_\{rk\}=\\mathbb\{E\}\_\{w\}\[z\_\{fr\}R\_\{fk\}\]\.\(49\)Then exactly

βr​\(K\)−βr​\(0\)=∑k=0K−1\{q¯r​k⊤​Δ​θk\+R¯r​k\}\.\\beta\_\{r\}\(K\)\-\\beta\_\{r\}\(0\)=\\sum\_\{k=0\}^\{K\-1\}\\\{\\bar\{q\}\_\{rk\}^\{\\top\}\\Delta\\theta\_\{k\}\+\\bar\{R\}\_\{rk\}\\\}\.\(50\)
Zero\-remainder criterion\.If every margin is continuously differentiable on thekkth update segment andq~f​k=∫01∇mf​\(θk\+s​Δ​θk\)​𝑑s,\\widetilde\{q\}\_\{fk\}=\\int\_\{0\}^\{1\}\\nabla m\_\{f\}\(\\theta\_\{k\}\+s\\Delta\\theta\_\{k\}\)\\,\\mathrm\{d\}s,then

Rf​k=0⟺\(q~f​k−qf​k\)⊤Δθk=0\.R\_\{fk\}=0\\quad\\Longleftrightarrow\\quad\(\\widetilde\{q\}\_\{fk\}\-q\_\{fk\}\)^\{\\top\}\\Delta\\theta\_\{k\}=0\.\(51\)Thusqf​k=q~f​kq\_\{fk\}=\\widetilde\{q\}\_\{fk\}is sufficient but not necessary: one update identifies only the covector projection along its actual displacement\.

###### Proof\.

Equation \([48](https://arxiv.org/html/2609.11063#as1_S8.E48)\) follows by summingβr​\(k\+1\)−βr​\(k\)\\beta\_\{r\}\(k\+1\)\-\\beta\_\{r\}\(k\)\. SubstitutingΔ​mf​k=qf​k⊤​Δ​θk\+Rf​k\\Delta m\_\{fk\}=q\_\{fk\}^\{\\top\}\\Delta\\theta\_\{k\}\+R\_\{fk\}and using finite linearity gives Eq\. \([50](https://arxiv.org/html/2609.11063#as1_S8.E50)\)\. Along the update segment, the chain rule and fundamental theorem of calculus giveΔ​mf​k=q~f​k⊤​Δ​θk\\Delta m\_\{fk\}=\\widetilde\{q\}\_\{fk\}^\{\\top\}\\Delta\\theta\_\{k\}; subtracting the supplied linear term gives Eq\. \([51](https://arxiv.org/html/2609.11063#as1_S8.E51)\)\. ∎

The endpoint telescope requires no smoothness and is the primary exact finite\-step identity\. Using a left\-endpoint or fixed\-initialisation tangent is also exact only when its remainder is retained\. A zero pooled remainder can conceal cancellation among nonzero fact\-level remainders\.

#### 8\.2Signed rate in a positive\-semidefinite geometry

Fix a mode and time, abbreviateq=q¯rq=\\bar\{q\}\_\{r\}andy=θ˙y=\\dot\{\\theta\}\(ory=Δ​θky=\\Delta\\theta\_\{k\}\), and letA⪰0A\\succeq 0be a declared local metric or effective optimiser geometry\. Supply solve certificates

A​x=q,A​y=g,Ax=q,\\qquad Ay=g,\(52\)whereggis the signed update covector whose metric solve is the actual velocity\. Define

S=q⊤​x,E=g⊤​y=y⊤​A​y,V=q⊤​y,a=VS​E\.S=q^\{\\top\}x,\\qquad E=g^\{\\top\}y=y^\{\\top\}Ay,\\qquad V=q^\{\\top\}y,\\qquad a=\\frac\{V\}\{\\sqrt\{S\}\\sqrt\{E\}\}\.\(53\)
###### Theorem 19\(positive\-semidefinite projected\-margin rate factorisation\)\.

If Eq\. \([52](https://arxiv.org/html/2609.11063#as1_S8.E52)\) holds andS,E\>0S,E\>0, then

V=S​E​a,\|a\|≤1,V2≤S​E\.V=\\sqrt\{S\}\\sqrt\{E\}\\,a,\\qquad\|a\|\\leq 1,\\qquad V^\{2\}\\leq SE\.\(54\)The sign ofaarecords whether the update increases or decreases the weighted margin projection\. For singularAA, the solve certificates requireq,g∈Range⁡\(A\)q,g\\in\\operatorname\{Range\}\(A\); null\-space choices do not alterS,ES,EorVV\. IfS=0S=0orE=0E=0, thenV=0V=0but the normalised alignment is undefined\.

###### Proof\.

Becauseq=A​xq=Axandg=A​yg=Ay, one hasV=x⊤​A​yV=x^\{\\top\}Ay,S=x⊤​A​xS=x^\{\\top\}AxandE=y⊤​A​yE=y^\{\\top\}Ay\. Cauchy–Schwarz for the positive\-semidefinite form givesV2≤S​EV^\{2\}\\leq SE; the signed factorisation and unit bound follow from the definition ofaa\. ∎

#### 8\.3Conditional first\-passage and logarithmic\-time relations

Let two projected\-margin trajectories start from the same levelβ0\\beta\_\{0\}, and putCj​\(k\)=βj​\(k\)−β0C\_\{j\}\(k\)=\\beta\_\{j\}\(k\)\-\\beta\_\{0\}forj∈\{s,d\}j\\in\\\{s,d\\\}\.

###### Theorem 20\(cumulative\-change first\-passage ordering\)\.

Fix a common thresholdhh\.

Exact dominance\.If the deep path first reacheshhat indexkkandCs​\(j\)≥Cd​\(j\)C\_\{s\}\(j\)\\geq C\_\{d\}\(j\)for0≤j≤k0\\leq j\\leq k, then the shallow path has reachedhhby indexkk\. No monotonicity of either path is required\.

Uniform\-error form\.Ifε≥0\\varepsilon\\geq 0andCd​\(j\)≤Cs​\(j\)\+εC\_\{d\}\(j\)\\leq C\_\{s\}\(j\)\+\\varepsilonthroughkk, then a deep first crossing ofh\+εh\+\\varepsilonimplies a shallow crossing ofhhbykk\. The same pointwise argument gives inclusion of continuous\-time hit sets and hence orders their infima whenever the deep hit set is nonempty\.

###### Proof\.

At the deep crossing, exact dominance givesβs​\(k\)=β0\+Cs​\(k\)≥β0\+Cd​\(k\)=βd​\(k\)≥h\\beta\_\{s\}\(k\)=\\beta\_\{0\}\+C\_\{s\}\(k\)\\geq\\beta\_\{0\}\+C\_\{d\}\(k\)=\\beta\_\{d\}\(k\)\\geq h\. Under uniform error,βs​\(k\)≥βd​\(k\)−ε≥h\\beta\_\{s\}\(k\)\\geq\\beta\_\{d\}\(k\)\-\\varepsilon\\geq h\. Earlier or later increments are irrelevant\. ∎

This theorem concerns ordinary first entry\. The empirical acquisition time in Eq\. \([42](https://arxiv.org/html/2609.11063#as1_S7.E42)\) requires the margin to remain above threshold at all later observed checkpoints\. Ordering that persistent acquisition time requires an additional persistence argument; sparse local rate measurements alone do not establish cumulative\-change dominance\.

###### Theorem 21\(conditional exponential response\)\.

Suppose a projected\-margin trajectory has the normalised response

βr​\(t\)=βr,∞−\(βr,∞−βr,0\)​e−κr​t,κr\>0,\\beta\_\{r\}\(t\)=\\beta\_\{r,\\infty\}\-\(\\beta\_\{r,\\infty\}\-\\beta\_\{r,0\}\)e^\{\-\\kappa\_\{r\}t\},\\qquad\\kappa\_\{r\}\>0,withβr,∞\>βr,0\\beta\_\{r,\\infty\}\>\\beta\_\{r,0\}\. At fractional threshold0<ρ<10<\\rho<1,

Tr​\(ρ\)=−log⁡\(1−ρ\)κr\.T\_\{r\}\(\\rho\)=\\frac\{\-\\log\(1\-\\rho\)\}\{\\kappa\_\{r\}\}\.\(55\)Consequentlyκs≥κd\>0\\kappa\_\{s\}\\geq\\kappa\_\{d\}\>0impliesTs​\(ρ\)≤Td​\(ρ\)T\_\{s\}\(\\rho\)\\leq T\_\{d\}\(\\rho\), and

log2⁡Td​\(ρ\)−log2⁡Ts​\(ρ\)=log2⁡κsκd\.\\log\_\{2\}T\_\{d\}\(\\rho\)\-\\log\_\{2\}T\_\{s\}\(\\rho\)=\\log\_\{2\}\\frac\{\\kappa\_\{s\}\}\{\\kappa\_\{d\}\}\.\(56\)The threshold\-independent translation follows from the shared normalised response shape and constant rates; it is not implied by the rate identity alone\.

###### Proof\.

Solving1−e−κr​t=ρ1\-e^\{\-\\kappa\_\{r\}t\}=\\rhogives Eq\. \([55](https://arxiv.org/html/2609.11063#as1_S8.E55)\)\. Positivity of−log⁡\(1−ρ\)\-\\log\(1\-\\rho\)reverses the rate ordering into a time ordering\. Taking the time ratio and its logarithm gives Eq\. \([56](https://arxiv.org/html/2609.11063#as1_S8.E56)\)\. ∎

###### Corollary 11\(distributional scaling\)\.

LetTs,TdT\_\{s\},T\_\{d\}be nonnegative random acquisition times\. IfTd​=𝑑​c​TsT\_\{d\}\\overset\{d\}\{=\}cT\_\{s\}forc\>0c\>0, equivalentlyFd​\(t\)=Fs​\(t/c\)F\_\{d\}\(t\)=F\_\{s\}\(t/c\)for alltt, then every lower quantileQj​\(p\)=inf\{t:Fj​\(t\)≥p\}Q\_\{j\}\(p\)=\\inf\\\{t:F\_\{j\}\(t\)\\geq p\\\}satisfies

Qd​\(p\)=c​Qs​\(p\),0<p<1\.Q\_\{d\}\(p\)=cQ\_\{s\}\(p\),\\qquad 0<p<1\.\(57\)For positive quantiles this is a constant logarithmic\-time translation bylog2⁡c\\log\_\{2\}c\.

###### Proof\.

Multiplication byc\>0c\>0preserves order, so scaling maps the entire CDF level set for the shallow condition onto the corresponding level set for the deep condition\. Its infimum is therefore multiplied bycc\. ∎

Population\-level scaling need not be a same\-fact multiplier\. Conversely, a shift over one quantile band does not by itself establish full distributional scaling\.

###### Theorem 22\(learner\-independent depth monotonicity is impossible\)\.

Consider the positive\-rate cumulative\-change modelTj=Wj/κjT\_\{j\}=W\_\{j\}/\\kappa\_\{j\}, with required changeWj\>0W\_\{j\}\>0and effective rateκj\>0\\kappa\_\{j\}\>0\. Within this model there exist a fixed shallow/deep evidence pair with nominal depthsds<ddd\_\{s\}<d\_\{d\}and two positive rate profiles whose acquisition orders are opposite\. Thus nominal depth alone cannot order acquisition time throughout this model without an assumption on learner\-relative effective rates or an equivalent accessibility object\.

###### Proof\.

Give both items one unit of required change\. The rate profile\(κs,κd\)=\(2,1\)\(\\kappa\_\{s\},\\kappa\_\{d\}\)=\(2,1\)has\(Ts,Td\)=\(1/2,1\)\(T\_\{s\},T\_\{d\}\)=\(1/2,1\), whereas the profile\(1,2\)\(1,2\)has\(Ts,Td\)=\(1,1/2\)\(T\_\{s\},T\_\{d\}\)=\(1,1/2\)\. The evidence depths are unchanged and both rate profiles are positive\. ∎

Randomised evidence\-depth assignment identifies the causal acquisition effect in Supplementary Note[7](https://arxiv.org/html/2609.11063#as1_S7); the weighted\-margin\-projection identities provide conditional path accounting for the declared evidence projection\.

### 9Geometric motion, margins and local objective rate

###### Theorem 23\(continuous scores and discontinuous decisions\)\.

Letρ⁡\(p,q\)=∥p−q∥2\\rho\(p,q\)=\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}\.

Continuous score\.If a scoreSSisLL\-Lipschitz in this metric, then

\|S⁡\(p\)−S⁡\(q\)\|≤L​ρ​\(p,q\)\.\|S\(p\)\-S\(q\)\|\\leq L\\rho\(p,q\)\.\(58\)
Discrete decision\.For a discrete argmax decision, letyybe the unique maximiser ofppandm=py−maxj≠y⁡pjm=p\_\{y\}\-\\max\_\{j\\neq y\}p\_\{j\}its margin\. If the maximiser changes underqq, then

TV⁡\(p,q\)≥m/2,ρ⁡\(p,q\)≥m/2\.\\operatorname\{TV\}\(p,q\)\\geq m/2,\\qquad\\rho\(p,q\)\\geq m/2\.\(59\)

###### Proof\.

Equation \([58](https://arxiv.org/html/2609.11063#as1_S9.E58)\) is the definition of Lipschitz continuity\. Ifz≠yz\\neq ymaximisesqq, thenqz≥qyq\_\{z\}\\geq q\_\{y\}and

m≤py−pz≤\|py−qy\|\+\|pz−qz\|≤∥p−q∥1=2​TV⁡\(p,q\)\.m\\leq p\_\{y\}\-p\_\{z\}\\leq\|p\_\{y\}\-q\_\{y\}\|\+\|p\_\{z\}\-q\_\{z\}\|\\leq\\lVert p\-q\\rVert\_\{1\}=2\\operatorname\{TV\}\(p,q\)\.FinallyTV⁡\(p,q\)≤ρ⁡\(p,q\)\\operatorname\{TV\}\(p,q\)\\leq\\rho\(p,q\)follows from∥p−q∥1≤2​∥p−q∥2\\lVert p\-q\\rVert\_\{1\}\\leq 2\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\}\. This proves Eq\. \([59](https://arxiv.org/html/2609.11063#as1_S9.E59)\)\. ∎

The continuous bound is checked without violation in 5,704 implementation cases\. The discrete margin theorem has zero violations over 45,315 observed cross\-seed flips, or 90,630 oriented checks\. On the held\-out LAMBADA battery\[[4](https://arxiv.org/html/2609.11063#as1_bib.bib4)\]the flip frequency falls from0\.5190\.519to zero as the normalised marginz=m/\(2​ρ\)z=m/\(2\\rho\)crosses one, where zero is forced by the theorem\. This connects a smooth geometric acquisition trajectory to an apparently abrupt benchmark transition\.

###### Theorem 24\(Cauchy–Schwarz bound for the objective rate\)\.

LetAAbe positive definite, letqqbe a behaviour covector andgga training\-loss covector, and solveA​x=qAx=q,A​y=gAy=g\. Define

sensitivity=q⊤​x,update norm2=g⊤​y,objective rate=q⊤​y\.\\text\{sensitivity\}=q^\{\\top\}x,\\qquad\\text\{update norm\}^\{2\}=g^\{\\top\}y,\\qquad\\text\{objective rate\}=q^\{\\top\}y\.Then

objective rate2≤sensitivity×update norm2\.\\text\{objective rate\}^\{2\}\\leq\\text\{sensitivity\}\\times\\text\{update norm\}^\{2\}\.\(60\)When both quadratic forms are positive, definealignment=⟨x,y⟩A/\(∥x∥A​∥y∥A\)\\text\{alignment\}=\\langle x,y\\rangle\_\{A\}/\(\\lVert x\\rVert\_\{A\}\\lVert y\\rVert\_\{A\}\)\. Then exactly

objective rate2=sensitivity×update norm2×alignment2\.\\text\{objective rate\}^\{2\}=\\text\{sensitivity\}\\times\\text\{update norm\}^\{2\}\\times\\text\{alignment\}^\{2\}\.\(61\)Equivalently,objective rate2/sensitivity≤update norm2\\text\{objective rate\}^\{2\}/\\text\{sensitivity\}\\leq\\text\{update norm\}^\{2\}\.

###### Proof\.

Use theAA\-inner product⟨u,v⟩A=u⊤​A​v\\langle u,v\\rangle\_\{A\}=u^\{\\top\}Av\. Thensensitivity=∥x∥A2\\text\{sensitivity\}=\\lVert x\\rVert\_\{A\}^\{2\},update norm2=∥y∥A2\\text\{update norm\}^\{2\}=\\lVert y\\rVert\_\{A\}^\{2\}andobjective rate=q⊤​y=x⊤​A​y=⟨x,y⟩A\\text\{objective rate\}=q^\{\\top\}y=x^\{\\top\}Ay=\\langle x,y\\rangle\_\{A\}\. Cauchy–Schwarz proves Eq\. \([60](https://arxiv.org/html/2609.11063#as1_S9.E60)\); dividing its two sides by the product of the positive quadratic forms and using the definition of alignment gives Eq\. \([61](https://arxiv.org/html/2609.11063#as1_S9.E61)\)\. ∎

Objective sensitivity alone therefore cannot determine acquisition rate\. WithA=IA=I,q=\(1,0\)q=\(1,0\)and unit training covectors\(1,0\)\(1,0\)and\(0,1\)\(0,1\), sensitivity and squared update norm are identical but the objective rate is one or zero\. On 80 held\-out Pythia\-70M examples, evaluated with a declared final\-read\-out preconditioner, mean alignment squared is0\.68110\.6811\(95% interval0\.65030\.6503–0\.71170\.7117\)\. The factorisation holds exactly and the correlation between log sensitivity and log squared objective rate is0\.5410\.541\. The missing quantity linking controllability to learning is objective–update alignment\.

###### Proposition 6\(read\-out subspace kinematics\)\.

LetQ⁡\(t\)Q\(t\)have orthonormal columns and setP⁡\(t\)=Q⁡\(t\)​Q​\(t\)⊤P\(t\)=Q\(t\)Q\(t\)^\{\\top\}andB=\(I−P\)​Q˙B=\(I\-P\)\\dot\{Q\}\. Then

P˙=B​Q⊤\+Q​B⊤,Q⊤​B=0,∥P˙∥F2=2​∥B∥F2\.\\dot\{P\}=BQ^\{\\top\}\+QB^\{\\top\},\\qquad Q^\{\\top\}B=0,\\qquad\\lVert\\dot\{P\}\\rVert\_\{\\rm F\}^\{2\}=2\\lVert B\\rVert\_\{\\rm F\}^\{2\}\.\(62\)Only the normal componentBBrotates the subspace\. In particular, an update confined to the current column span, including scalar decoupled weight decay, produces no rotation\.

###### Proof\.

DifferentiateP=Q​Q⊤P=QQ^\{\\top\}\. DifferentiatingQ⊤​Q=IQ^\{\\top\}Q=Ishows thatQ⊤​Q˙Q^\{\\top\}\\dot\{Q\}is skew\-symmetric, so its two contributions toP˙\\dot\{P\}cancel\. The remaining normal component isBB, giving the first identity andQ⊤​B=0Q^\{\\top\}B=0\. Orthogonality makes the two summands Frobenius\-orthogonal and equal in norm, which proves the final identity\. IfQ˙\\dot\{Q\}is in the span ofQQ, thenB=0B=0\. ∎

The same approximation residual that governs static recovery also controls how strongly unresolved language can rotate the read\-out subspace\.

###### Theorem 25\(language residual bounds read\-out rotation\)\.

Fix a strictly positive reference lawρ\\rhoand put

Pρ=I−ρ​ρ⊤,Cρ=Pρ​diag⁡\(ρ\)1/2,κρ=∥Cρ∥op\.P\_\{\\rho\}=I\-\\sqrt\{\\rho\}\\sqrt\{\\rho\}^\{\\,\\top\},\\qquad C\_\{\\rho\}=P\_\{\\rho\}\\operatorname\{diag\}\(\\rho\)^\{1/2\},\\qquad\\kappa\_\{\\rho\}=\\lVert C\_\{\\rho\}\\rVert\_\{\\rm op\}\.For an untied affine read\-outWtW\_\{t\}, letAt=Cρ​WtA\_\{t\}=C\_\{\\rho\}W\_\{t\},Rt=At⊤​AtR\_\{t\}=A\_\{t\}^\{\\top\}A\_\{t\},Ut=AtRt−1/2U\_\{t\}=A\_\{t\}R\_\{t\}^\{\-1/2\}andPt=Ut​Ut⊤P\_\{t\}=U\_\{t\}U\_\{t\}^\{\\top\}on a constant\-full\-rank interval\. Let the true next\-token law beπ~X\\widetilde\{\\pi\}\_\{X\}, letπX\\pi\_\{X\}be a declared resolved core, and define

ζ=𝔼X∥π~X−πX∥2,ψt\(X\)=Rt−1/2ht\(X\),Ψt=\{𝔼∥ψt\(X\)∥2\}1/2\.\\zeta=\\mathbb\{E\}\_\{X\}\\lVert\\sqrt\{\\widetilde\{\\pi\}\_\{X\}\}\-\\sqrt\{\\pi\_\{X\}\}\\rVert^\{2\},\\qquad\\psi\_\{t\}\(X\)=R\_\{t\}^\{\-1/2\}h\_\{t\}\(X\),\\qquad\\Psi\_\{t\}=\\\{\\mathbb\{E\}\\lVert\\psi\_\{t\}\(X\)\\rVert^\{2\}\\\}^\{1/2\}\.Under the population read\-out flow

W˙t=γt𝔼\[\(eY−pt\)ht\(X\)⊤\]−λtWt\+𝒟t,Y∣X∼π~X,γt≥0,\\dot\{W\}\_\{t\}=\\gamma\_\{t\}\\mathbb\{E\}\[\(e\_\{Y\}\-p\_\{t\}\)h\_\{t\}\(X\)^\{\\top\}\]\-\\lambda\_\{t\}W\_\{t\}\+\\mathcal\{D\}\_\{t\},\\qquad Y\\mid X\\sim\\widetilde\{\\pi\}\_\{X\},\\qquad\\gamma\_\{t\}\\geq 0,the horizontal velocityBt=\(I−Pt\)A˙tRt−1/2B\_\{t\}=\(I\-P\_\{t\}\)\\dot\{A\}\_\{t\}R\_\{t\}^\{\-1/2\}decomposes exactly as

Bt=Btcore\+Etlang\+Etmech,B\_\{t\}=B\_\{t\}^\{\\rm core\}\+E\_\{t\}^\{\\rm lang\}\+E\_\{t\}^\{\\rm mech\},\(63\)where

Btcore\\displaystyle B\_\{t\}^\{\\rm core\}=γt​\(I−Pt\)​Cρ​𝔼​\[\(πX−pt\)​ψt​\(X\)⊤\],\\displaystyle=\\gamma\_\{t\}\(I\-P\_\{t\}\)C\_\{\\rho\}\\mathbb\{E\}\[\(\\pi\_\{X\}\-p\_\{t\}\)\\psi\_\{t\}\(X\)^\{\\top\}\],Etlang\\displaystyle E\_\{t\}^\{\\rm lang\}=γt​\(I−Pt\)​Cρ​𝔼​\[\(π~X−πX\)​ψt​\(X\)⊤\],\\displaystyle=\\gamma\_\{t\}\(I\-P\_\{t\}\)C\_\{\\rho\}\\mathbb\{E\}\[\(\\widetilde\{\\pi\}\_\{X\}\-\\pi\_\{X\}\)\\psi\_\{t\}\(X\)^\{\\top\}\],Etmech\\displaystyle E\_\{t\}^\{\\rm mech\}=\(I−Pt\)Cρ𝒟tRt−1/2\.\\displaystyle=\(I\-P\_\{t\}\)C\_\{\\rho\}\\mathcal\{D\}\_\{t\}R\_\{t\}^\{\-1/2\}\.Scalar weight decay contributes exactly zero, and

∥Etlang∥F≤2​γt​κρ​Ψt​ζ,∥P˙tactual−P˙tcore\+mech∥F≤2​2​γt​κρ​Ψt​ζ\.\\lVert E\_\{t\}^\{\\rm lang\}\\rVert\_\{\\rm F\}\\leq 2\\gamma\_\{t\}\\kappa\_\{\\rho\}\\Psi\_\{t\}\\sqrt\{\\zeta\},\\qquad\\lVert\\dot\{P\}\_\{t\}^\{\\rm actual\}\-\\dot\{P\}\_\{t\}^\{\\rm core\+mech\}\\rVert\_\{\\rm F\}\\leq 2\\sqrt\{2\}\\gamma\_\{t\}\\kappa\_\{\\rho\}\\Psi\_\{t\}\\sqrt\{\\zeta\}\.\(64\)

###### Proof\.

Conditioning onXXgives𝔼⁡\[eY∣X\]=π~X\\mathbb\{E\}\[e\_\{Y\}\\mid X\]=\\widetilde\{\\pi\}\_\{X\}\. ApplyCρC\_\{\\rho\}to the flow, substitute intoBtB\_\{t\}, add and subtractπX\\pi\_\{X\}, and use\(I−Pt\)​At=0\(I\-P\_\{t\}\)A\_\{t\}=0\. This proves Eq\. \([63](https://arxiv.org/html/2609.11063#as1_S9.E63)\), including the zero contribution of−λt​At\-\\lambda\_\{t\}A\_\{t\}\. For probability vectors,

∥p−q∥2≤2​∥p−q∥2,\\lVert p\-q\\rVert\_\{2\}\\leq 2\\lVert\\sqrt\{p\}\-\\sqrt\{q\}\\rVert\_\{2\},because\|pi−qi\|=\|pi−qi\|\(pi\+qi\)\|p\_\{i\}\-q\_\{i\}\|=\|\\sqrt\{p\}\_\{i\}\-\\sqrt\{q\}\_\{i\}\|\(\\sqrt\{p\}\_\{i\}\+\\sqrt\{q\}\_\{i\}\)and the final factor is at most two\. Projection is contractive, so Frobenius Cauchy–Schwarz gives

∥Etlang∥F≤γt​κρ​\{𝔼​∥π~X−πX∥22\}1/2​Ψt≤2​γt​κρ​Ψt​ζ\.\\lVert E\_\{t\}^\{\\rm lang\}\\rVert\_\{\\rm F\}\\leq\\gamma\_\{t\}\\kappa\_\{\\rho\}\\\{\\mathbb\{E\}\\lVert\\widetilde\{\\pi\}\_\{X\}\-\\pi\_\{X\}\\rVert\_\{2\}^\{2\}\\\}^\{1/2\}\\Psi\_\{t\}\\leq 2\\gamma\_\{t\}\\kappa\_\{\\rho\}\\Psi\_\{t\}\\sqrt\{\\zeta\}\.Finally, Proposition[6](https://arxiv.org/html/2609.11063#as1_Thmproposition6)applied to the velocity differenceEtlangE\_\{t\}^\{\\rm lang\}multiplies its Frobenius norm by2\\sqrt\{2\}, proving the projector bound\. ∎

A controlled smooth\-path experiment supports the projector identity, the scalar\-decay null and Eq\. \([64](https://arxiv.org/html/2609.11063#as1_S9.E64)\) in all 24 held\-out seeds\. The corpusnn\-gram, evidence\-depth and held\-out\-loss alignment experiments establish the acquisition\-time results above\.

###### Cyclic concepts and relation\-induced probability displacements\.

For a cyclic registry \(weekdays, months\), the advance\-one\-step map is fitted as the metric isometry \(rotation\) in the Fourier\-1 phase representation that best carries each member’s distribution to the next\. The rotation angle is compared with the ideal2​π/n2\\pi/n\(weekdays0\.9020\.902vs2​π/7=0\.8982\\pi/7=0\.898\)\. The off\-rotation component is the metric\-isometry defect reported in the double dissociation \(0\.290\.29for cyclic concepts,1\.021\.02for relations\), and the closure defect is the Fisher–Rao distance between the start point and the composition ofnnsteps\.

Relations \(capital\-of, gender, tense, comparative\) are scored by probability\-displacement alignment, the mean cosine alignment of the probability\-displacement directions across instances \(raw0\.490\.49for relations, near zero for cyclic\), reported against a permuted\-geometry control that preserves marginals \(0\.330\.33\)\.

## Part IV: Controllable

### 10Minimum\-disturbance control and coordinate dependence of Euclidean cost

This section collects consequences of the chain that the operations of the main text use directly\.

LetAAbe a positive\-definite local cost metric,q≠0q\\neq 0a behaviour covector andb≠0b\\neq 0a desired first\-order change, so feasible interventions satisfyq⊤​δ=bq^\{\\top\}\\delta=b\. For the undamped output problemA=GA=Gon the resolved tangent space; for the stable finite\-resolution problemA=G\+α​RA=G\+\\alpha R, whereRRis the declared reference metric\.

###### Theorem 26\(minimum disturbance and the exact relative cost of Euclidean control\)\.

Minimum\.The unique minimum\-cost intervention is

δN=b​A−1​qq⊤​A−1​q,12​δN⊤​A​δN=b22​q⊤​A−1​q\.\\delta\_\{\\rm N\}=\\frac\{bA^\{\-1\}q\}\{q^\{\\top\}A^\{\-1\}q\},\\qquad\\frac\{1\}\{2\}\\delta\_\{\\rm N\}^\{\\top\}A\\delta\_\{\\rm N\}=\\frac\{b^\{2\}\}\{2q^\{\\top\}A^\{\-1\}q\}\.\(65\)
Exact gap\.Every feasibleδ\\deltaobeys the exact gap identity

12​δ⊤​A​δ−b22​q⊤​A−1​q=12​∥δ−δN∥A2\.\\frac\{1\}\{2\}\\delta^\{\\top\}A\\delta\-\\frac\{b^\{2\}\}\{2q^\{\\top\}A^\{\-1\}q\}=\\frac\{1\}\{2\}\\lVert\\delta\-\\delta\_\{\\rm N\}\\rVert\_\{A\}^\{2\}\.\(66\)
Euclidean\-to\-natural cost ratio\.In particular, the Euclidean stepδE=b​q/\(q⊤​q\)\\delta\_\{\\rm E\}=bq/\(q^\{\\top\}q\)has cost ratio

ΓA​\(q\):=δE⊤​A​δEδN⊤​A​δN=\(q⊤​A​q\)​\(q⊤​A−1​q\)\(q⊤​q\)2≥1\.\\Gamma\_\{A\}\(q\):=\\frac\{\\delta\_\{\\rm E\}^\{\\top\}A\\delta\_\{\\rm E\}\}\{\\delta\_\{\\rm N\}^\{\\top\}A\\delta\_\{\\rm N\}\}=\\frac\{\(q^\{\\top\}Aq\)\(q^\{\\top\}A^\{\-1\}q\)\}\{\(q^\{\\top\}q\)^\{2\}\}\\geq 1\.\(67\)
Spectral bound\.If the spectrum ofAAlies in\[m,M\]\[m,M\], then

1≤ΓA​\(q\)≤\(M\+m\)24​M​m\.1\\leq\\Gamma\_\{A\}\(q\)\\leq\\frac\{\(M\+m\)^\{2\}\}\{4Mm\}\.\(68\)Equality on the left holds precisely whenqqis supported on one eigenspace ofAA\.

Damping\.ForB=B⊤⪰0B=B^\{\\top\}\\succeq 0,α\>0\\alpha\>0andAα=B\+α​IA\_\{\\alpha\}=B\+\\alpha I,ΓAα​\(q\)\\Gamma\_\{A\_\{\\alpha\}\}\(q\)and its worst\-case ceiling are non\-increasing inα\\alphaand converge to one: damping removes the exploitable anisotropy together with the natural\-gradient advantage\.

###### Proof\.

Lagrange stationarity for minimising12​δ⊤​A​δ\\tfrac\{1\}\{2\}\\delta^\{\\top\}A\\deltasubject toq⊤​δ=bq^\{\\top\}\\delta=bgivesA​δ=λ​qA\\delta=\\lambda q; imposing the constraint gives Eq\. \([65](https://arxiv.org/html/2609.11063#as1_S10.E65)\)\. For any other feasibleδ=δN\+e\\delta=\\delta\_\{\\rm N\}\+e,q⊤​e=0q^\{\\top\}e=0and henceδN⊤​A​e=b​q⊤​e/\(q⊤​A−1​q\)=0\\delta\_\{\\rm N\}^\{\\top\}Ae=bq^\{\\top\}e/\(q^\{\\top\}A^\{\-1\}q\)=0\. Expanding the quadratic proves Eq\. \([66](https://arxiv.org/html/2609.11063#as1_S10.E66)\); positive definiteness gives uniqueness\. SubstitutingδE\\delta\_\{\\rm E\}andδN\\delta\_\{\\rm N\}proves Eq\. \([67](https://arxiv.org/html/2609.11063#as1_S10.E67)\)\.

For the bounds, diagonaliseAAand putai=qi2/\(q⊤​q\)a\_\{i\}=q\_\{i\}^\{2\}/\(q^\{\\top\}q\), so∑iai=1\\sum\_\{i\}a\_\{i\}=1\. Then

ΓA​\(q\)=\(∑iai​λi\)​\(∑iaiλi\)\.\\Gamma\_\{A\}\(q\)=\\left\(\\sum\_\{i\}a\_\{i\}\\lambda\_\{i\}\\right\)\\left\(\\sum\_\{i\}\\frac\{a\_\{i\}\}\{\\lambda\_\{i\}\}\\right\)\.Cauchy–Schwarz gives the lower bound\. Moreover\(λi−m\)​\(M−λi\)≥0\(\\lambda\_\{i\}\-m\)\(M\-\\lambda\_\{i\}\)\\geq 0impliesλi\+m​M/λi≤M\+m\\lambda\_\{i\}\+mM/\\lambda\_\{i\}\\leq M\+m\. Averaging and applying the arithmetic–geometric mean inequality gives

2​m​M​ΓA​\(q\)≤∑iai​λi\+m​M​∑iaiλi≤M\+m,2\\sqrt\{mM\\Gamma\_\{A\}\(q\)\}\\leq\\sum\_\{i\}a\_\{i\}\\lambda\_\{i\}\+mM\\sum\_\{i\}\\frac\{a\_\{i\}\}\{\\lambda\_\{i\}\}\\leq M\+m,which proves Eq\. \([68](https://arxiv.org/html/2609.11063#as1_S10.E68)\)\. Equality in the lower Cauchy–Schwarz bound requires a common eigenvalue whereverai\>0a\_\{i\}\>0\.

Finally, withxi=λi​\(B\)\+αx\_\{i\}=\\lambda\_\{i\}\(B\)\+\\alphaand weighted averages denoted by𝔼a\\mathbb\{E\}\_\{a\},

dd​α​ΓAα​\(q\)=𝔼a​\(x−1\)−𝔼a​\(x\)​𝔼a​\(x−2\)≤0,\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}\\Gamma\_\{A\_\{\\alpha\}\}\(q\)=\\mathbb\{E\}\_\{a\}\(x^\{\-1\}\)\-\\mathbb\{E\}\_\{a\}\(x\)\\mathbb\{E\}\_\{a\}\(x^\{\-2\}\)\\leq 0,becausexxandx−2x^\{\-2\}are oppositely ordered, so𝔼a​\(x​x−2\)≤𝔼a​\(x\)​𝔼a​\(x−2\)\\mathbb\{E\}\_\{a\}\(x\\,x^\{\-2\}\)\\leq\\mathbb\{E\}\_\{a\}\(x\)\\mathbb\{E\}\_\{a\}\(x^\{\-2\}\)\. The spectral interval is\[λmin​\(B\)\+α,λmax​\(B\)\+α\]\[\\lambda\_\{\\min\}\(B\)\+\\alpha,\\lambda\_\{\\max\}\(B\)\+\\alpha\], whose ratio and Kantorovich ceiling decrease to one\. ∎

Equation \([66](https://arxiv.org/html/2609.11063#as1_S10.E66)\) is the local off\-target decomposition used in the experiments: the first term is total regularised disturbance at matched objective change, the second is the unavoidable cost of that change, and their difference is exactly half the squaredAA\-distance from the natural step\. On a positive\-semidefiniteGG, the same statement holds on its resolved range withG−1G^\{\-1\}replaced byG†G^\{\\dagger\}, providedq∈Range⁡\(G\)q\\in\\operatorname\{Range\}\(G\)andq≠0q\\neq 0\.

At finite damping, the empirical battery measures realised output KL, whereas Eq\. \([67](https://arxiv.org/html/2609.11063#as1_S10.E67)\) compares regularised costs inAA\. The comparison therefore holdsA=G\+α​IA=G\+\\alpha Ifixed and, forΠG​\(u\)=\(u⊤​G​u\)/\(q⊤​u\)2\\Pi\_\{G\}\(u\)=\(u^\{\\top\}Gu\)/\(q^\{\\top\}u\)^\{2\}, predicts the local Euclidean\-to\-damped\-natural\-gradientGG\-cost ratio with

Rpred​\(q\)=ΠG​\(q\)ΠG​\(A−1​q\)\.R\_\{\\rm pred\}\(q\)=\\frac\{\\Pi\_\{G\}\(q\)\}\{\\Pi\_\{G\}\(A^\{\-1\}q\)\}\.The two ratios agree in the undamped limit and in special finite\-damping cases, such as an objective supported on one eigenspace; they generally differ at non\-zero damping\. The held\-out evaluation covers 1,188 predictor cells at three matched target magnitudes, giving 3,564 deployed responses, of which 3,515 are usable \(98\.6% coverage\)\. Across the usable responses, the median ratio of measured to predicted advantage is0\.9670\.967\(95% interval0\.9570\.957–0\.9750\.975\), and the log–log slope of measured on predicted advantage is0\.9820\.982\(95% interval0\.9660\.966–0\.9970\.997\)\. At intervention fractionsΔ​C/C=0\.01,0\.03,0\.10\\Delta C/C=0\.01,0\.03,0\.10, the model\-balanced geometric means of the measured\-to\-predicted ratio are0\.9180\.918,0\.8600\.860and0\.7550\.755, respectively; every one of the 11 within\-model paths decreases at both adjacent steps\. This quantifies the expected attenuation of a local prediction as intervention magnitude grows\. The objective\-resolved metric anisotropy therefore quantitatively predicts intervention cost across the cross\-model battery; a scalar condition number alone does not supply this prediction\.

###### Proposition 7\(chart\-covariant control\)\.

Under an invertible chart changeh′=S​hh^\{\\prime\}=Sh, let

A′=S−⁣⊤​A​S−1,q′=S−⁣⊤​q,δ′=S​δ\.A^\{\\prime\}=S^\{\-\\top\}AS^\{\-1\},\\qquad q^\{\\prime\}=S^\{\-\\top\}q,\\qquad\\delta^\{\\prime\}=S\\delta\.Then objective change, cost and the natural step are invariant:q′⁣⊤​δ′=q⊤​δq^\{\\prime\\top\}\\delta^\{\\prime\}=q^\{\\top\}\\delta,δ′⁣⊤​A′​δ′=δ⊤​A​δ\\delta^\{\\prime\\top\}A^\{\\prime\}\\delta^\{\\prime\}=\\delta^\{\\top\}A\\delta,δN′=S​δN\\delta^\{\\prime\}\_\{\\rm N\}=S\\delta\_\{\\rm N\}\. The cost ratio of any transported comparatorδC′=S​δC\\delta^\{\\prime\}\_\{\\rm C\}=S\\delta\_\{\\rm C\}to the natural step is likewise invariant\. By contrast, the fresh coordinate\-Euclidean comparatorq′/\(q′⁣⊤​q′\)q^\{\\prime\}/\(q^\{\\prime\\top\}q^\{\\prime\}\)is generally not the transport ofq/\(q⊤​q\)q/\(q^\{\\top\}q\), soΓA′​\(q′\)\\Gamma\_\{A^\{\\prime\}\}\(q^\{\\prime\}\)need not equalΓA​\(q\)\\Gamma\_\{A\}\(q\): coordinate\-Euclidean control depends on the chosen chart\. IfA=G\+α​RA=G\+\\alpha R, covariance requiresR′=S−⁣⊤​R​S−1R^\{\\prime\}=S^\{\-\\top\}RS^\{\-1\}\. Resetting the reference to a fresh identity in the new chart instead maps back to the native referenceS⊤​SS^\{\\top\}S\.

###### Proof\.

The first two identities follow by substitution\. SinceA′−1=S​A−1​S⊤A^\{\\prime\-1\}=SA^\{\-1\}S^\{\\top\},A′−1​q′=S​A−1​qA^\{\\prime\-1\}q^\{\\prime\}=SA^\{\-1\}qandq′⁣⊤​A′−1​q′=q⊤​A−1​qq^\{\\prime\\top\}A^\{\\prime\-1\}q^\{\\prime\}=q^\{\\top\}A^\{\-1\}q, proving covariance of Eq\. \([65](https://arxiv.org/html/2609.11063#as1_S10.E65)\)\. The cost ratio is the ratio of two invariant quadratic costs only when both interventions are transported; the coordinate\-Euclidean vectors do not obey that transport law in general\. Finally, a fresh new\-chart identity contributesα​I\\alpha ItoA′A^\{\\prime\}; multiplying the equation byS⊤S^\{\\top\}and writingδ′=S​δ\\delta^\{\\prime\}=S\\deltagives the native termα​S⊤​S​δ\\alpha S^\{\\top\}S\\delta\. ∎

#### 10\.1Relation to direct preference optimisation

A natural policy gradient for a KL\-regularised reinforcement\-learning\-from\-human\-feedback \(RLHF\) objective has the formF−1​∇θ\(reward\)F^\{\-1\}\\nabla\_\{\\theta\}\(\\text\{reward\}\), whereFFis the policy Fisher matrix\. Direct preference optimisation \(DPO\)\[[5](https://arxiv.org/html/2609.11063#as1_bib.bib5)\]parameterises an implicit reward throughβ​log⁡\[π/πref\]\\beta\\log\[\\pi/\\pi\_\{\\mathrm\{ref\}\}\]and optimises model parameters\. This inference\-time construction instead uses the activation\-space pullback metric and the preference covector,δ​h=\(G\+α​I\)−1​∇h\[log⁡p⁡\(yw∣x\)−log⁡p⁡\(yl∣x\)\]\\delta h=\(G\+\\alpha I\)^\{\-1\}\\nabla\_\{h\}\[\\log p\(y\_\{w\}\\mid x\)\-\\log p\(y\_\{l\}\\mid x\)\]\. The resulting activation\-space natural\-gradient update is distinct from DPO’s parameter update\.

#### 10\.2Mass\-preserving Fisher coarse\-graining and interpretable partitions

Grouping the support tokens intoKKcells and replacing each token’s read\-out row by its group mean gives a coarse output FisherHcoarse=Covg​\(w¯g\)H\_\{\\text\{coarse\}\}=\\mathrm\{Cov\}\_\{g\}\(\\bar\{w\}\_\{g\}\), the between\-group covariance of the group\-mean unembeddings, which by the law of total covariance equalsHHminus the within\-group term and is exact whenKKreaches the support size\. For the exponential\-tilt family defined by a scalar token valueϕ\\phi, a partition preserves all Fisher information about the tilt whenϕ\\phiis constant within each cell\. Approximating the complete steering update also requires retaining read\-out covariance\. Bins ofϕ\\phiare therefore refined by probability\-weighted clusters of the read\-out directionswaw\_\{a\}and evaluate the resulting update against the ungrouped solve\. In the measured cost model, the exact matrix\-free solve is less expensive per step than the tested partitions fine enough to reproduce its update \(main\-text Methods\)\.

For sentiment, the resulting interpretable cells correspond approximately to sentiment level crossed with lexical role\. This partition serves interpretation rather than scalability: at the same update fidelity, the tested token groupings require several times more cells than the fixed\-tolerance Krylov solve requires iterations\.

#### 10\.3The full cubic correction and the erosion predictor

Letη⁡\(h\)\\eta\(h\)be the output natural parameters and take a straight activation pathh⁡\(t\)=h0\+t​δ​hh\(t\)=h\_\{0\}\+t\\delta h\. Writea=D​η​\[δ​h\]a=D\\eta\[\\delta h\],b=D2​η​\[δ​h,δ​h\]b=D^\{2\}\\eta\[\\delta h,\\delta h\]andC=diag⁡\(p\)−p​p⊤C=\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\. Direct expansion of output KL gives

KL\(p0∥pt\)=12t2a⊤Ca\+16t3\{T\(a,a,a\)\+3a⊤Cb\}\+O\(t4\),\\mathrm\{KL\}\(p\_\{0\}\\\|p\_\{t\}\)=\\tfrac\{1\}\{2\}t^\{2\}a^\{\\top\}Ca\+\\tfrac\{1\}\{6\}t^\{3\}\\\!\\left\\\{T\(a,a,a\)\+3a^\{\\top\}Cb\\right\\\}\+O\(t^\{4\}\),\(69\)whereT⁡\(a,a,a\)=𝔼p​\[\(a−𝔼p​a\)3\]T\(a,a,a\)=\\mathbb\{E\}\_\{p\}\[\(a\-\\mathbb\{E\}\_\{p\}a\)^\{3\}\]is the Amari–Chentsov contraction\. The first cubic term is the third central moment pulled back through the Jacobian\. The second is the directional Hessian of the network above the intervention; it vanishes at an affine read\-out and is generally non\-zero at an internal layer\. The erosion analysis computes the first component exactly from one Jacobian–vector product and one matrix multiplication; restricted to the moderate\-intervention regime, in which matched targets are reached along monotone segments of the scan, its rank correlation with gentle\-to\-aggressive erosion is\+0\.49\+0\.49to\+0\.75\+0\.75across 410M–2\.8B, and the association reverses sign beyond that regime\. In a 410M full\-curvature analysis \(four prompts, layers 4, 12 and 20 plus affine final\-layer anchors\), adding the network\-Hessian term reduces mean absolute error against a parity\-cancelling directional finite\-difference estimate from 0\.345 to 0\.0176, a ratio of 0\.051; the affine anchors return the map\-Hessian term exactly zero\. The full scale\-ladder erosion analysis therefore reports the Amari–Chentsov and network\-Hessian components separately and jointly\.

###### Scan and regime comparisons\.

The analysis uses a two\-segment scan grid extending the intervention range sixteenfold \(so that unreached matched targets become genuine exclusions after scan\-boundary artefacts are removed\), a fixed sign\-consistent crossing bracket, and average\-tie rank correlations\.

The association reverses on the pooled reach\-extended population\. Supplementary Note[10](https://arxiv.org/html/2609.11063#as1_S10)analyses the moderate\-intervention regime, comprising items whose crossings lie on monotone scan segments, with a partial rank correlation controlling the gentle\-regime advantage separating the predictive signal from its mechanical coupling to the outcome’s denominator\. On the 410M subset reaching targets within the narrower scan range, the correlation is\+0\.74\+0\.74\.

#### 10\.4Relinearised finite intervention paths

Both Fisher and Euclidean paths were relinearised after every accepted step, with an exact prompt\-KL trust region of0\.020\.02nats and common cumulative objective checkpoints of0\.050\.05,0\.10\.1,0\.20\.2and0\.40\.4nats\. Expansion and bisection matched the reached changes to within 1%\. Cumulative local cost is the sum ofKL\(pj−1∥pj\)\\mathrm\{KL\}\(p\_\{j\-1\}\\\|p\_\{j\}\)over accepted steps; it measures the path and is distinct from endpoint or unrelated\-prompt disturbance\. The completed panel contains 216 prompts in 18 model–objective cells \(16 models from six families\)\. All 864 prompt–target pairs reach the common checkpoints\. Cell–target mean log Euclidean/Fisher path\-cost ratios range from2\.44032\.4403to4\.84524\.8452, giving geometric\-mean ratios11\.4811\.48–127\.12127\.12\. All 72 paired 95% prompt\-bootstrap intervals are positive in log space\. The full set of ratios is displayed in main Figureb\.

#### 10\.5Reusable interventions with a reference metric

BLOOM\-560M, Pythia\-410M, Pythia\-1\.4B and Qwen2\-1\.5B were each given four donor prompts per objective, eight unseen target prompts per objective and four reference prompt–continuation pairs\. Truth preference and anti\-sycophancy used continuation log\-odds objectives\. A single activation update per model and objective was shared across prompts, injected at the final prompt position after blockround⁡\(0\.45​L\)\\operatorname\{round\}\(0\.45L\)forLLblocks \(zero\-based index\)\. The mean donor objective gradient was preconditioned by the reference\-prompt Fisher, by the donor\-prompt Fisher, or left Euclidean\. All arms recomputed their gradients and metrics after accepted steps and reached matched donor effects of0\.050\.05,0\.10\.1and0\.20\.2nats within 1\.5%\. Fisher damping was0\.030\.03times the estimated largest eigenvalue\. No target gradient or target\-specific coefficient was used\.

The primary cost was mean sequence KL over the reference continuations, evaluated by teacher forcing and summing over continuation positions\. Reference prompts also define the metric, so the readout evaluates continuation disturbance on that reference set\. Across all 24 model–objective–checkpoint settings, the Euclidean/reference\-Fisher ratio is2\.89862\.8986–6\.28826\.2882; the donor\-Fisher/reference\-Fisher ratio is at least2\.1292\.129\. All six model\-bootstrap intervals for the Euclidean/reference\-Fisher log ratio are positive, with the smallest lower bound1\.14291\.1429\. The smallest reference\-Fisher mean objective change on unseen targets is0\.0127580\.012758nats; all 24 means are positive\. These are matched\-donor comparisons: held\-out target effects are measured rather than matched\.

A two\-objective solve requests donor increments of0\.050\.05nats for truth preference and anti\-sycophancy using the same reference metric\. Both mean target effects are positive in each of the four models and reference\-sequence KL is lower than for the corresponding Euclidean composition\. The reusable object is a within\-model activation update, applied after construction to new prompts\. The experiment does not require aligned hidden coordinates between models\.

### 11Matrix\-free computation

The intervention algorithms avoid materialising the full matricesHH,JJandG=J⊤​H​JG=J^\{\\top\}HJ\. Two implementations of the native\-reference solve are used\(G\+α​I\)−1​q\(G\+\\alpha I\)^\{\-1\}q; both apply the relevant linear operators to vectors, and a general referenceRRis handled by whitening or generalised preconditioning\.

###### Proposition 8\(dimension\-free CG bound at relative damping\)\.

LetG⪰0G\\succeq 0withλmax​\(G\)\>0\\lambda\_\{\\max\}\(G\)\>0, setα=c​λmax​\(G\)\\alpha=c\\lambda\_\{\\max\}\(G\)withc\>0c\>0, and solve\(G\+α​I\)​x=b\(G\+\\alpha I\)x=bby conjugate gradients\. The regularised condition number satisfiesκ≤\(1\+c\)/c\\kappa\\leq\(1\+c\)/c\. Ifx0≠x∗x\_\{0\}\\neq x\_\{\*\}, then in exact arithmetic

∥xt−x∗∥G\+α​I∥x0−x∗∥G\+α​I≤2​\(κ−1κ\+1\)t\.\\frac\{\\lVert x\_\{t\}\-x\_\{\*\}\\rVert\_\{G\+\\alpha I\}\}\{\\lVert x\_\{0\}\-x\_\{\*\}\\rVert\_\{G\+\\alpha I\}\}\\leq 2\\left\(\\frac\{\\sqrt\{\\kappa\}\-1\}\{\\sqrt\{\\kappa\}\+1\}\\right\)^\{t\}\.The residual obeys the same bound in the inverse\-operator norm and, in Euclidean relative norm, the right side multiplied by at mostκ\\sqrt\{\\kappa\}\. Hence fixed relative damping and tolerance give a worst\-case iteration bound independent of width\. Atc=10−2c=10^\{\-2\}, the standard bound gives at most8585iterations for Euclidean relative residual10−610^\{\-6\}\.

###### Proof\.

The eigenvalues ofG\+α​IG\+\\alpha Ilie in\[α,λmax​\(G\)\+α\]\[\\alpha,\\lambda\_\{\\max\}\(G\)\+\\alpha\]\. The displayed inequality is the standard Chebyshev convergence bound for conjugate gradients\. Sincert=\(G\+α​I\)​\(x∗−xt\)r\_\{t\}=\(G\+\\alpha I\)\(x\_\{\*\}\-x\_\{t\}\), its inverse\-operator norm equals the error norm; Euclidean norm equivalence costs at mostκ\\sqrt\{\\kappa\}\. ∎

*Low\-rank Woodbury\.*Restricting the output Fisher to the top\-\|S\|\|S\|tokens by mass and renormalising their probabilities defines a truncated\-support approximationHS=B​B⊤H\_\{S\}=BB^\{\\top\}, factored exactly asB=WU\[S,:\]⊤\(diag\(pS\)−pSpS⊤\)1/2∈ℝd×\|S\|B=W\_\{U\}\[S,:\]^\{\\top\}\\,\(\\mathrm\{diag\}\(p\_\{S\}\)\-p\_\{S\}p\_\{S\}^\{\\top\}\)^\{1/2\}\\in\\mathbb\{R\}^\{d\\times\|S\|\}\. FormingM=J⊤​BM=J^\{\\top\}Bcosts\|S\|\|S\|vector–Jacobian products \(one per column ofBB\), after whichGS=M​M⊤G\_\{S\}=MM^\{\\top\}and the damped solve reduces, by the Woodbury identity, to an\|S\|×\|S\|\|S\|\\times\|S\|system:\(GS\+α​I\)−1​q=α−1​\(q−M​\(α​I\+M⊤​M\)−1​M⊤​q\)\(G\_\{S\}\+\\alpha I\)^\{\-1\}q=\\alpha^\{\-1\}\\\!\\big\(q\-M\(\\alpha I\+M^\{\\top\}M\)^\{\-1\}M^\{\\top\}q\\big\)\.

*Exact conjugate gradients\.*For the full vocabulary,G​vGvis computed in two passes: a Jacobian–vector product \(JVP\) returnsJ​vJv, the exact output\-Fisher productH⁡\(J​v\)=WU⊤​\(diag⁡\(p\)−p​p⊤\)​WU​\(J​v\)H\(Jv\)=W\_\{U\}^\{\\top\}\(\\mathrm\{diag\}\(p\)\-pp^\{\\top\}\)W\_\{U\}\(Jv\)is applied in closed form, and a vector–Jacobian product \(VJP\) returnsJ⊤​\(⋅\)J^\{\\top\}\(\\cdot\)\. Conjugate gradients onG\+α​IG\+\\alpha Ithen solves the native\-reference system\. At fixed tolerance, iteration count measures convergence of the solver at that resolution\. Both routes use explicit automatic\-differentiation primitives \(reverse\-mode VJP viaautograd\.gradand forward\-mode JVP via double backward\)\.

1. 1\.Power\-iterate a few steps to estimateλmax​\(G\)\\lambda\_\{\\max\}\(G\); setα=c​λmax\\alpha=c\\,\\lambda\_\{\\max\},c∼10−2c\\\!\\sim\\\!10^\{\-2\}, fixing damping relative to the spectrum\.
2. 2\.Solve\(G\+α​I\)−1​q\(G\+\\alpha I\)^\{\-1\}qby Woodbury \(low\-rank\) or conjugate gradients \(CG; exact\)\.
3. 3\.Range\-restrict via Lanczos: project onto the eigenspace retained by numerical spectral truncation, so the step carries no null\-space \(output\-inert\) component\.

For the aggregated training FisherF=∑xJx⊤​Hx​JxF=\\sum\_\{x\}J\_\{x\}^\{\\top\}H\_\{x\}J\_\{x\}, the additiveFF\-vector product is accumulated in chunks of examples, holding only one example’s autograd graph in memory at a time\. All solves run in float32:GGis severely ill\-conditioned \(κ∼105\\kappa\\sim 10^\{5\}–10710^\{7\}\), and in the evaluated Pythia\-410M and Pythia\-1\.4B cases the bfloat16 and float32 recovered directions were only weakly to moderately aligned \(absolute cosines0\.0560\.056–0\.5420\.542across two contexts per model\)\.

###### Dense spectral computation\.

At the evaluated widths \(d≤1024d\\leq 1024\), forming the truncated operators directly and diagonalising them is faster than the matrix\-free route for spectra\. The matrix\-free implementation is retained for its asymptotic storage scaling: additional operator storage grows linearly with activation dimension and omits a vocabulary\-by\-activation Jacobian, while dense\-operator storage grows quadratically\. Output evaluation itself still scales with vocabulary size\.

## Part V: Evidence, translation and reproducibility

### 12Experimental protocols

#### 12\.1Experimental design and statistical controls

The empirical analyses separate model\-selection or exploratory data from held\-out evaluation data where applicable\. For held\-out batteries, the population, split, unit of analysis, endpoints and numerical tolerances are fixed before evaluation\. For thresholded endpoints, response\-independent resolution was measured against the instrument floor before evaluation\. Reported summaries are reconstructed from the underlying rows\.

The estimand determines the unit of replication\. Context batteries are resampled by context; fact\-acquisition studies cross training seed with fact or reciprocal\-fact block; intervention frontiers compare methods on the same prompt and objective sets; dictionary analyses use three paired seeds\. Intervals use the analysis\-specific resampling units described in Methods and figure captions; paired comparisons share each resampling draw across their arms\. For cross\-model agreement, factual\-relation alignments, corpus mediation, residual eigenvector overlap and band profiles, 95% percentile intervals use 5,000 full\-size context draws with replacement, paired across fixed model rosters and contrasts\. Distance\-matrix analyses average tied ranks and exclude comparisons between repeated copies of the same original context; convergence averages the 29 displayed model pairs\. Separately labelled 80%\-subsample ranges describe stability under context removal\. Nulls retain the nuisance structure relevant to the claim: permutation within objective–layer strata for the intervention\-cost\-ratio prediction, displacement\-magnitude\-matched rotation for shared eigenspaces, and fixed\-half label randomisation for the acquisition controls\.

Three batteries deliberately test distinct boundaries of the theory\. Identification uses both local curvature and global profiled\-risk certificates\. Shared geometry is evaluated on natural, templated and factual\-relation contexts, with a common\-outcome subset for the quantitative risk bounds\. Acquisition combines a real\-checkpoint forecast applied without refitting, randomised evidence\-depth assignment and cross\-size alignment by held\-out loss\. The components are kept separate in the raw analyses and combined only in the synthesis of Supplementary Note[7](https://arxiv.org/html/2609.11063#as1_S7)\.

#### 12\.2Matched\-effect comparisons

Comparisons reported at matched response are made at the same realised change in the scalar objectiveϕ\\phi\. For each method and prompt, binary search over the step scale is used until the realisedΔ​ϕ\\Delta\\phireaches a target\. When the objective changes in discrete increments, the two steps that bracket the target are interpolated and all other quantities evaluated at that matchedΔ​ϕ\\Delta\\phi\[[6](https://arxiv.org/html/2609.11063#as1_bib.bib6),[7](https://arxiv.org/html/2609.11063#as1_bib.bib7)\]\. The instruction\-model frontiers instead evaluate each prompt\-specific direction on a method\-specific strength grid shared across the 12 prompts, average behaviour and off\-target KL at each grid point, and compare interpolated frontiers at equal aggregate mean behaviour change\. The amortised contrastive\-activation\-addition baseline uses the mean covector direction over the contrast set and the same frontier convention\. The six\-size Pythia factual sweep reports its three requested shifts as nominal targets\.

Off\-target change is reported under three conventions\. The first is a concept\-decomposed Kullback–Leibler \(KL\) divergence that separates the intended change on the objective tokens from changes elsewhere\. The second,off\_target\_kl\_set, evaluates the KL divergence over the complement of the objective’s support\. The third, used for scalar objectives, is the second\-order Fisher residualKL\(p0∥ps\)−Δϕ2/\(2Varp0ϕ\)\\mathrm\{KL\}\(p\_\{0\}\\\|p\_\{s\}\)\-\\Delta\\phi^\{2\}/\(2\\,\\mathrm\{Var\}\_\{p\_\{0\}\}\\phi\), which subtracts the local quadratic disturbance predicted by the intended change alone\. For finite KL divergence, this residual approximates rather than exactly decomposes off\-target change\. The win rate is the fraction of prompts for which the natural\-gradient step has lower off\-target change at the comparison setting\.

For matched\-response analyses, advantages are ratios of off\-target change at matched objective change\. In the nominal\-target Pythia sweep, ratios and win rates use the two arms returned at each requested target setting\. Confidence intervals are paired bootstraps over a fixed set of 12 shared contrastive pairs, resampled in log\-ratio space and reported as 95% intervals\. Because the same 12 pairs are used at every model scale, depth and width comparisons are paired and tested in the same space\. The full prompt sets, contrastive pairs, random seeds and exact model revisions for each analysis are available from the corresponding author on reasonable request\.

### 13Knowledge\-editing details

Each fact is edited by a contrastive natural\-gradient step gated by a subject\-string rule and evaluated on CounterFact with GPT\-2\-XL\. The rule activates when a prompt contains the edit subject after light text normalisation and otherwise leaves the model unchanged\. On the 100\-edit sample, it activates on all edit and paraphrase prompts \(100% paraphrase activation rate\) and on 0\.2% of neighbourhood prompts\. Under the standard EasyEdit metrics\[[8](https://arxiv.org/html/2609.11063#as1_bib.bib8)\]—efficacy score \(ES\), paraphrase score \(PS\), neighbourhood score \(NS\) and their harmonic meanSS—the routed Fisher system obtainsS=100\.0S=100\.0\(ES/PS/NS=100/100/100=100/100/100\)\. For context, the same benchmark implementation givesS=85\.8S=85\.8for ROME\[[9](https://arxiv.org/html/2609.11063#as1_bib.bib9)\]\(98/77/8598/77/85\),S=70\.8S=70\.8for MEMIT\[[10](https://arxiv.org/html/2609.11063#as1_bib.bib10)\]\(86/50/9386/50/93\),S=43\.8S=43\.8for GRACE\[[11](https://arxiv.org/html/2609.11063#as1_bib.bib11)\]\(92/22/8292/22/82\) andS=30\.2S=30\.2for SERAC\[[12](https://arxiv.org/html/2609.11063#as1_bib.bib12)\]\(20/27/8120/27/81\)\. Because routing and method\-access conditions differ, these values provide benchmark context rather than a controlled method ranking: the Fisher system computes a query\-time edit gated by a subject\-string rule, whereas the comparators are persisted editors\.

A learned scope\-classifier variant replaces the subject\-string test with a trained classifier\. It obtainsS=89\.0S=89\.0and covers99\.3%99\.3\\%of subject aliases, whereas the subject\-string rule obtainsS=92\.5S=92\.5at32%32\\%alias coverage\. The subject\-string rule is the default for the reported locality set\.

A stricter probability\-margin protocol on 300 edits, in which an edit counts only if the new fact’s probability exceeds the old fact’s, givesS=92\.8S=92\.8\(ES100100, PS99\.599\.5, NS81\.581\.5\)\. The NS is indistinguishable at the displayed precision from the unedited\-model baseline of81\.681\.6\. The apply\-operator ablation replaces the pullback\-metric step with Euclidean and unpreconditioned counterparts at matched edit efficacy \(P=0\.5P=0\.5\) and measures off\-target KL divergence on unrelated prompts: Fisher0\.8720\.872\(1\.0×1\.0\\times\), Euclidean3\.7003\.700\(4\.2×4\.2\\times\) and unpreconditioned1\.2171\.217\(1\.4×1\.4\\times\)\.

The fixed routed\-edit comparison removes the query\-time\-solve asymmetry\. On GPT\-2, each Fisher and Euclidean delta is solved once on a canonical CounterFact prompt, fixed and reused, and both arms use the same gating rule\. Both operators attain the fixed\+5\+5log\-odds calibration shift on all 30 held\-out edits\. The paired geometric\-mean ratio of Euclidean to Fisher other\-token KL divergence is10\.5710\.57\(95% bootstrap interval8\.528\.52–13\.3713\.37\)\. This fixed routed\-edit result directly compares Fisher and Euclidean application\. Ranking against ROME, MEMIT or other persisted editors requires a common fixed\-edit protocol and gating rule\. Supplementary Table[3](https://arxiv.org/html/2609.11063#as1_S13.T3)assembles the matched fixed\-edit comparison and the separate query\-time results\.

Supplementary Table 3:Knowledge editing in full\(CounterFact\)\. Top: the matched fixed\-edit comparison on GPT\-2, with deltas fixed after one solve and a shared gating rule\. The remaining sections report the separate GPT\-2\-XL query\-time\-routed benchmark: standard EasyEdit metrics \(100 edits\), the stricter probability\-margin protocol \(300 edits\) and the earlier apply\-operator screen\. The cross\-method EasyEdit values use common benchmark scores under different routing and data\-access conditions\.*Matched fixed routed edits \(GPT\-2; 30 held\-out edits; shared gating rule\)*Comparison\+5\+5shift attainedother\-token KL advantagedelta useFisher versus Euclidean30/30 each10\.57×10\.57\\times\(8\.528\.52–13\.3713\.37\)solved once, reused*EasyEdit standard metrics \(GPT\-2\-XL; 100 edits; descriptive\)*MethodEfficacyParaphraseLocalitySSRouted Fisher \(ours, training\-free\)100\.0100\.0100\.0100\.0ROME98\.077\.085\.085\.8MEMIT86\.050\.093\.070\.8GRACE92\.022\.082\.043\.8SERAC20\.027\.081\.030\.2*Probability\-margin protocol \(300 edits, mass\-edit\)*EfficacyParaphraseNeighbourhood \(base81\.681\.6\)SSRouted Fisher100\.099\.581\.592\.8*Apply\-operator ablation at matched efficacyP⁡\(target\)=0\.5P\(\\mathrm\{target\}\)\{=\}0\.5\(100 edits\)*off\-target KL \(median\)fluencytarget reachedFisher apply0\.8720\.872\(1\.0×\)\(1\.0\\times\)3\.84100%Euclidean apply3\.7003\.700\(4\.2×\)\(4\.2\\times\)3\.7494%Unpreconditioned apply1\.2171\.217\(1\.4×\)\(1\.4\\times\)3\.80100%*Subject\-string gating*: paraphrase 100%; neighbourhood 0\.2%\.
### 14The instruct\-model programme

Three alignment objectives are steered by the same untuned preference covectorq=∇h\[log⁡p⁡\(yw\)−log⁡p⁡\(yl\)\]q=\\nabla\_\{h\}\[\\log p\(y\_\{w\}\)\-\\log p\(y\_\{l\}\)\]\(main\-text Methods\): sycophancy \(honest\-over\-flattering continuations\), truthfulness \(capital\-city true/false pairs\) and positive writing style\. Capability retention is measured as off\-target KL divergence on held\-out neutral prompts that are semantically distinct from the steered attribute\. The 20\-item multiple\-choice check changes in increments of1/20=0\.051/20=0\.05and showed no Fisher–Euclidean difference\. This discrete endpoint did not resolve the distributional difference captured by off\-target KL, which differed by a factor of3636–5050on Qwen2\. For localised objectives, the analyses measure on\-prompt off\-target KL divergence\. For the diffuse style objective, intended changes also affect this quantity, so retention is measured on the held\-out neutral prompts\.

At equal interpolated aggregate mean behaviour change, the natural\-gradient update reduces off\-target KL divergence relative to the Euclidean update by factors of88–7474for sycophancy,3030–190190for truthfulness and44–550550for style\. Contrastive activation addition\[[13](https://arxiv.org/html/2609.11063#as1_bib.bib13)\]often does not reach the target change for these objectives\. Objectives combine linearly as\(G\+α​I\)−1​\(w1​q1\+w2​q2\)\(G\+\\alpha I\)^\{\-1\}\(w\_\{1\}q\_\{1\}\+w\_\{2\}q\_\{2\}\), with one weight per objective; the weight required for a given balance is model\-dependent\. The results replicate on Qwen1\.5\-1\.8B\-Chat\.

Supplementary Figure 2:Instruction\-model control frontiers\.Left, anti\-sycophancy; right, capital\-city truth preferences\. Qwen2\-1\.5B\-Instruct and Qwen1\.5\-1\.8B\-Chat each contribute the complete 19\-point sweep: seven natural\-gradient, six Euclidean\-gradient and six contrastive\-activation\-addition settings\. Cost is off\-target KL on held\-out neutral prompts; the horizontal coordinate is mean objective change\. Interpolated comparisons match aggregate effects\. The style and composition frontiers are shown in Extended Data Figure\.
### 15Sparse\-autoencoder feature ablations

The feature experiment uses one Top\-kksparse autoencoder at layer 12 of Pythia\-410M\. Feature selection uses 18 discovery contexts; evaluation uses 18 disjoint context clusters\. Of 71 selected features, 57 are active in evaluation, giving 303 unique context–feature ablations\. Each intervention sets the active code to zero before decoding\. The measured cost is exact full\-vocabularyKL\(p0∥pzero\)\\mathrm\{KL\}\(p\_\{0\}\\\|p\_\{\\rm zero\}\); the Fisher prediction12​Δ​h⊤​G​Δ​h\\tfrac\{1\}\{2\}\\Delta h^\{\\top\}G\\Delta his computed before observing the causal outcome\.

Across the 303 interventions, the Fisher prediction has Spearman correlation0\.9970\.997with measured KL, compared with0\.8960\.896for activation magnitude\. Their paired difference is0\.1010\.101\(95% cluster\-bootstrap interval0\.0690\.069–0\.1360\.136\)\. For outcomes independent of the distributional cost approximation, the objective\-effect\-to\-Fisher\-cost ratio correlates with the held\-out selected\-token probability\-change fraction at0\.9350\.935versus0\.5060\.506for attribution\-patching magnitude, and with sign agreement at0\.7460\.746versus0\.3540\.354\. The corresponding paired\-difference intervals are0\.2960\.296–0\.5530\.553and0\.2890\.289–0\.4920\.492\. The objective\-effect\-to\-Fisher\-cost ratio is the stronger correlate of the selected\-token probability\-change fraction and sign agreement, whereas attribution\-patching magnitude is the stronger correlate of absolute signed effect \(0\.9870\.987versus0\.5250\.525\)\.

### 16Dictionary\-learning details

Top\-kksparse autoencoders\[[14](https://arxiv.org/html/2609.11063#as1_bib.bib14)\]are trained with only the decoder update preconditioned by the natural gradient in the code geometry\[[15](https://arxiv.org/html/2609.11063#as1_bib.bib15)\],Δ​D∝\(𝔼⁡\[a​a⊤\]\+ε​I\)−1​∇Dℒ\\Delta D\\propto\(\\mathbb\{E\}\[aa^\{\\top\}\]\+\\varepsilon I\)^\{\-1\}\\nabla\_\{D\}\\mathcal\{L\}, whereaadenotes the sparse code\. The code Gram matrix𝔼⁡\[a​a⊤\]\\mathbb\{E\}\[aa^\{\\top\}\]is not formed explicitly; the preconditioner is applied to each batch by a Woodbury solve on the low\-rank batch code matrix and agrees with the explicit inverse to7×10−137\\times 10^\{\-13\}\. Residual\-stream activations have heavy\-tailed norms, with a maximum approximately 15 times the median in the training data\. In the reported configuration, per\-token norms are winsorised at approximately the 95th percentile before training\.

Comparisons use three paired seeds\. Mean feature activation frequency is approximatelyk/Lk/Lby construction for a Top\-kkautoencoder with dictionary sizeLLand is reported as a diagnostic\. Because recovery alone need not detect poorly used features, reconstruction variance explained and the fraction of features activated at least once are both reported alongside it\. Disentanglement is scored by Hungarian assignment of learned decoder atoms to planted fine\-grained features using absolute cosine similarity, followed by the mean matched cosine\. In the evaluated 410M configuration withβ=0\.5\\beta=0\.5, natural\-gradient preconditioning increases mean matched cosine from0\.1670\.167to0\.4800\.480\(\+0\.313\+0\.313\), with an increase in all three seeds\. Mean reconstruction variance explained increases from0\.5350\.535to0\.6220\.622, and the fraction of features activated at least once from21\.8%21\.8\\%to62\.3%62\.3\\%\. Scale comparisons hold the per\-feature training budget fixed\.

### 17Cross\-model convergence: protocols and controls

Three context batteries are used\. The natural battery is 200 sentence prefixes drawn from the held\-out WikiText test split\[[16](https://arxiv.org/html/2609.11063#as1_bib.bib16)\], cut mid\-sentence at 8–20 words; the templated battery is 160 contexts from eight topic templates; the semantic battery is 120 prompts across eight factual relations whose next token is the answer\. The natural battery supplies the primary estimates\.

Agreement between two models is the Spearman correlation between the upper triangles of their context\-by\-context Fisher–Rao distance matrices\. On the templated battery, output\-distribution geometry gives higher agreement than mid\-layer activations under three measures used in the representation\-similarity literature: mutualkk\-nearest\-neighbour alignment \(0\.778 versus 0\.710\), linear centred\-kernel alignment \(0\.975 versus 0\.935\) and distance\-matrix agreement \(0\.935 versus 0\.755\)\. In the reparameterisation experiment, each model’s mid\-layer states are transformed by an independent random invertible map with condition numberκ\\kappaswept from 1 to10410^\{4\}; output\-distribution agreement is constant to machine precision at everyκ\\kappawhile centred\-kernel alignment and mutualkk\-nearest\-neighbour alignment degrade by 10% and 15% byκ=104\\kappa=10^\{4\}and cosine distance\-matrix agreement by 28% byκ=103\\kappa=10^\{3\}\(its apparent recovery atκ=104\\kappa=10^\{4\}coincides with near\-singular transformed activations\)\.

The corpus\-specificity control illustrates why the natural battery is primary: the same\-corpus mediation advantage is\+0\.27\+0\.27\(95% interval\+0\.14\+0\.14to\+0\.38\+0\.38\) on the templated battery but falls to\+0\.03\+0\.03\(−0\.04\-0\.04to\+0\.10\+0\.10\) on natural text at matched tokenizer and scale; the interval spans zero\.

Reference mediation conditions each target\-model pair’s agreement on the geometry of one reference model\. The mediated fraction rises from 7% with a 70\-million\-parameter reference to a peak of 54% with a 2\.8\-billion\-parameter reference and then saturates, with 7\-billion\-parameter references from other families mediating 41–42%\. The qualitative result is stable across the tested output metrics: cross\-model agreement is 0\.952 under Fisher–Rao, 0\.947 under Jensen–Shannon, 0\.935 under total variation and 0\.892 under a Euclidean metric on probabilities\. The selection of Fisher–Rao specifically follows from the reparameterisation result and Chentsov’s theorem\.

As a separate activation\-space grounding control, over 100 concrete concepts the mean cosine\-distance geometry of last\-token, last\-hidden\-state representations from Pythia\-1\.4B and GPT\-2 Large agrees with the representational geometry of DINOv2\[[17](https://arxiv.org/html/2609.11063#as1_bib.bib17)\], evaluated on CIFAR\-100\[[18](https://arxiv.org/html/2609.11063#as1_bib.bib18)\], at 0\.385 \(permutationP<0\.001P<0\.001\), 84% of which survives partialling out textual co\-occurrence; the result is stable across two vision models and two language models\.

A recent critique shows that representational\-similarity metrics inflate with model scale, because spectral measures such as centred\-kernel alignment carry a non\-vanishing null baseline that grows with representation dimension, and recommends recalibrating every score against a permutation null\[[19](https://arxiv.org/html/2609.11063#as1_bib.bib19)\]\. That calibration was applied to the reported metrics \(200200row\-permutations per model pair; ten models from7070million to77billion parameters across six families, thirty\-five cross\-family pairs on the natural battery\)\. The rank agreement of distance matrices used here has a permutation null of0\.0010\.001, independent of dimension, whereas centred\-kernel alignment on the same mid\-layer representations carries a null baseline of0\.3160\.316, reproducing the confound: across the ladder its raw score more than doubles with scale \(0\.350\.35to0\.730\.73\), the inflation identified by the critique, while the output\-distribution rank agreement stays high and flat \(calibrated0\.850\.85to0\.890\.89\)\.

After calibration, the Fisher–Rao output\-distribution agreement is essentially unchanged \(0\.8790\.879to a calibrated0\.8690\.869, retaining99%99\\%\) and remains the strongest agreement among the representations compared, whereas mid\-layer centred\-kernel alignment is reduced most strongly \(retaining74%74\\%\) and has the weakest significance\. Thus, the dimension\-dependent null identified by the critique affects centred\-kernel alignment; the rank correlation of distance matrices used for the primary output\-distribution analysis remains stable\.

### 18The displacement\-magnitude\-matched null and the anisotropy analysis

Claims that two models share eigenvector structure require a null that preserves displacement magnitude\. The decision axes of the output Fisher metric, expressed as probability displacements over a common vocabulary, assign each word a displacement magnitude and a direction within the spectrum of the metric\. A shuffled\-word null destroys both and makes sharing look ubiquitous: the top\-six cross\-family subspace overlap is 0\.698 against a shuffled null of 0\.025\. A displacement\-magnitude\-matched null preserves each word’s displacement magnitude and randomises only the directions; against it the same overlap has a null of 0\.636, so the residual eigenvector overlap is\+0\.063\+0\.063\(95% interval\+0\.056\+0\.056to\+0\.070\+0\.070\), about 9% of the raw overlap\. The remaining 91% is the shared displacement\-magnitude structure, which follows from the shared output distributions alone\. The residual component is concentrated in the top modes \(templated battery, where per\-band residuals are resolvable\), decaying from\+0\.080\+0\.080in the first five to\+0\.013\+0\.013past mode ninety, and differs by training lineage:\+0\.088\+0\.088within a model family against\+0\.064\+0\.064across families, with a shared tokenizer alone conferring no advantage \(\+0\.061\+0\.061\)\.

Separately, raw cosine similarities of embedding rows are dominated by a shared mean direction and can invert cross\-model comparisons outright: one model’s raw unembedding geometry anti\-correlated with every other model’s before correction, and the cross\-family agreement rose from 0\.498 to 0\.885 after subtracting the vocabulary mean\. Covariance\-based quantities are immune: the categorical factor of the output Fisher metric annihilates the all\-ones direction, so adding any constant vector to all unembedding rows leaves the metric, its eigenvectors and the displacement analysis exactly unchanged\. Accordingly, the shared\-structure residual computed from raw and mean\-centred unembeddings is identical \(\+0\.063\+0\.063in both cases\), whereas cosine analyses require mean\-centring for this comparison\.

An objective\-level robustness analysis probes this link beyond the band\-level co\-localisation of the main text\. Across four small models \(GPT\-2, GPT\-Neo\-125M, Pythia\-70M and Pythia\-410M; 180 model\-pair×\\timesobjective items\), partial Spearman associations conditioned on generalised rank fraction and log generalised eigenvalue are tested against Freedman–Lane residual permutation nulls within model\-pair×\\timesobjective blocks \(5,000 permutations, Holm multiplicity adjustment\)\. The Fisher\-cost–sharing and objective\-sensitivity–sharing associations survive \(both adjustedp=0\.001p=0\.001, and the direction is stable under leave\-one\-model\-out and leave\-one\-objective\-out re\-analysis\), while the direct item\-level cost–meaning association at matched training effect does not \(observed−0\.016\-0\.016, adjustedp=1\.0p=1\.0\) and sharing–meaning is marginal \(adjustedp=0\.074p=0\.074\)\. The association with semantic relevance is therefore supported at the band level of the main text, where answer\-relevant motion concentrates in the same top modes\. Because model pairs share endpoints, the pooled analysis uses a per\-block permutation scheme that preserves this dependence\.

### 19Meaning decomposition and transfer probes

The consensus geometry is the mean of the models’ rank\-transformed distance matrices, and a model’s residual is its matrix with the consensus partialled out\. Against the external semantic geometry defined by theall\-MiniLM\-L6\-v2Sentence\-BERT encoder\[[3](https://arxiv.org/html/2609.11063#as1_bib.bib3)\], the residual has negligible alignment on each battery \(−0\.001\-0\.001on natural,\+0\.003\+0\.003on templated,\+0\.051\+0\.051on the semantic battery\), while the consensus carries 0\.044, 0\.102 and 0\.440 respectively\. The alignment increases when the next token directly encodes the answer to a factual relation\. On the semantic battery the prompts within a relation share a template, so the surface geometry is itself strongly aligned \(0\.614\); the consensus retains\+0\.231\+0\.231\(95% interval\+0\.159\+0\.159to\+0\.316\+0\.316\) after partialling the surface out, and the surface retains\+0\.518\+0\.518after partialling the meaning out, so template and meaning are separate strong drivers there\.

Transfer probes use relative representations: each prompt is its vector of Fisher–Rao distances to a fixed 40\-prompt anchor set,zz\-scored per model, and a multinomial logistic probe for the eight\-way topic label is trained on one model and tested on held\-out prompts of another, over all ordered model pairs and five folds\. Accuracy is 0\.724 within model, 0\.659 across models and 0\.700 for the consensus alone, against a chance level of 0\.125; the residual supports 0\.407 within its own model and 0\.091, below chance, across models\.

#### 19\.1Native\-sample human semantic geometry

The native\-sample panel uses 512 sentence\-final contexts from the Peelle et al\. completion norms\[[20](https://arxiv.org/html/2609.11063#as1_bib.bib20)\]\. At each of six Pythia sizes, 32 samples per context were drawn at temperature one from the full next\-token law, without top\-kkor nucleus filtering\. Sampling had an eight\-token limit; completions were parsed at the first punctuation, newline or end\-of\-sequence boundary\. All samples, including empty completions and those reaching the limit, entered the analysis\. The limit was reached by27\.227\.2–36\.3%36\.3\\%of samples across models\.

Each normalised completion was encoded by the fixed 384\-dimensionalall\-MiniLM\-L6\-v2encoder, then mapped to 768 random Fourier features for an equal mixture of Gaussian kernels with bandwidths0\.250\.25,0\.50\.5and1\.01\.0\. Sixteen disjoint folds contained 32 contexts each\. Completion feature means define the semantic distributional representation; its squared Euclidean distance and centred Gram\-matrix CKA compare model and human predictions\. The four pairings of independent human\-participant halves and model\-sample halves were averaged\. CKA was divided by the square root of human and model split\-half CKA, using the same fold\. Thus these endpoints describe semantic kernel geometry, whereas the native next\-token Fisher–Rao distances in the cross\-model panel are computed directly from probabilities\.

For 70M, 160M, 410M, 1B, 1\.4B and 2\.8B, mean squared model–human semantic distances are0\.338850\.33885,0\.326340\.32634,0\.308410\.30841,0\.303160\.30316,0\.300460\.30046and0\.295760\.29576\. The slope against natural\-log parameter count is−0\.011809\-0\.011809\(95% paired fold\-bootstrap interval−0\.013017\-0\.013017to−0\.010627\-0\.010627\)\. Reliability\-corrected semantic CKA values are0\.821010\.82101,0\.826800\.82680,0\.841620\.84162,0\.844500\.84450,0\.845000\.84500and0\.840050\.84005, with slope0\.0062900\.006290\(0\.0000700\.000070–0\.0128800\.012880\); the overall trend need not be monotone between adjacent sizes\. A character 3–5\-gram feature control also gives decreasing model–human distance, with slope−0\.012915\-0\.012915\(−0\.014727\-0\.014727to−0\.011243\-0\.011243\), while its CKA slope interval includes zero\.

#### 19\.2Predictive fit and developmental human alignment

A separate 384\-context subset of the same corpus was divided into 16 folds\. Model probabilities were evaluated on complete human\-observed completion\-plus\-end\-of\-sequence events and normalised within this support; token\-sequence collisions were retained\. Risk is the context mean of∑y\(py−hy\)2\\sum\_\{y\}\(\\sqrt\{p\_\{y\}\}\-\\sqrt\{h\_\{y\}\}\)^\{2\}for model and human conditional lawsppandhh\. CKA is averaged over the two human halves and divided by the square root of human split\-half reliability; the model feature means are deterministic conditional expectations\. For each fold, a linear relation between conditional root\-probability risk and corrected semantic CKA was fitted on six Pythia sizes and five Pythia\-160M checkpoints\. Its risk slope was−0\.162883\-0\.162883\(95% fold\-bootstrap interval−0\.202616\-0\.202616to−0\.125890\-0\.125890\)\. Applied without refitting to five OLMo\-2\-1B checkpoints\[[21](https://arxiv.org/html/2609.11063#as1_bib.bib21)\]and BLOOM\-560M, GPT\-2 Large, GPT\-Neo\-1\.3B, Mamba\-130M and Qwen2\-1\.5B, the relation gives correlation0\.7583200\.758320and root\-mean\-square error0\.0294650\.029465over all 160 model–fold observations\. These rows share contexts and checkpoints and do not constitute 160 independently trained models\.

OLMo alignment increases across checkpoint order, with slope0\.0188910\.018891\(0\.0134380\.013438–0\.0250560\.025056\)\. The Pythia trajectory slope is0\.0026630\.002663\(−0\.002270\-0\.002270–0\.0081020\.008102\)\. The residual log\-parameter coefficient after conditional risk is0\.0028720\.002872\(−0\.005020\-0\.005020–0\.0098760\.009876\)\. Together with the cross\-family prediction, these comparisons support predictive fit as an organising variable for human alignment\. The conditional\-law analysis and the native\-sampling analysis evaluate different completion distributions\.

#### 19\.3Leading directions in an independent human probability geometry

The de Varda et al\. corpus\[[22](https://arxiv.org/html/2609.11063#as1_bib.bib22)\]supplies 1,726 context positions in 205 sentences\. Human counts used the released exclusion, first\-word and spelling\-correction rules\. Both model and human laws were mapped to 64 semantic outcome bins fixed independently of this corpus, plus a residual outcome retaining the remaining mass\. Human laws used a Dirichlet posterior mean with total concentration0\.50\.5and a fixed external reference lawρ\\rho\. Coordinates were the centred log probabilities weighted byρ\\sqrt\{\\rho\}, as in the standardised read\-out representation\. Four rotations of whole\-sentence folds separated estimation of the human reference, estimation of model directions, calibration and final evaluation\. The leading four model directions were learned without human target responses\.

Directional recovery was evaluated from the component of human coordinate variation orthogonal to the model\-derived frame\. WritingRorientationR\_\{\\mathrm\{orientation\}\}for this component andmzm\_\{z\}for the smallest eigenvalue of human in\-frame variation, the directional lower bound is1−Rorientation/\(4​mz\)1\-R\_\{\\mathrm\{orientation\}\}/\(4m\_\{z\}\)\. Contributions were summed over rotations before taking the ratio\. Simultaneous one\-sided 95% lower bounds over the seven families used 2,048 whole\-sentence bootstrap draws\. For Pythia, Mamba, RWKV, GPT\-2, GPT\-Neo, Qwen2 and BLOOM, they are0\.340550\.34055,0\.379350\.37935,0\.374570\.37457,0\.383060\.38306,0\.319370\.31937,0\.411160\.41116and−0\.00120\-0\.00120\. Direct directional capture exceeds a Haar\-random four\-dimensional subspace by0\.500\.50–0\.610\.61across the seven families\. The direction result also retains its sign when human participants used for reference estimation and evaluation are disjoint\.

These measurements concern directions\. Recovering the full human probability law additionally requires the appropriate origin and coordinates within the frame\. Full\-reconstruction bounds remain non\-positive in this corpus under the evaluated native and low\-rank calibrated maps; the directional result therefore establishes shared leading structure rather than full\-law recovery\.

#### 19\.4Independent human\-completion replication and model\-only calibration

The independent analysis used 640 word positions selected without response values from five DERCo narratives\[[23](https://arxiv.org/html/2609.11063#as1_bib.bib23)\]\. The selected positions contain 64,000 responses from 499 participant identities\. For Pythia 70M, 160M, 410M, 1B, 1\.4B and 2\.8B, cross\-fitted native model–human semantic distances are0\.233480\.23348,0\.224500\.22450,0\.216180\.21618,0\.211510\.21151,0\.200840\.20084and0\.203350\.20335\. The equal\-story slope against natural\-log parameter count is−0\.008725\-0\.008725\(95% interval−0\.012022\-0\.012022to−0\.005556\-0\.005556; Extended Data Fig\.b\)\. The contexts and completion units differ from the Peelle analysis above, so absolute distances are not pooled across corpora; the prospectively fixed comparison is the within\-corpus slope\.

Across the five ordered training stages, Pythia\-160M distances are0\.343800\.34380,0\.265620\.26562,0\.226470\.22647,0\.226040\.22604and0\.223000\.22300, while OLMo\-2\-1B distances are0\.304030\.30403,0\.214410\.21441,0\.200430\.20043,0\.199280\.19928and0\.194410\.19441\. The final\-minus\-initial changes are−0\.12080\-0\.12080\(−0\.13529\-0\.13529to−0\.10470\-0\.10470\) and−0\.10962\-0\.10962\(−0\.12571\-0\.12571to−0\.09223\-0\.09223\), respectively \(Extended Data Fig\.c\)\. In the held\-family conditional analysis, the risk\-based predictor has root\-mean\-square error0\.038250\.03825, compared with0\.061870\.06187for a constant and0\.049910\.04991for log model size\. Its paired advantage over the constant is0\.023630\.02363\(0\.002640\.00264–0\.041430\.04143\); the advantage over size is0\.011660\.01166and is not resolved \(−0\.00203\-0\.00203–0\.036380\.03638\)\. Thus the independent data establish improvement over a constant predictor but do not distinguish risk from size for this endpoint\.

The model\-only calibration retained all 64 fixed semantic bins plus the residual outcome and was learned entirely from model probability laws on the earlier corpus\. Its equal\-story expected human log\-score gains over each native law are0\.045200\.04520nats for Pythia,0\.040780\.04078for Mamba,0\.038650\.03865for RWKV,0\.022020\.02202for GPT\-2,0\.019620\.01962for GPT\-Neo,−0\.00297\-0\.00297for Qwen2 and0\.050740\.05074for BLOOM \(Extended Data Fig\.d\)\. Five of the six positive family means are positive in every story and GPT\-Neo is positive in four of five; Qwen2 is positive in three\. By contrast, maps trained on human responses from the earlier corpus reduce log score in all seven families \(−0\.18212\-0\.18212to−0\.10584\-0\.10584nats\)\. The transferable gain is therefore specific to the model/source\-law calibration rather than generic cross\-corpus fitting\. These are effects over five fixed observed stories, not a population\-of\-stories threshold claim, and the coarse outcomes do not establish individual\-word recovery\.

### 20Reproducibility map

Supplementary Table 4:Evidence map for the central claim chain\.The table organises each strand by its evidence and role in the manuscript\.StrandEvidence and role in the manuscriptOrigin and identificationCoordinate\-covariance tests, across\-scale curvature and profile calibration, initialisation controls and standardised\-frame certificates establish the canonical metric and its identified content\.Shared geometryCross\-model agreement and meaning batteries, exact attenuation accounting, common\-outcome and common\-byte comparisons, held\-out prediction, independent human completion comparisons and fresh model\-only calibration establish shared predictive structure\.Spectral anatomyCore\-to\-tail response measurements, effective\-dimension agreement without fitted parameters, inheritance bounds, singleton\-head/cluster\-bulk anatomy and fresh\-context tests and the probability/read\-out comparison establish how language statistics shape the spectrum\.Acquisition scheduleCross\-size alignment by held\-out loss, corpusnn\-gram forecasts applied without refitting, randomised logarithmic\-time depth shifts, crossed architecture–construction generalisation and real\-checkpoint observational composition connect corpus evidence to acquisition time\.Motion and learningContinuous motion checks, the exact flip\-margin identity, objective–update factorisation and read\-out projector kinematics connect geometric motion to behavioural change\.Control geometryChart\-covariant damping and the prediction of natural\-versus\-Euclidean intervention cost without fitted parameters establish the minimum\-disturbance law and its relative cost; relinearised paths and reusable reference\-metric updates extend its measured scope\.TranslationInstruction\-model frontiers, dictionary learning, sparse\-autoencoder feature ablations, matched fixed routed edits and a held\-out low\-rank adaptation comparison test the same correction across model operations\.Pretrained\-model experiments include publicly available checkpoints from the Pythia suite\[[24](https://arxiv.org/html/2609.11063#as1_bib.bib24)\], GPT\-2\[[25](https://arxiv.org/html/2609.11063#as1_bib.bib25)\], Qwen2\[[26](https://arxiv.org/html/2609.11063#as1_bib.bib26)\], Qwen1\.5\[[27](https://arxiv.org/html/2609.11063#as1_bib.bib27)\], GPT\-Neo\[[28](https://arxiv.org/html/2609.11063#as1_bib.bib28)\], Mistral\[[29](https://arxiv.org/html/2609.11063#as1_bib.bib29)\], Mamba\[[30](https://arxiv.org/html/2609.11063#as1_bib.bib30)\], RWKV\[[31](https://arxiv.org/html/2609.11063#as1_bib.bib31)\], BLOOM\[[32](https://arxiv.org/html/2609.11063#as1_bib.bib32)\]and StarCoder2\[[33](https://arxiv.org/html/2609.11063#as1_bib.bib33)\]\. The main\-text randomised causal acquisition results use explicitly controlled synthetic systems\. Model execution and intervention solves use float32 unless an analysis specifies float64 dense spectral arithmetic\. Most GPU\-resident model\-forward runs use one NVIDIA RTX 5000 Ada with 32 GB of memory, with additional model runs on NVIDIA A100 GPUs and CPU execution for dense statistical analyses\. A full GPU\-resident pullback solve fits through 2\.8 billion parameters\. At 6\.9 billion, the evidence is limited to graph\-free output\-Fisher spectra, a CPU\-only mid\-layer full\-vocabulary pullback calibration and a descriptive late\-layer CPU\-offload iteration analysis\. Given a random seed, the conjugate\-gradient and Woodbury solvers are deterministic; seeds are logged for bootstrap resampling, dictionary learning and Monte Carlo rollout values\. The map above organises the central claim strands\. Materials supporting the reported analyses, including model revisions, prompt, pair and objective\-set listings, relevant analysis code and summary logs, are available from the corresponding author on reasonable request; code and data availability are stated in the main text\.

### Supplementary References

- \[1\]N\. N\. Chentsov\.*Statistical Decision Rules and Optimal Inference*, volume 53 of*Translations of Mathematical Monographs*\.American Mathematical Society, 1982\.
- \[2\]Nihat Ay, Jürgen Jost, Hông Vân Lê, and Lorenz Schwachhöfer\.*Information Geometry*, volume 64 of*Ergebnisse der Mathematik und ihrer Grenzgebiete\. 3\. Folge*\.Springer, Cham, 2017\.ISBN 978\-3\-319\-56478\-4\.doi:10\.1007/978\-3\-319\-56478\-4\.
- \[3\]Nils Reimers and Iryna Gurevych\.Sentence\-BERT: Sentence embeddings using siamese BERT\-networks\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing*, pages 3982–3992\. Association for Computational Linguistics, 2019\.doi:10\.18653/v1/D19\-1410\.
- \[4\]Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc\-Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández\.The LAMBADA dataset: Word prediction requiring a broad discourse context\.In*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics*, pages 1525–1534\. Association for Computational Linguistics, 2016\.doi:10\.18653/v1/P16\-1144\.
- \[5\]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\.Direct preference optimization: Your language model is secretly a reward model\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- \[6\]Sihan Wang, Jiayi Zhao, Qingyan Cao, Hongbo Yao, and Lin Shu\.FishBack: Pullback Fisher geometry for optimal activation steering in transformers, 2026\.arXiv:2605\.17231\.
- \[7\]Kiho Park, Todd Nief, Yo Joong Choe, and Victor Veitch\.The information geometry of softmax: Probing and steering\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.arXiv:2602\.15293\.
- \[8\]Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen\.EasyEdit: An easy\-to\-use knowledge editing framework for large language models, 2024\.arXiv:2308\.07269\.
- \[9\]Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\.Locating and editing factual associations in GPT\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- \[10\]Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau\.Mass\-editing memory in a transformer\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- \[11\]Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi\.Aging with GRACE: Lifelong model editing with discrete key\-value adaptors\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- \[12\]Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D\. Manning, and Chelsea Finn\.Memory\-based model editing at scale\.In*Proceedings of the 39th International Conference on Machine Learning \(ICML\)*, 2022\.
- \[13\]Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner\.Steering Llama 2 via contrastive activation addition\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15504–15522\. Association for Computational Linguistics, 2024\.doi:10\.18653/v1/2024\.acl\-long\.828\.
- \[14\]Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu\.Scaling and evaluating sparse autoencoders, 2024\.arXiv:2406\.04093\.
- \[15\]Shun\-ichi Amari\.Natural gradient works efficiently in learning\.*Neural Computation*, 10\(2\):251–276, 1998\.
- \[16\]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- \[17\]Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V\. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El\-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po\-Yao Huang, Shang\-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski\.DINOv2: Learning robust visual features without supervision\.*Transactions on Machine Learning Research*, 2024\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=a68SUt6zFt](https://openreview.net/forum?id=a68SUt6zFt)\.arXiv:2304\.07193\.
- \[18\]Alex Krizhevsky\.Learning multiple layers of features from tiny images\.Technical report, University of Toronto, 2009\.
- \[19\]Fabian Gröger, Shuo Wen, and Maria Brbić\.Revisiting the platonic representation hypothesis: An aristotelian view\.In*Proceedings of the 43rd International Conference on Machine Learning \(ICML\)*, 2026\.arXiv:2602\.14486\.
- \[20\]J\. E\. Peelle, R\. L\. Miller, C\. S\. Rogers, B\. Spehar, M\. S\. Sommers, and K\. J\. Van Engen\.Completion norms for 3085 English sentence contexts\.*Behavior Research Methods*, 52:1795–1799, 2020\.doi:10\.3758/s13428\-020\-01351\-1\.
- \[21\]Allen Institute for AI\.OLMo 2 1B: model and training checkpoints\.[https://huggingface\.co/allenai/OLMo\-2\-0425\-1B](https://huggingface.co/allenai/OLMo-2-0425-1B), 2025\.
- \[22\]Andrea Gregor de Varda, Marco Marelli, and Simona Amenta\.Cloze probability, predictability ratings, and computational estimates for 205 English sentences, aligned with existing EEG and reading time data\.*Behavior Research Methods*, 56:5190–5213, 2024\.doi:10\.3758/s13428\-023\-02261\-8\.
- \[23\]Boi Mai Quach, Cathal Gurrin, and Graham Healy\.DERCo: A dataset for human behaviour in reading comprehension using EEG\.*Scientific Data*, 11:1104, 2024\.doi:10\.1038/s41597\-024\-03915\-8\.
- \[24\]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, 2023\.
- \[25\]Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever\.Language models are unsupervised multitask learners\.Technical report, OpenAI, 2019\.
- \[26\]An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan\.Qwen2 technical report, 2024\.arXiv:2407\.10671\.
- \[27\]Qwen Team\.Introducing Qwen1\.5\.Qwen blog, February 2024\.URL[https://qwenlm\.github\.io/blog/qwen1\.5/](https://qwenlm.github.io/blog/qwen1.5/)\.
- \[28\]Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman\.GPT\-Neo: Large scale autoregressive language modeling with mesh\-TensorFlow, 2021\.
- \[29\]Albert Q\. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie\-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed\.Mistral 7b, 2023\.arXiv:2310\.06825\.
- \[30\]Albert Gu and Tri Dao\.Mamba: Linear\-time sequence modeling with selective state spaces\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=tEYskw1VY2](https://openreview.net/forum?id=tEYskw1VY2)\.
- \[31\]Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui\-Jie Zhu\.RWKV: Reinventing RNNs for the transformer era\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 14048–14077\. Association for Computational Linguistics, 2023\.doi:10\.18653/v1/2023\.findings\-emnlp\.936\.
- \[32\]BigScience Workshop\.BLOOM: A 176b\-parameter open\-access multilingual language model\.*Journal of Machine Learning Research*, 25\(422\):1–74, 2024\.
- \[33\]Anton Lozhkov, Raymond Li, Loubna Ben Allal, et al\.StarCoder 2 and The Stack v2: The next generation, 2024\.arXiv:2402\.19173\.