A Controlled Study of Attention-Only Transformers

# A Controlled Study of Attention-Only Transformers
Source: [https://arxiv.org/html/2607.18363](https://arxiv.org/html/2607.18363)
Henry Ndubuaku, Karen Mosoyan, Jakub Mroz, Noah Cylich, Satyajit Kumar, Parkirat Sandhu, Roman Shemet, Justin H Lee

###### Abstract

Feed\-forward networks hold two thirds of a transformer’s non\-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once\. We pretrain attention\-only decoder transformers \(Simple Attention Networks, SANs\) against standard transformers matched separately for parameter count, training FLOPs, and depth \(2 to 48 layers across arms\), with per\-arm learning\-rate sweeps, for up to 105B tokens \(1\.5 epochs of a 68B\-token reasoning\-dense corpus\) at 6M to 87M total parameters\. Deleting feed\-forward layers in place is costly: the standard transformer leads by 0\.47 nats at matched depth, and by 0\.26 nats at matched training FLOPs, where attention spends compute on the parameter\-free quadratic term and so carries fewer parameters at equal cost\. Reallocating the freed budget into attention depth closes the gap: at matched parameter count the difference is 0\.006 nats, 0\.27 percent of loss, reproducible to one part in ten thousand across clean seed pairs, shrinking across separately trained 5B, 30B, and 105B budgets, and holding near 0\.02 nats across a 29x non\-embedding size range at a fixed 31\.5B\-token budget\. Three independent measurements localize the remaining gap to parametric recall: loss over token regions, loss over task types, and zero\-shot benchmarks\. Attention\-only models are better on context\-grounded answers and worse where knowledge must come from weights\. Weight spectra show why: routing matrices \(Q/K\) crystallize in the first quarter of training while content matrices accumulate rank through the stable phase, and removing feed\-forward layers relocates this accumulation to the attention output projection\. QK\-normalization, not feed\-forward layers or residual gating, keeps 48\-layer attention\-only stacks trainable\. The gap is concentrated, not diffuse: low\-context query prediction carries a per\-token deficit five times the aggregate but only 8 percent of corpus loss, and the token\-weighted decomposition reproduces its own aggregate to within 2 percent; by the largest budget the deficit localizes there entirely on that sample\. A pre\-registered test confirms the account: it predicts a 0\.02 to 0\.05 nat gap on knowledge\-dense web text, and a matched pair trained on fineweb\-edu measures 0\.040\. Within the tested regime, attention does the rest\.

## Introduction

The feed\-forward network \(FFN\) has been a fixed component of the transformer block since its introduction\(Vaswani et al\.[2017](https://arxiv.org/html/2607.18363#bib.bib33)\), and in modern decoder configurations it holds roughly two thirds of non\-embedding parameters\. A substantial interpretability literature argues that these layers act as the model’s parametric memory: key–value stores over training facts\(Geva et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib10),[2022](https://arxiv.org/html/2607.18363#bib.bib9)\), the locus of editable factual associations\(Meng et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib19)\), and the home of identifiable knowledge neurons\(Dai et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib2)\)\. Yet the converse experiment, removing the FFN entirely and measuring what is actually lost, has never been run under controls\. The closest attempt folded equivalent capacity back into attention as persistent memory vectors\(Sukhbaatar et al\.[2019](https://arxiv.org/html/2607.18363#bib.bib31)\); probing and editing studies localize function without deleting it; and the classical theory of pure attention\(Dong, Cordonnier, and Loukas[2021](https://arxiv.org/html/2607.18363#bib.bib4)\)analyzes a setting without residual connections that no one trains\.

The experiment is harder than it looks because removing the FFN perturbs three quantities at once: parameter count, compute per token, and the number of nonlinear composition steps\. A single\-control comparison is therefore confounded no matter which axis it fixes\. We run the necessity test with three separate matchings, each holding one axis fixed \(Table[1](https://arxiv.org/html/2607.18363#Sx3.T1)\), give every arm its own learning\-rate sweep with a boundary\-extension protocol so that no arm competes at another arm’s preferred hyperparameters, and calibrate the experiment’s noise floor with an optimizer\-equivalent control pair\. All predictions were registered before the corresponding measurements were made, and one, the gap on knowledge\-dense text, was registered numerically before the run was launched\.

The result is a monotone sequence that decomposes the question\. Deleting feed\-forward layers in place, at matched depth, costs 0\.47 nats of validation loss\. At matched training FLOPs, where attention’s quadratic term crowds out parameters, the standard transformer still leads by 0\.26 nats\. But at matched parameter count, with the freed budget reallocated into attention depth, the gap collapses to a small, highly reproducible\+0\.0055\+0\.0055and\+0\.0054\+0\.0054nats on clean seed pairs \(0\.27% of loss, agreeing to10−410^\{\-4\}\), and it shrinks monotonically across separately trained 5B, 30B, and 105B token budgets while staying near 0\.02 nats across a 29×\\timesnon\-embedding parameter range at a fixed 31\.5B\-token budget\. The FFN’s parameters matter; its functional form largely does not, on this distribution\.

What remains of the gap is not diffuse\. Decomposing the loss over token regions and task types, and separately over zero\-shot benchmarks, the deficit concentrates on low\-context prediction: tokens for which the context offers little to route, so that only parametric knowledge helps\. At the largest budget the attention\-only model is ahead on every answer region of the decomposition sample, including memorization exercises, and behind only on low\-context query tokens; on benchmarks, the one task whose answer sits in a provided passage favors the SAN with growing margin, while the one task demanding out\-of\-distribution recall favors the FFN model at every budget\. A weight\-spectrum analysis supplies the mechanism: in every model, routing matrices \(Q/K\) spectrally crystallize within the first quarter of training, while content matrices accumulate stable rank through the stable phase; removing the FFN relocates this accumulation to the attention output projection, the attention\-only model’s only write path into the residual stream\.

Our contributions: \(i\) the first controlled necessity test of the transformer FFN, with parameter, FLOP, and depth matchings, per\-arm learning\-rate fairness, and a calibrated noise floor; \(ii\) a three\-way localization of the residual gap to low\-context prediction, confirmed by a numerically pre\-registered out\-of\-distribution test \(predicted 0\.02–0\.05 nats on fineweb\-edu; measured 0\.040\); \(iii\) the identification of QK\-normalization, not feed\-forward layers or residual gating, as the component that keeps deep attention\-only stacks trainable; and \(iv\) a weight\-spectrum account in which routing crystallizes early, content accumulates through the stable phase, and storage relocates when the FFN is removed\.

## Related Work

The FFN as memory\.Geva et al\. \([2021](https://arxiv.org/html/2607.18363#bib.bib10)\)identify FFN layers as key–value memories;Geva et al\. \([2022](https://arxiv.org/html/2607.18363#bib.bib9)\)refine the mechanism; ROME\(Meng et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib19)\)and knowledge neurons\(Dai et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib2)\)localize editable facts in mid\-layer FFNs\. These are probing and editing results; ours is the causal complement, showing by deletion that the capability lost is precisely parametric storage\. Attention weights store associations too\(Elhage et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib6)\), so storage is a tendency with causal evidence rather than an exclusive location; our decomposition measures the tendency’s size instead of assuming it\.Pires et al\. \([2023](https://arxiv.org/html/2607.18363#bib.bib26)\)find FFNs redundant across layers;He and Hofmann \([2024](https://arxiv.org/html/2607.18363#bib.bib12)\)remove other block components via signal\-propagation arguments\.

Attention\-only models\.Sukhbaatar et al\. \([2019](https://arxiv.org/html/2607.18363#bib.bib31)\)remove the FFN but restore its capacity as persistent memory vectors, leaving the necessity question open\. The mechanistic\-interpretability literature studies small attention\-only transformers as an analyzable model class\(Elhage et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib6); Olsson et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib24)\), establishing the QK/OV routing\-versus\-content decomposition that our spectral dynamics confirm at realistic depth\.Dong, Cordonnier, and Loukas \([2021](https://arxiv.org/html/2607.18363#bib.bib4)\)prove rank collapse for attention stacks without residuals; our 48\-layer results and rank measurements bound where that analysis applies to trained practice, and identify QK\-normalization\(Henry et al\.[2020](https://arxiv.org/html/2607.18363#bib.bib13); Dehghani et al\.[2023](https://arxiv.org/html/2607.18363#bib.bib3)\)as the operative stabilizer\.

The dual ablation\.MLP\-only architectures\(Tolstikhin et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib32); Liu et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib17)\)map the attention\-free direction; MetaFormer\(Yu et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib34)\)argues the block frame matters more than the mixer\. We supply the missing attention\-only side for language modeling: deleting the channel MLP costs 0\.006 nats at matched parameters, with a specific, predictable residual profile\.

Components and optimizer\.Our scalar residual gates belong to the Highway/ReZero/LayerScale family\(Srivastava, Greff, and Schmidhuber[2015](https://arxiv.org/html/2607.18363#bib.bib30); Bachlechner et al\.[2020](https://arxiv.org/html/2607.18363#bib.bib1)\); sandwich normalization followsShleifer, Weston, and Ott \([2021](https://arxiv.org/html/2607.18363#bib.bib29)\); zero\-centered norm gains followOLMo Team \([2024](https://arxiv.org/html/2607.18363#bib.bib23)\); gated attention outputs are studied byQiu et al\. \([2025](https://arxiv.org/html/2607.18363#bib.bib28)\), and depth\-wise residual reweighting byKimi Team \([2026](https://arxiv.org/html/2607.18363#bib.bib16)\)\. We train with Muon\(Jordan et al\.[2024](https://arxiv.org/html/2607.18363#bib.bib15); Liu et al\.[2025](https://arxiv.org/html/2607.18363#bib.bib18)\)against an AdamW control\.

Data regime\.We pretrain on SYNTH\(PleIAs[2025](https://arxiv.org/html/2607.18363#bib.bib27)\), a reasoning\-dense synthetic corpus, deliberately overtrained relative to compute\-optimal\(Hoffmann et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib14)\), in the small\-model synthetic\-pretraining lineage ofEldan and Li \([2023](https://arxiv.org/html/2607.18363#bib.bib5)\); Gunasekar et al\. \([2023](https://arxiv.org/html/2607.18363#bib.bib11)\); repetition behavior extendsMuennighoff et al\. \([2023](https://arxiv.org/html/2607.18363#bib.bib21)\)to synthetic data\. Chain\-of\-thought expressivity results\(Merrill and Sabharwal[2024](https://arxiv.org/html/2607.18363#bib.bib20); Feng et al\.[2023](https://arxiv.org/html/2607.18363#bib.bib7)\)motivate the trace\-formatted regime, with the caveat that those constructions use MLPs\.

## Experimental Setup

Architecture\.A SAN block is a pre\-norm attention block with the FFN deleted, exactly:

ZCN​\(z\)\\displaystyle\\mathrm\{ZCN\}\(z\)=\(1\+γ\)⊙z/RMS​\(z\),γ​init​0\\displaystyle=\(1\+\\gamma\)\\odot z/\\mathrm\{RMS\}\(z\),\\quad\\gamma\\ \\text\{init \}0u\\displaystyle u=ZCN​\(x\)\\displaystyle=\\mathrm\{ZCN\}\(x\)q,k,v\\displaystyle q,k,v=Wq​u,Wk​u,Wv​u\(GQA: 8Q, 4KV\)\\displaystyle=W\_\{q\}u,\\ W\_\{k\}u,\\ W\_\{v\}u\\quad\\text\{\(GQA: 8Q, 4KV\)\}q,k\\displaystyle q,k=RoPE​\(ZCNh​\(q\),ZCNh​\(k\)\)\\displaystyle=\\mathrm\{RoPE\}\\big\(\\mathrm\{ZCN\}\_\{h\}\(q\),\\ \\mathrm\{ZCN\}\_\{h\}\(k\)\\big\)A\\displaystyle A=softmax​\(q​k⊤/dh\+M\)\\displaystyle=\\mathrm\{softmax\}\\big\(qk^\{\\top\}/\\sqrt\{d\_\{h\}\}\+M\\big\)y\\displaystyle y=x\+σ​\(g\)⋅Wo​\(A​v\)\\displaystyle=x\+\\sigma\(g\)\\cdot W\_\{o\}\(A\\,v\)withMMthe causal, document\-boundary mask over packed 2048\-token rows,gga scalar per sublayer \(init 0, so every branch starts at half strength\), tied embeddings, and a final ZCN before the head\. The control arm inserts a SwiGLU FFN,y′=y\+σ​\(g2\)⋅Wdown​SwiGLU​\(Wgate​u~,Wup​u~\)y^\{\\prime\}=y\+\\sigma\(g\_\{2\}\)\\cdot W\_\{\\mathrm\{down\}\}\\,\\mathrm\{SwiGLU\}\(W\_\{\\mathrm\{gate\}\}\\tilde\{u\},W\_\{\\mathrm\{up\}\}\\tilde\{u\}\)withdf​f=4​dd\_\{ff\}=4d, after attention; both arms share one implementation\. The complete inventory of per\-position nonlinearities in the SAN is the attention softmax and the normalizations: there is no learned per\-position feature map, which is the entire experimental manipulation\. Four short results make precise what the manipulation removes\.

###### Proposition 1\(Conditional linearity\)\.

For any fixed row\-stochastic attention patternAA, the map from the normalized context\(u^1,…,u^T\)\(\\hat\{u\}\_\{1\},\\dots,\\hat\{u\}\_\{T\}\)to the layer’s updates is linear\.

###### Proof\.

It is the composition of the linear mapsWvW\_\{v\},v↦A​vv\\mapsto Av, andWoW\_\{o\}, scaled by the constantσ​\(g\)\\sigma\(g\)\. ∎

###### Proposition 2\(Simplex transport\)\.

Per head, the pre\-projection update at positioniilies in the convex hull of\{Wv​u^j:j≤i\}\\\{W\_\{v\}\\hat\{u\}\_\{j\}:j\\leq i\\\}\.

###### Proof\.

Softmax rows are nonnegative and sum to one\. ∎

A SAN layer selects and transports content present in context; it cannot synthesize representations unsupported by it\. This is the formal sense of “context\-grounded” used throughout\.

###### Lemma 1\(Scalar bottleneck\)\.

At sequence length 1, the entire depth\-LLnetwork is the linear mapxL=\[∏ℓ\(I\+B~ℓ/ρℓ\)\]​x0x\_\{L\}=\\big\[\\prod\_\{\\ell\}\(I\+\\tilde\{B\}\_\{\\ell\}/\\rho\_\{\\ell\}\)\\big\]x\_\{0\}, whereB~ℓ=σ​\(gℓ\)​Wo​Wv​diag​\(1\+γℓ\)\\tilde\{B\}\_\{\\ell\}=\\sigma\(g\_\{\\ell\}\)W\_\{o\}W\_\{v\}\\,\\mathrm\{diag\}\(1\+\\gamma\_\{\\ell\}\)is constant andρℓ=RMS​\(xℓ\)\\rho\_\{\\ell\}=\\mathrm\{RMS\}\(x\_\{\\ell\}\): a linear map modulated by exactlyLLscalars\.

###### Proof\.

A=\[1\]A=\[1\]and RoPE at position 0 is the identity, so each block computesxℓ\+1=\(I\+B~ℓ/ρℓ\)​xℓx\_\{\\ell\+1\}=\(I\+\\tilde\{B\}\_\{\\ell\}/\\rho\_\{\\ell\}\)\\,x\_\{\\ell\}; induct over layers\. ∎

All per\-position nonlinearity flows through thisLL\-scalar bottleneck; cross\-token attention is the only escape, and pricing that restriction is what the experiments do\.

###### Theorem 1\(Bounded stream variance at init\)\.

Under the implemented initialization \(exact RMS normalization, gatesσ​\(0\)=12\\sigma\(0\)=\\tfrac\{1\}\{2\}, output projections i\.i\.d\. mean\-zero with variances2/\(2​N\)s^\{2\}/\(2N\)\),𝔼​‖xN‖2=𝔼​‖x0‖2\+Θ​\(1\)\\mathbb\{E\}\\\|x\_\{N\}\\\|^\{2\}=\\mathbb\{E\}\\\|x\_\{0\}\\\|^\{2\}\+\\Theta\(1\)uniformly in depthNN\.

###### Proof\.

Cross terms𝔼​⟨xℓ,bℓ⟩\\mathbb\{E\}\\langle x\_\{\\ell\},b\_\{\\ell\}\\ranglevanish becauseWoW\_\{o\}is mean\-zero and independent of everything upstream, so second moments add\. Each branch input is RMS\-normalized, so its second moment is bounded independent of the stream magnitude; row\-stochasticAAcannot increase the maximum value norm \(Proposition 2\); the1/\(2​N\)1/\(2N\)output variance then gives𝔼​‖bℓ‖2=Θ​\(1/N\)\\mathbb\{E\}\\\|b\_\{\\ell\}\\\|^\{2\}=\\Theta\(1/N\), and the sum overNNlayers telescopes toΘ​\(1\)\\Theta\(1\)\. ∎

Each hypothesis is load\-bearing in the proof; empirically, however, the gate factor proved dispensable \(Table[6](https://arxiv.org/html/2607.18363#Sx7.T6)\), so the initialization scaling and normalization carry the result in practice\.

Matchings\.Table[1](https://arxiv.org/html/2607.18363#Sx3.T1)gives the three controls\. Removing the FFN from a 20\-layer model \(iso\-depth\) deletes 72% of its parameters; matching parameters instead \(iso\-param\) reallocates the budget into depth, 20 attention\-only layers against 4 standard blocks; matching training FLOPs \(iso\-FLOP\) sits between, because attention pays a parameter\-free quadratic cost in sequence length\. No single matching is the fair one; we report all three\.

Table 1:The three matchings \(ff= FFN width; GF/tok = training GFLOPs per token\)\. Removing the FFN perturbs parameters, compute, and depth at once; each control fixes one axis\. Iso\-param match is exact to<<0\.04%\.Fairness protocol\.Every arm×\\timesoptimizer cell received its own learning\-rate sweep at 5B tokens \(0\.5×\\times/1×\\times/2×\\timesaround a base point, with a boundary rule extending any sweep\-edge winner until its curve turned; 17 cells total, Table[2](https://arxiv.org/html/2607.18363#Sx3.T2)\)\. The locked rates are interior minima for all four arms\. Global batch \(512 rows, 1\.05M tokens/step\), schedule shape \(warmup\-stable\-decay\), data order, validation set, and all masking are identical across arms\. Where a training phase ran on 7\-GPU nodes, batch and learning rates were frame\-matched so effective values equal the 8\-GPU locks \(511 vs\. 512 rows, 0\.2%\)\.

Table 2:The learning\-rate fairness sweep \(val loss at 5B tokens; 1×\\timesis Muon 0\.02, AdamW 3e\-4\)\. Every lock is an interior minimum\. At the most extreme rate probed, the FFN model’s FFN gates collapsed to mean 0\.07 while its attention gates stayed structured: under rate stress the standard transformer self\-pruned toward attention\-only form\.Noise floor\.Our zero\-centered RMSNorm and standard RMSNorm are optimizer\-equivalent under kernel\-only weight decay \(the gain parameterizations differ by a constant shift, and Adam\-family updates are shift\-invariant\)\. Training both anyway calibrates the same\-seed noise floor:Δ=0\.0015\\Delta=0\.0015nats\. Every noise comparison below is against this floor or the three\-seed spread\. The pipeline’s determinism is strong: a mid\-training restore of one run from its 40% checkpoint, with cold optimizer state, reproduced the original final validation loss exactly \(1\.5982\)\.

Evaluation, and two distinct pools\.Headline validation loss \(Table[3](https://arxiv.org/html/2607.18363#Sx4.T3)\) is computed on a fixed held\-out document pool, disjoint from training\. The region and exercise decomposition \(Table[4](https://arxiv.org/html/2607.18363#Sx5.T4)\) requires exercise metadata that exists only in the raw corpus, so it uses a second, deterministic 20k\-document sample of the corpus head: approximately 99\.9% of these documents were seen in training by both arms \(about 1\.5 times\), and documents longer than the context window are dropped by whole\-document packing, a mild selection toward shorter documents\. Decomposition comparisons are therefore paired contrasts on identical, equally exposed tokens, valid for localizing the gap but not held\-out measurements, and the two pools’ aggregates are not interchangeable \(we reconcile them below\)\. Zero\-shot benchmarks use lm\-evaluation\-harness\(Gao et al\.[2023](https://arxiv.org/html/2607.18363#bib.bib8)\); weight spectra are computed over milestone checkpoints\. Code, training curves, evaluation reports, and all checkpoints, including quarter\-milestones, accompany this work\.

## The Cost of Removing Feed\-Forward Layers

Table 3:Deleting FFNs in place \(iso\-depth\) costs 0\.47 nats; at matched FLOPs 0\.26; reallocating the budget into attention depth leaves a small, seed\-reproducible 0\.006 nats\.†Includes one run with a documented terminal instability \(see Discussion\); excluding it,2\.0650±0\.00062\.0650\\pm 0\.0006\.‡Clean\-pair pairedΔ\\Delta:\+0\.0055\+0\.0055\(s42\),\+0\.0054\+0\.0054\(s44\); incident pair−0\.0491\-0\.0491\. Same\-seed noise floor: 0\.0015\.Table[3](https://arxiv.org/html/2607.18363#Sx4.T3)and Figure[1](https://arxiv.org/html/2607.18363#Sx4.F1)give the main result as a monotone sequence\. Deleting FFNs in place \(iso\-depth control, 87M→\\rightarrow24M parameters\) costs 0\.470 nats\. At matched compute \(iso\-FLOP\), where the FFN arm carries 1\.8×\\timesthe parameters, it leads by 0\.263\. At matched parameters the gap is\+0\.0055\+0\.0055and\+0\.0054\+0\.0054nats on the two clean seed pairs, agreeing to10−410^\{\-4\}; the third pair contains a documented terminal\-phase instability in the FFN run \(gradient\-norm ramp at the learning\-rate floor; Table[3](https://arxiv.org/html/2607.18363#Sx4.T3)\) and reverses the sign\. The sequence orders exactly by how much parameter budget the control lets attention reclaim\.

![Refer to caption](https://arxiv.org/html/2607.18363v1/x1.png)Figure 1:Validation loss over 105B tokens: three SAN seeds, three FFN iso\-param seeds, and the iso\-FLOP and iso\-depth controls\. The iso\-param pair is visually indistinguishable through training\.The token axis\.The iso\-param gap falls from 0\.046 nats at 5B tokens to 0\.019 at 30B to 0\.0055 at 105B, each budget separately trained and separately tuned \(different schedule horizons\), so this is not one curve read at three points\. More training closes the gap\.

The parameter axis\.At a fixed 31\.5B\-token budget across five matched size pairs \(Figure[2](https://arxiv.org/html/2607.18363#Sx4.F2)\), the gap is−0\.045\-0\.045nats at the smallest size \(the 2\-layer FFN partner hits a depth floor first\), crosses zero, and plateaus at≈\\approx0\.02 nats from 16M to 57M non\-embedding parameters\. The gap does not grow with scale in the tested range\.

![Refer to caption](https://arxiv.org/html/2607.18363v1/x2.png)Figure 2:Iso\-token scaling \(31\.5B tokens per size\)\. The SAN wins at the smallest size; from 16M non\-embedding parameters upward the gap is flat at≈\\approx0\.02 nats\.Repetition\.Constraining both arms to 2M, 8M, or 32M unique documents at fixed tokens \(up to 18 epochs\) costs at most 0\.010 nats and shows no architecture×\\timesrepetition interaction \(Figure[3](https://arxiv.org/html/2607.18363#Sx4.F3)\), extending data\-constrained scaling results\(Muennighoff et al\.[2023](https://arxiv.org/html/2607.18363#bib.bib21)\)to synthetic data\.

![Refer to caption](https://arxiv.org/html/2607.18363v1/x3.png)Figure 3:Unique\-document constraints at a fixed 31\.5B\-token budget\. Both arms are nearly repetition\-insensitive, with no architecture interaction\.
## Where the Gap Lives

querytraceansweralltoken share \(%\)5\.857\.235\.9100loss share \(%\)7\.955\.436\.6100Δ\\Delta@31B\+\.052\+\.052\+\.008\+\.008\+\.011\+\.011\+\.011\+\.011Δ\\Delta@105B \(clean\)\+\.038\+\.038−\.004\-\.004−\.007\-\.007−\.0025\-\.0025Table 4:Token\-weighted decomposition on the decomposition sample \(Setup; training\-exposed, not the held\-out pool\):∑\\sumshare×\\timesregion\-Δ\\Deltareproduces the sample aggregate to within 2%\. Point estimates; per\-block losses were not retained\. The deficit concentrates on low\-context query tokens and localizes there entirely by 105B; the same account predicts the fineweb gap \(measured 0\.040, pre\-registered window 0\.02–0\.05\)\.Token regions\.Every corpus document has query, reasoning\-trace, and answer regions delimited by atomic marker tokens\. Table[4](https://arxiv.org/html/2607.18363#Sx5.T4)and Figure[4](https://arxiv.org/html/2607.18363#Sx5.F4)decompose the iso\-param gap on the decomposition sample\. At 31B tokens the per\-token deficit on query regions \(\+0\.052\+0\.052\) is five times the sample aggregate while carrying 8% of its loss; the token\-weighted sum of region gaps reproduces the sample aggregate to within 2%, so the decomposition is exhaustive\. By 105B the localization is complete on this sample: the SAN leads on every answer region, including memorization exercises, and on traces; the deficit survives only on query tokens \(\+0\.038\+0\.038\), the positions with the least context to route from\. Note the pool distinction of the Setup: the decomposition sample’s 105B aggregate \(−0\.0025\-0\.0025, SAN ahead\) differs from the held\-out validation gap \(\+0\.0055\+0\.0055, FFN ahead\) because the samples differ in composition and training exposure; both are parity\-scale, the within\-sample paired contrasts that localize the gap are unaffected, and the two aggregates are not interchangeable\. Across the size ladder the query deficit is the invariant, positive at every size and budget while every other region changes sign \(Figure[4](https://arxiv.org/html/2607.18363#Sx5.F4)b\)\. The query\-localized account is itself a revision: the registered prediction \(gap largest on memorization answers\) was falsified by these measurements, and only the revised account’s fineweb consequence below was pre\-registered\.

![Refer to caption](https://arxiv.org/html/2607.18363v1/x4.png)Figure 4:\(a\) Region\-level gap at 31B vs\. 105B tokens: with training the deficit localizes entirely to low\-context query prediction\. \(b\) Across matched size pairs, the query deficit is the only invariantly FFN\-favored quantity\.Task types\.On answer regions by exercise, the sign flips in the storage\-account’s direction: at 31B the SAN is already better on rag, editing, and math answers \(retrievable from prompt or trace\) and worse on memorization and free\-form generation; by 105B it leads on all of them\.

Table 5:The storage/routing split at task level\. Lambada \(out\-of\-distribution recall for SYNTH\-trained models\) favors the FFN at every budget, but is non\-monotone in FFN capacity: iso\-depth \(87M\), the best in\-distribution model, is the worst at it \(0\.111, ppl 964\)\. Sciq \(support passage in context\) favors the SAN with a margin that grows with training in the pre\-registered direction \(seed ranges non\-overlapping at 105B\) and falls monotonically with FFN capacity\. Trained on knowledge\-dense fineweb\-edu, the lambada preference reverses: a task’s storage/routing identity is relative to the training distribution\. Remaining tasks are at chance at these scales\.Benchmarks\.Table[5](https://arxiv.org/html/2607.18363#Sx5.T5)shows the same split at task level\. Lambada, predicting a specific content word and, for SYNTH\-trained models, an out\-of\-distribution recall task, favors the FFN arm at every budget\. Sciq, whose answer sits in a provided support passage, favors the SAN, and the margin grows with training in a direction we registered in advance: from 31B to 105B the SAN improves \(0\.725→\\rightarrow0\.742\) while the FFN model regresses \(0\.702→\\rightarrow0\.661\), with non\-overlapping seed ranges\. The remaining tasks sit at chance at these scales\.

A pre\-registered out\-of\-distribution test\.The account so far says the FFN’s advantage is low\-context prediction\. Natural web text is mostly low\-context prediction\. Before launching the run, we registered the prediction that an iso\-param pair trained on fineweb\-edu\(Penedo et al\.[2024](https://arxiv.org/html/2607.18363#bib.bib25)\)would show a gap of 0\.02 to 0\.05 nats, bounded above by the pure query\-region deficit and well above the SYNTH aggregate\. The measured gap is 0\.0398\. Instructively, the same pair reverses on lambada \(SAN 0\.203 vs\. FFN 0\.181\): trained on distribution\-matched text, the passage suffices to infer the final word, and lambada becomes a routing task\. A task’s storage\-versus\-routing identity is relative to the match between training distribution and task, a point the discussion returns to\.

## Mechanism: Weight\-Spectrum Dynamics

![Refer to caption](https://arxiv.org/html/2607.18363v1/x5.png)Figure 5:Stable rank \(‖W‖F2/‖W‖22\\\|W\\\|\_\{F\}^\{2\}/\\\|W\\\|\_\{2\}^\{2\}, layer mean\) of weight matrices over training\. \(a\) Muon holds Q and output\-projection spectra 2–3×\\timesflatter than AdamW in the SAN\. \(b\) Routing matrices \(Q\) crystallize by a quarter of training in every model; write\-path matrices \(SANWoW\_\{o\}, FFN down\-projection\) accumulate rank through the stable phase, contracting only in the learning\-rate decay tail\.Figure[5](https://arxiv.org/html/2607.18363#Sx6.F5)tracks the stable rank of every weight matrix over milestone checkpoints\. Two regularities hold across all architectures, sizes, and budgets we trained\. First, routing matrices \(Q/K\) spectrally crystallize within the first quarter of training and do not move thereafter; the optimizer sets the level \(Muon holds them 2–3×\\timesflatter than AdamW, alike in both architectures; the value projection is optimizer\-invariant\), but the schedule is universal\. The Muon side is the a\-priori expectation: for orthogonalized updates, Weyl’s inequality bounds every singular value’s per\-step drift by the learning rate, though the bound controls drift only and does not by itself predict the level separation we observe\. Second, the matrices that write content into the residual stream accumulate stable rank for as long as the stable phase lasts: the FFN down\-projection in FFN models, and in attention\-only models the output projectionWoW\_\{o\}, which inherits the role\. This is the training\-dynamics face of the QK/OV routing\-versus\-content decomposition\(Elhage et al\.[2021](https://arxiv.org/html/2607.18363#bib.bib6)\): selection structure is learned early and frozen; content capacity accrues indefinitely, in whichever matrices can hold it\. Representation effective rank stays high everywhere \(minimum layer rank 173 of 512 at 20 layers\), so the classical collapse regime\(Dong, Cordonnier, and Loukas[2021](https://arxiv.org/html/2607.18363#bib.bib4); Noci et al\.[2022](https://arxiv.org/html/2607.18363#bib.bib22)\)is never approached with residuals and normalization in place\.

## Component Ablations

VariantVal lossΔ\\Deltavs gatedgated \(baseline, 20L\)2\.1343—\+\+sandwich norm2\.1251−0\.0092\-0\.0092standard residual2\.1330−0\.0013\-0\.0013ReZero residual2\.1375\+0\.0032\+0\.0032RMSNormγ=1\\gamma\{=\}1\(control\)2\.1358\+0\.0015\+0\.0015no residual2\.8826\+0\.75\+0\.75no QK\-norm*diverges*\(8\.28\)—depth 8/32/48L, gated2\.168/2\.153/2\.175U\-shapedstandard residual2\.164/2\.150/2\.177gates neutralFFN 20L gated/standard1\.6977/1\.6930neutralTable 6:QK\-norm is the load\-bearing component; gates are performance\-neutral at every depth and in both architectures \(their value was diagnostic\); sandwich norm is the only improvement\. The RMSNorm control is optimizer\-equivalent by construction and calibrates the same\-seed noise floor \(0\.0015\)\.Table[6](https://arxiv.org/html/2607.18363#Sx7.T6)isolates the components\. Three results matter\. First, QK\-normalization is load\-bearing: removing it diverges outright at the tuned learning rate, the only divergence in the study and a finding none of our predictions anticipated; the claim is scoped to the tuned rate\. Second, residual gating is performance\-neutral everywhere, at 20–48 layers in the SAN arm and in the FFN mirror, matching ReZero\-family expectations\(Bachlechner et al\.[2020](https://arxiv.org/html/2607.18363#bib.bib1)\); the gates’ value was diagnostic, as their trajectories exposed the FFN arm’s self\-pruning under learning\-rate stress \(Table[2](https://arxiv.org/html/2607.18363#Sx3.T2)\)\. Third, the only variant to beat the baseline is sandwich normalization\(Shleifer, Weston, and Ott[2021](https://arxiv.org/html/2607.18363#bib.bib29)\)at−0\.009\-0\.009nats\. Depth at iso\-param is U\-shaped with a 20\-layer optimum, and 48\-layer attention\-only stacks train without incident\. The optimizer interaction is budget\-dependent: a full crossover at 5B tokens \(each architecture preferring a different optimizer, Table[2](https://arxiv.org/html/2607.18363#Sx3.T2)\) washes out entirely by 30B \(Table[7](https://arxiv.org/html/2607.18363#Sx7.T7)\)\.

Table 7:The optimizer×\\timesarchitecture grid at 30B tokens, each cell at its own tuned rate: within\-architecture deltas are≤0\.002\\leq 0\.002nats, against a 5B\-token crossover of 0\.022 and 0\.017 \(Table[2](https://arxiv.org/html/2607.18363#Sx3.T2)\)\. The interaction is a short\-horizon phenomenon\.A recipe, in one paragraph\.For attention\-only stacks at these scales: QK\-normalization is mandatory \(its removal is the study’s only divergence\); post\-attention sandwich normalization is a free improvement; scalar residual gates cost nothing and expose useful training diagnostics, but standard residuals match them at every depth tested; depth prefers 20 layers at this width, and 48 layers remain trainable\. More consequential than any of these, tune the learning rate per architecture: the optimal Muon rate differs by 2×\\timesbetween the arms, and a shared rate silently biases any comparison\.

## Discussion and Limitations

What the FFN buys\.Under controls, the transformer’s FFN is not a distinct computational primitive whose removal breaks the model; it is parameter capacity attached to a write path\. Give the same capacity to attention depth and, on reasoning\-dense data, what remains is a small, highly reproducible cost: 0\.006 nats, 0\.27% of loss, agreeing to10−410^\{\-4\}across clean seed pairs, well outside the paired\-seed spread and 3\.7×\\timesthe same\-seed floor, reversing sign only under a documented FFN\-arm instability\. We claim a tiny, real effect, not a null\. What is lost specifically is performance on tokens where context provides nothing to route, and the loss relocates, in weight space, to whichever matrices write content into the stream\. Separately, every instability event in the study occurred in the FFN arm and none in the SAN arm \(gate self\-pruning under learning\-rate stress, Table[2](https://arxiv.org/html/2607.18363#Sx3.T2); one terminal\-phase divergence, Table[3](https://arxiv.org/html/2607.18363#Sx4.T3)\), a practitioner\-relevant robustness asymmetry we report as observation\.

Distribution\-relative task identity\.The fineweb reversal on lambada and the iso\-depth model’s pattern \(best in\-distribution loss, worst out\-of\-distribution recall\) argue that storage\-versus\-routing is not a fixed property of a task: it is a property of the match between the task and the training distribution\. This reframes small\-model benchmark comparisons generally, and it cautions against reading our SYNTH results as distribution\-free\.

When would one use a SAN?At matched parameters the SAN pays≈\\approx2×\\timesFLOPs per token at 2048 context; iso\-FLOP favors the FFN\. The trade favors attention\-only models where parameters, not FLOPs, bind \(on\-device memory limits\), where architectural simplicity has analysis value \(a SAN is one repeated, interpretable primitive\), and on trace\-rich distributions\. We claim regime, not superiority\.

Limitations\.All results are at or below 87M total parameters and 105B tokens, on one reasoning\-dense corpus plus one knowledge\-dense control pair; MMLU\-class benchmarks are at chance throughout and cannot yet discriminate\. Size\-flatness is measured at 31\.5B tokens and token convergence at the base size only; their conjunction at larger scale is extrapolation\. The region decomposition reports point estimates \(per\-block intervals were not retained\), and its confirmation of the revised account rests on a single out\-of\-distribution point\. The storage account itself predicts a wider gap on storage\-heavy mixtures at larger scale, which is the natural next experiment\. Of the eight predictions fixed before their measurements, two were falsified outright; the central localization finding emerged from one of those failures\.

## References

- Bachlechner et al\. \(2020\)Bachlechner, T\.; Majumder, B\. P\.; Mao, H\. H\.; Cottrell, G\. W\.; and McAuley, J\. 2020\.ReZero Is All You Need: Fast Convergence at Large Depth\.*arXiv preprint arXiv:2003\.04887*\.
- Dai et al\. \(2022\)Dai, D\.; Dong, L\.; Hao, Y\.; Sui, Z\.; Chang, B\.; and Wei, F\. 2022\.Knowledge Neurons in Pretrained Transformers\.In*Proceedings of ACL*\.
- Dehghani et al\. \(2023\)Dehghani, M\.; Djolonga, J\.; Mustafa, B\.; et al\. 2023\.Scaling Vision Transformers to 22 Billion Parameters\.In*International Conference on Machine Learning*\.
- Dong, Cordonnier, and Loukas \(2021\)Dong, Y\.; Cordonnier, J\.\-B\.; and Loukas, A\. 2021\.Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth\.In*International Conference on Machine Learning*\.
- Eldan and Li \(2023\)Eldan, R\.; and Li, Y\. 2023\.TinyStories: How Small Can Language Models Be and Still Speak Coherent English?*arXiv preprint arXiv:2305\.07759*\.
- Elhage et al\. \(2021\)Elhage, N\.; Nanda, N\.; Olsson, C\.; et al\. 2021\.A Mathematical Framework for Transformer Circuits\.*Transformer Circuits Thread*\.
- Feng et al\. \(2023\)Feng, G\.; Zhang, B\.; Gu, Y\.; Ye, H\.; He, D\.; and Wang, L\. 2023\.Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective\.In*Advances in Neural Information Processing Systems*\.
- Gao et al\. \(2023\)Gao, L\.; Tow, J\.; Abbasi, B\.; et al\. 2023\.A Framework for Few\-Shot Language Model Evaluation\.https://github\.com/EleutherAI/lm\-evaluation\-harness\.
- Geva et al\. \(2022\)Geva, M\.; Caciularu, A\.; Wang, K\. R\.; and Goldberg, Y\. 2022\.Transformer Feed\-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space\.In*Proceedings of EMNLP*\.
- Geva et al\. \(2021\)Geva, M\.; Schuster, R\.; Berant, J\.; and Levy, O\. 2021\.Transformer Feed\-Forward Layers Are Key\-Value Memories\.In*Proceedings of EMNLP*\.
- Gunasekar et al\. \(2023\)Gunasekar, S\.; Zhang, Y\.; Aneja, J\.; et al\. 2023\.Textbooks Are All You Need\.*arXiv preprint arXiv:2306\.11644*\.
- He and Hofmann \(2024\)He, B\.; and Hofmann, T\. 2024\.Simplifying Transformer Blocks\.In*International Conference on Learning Representations*\.
- Henry et al\. \(2020\)Henry, A\.; Dachapally, P\. R\.; Pawar, S\. S\.; and Chen, Y\. 2020\.Query\-Key Normalization for Transformers\.In*Findings of EMNLP*\.
- Hoffmann et al\. \(2022\)Hoffmann, J\.; Borgeaud, S\.; Mensch, A\.; et al\. 2022\.Training Compute\-Optimal Large Language Models\.*arXiv preprint arXiv:2203\.15556*\.
- Jordan et al\. \(2024\)Jordan, K\.; Jin, Y\.; Boza, V\.; You, J\.; Cesista, F\.; Newhouse, L\.; and Bernstein, J\. 2024\.Muon: An Optimizer for Hidden Layers in Neural Networks\.https://kellerjordan\.github\.io/posts/muon/\.
- Kimi Team \(2026\)Kimi Team\. 2026\.Attention Residuals\.*arXiv preprint arXiv:2603\.15031*\.
- Liu et al\. \(2021\)Liu, H\.; Dai, Z\.; So, D\.; and Le, Q\. V\. 2021\.Pay Attention to MLPs\.In*Advances in Neural Information Processing Systems*\.
- Liu et al\. \(2025\)Liu, J\.; Su, J\.; Yao, X\.; et al\. 2025\.Muon Is Scalable for LLM Training\.*arXiv preprint arXiv:2502\.16982*\.
- Meng et al\. \(2022\)Meng, K\.; Bau, D\.; Andonian, A\.; and Belinkov, Y\. 2022\.Locating and Editing Factual Associations in GPT\.In*Advances in Neural Information Processing Systems*\.
- Merrill and Sabharwal \(2024\)Merrill, W\.; and Sabharwal, A\. 2024\.The Expressive Power of Transformers with Chain of Thought\.In*International Conference on Learning Representations*\.
- Muennighoff et al\. \(2023\)Muennighoff, N\.; Rush, A\. M\.; Barak, B\.; et al\. 2023\.Scaling Data\-Constrained Language Models\.In*Advances in Neural Information Processing Systems*\.
- Noci et al\. \(2022\)Noci, L\.; Anagnostidis, S\.; Biggio, L\.; Orvieto, A\.; Singh, S\. P\.; and Lucchi, A\. 2022\.Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse\.In*Advances in Neural Information Processing Systems*\.
- OLMo Team \(2024\)OLMo Team\. 2024\.2 OLMo 2 Furious\.*arXiv preprint arXiv:2501\.00656*\.
- Olsson et al\. \(2022\)Olsson, C\.; Elhage, N\.; Nanda, N\.; et al\. 2022\.In\-Context Learning and Induction Heads\.*Transformer Circuits Thread*\.
- Penedo et al\. \(2024\)Penedo, G\.; Kydlíček, H\.; Lozhkov, A\.; et al\. 2024\.The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale\.*arXiv preprint arXiv:2406\.17557*\.
- Pires et al\. \(2023\)Pires, T\. P\.; Lopes, A\. V\.; Assogba, Y\.; and Setiawan, H\. 2023\.One Wide Feedforward Is All You Need\.*arXiv preprint arXiv:2309\.01826*\.
- PleIAs \(2025\)PleIAs\. 2025\.SYNTH: A Generalist Synthetic Dataset for Reasoning\-First Pretraining\.https://huggingface\.co/datasets/PleIAs/SYNTH\.
- Qiu et al\. \(2025\)Qiu, Z\.; Wang, Z\.; Zheng, B\.; et al\. 2025\.Gated Attention for Large Language Models: Non\-Linearity, Sparsity, and Attention\-Sink\-Free\.*arXiv preprint arXiv:2505\.06708*\.
- Shleifer, Weston, and Ott \(2021\)Shleifer, S\.; Weston, J\.; and Ott, M\. 2021\.NormFormer: Improved Transformer Pretraining with Extra Normalization\.*arXiv preprint arXiv:2110\.09456*\.
- Srivastava, Greff, and Schmidhuber \(2015\)Srivastava, R\. K\.; Greff, K\.; and Schmidhuber, J\. 2015\.Highway Networks\.*arXiv preprint arXiv:1505\.00387*\.
- Sukhbaatar et al\. \(2019\)Sukhbaatar, S\.; Grave, E\.; Lample, G\.; Jegou, H\.; and Joulin, A\. 2019\.Augmenting Self\-Attention with Persistent Memory\.*arXiv preprint arXiv:1907\.01470*\.
- Tolstikhin et al\. \(2021\)Tolstikhin, I\. O\.; Houlsby, N\.; Kolesnikov, A\.; et al\. 2021\.MLP\-Mixer: An All\-MLP Architecture for Vision\.In*Advances in Neural Information Processing Systems*\.
- Vaswani et al\. \(2017\)Vaswani, A\.; Shazeer, N\.; Parmar, N\.; Uszkoreit, J\.; Jones, L\.; Gomez, A\. N\.; Kaiser, Ł\.; and Polosukhin, I\. 2017\.Attention Is All You Need\.In*Advances in Neural Information Processing Systems*\.
- Yu et al\. \(2022\)Yu, W\.; Luo, M\.; Zhou, P\.; Si, C\.; Zhou, Y\.; Wang, X\.; Feng, J\.; and Yan, S\. 2022\.MetaFormer Is Actually What You Need for Vision\.In*Proceedings of CVPR*\.

## Appendix A: Additional Formal Results

Propositions 1–2, the scalar\-bottleneck lemma, and the bounded\-variance theorem are stated and proved in the body of the paper\. Two further results inform component choices and the optimizer analysis\. Notation as in the main text: norm gain\(1\+γ\)\(1\+\\gamma\)withγ\\gammainit 0\.

###### Proposition 3\(Weight\-decay fixed point of norm gains\)\.

UnderL2L\_\{2\}decay, the penalty onγ\\gammain the\(1\+γ\)\(1\+\\gamma\)parameterization is minimized at gain 1 \(identity\); under the standard parameterization \(gainhh, init 1\) it is minimized ath=0h=0, the zero map\.

###### Proof\.

Both objectives are strictly convex with the stated minimizers\. ∎

In the study’s fixed frame, weight decay applies to kernels only, so the two parameterizations are additionally*optimizer\-equivalent*: they differ by a constant shift, and Adam\-family updates are shift\-invariant\. Training both anyway yieldedΔ=0\.0015\\Delta=0\.0015nats, the same\-seed noise floor used throughout\. The proposition’s content, that zero\-centering makes “do nothing” the regularizer’s fixed point, becomes operative only in frames that decay norm gains\.

###### Lemma 2\(Muon bounds per\-step spectral drift, and only that\)\.

Let the update beΔ=−η​P\\Delta=\-\\eta PwithPPthe polar factor of the momentum gradient \(Newton–Schulz approximation errorε\\varepsilon\)\. By Weyl’s inequality for singular values, for everyii,\|σi​\(W\+Δ\)−σi​\(W\)\|≤‖Δ‖2=η​\(1\+ε\)\|\\sigma\_\{i\}\(W\+\\Delta\)\-\\sigma\_\{i\}\(W\)\|\\leq\\\|\\Delta\\\|\_\{2\}=\\eta\(1\+\\varepsilon\); afterttsteps every singular value lies withint​ηt\\etaof its initialization\.

This bounds drift; it does not prove spectra stay balanced nor that balance improves loss\. The measured outcome is that Muon holds routing\-matrix spectra 2–3×\\timesflatter than AdamW in both architectures alike: the effect is real but not architecture\-differential\.

## Appendix B: Registration Record

All hypotheses and predictions were fixed before the corresponding measurements; the fineweb\-edu prediction was fixed numerically before that run launched\. Outcomes against the registered predictions:

P1Confirmed\.Iso\-param gap within the seed band at 105B \(clean pairs\+0\.0055/\+0\.0054\+0\.0055/\+0\.0054\)\.P2Confirmed at 31B\(trace gap smallest\); at 105B trace and answer both reach parity, superseded by full localization\.P3Falsified as stated\.The gap peaks on low\-context query tokens, not memorization answers; the restated account then made the confirmed fineweb prediction\.P4Gates component falsifiedby its own criterion \(gated≈\\approxungated everywhere\); trainability confirmed; QK\-norm emerged, unpredicted, as the load\-bearing component\.P5Not exercised: the fixed frame decays kernels only; the measurement instead calibrated the noise floor \(0\.0015\)\.P6Budget\-dependent: full optimizer×\\timesarchitecture crossover at 5B; gone by 30B\.P7Half\-confirmed: spectra flatter under Muon everywhere, but not differentially in the SAN; representation rank high everywhere \(minimum 173/512 at 20 layers\)\.P8Falsified: no architecture×\\timesrepetition interaction; both arms nearly repetition\-insensitive to 18 epochs\.Table 8:Registered predictions vs\. outcomes\. The instruments were designed so that predictions could fail; two did, and one failure produced the paper’s central finding\.
## Appendix C: Learning\-Rate Probes for the Size Ladder

The size ladder reuses the base\-tuned rates at every size\. Before interpreting its curve shape, 3\-point rate probes \(0\.5×\\times/1×\\times/2×\\timesof the lock, 2\.5k\-step horizon, since a 30k\-horizon run’s early validation is not schedule\-comparable\) at the smallest and largest sizes confirmed the transferred lock is in the flat region at tiny \(2\.851/2\.856/2\.857\) and an interior minimum at xl \(2\.074/2\.066/2\.078\); no per\-size sweeps were needed\.

## Appendix D: Per\-Seed Downstream Results

Table 9:Per\-seed 0\-shot results behind the 105B means of the main text\. Sciq seed ranges do not overlap between arms \(SAN 0\.718–0\.757 vs\. FFN 0\.650–0\.673\)\.†The seed with the terminal\-phase instability\.Table[9](https://arxiv.org/html/2607.18363#Sx12.T9)gives the per\-seed values behind the 105B means reported in the body of the paper\. The capacity\-axis rows \(iso\-FLOP, iso\-depth\) and the fineweb\-trained pair appear in the body of the paper’s downstream table\.

## Appendix E: Repetition: Exact Values

Validation loss at 30k steps \(31\.5B tokens\) under unique\-document constraints:

Table 10:Both arms are nearly repetition\-insensitive; within\-arm non\-monotonicity \(∼\\sim0\.005–0\.009\) is comparable to data\-subset variance, and there is no architecture interaction\.
## Appendix F: Gate Trajectories and FFN\-Arm Observations

![Refer to caption](https://arxiv.org/html/2607.18363v1/x6.png)Figure 6:Gate values over training, SAN 105B run \(layers×\\timestime\)\. Early layers self\-attenuate and late layers sharpen under Muon\.Although performance\-neutral, the gates were diagnostically productive\. Three independent observations of FFN\-arm fragility: \(i\) at 16×\\timesthe tuned rate, all FFN gates collapsed \(mean 0\.07\) while attention gates stayed structured; \(ii\) at the tuned rate under Muon, FFN gates ended low \(mean 0\.16–0\.17\) in three separate runs, echoing the same self\-pruning; \(iii\) one of three FFN seeds at 105B suffered a terminal\-phase instability, its gradient norm tripling over the final 800 steps at the learning\-rate floor \(Figure[7](https://arxiv.org/html/2607.18363#Sx14.F7)\); no SAN run showed any of these behaviors\. We report these as observations, not claims\.

![Refer to caption](https://arxiv.org/html/2607.18363v1/x7.png)Figure 7:The FFN iso\-param seed\-43 terminal instability\. The run tracked its sibling seed to within 0\.002 nats through step 99,000\. Its final value is included, flagged, in all statistics\.
## Appendix G: Depth Ladder

![Refer to caption](https://arxiv.org/html/2607.18363v1/x8.png)Figure 8:Iso\-parameter depth ladder for the SAN arm, gated vs\. standard residuals: U\-shaped in depth with a 20\-layer optimum; gates neutral at every depth; 48 layers train without incident\.
## Appendix H: Reproducibility Details

Infrastructure\.Training ran on 8×\\timesH100 80GB nodes \(two phases on 7×\\timesH100, frame\-matched below\) with single\-node data parallelism, bfloat16 activations, JAX 0\.10 with CUDA 12\.9 on Ubuntu 22\.04; evaluation on a single GPU\.

Frame matching\.All headline results use a fixed global batch of 512 rows \(64/device×\\times8\) of 2048 tokens\. Phases run on 7\-GPU nodes used 73 rows/device \(511 total,−0\.2%\-0\.2\\%\) with learning rates rescaled so effective values equal the 8\-device locks \(Adam scales linearly in device count, Muon as its square root\)\. A base\-size pair rerun across frames agreed to 0\.0033 nats\.

Determinism and recovery\.Checkpoints store parameters, not optimizer state\. One run was resumed from its 40%\-checkpoint after an artifact loss; despite cold optimizer state at the stable\-phase rate, it reproduced the original final validation loss exactly \(1\.5982\), suggesting the warm\-restart transient is fully absorbed by the learning\-rate decay phase\.

Statistics\.Headline comparisons use three seeds per arm with paired per\-seed differences; the same\-seed noise floor \(0\.0015 nats\) comes from the optimizer\-equivalent normalization pair\. Per\-block losses were not retained, so block\-level bootstrap intervals are not reported; seed\-level agreement \(clean pairs matching to10−410^\{\-4\}\) is the operative evidence\.

Artifacts\.Code, tokenizer, corpus tokenization manifests, all checkpoints with 25% milestones, training logs, and every evaluation report accompany this work\.