A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers

# A Reproducibility Study of Partial Residual Ablations: Removing the Attention Skip Causes Collapse; Removing the FFN Skip Recovers at 10M but Remains Unresolved at 124M
Source: [https://arxiv.org/html/2608.14689](https://arxiv.org/html/2608.14689)
###### Abstract

I study what happens when attention\-skip and FFN\-skip connections are selectively removed from Pre\-LN transformer blocks, training GPT\-style models at 10M \(TinyShakespeare\) and 124M \(OpenWebText\) parameters\. The main finding is an asymmetry: removing the attention skip \(FFNOnly configuration\) causes deterministic collapse to the No\-Residual floor across every seed and environment tested \(mean 3\.350±\\pm0\.002, 3 seeds at 10M\)\. Removing the FFN skip \(AttnOnly\) shows a confirmed recovery effect at 10M: a controlled 8\-seed sweep under forced determinism on an A100 gives mean 1\.580±\\pm0\.003, with all seeds remaining at least 1\.76 validation\-loss units below the collapse floor\. An intermediate 3\-seed reproduction on different hardware gave 2\.591±\\pm0\.544 – one seed reaching the floor – and remains an unresolved cross\-environment discrepancy\. At 124M, three AttnOnly seeds give mean 6\.151, std 1\.105; this is suggestive but not yet confirmed by a controlled sweep\. Mid\-experiment, I identified and corrected a measurement confound: gain applied as a runtime multiplier is absorbed by AdamW within∼\\sim200 steps, making all gain values equivalent\. I propose a cross\-position routing hypothesis for the asymmetry and release all code, checkpoints, and data – including the non\-reproducing runs – for independent investigation\.

## 1Introduction

Residual connections, introduced by He et al\.\[[1](https://arxiv.org/html/2608.14689#bib.bib1)\]for image classification, are now a defining structural feature of transformer language models\[[2](https://arxiv.org/html/2608.14689#bib.bib2),[3](https://arxiv.org/html/2608.14689#bib.bib3)\]\. In the Pre\-LN formulation each block applies two sublayers with identity shortcuts:

xk\+1/2\\displaystyle x^\{k\+1/2\}=xk\+Attn​\(LN1​\(xk\)\)\(attn\_res = True\)\\displaystyle=x^\{k\}\+\\mathrm\{Attn\}\(\\mathrm\{LN\}\_\{1\}\(x^\{k\}\)\)\\quad\(\\texttt\{attn\\\_res = True\}\)\(1\)xk\+1\\displaystyle x^\{k\+1\}=xk\+1/2\+FFN​\(LN2​\(xk\+1/2\)\)\(ffn\_res = True\)\\displaystyle=x^\{k\+1/2\}\+\\mathrm\{FFN\}\(\\mathrm\{LN\}\_\{2\}\(x^\{k\+1/2\}\)\)\\quad\(\\texttt\{ffn\\\_res = True\}\)\(2\)The skip connections serve two purposes: they provide a gradient highway that prevents vanishing gradients in deep networks, and they allow each layer to learn residual transformations rather than full mappings\[[1](https://arxiv.org/html/2608.14689#bib.bib1)\]\. The block Jacobian with a residual connection contains an identity term:

∂xℓ\+1∂xℓ=I\+∂f​\(xℓ\)∂xℓ\.\\frac\{\\partial x\_\{\\ell\+1\}\}\{\\partial x\_\{\\ell\}\}=I\+\\frac\{\\partial f\(x\_\{\\ell\}\)\}\{\\partial x\_\{\\ell\}\}\.\(3\)This identity term guarantees gradient flow regardless of sublayer behavior\. Without it, gradients must traverse the full composition of nonlinear transformations and typically vanish\. Setting either residual flag toFalseremoves the corresponding identity path, giving four configurations: Full Residual \(both skips\), AttnOnly \(attention skip only\), FFNOnly \(FFN skip only\), and No Residual \(neither skip\)\.

My initial experiments with a nanoGPT\-style implementation reveal an apparently clean result: all three partial configurations collapse identically, reaching validation loss≈\\approx3\.35 at 10M scale and≈\\approx7\.4 at 124M – indistinguishable from No Residual\. I treat this as a starting observation, not a conclusion, and ask whether the collapse is architectural or an artifact of the specific implementation and evaluation methodology\.

#### Contributions\.

\(1\) A two\-scale controlled ablation \(10M and 124M\) establishing the joint\-collapse observation under nanoGPT and its gradient\-starvation signature\. \(2\) A cleaner reimplementation \(ResidualGPT\) with fixed validation batches revealing a 2\.09×\\timesasymmetry at 10M scale, confirmed by a controlled 8\-seed deterministic sweep: AttnOnly mean 1\.580±\\pm0\.003, FFNOnly mean 3\.350±\\pm0\.002, gap of 1\.770 validation\-loss units, no seed approaching the collapse floor\. An intermediate 3\-seed reproduction on different hardware gave 2\.591±\\pm0\.544 and is retained as an unresolved cross\-environment discrepancy rather than omitted\. \(3\) A scale observation at 124M \(FFNOnly/AttnOnly = 1\.56×\\timesat seed 1337\) with two additional AttnOnly seeds showing large variance, motivating a controlled multi\-seed 124M sweep as the next step\. \(4\) Identification and honest reporting of a measurement confound \(runtime gain scaling vs\. weight\-level initialization\) discovered and corrected mid\-experiment\. \(5\) A mechanistic hypothesis with a falsifiable prediction; the falsification test was run under the anomalous intermediate environment and is inconclusive pending a rerun under the clean sweep conditions\. \(6\) Full release of code, checkpoints, and all experimental data, including the intermediate non\-reproducing runs, so the complete investigation trail can be independently examined\.

## 2Background

### 2\.1Residual Networks and Gradient Flow

He et al\.\[[1](https://arxiv.org/html/2608.14689#bib.bib1)\]showed that identity shortcuts allow training of very deep networks by ensuring gradient magnitude at layerℓ\\ellis at least as large as at layerℓ\+1\\ell\+1\. The residual stream formalization by Elhage et al\.\[[4](https://arxiv.org/html/2608.14689#bib.bib4)\]extended this to transformers: each sublayer reads from and writes to a shared vector that accumulates information across blocks\. Removing a residual connection disrupts this compositional structure, not merely the optimization landscape\.

### 2\.2Initialization and Residual Coupling

Standard initialization \(std = 0\.02\) implicitly assumes the identity path dominates early training\. T\-Fixup\[[5](https://arxiv.org/html/2608.14689#bib.bib5)\]and GPT\-2 output scaling\[[3](https://arxiv.org/html/2608.14689#bib.bib3)\]demonstrate that initialization scale and residual connections are tightly coupled\. When a residual is removed, this assumption breaks: the sublayer output is no longer stabilized by the identity path at step zero\. I exploit this coupling in my falsification experiment, testing whether initialization adjustment can rescue partial\-residual configurations\.

## 3Experimental Setup

### 3\.1Two Implementations

I report results from two implementations\. The nanoGPT implementation \(no weight tying, random validation batches at every eval, manual dot\-product attention\) was used for the initial joint\-collapse observation at both scales\. The ResidualGPT implementation uses weight tying between the token embedding and language model head,F\.scaled\_dot\_product\_attention\(flash attention\), fixed validation batches \(created once with seed 99,991 – identical across all configurations\), and per\-layer activation storage for diagnostics\. All asymmetry claims use ResidualGPT as the primary implementation\. Implementation differences are explicitly reported wherever they affect results\.

### 3\.2Hyperparameters

Table[1](https://arxiv.org/html/2608.14689#S3.T1)summarizes all hyperparameters\.

Table 1:Hyperparameters for all experiments\.Parameter10M124MSharedn\_layer612lr: 3×\\times10\-4\(nanoGPT: constant; ResidualGPT 10M: cosine, 100\-step warmup, floor 3×\\times10\-5\) / 6×\\times10\-4→\\rightarrow6×\\times10\-5cosine, 2,000\-step warmup \(ResidualGPT 124M\)n\_embd384768β1\\beta\_\{1\}=0\.9,β2\\beta\_\{2\}=0\.95, weight decay=0\.1n\_head612Grad clip=1\.0, dropout=0\.2 \(10M\) / 0\.1 \(124M\)block\_size2561024Optimizer: AdamWBatch size6416Grad accum: 1 \(10M\) / 8 \(124M\)Train steps3,00020,000Parameters≈\\approx10\.7M / 124\.4MVal interval100500Dataset: TinyShakespeare / OpenWebText

### 3\.3Datasets

The 10M experiments use TinyShakespeare with character\-level tokenization \(vocab=65, train=1\.0M tokens, val=111K\)\. The 124M experiments use OpenWebText\[[6](https://arxiv.org/html/2608.14689#bib.bib6)\]with BPE tokenization viatiktoken\(vocab=50,257,∼\\sim113M tokens from 100,000 documents\)\. Fixed validation batches \(seed 99,991\) are used in all ResidualGPT runs, ensuring identical evaluation data across all configurations\.

### 3\.4Configurations

Four configurations are tested in all experiments: Full Residual \(both skips active\), No Residual \(neither skip\), AttnOnly \(attention skip kept, FFN skip removed\), and FFNOnly \(FFN skip kept, attention skip removed\)\. All hyperparameters are held constant across configurations\. The only variable is which residual connections are active\.

## 4The Initial Observation: Joint Collapse

Table[2](https://arxiv.org/html/2608.14689#S4.T2)shows the nanoGPT results at 10M scale\. The three partial configurations collapse symmetrically to validation loss≈\\approx3\.35, a 2\.27×\\timesdegradation relative to Full Residual and indistinguishable from No Residual\. This is the joint\-collapse observation\.

Table 2:nanoGPT implementation, 10M params, TinyShakespeare, seed 1337\.ConfigAttnFFNBest valPPLFinal valRatioFull Residual✓\\checkmark✓\\checkmark1\.47514\.371\.47511\.00×\\timesAttnOnly✓\\checkmark×\\times3\.353428\.603\.35342\.27×\\timesFFNOnly×\\times✓\\checkmark3\.347528\.433\.34752\.27×\\timesNo Residual×\\times×\\times3\.352328\.573\.35232\.27×\\timesFigure[1](https://arxiv.org/html/2608.14689#S4.F1)shows val loss curves at both 10M and 124M nanoGPT scale\. The pattern is identical: Full Residual descends to low loss while all partial configurations plateau immediately after an initial drop\. At 124M, degradation ratios \(2\.13×\\timesto 2\.18×\\times\) are consistent with 10M \(2\.27×\\times\), confirming the joint\-collapse observation is robust across scale, dataset, and tokenization scheme\. In Table[2](https://arxiv.org/html/2608.14689#S4.T2), Best val and Final val coincide for all four 10M configurations: validation loss was still monotonically improving at the last logged checkpoint \(step 2,999\) for each configuration, so the final measurement is also the best one observed during training\.

![Refer to caption](https://arxiv.org/html/2608.14689v1/figure1_joint_collapse.png)Figure 1:Validation loss curves at 10M \(left, TinyShakespeare\) and 124M \(right, OpenWebText\) using the nanoGPT implementation\. All partial configurations plateau identically, indistinguishable from No Residual at both scales\.### 4\.1Mechanistic Signature of the Joint Collapse

Despite identical validation losses, the four configurations show qualitatively different internal behavior \(Figure[2](https://arxiv.org/html/2608.14689#S5.F2)\)\. Layer 0 gradient norm collapses to exactly 0\.000 from step 300 onward for all partial configurations, while Full Residual maintains≈\\approx0\.114 throughout\. This is the gradient highway effect: the identity term in Equation[1](https://arxiv.org/html/2608.14689#S1.E1)provides a direct gradient path to early layers regardless of sublayer behavior\. Hidden\-state norm growth ratios diverge significantly: Full Residual grows 1\.38×\\times, while AttnOnly grows 14\.03×\\timesand FFNOnly 5\.37×\\times\. This divergence – despite identical losses – is the first indication that the two partial configurations are failing for different reasons\.

## 5Challenging the Observation

### 5\.1A Measurement Confound: Runtime Scaling vs\. Weight Initialization

My first attempt to test initialization sensitivity applied gain as a runtime multiplier: output = g×\\timessublayer\(x\) at every forward pass, for g∈\\in\{0\.25, 0\.5, 2\.0, 4\.0\}\. At g = 2\.0, AttnOnly appeared to recover to best val 1\.54\. I identified this as a confound before reporting the result: AdamW’s per\-parameter adaptive learning rate compensates for a constant multiplicative reparameterization within≈\\approx200 steps, making all gain values functionally equivalent\. Loss curves across all gain values overlapped completely \(total spread<<0\.002 at any step\)\. The apparent recovery was an optimizer artifact, not a genuine initialization effect\. I corrected the implementation to use weight\-level reinitialization applied before optimizer construction, and rebuilt the experiment\. This confound is reported here in full as a methodological contribution: it is easy to miss, produces convincing\-looking results, and requires careful experimental design to avoid\.

![Refer to caption](https://arxiv.org/html/2608.14689v1/figure3_mechanistic_diagnostics.png)Figure 2:Mechanistic diagnostics from 10M nanoGPT\. \(a\) Gradient starvation: Layer 0 norm collapses to 0\.000 at step 300 for all partial configs; Full Residual sustains≈\\approx0\.114\. \(b\) Representation drift: AttnOnly 14\.0×\\times, FFNOnly 5\.4×\\times, Full Residual 1\.4×\\times\. \(c\) Training outcome \(ResidualGPT, not nanoGPT\): AttnOnly’s originally recorded value \(1\.600\) alongside a subsequent reproduction \(2\.591\) that did not recover it; FFNOnly \(3\.349\) reproduces closely and is shown unhatched\.
### 5\.2Implementation Difference: Fixed Validation Batches

The nanoGPT implementation evaluates on randomly sampled validation batches at every checkpoint\. For a slowly\-learning model such as AttnOnly, this introduces high evaluation variance: a model that is genuinely improving can appear to have plateaued under noisy evaluation\. ResidualGPT creates fixed validation batches once at startup \(seed 99,991\) and reuses the identical samples at every evaluation across all four configurations\. This eliminates evaluation noise as a confound and is the most likely explanation for why AttnOnly shows 3\.35 in nanoGPT and 1\.58 in ResidualGPT on the same dataset\. I verify thatset\_global\_seed\(\)does not reseed the batcher generator, confirming fixed val batches are identical across all four configs\. This explanation is not formally isolated from other differences between the two implementations: ResidualGPT also differs from nanoGPT in weight tying between the token embedding and output head, use of flash attention \(F\.scaled\_dot\_product\_attention\) versus a manual implementation, and minor optimizer/initialization details\. I have not run an ablation that holds all of these fixed except validation batching, so the fixed\-validation\-batch explanation should be read as the most plausible hypothesis given the available evidence, not as a demonstrated causal mechanism; any of the bundled differences, individually or in combination, could instead be responsible\.

## 6The Central Observation: A Confirmed 10M Asymmetry and an Unresolved 124M Scaling Behavior

### 6\.110M Scale — ResidualGPT

Table[3](https://arxiv.org/html/2608.14689#S6.T3)shows the ResidualGPT primary results at 10M scale, as originally recorded: AttnOnly was measured below the collapse floor while FFNOnly collapsed\. A later intermediate reproduction on different hardware did not recover this statistic, but a subsequent controlled 8\-seed deterministic sweep confirmed the 10M effect: AttnOnly mean 1\.580±\\pm0\.003 across 8 seeds, FFNOnly mean 3\.350±\\pm0\.002, gap of 1\.770 val loss units, no seed approaching the floor\. See Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)for the full three\-phase account\.

Table 3:ResidualGPT implementation, 10M params, TinyShakespeare\. Mean±\\pmstd across 3 seeds where available\. AttnOnly values are as originally recorded; see Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)for a subsequent reproduction attempt that did not recover this exact statistic\.ConfigBest val loss±\\pmstdPPLSeedsvs\. Full ResidualFull Residual1\.5017 –4\.491337– \(baseline\)AttnOnly1\.5999 0\.01494\.951337, 42, 123\+6\.5% \[ORIGINALLY RECORDED BELOW FLOOR\]FFNOnly3\.3493 0\.000928\.481337, 42, 123\+123\.0% \[COLLAPSES\]No Residual3\.3564 –28\.691337\+123\.5% \[COLLAPSES\]Figure[3](https://arxiv.org/html/2608.14689#S6.F3)visualizes the apparent asymmetry, alongside a subsequent per\-seed reproduction attempt that did not recover the original AttnOnly value – and in one seed, reached the collapse floor itself \(see Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)for the full account\)\. As originally recorded, the FFNOnly/AttnOnly ratio is 2\.09×\\times\. FFNOnly and No Residual are separated by only 0\.007 loss units – well within measurement noise, and reproduced closely\. The FFNOnly std = 0\.0009 is roughly sixteen times tighter than AttnOnly’s originally recorded std = 0\.0149, indicating FFNOnly collapse is essentially deterministic; AttnOnly’s recovery is not similarly characterizable by a tight std, since a subsequent 3\-seed reproduction under forced determinism gave std = 0\.544, two orders of magnitude larger, with the worst seed indistinguishable from the FFNOnly floor\.

![Refer to caption](https://arxiv.org/html/2608.14689v1/figure2_10M_asymmetry.png)Figure 3:Apparent asymmetry at 10M scale, original record and per\-seed reproduction\. AttnOnly’s original mean \(1\.5999\) was recorded well below the collapse floor; reproduction under forced determinism instead shows substantial per\-seed spread, with seed 42 \(3\.3542\) reaching the FFNOnly/No Residual collapse floor itself \(3\.3493/3\.3564\) rather than staying clearly apart from it\. FFNOnly reproduces closely at the floor in every attempt\. ResidualGPT – 10M – TinyShakespeare\. See Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)for the full reproduction account\.![Refer to caption](https://arxiv.org/html/2608.14689v1/figure4_3seed_stability.png)Figure 4:Phase 3 controlled 8\-seed sweep – 10M ResidualGPT, TinyShakespeare\. \(a\) AttnOnly best val loss across 8 independent seeds under forced determinism \(A100, PyTorch 2\.12\.0\+cu130, CUDA 13\.0\)\. Mean = 1\.580, std = 0\.003; no seed within 1\.76 val loss units of the FFNOnly floor \(dotted red line, mean 3\.350\)\. \(b\) Summary across all four configs: FullResidual anchor \(1\.499\), AttnOnly \(1\.580±\\pm0\.003, 8 seeds\), FFNOnly \(3\.350±\\pm0\.002, 3 seeds\), NoResidual anchor \(3\.356\)\. The 10M asymmetry is confirmed: AttnOnly tightly avoids the collapse floor that FFNOnly reliably reaches\.
### 6\.2124M Scale Verification — ResidualGPT

Table[4](https://arxiv.org/html/2608.14689#S6.T4)shows the 124M ResidualGPT results at the originally recorded seed \(1337\)\. This was originally read as the critical scale confirmation: does AttnOnly convergence persist at GPT\-2 scale on a large web corpus? At seed 1337, the recorded value \(4\.899\) sits clearly apart from the floor; two additional seeds run subsequently \(Figure[5](https://arxiv.org/html/2608.14689#S6.F5), Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)\) complicate this picture, with seed 42 \(7\.586\) landing close enough to the floor \(7\.666/7\.689\) that the qualitative distinction is no longer clearly established at this scale either\.

Table 4:ResidualGPT implementation, 124M params, OpenWebText, seed 1337, 20,000 steps\. Two additional AttnOnly seeds \(42, 123\), run subsequently, are shown in Figure[5](https://arxiv.org/html/2608.14689#S6.F5)and discussed in Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3); they are not included in this table because Table[4](https://arxiv.org/html/2608.14689#S6.T4)reports each configuration at its originally tested seed\.ConfigBest val lossPPLRatio / ResultFull Residual3\.263026\.131\.00×\\times/ BaselineAttnOnly4\.8994134\.211\.50×\\times/ ORIGINALLY RECORDED BELOW FLOOR \(\+50\.1% above Full Res, seed 1337\)FFNOnly7\.66622,1352\.35×\\times/ COLLAPSES \(gap from No Res = 0\.023\)No Residual7\.68872,1842\.36×\\times/ Floor![Refer to caption](https://arxiv.org/html/2608.14689v1/figure5_124M_scale_verification.png)Figure 5:124M ResidualGPT results, now with three AttnOnly seeds\. \(a\) AttnOnly seed 1337 \(originally recorded\) converges to 4\.899; seeds 42 and 123, run subsequently, give 7\.586 and 5\.967 – seed 42 in particular sits just below the FFNOnly/No Residual floor \(dotted line\), close enough that the qualitative distinction is no longer clearly established\. FFNOnly collapses to 7\.666 and reproduces closely\. \(b\) The AttnOnly gap above Full Residual, comparing the originally recorded seed against the range from the other two seeds: at 10M the range is 41\.8–123\.4% versus 6\.5% recorded; at 124M the range is 50\.2–132\.5% versus 50\.1% recorded\. The qualitative distinction between AttnOnly and FFNOnly, clear in the originally recorded seeds, is now confirmed by the 8\-seed sweep \(Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)\): AttnOnly mean 1\.580±\\pm0\.003 across 8 seeds, FFNOnly mean 3\.350±\\pm0\.002, gap of 1\.770 val loss units\. The 124M result shows higher seed variance and warrants further investigation\.
### 6\.3Cross\-Scale Summary

Table[5](https://arxiv.org/html/2608.14689#S6.T5)summarizes the asymmetry across both scales and implementations\.

Table 5:Cross\-scale asymmetry summary\. ResidualGPT AttnOnly values are as originally recorded\. At 10M, a subsequent 8\-seed sweep under forced determinism \(seeds 1337, 42, 123, 0, 7, 99, 2024, 512\) confirmed the asymmetry: mean 1\.580±\\pm0\.003, no seed approaching the FFNOnly floor \(mean 3\.350±\\pm0\.002\)\. An intermediate 3\-seed reproduction on different hardware gave 2\.591±\\pm0\.544 and was identified as an unresolved cross\-environment discrepancy\. At 124M, three AttnOnly seeds give mean 6\.151, std 1\.105, suggesting higher seed sensitivity at scale; a controlled multi\-seed 124M sweep remains as future work\.ScaleImpl\.AttnOnly best valFFNOnly best valAttnOnly gapAsym\. ratio10MnanoGPT3\.353 \(collapses\)3\.348 \(collapses\)\+127\.3%1\.00×\\times10MResidualGPT1\.600±\\pm0\.0153\.349±\\pm0\.001\+6\.5%2\.09×\\times10MResidualGPT Phase 3 \(8\-seed sweep\)1\.580±\\pm0\.0033\.350±\\pm0\.002\+5\.4%2\.12×\\times124MnanoGPT7\.433 \(collapses\)7\.430 \(collapses\)\+113\.5%1\.00×\\times124MResidualGPT4\.8997\.666\+50\.1%1\.56×\\times

Two observations from Table[5](https://arxiv.org/html/2608.14689#S6.T5): \(1\) As originally recorded, the asymmetry’s direction appeared consistent across all conditions where it is detectable – AttnOnly always converging short of the collapse floor, FFNOnly always collapsing to it, in ResidualGPT\. The 10M direction is now confirmed across 8 seeds \(mean gap 1\.770 val loss units, std 0\.003\) under controlled deterministic conditions; the 124M direction remains suggestive but not yet confirmed given high seed variance\. \(2\) As originally recorded, the asymmetry magnitude decreases from 2\.09×\\timesat 10M to 1\.56×\\timesat 124M, and the AttnOnly gap above Full Residual increases from 6\.5% to 50\.1%\. The 10M magnitude is now confirmed under the controlled 8\-seed sweep \(mean 1\.580±\\pm0\.003 vs\. originally recorded 1\.600±\\pm0\.015\)\. The 124M magnitude remains unresolved because additional seeds showed high variance \(mean 6\.151, std 1\.105\); the scale trend should be read as a hypothesis suggested by the original recordings rather than a confirmed magnitude until a controlled 124M multi\-seed sweep is completed\.

## 7Mechanistic Hypothesis

Both AttnOnly and FFNOnly exhibit identical gradient starvation at Layer 0 \(norm = 0\.000 from step 300 onward\) in the nanoGPT implementation\. Yet, in the originally recorded runs and confirmed by a subsequent 8\-seed deterministic sweep, AttnOnly recovered and FFNOnly did not – a distinction that an intermediate 3\-seed reproduction on different hardware did not recover but that the controlled 8\-seed sweep subsequently confirmed\. The hypothesis below was developed to explain this distinction\. The 10M version of the effect is now confirmed; the mechanism remains hypothetical and the 124M scaling behavior remains unresolved\. I propose the explanation lies in a structural property of the attention sublayer that the FFN sublayer does not share\. I label this a working hypothesis, not a proven account\.

### 7\.1Gradient Starvation is Universal But Not Predictive of Outcome

Figure[2](https://arxiv.org/html/2608.14689#S5.F2)shows the gradient norm at Layer 0 \(the earliest layer\)\. Full Residual maintains≈\\approx0\.114 throughout training\. All three partial configurations collapse to 0\.000 at step 300 and remain there\. This pattern holds at both 10M and 124M scale, and is consistent with the theory: removing the identity term from Equation[1](https://arxiv.org/html/2608.14689#S1.E1)eliminates the direct gradient path, forcing gradients through the full composition of nonlinear transformations which collapses near\-zero at initialization\. Gradient starvation is thus a necessary consequence of removing any residual, not a distinguishing factor between AttnOnly and FFNOnly\.

### 7\.2The Pointwise FFN Cannot Reconstruct Lost Identity Information

The FFN sublayer operates pointwise\. For inputX∈ℝT×dX\\in\\mathbb\{R\}^\{T\\times d\}, at each positionttindependently:

FFN​\(X\)t=W2​GELU​\(W1​xt\+b1\)\+b2\.\\mathrm\{FFN\}\(X\)\_\{t\}=W\_\{2\}\\,\\mathrm\{GELU\}\(W\_\{1\}x\_\{t\}\+b\_\{1\}\)\+b\_\{2\}\.\(4\)Each positionttis transformed using only its own representationxtx\_\{t\}, with no access to information from other positions\. Without a residual skip, if early representations fail to carry useful signal forward, there is no recovery mechanism available to the FFN: the pointwise transformation cannot use cross\-position information to compensate for the missing identity path\. If this account is correct, it would explain why FFNOnly≈\\approxNo Residual \(gap = 0\.007 at 10M, 0\.023 at 124M\): the FFN residual preserves per\-token local identity but cannot substitute for the cross\-position routing the attention residual would provide\.

### 7\.3Self\-Attention Provides Implicit Cross\-Position Routing

The attention sublayer aggregates across positions:

Attn​\(X\)t=∑j=1tαt​j​\(X\)⋅Vjwhere​Vj=WV​xj\.\\mathrm\{Attn\}\(X\)\_\{t\}=\\sum\_\{j=1\}^\{t\}\\alpha\_\{tj\}\(X\)\\cdot V\_\{j\}\\qquad\\text\{where \}V\_\{j\}=W\_\{V\}x\_\{j\}\.\(5\)where attention weightsαt​j\\alpha\_\{tj\}depend on all positions 1 throughttvia the query\-key product\. Positiontt’s output at layerℓ\+1\\ell\+1aggregates representations from all preceding positions at layerℓ\\ell, weighted by learned content similarity\. The model can learnαt​t≈1\\alpha\_\{tt\}\\approx 1to approximate self\-identity – not a free skip connection, but a learnable substitute that is available to attention and unavailable to the FFN\. I call this implicit cross\-position routing\.

This hypothesis is consistent with three observations: \(1\) AttnOnly hidden norm grows 14\.03×\\timesvs\. FFNOnly 5\.37×\\times– AttnOnly representations are being actively restructured through learned routing patterns, not passively failing\. \(2\) As originally recorded, the asymmetry magnitude decreases at 124M \(AttnOnly gap \+50\.1% vs\. \+6\.5% at 10M\), suggesting routing capacity through attention is finite and harder to maintain across 12 layers than 6 – though this specific scale trend did not reproduce when additional seeds were tested at both scales \(Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)\), so it is better read as a hypothesis the original recordings suggested than a confirmed pattern\. \(3\) FFNOnly≈\\approxNo Residual at both scales: the FFN residual cannot create cross\-position routing where none existed, and this observation reproduced closely in every attempt\.

### 7\.4Falsifiable Prediction

If cross\-position routing explains AttnOnly recovery, replacing self\-attention with a purely local mixer \(sliding\-window convolution, position\-blind MLP, or Mamba\-style state\-space model with no explicit cross\-position interaction\) should eliminate AttnOnly recovery: the implicit routing mechanism would no longer exist, and AttnOnly should collapse like FFNOnly\. This prediction distinguishes the routing account from simpler alternatives such as “attention simply has larger gradients at initialization\.” I attempted this test with a position\-blind MLP mixer at 10M scale \(seed 1337, forced determinism\): the mixer converged to 2\.487, statistically indistinguishable from this same environment’s own AttnOnly reproduction \(2\.591±\\pm0\.544 across seeds; see Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)\) rather than the originally recorded AttnOnly value \(1\.580\) or the FFNOnly floor \(3\.349\)\. Because the AttnOnly baseline itself did not reproduce in this environment, this result cannot presently confirm or falsify the routing hypothesis – it is equally consistent with “the mixer behaves like AttnOnly” and “both are reflecting the same unresolved cross\-environment discrepancy\.” Because this test was run under the anomalous Phase 2 environment, it should not be treated as a conclusive test of the routing hypothesis\. A clean falsification requires rerunning the local\-mixer experiment under the same Phase 3 environment that produced AttnOnly = 1\.580±\\pm0\.003\.

## 8Discussion

### 8\.1What the Paper Claims

AttnOnly was originally recorded converging well short of the FFNOnly/No Residual floor at both scales\. At 10M, this is now confirmed: a controlled 8\-seed sweep under forced determinism gives AttnOnly mean 1\.580±\\pm0\.003 across seeds 1337, 42, 123, 0, 7, 99, 2024, and 512 – no seed approaching the collapse floor, gap of 1\.770 val loss units from FFNOnly\. An intermediate 3\-seed reproduction on different hardware gave 2\.591±\\pm0\.544 and is retained as an unresolved cross\-environment discrepancy rather than the final characterization\. At 124M, the relationship is not yet established: three AttnOnly seeds give 4\.899, 7\.586, and 5\.967 \(mean 6\.151, std 1\.105\), which is too variable to characterize without a controlled multi\-seed sweep\. What is confirmed: FFNOnly collapses to 3\.349±\\pm0\.001 \(10M, 3 seeds\) and 7\.666 \(124M, seed 1337\)\. What is confirmed at 10M: AttnOnly avoids that floor by a clear and seed\-stable margin \(1\.580±\\pm0\.003, 8 seeds\)\. What remains unresolved: AttnOnly behavior at 124M scale, the mechanism behind the asymmetry, and the cause of the intermediate cross\-environment reproduction discrepancy – see Section[8\.3](https://arxiv.org/html/2608.14689#S8.SS3)for the full account\.

Gradient starvation at Layer 0 is universal: all partial configurations exhibit zero gradient norm from step 300 onward in the nanoGPT implementation at both 10M and 124M scale\.

Fixed validation batches explain the nanoGPT/ResidualGPT discrepancy for AttnOnly\. This is not formally proven; ablating each implementation difference individually is deferred\.

Cross\-position routing explains why AttnOnly recovers while FFNOnly does not\. Consistent with all diagnostic evidence; no formal proof; not tested in alternative architectures\.

### 8\.2On Research Practice

The runtime gain confound is the most methodologically important event in this project\. I applied gain as a runtime multiplier, observed apparent recovery, identified the optimizer\-reparameterization mechanism that made all gain values equivalent, corrected the implementation, and rebuilt the experiment from scratch\. The corrected result – an apparent asymmetry that later exposed a deeper reproducibility gap – is more precise and more interesting than the confounded one\. Reporting this sequence is a deliberate choice: the arc from observation to confound to correction to an honestly reported reproduction failure is the research, not merely its output\.

### 8\.3Limitations

The reproduction investigation went through three phases\. Phase 1 \(non\-deterministic\): three AttnOnly runs at seed 1337 on different hardware under default CUDA kernels gave 1\.97, 1\.94, and 2\.06 – confirmed non\-deterministic\. Phase 2 \(forced determinism, new hardware\):torch\.use\_deterministic\_algorithms\+ fixed cuBLAS workspace eliminated run\-to\-run variance but did not close the gap\. Seeds 1337/42/123 gave 2\.289, 3\.354, 2\.130 \(mean 2\.591, std 0\.544\); one seed reached the FFNOnly floor\. No record of the original software environment survived\. Phase 3 \(8\-seed controlled sweep\): seeds 1337, 42, 123, 0, 7, 99, 2024, 512 on a clean A100\-PCIE\-40GB \(PyTorch 2\.12\.0\+cu130, CUDA 13\.0, forced determinism throughout\)\. AttnOnly: mean 1\.580, std 0\.003, min 1\.577, max 1\.586\. FFNOnly control \(3 seeds\): mean 3\.350, std 0\.002\. Mean\-to\-mean gap: 1\.770\. All AttnOnly seeds remained at least 1\.76 val loss units below the FFNOnly floor \(closest: 1\.586, distance 1\.764\)\. The Phase 2 intermediate reproduction remains an unresolved cross\-environment discrepancy – it differs from both the original run and the Phase 3 sweep, and the exact cause is unknown\. The 10M asymmetry is confirmed\. The 124M result remains partially unresolved: seeds 1337/42/123 gave 4\.899, 7\.586, 5\.967 \(mean 6\.151, std 1\.105\)\. A controlled multi\-seed 124M sweep has not been run and is the most direct next experiment\. Additional limitations: training budgets \(3K steps at 10M, 20K at 124M\) are below full convergence; Pre\-LN GPT\-style transformers only \(Post\-LN, RMSNorm, SwiGLU, RoPE untested\); the local\-mixer falsification test \(Figure[2](https://arxiv.org/html/2608.14689#S5.F2)\) was run under Phase 2 conditions and must be rerun under Phase 3 conditions to be interpretable\. He & Hofmann\[[8](https://arxiv.org/html/2608.14689#bib.bib8)\]study the same question – what block components can be removed without loss of training speed – in decoder\-only and BERT encoder\-only architectures\. Their findings partially corroborate this paper’s results, with an important difference in scope\. On attention\-skip removal \(their Section 4\.1\): removing the attention skip while keeping the MLP skip is, by this paper’s definitions in Section 3\.4, the FFNOnly configuration\. He & Hofmann confirm that naively removing the attention skip from a standard Pre\-LN block causes rank collapse and harms trainability – directly consistent with this paper’s FFNOnly collapse finding\. They go further by showing that with Shaped Attention \(initializing the attention matrix with a dominant identity component\), full training speed is recoverable even without the attention skip; this paper does not apply such modifications and does not study that recovery path\. On MLP\-skip removal \(their Figure 25\): removing the MLP skip while keeping the attention skip is this paper’s AttnOnly configuration\. He & Hofmann find that MLP\-skip removal without modification causes “significant losses of training speed” across tested activations but does not produce catastrophic, irreversible collapse – consistent with this paper’s AttnOnly finding, and a second independent corroboration of the asymmetry\. The key distinction between this paper and He & Hofmann \(2024\): this paper measures what happens when residual skips are removed from a standard Pre\-LN block with no compensating modifications; He & Hofmann measure whether, with architectural modifications, such removal can be made harmless\. Both are valid questions addressing different aspects of the same structural property\.

## 9Conclusion

I set out to characterize what happens when transformer residual connections are selectively removed\. I began with the observation that all partial configurations collapse under nanoGPT\. I challenged that observation, caught a confound in my own experiment, corrected it, and observed a striking asymmetry: removing the attention skip connection caused complete, deterministic collapse \(FFNOnly\), which reproduced closely across every seed and every environment tested \(std = 0\.0009 at 10M, gap from No Residual = 0\.023 at 124M\); removing the FFN skip connection was originally recorded converging well short of that collapse floor \(AttnOnly, 1\.600±\\pm0\.015 at 10M\)\. That second half of the asymmetry initially did not hold under an intermediate reproduction attempt on different hardware \(mean 2\.591±\\pm0\.544, one seed reaching the floor\)\. A subsequent controlled 8\-seed sweep under forced determinism on a clean A100 instance resolved the question: AttnOnly mean 1\.580±\\pm0\.003 across 8 independent seeds, gap from FFNOnly floor of 1\.770 val loss units, no seed approaching collapse\. The intermediate failure was an unresolved cross\-environment discrepancy; the original observation was accurate\. At 124M, three AttnOnly seeds give mean 6\.151, std 1\.105 – the pattern is suggestive but not yet characterized by a controlled multi\-seed sweep, and this remains the most direct next experiment\.

The asymmetry I originally set out to report is confirmed at 10M: FFNOnly’s collapse is robust and environment\-independent; AttnOnly’s recovery is real, seed\-stable, and reproducible under controlled conditions\. The mechanism – cross\-position routing as a substitute identity path – remains a working hypothesis that generated a falsifiable prediction \(Section 7\.4\) that itself could not be conclusively tested under the anomalous intermediate environment and should be rerun under the clean sweep conditions\. At 124M, the pattern remains suggestive but not yet established because AttnOnly shows large seed variance and lacks a controlled multi\-seed sweep across all configurations\. Beyond the specific finding: designing experiments to disprove your own hypothesis, following through when the experiment reveals a confound, retaining and honestly reporting an intermediate reproduction failure, and running the additional sweep that resolves it – rather than quietly omitting any of these steps – is the actual research contribution of this paper, independent of where the 124M question eventually lands\.

## Code Availability

## References

- \[1\]He, K\., Zhang, X\., Ren, S\., & Sun, J\. \(2016\)\. Deep residual learning for image recognition\.*CVPR 2016*, pp\. 770–778\.
- \[2\]Vaswani, A\., Shazeer, N\., Parmar, N\., et al\. \(2017\)\. Attention is all you need\.*NeurIPS 2017*\.
- \[3\]Radford, A\., Wu, J\., Child, R\., et al\. \(2019\)\. Language models are unsupervised multitask learners\. OpenAI Blog\.
- \[4\]Elhage, N\., Nanda, N\., Olsson, C\., et al\. \(2021\)\. A mathematical framework for transformer circuits\. Transformer Circuits Thread\.
- \[5\]Huang, X\., Dhariwal, P\., Song, J\., & Ermon, S\. \(2020\)\. Improving deep transformer with depth\-scaled initialization and merged attention\.*EMNLP 2020*\.
- \[6\]Gokaslan, A\. & Cohen, V\. \(2019\)\. OpenWebText Corpus\.[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)\.
- \[7\]Karpathy, A\. \(2022\)\. nanoGPT\.[https://github\.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)\.
- \[8\]He, B\. & Hofmann, T\. \(2024\)\. Simplifying Transformer Blocks\. International Conference on Learning Representations \(ICLR 2024\)\. arXiv:2311\.01906\.