Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

# What Transfers Between Transformer Sizes — and What Doesn’t
Source: [https://arxiv.org/html/2608.02829](https://arxiv.org/html/2608.02829)
## Wiring Beats Blending: What Transfers Between Transformer Sizes — and What Doesn’t

###### Abstract

Model families train every size from scratch\. Can a pretrained large model be converted into a smaller sibling? We characterize the 1\.4B→\\to410M conversion in the Pythia family end\-to\-end: \(i\) representations align strongly across sizes \(ridgeR2=0\.84R^\{2\}=0\.84\) while parameters align weakly; \(ii\) dense weight projection is functionally destructive — provably not an assembly artifact — because basis mixing breaks rotary, per\-head, GELU, and LayerNorm structure; \(iii\) after the best\-fit linear operator, weight residuals are statistically indistinguishable from noise under shuffle controls; \(iv\) conversion value therefore lives in*initialization*\. In matched\-budget continued pre\-training we decompose conversion into two independent levers — least\-squarescompensation\(function: best zero\-shot\) and variance\-preservingrescale\(dynamics: best endpoints\)\. Compensation is a*token\-efficient, low\-budget*win rather than a universal one: at 30M tokens it beats the strongest subcloning variant on both a width\-reduced pair \(84\.0±1\.884\.0\\pm 1\.8vs\.89\.7±3\.789\.7\\pm 3\.7, 3/3 seeds\) and a held\-out depth\-reduced pair \(109\.3109\.3vs\.117\.9117\.9, 3/3 seeds\), reaching a given quality with fewer tokens; at a33×33\\timeslarger budget the two*converge to parity*\(40\.040\.0vs\.40\.040\.0\), both far ahead of from\-scratch, which transfer initialization always beats — by up to18×18\\timesat low budget, the margin narrowing at convergence and at the largest scale\. We further*map the method’s boundary*: at∼\\sim5×\\timesthe donor scale \(6\.9B→\\to1\.4B\) stacking both levers over\-corrects, which we trace to ill\-conditioning of the compensation solve at large width, pointing to dimension\-aware regularization as the fix\. Code, checkpoints, and the frozen evaluation corpus are released\.

## 1Introduction

Transformer families ship as ladders of discrete sizes — Pythia alone spans 70M to 12B parameters\(Biderman et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib3)\)— because different deployment budgets need different models\. Under current practice every rung is bought separately: each size is a full pretraining run with its own GPU and energy bill, even though the family members are trained on identical data with a shared tokenizer and architecture, differing only in width and depth\. The siblings are, by construction, solutions to the same problem at different capacities; training each from nothing treats them as unrelated\. This paper asks the question the ladder’s cost makes practical and its construction makes tractable:*what actually transfers between two sizes of the same pretrained model family — and through what operation can it be carried into the smaller shape?*

We answer end\-to\-end on the Pythia suite, where identical data and tokenizer isolate size as the only variable \(§[3](https://arxiv.org/html/2608.02829#S3)\), in four steps\. First, we locate the cross\-size relation \(§[4](https://arxiv.org/html/2608.02829#S4)\): the smaller model’s activations are largely a linear image of the larger’s \(held\-out ridgeR2=0\.84R^\{2\}=0\.84\), its weights are not\. Second, we diagnose why densely projecting the large weights into the small shape is functionally destructive \(§[5](https://arxiv.org/html/2608.02829#S5)\): a bit\-exact control rules out assembly, leaving the projection mathematics, whose basis mixing breaks rotary, per\-head, GELU, and LayerNorm structure\. Third, what the best\-fit linear operator misses is indistinguishable from noise under shuffle controls \(§[6](https://arxiv.org/html/2608.02829#S6)\), so no zero\-shot fix exists\. Conversion value must therefore live in*initialization*, and the fourth step is a matched\-budget recovery race \(§[7](https://arxiv.org/html/2608.02829#S7)\) that decomposes conversion into two independent levers over structure\-respecting selection — closed\-form least\-squares*compensation*\(repairs what the network computes\) and variance\-preserving*rescale*\(repairs the scale the optimizer sees\) — which, acting on disjoint weight paths, stack\.

Our contributions are:

- •A controlled characterization of what transfers across sizes\.Within a family trained on identical data, representations align strongly under a linear map \(held\-out ridgeR2=0\.84R^\{2\}=0\.84\) while parameters align weakly \(R2=0\.25R^\{2\}=0\.25–0\.390\.39in the cleanest, embedding case\) — the gap the rest of the paper pivots on \(§[4](https://arxiv.org/html/2608.02829#S4)\)\.
- •A mechanically verified diagnosis of why dense projection fails\.Rebuilding the donor from its own extracted weights reproduces its logits bit\-exactly, so the projected model’s collapse is attributable to basis mixing alone: dense maps break rotary pairing, per\-head attention, elementwise GELU, and per\-feature LayerNorm — LayerNorm the largest single factor \(§[5](https://arxiv.org/html/2608.02829#S5)\)\.
- •A null result under controls\.After the best\-fit linear operator, weight residuals carry no cross\-layer direction, no patch\-learnable signal, and no cross\-pair transfer \(spectral, consistency, learnability, transfer tests vs\. shuffle controls with bootstrap CIs,p≥0\.99p\\geq 0\.99\): no learned zero\-shot correction exists at this granularity \(§[6](https://arxiv.org/html/2608.02829#S6)\)\.
- •A two\-lever decomposition of conversion, and a method that stacks them\.At matched budget, least\-squares compensation is the*function*lever \(best zero\-shot\) and variance\-preserving rescale the*training\-dynamics*lever \(best endpoints\); their combination — Compensated Selection — dominates the strongest variant of weight subcloning\(Samragh et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib12)\)on both axes and in every seed \(84\.0±1\.884\.0\\pm 1\.8vs\.89\.7±3\.789\.7\\pm 3\.7final perplexity, 3/3 paired wins; zero\-shot18\.518\.5k vs\.61\.961\.9k\) and finishes 18×\\timesbetter than from scratch at 30M tokens; the ordering holds out\-of\-domain and at2×2\\timesthe training context, and the margin over from scratch \(∼\\sim13×\\times\) generalizes with three seeds to a held\-out depth\-dominated pair \(§[7](https://arxiv.org/html/2608.02829#S7)\)\.

## 2Related work

##### Downscaling pretrained models\.

The closest prior work is Weight Subcloning\(Samragh et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib12)\), which initializes a smaller transformer by ranking units of a larger pretrained one, copying selected rows and columns, and rescaling — establishing that selection\-based initialization accelerates training of the target size\. Sheared LLaMA\(Xia et al\.,[2024](https://arxiv.org/html/2608.02829#bib.bib16)\)learns structured pruning masks jointly with continued pre\-training at a fraction of from\-scratch compute; it optimizes the full pipeline, whereas we hold the training recipe fixed across arms and isolate what the*initialization*alone contributes\. Our closed\-form compensation belongs to the pruning\-with\-reconstruction lineage — adjusting surviving weights to absorb the function of removed ones, from Optimal Brain Surgeon\(Hassibi & Stork,[1993](https://arxiv.org/html/2608.02829#bib.bib6)\)to structured LLM pruning like LLM\-Pruner\(Ma et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib10)\)— applied once, in closed form, at conversion time\. Knowledge distillation\(Hinton et al\.,[2015](https://arxiv.org/html/2608.02829#bib.bib7)\)transfers function through a teacher’s outputs and requires a full training run; we study the complementary question of what transfers through*weights*at negligible cost, and the two are combinable\.

##### Growing pretrained models\.

The mirror direction — initializing a*larger*model from a smaller one — has a longer history: Net2Net\(Chen et al\.,[2016](https://arxiv.org/html/2608.02829#bib.bib5)\)introduced function\-preserving widening and deepening, bert2BERT\(Chen et al\.,[2022](https://arxiv.org/html/2608.02829#bib.bib4)\)adapted it to transformers, staged training\(Shen et al\.,[2022](https://arxiv.org/html/2608.02829#bib.bib13)\)formalized growth operators that preserve loss and dynamics, and LiGO\(Wang et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib15)\)learns a structured linear map from small weights to the large initialization\. That linear operators carry useful signal upward is consistent with our finding that, downward, even a fitted dense projection initializes far better than random; the asymmetry we document is that dense basis mixing destroys the rotary, per\-head, and normalization structure that selection preserves \(§[5](https://arxiv.org/html/2608.02829#S5)\)\.

##### Representation similarity and parameter symmetries\.

Our analyses use standard similarity tools: CKA\(Kornblith et al\.,[2019](https://arxiv.org/html/2608.02829#bib.bib8)\)and SVCCA\(Raghu et al\.,[2017](https://arxiv.org/html/2608.02829#bib.bib11)\)measure representation alignment, and model stitching\(Lenc & Vedaldi,[2015](https://arxiv.org/html/2608.02829#bib.bib9); Bansal et al\.,[2021](https://arxiv.org/html/2608.02829#bib.bib2)\)tests functional interchangeability through a trained adapter\. Work on permutation symmetries and model merging\(Ainsworth et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib1)\)shows that networks are naturally compared*modulo*the transformations under which the architecture is invariant\. Our selection\-versus\-blending result is an instance of the same principle: structured selection composes a permutation — an element of the architecture’s symmetry group, restricted to respect head boundaries and rotary frequency pairs\(Su et al\.,[2024](https://arxiv.org/html/2608.02829#bib.bib14)\)— with coordinate deletion, so every surviving unit keeps the exact nonlinear and positional semantics the architecture assigns it; dense projection mixes coordinates, exits the group, and is punished for it\.

##### Positioning\.

Each ingredient above exists in isolation: selection\-based initialization, least\-squares reconstruction, linear growth operators, representation similarity\. What is new is the*setting*— conversion between sizes of a single family trained on identical data, removing data and tokenizer confounds — together with rotary\-frequency\-matched selection and a controlled end\-to\-end characterization, from representation alignment through projection diagnosis and residual null tests to a matched\-budget race\. Because our subcloning baseline re\-implements the recipe ofSamragh et al\. \([2023](https://arxiv.org/html/2608.02829#bib.bib12)\), we disclose three fidelity differences and label it*subcloning\-style*: \(a\) we score attention heads and MLP units by weight norms and residual lanes by activation variance, where they score by activation magnitudes; \(b\) when depth must shrink we remove blocks at even stride rather than from the middle; and \(c\) we evaluate theird/d′\\sqrt\{d/d^\{\\prime\}\}weight rescale directly rather than adopting it wholesale \(§[7](https://arxiv.org/html/2608.02829#S7)\) — applied to every matrix it collapses zero\-shot quality because LayerNorm already renormalizes most read paths, while its variance\-preserving motivation holds exactly on the projections no norm protects, the same two families our compensation re\-fits\.

## 3Experimental setup

##### Model family and conversion pairs\.

We study the Pythia suite\(Biderman et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib3)\): GPT\-NeoX models trained on identical data \(the Pile\) with a shared tokenizer, isolating*size*as the only variable between family members\. Our primary pair converts1\.4B→\\,\\to\\,410M\(24 layers and 16 heads in both; only widths shrink: residual2048→10242048\\\!\\to\\\!1024, head dim128→64128\\\!\\to\\\!64, MLP8192→40968192\\\!\\to\\\!4096\), so layer correspondence is one\-to\-one and the analysis isolates width\. A held\-out pair,410M→\\,\\to\\,160M, exercises the other axes: depth halves \(24→1224\\\!\\to\\\!12blocks\), the head*count*drops \(16→1216\\\!\\to\\\!12; head dim unchanged\), and width cuts are mild \(∼\\sim25%\)\. A third pair,6\.9B→\\,\\to\\,1\.4B, tests scale \(all three axes reduced; §[7](https://arxiv.org/html/2608.02829#S7)\)\.

##### Frozen evaluation corpus\.

All representation analyses probe every model with the*same*inputs: 10,000 sequences of 128 tokens from the Pile \(pile\-uncopyrighted\), tokenized once and frozen for the project\. The first 500 sequences \(all token positions\) form the similarity subset \(§[4](https://arxiv.org/html/2608.02829#S4)\); the rest contribute mean\-pooled and last\-token activations\. Continued pre\-training streams the same corpus but skips the frozen\-set region \(contamination guard\), and each seed trains on a disjoint stream offset\.

##### Behavioral evaluation\.

Language\-modeling quality is strided sliding\-window perplexity on WikiText\-103 validation \(every token scored once\)\. Hardening evaluations add C4 perplexity \(out\-of\-domain\), perplexity at2×2\\timesthe training context \(2048\), and zero\-shot accuracy on LAMBADA, ARC\-Easy, HellaSwag, and PIQA vialm\-eval\. Reference models anchor every table; anchor scores match published Pythia numbers\. All forward passes are greedy and seeded, bit\-identical on repetition; weight extraction is validated by exact reconstruction \(head\-aware QKV split/re\-fuse to bit\-identical equality; a model rebuilt from its own extracted weights reproduces the original logits with zero difference, §[5](https://arxiv.org/html/2608.02829#S5)\), so downstream quality changes are attributable to conversion mathematics, not assembly\.

##### Training protocol\.

Every conversion arm within an experiment shares*identical*data order, optimizer, cosine schedule, and token budget, so within an experiment the only difference across arms is the starting weights; initialization construction is negligible against any training budget\. Full hyperparameters, budgets, and hardware are in Appendix[C](https://arxiv.org/html/2608.02829#A3)\.

## 4Representations align, parameters do not

A size conversion presupposes that the two family members are related; the question is*where*the relation lives\. On the primary width\-only pair \(1\.4B→\\,\\to\\,410M, same depth and head count; §[3](https://arxiv.org/html/2608.02829#S3)\) we measure the relation in the representations both models compute and in the parameters they store, and find it in only one\.

##### Layer correspondence saturates under CKA\.

Linear CKA between all24×2424\\times 24layer pairs \(similarity subset, all token positions; §[3](https://arxiv.org/html/2608.02829#S3)\) gives a diagonal mean of0\.8830\.883, yet only12\.5%12\.5\\%of layers place their single best match on the diagonal\. The heatmap is one large saturated block \(App\. Fig\.[2](https://arxiv.org/html/2608.02829#A1.F2), left\) — layers 0–2 distinct, the middle band \(layers∼\\sim4–22\) mutually≈1\.0\\approx 1\.0, layer 23 distinct — so linear CKA confirms only*coarse*early/mid/late correspondence \(saturation in the middle band is a known limitation of the metric\)\. This motivates a directional map: how much of the small model’s representation is a linear image of the large model’s?

##### Activations are largely a linear image across widths\.

Per layer we fit a large→\\,\\to\\,small map on pooled activations \(8k train / 2k held\-out\) and score held\-outR2R^\{2\}under two operators:*Procrustes*\(rotation plus one global scale\) and*ridge*\(a full linear map2048→10242048\\\!\\to\\\!1024\)\. Ridge reaches a mean held\-outR2=0\.844R^\{2\}=0\.844\(App\. Table[3](https://arxiv.org/html/2608.02829#A1.T3)\): the 410M pooled representation is largely a linear projection of the 1\.4B’s, capturing84%84\\%of held\-out variance\. Procrustes already reaches0\.7170\.717, so most of the relationship is a change of basis and the∼\\sim0\.13 gap is genuine non\-orthogonal reshaping\. Both maps dip together at layers 3–5, the layers CKA flagged as distinct \(App\. Fig\.[2](https://arxiv.org/html/2608.02829#A1.F2), right\)\.

##### Raw parameters do not\.

The token embeddings are the one weight matrix where the two sizes share an axis \(the vocabulary\) and differ only in width, so a single map applies \(rows split80/2080/20\)\. Even in this cleanest case a linear map explains only0\.250\.25–0\.390\.39of weight variance \(App\. Table[3](https://arxiv.org/html/2608.02829#A1.T3)\): ridgeR2=0\.248R^\{2\}=0\.248on the input embedding and0\.3860\.386on the output embedding, against0\.8440\.844for activations\.

##### The representation–parameter gap\.

Two independently trained family members reach closely related*representations*while their*parameters*are only loosely a linear map of one another\. This gap is the paper’s pivot: if representations align but weights do not, projecting the large weights into the small shape \(§[5](https://arxiv.org/html/2608.02829#S5)\) must leave a substantial residual, and whether that residual is learnable or noise \(§[6](https://arxiv.org/html/2608.02829#S6)\) decides where conversion value comes from\.

## 5Why dense projection fails

If representations align across sizes, the natural attempt is to project the large model’s weights into the small model’s shape and read off a working small model\. We formulate this projection from only the size relationship \(never the target’s weights\), measure the residual, run the assembled model, and diagnose why it breaks\. The verdict: dense blending destroys the network’s structured computation, and a bit\-exact control places the destruction in the projection math, not the assembly\.

##### Target\-free projection\.

Each weight is mapped by a pair of structured linear operators,

W^=Pout​W​Pin⊤,P∙∈\{Pres,PQ​K,PV​O,PMLP\},\\hat\{W\}\\;=\\;P\_\{\\text\{out\}\}\\,W\\,P\_\{\\text\{in\}\}^\{\\top\},\\qquad P\_\{\\bullet\}\\in\\\{\\,P\_\{\\text\{res\}\},\\,P\_\{QK\},\\,P\_\{VO\},\\,P\_\{\\text\{MLP\}\}\\,\\\},\(1\)whose bases are built*only*from the large model and the size relationship, never from the 410M target\. The residual axis uses a global activation\-PCA basisPresP\_\{\\text\{res\}\}; internal axes use SVD bases*shared between interacting weights*so composed operations stay consistent —Q,KQ,Kshare one,V,OV,Oshare one, andMLP\_UP/MLP\_DOWNshare a joint basis from the column stack\[UP,DOWN⊤\]\[\\mathrm\{UP\},\\,\\mathrm\{DOWN\}^\{\\top\}\]\. Sharing is not cosmetic: a per\-weight basis for the MLP hidden axis cannot span the target, since one8192×20488192\\times 2048matrix has rank≤2048<4096\\leq 2048<4096, so the interacting pair must be factorized together\.

##### The projected weights miss the target solution\.

App\. Table[5](https://arxiv.org/html/2608.02829#A1.T5)reports the relative Frobenius error ofW^\\hat\{W\}against the actual 410M weights, mean over 24 layers, for the target\-free projection of Eq\.[1](https://arxiv.org/html/2608.02829#S5.E1)and — as a diagnostic upper bound — a shared operator\(A,B\)\(A,B\)*fitted*with access to the target\. Every target\-free error exceeds11\(worse than predicting zero\): the projected weights share almost nothing with the 410M’s particular solution, confirming the activation\-versus\-weight gap of §[4](https://arxiv.org/html/2608.02829#S4)at every layer\. Even the fitted operator only reaches0\.660\.66–0\.800\.80, so at best∼\\sim30% of weight variance is linearly explainable across sizes with one shared map\.

##### Behavioral anchors\.

We assemble a 410M\-shaped model from the projected 1\.4B weights and score strided WikiText\-103 perplexity against reference and anchor models \(App\. Table[5](https://arxiv.org/html/2608.02829#A1.T5)\)\. Greedy generation is deterministic but word salad, and the acceptance bar — beat random init — is not met zero\-shot\. The failure’s*shape*is instructive: random init produces near\-uniform logits \(perplexity on the order of vocabulary size\), whereas the projected model is confidently wrong, far worse than uniform\. This is not a plumbing bug: rebuilding the 1\.4B from its own extracted weights with*no*projection reproduces the real model’s logits withmax⁡\|diff\|=0\\max\|\\text\{diff\}\|=0\(bit\-exact; previewed in §[3](https://arxiv.org/html/2608.02829#S3)\), so the entire quality loss in App\. Table[5](https://arxiv.org/html/2608.02829#A1.T5)is attributable to Eq\.[1](https://arxiv.org/html/2608.02829#S5.E1), not to extraction, head\-aware QKV split/fuse, embeddings, or LayerNorm plumbing\.

##### Structure mixing breaks the computation\.

A dense basis change on any internal axis silently changes what the block*computes*, because every structured or nonlinear operation is tied to specific coordinates\.*Rotary embeddings*act on fixed per\-head dimension pairs, so head\-space mixing distorts positions\.*Attention is per\-head*, yet our global2048→10242048\\\!\\to\\\!1024head\-space basis mixes dimensions*across*heads\.*GELU is elementwise*in the MLP hidden space, soGELU​\(B​x\)≠B​GELU​\(x\)\\mathrm\{GELU\}\(Bx\)\\neq B\\,\\mathrm\{GELU\}\(x\)\.*LayerNorm gains and biases are per\-feature*, so a rotated stream no longer matches any diagonal gain — which is why the projected\-LN variant explodes to101310^\{13\}\. Function\-preserving zero\-shot conversion therefore needs structure\-respecting*selection*— keep or drop whole heads, hidden units, and features \(the architecture’s own symmetry group\) — not dense blending; dense projections are useful only as*initializations*for brief fine\-tuning \(§[7](https://arxiv.org/html/2608.02829#S7)\)\.

## 6Post\-operator residuals are noise

Section[5](https://arxiv.org/html/2608.02829#S5)shows that even the strongest shared linear operator explains at best∼\\sim30% of weight variance \(relative error0\.660\.66–0\.800\.80\), leaving the majority as a residual\. Is that residual a structured, learnable correction — or noise? We define the residual per weight as

Δ​\(l,type\)=W410​M−A​W1\.4​B​B⊤,\\Delta\(l,\\text\{type\}\)\\;=\\;W\_\{410\\text\{M\}\}\\;\-\\;A\\,W\_\{1\.4\\text\{B\}\}\\,B^\{\\top\},\(2\)using the shared fitted operator\(A,B\)\(A,B\)per type — the strongest\-alignment condition of §[5](https://arxiv.org/html/2608.02829#S5), so whatever remains is what no single linear size\-map can explain and the verdict is conservative: if any signal survives the best linear map, these tests should find it\. Four tests probeΔ\\Delta, each against a matched control\.

##### Test 1 — spectral \(effective rank\)\.

App\. Table[6](https://arxiv.org/html/2608.02829#A1.T6)compares the effective rank ofΔ\\Deltaagainst a per\-type shuffled control and a shape/scale\-matched Gaussian control\.Δ\\Deltasits2\.82\.8–6\.4%6\.4\\%below both controls in every type: a*faint*spectral concentration, so the residual is not perfectly isotropic noise \(App\. Fig\.[3](https://arxiv.org/html/2608.02829#A1.F3)\)\.

##### Test 2 — cross\-layer consistency\.

The mean pairwise cosine of vectorizedΔ\\Deltaacross layers is≈±0\.0002\\approx\\pm 0\.0002in every type, identical to the shuffle control: no direction is shared across layers, so no single correction is uniformly missing\.

##### Test 3 — learnability \(decisive\)\.

A patch predictorR​\(W^​patch\+type/layer/position\)→Δ​patchR\(\\hat\{W\}\\text\{ patch\}\+\\text\{type/layer/position\}\)\\to\\Delta\\text\{ patch\}is trained on 18 layers, scored on 6 stratified held\-out layers as error reduction over predicting zero; the control retrains onΔ\\Deltashuffled across layers, with a 1000\-sample bootstrap on the real−\-control gap\. Neither a linear nor an MLP\-512 predictor beats zero \(reductions−0\.0004\-0\.0004/−0\.0003\-0\.0003\), and real and control are indistinguishable \(gap CI≈95\[−0\.0000,−0\.0000\]\{\}\_\{95\}\\approx\[\-0\.0000,\-0\.0000\],p≥0\.99p\\geq 0\.99; Table[1](https://arxiv.org/html/2608.02829#S6.T1)\)\. No patch\-learnable signal\.

##### Test 4 — transfer\.

On the held\-out 410M→\\,\\to\\,160M pair the shared operators fit at relative error0\.560\.56–0\.740\.74\(smaller width gap\)\. A predictor trained on all primary\-pair layers, evaluated on the held\-out pair, reduces error by−0\.0004\-0\.0004/−0\.0003\-0\.0003, again indistinguishable from its shuffle control \(p=1\.00p=1\.00; Table[1](https://arxiv.org/html/2608.02829#S6.T1)\)\. Nothing transfers, consistent with Test 3\.

##### Reconciling the faint spectral signal\.

Tests 1 and 3 are not in tension: the2\.82\.8–6\.4%6\.4\\%spectral deficit is a*generic*statistical trace, not a predictor\-exploitable correspondence, and a behavioral cross\-check confirms the destruction lives in the dense projection, not the residual \(Appendix[B\.2](https://arxiv.org/html/2608.02829#A2.SS2)\)\.

##### Handoff to §[7](https://arxiv.org/html/2608.02829#S7)\.

Within a family the sizes share representation geometry but not parameter solutions, and what the best linear size\-map misses behaves as noise \(no cross\-layer direction, no patch\-learnable mapping, no cross\-pair transfer; only a faint generic spectral fingerprint\), so conversion value must come from*initialization*plus brief fine\-tuning — the matched\-budget recovery race of §[7](https://arxiv.org/html/2608.02829#S7)\.

Table 1:Tests 3–4 \(learnability and transfer\)\. Patch\-predictor error reduction over predicting zero vs\. a label\-shuffled control; 1000\-sample bootstrap CI on the real−\-control gap\. No predictor beats zero or separates from its control\.CIs are reported for the in\-pair held\-out test only; the transfer test reportspp\(M4\)\. Reductions are negative because no predictor improves on predicting zero\.

## 7Matched\-budget conversion: two independent levers

The characterization so far is diagnostic: representations align \(§[4](https://arxiv.org/html/2608.02829#S4)\), dense projection is destructive \(§[5](https://arxiv.org/html/2608.02829#S5)\), and its post\-operator residuals are noise \(§[6](https://arxiv.org/html/2608.02829#S6)\)\. The constructive claim follows\. If conversion value lives in*initialization*, the decisive test is a matched\-budget recovery race: every candidate gets the same target shape, data, schedule, and token budget, and continued pre\-training reveals which starting weights convert fastest\. The init decomposes into two*independent*levers — least\-squarescompensation, which fixes what the init*computes*\(best zero\-shot\), and variance\-preservingrescale, which fixes the*scale the optimizer sees*\(best endpoints\)\. At a fixed low budget stacking both dominates the strongest single\-lever variant on both axes and in every seed — a token\-efficiency edge that closes to parity at convergence, and inverts at the largest scale \(§[7\.5](https://arxiv.org/html/2608.02829#S7.SS5), §[7\.6](https://arxiv.org/html/2608.02829#S7.SS6)\)\.

##### Protocol\.

All arms share the target shape \(410M\), data order, optimizer, schedule, and token budget; only the starting weights differ \(§[3](https://arxiv.org/html/2608.02829#S3)\)\. Crucially,*every*init is obtainable without the trained target — selection scores, compensation moments, and rescale factors are read off the donor 1\.4B and the frozen corpus, so no arm leaks target weights\. Zero\-shot columns are quick\-eval perplexity \(50k WikiText tokens\) att=0t\{=\}0; final columns are full strided WikiText\-103 perplexity at 30M tokens\. Unless a±\\pmappears, a cell is single\-seed \(footnoted\)\.

### 7\.1The ablation ladder

Table[2](https://arxiv.org/html/2608.02829#S7.T2)walks a seven\-arm ladder from an untrained model to the full method\. Top to bottom it is monotone in the endpoint — each rung improves the final perplexity — but the zero\-shot column is emphatically*not*monotone, and that dissonance is the section’s main result\.

Table 2:Seven\-arm ablation ladder, primary pair \(1\.4B→\\to410M\), 30M tokens\. Zero\-shot is quick\-eval perplexity att=0t\{=\}0; final is full WikiText\-103 perplexity at budget\. Rows marked “\+\+” build on the same selection; best zero\-shot ishybrid, best endpointhybrid\_rs\. All cells single\-seed \(top two arms re\-run across seeds in App\. Table[8](https://arxiv.org/html/2608.02829#A1.T8)\)\.##### Both transfer inits beat from scratch\.

![Refer to caption](https://arxiv.org/html/2608.02829v1/figures/m5_curves.png)Figure 1:Recovery ladder \(1\.4B→\\to410M\)\.WikiText\-103 perplexity \(log scale\) vs\. tokens, seven arms of Table[2](https://arxiv.org/html/2608.02829#S7.T2)\(dashed==real Pythia\-410M,≈15\.6\\approx 15\.6; 30M tokens==∼\\sim0\.01% of pretraining, so this measures*relative*recovery speed\)\.Even bare structured selection \(subclone, 355\.8\) recovers4\.3×4\.3\\timesfaster than random \(1,519\.0\) at matched compute, and dense projection \(1,054\.5\) — despite its scrambled wiring — still beats from scratch1\.4×1\.4\\times\. Selection preserves valid wiring \(rotary pairing, per\-head attention, elementwise GELU, per\-feature LayerNorm\) and pays off throughout training:subcloneseparates fromprojectionimmediately and the gap widens monotonically \(Fig\.[1](https://arxiv.org/html/2608.02829#S7.F1)\)\. The full ladder closes at1,519\.0/83\.0=18×1\{,\}519\.0/83\.0=18\\timesbetter than from scratch\.

### 7\.2Two independent levers

Reading the ladder by*construction*exposes two orthogonal axes over the selection base\.

##### Compensation is the function lever\.

The hybrid re\-fits the two purely\-linear, LayerNorm\-free read\-out paths \(OOand the MLP down\-projection\) by closed\-form ridge least squares \(§[7\.4](https://arxiv.org/html/2608.02829#S7.SS4)\), so surviving units absorb the dropped units’ correlated contribution\. This strictly improves*both*axes: zero\-shot11,509→9,66411\{,\}509\\to 9\{,\}664\(best in the table\) and endpoint355\.8→114\.1355\.8\\to 114\.1\(3\.1×3\.1\\timesoversubclone\)\. A better function is better everywhere\.

##### Rescale is the training\-dynamics lever\.

The reference recipe’s variance\-preserving factordin/din′\\sqrt\{d\_\{\\text\{in\}\}/d\_\{\\text\{in\}\}^\{\\prime\}\}\(Samragh et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib12)\)does the opposite\. On the LayerNorm\-free read\-out paths only \(subclone\_iso\) it*worsens*zero\-shot11,509→26,66911\{,\}509\\to 26\{,\}669yet*improves*the endpoint355\.8→208\.2355\.8\\to 208\.2; on every input\-cut path \(subclone\_rs\) it collapses zero\-shot to61,91261\{,\}912—5\.4×5\.4\\timesworse than bare selection — while driving the endpoint to86\.486\.4, past even the compensated hybrid\. It does not fix the function — a downstream LayerNorm renormalizes each rescaled read path, so the scalar cancels in the forward pass and merely perturbs zero\-shot alignment — but it fixes the variance the optimizer starts from, and along this axis the trade is monotone: more rescale, worse start, better endpoint\.

##### The levers stack\.

The two levers act on disjoint quantities — compensation on the function of the two LN\-free read\-out paths, rescale on the scale of the LN\-fronted read\-in paths \(Q,K,VQ,K,V, MLP up\-projection, output embedding\) — so they compose\.hybrid\_rsapplies compensation where it is ridge\-optimal and the reference\-recipe scale where compensation cannot reach, dominating the strongest single\-lever arm \(subclone\_rs\) on*both*axes: zero\-shot18,459<61,91218\{,\}459<61\{,\}912and endpoint83\.0<86\.483\.0<86\.4\.

### 7\.3Seeded verdict

The two rescale leaders are close on the single seed \(86\.486\.4vs\.83\.083\.0\), so we re\-run both across three data\-draw seeds\. The separation is clean:hybrid\_rs\(84\.0±1\.884\.0\\pm 1\.8final WikiText\-103 perplexity\) wins every paired comparison againstsubclone\_rs\(89\.7±3\.789\.7\\pm 3\.7\) \(3/33/3\), and its*worst*seed \(86\.086\.0\) still beatssubclone\_rs’s*best*seed \(86\.486\.4\) — the seed distributions do not overlap \(per\-seed detail in App\. Table[8](https://arxiv.org/html/2608.02829#A1.T8)\)\.

This upset \(subclone\_rsat86\.486\.4beating the compensation\-only hybrid at114\.1114\.1\) corrects our own earlier reading that compensation subsumes rescale; the reference recipe is vindicated exactly on the LN\-fronted read\-in paths compensation leaves alone, so the principled method is the stack, not either lever \(Appendix[B\.1](https://arxiv.org/html/2608.02829#A2.SS1)\)\.

### 7\.4Compensated Selection

Algorithm[1](https://arxiv.org/html/2608.02829#alg1)gives the full construction \(per\-coordinate scoring in the listing\)\.*Selection*keeps whole coordinates so each subspace’s symmetry survives: with rotary half\-width16→816\\to 8the small frequency ladder is exactly every other large frequency, so we keep rotary pairs at stride22to preserve the\(cos,sin\)\(\\cos,\\sin\)pairing\(Su et al\.,[2024](https://arxiv.org/html/2608.02829#bib.bib14)\)\. When the head*count*shrinks \(the held\-out pair\) selection is promoted to whole heads shared byQ,K,V,OQ,K,V,O; when depth shrinks, evenly strided blocks are kept with their norms and biases\.

*Compensation*is the closed\-form optimum for a pruned linear map\. For a mapMMreading a signal with measured second momentΣ\\Sigma, the ridge least\-squares substitute on the kept columns𝒦\\mathcal\{K\}is

M′=arg⁡minM′⁡𝔼​∥M​x−M′​x𝒦∥2=M​Σ:,𝒦​\(Σ𝒦,𝒦\+λ​I\)−1,M^\{\\prime\}\\;=\\;\\arg\\min\_\{M^\{\\prime\}\}\\;\\mathbb\{E\}\\,\\big\\lVert Mx\-M^\{\\prime\}x\_\{\\mathcal\{K\}\}\\big\\rVert^\{2\}\\;=\\;M\\,\\Sigma\_\{:,\\mathcal\{K\}\}\\,\\big\(\\Sigma\_\{\\mathcal\{K\},\\mathcal\{K\}\}\+\\lambda I\\big\)^\{\-1\},\(3\)with plain subcloning as the special caseM′=M:,𝒦M^\{\\prime\}=M\_\{:,\\mathcal\{K\}\}\. We apply it at the two spots where the reconstruction target is well\-defined —OOand the MLP down\-projection, both purely linear with no normalization between cut and read — and leave the LN\-fronted read\-in paths to the optional rescale lever\. Least\-squares reconstruction after pruning is classical\(Hassibi & Stork,[1993](https://arxiv.org/html/2608.02829#bib.bib6); Ma et al\.,[2023](https://arxiv.org/html/2608.02829#bib.bib10)\); new here are the setting \(family*size conversion*\), the rotary\-frequency\-matched selection, and the two\-lever decomposition\.

Algorithm 1Compensated Selection \(hybrid\_rs\)0:donor weights

WLW\_\{L\}; frozen\-corpus activation variances and second moments

Σ\\Sigma; target shape \(residual

d′d^\{\\prime\}, head dim, MLP width, block map\)

0:installable small\-model weights \(no target\-model access\)

1:

ℛ←\\mathcal\{R\}\\leftarrowtop\-

d′d^\{\\prime\}residual features by mean\-over\-layer activation variance \{one global basis\}

2:foreach kept block

bbdo

3:

𝒦q​k←\\mathcal\{K\}\_\{qk\}\\leftarrowfrequency\-matched rotary pairs \(stride

22\)

∪\\cuptop non\-rotary dims by

∥q∥\+∥k∥\\lVert q\\rVert\{\+\}\\lVert k\\rVert
4:

𝒦v​o←\\mathcal\{K\}\_\{vo\}\\leftarrowtop head dims by

∥Vrow∥⋅∥Ocol∥\\lVert V\_\{\\text\{row\}\}\\rVert\\cdot\\lVert O\_\{\\text\{col\}\}\\rVert\{whole heads if head count shrinks\}

5:

𝒦m​l​p←\\mathcal\{K\}\_\{mlp\}\\leftarrowtop units by

∥uprow∥⋅∥downcol∥\\lVert\\text\{up\}\_\{\\text\{row\}\}\\rVert\\cdot\\lVert\\text\{down\}\_\{\\text\{col\}\}\\rVert
6:slice

WLW\_\{L\}block

bbto

\(ℛ,𝒦q​k,𝒦v​o,𝒦m​l​p\)\(\\mathcal\{R\},\\mathcal\{K\}\_\{qk\},\\mathcal\{K\}\_\{vo\},\\mathcal\{K\}\_\{mlp\}\); LayerNorms and biases ride along

7:

O′←LScomp​\(O,Σattn\-in,𝒦v​o\)O^\{\\prime\}\\leftarrow\\text\{LScomp\}\(O,\\,\\Sigma\_\{\\text\{attn\-in\}\},\\,\\mathcal\{K\}\_\{vo\}\);

MLP\_down′←LScomp​\(MLP\_down,Σmlp\-in,𝒦m​l​p\)\\text\{MLP\\\_down\}^\{\\prime\}\\leftarrow\\text\{LScomp\}\(\\text\{MLP\\\_down\},\\,\\Sigma\_\{\\text\{mlp\-in\}\},\\,\\mathcal\{K\}\_\{mlp\}\)\{Eq\.[3](https://arxiv.org/html/2608.02829#S7.E3), function lever\}

8:ifrescalethen

9:multiply LN\-fronted read\-in paths

Q,K,V,MLP\_upQ,K,V,\\text\{MLP\\\_up\}\(and output embedding\) by

din/din′\\sqrt\{d\_\{\\text\{in\}\}/d\_\{\\text\{in\}\}^\{\\prime\}\}\{dynamics lever\}

10:endif

11:endfor

12:returnassembled small model

### 7\.5Hardening

The headline is one metric at one budget on one pair; we stress it on all three\.

##### Every metric, same ordering\.

App\. Table[7](https://arxiv.org/html/2608.02829#A1.T7)adds C4 perplexity \(out\-of\-domain\), perplexity at2×2\\timesthe training context \(2048\), and four zero\-shot tasks\. The orderinghybrid<subclone<projection<random\\text\{hybrid\}<\\text\{subclone\}<\\text\{projection\}<\\text\{random\}holds on every metric\. Perplexity at 2048 tracks each model’s 1024 number, so rotary\-frequency\-matched selection extrapolates past its training length without penalty\. LAMBADA \(floor≈0\\approx 0\) is the sharpest ability gap \(hybrid14×14\\timessubclone\); the multiple\-choice tasks sit near guess floors, so 30M tokens buys language modeling, not task reasoning; anchor scores match published Pythia numbers\.

##### The lead persists at longer budget\.

Re\-running the two leaders at 100M tokens, hybrid finishes at62\.362\.3and subclone at103\.1103\.1; hybrid crosses subclone’s*100M*endpoint at∼\\sim38M tokens, a∼\\sim2\.6×2\.6\\timestoken\-efficiency gain\.111100M cells are single\-seed; cross\-horizon comparisons are schedule\-confounded \(finals compared only within a horizon\)\.

##### At convergence, the edge closes\.

Pushing the primary pair to 1B tokens \(33×33\\timesthe headline budget\), the two rescale leaders finish level —hybrid\_rs40\.040\.0vs\.subclone\_rs40\.040\.0\(an exact tie\) — both∼\\sim1\.4×1\.4\\timesahead of from scratch \(57\.457\.4\) and far nearer the real 410M \(≈15\.6\\approx 15\.6\) than at 30M\. Compensation’s advantage is thus a low\-budget, token\-efficiency effect: it reaches a given quality with fewer tokens but is redundant with rescale once the budget is large, the two levers improving the same underlying conditioning so that at convergence one suffices\.

##### A held\-out pair, and the method’s boundary\.

The 410M→\\to160M pair exercises axes the primary pair does not \(depth\-dominated: depth halves, head*count*16→1216\\to 12with whole\-head selection, mild width cuts; §[3](https://arxiv.org/html/2608.02829#S3)\)\. Across three seeds \(final WikiText\-103 perplexity at 30M tokens: hybrid113\.2±4\.5113\.2\\pm 4\.5, subclone118\.5±3\.3118\.5\\pm 3\.3, random1,505\.2±57\.91\{,\}505\.2\\pm 57\.9; per\-seed detail in App\. Table[9](https://arxiv.org/html/2608.02829#A1.T9)\), transfer beats from scratch∼\\sim13×13\\timeson every seed — the core claim generalizes, with error bars, to an untuned regime\. The full stack keeps its low\-budget edge here too:hybrid\_rs\(109\.3109\.3\) beatssubclone\_rs\(117\.9117\.9\) on all three seeds\. Compensation*alone*, though, is neutral — hybrid and subclone here*tie*\(overlapping±1​σ\\pm 1\\sigma\), unlike the3\.1×3\.1\\timescompensation gap on the width pair — compensation’s*width*\-repair edge is neutral when damage is depth\-dominated \(§[9](https://arxiv.org/html/2608.02829#S9)\), costing nothing here \(zero\-shot starts at hybrid≈\\approx30k vs\.9\.79\.7k on the width pair\)\. The tie is budget\-robust \(100M tokens, seed 0: hybrid77\.177\.1vs\. subclone74\.474\.4, a 3\.6% gap inside seed noise, leader flipped\) and holds on every other metric, with random far below and the real\-160M anchor matching published Pythia numbers \(single\-seed; App\.[A\.6](https://arxiv.org/html/2608.02829#A1.SS6.SSS0.Px1)\)\.

### 7\.6Scale, and the method’s boundary

The scale pair \(6\.9B→\\to1\.4B\) reduces all three axes at once at∼\\sim5×\\timesthe donor size \(30M tokens; App\. Table[10](https://arxiv.org/html/2608.02829#A1.T10)\)\. Here the stack*inverts*: rescale\-onlysubclone\_rsis best \(572572\), compensation\-onlyhybridnext \(776776\), and stacking both \(hybrid\_rs,1,2131\{,\}213\) is worst among transfer arms — though all still beat from scratch \(1,4131\{,\}413\), so the core claim survives its worst case\. Each lever alone is safe; only their combination over\-corrects, and only at this scale\. A*preliminary*control — converting a smaller \(1\.4B\) donor to targets at the*same*reduction ratios — did*not*reproduce the over\-correction, suggesting the cause is*absolute*scale rather than the reduction geometry: the compensation solve \(Eq\.[3](https://arxiv.org/html/2608.02829#S7.E3)\) inverts a far larger, worse\-conditioned matrix at81928192\-wide inputs than in the smaller cases \(≤4096\\leq 4096\), which its magnitude\-scaled ridge under\-regularizes\. A controlled study across donor scales, and the dimension\-aware fix, are left to future work \(§[9](https://arxiv.org/html/2608.02829#S9)\)\.

##### Compute accounting\.

Initialization construction is negligible and included in the accounting \(selection: norms plus top\-kk, seconds; compensation moments over 1,000 frozen\-corpus sequences,∼\\sim75 s; closed\-form solves, seconds; rescale, free\)\. Every arm then trains on the*same*30M tokens under the*same*schedule \(∼\\sim55 min each on a single GB10 at∼\\sim9\.4k tok/s\), so all standings \(Table[2](https://arxiv.org/html/2608.02829#S7.T2); App\. Table[8](https://arxiv.org/html/2608.02829#A1.T8)\) compare equal\-budget endpoints differing only in starting weights\.

## 8Conclusion

We asked what actually transfers when a pretrained transformer is converted into a smaller sibling, and answered it in a controlled, single\-family setting: representations align across sizes but parameters do not, dense weight projection is provably destructive because it breaks the architecture’s load\-bearing wiring, and after the best\-fit linear operator the weight residuals are indistinguishable from noise\. The value of a conversion therefore lives in*initialization*, not in any learnable zero\-shot weight correction\. Decomposing that initialization into two independent levers — least\-squares*compensation*\(what the network computes\) and variance\-preserving*rescale*\(the scale the optimizer sees\) — lets us state the practical rule precisely: compensation is a token\-efficient, low\-budget win that stacks with rescale to dominate subcloning at small budgets on both a width\- and a depth\-reduced pair, ties it once the budget is large enough to converge, and, stacked, over\-corrects only at the largest donor scale, where the compensation solve is ill\-conditioned\. For a practitioner spinning up a new size, the takeaway is concrete: initialize by structure\-respecting selection, compensate and rescale on disjoint paths for a strong low\-budget start, and fall back to the single robust rescale lever at large scale until a dimension\-aware regularizer closes that gap\.

## 9Limitations and future work

##### One family, one wiring\.

All experiments are within the Pythia suite \(GPT\-NeoX: LayerNorm, elementwise GELU, rotary attention\) — deliberate \(identical data and tokenizer isolate size\), but the two\-lever decomposition leans on those wiring details: which paths a norm fronts sets where each lever is well\-posed \(§[7\.4](https://arxiv.org/html/2608.02829#S7.SS4)\)\. Llama\-/Qwen\-class models change the load\-bearing parts \(RMSNorm, gated SwiGLU, grouped\-query attention\), so porting needs a per\-architecture safe\-wiring analysis; the characterization tools are architecture\-agnostic\.

##### Budgets, seeds, and open directions\.

Our recovery budgets \(30M primary, 100M persistence, 1B convergence\) span∼\\sim0\.01–0\.3% of Pythia’s 300B\-token pretraining; at 1B tokens converted models reach40\.040\.0\(vs\. the real 410M’s≈15\.6\\approx 15\.6\) — much closer than at 30M \(83\.083\.0\) but still short of full pretraining, so most claims concern*relative recovery at matched budget*rather than end quality\. Decisive comparisons carry three data\-draw seeds \(App\. Tables[8](https://arxiv.org/html/2608.02829#A1.T8)and[9](https://arxiv.org/html/2608.02829#A1.T9)\); remaining cells are single\-seed\. Two directions stay open\. First, compensation touches only the two LN\-free read\-out paths; on the LN\-fronted read\-in paths the same renormalization that collapses blanket\-rescale zero\-shot \(§[7\.2](https://arxiv.org/html/2608.02829#S7.SS2)\) makes the least\-squares target ill\-defined, so normalization\-aware compensation there is open\. Second, the held\-out pair marks a harder boundary \(§[7\.5](https://arxiv.org/html/2608.02829#S7.SS5)\): compensation repairs width cuts but not deleted blocks; reduction\-aware conversion and width\-vs\-depth budget allocation \(our boundary favors width\) are untried\.

##### Compensation conditioning at scale\.

The two\-lever stack, which dominates at low budget on both smaller pairs,*over\-corrects*on the 6\.9B→\\to1\.4B pair \(§[7\.6](https://arxiv.org/html/2608.02829#S7.SS6)\): the least\-squares compensation \(Eq\.[3](https://arxiv.org/html/2608.02829#S7.E3)\) inverts a second\-moment matrix whose conditioning worsens with the kept width, and its magnitude\-scaled ridge under\-regularizes the low\-variance directions at81928192\-wide inputs\. A preliminary small\-donor control at matched reduction ratios did not reproduce the effect, suggesting the cause is absolute scale rather than the reduction geometry\. The direct remedy — a spectrum\- or dimension\-aware ridge \(settingλ\\lambdafrom the eigenvalue spread rather than the mean diagonal, or clamping the compensated\-weight norm\) — is untested here and left to future work; until then, the robust single\-leversubclone\_rsis the safe default at large donor scale\.

#### Reproducibility statement

All experiments use the public Pythia suite and thepile\-uncopyrighted, WikiText\-103, and C4 corpora\. Every result comes from a seeded, deterministic pipeline: weight extraction is validated by exact \(bit\-identical\) reconstruction, forward passes are greedy and reproducible per device, and each reported comparison fixes data order, optimizer, schedule, and token budget across arms — varying only the initialization\. The frozen10,000×12810\{,\}000\\times 128evaluation corpus is built once from a recorded stream offset and contamination\-guarded against the continued\-pre\-training stream; decisive comparisons report three data\-draw seeds\. The full code, all configs and seeds, theuv\.lockenvironment pin, the frozen\-corpus recipe, and the released checkpoints will be made publicly available at[https://github\.com/rsdpyenugula/ScaleOp](https://github.com/rsdpyenugula/ScaleOp)upon publication\.

## References

- Ainsworth et al\. \(2023\)Samuel K\. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa\.Git Re\-Basin: Merging models modulo permutation symmetries\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Bansal et al\. \(2021\)Yamini Bansal, Preetum Nakkiran, and Boaz Barak\.Revisiting model stitching to compare neural representations\.In*Advances in Neural Information Processing Systems 34 \(NeurIPS\)*, 2021\.
- Biderman et al\. \(2023\)Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, pp\. 2397–2430\. PMLR, 2023\.
- Chen et al\. \(2022\)Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu\.bert2BERT: Towards reusable pretrained language models\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2022\.
- Chen et al\. \(2016\)Tianqi Chen, Ian Goodfellow, and Jonathon Shlens\.Net2Net: Accelerating learning via knowledge transfer\.In*International Conference on Learning Representations \(ICLR\)*, 2016\.
- Hassibi & Stork \(1993\)Babak Hassibi and David G\. Stork\.Second order derivatives for network pruning: Optimal brain surgeon\.In*Advances in Neural Information Processing Systems 5 \(NIPS\)*, pp\. 164–171, 1993\.
- Hinton et al\. \(2015\)Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- Kornblith et al\. \(2019\)Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton\.Similarity of neural network representations revisited\.In*Proceedings of the 36th International Conference on Machine Learning \(ICML\)*, pp\. 3519–3529\. PMLR, 2019\.
- Lenc & Vedaldi \(2015\)Karel Lenc and Andrea Vedaldi\.Understanding image representations by measuring their equivariance and equivalence\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pp\. 991–999, 2015\.
- Ma et al\. \(2023\)Xinyin Ma, Gongfan Fang, and Xinchao Wang\.LLM\-Pruner: On the structural pruning of large language models\.In*Advances in Neural Information Processing Systems 36 \(NeurIPS\)*, 2023\.
- Raghu et al\. \(2017\)Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl\-Dickstein\.SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability\.In*Advances in Neural Information Processing Systems 30 \(NeurIPS\)*, 2017\.
- Samragh et al\. \(2023\)Mohammad Samragh, Mehrdad Farajtabar, Sachin Mehta, Raviteja Vemulapalli, Fartash Faghri, Devang Naik, Oncel Tuzel, and Mohammad Rastegari\.Weight subcloning: direct initialization of transformers using larger pretrained ones\.*arXiv preprint arXiv:2312\.09299*, 2023\.
- Shen et al\. \(2022\)Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy\.Staged training for transformer language models\.In*Proceedings of the 39th International Conference on Machine Learning \(ICML\)*, pp\. 19893–19908\. PMLR, 2022\.
- Su et al\. \(2024\)Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568:127063, 2024\.
- Wang et al\. \(2023\)Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David D\. Cox, Zhangyang Wang, and Yoon Kim\.Learning to grow pretrained models for efficient transformer training\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Xia et al\. \(2024\)Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen\.Sheared LLaMA: Accelerating language model pre\-training via structured pruning\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.

## Appendix ADeferred tables and figures

This appendix collects tables and figures deferred from the main text for space\. All numbers are referenced from, and discussed in, the sections indicated\.

### A\.1Alignment across sizes \(§[4](https://arxiv.org/html/2608.02829#S4)\)

![Refer to caption](https://arxiv.org/html/2608.02829v1/figures/cka_heatmap.png)

![Refer to caption](https://arxiv.org/html/2608.02829v1/figures/maps_r2.png)

Figure 2:Alignment across sizes for the primary pair \(1\.4B→\\,\\to\\,410M\)\.\(left\)Linear CKA between all24×2424\\times 24layer pairs: a saturated middle band \(layers∼\\sim4–22 mutually≈1\.0\\approx 1\.0\) confirms coarse correspondence but resolves no sharp per\-layer match\.\(right\)Held\-outR2R^\{2\}of the per\-layer activation maps: ridge \(full linear\) and Procrustes \(rotation\+\+scale\) track each other and dip together at layers 3–5, the layers CKA also flags as distinct\.Table 3:Alignment across sizes, held\-out\. Activations align strongly under a linear map; raw parameters, even in the cleanest \(embedding\) case, do not\. Best \(highest\) map per row inbold\.†For activations, the diagonal mean of the24×2424\\times 24linear\-CKA matrix, which saturates in the middle band \(Figure[2](https://arxiv.org/html/2608.02829#A1.F2), left\); for embeddings, linear CKA before mapping\.

### A\.2Projection diagnostics \(§[5](https://arxiv.org/html/2608.02829#S5)\)

Table 4:Relative Frobenius error of projected weights vs\. the actual 410M weights \(mean over 24 layers\)\. Every target\-free error exceeds11\(worse than zero\); lower per row inbold\.†A single\(A,B\)\(A,B\)shared across layers per type; it peeks at the target and so upper\-bounds the linearly explainable fraction\. A*per\-layer*fit is degenerate \(error→0\\to 0around any full\-rankWW\) and is not reported\.

Table 5:Zero\-shot dense\-projected quality \(strided WikiText\-103 perplexity\); the projected model does not beat random init\. Best \(lowest\) inbold\.With fresh*neutral*LayerNorms \(gain 1, bias 0\) the same projected matrices score∼\\sim12\.3k quick\-perplexity —∼\\sim15×\\timesbetter than the 181,242 row and∼\\sim5×\\timesbetter than random init — so much of the collapse is LayerNorm mismatch, not weight destruction alone \(§[7](https://arxiv.org/html/2608.02829#S7)\); the structure\-mixing diagnosis still holds directionally\.

### A\.3Spectral residual test \(§[6](https://arxiv.org/html/2608.02829#S6)\)

Table 6:Test 1 \(spectral\)\. Effective rank of the residualΔ\\Deltavs\. a per\-type shuffled control and a shape/scale\-matched Gaussian control\.Δ\\Deltasits2\.82\.8–6\.4%6\.4\\%below both in every type — a faint, non\-isotropic concentration, not a learnable direction\.![Refer to caption](https://arxiv.org/html/2608.02829v1/figures/delta_spectra.png)Figure 3:Test 1 \(spectral\)\. Singular\-value spectrum of the residualΔ\\Deltaagainst shuffled and Gaussian controls\.Δ\\Deltaconcentrates2\.82\.8–6\.4%6\.4\\%more than either control in every weight type — a faint, generic non\-isotropy that carries no layer\-specific, predictable signal \(Table[1](https://arxiv.org/html/2608.02829#S6.T1)\)\.
### A\.4Extended evaluation, primary pair \(§[7\.5](https://arxiv.org/html/2608.02829#S7.SS5)\)

Table 7:Extended evaluation of the 30M primary\-pair checkpoints and the real\-410M reference\. Guess floors: arc\_easy0\.250\.25, hellaswag0\.250\.25, piqa0\.500\.50, lambada≈0\\approx 0\(open\-vocabulary\)\. Best converted arm inbold; the ordering holds on all six metrics\. Single\-seed checkpoints\. This table predates the rescale\-lever ablation and reports the compensation contrast \(hybridvs\.subclone\); the rescale gains of Tables[2](https://arxiv.org/html/2608.02829#S7.T2)–[8](https://arxiv.org/html/2608.02829#A1.T8)stack on top of it\.
### A\.5Primary\-pair seeded verdict \(§[7\.3](https://arxiv.org/html/2608.02829#S7.SS3)\)

Table 8:Primary\-pair final WikiText\-103 perplexity across three data\-draw seeds \(30M tokens; identical budget/schedule per seed\)\.hybrid\_rswins every paired comparison; its worst seed beatssubclone\_rs’s best\.
### A\.6Held\-out pair per\-seed detail \(§[7\.5](https://arxiv.org/html/2608.02829#S7.SS5)\)

Table 9:Held\-out pair \(410M→\\to160M\), final WikiText\-103 perplexity across three data\-draw seeds \(30M tokens\)\. Transfer beats from scratch∼\\sim13×13\\timeson every seed; hybrid and subclone are a statistical tie on this depth\-dominated pair \(overlapping±1​σ\\pm 1\\sigma\)\.##### Metric robustness \(held\-out pair\)\.

On the held\-out 410M→\\to160M pair the hybrid\-vs\-subclone tie holds on every metric beyond WikiText\-103: C4 perplexity91\.191\.1vs\.94\.294\.2, perplexity at2×2\\timescontext111\.8111\.8vs\.110\.2110\.2, LAMBADA11\.0%11\.0\\%vs\.10\.7%10\.7\\%\(others within noise\), while random sits at C4514514and LAMBADA0; the real\-160M anchor \(LAMBADA35\.4%35\.4\\%\) matches published Pythia numbers\. These cells are single\-seed\.

![Refer to caption](https://arxiv.org/html/2608.02829v1/figures/m5b_curves.png)Figure 4:Held\-out pair \(410M→\\to160M\), three seeds\.Full WikiText\-103 perplexity \(log scale\) versus tokens; bands span the three data\-draw seeds and the dashed line marks the real Pythia\-160M reference\. Transfer inits \(hybrid, subclone\) sit an order of magnitude below random throughout and overlap each other — the depth\-dominated regime where compensation’s width\-repair edge is neutral \(Table[9](https://arxiv.org/html/2608.02829#A1.T9)\), yet the∼\\sim13×13\\timesmargin over from scratch is preserved on every seed\.

### A\.7Scale pair \(§[7\.6](https://arxiv.org/html/2608.02829#S7.SS6)\)

Table 10:Scale pair \(6\.9B→\\to1\.4B\), 30M tokens, single seed\. The two levers that stack at smaller scale here*anti*\-synergize — each alone beats the combination — though every transfer arm still beats from scratch \(§[7\.6](https://arxiv.org/html/2608.02829#S7.SS6)\)\.InitConstructionFinal ppl \(30M\)randomfrom scratch1,413hybrid\_rscompensation\+\+rescale1,213hybridcompensation only776subclone\_rsrescale only572

## Appendix BDeferred discussion

### B\.1An honest correction \(§[7\.3](https://arxiv.org/html/2608.02829#S7.SS3)\)

The rescale lever corrects our own earlier reading\. When the hybrid first won at114\.1114\.1we argued that least\-squares compensation*subsumed*the reference recipe’s scalar rescale, since a ridge\-optimal map beats a scalar on the paths it re\-fits\. The upset —subclone\_rsat86\.486\.4, beating the hybrid with no compensation — showed that is only half right: compensation dominates rescale*on the two read\-out paths it touches*, but the LN\-fronted read\-in paths it leaves alone still benefit from the reference recipe’s variance correction as a training\-dynamics prior\. Blanket rescale collapses zero\-shot yet wins on dynamics, so the reference recipe is vindicated exactly where our first account wrote it off, and the principled method is the stack, not either lever alone\.

### B\.2Reconciling the faint spectral signal \(§[6](https://arxiv.org/html/2608.02829#S6)\)

Tests 1 and 3 are not in tension: the2\.82\.8–6\.4%6\.4\\%spectral deficit is a*generic*statistical trace \(mild row/column\-norm heterogeneity\), not a correspondence betweenW^\\hat\{W\}andΔ\\Deltaany predictor can exploit\. A behavioral cross\-check agrees: an assembled 410M whose six matrix types are operator\-projected \(with real biases, LayerNorms, and embeddings\) scores35,21235\{,\}212perplexity, and the learned correction moves it only∼\\sim2% \(to34,37334\{,\}373\); both remain non\-functional\. Notably the operator matrices alone beat random init \(3535k vs\.6666k\) once the surrounding tensors are real — unlike §[5](https://arxiv.org/html/2608.02829#S5)’s fully projected model — again locating the destruction in the dense projection, not the residual\. \(The correction’s gains on the 18 training layers are partly memorized; only the held\-out layers carry genuine predictions, and there the reduction is zero\.\)

## Appendix CTraining protocol details

Every conversion arm within an experiment shares*identical*data order, optimizer \(AdamW,β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay0\.10\.1\), cosine schedule with 100\-step warmup, gradient clipping at1\.01\.0, bf16 autocast, and token budget \(30M primary; 100M persistence checks\); the learning rate equals the target size’s original pretraining rate\. Initialization construction costs are negligible against any training budget \(selection: seconds; moments pass for compensation:∼\\sim75 s; closed\-form solves: seconds\) and are included in the compute accounting\. All experiments ran on a single NVIDIA GB10 \(128 GB unified memory\); any 24 GB\+ CUDA GPU reproduces them\.