SPECTRUM: Proximal Spectral Modulation for Looped Self-Distillation

# Proximal Spectral Modulationfor Looped Self-Distillation
Source: [https://arxiv.org/html/2610.07237](https://arxiv.org/html/2610.07237)
## SPECTRUM: Proximal Spectral Modulation for Looped Self\-Distillation

WenJie ChenAffiliation:Fudan UniversityJiaquan ZhangAffiliation:Fudan UniversityGuangya HaoAffiliation:University of CambridgeZihang ZengAffiliation:Fudan UniversityPengze LiAffiliation:Fudan UniversityXi ChenAffiliation:Fudan University

###### Abstract

A model that learns from its own outputs inherits more than their correctness: it inherits which solutions it produces\. We formulate Looped Self\-Distillation, a self\-evolution framework for code generation in which a model repeatedly generates and learns from its own raw outputs, under a fixed information budget, without ongoing external assessment or test\-based selection of the generated samples\. We identify a consequential separation: correctness can improve while the breadth of correct implementations contracts\. We introduceSpectrum, which re\-estimates loss\-sensitive key/value geometry from a fixed reference anchor at each round and converts it into full\-rank proximal spectral modulation\. All generated completions train a single student, whose subsequent inference requires no intervention\. After five rounds of experiments on MBPP,Spectrumretains89\.9%89\.9\\%of the initial model’s 64\-sample correct AST richness, compared with66\.4%66\.4\\%for Vanilla self\-distillation and65\.5%65\.5\\%for a subspace\-projection control\. The advantage persists at matched correct\-sample counts\. Without further training or recalibration, the resulting student also achieves higher matched\-correct richness than Vanilla SD on HumanEval\+ and APPS Intro, demonstrating transfer of the diversity benefit\. These findings establish correct\-solution retention as a complementary objective of recursive self\-improvement \(RSI\) and show that generation\-time intervention can improve the solution repertoire retained by subsequent students\.

## 1Introduction

Iterative self\-distillation has improved rationales and code\-data synthesizers\([Rao et al\., 2023](https://arxiv.org/html/2610.07237#bib.bib21);[Zhang et al\., 2025](https://arxiv.org/html/2610.07237#bib.bib22)\)\. Existing approaches use correctness selection, reward\-conditioned revision, evolving teachers, or decoding\-based distribution shaping\([Zelikman et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib12);[Singh et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib13);[He et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib14);[Sang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib23);[Zhang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib2)\)\. We ask:*what does a model retain when it repeatedly learns from its own outputs without new external assessments of those outputs?*

Each student supplies its next training distribution\. Higher correctness need not make it a better source of diverse correct solutions \(Figure[1](https://arxiv.org/html/2610.07237#S1.F1)\)\. Recursive self\-improvement \(RSI\) therefore calls for measuring whether alternative successful behaviors survive and transfer beyond the training distribution\. We formulate*Looped Self\-Distillation under a fixed information budget*\. The evolving model generates a corpus, learns from every raw completion, and becomes the next generator\. No new annotations, verifier rewards, judge scores, or quality\-based sample selection enter training\. An optional reference anchor supplies repeated calibration but is never expanded with newly judged samples\. Alternative proofs, plans, and programs make solution diversity relevant across reasoning tasks\. For coding agents that use language models to generate candidate programs, repeated self\-distillation changes which alternatives those models can supply\. We study code generation because executable tests and program structure make solution breadth measurable\([Chen et al\., 2021](https://arxiv.org/html/2610.07237#bib.bib8);[Liu et al\., 2023](https://arxiv.org/html/2610.07237#bib.bib6)\), and multiple implementations support program search\([Li et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib18)\)\. Even the entire pass@kkcurve cannot identify the distribution within correct outputs: redistributing their probability while holding total correctness fixed leaves pass@kkunchanged \(Proposition[1](https://arxiv.org/html/2610.07237#Thmproposition1)\)\.

![Refer to caption](https://arxiv.org/html/2610.07237v1/figures/star.png)Figure 1:Each student inherits a distribution of continuations\.A shared code prefix admits different continuation forms\. The schematic histograms depict concentration under repeated Vanilla SD and broader retention underSpectrum\. Heights are conceptual; the experiments measure complete correct programs, rather than the displayed continuation starts \(Figure[3](https://arxiv.org/html/2610.07237#S4.F3)\)\.Our experiments expose a separation between correctness and breadth\. Across five rounds, Vanilla SD’s pass@1 rises while correct implementation richness contracts \(Figure[3](https://arxiv.org/html/2610.07237#S4.F3); Appendix[D](https://arxiv.org/html/2610.07237#A4)\)\. In the final 64\-sample evaluation, pass@1 increases from37\.94%37\.94\\%to41\.45%41\.45\\%, while correct AST richness falls from12\.80212\.802to8\.5068\.506classes per task\. This connects recursive synthetic\-data degradation\([Shumailov et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib3);[Gerstgrasser et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib4)\)and self\-distillation diversity loss\([Nicolicioiu et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib24)\)to repeated learning from unselected outputs\.

We introduceSpectrumto improve retention\. Building on reference\-gradient calibration\([Hao et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib1)\), it re\-estimates the student’s loss\-sensitive K/V geometry from the same anchor each round\. A full\-rank proximal gain attenuates less\-sensitive directions while retaining a positive local response in every direction\. The modulated model generates the corpus; native weights are restored before one LoRA learns from all completions\([Hu et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib5)\)\. After five rounds,Spectrumretains89\.9%89\.9\\%of initial 64\-sample richness versus Vanilla SD’s66\.4%66\.4\\%, and improves pass@64 by2\.22\.2percentage points over Vanilla SD\. Pass@1 is2\.392\.39points above initialization and1\.121\.12below Vanilla SD\. Matched\-correct richness gains transfer to HumanEval\+ and APPS Intro\. Our contributions are:

1. 1\.A framework for Looped Self\-Distillation\.Repeated learning from raw self\-generated corpora under a fixed information budget, without ongoing external assessment or selection of generated samples\.
2. 2\.SPECTRUM: proximal spectral modulation\.Per\-round loss\-sensitive geometry and a full\-rank generation operator transfer a temporary intervention into one native student\.
3. 3\.Solution retention across rounds and benchmarks\.Longitudinal, sampling\-budget, generation\-design, and transfer analyses establish improved retention within correct implementations\.

## 2Preliminaries

### 2\.1Self\-distillation from model\-generated outputs

SSD learns from self\-generated code through cross\-entropy training, using temperature and truncation to shape its synthetic distribution without correctness\-based selection\([Zhang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib2)\)\. We study this sequence\-level learning mechanism when the updated student repeatedly becomes its own data source\.

### 2\.2Looped Self\-Distillation

We formulate*Looped Self\-Distillation*as repeated generation and learning under a fixed information budget\. Letθt\\theta\_\{t\}denote the current student,𝒬\\mathcal\{Q\}a fixed set of synthesis prompts, and𝒢\\mathcal\{G\}a generation constructor\. One round consists of

π~t\\displaystyle\\widetilde\{\\pi\}\_\{t\}=𝒢⁡\(θt,𝒜\),\\displaystyle=\\mathcal\{G\}\(\\theta\_\{t\};\\mathcal\{A\}\),\(1\)𝒟t\\displaystyle\\mathcal\{D\}\_\{t\}=\(\(x,Yx,s\)\)x∈𝒬,s=1,…,S,Yx,s∼π~t\(⋅∣x\),\\displaystyle=\\bigl\(\(x,Y\_\{x,s\}\)\\bigr\)\_\{x\\in\\mathcal\{Q\},\\ s=1,\\ldots,S\},\\qquad Y\_\{x,s\}\\sim\\widetilde\{\\pi\}\_\{t\}\(\\cdot\\mid x\),θt\+1\\displaystyle\\theta\_\{t\+1\}=ℒ⁡\(θt,𝒟t\),\\displaystyle=\\mathcal\{L\}\(\\theta\_\{t\},\\mathcal\{D\}\_\{t\}\),where𝒟t\\mathcal\{D\}\_\{t\}is an indexed corpus retaining duplicate records andℒ\\mathcal\{L\}learns from every generated completion through cross\-entropy training\. The optional reference information𝒜\\mathcal\{A\}is fixed before the loop\. Newly generated training samples receive no ongoing external evaluation or supervision: no verifier or judge determines their selection or training weights\.

The loop closes whenθt\+1\\theta\_\{t\+1\}becomes the next generator, exposing how an update changes the distribution available for future learning\. This specifies a learning mechanism within the broader RSI agenda\. Our implementation uses one LoRA per round\([Hu et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib5)\); the outer data–parameter loop differs from recurrent\-depth transformers that repeat computation within inference\([Dehghani et al\., 2019](https://arxiv.org/html/2610.07237#bib.bib16)\)\.

### 2\.3Correctness and correct\-solution breadth

For a taskxx, letvx​\(y\)∈\{0,1\}v\_\{x\}\(y\)\\in\\\{0,1\\\}denote passing its recorded verifier, and letϕ⁡\(y\)\\phi\(y\)identify a normalized AST class\. Letpθp\_\{\\theta\}denote raw model likelihood andπθ\\pi\_\{\\theta\}the fixed decoding policy induced by the model\. Defineaθ\(x\)=Prπθ\[vx\(Y\)=1\]a\_\{\\theta\}\(x\)=\\Pr\_\{\\pi\_\{\\theta\}\}\[v\_\{x\}\(Y\)=1\]and, whenaθ​\(x\)\>0a\_\{\\theta\}\(x\)\>0,

qθ,j​\(x\)=Prπθ⁡\[ϕ⁡\(Y\)=j∣vx​\(Y\)=1\],Pr⁡\[vx​\(Y\)=1,ϕ⁡\(Y\)=j\]=aθ​\(x\)​qθ,j​\(x\)\.q\_\{\\theta,j\}\(x\)=\\Pr\_\{\\pi\_\{\\theta\}\}\[\\phi\(Y\)=j\\mid v\_\{x\}\(Y\)=1\],\\qquad\\Pr\[v\_\{x\}\(Y\)=1,\\phi\(Y\)=j\]=a\_\{\\theta\}\(x\)q\_\{\\theta,j\}\(x\)\.\(2\)The first factor measures success frequency; the second describes the distribution within successful outputs\. Suppressxxand assume independent draws from a fixed decoding policy below\.

###### Proposition 1\(Correctness does not identify correct\-solution breadth\)\.

For fixeda\>0a\>0, pass@k=1−\(1−a\)kk=1\-\(1\-a\)^\{k\}is independent ofqq\. Expected correct\-class counts underkktotal draws andbbcorrect draws are

Ck​\(a,q\)=∑j\[1−\(1−a​qj\)k\],Db​\(q\)=∑j\[1−\(1−qj\)b\]\.C\_\{k\}\(a,q\)=\\sum\_\{j\}\[1\-\(1\-aq\_\{j\}\)^\{k\}\],\\qquad D\_\{b\}\(q\)=\\sum\_\{j\}\[1\-\(1\-q\_\{j\}\)^\{b\}\]\.\(3\)Identical pass@kkat everykkcan coexist with differentDbD\_\{b\}for anyb≥2b\\geq 2\.

We callCkC\_\{k\}*correct AST richness@kk*andDbD\_\{b\}*correct\-count\-matched AST richness@bb*\. Finite\-pool rarefaction estimators\([Hurlbert, 1971](https://arxiv.org/html/2610.07237#bib.bib15)\)and the proof appear in Appendix[A](https://arxiv.org/html/2610.07237#A1)\. Conditional comparisons use tasks eligible for both methods and report their number\. Across rounds, relative richness retention is the ratio of task\-meanCkC\_\{k\}to its initial\-model value\. It measures expected class counts at a fixed budget, rather than identity overlap with the initial classes\.

## 3SPECTRUM: Proximal Spectral Modulation

Spectrumtemporarily modulates the current model’s K/V representations to generate its next training corpus\. Each round recalibrates the geometry from the fixed anchor, generates under a full\-rank spectral operator, restores native weights, and trains one student on all raw completions \(Figure[2](https://arxiv.org/html/2610.07237#S3.F2)\)\.

![Refer to caption](https://arxiv.org/html/2610.07237v1/figures/final.png)Figure 2:SPECTRUM within Looped Self\-Distillation\.Fixed\-anchor completion gradients determine a per\-round K/V operator, folded into temporary generation weights\. All raw completions train one LoRA student after native weights are restored\. The merged student starts the next round and supports native inference\. Dashed arrows denote calibration gradients; the bottom loop carries updated parameters\. Geometry and code examples are schematic\.### 3\.1Loss\-sensitive key/value geometry

Let𝒜=\{\(xi,yi⋆\)\}\\mathcal\{A\}=\\\{\(x\_\{i\},y\_\{i\}^\{\\star\}\)\\\}be the reference anchor fixed before the loop\. For the current studentθt\\theta\_\{t\}, letℓi​\(θt\)\\ell\_\{i\}\(\\theta\_\{t\}\)denote the mean negative log likelihood of reference completionyi⋆y\_\{i\}^\{\\star\}\. Prompt and padding targets are masked\. Following reference\-loss\-gradient calibration of K/V directions\([Hao et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib1)\), collect the gradient vectorgi​u\(ℓ\)∈ℝdℓg\_\{iu\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\_\{\\ell\}\}with respect to the native linear output of each selected K/V moduleℓ\\ellat nonpadding positionuu\. Each module is calibrated separately\. Its uncentered gradient second moment and normalized geometry are

Ct,ℓ=1Mℓ​∑i,u,nonpadgi​u\(ℓ\)​gi​u\(ℓ\)⊤,C¯t,ℓ=Ct,ℓλmax​\(Ct,ℓ\)\.C\_\{t,\\ell\}=\\frac\{1\}\{M\_\{\\ell\}\}\\sum\_\{i,u,\\mathrm\{nonpad\}\}g\_\{iu\}^\{\(\\ell\)\}g\_\{iu\}^\{\(\\ell\)\\top\},\\qquad\\overline\{C\}\_\{t,\\ell\}=\\frac\{C\_\{t,\\ell\}\}\{\\lambda\_\{\\max\}\(C\_\{t,\\ell\}\)\}\.\(4\)HereMℓM\_\{\\ell\}counts all included positions, including zero gradients\. Nonzero calibration signal gives0⪯C¯t,ℓ⪯𝐈0\\preceq\\overline\{C\}\_\{t,\\ell\}\\preceq\\mathbf\{I\}\. Prompt positions are included because they influence completion loss despite their masked prediction targets\.

For a unit directionvv,v⊤​Ct,ℓ​vv^\{\\top\}C\_\{t,\\ell\}vis the mean squared directional derivative\(gi​u\(ℓ\)⊤​v\)2\(g\_\{iu\}^\{\(\\ell\)\\top\}v\)^\{2\}\. A finite anchor observes only part of the model’s behavior, so low measured sensitivity need not imply uselessness for other valid implementations\. We use this statistic for continuous attenuation and re\-estimate it under each updated student\.

### 3\.2Full\-rank proximal spectral modulation

Suppressing round and module indices, consider a row activationh∈ℝ1×dh\\in\\mathbb\{R\}^\{1\\times d\}\. We choose the quadratic penaltyz⁡\(𝐈−C¯\)​z⊤z\(\\mathbf\{I\}\-\\overline\{C\}\)z^\{\\top\}and define

z⋆\\displaystyle z^\{\\star\}=arg​minz∈ℝ1×d⁡\{12​‖z−h‖22\+τ2​z​\(𝐈−C¯\)​z⊤\}=hTτ,\\displaystyle=\\argmin\_\{z\\in\\mathbb\{R\}^\{1\\times d\}\}\\left\\\{\\frac\{1\}\{2\}\\\|z\-h\\\|\_\{2\}^\{2\}\+\\frac\{\\tau\}\{2\}z\(\\mathbf\{I\}\-\\overline\{C\}\)z^\{\\top\}\\right\\\}=hT\_\{\\tau\},\(5\)Tτ\\displaystyle T\_\{\\tau\}=\[𝐈\+τ\(𝐈−C¯\)\]−1,τ≥0\.\\displaystyle=\[\\mathbf\{I\}\+\\tau\(\\mathbf\{I\}\-\\overline\{C\}\)\]^\{\-1\},\\qquad\\tau\\geq 0\.This quadratic proximal map\([Parikh and Boyd, 2014](https://arxiv.org/html/2610.07237#bib.bib10)\)limits departure from the native activation while penalizing less\-sensitive directions more strongly at fixed norm\.

WritingC¯=U​diag⁡\(μj\)​U⊤\\overline\{C\}=U\\diag\(\\mu\_\{j\}\)U^\{\\top\}gives the closed\-form response

Tτ=U​diag⁡\(11\+τ⁡\(1−μj\)\)​U⊤\.T\_\{\\tau\}=U\\diag\\\!\\left\(\\frac\{1\}\{1\+\\tau\(1\-\\mu\_\{j\}\)\}\\right\)U^\{\\top\}\.\(6\)The highest\-sensitivity directions retain unit gain; all others retain positive gains at finiteτ\\tau\. Strength varies continuously without a prescribed rank\.

###### Proposition 2\(Local distinguishability and calibration stability\)\.

Let0⪯C¯⪯𝐈0\\preceq\\overline\{C\}\\preceq\\mathbf\{I\}and0≤τ<∞0\\leq\\tau<\\infty\. Equation \([5](https://arxiv.org/html/2610.07237#S3.E5)\) has a unique minimizer, and the gains ofTτT\_\{\\tau\}lie in\[1/\(1\+τ\),1\]\[1/\(1\+\\tau\),1\]\. For any row\-vector differenceδ\\delta,

‖δ‖21\+τ≤‖δ​Tτ‖2≤‖δ‖2,‖Tτ−𝐈‖2≤τ1\+τ\.\\frac\{\\\|\\delta\\\|\_\{2\}\}\{1\+\\tau\}\\leq\\\|\\delta T\_\{\\tau\}\\\|\_\{2\}\\leq\\\|\\delta\\\|\_\{2\},\\qquad\\\|T\_\{\\tau\}\-\\mathbf\{I\}\\\|\_\{2\}\\leq\\frac\{\\tau\}\{1\+\\tau\}\.\(7\)For0⪯A,B⪯𝐈0\\preceq A,B\\preceq\\mathbf\{I\}, the corresponding operators satisfy‖Tτ​\(A\)−Tτ​\(B\)‖2≤τ​‖A−B‖2\\\|T\_\{\\tau\}\(A\)\-T\_\{\\tau\}\(B\)\\\|\_\{2\}\\leq\\tau\\\|A\-B\\\|\_\{2\}\.

The operator also increases or preserves relative alignment with the measured sensitivity geometry\. DefinesC¯​\(h\)=h​C¯​h⊤/‖h‖22s\_\{\\overline\{C\}\}\(h\)=h\\overline\{C\}h^\{\\top\}/\\\|h\\\|\_\{2\}^\{2\}forh≠0h\\neq 0\. Monotonicity of the gains inμj\\mu\_\{j\}yields

sC¯​\(h​Tτ\)≥sC¯​\(h\)\.s\_\{\\overline\{C\}\}\(hT\_\{\\tau\}\)\\geq s\_\{\\overline\{C\}\}\(h\)\.\(8\)These local properties establish bounded, sensitivity\-aligned modulation and calibration stability \(Appendix[B](https://arxiv.org/html/2610.07237#A2)\)\. Retention after attention, decoding, and student learning is measured experimentally\.

### 3\.3Distilling the modulated generator

For a native linear moduleh=x​W⊤\+b⊤h=xW^\{\\top\}\+b^\{\\top\}, the row operator can be folded into its parameters:

W′=Tτ⊤​W,b′=Tτ⊤​b,x​W′⁣⊤\+b′⁣⊤=h​Tτ\.W^\{\\prime\}=T\_\{\\tau\}^\{\\top\}W,\\qquad b^\{\\prime\}=T\_\{\\tau\}^\{\\top\}b,\\qquad xW^\{\\prime\\top\}\+b^\{\\prime\\top\}=hT\_\{\\tau\}\.\(9\)Folding acts at native K/V outputs before rotary transformations and key normalization, requiring no generation\-time activation hooks\. Keys affect query–key compatibility; values affect the information aggregated by attention\.

After generating𝒟t\\mathcal\{D\}\_\{t\}, we restore the original weightsθt\\theta\_\{t\}and fit one LoRA adapter\([Hu et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib5)\)\. Every raw record enters training, including unsuccessful, empty, and duplicate completions\. For tokenized recordz=\(x,y\)z=\(x,y\), let𝒯⁡\(z\)\\mathcal\{T\}\(z\)contain its nonpadding causal targets, and letΔ\\Deltadenote the effective LoRA update\. The student objective is

ℒt\(Δ\)=−1\|𝒟t\|∑z∈𝒟t1\|𝒯⁡\(z\)\|∑u∈𝒯⁡\(z\)logpθt\+Δ\(zu∣z<u\)\.\\mathcal\{L\}\_\{t\}\(\\Delta\)=\-\\frac\{1\}\{\|\\mathcal\{D\}\_\{t\}\|\}\\sum\_\{z\\in\\mathcal\{D\}\_\{t\}\}\\frac\{1\}\{\|\\mathcal\{T\}\(z\)\|\}\\sum\_\{u\\in\\mathcal\{T\}\(z\)\}\\log p\_\{\\theta\_\{t\}\+\\Delta\}\(z\_\{u\}\\mid z\_\{<u\}\)\.\(10\)SFT includes prompt targets, whereas calibration uses completion targets only\. Every record contributes an equally weighted token\-mean loss; the operator changes sequence frequencies, not correctness\-based training weights\.

The merged adapter yieldsθt\+1\\theta\_\{t\+1\}\. Restoring native weights prevents direct composition of operators across rounds; their influence passes through the corpora\. Final inference uses neither operator nor anchor\.

#### Fixed reference information, no ongoing external feedback\.

The reference anchor is supplied before the loop and reused unchanged\. Recalibration acquires no new annotations, external\-model judgments, or verifier feedback\. All generated records retain equal training weight; tests only evaluate the resulting students\.

## 4Experiments

### 4\.1Protocol and research questions

#### Learning and evaluation\.

We study five rounds with Qwen2\.5\-Coder\-1\.5B\-Instruct\([Hui et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib9)\)on MBPP\([Austin et al\., 2021](https://arxiv.org/html/2610.07237#bib.bib7)\)\. After removing three exact normalized train/test prompt overlaps, the disjoint splits contain 291 synthesis, 50 calibration, 30 validation, and 500 evaluation tasks\. Each round generates 16 raw completions per synthesis task and fits one LoRA epoch: 4,656 records per round, 23,280 overall\.Spectrumusesτ=1\\tau=1, middle and last decoder\-layer K/V modules, and LoRA rank 8\. The anchor and hyperparameters stay fixed\.

Native evaluation uses temperature0\.80\.8, top\-pp0\.950\.95, and no top\-kktruncation\. Every round has 16 samples per task; a separate final pool has 64 for the initial, Vanilla SD, projection, andSpectrumcheckpoints\. SSD is compared under the common 16\-sample protocol\. We report pass@kk,CkC\_\{k\}, andDbD\_\{b\}, using common eligible tasks for paired conditional contrasts\. The longitudinal study uses one training run; intervals use 2,000 task\-bootstrap resamples\. Appendix[C](https://arxiv.org/html/2610.07237#A3)gives complete settings, and Appendix[E](https://arxiv.org/html/2610.07237#A5)provides independent initial\-model context across four models and five benchmarks\.

#### Comparators and questions\.

Vanilla SD uses standard sampling; SSD shapes generation through temperature1\.51\.5, top\-pp0\.80\.8, and top\-kk20\([Zhang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib2)\)\. A top\-half rank projector is a generation\-design ablation adapted from[Hao et al\. \(2026\)](https://arxiv.org/html/2610.07237#bib.bib1)\. Projection andSpectrumshare the 50 references and calibration procedure; Vanilla SD and SSD use no anchor\. Synthesis tasks, record counts, and optimization schedules are shared\. We ask what survives successive updates \(RQ1\), whether gains persist within correct outputs \(RQ2\), how generation design changes the result \(RQ3\), and whether retention transfers \(RQ4\)\. A separate one\-round study supplies complementary evidence under a different training budget\.

### 4\.2RQ1: What do successive students retain?

Figure 3:Generation design separates correctness from solution retention\.Native students on 500 MBPP tasks, 16 samples per task per round\. \(a\)C16C\_\{16\}relative to each recorded initialization; \(b\) pass@1; \(c\) pass@16\. All panels show point estimates from one training trajectory per design\. SSD’s initial model is sampled independently\. Additional round\-level results appear in Appendix[D](https://arxiv.org/html/2610.07237#A4)\.Vanilla SD’s pass@1 rises from37\.92%37\.92\\%to41\.30%41\.30\\%whileC16C\_\{16\}falls from4\.0164\.016to3\.0903\.090\.Spectruminstead retains89\.8%89\.8\\%of initialC16C\_\{16\}, versus Vanilla SD’s76\.9%76\.9\\%, with greater richness at every learned round \(Figure[3](https://arxiv.org/html/2610.07237#S4.F3)a\)\. Its correctness also improves: pass@1 ends at40\.18%40\.18\\%, and pass@16 at67\.2%67\.2\\%versus Vanilla SD’s66\.0%66\.0\\%\. The joint trajectory identifies a student that improves success while retaining more of its correct repertoire\.

Table 1:Final native students after five MBPP rounds\.Independent 64\-sample evaluation on 500 tasks\. Bold marks the better trained\-student estimate\.D4D\_\{4\}is a marginal mean over 322/321/323 eligible tasks, respectively; paired contrasts follow separately\.The separate 64\-sample pool confirms the retention advantage \(Table[1](https://arxiv.org/html/2610.07237#S4.T1)\):11\.51011\.510correct AST classes versus8\.5068\.506, or35\.3%35\.3\\%more, with paired difference3\.0043\.004\[2\.568,3\.494\]\[2\.568,3\.494\]\. Retention reaches89\.9%89\.9\\%versus66\.4%66\.4\\%\.Spectrumsolves 363 tasks versus 352, a pass@64 gain of2\.22\.2percentage points\[0\.4,4\.2\]\[0\.4,4\.2\]\. Pass@1 is2\.392\.39points above initialization and1\.121\.12below Vanilla SD\[−1\.61,−0\.63\]\[\-1\.61,\-0\.63\]\. Thus richer native students solve more tasks at high budget, with a smaller single\-draw success rate than Vanilla SD\.

### 4\.3RQ2: Does the advantage persist within correct outputs?

Matching successful draws isolates breadth within correct outputs\. At four correct draws,Spectrumexceeds Vanilla SD by0\.2950\.295expected AST classes\[0\.249,0\.343\]\[0\.249,0\.343\]on 316 common eligible tasks\. At eight and sixteen correct draws, the paired gains are0\.7800\.780and1\.8161\.816on 296 and 254 tasks \(Figure[4](https://arxiv.org/html/2610.07237#S4.F4)c\)\. The advantage therefore persists after equalizing correct\-sample counts; each budget uses its own common eligible population\.

![Refer to caption](https://arxiv.org/html/2610.07237v1/figures/exp2.png)Figure 4:\(a,b\) Correct AST richness and pass@kkfrom the same final 64\-sample pools on 500 MBPP tasks\. Budget groups are categorical\. \(c\) PairedSpectrum\-minus\-Vanilla SDDbD\_\{b\}gains; whiskers are 95% task\-bootstrap intervals on 316, 296, and 254 common eligible tasks forb=4,8,16b=4,8,16\.The total\-budget curves provide the complementary view\. Atk=1k=1,C1C\_\{1\}equals correctness and favors Vanilla SD\. At every displayed larger budget,Spectrumproduces more distinct correct structures, with its largest pass@kkadvantage atk=64k=64\(Figure[4](https://arxiv.org/html/2610.07237#S4.F4)a,b\)\. These endpoints distinguish breadth bought by a generation budget from variety conditional on success\.

### 4\.4RQ3: How does generation design affect retention and correctness?

Table 2:Generation designs and realized training tokens\.Round\-five native students, 500 MBPP tasks, 16 evaluation draws\. Success and richness are higher\-is\-better;D4D\_\{4\}uses each method’s eligible tasks \(Appendix[D](https://arxiv.org/html/2610.07237#A4)\)\. SFT tokens sum five rounds, in millions\. All designs use 23,280 training records; these are not matched\-compute measurements\.#### Improving correctness relative to decoding\-based SSD\.

Spectrumachieves the highest pass@16 among the evaluated designs\. Across the five\-round trajectories, its pass@1 improves by2\.252\.25points from its recorded initialization, whereas SSD declines by1\.851\.85points \(Figure[3](https://arxiv.org/html/2610.07237#S4.F3)b\)\. At round five,Spectrumexceeds SSD by4\.214\.21points in pass@1 and1\.601\.60in pass@16\. TheirC8C\_\{8\}point estimates are nearly identical \(2\.0912\.091\), while SSD has higherC16C\_\{16\}and marginalD4D\_\{4\}\(Table[2](https://arxiv.org/html/2610.07237#S4.T2)\)\. The designs occupy distinct correctness–richness operating points\.

The accuracy gain uses8\.1%8\.1\\%fewer generated completion tokens and6\.4%6\.4\\%fewer SFT tokens than SSD\. Relative to Vanilla SD, generated tokens are9\.9%9\.9\\%higher\. These are realized text budgets, not wall\-clock or total\-compute comparisons:Spectrumalso performs calibration \(Appendix[G](https://arxiv.org/html/2610.07237#A7)\)\.

#### Continuous modulation versus projection\.

With the same reference anchor, calibration, module locations, and learning schedule, continuous modulation retains89\.9%89\.9\\%of initialC64C\_\{64\}versus projection’s65\.5%65\.5\\%\. It gains3\.1223\.122classes and2\.42\.4pass@64 points, with pass@1 lower by1\.431\.43points\. On 251 common eligible tasks in the 16\-sample pool, pairedD4D\_\{4\}gains are0\.3120\.312\[0\.253,0\.377\]\[0\.253,0\.377\]for exact programs and0\.2930\.293\[0\.233,0\.353\]\[0\.233,0\.353\]for AST classes; the coarser control\-flow gain is0\.0420\.042\[−0\.011,0\.095\]\[\-0\.011,0\.095\]\. The evidence is strongest for implementation breadth\. Attenuation strengths differ, so this contrast evaluates the complete operators\.

### 4\.5RQ4: Does retained diversity transfer across benchmarks?

We freeze the MBPP\-trained final students and evaluate 164 HumanEval\+ tasks and a fixed 200\-task APPS Intro subset with 16 samples per task, without further training or recalibration\. HumanEval\+ uses EvalPlus 0\.3\.1\([Liu et al\., 2023](https://arxiv.org/html/2610.07237#bib.bib6)\); APPS uses the recorded Python function\-interface adaptation\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.07237#bib.bib17)\)\(Appendix[C](https://arxiv.org/html/2610.07237#A3)\)\.

Table 3:Matched\-correct richness transfers to both benchmarks\.Spectrumminus Vanilla SD after MBPP learning\.D4D\_\{4\}uses common eligible tasks; accuracy differences use all tasks\. Intervals are 95% paired task\-bootstrap intervals\. Full absolute results appear in Table[9](https://arxiv.org/html/2610.07237#A4.T9)\.The paired richness advantage is positive on both benchmarks \(Table[3](https://arxiv.org/html/2610.07237#S4.T3)\)\. On HumanEval\+,Spectrumalso matches Vanilla SD’s pass@16 \(86\.59%86\.59\\%\) and raises theC16C\_\{16\}point estimate from5\.8485\.848to6\.0006\.000\. APPSC16C\_\{16\}falls from1\.9301\.930to1\.8451\.845, alongside its lower accuracy\. Thus the transferable benefit lies in matched\-correct implementation breadth, with benchmark\-dependent success rates\. Appendix[F](https://arxiv.org/html/2610.07237#A6)shows recorded equal\-correct\-count examples\.

### 4\.6Complementary evidence from a one\-round study

A one\-round study evaluates 64 MBPP tasks with 64 samples each\. It uses one generate–learn cycle, 128 synthesis tasks with one completion each, and five SFT epochs, separately from the five\-round protocol\.Spectrumimproves SSD’s pass@32, pass@64, and matched\-correct richness \(Table[4](https://arxiv.org/html/2610.07237#S4.T4)\); the recorded pairedD4D\_\{4\}gain is0\.03120\.0312\[0\.0095,0\.0550\]\[0\.0095,0\.0550\], with pass@1 lower by0\.2440\.244points\. This supplies complementary evidence under another training budget\. Appendix[H](https://arxiv.org/html/2610.07237#A8)reports all four designs and the code\-extraction and verification protocol\.

Table 4:One\-round comparison with SSD\.64 MBPP tasks, 64 samples per task; a separate training protocol\. Values are the archived rounded summaries\. Bold marks the larger estimate\. Full comparisons and reported intervals are in Appendix[H](https://arxiv.org/html/2610.07237#A8)\.

## 5Related work

#### Sequence\-level self\-distillation\.

SSD fits self\-generated, unverified code after shaping its distribution through temperature and truncation\([Zhang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib2)\)\. This is our closest learning paradigm\.Spectruminstead shapes generation with fixed reference\-loss geometry and measures what native students retain across complete generation–learning cycles\. Vanilla SD and the adapted SSD recipe use no reference completions\.

#### Recursive self\-improvement and evolving teachers\.

STaR and ReST\-EM learn from solutions selected through correctness feedback\([Zelikman et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib12);[Singh et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib13)\)\. Iterative self\-distillation has explicit precedents:[Rao et al\. \(2023\)](https://arxiv.org/html/2610.07237#bib.bib21)use critics and filtering for contexts and rationales, while SCoder improves a code\-data synthesizer through checkpoint sampling, scoring, and selection\([Zhang et al\., 2025](https://arxiv.org/html/2610.07237#bib.bib22)\)\. SD\-Zero distills reward\-conditioned self\-revision with teacher synchronization\([He et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib14)\); CRISP refreshes an instruction\-conditioned teacher for token\-level KL training without correctness filtering\([Sang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib23)\)\. Our setting fixes synthesis prompts and optional reference information, retains every raw completion, and examines successive students as future data sources\.

#### Diversity under recursive learning\.

[Nicolicioiu et al\. \(2026\)](https://arxiv.org/html/2610.07237#bib.bib24)show functional and semantic diversity loss with sampled correct demonstrations supplied to a self\-teacher\. Our loop omits that demonstration\-conditioned feedback\. UA\-RL rewards rare strategies\([Hu et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib11)\);Spectrummodulates generation without scoring or labeling generated strategies\. Recursive model\-collapse studies examine synthetic\-data degradation, and data accumulation can mitigate it\([Shumailov et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib3);[Gerstgrasser et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib4)\)\. We hold the historical\-data policy fixed and jointly measure pass@kk,CkC\_\{k\}, andDbD\_\{b\}to separate correctness from breadth within correct outputs\.

## 6Limitations

Our evaluation has three limitations\. First, AST richness captures implementation structure rather than semantic algorithm identity\. Structurally different programs may share an underlying algorithm, so semantic and behavioral comparisons would provide a more complete account of solution diversity\. Second, finite tests and sample budgets can miss invalid behaviors and rare implementations, motivating broader behavioral checks and adaptive sampling\. Third, diversity requires task\-specific interpretation: Python AST classes do not directly correspond to distinct proofs, plans, or explanations across languages\. Extending this framework requires representations that capture meaningful alternatives within each domain\.

## 7Conclusion and future work

We introduce Looped Self\-Distillation as a framework for repeated self\-improvement under a fixed information budget, without ongoing external assessment of generated samples\. Each student supplies its next training distribution, and correctness can improve while correct\-solution breadth contracts\.Spectrumaddresses this retention problem through evolving K/V geometry and full\-rank proximal modulation, transferring a temporary generation intervention into a native student through unfiltered data\. After five MBPP rounds, it retains89\.9%89\.9\\%of initial 64\-sample richness versus Vanilla SD’s66\.4%66\.4\\%\. Matched\-correct richness gains also appear on HumanEval\+ and APPS Intro\.

Retention and transfer matter for long\-horizon RSI\. Future work will investigate whether lifelong agents can consolidate experience into reusable concepts and strategies, then recombine them to solve unfamiliar problems without additional task\-specific training\.

## References

- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.Note:arXiv preprintExternal Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732),[Document](https://dx.doi.org/10.48550/arXiv.2108.07732)Cited by:[§4\.1](https://arxiv.org/html/2610.07237#S4.SS1.SSS0.Px1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. Ponde de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.Note:arXiv preprintExternal Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374),[Document](https://dx.doi.org/10.48550/arXiv.2107.03374)Cited by:[§A\.2](https://arxiv.org/html/2610.07237#A1.SS2.p1.2),[§1](https://arxiv.org/html/2610.07237#S1.p2.1)\.
- Dehghaniet al\.\(2019\)M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. KaiserUniversal transformers\.InInternational Conference on Learning Representations,External Links:1807\.03819,[Link](https://arxiv.org/abs/1807.03819)Cited by:[§2\.2](https://arxiv.org/html/2610.07237#S2.SS2.p2.1)\.
- Gemma Teamet al\.\(2025\)Gemma Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova,et al\.Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786),[Document](https://dx.doi.org/10.48550/arXiv.2503.19786)Cited by:[Appendix E](https://arxiv.org/html/2610.07237#A5.p1.1)\.
- Gerstgrasseret al\.\(2024\)M\. Gerstgrasser, R\. Schaeffer, A\. Dey, R\. Rafailov, H\. Sleight, J\. Hughes, T\. Korbak, R\. Agrawal, D\. Pai, A\. Gromov, D\. A\. Roberts, D\. Yang, D\. L\. Donoho, and S\. KoyejoIs model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data\.Note:arXiv preprintExternal Links:2404\.01413,[Link](https://arxiv.org/abs/2404.01413),[Document](https://dx.doi.org/10.48550/arXiv.2404.01413)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p3.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px3.p1.1)\.
- Guoet al\.\(2024\)D\. Guo, Q\. Zhu, D\. Yang, Z\. Xie, K\. Dong, W\. Zhang, G\. Chen, X\. Bi, Y\. Wu, Y\. K\. Li, F\. Luo, Y\. Xiong, and W\. LiangDeepSeek\-Coder: when the large language model meets programming—the rise of code intelligence\.External Links:2401\.14196,[Link](https://arxiv.org/abs/2401.14196)Cited by:[Appendix E](https://arxiv.org/html/2610.07237#A5.p1.1)\.
- Haoet al\.\(2026\)G\. Hao, Y\. Shang, Y\. Long, Z\. Zhao, and H\. LiangSelf\-policy distillation via capability\-selective subspace projection\.Note:arXiv preprintExternal Links:2605\.22675,[Link](https://arxiv.org/abs/2605.22675),[Document](https://dx.doi.org/10.48550/arXiv.2605.22675)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p4.1),[§3\.1](https://arxiv.org/html/2610.07237#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.07237#S4.SS1.SSS0.Px2.p1.1)\.
- Heet al\.\(2026\)Y\. He, S\. Kaur, A\. Bhaskar, Y\. Yang, J\. Liu, N\. Ri, L\. Fowl, A\. Panigrahi, D\. Chen, and S\. AroraSelf\-distillation zero: self\-revision turns binary rewards into dense supervision\.External Links:2604\.12002,[Link](https://arxiv.org/abs/2604.12002)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, S\. Basart, S\. Kadavath, M\. Mazeika, A\. Arora, E\. Guo, C\. Burns, S\. Puranik, H\. He, D\. Song, and J\. SteinhardtMeasuring coding challenge competence with APPS\.External Links:2105\.09938,[Link](https://arxiv.org/abs/2105.09938)Cited by:[§4\.5](https://arxiv.org/html/2610.07237#S4.SS5.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9),2106\.09685Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p4.1),[§2\.2](https://arxiv.org/html/2610.07237#S2.SS2.p2.1),[§3\.3](https://arxiv.org/html/2610.07237#S3.SS3.p2.1)\.
- Huet al\.\(2026\)Z\. Hu, Y\. Wang, Y\. He, J\. Wu, Y\. Zhao, S\. Ng, C\. Breazeal, A\. T\. Luu, H\. W\. Park, and B\. HooiRewarding the rare: uniqueness\-aware RL for creative problem solving in LLMs\.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp\. 39765–39790\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1982),[Link](https://aclanthology.org/2026.findings-acl.1982/)Cited by:[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px3.p1.1)\.
- Huiet al\.\(2024\)B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu, K\. Dang, Y\. Fan, Y\. Zhang, A\. Yang, R\. Men, F\. Huang, B\. Zheng, Y\. Miao, S\. Quan, Y\. Feng, X\. Ren, X\. Ren, J\. Zhou, and J\. LinQwen2\.5\-Coder technical report\.Note:arXiv preprintExternal Links:2409\.12186,[Link](https://arxiv.org/abs/2409.12186),[Document](https://dx.doi.org/10.48550/arXiv.2409.12186)Cited by:[§4\.1](https://arxiv.org/html/2610.07237#S4.SS1.SSS0.Px1.p1.1)\.
- Hurlbert \(1971\)S\. H\. HurlbertThe nonconcept of species diversity: a critique and alternative parameters\.Ecology52\(4\),pp\. 577–586\.External Links:[Document](https://dx.doi.org/10.2307/1934145),[Link](https://esajournals.onlinelibrary.wiley.com/doi/10.2307/1934145)Cited by:[§A\.2](https://arxiv.org/html/2610.07237#A1.SS2.p1.2),[§2\.3](https://arxiv.org/html/2610.07237#S2.SS3.p2.1)\.
- Jainet al\.\(2024\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.External Links:2403\.07974,[Link](https://arxiv.org/abs/2403.07974)Cited by:[Appendix E](https://arxiv.org/html/2610.07237#A5.p1.1)\.
- Liet al\.\(2022\)Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. de Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov, J\. Molloy, D\. J\. Mankowitz, E\. Sutherland Robson, P\. Kohli, N\. de Freitas, K\. Kavukcuoglu, and O\. VinyalsCompetition\-level code generation with AlphaCode\.External Links:2203\.07814,[Document](https://dx.doi.org/10.1126/science.abq1158),[Link](https://arxiv.org/abs/2203.07814)Cited by:[Appendix E](https://arxiv.org/html/2610.07237#A5.p1.1),[§1](https://arxiv.org/html/2610.07237#S1.p2.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html),2305\.01210Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p2.1),[§4\.5](https://arxiv.org/html/2610.07237#S4.SS5.p1.1)\.
- Nicolicioiuet al\.\(2026\)A\. L\. Nicolicioiu, M\. Pezeshki, and A\. CourvilleOn\-policy self\-distillation with sampled demonstrations reduces output diversity\.External Links:2606\.26091,[Link](https://arxiv.org/abs/2606.26091v1)Cited by:[Appendix I](https://arxiv.org/html/2610.07237#A9.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.07237#S1.p3.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px3.p1.1)\.
- Parikh and Boyd \(2014\)N\. Parikh and S\. BoydProximal algorithms\.Foundations and Trends in Optimization1\(3\)\.External Links:[Document](https://dx.doi.org/10.1561/2400000003),[Link](https://web.stanford.edu/~boyd/papers/prox_algs.html)Cited by:[§3\.2](https://arxiv.org/html/2610.07237#S3.SS2.p1.3)\.
- Raoet al\.\(2023\)K\. Rao, L\. Jiang, V\. Pyatkin, Y\. Gu, N\. Tandon, N\. Dziri, F\. Brahman, and Y\. ChoiWhat makes it ok to set a fire? iterative self\-distillation of contexts and rationales for disambiguating defeasible social and moral situations\.InFindings of the Association for Computational Linguistics: EMNLP 2023,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.812),[Link](https://aclanthology.org/2023.findings-emnlp.812/)Cited by:[Appendix I](https://arxiv.org/html/2610.07237#A9.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.
- Sanget al\.\(2026\)H\. Sang, Y\. Xu, Z\. Zhou, R\. He, Z\. Wang, and J\. SunCRISP: compressed reasoning via iterative self\-policy distillation\.External Links:2603\.05433,[Link](https://arxiv.org/abs/2603.05433v7)Cited by:[Appendix I](https://arxiv.org/html/2610.07237#A9.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.
- Shumailovet al\.\(2024\)I\. Shumailov, Z\. Shumaylov, Y\. Zhao, N\. Papernot, R\. Anderson, and Y\. GalAI models collapse when trained on recursively generated data\.Nature631,pp\. 755–759\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07566-y),[Link](https://www.nature.com/articles/s41586-024-07566-y)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p3.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px3.p1.1)\.
- Singhet al\.\(2024\)A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu, J\. Harrison, J\. Lee, K\. Xu, A\. Parisi, A\. Kumar, A\. Alemi, A\. Rizkowsky, A\. Nova, B\. Adlam, B\. Bohnet, G\. Elsayed, H\. Sedghi, I\. Mordatch, I\. Simpson, I\. Gur, J\. Snoek, J\. Pennington, J\. Hron, K\. Kenealy, K\. Swersky, K\. Mahajan, L\. Culp, L\. Xiao, M\. L\. Bileschi, N\. Constant, R\. Novak, R\. Liu, T\. Warkentin, Y\. Qian, Y\. Bansal, E\. Dyer, B\. Neyshabur, J\. Sohl\-Dickstein, and N\. FiedelBeyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research\.External Links:2312\.06585,[Link](https://arxiv.org/abs/2312.06585)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. QiuQwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388),[Document](https://dx.doi.org/10.48550/arXiv.2505.09388)Cited by:[Appendix E](https://arxiv.org/html/2610.07237#A5.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. GoodmanSTaR: bootstrapping reasoning with reasoning\.External Links:2203\.14465,[Link](https://arxiv.org/abs/2203.14465)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)R\. Zhang, R\. H\. Bai, H\. Zheng, N\. Jaitly, R\. Collobert, and Y\. ZhangEmbarrassingly simple self\-distillation improves code generation\.Note:arXiv preprintExternal Links:2604\.01193,[Link](https://arxiv.org/abs/2604.01193),[Document](https://dx.doi.org/10.48550/arXiv.2604.01193)Cited by:[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.07237#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.07237#S4.SS1.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)X\. Zhang, C\. Zhou, L\. Hu, L\. Zhang, X\. Chen, H\. Fu, Y\. Yang, and M\. ZhangSCoder: progressive self\-distillation for bootstrapping small\-scale data synthesizers to empower code LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 20825–20841\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1136),[Link](https://aclanthology.org/2025.findings-emnlp.1136/)Cited by:[Appendix I](https://arxiv.org/html/2610.07237#A9.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.07237#S1.p1.1),[§5](https://arxiv.org/html/2610.07237#S5.SS0.SSS0.Px2.p1.1)\.

## Appendix AMetrics, estimators, and probability analysis

### A\.1Correctness and conditional implementation distributions

For a fixed task, letV=vx​\(Y\)V=v\_\{x\}\(Y\)andJ=ϕ⁡\(Y\)J=\\phi\(Y\)\. Equation \([2](https://arxiv.org/html/2610.07237#S2.E2)\) is simplyPr⁡\(V=1,J=j\)=Pr⁡\(V=1\)​Pr⁡\(J=j∣V=1\)\\Pr\(V=1,J=j\)=\\Pr\(V=1\)\\Pr\(J=j\\mid V=1\)\. Ifa=0a=0, unconditional correct richness is zero andqqis undefined\. An observed pool with no successes need not havea=0a=0; its matched\-correct estimator is unavailable\. For countably many classes the occupancy sums converge, because1−\(1−p\)k≤k​p1\-\(1\-p\)^\{k\}\\leq kpgivesCk≤k​aC\_\{k\}\\leq kaandDb≤bD\_\{b\}\\leq b\. These quantities count expected observed classes at the stated budget, not all possible implementations\.

#### Proof of Proposition[1](https://arxiv.org/html/2610.07237#Thmproposition1)\.

LetIjI\_\{j\}indicate observing at least one correct output from classjjinkkindependent draws\. Its expectation is1−\(1−a​qj\)k1\-\(1\-aq\_\{j\}\)^\{k\}\. Linearity of expectation givesCkC\_\{k\}\. The same argument forbbdraws from the conditional correct distribution yieldsDbD\_\{b\}\. Allkkdraws fail with probability\(1−a\)k\(1\-a\)^\{k\}, so pass@kkdepends only onaa\. Forq=\(1,0,…\)q=\(1,0,\\ldots\),Db=1D\_\{b\}=1\. For uniform mass onJ≥2J\\geq 2classes,Db=J⁡\[1−\(1−1/J\)b\]\>1D\_\{b\}=J\[1\-\(1\-1/J\)^\{b\}\]\>1whenb≥2b\\geq 2\. Hence identical correctness at every budget does not identify conditional breadth\.□\\square

More generally, spreading mass from a larger to a smaller conditional class, without reversing their order, cannot decreaseDbD\_\{b\}:f⁡\(q\)=1−\(1−q\)bf\(q\)=1\-\(1\-q\)^\{b\}is concave forb≥2b\\geq 2\. The same applies toCkC\_\{k\}at fixedaaandk≥2k\\geq 2\. This describes a change in a probability distribution; it does not assert that the spectral operator induces that change in every network\.

Fora\>0a\>0, the two richness endpoints are linked by the number of successful draws\. LetB∼Binomial⁡\(k,a\)B\\sim\\operatorname\{Binomial\}\(k,a\)and setD0​\(q\)=0D\_\{0\}\(q\)=0\. Conditioning onBBgives

Ck​\(a,q\)=𝔼⁡\[DB​\(q\)\]=∑b=0k\(kb\)​ab​\(1−a\)k−b​Db​\(q\)\.C\_\{k\}\(a,q\)=\\mathbb\{E\}\\bigl\[D\_\{B\}\(q\)\\bigr\]=\\sum\_\{b=0\}^\{k\}\\binom\{k\}\{b\}a^\{b\}\(1\-a\)^\{k\-b\}D\_\{b\}\(q\)\.\(11\)GivenB=bB=b, thebbcorrect classes are drawn fromqq, so their expected richness isDb​\(q\)D\_\{b\}\(q\)\. Thus total\-budget richness combines how many correct samples are obtained with how broadly those samples are distributed\. Matching the correct\-draw budget isolates the latter quantity on the eligible tasks\.

### A\.2Finite\-sample estimators

Suppose a task hasnngenerated outputs,mmof which pass its verifier, and correct\-class countsmjm\_\{j\}with∑jmj=m\\sum\_\{j\}m\_\{j\}=m\. We compute

pass​@​k^\\displaystyle\\widehat\{\\mathrm\{pass@\}k\}=1−\(n−mk\)\(nk\),\\displaystyle=1\-\\frac\{\\binom\{n\-m\}\{k\}\}\{\\binom\{n\}\{k\}\},1≤k≤n,\\displaystyle 1\\leq k\\leq n,\(12\)C^k\\displaystyle\\widehat\{C\}\_\{k\}=∑j\[1−\(n−mjk\)\(nk\)\],\\displaystyle=\\sum\_\{j\}\\left\[1\-\\frac\{\\binom\{n\-m\_\{j\}\}\{k\}\}\{\\binom\{n\}\{k\}\}\\right\],1≤k≤n,\\displaystyle 1\\leq k\\leq n,\(13\)D^b\\displaystyle\\widehat\{D\}\_\{b\}=∑j\[1−\(m−mjb\)\(mb\)\],\\displaystyle=\\sum\_\{j\}\\left\[1\-\\frac\{\\binom\{m\-m\_\{j\}\}\{b\}\}\{\\binom\{m\}\{b\}\}\\right\],1≤b≤m\.\\displaystyle 1\\leq b\\leq m\.\(14\)We set\(uv\)=0\\binom\{u\}\{v\}=0for nonnegative integersu<vu<v\. Equation \([12](https://arxiv.org/html/2610.07237#A1.E12)\) is the standard pass@kkestimator\([Chen et al\., 2021](https://arxiv.org/html/2610.07237#bib.bib8)\)\. Equations \([13](https://arxiv.org/html/2610.07237#A1.E13)\)–\([14](https://arxiv.org/html/2610.07237#A1.E14)\) are rarefaction estimators\([Hurlbert, 1971](https://arxiv.org/html/2610.07237#bib.bib15)\)\. A uniformly chosen subset ofkkoutputs misses correct classjjwith probability\(n−mjk\)/\(nk\)\\binom\{n\-m\_\{j\}\}\{k\}/\\binom\{n\}\{k\}\. Summing its complement gives the exact expected subset richness conditional on the recorded pool; under independent sampling it estimates the corresponding population quantity\. Conditioning on correct outputs gives the analogous calculation forDbD\_\{b\}\.

Atk=nk=n,C^n\\widehat\{C\}\_\{n\}is the number of observed correct classes\. Atk=1k=1,C^1=m/n=pass​@​1^\\widehat\{C\}\_\{1\}=m/n=\\widehat\{\\mathrm\{pass@\}1\}\. Atb=1b=1,D1=1D\_\{1\}=1wherever defined\. Thus neitherC1C\_\{1\}norD1D\_\{1\}distinguishes how probability is distributed among multiple correct classes\. The plotted budgets are those present in the archived estimates; we do not reconstruct a missing pass@32 value by interpolation\.

### A\.3AST fingerprints and concentration

Python programs are extracted using the recorded first\-fenced\-block rule\. The primary fingerprint parses the extracted program and conservatively normalizes its AST\. Formatting and docstrings are removed; local identifiers are normalized only when scope handling is unambiguous\. Function names, external identifiers, literals, and uncertain scopes are retained\. Consequently, distinct fingerprints can reflect structural implementation variants within one algorithm\. Conversely, a fingerprint does not establish functional equivalence beyond the recorded tests\. Parse failures do not silently become new correct classes\. Secondary exact\-program and control\-flow fingerprints provide different granularities\.

An additional recorded concentration statistic is correct\-output Simpson diversity,

S^=1−∑jmj​\(mj−1\)m⁡\(m−1\),m≥2\.\\widehat\{S\}=1\-\\frac\{\\sum\_\{j\}m\_\{j\}\(m\_\{j\}\-1\)\}\{m\(m\-1\)\},\\qquad m\\geq 2\.\(15\)It estimates the probability that two independently drawn correct outputs fall in different classes\. Its aggregate uses tasks with enough correct samples and complete labels\. It complements richness by emphasizing concentration, while sharing the limitations of the class definition\. Exact\-program, AST, and control\-flow metrics need not move equally; the coarser control\-flow comparisons do not establish a uniform increase in algorithmic families\.

### A\.4Task aggregation and uncertainty

Pass@kkandCkC\_\{k\}are macro\-averaged over the full task set\. A marginalDbD\_\{b\}mean usesℰb​\(M\)=\{x:mx​\(M\)≥b​and all correct outputs have AST labels\}\\mathcal\{E\}\_\{b\}\(M\)=\\\{x:m\_\{x\}\(M\)\\geq b\\text\{ and all correct outputs have AST labels\}\\\}for modelMMand reports its size\. A paired contrast betweenM1,M2M\_\{1\},M\_\{2\}averages task\-level differences onℰb​\(M1\)∩ℰb​\(M2\)\\mathcal\{E\}\_\{b\}\(M\_\{1\}\)\\cap\\mathcal\{E\}\_\{b\}\(M\_\{2\}\)\. Subtracting marginal means with different eligible sets is not this paired effect\. Figure[5](https://arxiv.org/html/2610.07237#A4.F5)b pairs each round with initialization atb=8b=8; its eligible population can change between rounds\. It is a pointwise trajectory, not a single fixed cohort\.

The five\-round MBPP intervals use 2,000 task\-bootstrap resamples\. Transfer intervals also use 2,000 resamples, pairing by exact task ID and conditioning on common eligibility\. Percentile endpoints are the 2\.5th and 97\.5th percentiles\. The analysis script records the means, intervals, populations, and input hashes\. These pointwise intervals describe evaluation\-task uncertainty conditional on the saved training runs, not variation across independent training runs; no multiplicity adjustment is applied\.

## Appendix BOperator derivation and mathematical scope

### B\.1Reference loss and positionwise sensitivity

Let𝒯i\\mathcal\{T\}\_\{i\}contain the nonpadding completion targets of referenceii\. The scalar differentiated during calibration is

ℓi\(θ\)=−1\|𝒯i\|∑u∈𝒯ilogpθ\(yi​u⋆∣xi,yi,<u⋆\)\.\\ell\_\{i\}\(\\theta\)=\-\\frac\{1\}\{\|\\mathcal\{T\}\_\{i\}\|\}\\sum\_\{u\\in\\mathcal\{T\}\_\{i\}\}\\log p\_\{\\theta\}\(y\_\{iu\}^\{\\star\}\\mid x\_\{i\},y\_\{i,<u\}^\{\\star\}\)\.\(16\)This loss is computed separately for each reference before collecting gradients\. For selected moduleℓ\\ell, lethi​u\(ℓ\)∈ℝdh\_\{iu\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\}be its native linear output at positionuu, written as a column here, and definegi​u\(ℓ\)=∇hi​u\(ℓ\)ℓi​\(θt\)g\_\{iu\}^\{\(\\ell\)\}=\\nabla\_\{h\_\{iu\}^\{\(\\ell\)\}\}\\ell\_\{i\}\(\\theta\_\{t\}\)\. The second moment in Equation \([4](https://arxiv.org/html/2610.07237#S3.E4)\) averages the outer products of these gradients over every nonpadding position, including prompt and zero\-gradient positions\. Completion masking selects the loss targets; it does not prevent prompt activations from influencing subsequent completion predictions\.

The statistic has a direct directional interpretation\. For a unit vectorvv, perturbing one selected output position byϵ​v\\epsilon vgives

ℓi​\(hi​u\(ℓ\)\+ϵ​v\)−ℓi​\(hi​u\(ℓ\)\)=ϵ​gi​u\(ℓ\)⊤​v\+o⁡\(ϵ\),\\ell\_\{i\}\\bigl\(h\_\{iu\}^\{\(\\ell\)\}\+\\epsilon v\\bigr\)\-\\ell\_\{i\}\\bigl\(h\_\{iu\}^\{\(\\ell\)\}\\bigr\)=\\epsilon\\,g\_\{iu\}^\{\(\\ell\)\\top\}v\+o\(\\epsilon\),\(17\)where the notation denotes an intervention on that intermediate output, with the remaining computation propagated normally\. Consequently,

v⊤​Ct,ℓ​v=1Mℓ​∑i,u,nonpad\(gi​u\(ℓ\)⊤​v\)2\.v^\{\\top\}C\_\{t,\\ell\}v=\\frac\{1\}\{M\_\{\\ell\}\}\\sum\_\{i,u,\\mathrm\{nonpad\}\}\\bigl\(g\_\{iu\}^\{\(\\ell\)\\top\}v\\bigr\)^\{2\}\.\(18\)This averages squared first\-order responses to individual positionwise perturbations\. It does not include the cross\-position terms arising from a coherent simultaneous perturbation of several outputs\. Opposite gradient signs contribute positively, soCt,ℓC\_\{t,\\ell\}measures sensitivity rather than a signed descent direction\. It is an uncentered gradient second moment, not a Hessian of the reference loss\.

Whenλmax​\(Ct,ℓ\)\>0\\lambda\_\{\\max\}\(C\_\{t,\\ell\}\)\>0, normalization produces0⪯C¯t,ℓ⪯𝐈0\\preceq\\overline\{C\}\_\{t,\\ell\}\\preceq\\mathbf\{I\}and a maximum eigenvalue of one\. Empty or zero\-signal calibration is rejected because the normalization would otherwise be undefined\. The same anchor is reused, but the gradients and normalized statistic are recomputed at the current student parameters\.

### B\.2Proof of Proposition[2](https://arxiv.org/html/2610.07237#Thmproposition2): proximal solution and bounds

For fixedC¯\\overline\{C\}and row activationhh, write the objective in Equation \([5](https://arxiv.org/html/2610.07237#S3.E5)\) as

Fh​\(z\)=12​‖z−h‖22\+τ2​z​\(𝐈−C¯\)​z⊤\.F\_\{h\}\(z\)=\\frac\{1\}\{2\}\\\|z\-h\\\|\_\{2\}^\{2\}\+\\frac\{\\tau\}\{2\}z\(\\mathbf\{I\}\-\\overline\{C\}\)z^\{\\top\}\.\(19\)Its Hessian isH=𝐈\+τ⁡\(𝐈−C¯\)H=\\mathbf\{I\}\+\\tau\(\\mathbf\{I\}\-\\overline\{C\}\)\. Ifμj∈\[0,1\]\\mu\_\{j\}\\in\[0,1\]are the eigenvalues ofC¯\\overline\{C\}, the eigenvalues ofHHare1\+τ⁡\(1−μj\)∈\[1,1\+τ\]1\+\\tau\(1\-\\mu\_\{j\}\)\\in\[1,1\+\\tau\]\. ThusFhF\_\{h\}is strongly convex and has a unique minimizer\. Its first\-order condition is

z−h\+τ​z​\(𝐈−C¯\)=0,z​H=h\.z\-h\+\\tau z\(\\mathbf\{I\}\-\\overline\{C\}\)=0,\\qquad zH=h\.\(20\)It follows thatz=h​H−1=h​Tτz=hH^\{\-1\}=hT\_\{\\tau\}\. DiagonalizingHHgives

Tτ=U​diag⁡\(11\+τ⁡\(1−μj\)\)​U⊤,T\_\{\\tau\}=U\\operatorname\{diag\}\\\!\\left\(\\frac\{1\}\{1\+\\tau\(1\-\\mu\_\{j\}\)\}\\right\)U^\{\\top\},\(21\)which is Equation \([6](https://arxiv.org/html/2610.07237#S3.E6)\)\.

All gains are positive at finiteτ\\tau, and lie in\[1/\(1\+τ\),1\]\[1/\(1\+\\tau\),1\]\. BecauseTτT\_\{\\tau\}is symmetric positive definite, its singular values are precisely these gains\. For any row vectorδ\\delta,

‖δ‖21\+τ≤‖δ​Tτ‖2≤‖δ‖2\.\\frac\{\\\|\\delta\\\|\_\{2\}\}\{1\+\\tau\}\\leq\\\|\\delta T\_\{\\tau\}\\\|\_\{2\}\\leq\\\|\\delta\\\|\_\{2\}\.\(22\)Moreover,

‖Tτ−𝐈‖2=maxj⁡τ⁡\(1−μj\)1\+τ⁡\(1−μj\)≤τ1\+τ\.\\\|T\_\{\\tau\}\-\\mathbf\{I\}\\\|\_\{2\}=\\max\_\{j\}\\frac\{\\tau\(1\-\\mu\_\{j\}\)\}\{1\+\\tau\(1\-\\mu\_\{j\}\)\}\\leq\\frac\{\\tau\}\{1\+\\tau\}\.\(23\)These establish Equation \([7](https://arxiv.org/html/2610.07237#S3.E7)\)\. The condition number is at most1\+τ1\+\\tau, andT0=𝐈T\_\{0\}=\\mathbf\{I\}\.

### B\.3Calibration stability

For normalized statisticsA,BA,Bwith spectra in\[0,1\]\[0,1\], writeHA=𝐈\+τ⁡\(𝐈−A\)H\_\{A\}=\\mathbf\{I\}\+\\tau\(\\mathbf\{I\}\-A\)andHB=𝐈\+τ⁡\(𝐈−B\)H\_\{B\}=\\mathbf\{I\}\+\\tau\(\\mathbf\{I\}\-B\)\. The inverse identity gives

Tτ​\(A\)−Tτ​\(B\)\\displaystyle T\_\{\\tau\}\(A\)\-T\_\{\\tau\}\(B\)=HA−1​\(HB−HA\)​HB−1\\displaystyle=H\_\{A\}^\{\-1\}\(H\_\{B\}\-H\_\{A\}\)H\_\{B\}^\{\-1\}\(24\)=τ​Tτ​\(A\)​\(A−B\)​Tτ​\(B\)\.\\displaystyle=\\tau T\_\{\\tau\}\(A\)\(A\-B\)T\_\{\\tau\}\(B\)\.\(25\)Both outer factors have operator norm at most one\. Submultiplicativity therefore yields

‖Tτ​\(A\)−Tτ​\(B\)‖2≤τ​‖A−B‖2\.\\\|T\_\{\\tau\}\(A\)\-T\_\{\\tau\}\(B\)\\\|\_\{2\}\\leq\\tau\\\|A\-B\\\|\_\{2\}\.\(26\)The argument requires neither an eigengap nor shared eigenvectors\. This completes the proof of Proposition[2](https://arxiv.org/html/2610.07237#Thmproposition2)\.□\\square

The result concerns normalized statistics\. To relate it to raw second moments, letC,D⪰0C,D\\succeq 0withc=λmax​\(C\)≥γ\>0c=\\lambda\_\{\\max\}\(C\)\\geq\\gamma\>0andd=λmax​\(D\)≥γ\>0d=\\lambda\_\{\\max\}\(D\)\\geq\\gamma\>0\. Since‖D‖2=d\\\|D\\\|\_\{2\}=dand\|c−d\|≤‖C−D‖2\|c\-d\|\\leq\\\|C\-D\\\|\_\{2\},

‖Cc−Dd‖2\\displaystyle\\left\\\|\\frac\{C\}\{c\}\-\\frac\{D\}\{d\}\\right\\\|\_\{2\}≤‖C−D‖2c\+‖D‖2​\|1c−1d\|\\displaystyle\\leq\\frac\{\\\|C\-D\\\|\_\{2\}\}\{c\}\+\\\|D\\\|\_\{2\}\\left\|\\frac\{1\}\{c\}\-\\frac\{1\}\{d\}\\right\|\(27\)=‖C−D‖2\+\|d−c\|c≤2​‖C−D‖2γ\.\\displaystyle=\\frac\{\\\|C\-D\\\|\_\{2\}\+\|d\-c\|\}\{c\}\\leq\\frac\{2\\\|C\-D\\\|\_\{2\}\}\{\\gamma\}\.\(28\)Combining the two estimates gives

‖Tτ​\(Cc\)−Tτ​\(Dd\)‖2≤2​τγ​‖C−D‖2\.\\left\\\|T\_\{\\tau\}\\\!\\left\(\\frac\{C\}\{c\}\\right\)\-T\_\{\\tau\}\\\!\\left\(\\frac\{D\}\{d\}\\right\)\\right\\\|\_\{2\}\\leq\\frac\{2\\tau\}\{\\gamma\}\\\|C\-D\\\|\_\{2\}\.\(29\)This is a deterministic perturbation bound under a nonvanishing normalization scale\. No sampling\-concentration claim for the finite\-anchor estimator is required\.

### B\.4Relative reference sensitivity

For a nonzero row vectorhh, define its sensitivity per unit squared norm as

sC¯​\(h\)=h​C¯​h⊤‖h‖22\.s\_\{\\overline\{C\}\}\(h\)=\\frac\{h\\overline\{C\}h^\{\\top\}\}\{\\\|h\\\|\_\{2\}^\{2\}\}\.\(30\)Equation \([8](https://arxiv.org/html/2610.07237#S3.E8)\) states that modulation cannot decrease this quantity\. To prove it, writeC¯=U​diag⁡\(μj\)​U⊤\\overline\{C\}=U\\operatorname\{diag\}\(\\mu\_\{j\}\)U^\{\\top\}and set

wj=\(h​uj\)2‖h‖22,rj=1\[1\+τ⁡\(1−μj\)\]2\.w\_\{j\}=\\frac\{\(hu\_\{j\}\)^\{2\}\}\{\\\|h\\\|\_\{2\}^\{2\}\},\\qquad r\_\{j\}=\\frac\{1\}\{\[1\+\\tau\(1\-\\mu\_\{j\}\)\]^\{2\}\}\.\(31\)The weights sum to one andrj\>0r\_\{j\}\>0\. Direct substitution yields

sC¯​\(h​Tτ\)−sC¯​\(h\)\\displaystyle s\_\{\\overline\{C\}\}\(hT\_\{\\tau\}\)\-s\_\{\\overline\{C\}\}\(h\)=𝔼w​\[μ​r\]𝔼w​\[r\]−𝔼w​\[μ\]\\displaystyle=\\frac\{\\mathbb\{E\}\_\{w\}\[\\mu r\]\}\{\\mathbb\{E\}\_\{w\}\[r\]\}\-\\mathbb\{E\}\_\{w\}\[\\mu\]\(32\)=Covw⁡\(μ,r\)𝔼w​\[r\]\\displaystyle=\\frac\{\\operatorname\{Cov\}\_\{w\}\(\\mu,r\)\}\{\\mathbb\{E\}\_\{w\}\[r\]\}\(33\)=∑i,jwi​wj​\(μi−μj\)​\(ri−rj\)2​𝔼w​\[r\]≥0\.\\displaystyle=\\frac\{\\sum\_\{i,j\}w\_\{i\}w\_\{j\}\(\\mu\_\{i\}\-\\mu\_\{j\}\)\(r\_\{i\}\-r\_\{j\}\)\}\{2\\mathbb\{E\}\_\{w\}\[r\]\}\\geq 0\.\(34\)The final inequality follows becauserjr\_\{j\}is nondecreasing inμj\\mu\_\{j\}\. Forτ\>0\\tau\>0, it is strictly increasing, so equality holds exactly when all eigendirections carrying nonzero energy fromhhshare the same eigenvalue\. Atτ=0\\tau=0, modulation is the identity\.

The relative increase does not require an increase in absolute sensitivity energy\. Indeed, since0<rj≤10<r\_\{j\}\\leq 1,

\(h​Tτ\)​C¯​\(h​Tτ\)⊤=∑jμj​rj​\(h​uj\)2≤∑jμj​\(h​uj\)2=h​C¯​h⊤\.\(hT\_\{\\tau\}\)\\overline\{C\}\(hT\_\{\\tau\}\)^\{\\top\}=\\sum\_\{j\}\\mu\_\{j\}r\_\{j\}\(hu\_\{j\}\)^\{2\}\\leq\\sum\_\{j\}\\mu\_\{j\}\(hu\_\{j\}\)^\{2\}=h\\overline\{C\}h^\{\\top\}\.\(35\)Thus modulation does not increase activation norm or absolute sensitivity energy, while sensitivity per unit squared norm does not decrease\. This is a property of a local quadratic statistic; it does not assert a decrease in the original reference loss\.

### B\.5Folding, finite precision, and the projection control

For row activations and a linear module with weightWWand biasbb,

\(x​W⊤\+b⊤\)​T=x​\(T⊤​W\)⊤\+\(T⊤​b\)⊤\.\(xW^\{\\top\}\+b^\{\\top\}\)T=x\(T^\{\\top\}W\)^\{\\top\}\+\(T^\{\\top\}b\)^\{\\top\}\.\(36\)Thus replacingWWandbbbyT⊤​WT^\{\\top\}WandT⊤​bT^\{\\top\}bimplements the same local map\. This is the algebra underlying temporary weight folding\. The implementation computes products in float32 and casts back to the model dtype; the exact invertibility statements concern real arithmetic\. The insertion point is the native K/V linear output before rotary position operations and any subsequent key normalization\. Grouped\-query dimensions come from each selected module’s actual output dimension\. The original weights are saved and restored before learning\.

The projection design usesPr=U1:rU1:r⊤P\_\{r\}=U\_\{1:r\}U\_\{1:r\}^\{\\top\}, with eigenvectors ordered by decreasing eigenvalue andrrequal to half the module output dimension in the completed run\. It preserves retained directions and nulls the remainder\. Atτ=1\\tau=1,Spectrum’s gains instead lie in\[1/2,1\]\[1/2,1\]\. The two designs differ in spectrum and effective strength, so this comparison does not match their distance from the identity\.

For completeness, the infinite\-strength limit of the proximal family is

limτ→∞Tτ=∑j:μj=1ujuj⊤\.\\lim\_\{\\tau\\to\\infty\}T\_\{\\tau\}=\\sum\_\{j:\\mu\_\{j\}=1\}u\_\{j\}u\_\{j\}^\{\\top\}\.\(37\)It projects onto the maximum\-eigenvalue eigenspace of the normalized statistic, not an arbitrary prescribed top\-rrspace\. Finite\-strength full rank is therefore the relevant property for the completed experiment\.

### B\.6Local operator properties and learned solution distributions

Positive gains preserve distinctions between inputs to the local linear map\. Subsequent attention, nonlinear layers, vocabulary projection, stochastic decoding, and finite\-data learning determine whether those distinctions affect output classes\. The operator results establish bounded modulation, calibration stability, and relative sensitivity enrichment\. The measurements ofCkC\_\{k\}andDbD\_\{b\}establish how the resulting students distribute their correct outputs\.

A separate idealization clarifies why learning from self\-generated sequences is not automatically an improvement step\. For a fixed promptxx, consider the completion\-only, unnormalized sequence\-NLL objective

Lt\(θ\)=−𝔼Y∼pθt\(⋅∣x\)\[logpθ\(Y∣x\)\]\.L\_\{t\}\(\\theta\)=\-\\mathbb\{E\}\_\{Y\\sim p\_\{\\theta\_\{t\}\}\(\\cdot\\mid x\)\}\\bigl\[\\log p\_\{\\theta\}\(Y\\mid x\)\\bigr\]\.\(38\)When the sequence distribution is normalized and differentiation can be interchanged with summation, its gradient at the generating parameters is

∇θLt\(θ\)\|θ=θt=−∑ypθt\(y∣x\)∇θlogpθ\(y∣x\)\|θ=θt=−∇θ∑ypθ\(y∣x\)\|θ=θt=0\.\\left\.\\nabla\_\{\\theta\}L\_\{t\}\(\\theta\)\\right\|\_\{\\theta=\\theta\_\{t\}\}=\-\\sum\_\{y\}p\_\{\\theta\_\{t\}\}\(y\\mid x\)\\left\.\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(y\\mid x\)\\right\|\_\{\\theta=\\theta\_\{t\}\}=\-\\left\.\\nabla\_\{\\theta\}\\sum\_\{y\}p\_\{\\theta\}\(y\\mid x\)\\right\|\_\{\\theta=\\theta\_\{t\}\}=0\.\(39\)This identity assumes exact model\-distribution sampling and the complete sequence log likelihood\. The recorded learner in Equation \([10](https://arxiv.org/html/2610.07237#S3.E10)\) includes prompt targets and uses example\-wise token averaging; a sequence\-dependent length normalization need not preserve the zero\-gradient identity\. The practical loop also uses a decoding distribution, finite corpora, a fixed optimization schedule, and, forSpectrum, reference\-calibrated generation\. Its improvement and retention trajectories are measured empirically\.

## Appendix CAlgorithm, information flow, and protocol

### C\.1A complete round

The following sequence implements a complete round ofSpectrum, separating the temporary generator from the native learner\.

1. 1\.Fix synthesis prompts𝒬\\mathcal\{Q\}, reference anchor𝒜\\mathcal\{A\}, held\-out evaluation tasks, decoding settings, and the initial checkpointθ0\\theta\_\{0\}before the loop\.
2. 2\.At roundtt, calculate reference\-completion loss underθt\\theta\_\{t\}, collect selected K/V\-output gradients, and formC¯t,ℓ\\overline\{C\}\_\{t,\\ell\}on the unchanged anchor\.
3. 3\.ConstructTt,ℓT\_\{t,\\ell\}with the fixed strengthτ\\tauand temporarily fold the operators into the current checkpoint’s native linear weights\.
4. 4\.SampleSScompletions for each synthesis prompt\. Keep every raw record\. No execution outcome, reward, uniqueness judgment, or AST label determines membership or weight in𝒟t\\mathcal\{D\}\_\{t\}\.
5. 5\.Restore the originalθt\\theta\_\{t\}weights\. Fit one LoRA adapter on𝒟t\\mathcal\{D\}\_\{t\}, merge it into the base weights, and saveθt\+1\\theta\_\{t\+1\}\.
6. 6\.Evaluateθt\+1\\theta\_\{t\+1\}through common native inference, storing all outputs and verification labels\. These labels do not affect the next training round\. Repeat from step 2 usingθt\+1\\theta\_\{t\+1\}\.

For Vanilla SD or SSD, the same generate–learn–update loop uses the corresponding standard or decoding\-shaped policy, omitting reference calibration and weight folding\. The anchor itself is never expanded with newly judged samples\. It is reused as supervision at every calibration step, so “fixed” describes the information source rather than an initialization\-only operation\. In the completed study, there is no paired intervention\-on generator diagnostic at every round; the native endpoints establish retained differences after learning, not the fraction of an immediate generator benefit transferred by SFT\.

### C\.2Data and prompts

The MBPP preparation starts from the full official splits\. Exact normalized prompt overlaps with held\-out prompts are removed before partitioning the training pool; the recorded excluded training IDs are 602, 704, and 872\. The resulting sizes are 291 synthesis, 50 calibration, 30 validation, and 500 evaluation tasks\. A Python signature stub is extracted mechanically from the reference AST to specify the interface\. Its implementation and evaluation assertions are not placed in the synthesis/evaluation prompt\. The calibration partition supplies complete reference solutions\.

The fixed MBPP prompt body is:

> \[task description\] Implement this Python function\. Return only the complete implementation\. \[signature stub\]

Brackets identify fields in a template, not missing manuscript results\. The same template is used across methods; it does not request multiple strategies or select a strategy\. Model\-native chat formatting is applied by the tokenizer\. Prompt and token provenance are stored in the experiment records\.

HumanEval\+ follows the recorded EvalPlus task interface and full base\-plus tests, using release v0\.1\.10 with EvalPlus 0\.3\.1\. APPS Intro is a fixed 200\-task evaluation subset using the stored function\-interface adapter\. For stdin/stdout tasks, the prompt requestssolve\(stdin: str\) \-\> str; the evaluator checks the returned output string against supplied cases using whitespace\-token equality\. This adapted interface, task subset, and evaluation budget are part of the result and differ from official benchmark leaderboard settings\. No HumanEval\+ or APPS references recalibrate the frozen MBPP students\.

Table 5:Completed training and evaluation settings\.The final 64\-sample MBPP supplement is a separate sample pool; only 16\-sample measurements exist for all historical rounds\.
### C\.3What the student optimizes

Letssdenote the concatenated tokenized prompt and generated completion, and let𝒯⁡\(s\)\\mathcal\{T\}\(s\)contain its nonpadding causal targets\. The recorded microbatch\-one schedule accumulates example\-level token\-mean losses,

ℓs\(θ\)=−1\|𝒯⁡\(s\)\|∑u∈𝒯⁡\(s\)logpθ\(su∣s<u\),ℒSFT=1\|𝒟t\|∑s∈𝒟tℓs\(θ\)\.\\ell\_\{s\}\(\\theta\)=\-\\frac\{1\}\{\|\\mathcal\{T\}\(s\)\|\}\\sum\_\{u\\in\\mathcal\{T\}\(s\)\}\\log p\_\{\\theta\}\(s\_\{u\}\\mid s\_\{<u\}\),\\qquad\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}=\\frac\{1\}\{\|\\mathcal\{D\}\_\{t\}\|\}\\sum\_\{s\\in\\mathcal\{D\}\_\{t\}\}\\ell\_\{s\}\(\\theta\)\.\(40\)Gradient accumulation follows the recorded minibatch schedule\. Including prompt targets and averaging each example by its own length are explicit choices, distinct from completion\-only calibration and globally token\-weighted corpus optimization\. Every record receives the same outer example weight; token normalization within each record follows its own length\. Correctness and source subspace do not determine training weights\. Truncation at the fixed context limit is a sequence\-length constraint, not quality filtering\. The completed training summaries record zero truncated training examples\.

### C\.4Information budgets and comparators

Vanilla SD and SSD do not use the reference anchor\.Spectrumand the projection control use the same 50 references at each calibration step\. Hence their direct comparison controls access to calibration information, whereas comparing either to Vanilla SD or SSD includes that difference in information use\. SSD here adapts the published decoding idea to the small\-model MBPP budget; it does not reproduce the original larger\-scale results\. UA\-RL, which uses rewards for rare strategies, is discussed as related work but is not a completed numerical comparator in this archive\.

## Appendix DComplete measured results

### D\.1Longitudinal MBPP results at 16 samples

Table[6](https://arxiv.org/html/2610.07237#A4.T6)contains all recorded rounds\. The initial SSD row is an independently sampled evaluation of the same starting model, retained separately from the original trajectory\. Small differences in initialization estimates are sampling/protocol\-stream differences, not different learned checkpoints\. No 64\-sample intermediate\-round results are inferred from these rows\.

Table 6:Every completed native\-student round on MBPP, 16 draws per task\.Full 500\-task means for pass@1 andC16C\_\{16\}; marginalD4D\_\{4\}means with eligibility\. SSD is the separate continuation\.MethodRoundpass@1 \(%\)C16C\_\{16\}D4D\_\{4\}EligibleInitial037\.924\.0163\.310262Vanilla SD138\.423\.8083\.173261Vanilla SD239\.153\.6343\.080269Vanilla SD339\.653\.3722\.895257Vanilla SD440\.793\.1902\.803270Vanilla SD541\.303\.0902\.747268Spectrum137\.843\.8483\.222261Spectrum238\.363\.7683\.159259Spectrum339\.053\.6623\.063262Spectrum439\.783\.5882\.991263Spectrum540\.183\.6062\.997263Projection control137\.923\.6923\.144257Projection control239\.203\.5923\.059267Projection control339\.403\.3142\.889256Projection control441\.093\.1622\.774266Projection control541\.353\.0662\.690262SSD initial037\.813\.9923\.311260SSD137\.803\.8463\.232260SSD236\.813\.7103\.200250SSD336\.603\.7243\.196256SSD437\.093\.7523\.181257SSD535\.963\.7103\.268255
### D\.2Paired retention trajectories

![Refer to caption](https://arxiv.org/html/2610.07237v1/figures/exp1.png)Figure 5:Within\-correct retention over successive updates\.Original 16\-sample MBPP comparison of Vanilla SD andSpectrum\. \(a\)C16C\_\{16\}relative to initialization; \(b\) pairedD8D\_\{8\}changes from initialization on each contrast’s common eligible tasks, with pointwise 95% task\-bootstrap intervals; \(c\) pass@16\. Panels \(a,c\) show point estimates\. The complete three\-design comparison is Figure[3](https://arxiv.org/html/2610.07237#S4.F3)\.
### D\.3Final projection comparison and correct\-count matching

Table 7:Final 64\-sample projection ablation\.Same 500 tasks, fixed reference access, and five\-round training schedule\. MarginalD4D\_\{4\}means are descriptive; eligible tasks are 321 and 323\.Table 8:Paired final MBPP richness contrasts,Spectrumminus Vanilla SD\.Common eligible tasks are selected separately for each correct\-draw budget\. Intervals are the archived 95% task\-bootstrap intervals\.SparseD64D\_\{64\}estimates are available on few fully correct tasks but do not form a primary claim: the pairedSpectrum–Vanilla SD population is only 29 tasks\. At the recorded 16\-sample round\-five endpoint, theSpectrum\-minus\-projection difference in exact\-programD4D\_\{4\}is0\.3120\.312with interval\[0\.253,0\.377\]\[0\.253,0\.377\], and the AST difference is0\.2930\.293\[0\.233,0\.353\]\[0\.233,0\.353\]\. The coarser control\-flow difference is0\.0420\.042\[−0\.011,0\.095\]\[\-0\.011,0\.095\]\. All three contrasts use 251 common eligible tasks and distinguish implementation structure from coarser control\-flow variation\.

### D\.4Transfer contrasts and the projection control

Table 9:Frozen\-student transfer after MBPP learning\.Sixteen samples per task, without transfer training or recalibration\. Success andC16C\_\{16\}average over all tasks\.D4D\_\{4\}is the marginal mean over each row’s eligible tasks; paired common\-task contrasts appear in the text\. Higher values indicate greater success or richness; eligibility is descriptive\.Table 10:Paired transfer effects ofSpectrumrelative to Vanilla SD\.Differences use all tasks for pass@1 andC16C\_\{16\}, and common eligible tasks forD4D\_\{4\}\. Intervals use 2,000 paired task\-bootstrap resamples of the saved task records\. Students receive no transfer training or recalibration\.The projection control’s HumanEval\+ scores are pass@152\.74%52\.74\\%, pass@1684\.76%84\.76\\%,C16=5\.921C\_\{16\}=5\.921, andD4=3\.473D\_\{4\}=3\.473on 115 eligible tasks\. Its APPS Intro scores are14\.88%14\.88\\%,34\.50%34\.50\\%,1\.8801\.880, and3\.6853\.685on 47 eligible tasks\. TheSpectrum\-minus\-projection pairedD4D\_\{4\}effects are0\.0390\.039\[−0\.021,0\.102\]\[\-0\.021,0\.102\]on 109 HumanEval\+ tasks and0\.0670\.067\[−0\.003,0\.142\]\[\-0\.003,0\.142\]on 42 APPS tasks\. The paired intervals specify the precision of these smaller transfer contrasts\.

## Appendix ECross\-model and cross\-benchmark baseline context

The completed matrix evaluates Qwen2\.5\-Coder\-1\.5B\-Instruct, DeepSeek\-Coder\-6\.7B\-Instruct\([Guo et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib20)\), Gemma\-3\-4B\-IT\([Gemma Team et al\., 2025](https://arxiv.org/html/2610.07237#bib.bib25)\), and Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2610.07237#bib.bib26)\)on MBPP, HumanEval\+, APPS Intro, CodeContests\([Li et al\., 2022](https://arxiv.org/html/2610.07237#bib.bib18)\), and LiveCodeBench\([Jain et al\., 2024](https://arxiv.org/html/2610.07237#bib.bib19)\)\. Table[11](https://arxiv.org/html/2610.07237#A5.T11)gives pass@16; Table[12](https://arxiv.org/html/2610.07237#A5.T12)reports the archived task\-bootstrap intervals\. These are initial\-model evaluations, independent of the sample pools used for the five\-round and frozen\-student comparisons\. Consequently, even the same initial checkpoint can have different finite\-sample estimates across tables\.

All twenty cells completed generation and verification\. Task counts are 500, 164, 200, 165, and 200, respectively\. MBPP pass@16 is estimated from 64 samples per task; the other benchmarks use 16\. HumanEval\+ uses official EvalPlus verification; APPS Intro, CodeContests, and LiveCodeBench use the recorded native adapted verifiers\. The Gemma–LiveCodeBench run uses eager attention\. These protocol choices define the reported benchmark setting\.

Table 11:Completed cross\-model baseline matrix\.Pass@16 \(%\) for initial models across five coding benchmarks\. Each column has a common recorded evaluation setting across models\. These rows measure initial\-model performance, not trainedSpectrumstudents\.Table 12:Archived uncertainty for the baseline matrix\.Pointwise 95% task\-bootstrap intervals for pass@16 on the percentage scale, conditional on the recorded model and sample pool\.
## Appendix FActual equal\-correct\-count examples

The compact archive retains correctness and program fingerprints but removes program text\. We therefore show actual class\-frequency records rather than reconstructing purported code samples\. For each sign of theSpectrum\-minus\-Vanilla SD observed AST\-count difference, choose the numerically lowest HumanEval task ID with equal correct counts of at least four\. This rule yields HumanEval/3 for a positive example and HumanEval/2 for a negative example \(Figure[6](https://arxiv.org/html/2610.07237#A6.F6)\)\. Selection conditions on the sign and is illustrative, not an estimate of prevalence\.

Figure 6:The same number of correct outputs can conceal different structures\.Actual saved round\-five HumanEval\+ samples\. \(a\) Both methods pass all 16 draws on HumanEval/3\. Vanilla SD has one AST class with count 16;Spectrumhas counts 15 and 1, givingD^4=1\\widehat\{D\}\_\{4\}=1and1\.251\.25\. \(b\) Both pass 12 of 16 draws on HumanEval/2; Vanilla SD has twelve singleton classes, whileSpectrumhas one doubleton and ten singletons\. TheirD^4\\widehat\{D\}\_\{4\}values are44and3\.9093\.909\. Classes are ranked separately within each method, so horizontal position does not assert matching class identity\. Both cases follow the stated lowest\-ID rule\.These examples instantiate the measurement distinction without supplying semantic algorithm labels\. At equal observednnandmm, every finite\-sample pass@kkestimator agrees between methods, while the correct\-class occupancy can differ in either direction\. Equal estimated pass curves are not proof that the underlying true success probabilities are identical\.

## Appendix GResources and realized sample lengths

All methods use the same 23,280 generated training records over five rounds\. Table[13](https://arxiv.org/html/2610.07237#A7.T13)records their realized lengths\.Spectrumgenerates9\.9%9\.9\\%more completion tokens than Vanilla SD; equal record count is therefore not equal token count or FLOPs\. Matching the number of correct evaluation samples does not control training length or compute\. It answers a distributional question about the saved students\.

Table 13:Training\-generation resources across five rounds\.Counts are summed from saved round budgets; mean length is generated tokens divided by records\. Length\-capped records remain in training\. SFT token counts include prompt targets under the recorded loss mask\.For a selected module of output dimensionddandMMgradient positions, dense second\-moment accumulation costsO⁡\(M​d2\)O\(Md^\{2\}\)and eigendecomposition costsO⁡\(d3\)O\(d^\{3\}\), excluding the reference forward/backward passes\. Dense folding of ad×pd\\times pweight costsO⁡\(d2​p\)O\(d^\{2\}p\)\. These operations are performed during calibration/construction, not at each generation token\. The final student has the original dense architecture\. Archived runtime records cover different hardware and stages; we make no matched wall\-clock or GPU\-memory efficiency claim from their combination\.

## Appendix HOne\-round self\-distillation study

This experiment uses Qwen2\.5\-Coder\-1\.5B\-Instruct with 128 synthesis tasks, one generated completion per task, one generate–learn cycle, and five SFT epochs\. Evaluation uses 64 held\-out MBPP tasks and 64 samples per task\. Five SFT epochs within a cycle are not five self\-distillation rounds\. Its task counts and training schedule differ from the main five\-round study, so the two experiments are reported separately\.

#### Code extraction and verification\.

For each saved completion, we extract the first Python,py, or untagged fenced code block and execute it against the task tests\. Surrounding explanatory text is ignored\. We re\-evaluate all four generation designs with this same extraction rule\.

Table 14:Complete one\-round comparison\.One generate–learn cycle, 64 MBPP tasks, 64 samples each\. Values are archived rounded summaries; bold marks the largest estimate at shown precision\. AST and exact\-program statistics are structural proxies\.Table 15:Paired one\-round effects relative to SSD\.Spectrumminus SSD, with pointwise task\-bootstrap intervals for the evaluated checkpoints\.Figure 7:Complementary evidence from the one\-round study\.\(a\) Pass@kkat recorded sampling budgets; \(b\) correct\-count\-matched ASTD4D\_\{4\}\. All four generation designs are included\. Marks use archived rounded point estimates from 64 tasks and 64 samples per task; marginal error bars are unavailable\. Table[15](https://arxiv.org/html/2610.07237#A8.T15)gives the recorded paired uncertainty\. This protocol is separate from the five\-round study\.The one\-round study combines higher high\-budget success and matched\-correct richness than SSD with slightly lower single\-draw correctness\. The full five\-round experiment subsequently measures a different operating point:Spectrumhas higher correctness than SSD, while SSD has higherC16C\_\{16\}and marginalD4D\_\{4\}\. Different synthesis budgets, task sets, and optimization schedules preclude attributing this difference solely to loop length\.

## Appendix IRelation to adjacent self\-distillation settings

#### Sequence\-level and token\-feedback distillation\.

Our learner uses sampled sequences as cross\-entropy targets\. Demonstration\-conditioned on\-policy distillation instead supplies dense teacher feedback conditioned on privileged context\.Spectrumshapes generation through fixed reference calibration, then trains one native student on its raw corpus\. Looped Self\-Distillation names the outer generate–learn–update process\.

#### Iterative self\-improvement\.

Iterative context/rationale learning, code\-data synthesis, and teacher refresh have been studied with different feedback mechanisms\([Rao et al\., 2023](https://arxiv.org/html/2610.07237#bib.bib21);[Zhang et al\., 2025](https://arxiv.org/html/2610.07237#bib.bib22);[Sang et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib23)\)\. Our study combines a fixed\-information protocol, a continuous spectral generation intervention, and longitudinal measurements of conditional correct\-implementation richness\. Demonstration\-conditioned diversity analyses\([Nicolicioiu et al\., 2026](https://arxiv.org/html/2610.07237#bib.bib24)\)address a different teacher\-feedback construction; the present results measure contraction and retention through repeated learning from unselected raw corpora\.