ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

# ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
Source: [https://arxiv.org/html/2607.13124](https://arxiv.org/html/2607.13124)
1\]ByteDance 2\]Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences 3\]University of Chinese Academy of Sciences\\contribution\[†\]Corresponding author

Qianhao YuanHongyu LinYaojie Lu Xianpei HanLe SunXiang LiMing XuJiarui LiXiuying Zhao\[\[\[

###### Abstract

Structured pruning is a hardware\-friendly way to compress LLMs, but it is mostly validated on multiple\-choice recognition tasks, while the same compressed checkpoints can collapse on the free\-form generation that deployment actually requires\. Two observations trace this gap\. First, greedypass@11nearly vanishes after compression, yetpass@kkrecovers substantially under repeated sampling: useful generations are demoted, not erased\. Second, the recoverable regime fails mainly through suffix repetition\. Recovery should therefore train on the compressed model’s own on\-policy states with dense token\-level supervision, which On\-Policy Distillation \(OPD\) provides by reusing the pre\-compression model as a frozen teacher\. However, long on\-policy rollouts spend early recovery budget on low\-information repetitive suffixes, delaying loss descent\. To mitigate this waste, we proposeShortOPD, a short\-to\-long OPD schedule that detects teacher\-confirmed repetitive suffixes, treats the surviving prefix as each rollout’s effective length, and allocates future rollout budgets to the effective lengths the policy can currently use\. Across math, code, and open\-ended generation, ShortOPD raises the compressed model’s score to about9×9\\timesits unrecovered value and1\.61\.6–4\.4×4\.4\\timesstandard recovery recipes \(SFT w/o KD, KD, and SeqKD\), and it matches a fixed81928192\-token rollout horizon within two points using a quarter of the training time \(8\.58\.5vs\.35\.935\.9hours\) and71%71\\%fewer rollout tokens\. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple\-choice benchmarks, a step closer to deployment\-ready generation quality\.

![[Uncaptioned image]](https://arxiv.org/html/2607.13124v1/x1.png)

Figure 1:Headline recovery on25%25\\%\-pruned Qwen3\-4B\-Instruct: domain\-mean generation scores from Table[3](https://arxiv.org/html/2607.13124#S4.T3)\(open\-ended judge scores×10\\times 10\)\. ShortOPD restores about two\-thirds of the unpruned teacher’s average of75\.275\.2\.
## 1Introduction

Serving LLMs at full size is expensive\[grattafiori2024llama3,qwen3\], and structured pruning is one of the most deployment\-friendly ways to cut this cost: it removes coupled parameter blocks while preserving ordinary dense transformer execution\[ma2023llmpruner,ashkboos2024slicegpt,men2025shortgpt,xia2024sheared,muralidharan2024minitron\], with no need for the dedicated kernel or hardware support that unstructured sparsity and low\-bit quantization lean on\[frantar2023sparsegpt,sun2024wanda,lin2024awq\]\. The obstacle is evaluation: most pruning work\[ma2023llmpruner,ashkboos2024slicegpt,men2025shortgpt,an2024flap,kim2024shortened,gromov2025unreasonable,muralidharan2024minitron\]reports strong retention on multiple\-choice recognition benchmarks such as MMLU and HellaSwag\[hendrycks2021mmlu,zellers2019hellaswag\], yet the same compressed checkpoints collapse on the free\-form generation that deployment actually requires\. Removing just44of3636layers from Qwen3\-4B\-Instruct\[qwen3\]with Block\-Influence depth pruning\[men2025shortgpt\]already devastates greedy generation across math, code, and open\-ended tasks \(Table[1](https://arxiv.org/html/2607.13124#S1.T1),pass@11versus teacher greedy\)\. This recognition\-generation gap is what blocks practical use of structured pruning\.††The experiments in this paper can be reproduced with public implementations of the two building blocks: structured pruning follows[https://github\.com/icip\-cas/ShortX](https://github.com/icip-cas/ShortX), and on\-policy distillation follows[https://github\.com/VisionOPD/Vision\-OPD](https://github.com/VisionOPD/Vision-OPD)\.

Table 1:Pruning collapses generation, but correct outputs stay reachable by sampling: best\-of\-kkscores of Qwen3\-4B\-Instruct with44of3636layers removed, against the number of sampled attemptskk\. Math/code reportpass@kk\(%\); open\-ended tasks report LLM\-judge best\-of\-kkscores on a11–1010scale\. Teacher is the unpruned model’s greedy score\.Bold: GSM8K crosses the teacher’s greedy score under sampling—correct trajectories are demoted, not erased\.Two observations identify the recovery signal\. First, the missing ability is demoted rather than erased\.wen2026illusionshow this for short QA answers; we find the same holds for entire generation trajectories: in Table[1](https://arxiv.org/html/2607.13124#S1.T1),pass@kk\[chen2021codex\]climbs steadily with more sampled attempts on every benchmark, and on GSM8K\[cobbe2021gsm8k\]it even reaches91%91\\%atk=64k\{=\}64, above the unpruned teacher’s greedy88%88\\%\. Correct trajectories thus remain in the compressed model’s own sampling distribution, within reach of on\-policy search\. Second, the repair must be token\-level: ShortGPT\-gen restores much of the lost quality by routing onlygeneratedtokens through the full model\[men2025shortgpt\], so generation failure is a chain of local next\-token mis\-rankings that compound along the student’s own decoding path\[ranzato2016sequence,agarwal2024gkd\]\. Sparse rewards\[shao2024deepseekmath,deepseekr1\]and fixed off\-policy labels\[kim2016sequence\]leave these local errors unspecified\. Recovery therefore needs dense distributional targets on the states the compressed model actually visits\.

OPD\[agarwal2024gkd,yuan2026vision\]is the direct instantiation of these requirements\. The compressed student samples rollouts from its own distribution\. Its frozen pre\-compression self scores the same trajectories as teacher, and the student matches the teacher’s next\-token distribution at every response position\. The recipe is on\-policy and dense, and it needs no labels, no verifier, and no external teacher\. The remaining difficulty is that early on\-policy rollouts are dominated by repetition\[holtzman2020degeneration,xu2022ditto\]\. Probing BI\-pruned Qwen3\-4B\-Instruct on a fixed192192\-prompt math/code/open\-ended set, suffix repetition reaches84%84\\%at our25%25\\%\-parameter testbed \(Figure[2](https://arxiv.org/html/2607.13124#S1.F2)a\); with stronger compression, outputs turn incoherent instead of looped \(Appendix[11](https://arxiv.org/html/2607.13124#S11)\), so we view25%25\\%as this model’s critical point for recoverable compression and adopt it as our main testbed\. These loop tails also carry little measured marginal distillation signal: repeated\-suffix tokens have a mean teacher–student generalized Jensen–Shannon divergence \(JSD\) of0\.00140\.0014versus0\.0510\.051on ordinary tokens \(about35×35\\timeslower\), and a mean teacher NLL of3×10−53\\times 10^\{\-5\}versus0\.680\.68\. Together with the conditional gradient analysis in Appendix[9](https://arxiv.org/html/2607.13124#S9), these measurements identify deep repeated suffixes as low\-information under the implemented OPD update: they consume early rollout and teacher compute under a long, fixed rollout horizonHH\(the per\-response token budget\) while providing little additional teacher–student correction\. Figure[2](https://arxiv.org/html/2607.13124#S1.F2)\(b\) follows the first100100steps of a fixed\-H=2048H\{=\}2048OPD run: during the early phase,5555–75%75\\%of rollouts end in suffix loops while the distillation loss remains at its lowest level, reaching its post\-warm\-up range only around step8080as repetition subsides\.

![Refer to caption](https://arxiv.org/html/2607.13124v1/x2.png)\(a\)Suffix repetition vs\. pruned depth
![Refer to caption](https://arxiv.org/html/2607.13124v1/x3.png)\(b\)Repetitive suffixes coincide with low loss

Figure 2:Repetition during early on\-policy recovery\.\(a\)Suffix\-repetition rate versus BI\-pruned depth on a fixed192192\-prompt probe; the star marks our main99\-layer setting\.\(b\)Distillation loss \(blue\) and suffix\-repetition rate \(red\) during fixed\-H=2048H\{=\}2048OPD; shading marks the repetition\-dominated warm\-up\. Figure[4](https://arxiv.org/html/2607.13124#S4.F4)\(a\) overlays ShortOPD\.We propose ShortOPD \(Short\-to\-Long On\-Policy Distillation\) to make OPD spend its early budget where the supervision is useful\. A lightweight controller tracks suffix repetition, truncation, and mean effective length\. High repetition opens the shrink gate, whose target is an exponential moving average \(EMA\) of effective length; low repetition and high truncation open the growth gate\. A second EMA moves the actual budget toward either target; both the first smooths noisy batch estimates, while the second prevents abrupt horizon changes\. Training therefore contains repetitive early rollouts and restores long generation as recovery proceeds, at no extra forward cost\. In practice, ShortOPD narrows the high\-repetition region, reduces tokens spent in repeated suffixes, and improves recovery over fixed short and fixed long horizons \(Figure[1](https://arxiv.org/html/2607.13124#S0.F1)\)\.

Our contributions are threefold:

- •We propose ShortOPD, a recipe for recovering structurally compressed LLMs built on our diagnosis of what recovery requires: supervision on the student’s own on\-policy states, dense token\-level teacher distributions, and a rollout budget matched to what the damaged policy can currently generate\. ShortOPD delivers the third ingredient with repetition\-gated, truncation\-aware horizon control\.
- •Extensive experiments demonstrate the effectiveness and efficiency of this recovery recipe\.On Qwen3\-4B\-Instruct with a45,44745\{,\}447\-prompt math/code/instruction corpus, ShortOPD restores about two\-thirds of the unpruned teacher’s generation score over eight task families, far ahead of SFT w/o KD, KD, SeqKD, and sparse\-reward RLVR under matched budgets, and it matches fixed rollout horizons of up to81928192tokens within two points while generating up to71%71\\%fewer rollout tokens; controlled comparisons confirm that the gain requiresbothon\-policy states and dense teacher distributions\.
- •The recovery corpus matters\.On\-policy distillation only repairs the states the student’s own search visits: leave\-one\-domain\-out ablations show that removing math or code prompts sharply damages recovery on the corresponding capabilities\. Corpus coverage is therefore a first\-class design choice for post\-compression recovery, not a detail\.

## 2Related Work

#### Structured pruning and the recognition\-generation gap\.

Depth pruning is among the most serving\-friendly LLM compression families: deleting whole transformer blocks preserves a standard dense architecture, with layers selected by Block Influence \(ShortGPT\[men2025shortgpt\]\), merged \(LaCo\[yang2024laco\]\), or removed under other criteria\[kim2024shortened,gromov2025unreasonable,yuan2025shortv\]\. Width pruning instead removes coupled structures inside layers\[ma2023llmpruner,ashkboos2024slicegpt,an2024flap,dery2024everybody\]; Sheared LLaMA\[xia2024sheared\]learns pruning masks with continued pre\-training, and Minitron\[muralidharan2024minitron,sreenivas2024minitron\]combines depth and width pruning with distillation\-based retraining\. One\-shot weight sparsification \(SparseGPT\[frantar2023sparsegpt\], Wanda\[sun2024wanda\]\) is complementary but needs sparse\-kernel support\. Across these lines, validation is dominated by recognition\-style benchmarks such as MMLU and HellaSwag\[hendrycks2021mmlu,zellers2019hellaswag\]\.wen2026illusionshow why this is deceptive: pruned models can still score multiple\-choice answers yet fail to produce them, the missing answers demoted rather than erased\. ShortGPT\-gen makes the generative side concrete by routing onlygeneratedtokens through the full depth, locating the damage along the decoding path\[men2025shortgpt\]\. We extend the demotion view from short answers to entire reasoning trajectories and focus on the step these works leave open: after the layers are gone, which training signal best restores generation?

#### Distillation and on\-policy recovery signals\.

Knowledge distillation\[hinton2015distilling\]trains a student to match a teacher; sequence\-level KD\[kim2016sequence\]trains on fixed teacher\-generated responses and is therefore off\-policy\. A line of work closes the resulting train\-inference mismatch by distilling on student states: imitation\-learning KD\[lin2020imitkd\], MiniLLM\[gu2024minillm\]with a reverse KL divergence objective, divergence generalizations\[wen2023fdistill,ko2024distillm\], and generalized on\-policy KD\[agarwal2024gkd\]\. Recovery retraining after pruning, however, still mostly follows the off\-policy recipe: Sheared LLaMA continues pre\-training on corpus data\[xia2024sheared\], and Minitron distills teacher logits on a fixed data blend and validates on recognition suites\[muralidharan2024minitron,sreenivas2024minitron\], so whether off\-policy logit distillation repairsgenerationis left open; our KD baseline instantiates exactly this recipe for a controlled comparison \(Section[4\.2](https://arxiv.org/html/2607.13124#S4.SS2)\)\. RL with verifiable rewards\[shao2024deepseekmath,deepseekr1,lambert2025tulu3,yu2025dapo\]is on\-policy but sparse: it offers no token\-level target when a heavily compressed model rarely samples a correct answer\. Vision\-OPD\[yuan2026vision\]introduced on\-policy distillation for multimodal models; we study the same principle after structural compression, where the privileged teacher is the model’s own pre\-compression self\. All of these methods distill ahealthygenerator; none addresses a damaged student whose rollouts collapse into repetition that the self\-teacher then endorses\.

## 3Method

We first formalize OPD as a recovery recipe for structurally compressed LLMs\. The experiments instantiate compression with Block\-Influence depth pruning because it is simple, hardware\-friendly, and representative of structured pruning, but the recovery objective only requires a compressed student and its pre\-compression self\. We then present ShortOPD, a short\-to\-long OPD schedule driven by repetition and truncation feedback; Figure[3](https://arxiv.org/html/2607.13124#S3.F3)gives the overview and Algorithm[1](https://arxiv.org/html/2607.13124#alg1)the procedure\. Finally, we place ShortOPD and its baselines in a single design space\.

### 3\.1Preliminaries: pruning and the OPD base recipe

#### Representative structured pruning operator\.

LetXi,tX\_\{i,t\}denote the hidden state of tokenttentering layerii\. ShortGPT’s Block Influence \(BI\) of layeriiis

BIi=1−𝔼X,t​\[Xi,t⊤​Xi\+1,t∥Xi,t∥2​∥Xi\+1,t∥2\],\\mathrm\{BI\}\_\{i\}\\;=\\;1\-\\mathbb\{E\}\_\{X,t\}\\\!\\left\[\\frac\{X\_\{i,t\}^\{\\top\}X\_\{i\+1,t\}\}\{\\lVert X\_\{i,t\}\\rVert\_\{2\}\\,\\lVert X\_\{i\+1,t\}\\rVert\_\{2\}\}\\right\],\(1\)so a lowBIi\\mathrm\{BI\}\_\{i\}marks a layer that barely changes the hidden state\. We remove the lowest\-BI layers until roughly25%25\\%of parameters are gone, obtaining the compressed studentπθ\\pi\_\{\\theta\}, and keep the original modelπT\\pi\_\{T\}as a frozen teacher\. The operator is only the experimental object: ShortOPD assumes nothing beyondπθ\\pi\_\{\\theta\}being a structurally compressed descendant ofπT\\pi\_\{T\}\.

#### On\-policy self\-distillation\.

Following the two implications of Section[1](https://arxiv.org/html/2607.13124#S1), the recovery objective should repair the student on the states it actually visits and at token granularity\. OPD satisfies both: for each promptxx, the student samples on\-policy rolloutsy∼πθ\(⋅∣x\)y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\), the frozen teacherπT\\pi\_\{T\}is run on the same trajectories, and the student matches the teacher’s next\-token distribution at every response position:

ℒOPD\(θ\)=𝔼x𝔼y∼πθ\(⋅∣x\)\[1\|y\|∑t=1\|y\|Dα\(πT\(⋅∣x,y<t\),πθ\(⋅∣x,y<t\)\)\],\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta\)=\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[\\frac\{1\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}D\_\{\\alpha\}\\\!\\big\(\\pi\_\{T\}\(\\cdot\\mid x,y\_\{<t\}\),\\,\\pi\_\{\\theta\}\(\\cdot\\mid x,y\_\{<t\}\)\\big\)\\right\],\(2\)whereDαD\_\{\\alpha\}is a generalized Jensen–Shannon divergence; we distill the top\-100100logits plus an aggregated tail mass, with clipped importance weighting against the rollout policy\. With a large teacher–student capacity gap, forward KL is mode\-covering and can force a limited student to spread probability mass across teacher modes, whereas reverse KL is mode\-seeking but may reduce diversity\[gu2024minillm,agarwal2024gkd\]\. This trade\-off is independent of whether the training trajectories are off\- or on\-policy and remains task\-dependent\[agarwal2024gkd\]\. We therefore use the balanced choiceα=0\.5\\alpha=0\.5, a bounded compromise that avoids extreme log\-ratio penalties when the teacher and damaged student differ sharply\. Equation \([2](https://arxiv.org/html/2607.13124#S3.E2)\) is on\-policy \(the expectation is over student samples\), dense \(every generated token receives a distributional target\), and reward\-free, so it applies equally to verifiable math/code prompts and open\-ended instruction prompts\. Because the teacher is the pre\-compression self, OPD pulls the student back toward its own original behavior on the trajectories it actually produces\.

### 3\.2Short\-to\-Long On\-Policy Distillation

![Refer to caption](https://arxiv.org/html/2607.13124v1/x4.png)Figure 3:ShortOPD closes a second control loop around OPD\.Distillation loop \(blue\):under the current budgetHH, the compressed student samples on\-policy rollouts \(finishing early, truncated atHH, or collapsing into a repetitive suffix\); the frozen pre\-compression teacher re\-scores the same tokens and returns a dense top\-KKdistribution target at every position\.Budget loop \(orange\):a token\-only terminal\-periodic probe reports severe repetition; its onset, locally refined by the existing OPD loss and teacher NLL, gives effective length\. The fraction of non\-repetitive rollouts that fillHHgives the clean truncation rate\. High repetition sets a shrink target from the EMA of mean effective length; low repetition and high clean truncation set a growth target\. The next budget is an EMA\-smoothed move toward that target\.Algorithm 1ShortOPD1:prompts

ℬ\\mathcal\{B\}, student

πθ\\pi\_\{\\theta\}, teacher

πT\\pi\_\{T\}, bounds

Hmin,HmaxH\_\{\\min\},H\_\{\\max\}, thresholds

ρlow,ρhigh,τ\\rho\_\{\\mathrm\{low\}\},\\rho\_\{\\mathrm\{high\}\},\\tau, margin

λ\\lambda, growth

γ↑\\gamma\_\{\\uparrow\}, EMA decays

β,βH\\beta,\\beta\_\{H\}
2:

H1←HmaxH\_\{1\}\\leftarrow H\_\{\\max\}; initialize EMAs

3:fortraining step

s=1,…,Ss=1,\\ldots,Sdo

4:sample rollouts

yi∼πθ\(⋅∣xi\)y\_\{i\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\_\{i\}\)with budget

HsH\_\{s\}
5:

ai←TerminalLoop⁡\(yi\)a\_\{i\}\\leftarrow\\operatorname\{TerminalLoop\}\(y\_\{i\}\);

rs←B−1​∑iair\_\{s\}\\leftarrow B^\{\-1\}\\sum\_\{i\}a\_\{i\}
6:

qs←B−1​∑i𝟙​\[\|yi\|=Hs∧ai=0\]q\_\{s\}\\leftarrow B^\{\-1\}\\sum\_\{i\}\\mathbb\{1\}\[\|y\_\{i\}\|=H\_\{s\}\\ \\land\\ a\_\{i\}=0\]
7:run

πT\\pi\_\{T\}on the same trajectories

8:update

θ\\thetawith the dense OPD loss over all generated tokens \(Eq\.[2](https://arxiv.org/html/2607.13124#S3.E2)\)

9:compute effective lengths

ℓi\\ell\_\{i\}and mean

LsL\_\{s\}; update

r¯s,q¯s,L¯s\\bar\{r\}\_\{s\},\\bar\{q\}\_\{s\},\\bar\{L\}\_\{s\}
10:if

r¯s\>ρhigh\\bar\{r\}\_\{s\}\>\\rho\_\{\\mathrm\{high\}\}then

11:

Hs∗←min⁡\(Hs,λ​L¯s\)H\_\{s\}^\{\*\}\\leftarrow\\min\(H\_\{s\},\\lambda\\bar\{L\}\_\{s\}\)
12:elseif

r¯s<ρlow\\bar\{r\}\_\{s\}<\\rho\_\{\\mathrm\{low\}\}and

q¯s\>τ\\bar\{q\}\_\{s\}\>\\tauthen

13:

Hs∗←γ↑​HsH\_\{s\}^\{\*\}\\leftarrow\\gamma\_\{\\uparrow\}H\_\{s\}
14:else

15:

Hs∗←HsH\_\{s\}^\{\*\}\\leftarrow H\_\{s\}
16:endif

17:

Hs\+1←clip\[Hmin,Hmax\]\(⌊βHHs\+\(1−βH\)Hs∗⌉16\)H\_\{s\+1\}\\leftarrow\\mathrm\{clip\}\_\{\[H\_\{\\min\},H\_\{\\max\}\]\}\(\\lfloor\\beta\_\{H\}H\_\{s\}\+\(1\-\\beta\_\{H\}\)H\_\{s\}^\{\*\}\\rceil\_\{16\}\)
18:endfor

Equation \([2](https://arxiv.org/html/2607.13124#S3.E2)\) implicitly assumes that the student’s rollouts are usable training states\. Under strong compression this assumption becomes horizon\-dependent: a long fixed budget reaches repetitive suffixes before the policy has recovered, while a permanently short budget truncates legitimate long generations later\. ShortOPD therefore keeps the globalresponse\_lengthas a static padding and context ceiling, but sets a per\-step sampling budgetHt≤HmaxH\_\{t\}\\leq H\_\{\\max\}\. The budget is not driven by batch\-mean loss, which is lowest precisely during the repetitive warm\-up \(Figure[2](https://arxiv.org/html/2607.13124#S1.F2)b\)\. Instead, ShortOPD observes three statistics at every step: how many rollouts end in a severe periodic loop, how many non\-repetitive rollouts exhaust the current budget, and the mean usable prefix before the detected loop\. Repetition is the first\-level gate: effective length may shorten the budget only when repetition is high, whereas low repetition and high clean truncation permit longer rollouts\.

#### Generation\-side feedback\.

For rolloutyiy\_\{i\}at stepss, the probe examines only its lastWWvalid tokens\. Letai=1a\_\{i\}=1when the terminal\-periodic detector of Appendix[8](https://arxiv.org/html/2607.13124#S8)finds a severe loop andai=0a\_\{i\}=0otherwise\. Letbi=1b\_\{i\}=1only when the rollout fills the current budget without such a loop, i\.e\.,\|yi\|=Hs\|y\_\{i\}\|=H\_\{s\}andai=0a\_\{i\}=0\. The structural loop onset is locally refined using the OPD divergence and teacher NLL to obtain effective lengthℓi\\ell\_\{i\}; these loss signals refine the boundary but never determineaia\_\{i\}\. We setℓi=\|yi\|\\ell\_\{i\}=\|y\_\{i\}\|when no severe loop is found\. The batch statistics and their EMAs are

rs=1B​∑i=1Bai,qs=1B​∑i=1Bbi,Ls=1B​∑i=1Bℓi,\(r¯s,q¯s,L¯s\)=EMAβ⁡\(rs,qs,Ls\)\.r\_\{s\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}a\_\{i\},\\qquad q\_\{s\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}b\_\{i\},\\qquad L\_\{s\}=\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\ell\_\{i\},\\qquad\(\\bar\{r\}\_\{s\},\\bar\{q\}\_\{s\},\\bar\{L\}\_\{s\}\)=\\operatorname\{EMA\}\_\{\\beta\}\(r\_\{s\},q\_\{s\},L\_\{s\}\)\.\(3\)The suffix restriction avoids reacting to repeated structure earlier in a valid response, and the EMA suppresses single\-batch fluctuations\. These statistics reuse tokens and teacher scores already produced by OPD and require no extra forward pass\. The effective\-length probe is control\-only: the current batch still receives the unmodified dense OPD loss of Equation \([2](https://arxiv.org/html/2607.13124#S3.E2)\) over all generated tokens\.

#### Repetition\-gated horizon control\.

The controller first forms a gated target

Hs∗=\{min⁡\(Hs,λ​L¯s\),r¯s\>ρhigh,γ↑​Hs,r¯s​<ρlow∧q¯s\>​τ,Hs,otherwise,Hs\+1=clip\[Hmin,Hmax\]\(⌊βHHs\+\(1−βH\)Hs∗⌉16\)\.H\_\{s\}^\{\*\}=\\begin\{cases\}\\min\(H\_\{s\},\\lambda\\bar\{L\}\_\{s\}\),&\\bar\{r\}\_\{s\}\>\\rho\_\{\\mathrm\{high\}\},\\\\ \\gamma\_\{\\uparrow\}H\_\{s\},&\\bar\{r\}\_\{s\}<\\rho\_\{\\mathrm\{low\}\}\\ \\land\\ \\bar\{q\}\_\{s\}\>\\tau,\\\\ H\_\{s\},&\\text\{otherwise\},\\end\{cases\}\\qquad H\_\{s\+1\}=\\mathrm\{clip\}\_\{\[H\_\{\\min\},H\_\{\\max\}\]\}\\\!\\Big\(\\Big\\lfloor\\beta\_\{H\}H\_\{s\}\+\(1\-\\beta\_\{H\}\)H\_\{s\}^\{\*\}\\Big\\rceil\_\{16\}\\Big\)\.\(4\)Hereλ\>1\\lambda\>1retains headroom beyond the observed usable prefix,γ↑\>1\\gamma\_\{\\uparrow\}\>1proposes growth, andβH\\beta\_\{H\}smooths both directions before rounding to multiples of1616\. The hysteresis bandρlow<ρhigh\\rho\_\{\\mathrm\{low\}\}<\\rho\_\{\\mathrm\{high\}\}prevents oscillation\. High repetition has priority even when truncation is also high, and its severity determines the shrink target throughL¯s\\bar\{L\}\_\{s\}\. Growth requires both clean suffixes and evidence that the current horizon is binding\. Consequently, short clean responses cannot reduce the budget merely by lowering average length\. We initialize atHmaxH\_\{\\max\}and checkpoint the current budget and all three EMAs across restarts\. Algorithm[1](https://arxiv.org/html/2607.13124#alg1)summarizes the loop\.

### 3\.3A design space for recovery signals

ShortOPD is one cell of a space spanned by two axes: trajectory source \(on\-policy student rollouts vs\. off\-policy fixed data\) and supervision \(dense teacher distribution vs\. hard labels or sparse reward\)\. Table[2](https://arxiv.org/html/2607.13124#S3.T2)summarizes where each recovery method sits\. Our Qwen3 experiments instantiate SFT w/o KD, SeqKD, KD, and ShortOPD on the same prompt distribution, and add a GSM8K\-only RLVR comparison for the sparse on\-policy cell:

Table 2:Recovery methods organized by the two design axes\. Only ShortOPD trains on the compressed student’s own states with a dense token\-level signal while steering the rollout budget away from low\-information suffixes; it requires neither ground\-truth responses nor a reward verifier\.- •SFT w/o KD: standard supervised finetuning on the corpus’s original golden responses\. This tests whether simply training the compressed model on the same data is enough\.
- •SeqKD\[kim2016sequence\]: SFT on fixed responses generated by the same pre\-compression teacherπT\\pi\_\{T\}\. This shares OPD’s teacher but removes the on\-policy trajectory source\.
- •KD\[hinton2015distilling,agarwal2024gkd\]: teacher\-forced token\-level distillation \(also known as word\-level KD\) that keeps SFT’s fixed sequences but replaces hard labels with the teacher’s next\-token distributions\. This shares OPD’s dense signal but removes the on\-policy trajectory source\.
- •RLVR\[schulman2017ppo,shao2024deepseekmath\]: on\-policy learning with a sparse correctness reward on GSM8K; we instantiate it with both PPO and GRPO\. This shares OPD’s on\-policy sampling but replaces dense teacher distributions with sparse answer rewards\.
- •ShortOPD \(ours\): on\-policy rollouts with dense self\-distillation targets and the repetition\-gated budget controller of Section[3\.2](https://arxiv.org/html/2607.13124#S3.SS2)\.

## 4Experiments

The experiments evaluate ShortOPD and its OPD foundation along four axes\. First, we compare recovered generation quality against practical post\-compression recovery baselines across two backbones and eight task families\. Second, we analyze the short\-to\-long horizon controller: its trajectory, its comparison against fixed short and fixed long horizons, and the repetition dynamics that drive it\. Third, we isolate the learning signal, separating dense on\-policy distillation from off\-policy SFT\-style recovery and sparse on\-policy reward learning\. Fourth, we test how recovery depends on the prompt\-domain mixture\.

### 4\.1Setup

#### Backbones and compression\.

We use Qwen3\-4B\-Instruct\-2507\[qwen3\], which has3636transformer layers\. The original model is the teacherπT\\pi\_\{T\}, and the compressed student is obtained by deleting the lowest\-BI layers until roughly25%25\\%of parameters are removed\. BI is computed on PG\-19 calibration documents\. The teacher is frozen throughout recovery\.

#### Recovery corpus construction\.

The recovery corpus contains45,44745\{,\}447prompts across three domains:

- •Math: GSM8K train\[cobbe2021gsm8k\]\(7,4737\{,\}473prompts\) and MATH train\[hendrycks2021math\]\(7,5007\{,\}500\), formatted as single\-turn problems with a final\-answer instruction; MATH training examples are kept disjoint from MATH\-500\.
- •Code:15,00015\{,\}000filtered NVIDIA OpenCodeInstruct\[opencodeinstruct\]prompts \(kept only with a positive unit\-test signal, deduplicated, and with function names overlapping HumanEval or MBPP test tasks removed\) plus474474unique MBPP\[austin2021program\]non\-test tasks\.
- •Open\-ended:15,00015\{,\}000instruction prompts sampled from ShareGPT/Vicuna\-style conversations and UltraChat\-200K\[vicuna2023,ding2023ultrachat\]\.

No HumanEval task or MBPP test task is used for recovery training\. All recovery methods share this prompt distribution but consume it differently: OPD and ShortOPD use only the prompt field, sampling on\-policy rollouts scored by the frozen teacher; SFT w/o KD trains on the original assistant responses; SeqKD trains on fixed responses generated by the same frozen teacher; and KD keeps SFT’s fixed sequences but replaces hard labels with the teacher’s next\-token distributions\. The comparisons therefore vary the recovery signal and trajectory source, not the prompt mixture\.

#### Recovery methods and schedules\.

All recovery methods start from the same compressed student and see one epoch of the same prompt distribution\. KD is a teacher\-forced forward\-KL baseline,KL\(πT\(⋅∣x<t\)∥πS\(⋅∣x<t\)\)\\mathrm\{KL\}\(\\pi\_\{T\}\(\\cdot\\mid x\_\{<t\}\)\\\|\\pi\_\{S\}\(\\cdot\\mid x\_\{<t\}\)\), with temperature1\.01\.0and no hard\-label cross\-entropy term; it instantiates, on our recovery corpus, the fixed\-context logit\-distillation recipe used by prune\-then\-distill pipelines such as Minitron\[muralidharan2024minitron,sreenivas2024minitron\], so the KD\-versus\-OPD contrast directly tests whether that recipe extends from recognition benchmarks to generation recovery\. For OPD, the student samples on\-policy responses; the pre\-compression teacher is then evaluated on the same trajectories, and the student applies the top\-100100\+tail generalized\-JSD distillation loss of Section[3\.1](https://arxiv.org/html/2607.13124#S3.SS1)\. The fixed horizons use a constant rollout budget \(20482048or81928192tokens\)\. ShortOPD uses the same loss and data with the repetition\-gated budget rule of Equation \([4](https://arxiv.org/html/2607.13124#S3.E4)\), initialized atH=2048H\{=\}2048so the budget must discover the usable range from the first step\. Full hyperparameters are in Appendix[7](https://arxiv.org/html/2607.13124#S7)\. The nominal one\-epoch schedule is710710steps\. All experiments run on a single node with8×8\\timesNVIDIA H20 GPUs\.

Table 3:Main generation recovery results on the25%25\\%\-pruned student\. Math/code are percentages; open\-ended tasks are judge scores on a11–1010scale\. Avg normalizes judge scores by×10\\times 10\. T% reports Avg as a percentage of the corresponding dense teacher Avg\. ShortOPD uses a repetition\-gated dynamic budget up to20482048tokens\.
#### Evaluation\.

We evaluate free\-form generation only, on eight task families in the same three domains:

- •Math: GSM8K and MATH\-500, scored by answer\-match accuracy\. MATH\-500 uses Qwen\-style long decoding \(a1616k generation cap\) with the recommended sampling parameters\.
- •Code: HumanEval and MBPP, scored by executionpass@11\.
- •Open\-ended: Alpaca, QA, Summarization, and MT\-Bench\[zheng2023mtbench\], judged on a11–1010scale by GPT\-5\.5, an independent judge that is neither the teacher nor a Qwen model\.

Tables report math/code scores as percentages and judge scores on the11–1010scale; the Avg column normalizes judge scores by×10\\times 10and averages the eight task families\.

### 4\.2Main generation recovery

Table[3](https://arxiv.org/html/2607.13124#S4.T3)reports the main results\. Structured compression alone nearly destroys generation: the normalized average drops from75\.1775\.17for the dense teacher to5\.715\.71after pruning\. One epoch of SFT w/o KD recovers some ability \(21\.1921\.19Avg\), SeqKD improves on it \(28\.6028\.60\), and KD adds a modest increase to30\.5230\.52, but all off\-policy baselines remain far below on\-policy distillation: ShortOPD reaches48\.4648\.46Avg, nearly9×9\\timesthe untrained score and an1818\-point gain over the best off\-policy baseline\.

The domain\-level pattern matters: on\-policy recovery is not a math trick, improving code execution, Alpaca, QA, summarization, and MT\-Bench simultaneously, and since ShortOPD uses no answer labels or verifiers, the improvement comes entirely from restoring the student distribution toward its own unpruned self on the student’s own states\.

![Refer to caption](https://arxiv.org/html/2607.13124v1/x5.png)Figure 4:Revisiting the Vanilla OPD warm\-up of Figure[2](https://arxiv.org/html/2607.13124#S1.F2)\(b\)\.\(a\)Distillation loss \(solid, left axis\) and suffix\-repetition rate \(dashed, right axis\) for Vanilla OPD and ShortOPD\. ShortOPD’s loss reaches its plateau within roughly4040–5050steps versus around8080for Vanilla: green shading marks ShortOPD’s warm\-up, red the additional Vanilla\-only delay\.\(b\)Wasted \(repetitive\-suffix\) tokens per rollout \(solid, left axis\) and rollout budgets \(dashed, right axis\)\. ShortOPD shrinks its budget to10241024during the repetitive warm\-up and restores it once clean truncations dominate \(shaded actions\), removing the wasted mass sooner; its effective length still catches Vanilla by step100100\.
### 4\.3ShortOPD: closed\-loop horizon control in practice

#### Closed\-loop recovery trajectory\.

Figure[4](https://arxiv.org/html/2607.13124#S4.F4)shows that ShortOPD shortens the repetitive warm\-up\. With Vanilla fixed atH=2048H\{=\}2048,5555–75%75\\%of rollouts repeat and the loss reaches its post\-warm\-up plateau only after roughly8080steps\. ShortOPD reaches this regime in about4040–5050steps while reducing wasted suffix tokens\. Its budget falls from20482048to10241024during high repetition and returns to20482048as clean truncations take over\. The controller therefore saves early rollout compute without sacrificing long\-generation capacity later in recovery\.

![Refer to caption](https://arxiv.org/html/2607.13124v1/x6.png)\(a\)Per\-step rollout generation time
![Refer to caption](https://arxiv.org/html/2607.13124v1/x7.png)\(b\)Quality vs\. rollout cost across schedules

Figure 5:What uncontrolled rollout budgets cost\.\(a\)Per\-step rollout\-generation time for Vanilla \(fixed 2048\) vs\. ShortOPD \(faint: raw; solid: moving average\) over the same first100100steps as Figure[4](https://arxiv.org/html/2607.13124#S4.F4); the shaded gap is saved warm\-up compute \(35\.4→25\.135\.4\\rightarrow 25\.1s per step on average\)\.\(b\)Across three schedules, the eight\-task Avg \(blue line\) varies by less than22points while the generated rollout tokens \(bars; ShortOPD in green, Vanilla fixed 8192 in red\) vary by more than3×3\\times\. Wall\-clock is annotated inside each bar\.
#### Vanilla versus gated budgets\.

Figure[5](https://arxiv.org/html/2607.13124#S4.F5)compares quality and cost across rollout schedules\. Relative to Vanilla fixed at20482048, ShortOPD lowers mean generation time over the first100100steps by29%29\\%\(35\.4→25\.135\.4\\rightarrow 25\.1s per step\) and end\-to\-end wall\-clock from11\.111\.1to8\.58\.5hours\. Across fixed20482048, ShortOPD, and fixed81928192, Avg varies only from48\.548\.5to50\.250\.2, whereas generated rollout tokens range from250250M to869869M\. ShortOPD finishes in8\.58\.5hours with250250M tokens, using76%76\\%less time than fixed81928192and24%24\\%less than fixed20482048while remaining within1\.71\.7Avg points of both\. In contrast, fixed81928192spends869869M tokens and35\.935\.9hours for only\+0\.6\+0\.6Avg over fixed20482048\.

### 4\.4Scaling

Table[4](https://arxiv.org/html/2607.13124#S4.T4)tests whether the recovery recipe of ShortOPD continues to improve with additional exposure\. Across one, two, and three epochs on the compressed Qwen3\-4B\-Instruct, gains are broad rather than concentrated in a single aggregate: GSM8K, MATH\-500, HumanEval, MBPP, Alpaca, QA, and MT\-Bench all improve from the first to the final checkpoint, with the overall Avg rising from48\.4648\.46to55\.4155\.41\(73\.7%73\.7\\%of the teacher\)\. On\-policy recovery is therefore scalable: more rollout exposure keeps converting into generation quality, and pushing this scaling further is a natural direction for future work\.

Table 4:Per\-domain epoch scaling for ShortOPD on the compressed Qwen3\-4B\-Instruct\. Math/code are percentages; open\-ended tasks are judge scores on a11–1010scale\. Avg normalizes judge scores by×10\\times 10, and T% is relative to the Instruct teacher Avg\.
### 4\.5Signal controls: dense versus sparse and off\-policy

The ShortOPD\-versus\-SeqKD and ShortOPD\-versus\-KD contrasts in Table[3](https://arxiv.org/html/2607.13124#S4.T3)test whether teacher information is sufficient without on\-policy states: with the same frozen teacher, moving its signal off\-policy costs19\.919\.9Avg points \(SeqKD\), and dense teacher\-forced distillation does not close the gap \(KD trails ShortOPD by17\.917\.9points\)\.

To isolate the supervision axis, we also run a GSM8K\-only comparison between sparse\-reward RLVR and ShortOPD from the same compressed Qwen3\-4B\-Instruct initialization, instantiating RLVR with both PPO and GRPO\. All methods use the same GSM8K train prompts, the same batch size, and the same one\-epoch step budget; the only intended difference is the learning signal\. Figure[6](https://arxiv.org/html/2607.13124#S4.F6)shows the result: sparse on\-policy RLVR barely moves GSM8K accuracy \(0\.230\.23with PPO,1\.591\.59with GRPO\), while ShortOPD on the same GSM8K\-only prompt set reaches37\.7637\.76; the multi\-domain ShortOPD run reaches62\.7062\.70and serves as the main\-reference point\. When the compressed model rarely samples a correct answer, a sparse reward has almost nothing to promote, whereas the dense teacher distribution corrects every token of every rollout regardless of correctness\.

![Refer to caption](https://arxiv.org/html/2607.13124v1/x8.png)Figure 6:Sparse versus dense supervision on the matched GSM8K\-only on\-policy comparison\. Sparse\-reward RLVR \(PPO and GRPO\) barely moves the compressed student, while ShortOPD on the same prompts restores a large fraction of accuracy; the multi\-domain ShortOPD run is shown for reference\.
### 4\.6Recovery\-corpus domain ablation

Because ShortOPD is reward\-free, the recovery corpus is only a prompt distribution; we test how much its composition matters\. Table[5](https://arxiv.org/html/2607.13124#S4.T5)recovers the same compressed student with leave\-one\-domain\-out recovery corpora\. To keep the comparison matched, each ablated corpus keeps the same number of prompts and the same710710optimizer steps as the full mixture by uniformly resampling from the two remaining domains; only the removed domain changes\. This isolates prompt composition from training exposure\.

Table 5:Leave\-one\-domain\-out recovery\-corpus ablation for ShortOPD on the compressed Qwen3\-4B\-Instruct\. Math/code are percentages; MATH uses the official MATH\-500 protocol\. Open\-ended tasks are judge scores on a11–1010scale; Avg normalizes them by×10\\times 10\.Redmarks the domain most directly damaged by the removed slice; the shaded row is the full\-mixture reference\.The resulting pattern is direct\. Removing math nearly eliminates mathematical recovery, dropping GSM8K from62\.7062\.70to8\.048\.04and official MATH\-500 from42\.0042\.00to6\.006\.00\. Removing code also lowers math, but collapses execution benchmarks \(HumanEval0\.610\.61, MBPP2\.332\.33\), showing that code rollouts provide a highly domain\-specific repair signal\. Removing general instruction data reduces both math and open\-ended quality in this run, despite improving the narrow code scores\. Thus ShortOPD’s dense token\-level signal can transfer across related states, but the on\-policy state distribution still needs coverage of the capabilities we want to recover\.

## 5Conclusion

Structured pruning can leave latent capability in the model’s search space while failing to promote it during generation, which makes recovery a distributional repair problem rather than ordinary supervised relearning\. On\-policy self\-distillation from the model’s own pre\-compression self is the right repair signal: on Qwen3\-4B\-Instruct it substantially outperforms SFT w/o KD, SeqKD, KD, and sparse\-reward RLVR on the same recovery prompts\. The remaining cost is early rollout quality: long horizons spend many steps on repetitive, low\-information suffixes before normal loss descent begins\. ShortOPD addresses this with a repetition\-gated, truncation\-aware short\-to\-long budget, matching fixed short and fixed long horizons within two points while generating a fraction of their rollout tokens and concentrating early teacher compute on higher\-signal prefixes\. These results motivate short\-to\-long on\-policy self\-distillation as a default recovery step after structured LLM compression\.

## 6Limitations

This work instantiates structured compression with BI depth pruning at about25%25\\%removed parameters on the Qwen3\-4B families; more pruning methods, model families, compression ratios, and larger models should be tested before claiming universality\. Finally, we do not yet establish the boundary where a model has been compressed too aggressively for light post\-compression recovery\.

## References

\\beginappendix

## 7Training details

#### Compression\.

Qwen3\-4B\-Instruct\-2507 has3636transformer blocks\. BI is calibrated on100100PG\-19 documents, and the nine lowest\-BI blocks are removed \(Table[6](https://arxiv.org/html/2607.13124#S7.T6)\); indices are reported before the retained blocks are renumbered at export\. The frozen original model serves as the teacher and the compressed student initializes the actor\.

Table 6:Pruned configuration\.
#### Main on\-policy recovery runs\.

Training uses the45,44745\{,\}447\-prompt math/code/open\-ended corpus of Section[4\.1](https://arxiv.org/html/2607.13124#S4.SS1)with rollout group88, train batch6464, maximum prompt and response lengths of40964096tokens each, rollout temperature0\.80\.8, and learning rate2×10−62\\times 10^\{\-6\}\. The objective is the top\-100100\+tail generalized JSD withα=0\.5\\alpha=0\.5and importance weights clipped at2\.02\.0; no policy\-gradient loss is used\. One epoch is710710steps; the main\-table checkpoint is step709709\.

#### Baselines and export\.

The offline baselines \(SFT w/o KD, SeqKD, KD\) train on the same prompt distribution with the method\-specific fixed responses of Section[4\.1](https://arxiv.org/html/2607.13124#S4.SS1), using sequence length81928192, batch6464, learning rate2×10−62\\times 10^\{\-6\}, and one epoch \(710710steps\); KD uses forward KL at temperature1\.01\.0with no hard\-label cross\-entropy term\. The GSM8K\-only comparison trains the RLVR baselines \(PPO and GRPO\) and ShortOPD on its7,4737\{,\}473prompts for one epoch \(117117steps\) with88rollouts per prompt and batch6464; PPO uses GAE with a critic initialized from the compressed student \(learning rate10−510^\{\-5\}\), while GRPO uses group\-relative advantages\. All runs checkpoint every5050steps and at the final step, and checkpoints are exported in HF format without optimizer states\.

#### Efficiency runs and controller\.

The ShortOPD and fixed\-horizon efficiency runs reuse the corpus, objective, batch size, rollout count, and learning rate above; they differ only in rollout temperature \(0\.70\.7\) and response ceiling \(20482048for ShortOPD and fixedH=2048H\{=\}2048;81928192for fixedH=8192H\{=\}8192\)\. These twoH≤2048H\{\\leq\}2048runs also provide the training trajectories shown in Figures[2](https://arxiv.org/html/2607.13124#S1.F2)\(b\),[4](https://arxiv.org/html/2607.13124#S4.F4), and[5](https://arxiv.org/html/2607.13124#S4.F5)\(a\)\. The controller checkpoints its current budget and all EMA state across restarts\. Table[7](https://arxiv.org/html/2607.13124#S7.T7)lists the detector and controller hyperparameters; symbols follow Algorithm[2](https://arxiv.org/html/2607.13124#alg2)\.

Table 7:Detector and ShortOPD controller hyperparameters\.Because low loss participates in defining effective length, it never opens the shrink gate: only the independent repetition\-rate threshold can do so\. The probe does not mask the loss; training always applies dense OPD to the tokens actually generated\.

## 8Terminal periodic\-loop detector

Algorithm[2](https://arxiv.org/html/2607.13124#alg2)gives the token\-only detector used by the controller\. It searches only the lastW=512W\{=\}512valid response tokens and tests periodsp∈\{1,…,10\}p\\in\\\{1,\\ldots,10\\\}\. For a given period, each token is compared with the tokenpppositions earlier\. This shifted comparison does not require the response to end on a cycle boundary, so a rollout truncated part\-way through its final cycle is still detected\. Requiring high agreement at the raw response end prevents repeated material followed by a clean ending from firing the controller\.

Algorithm 2Severe terminal periodic\-loop detection1:valid response tokens

yy; suffix window

W=512W\{=\}512; maximum period

P=10P\{=\}10; terminal anchor

A=32A\{=\}32; agreement

η=0\.9\\eta\{=\}0\.9; minimum cycles

C=3C\{=\}3; minimum tail

Lmin=64L\_\{\\min\}\{=\}64; severe length

Lsev=128L\_\{\\mathrm\{sev\}\}\{=\}128; severe fraction

ϕ=0\.30\\phi\{=\}0\.30
2:

z←LastW​\(y\)z\\leftarrow\\mathrm\{Last\}\_\{W\}\(y\);

n←\|z\|n\\leftarrow\|z\|;

𝒞←∅\\mathcal\{C\}\\leftarrow\\varnothing
3:for

p=1,…,min⁡\(P,n−1\)p=1,\\ldots,\\min\(P,n\-1\)do

4:

ck\(p\)←𝟙​\[zp\+k=zk\]c^\{\(p\)\}\_\{k\}\\leftarrow\\mathbb\{1\}\[z\_\{p\+k\}=z\_\{k\}\]for

k=1,…,n−pk=1,\\ldots,n\-p
5:if

Mean⁡\(LastA​\(c\(p\)\)\)<η\\operatorname\{Mean\}\(\\mathrm\{Last\}\_\{A\}\(c^\{\(p\)\}\)\)<\\etathen

6:continue⊳\\trianglerightthe periodic pattern does not reach the response end

7:endif

8:

kp←max⁡\{k:Mean⁡\(Lastk​\(c\(p\)\)\)≥η\}k\_\{p\}\\leftarrow\\max\\\{k:\\operatorname\{Mean\}\(\\mathrm\{Last\}\_\{k\}\(c^\{\(p\)\}\)\)\\geq\\eta\\\}
9:

Lp←kp\+pL\_\{p\}\\leftarrow k\_\{p\}\+p;

up←Mean⁡\(Lastkp​\(c\(p\)\)\)u\_\{p\}\\leftarrow\\operatorname\{Mean\}\(\\mathrm\{Last\}\_\{k\_\{p\}\}\(c^\{\(p\)\}\)\)
10:if

Lp≥LminL\_\{p\}\\geq L\_\{\\min\}and

Lp/p≥CL\_\{p\}/p\\geq Cand

\(Lp≥Lsev\(L\_\{p\}\\geq L\_\{\\mathrm\{sev\}\}or

Lp/n≥ϕ\)L\_\{p\}/n\\geq\\phi\)then

11:add

\(Lp,up,p\)\(L\_\{p\},u\_\{p\},p\)to

𝒞\\mathcal\{C\}
12:endif

13:endfor

14:if

𝒞=∅\\mathcal\{C\}=\\varnothingthen

15:returnnot\-severe,

\|y\|\|y\|
16:endif

17:

\(L∗,u∗,p∗\)←arg⁡max\(L,u,p\)∈𝒞⁡\(L,u,−p\)\(L^\{\*\},u^\{\*\},p^\{\*\}\)\\leftarrow\\arg\\max\_\{\(L,u,p\)\\in\\mathcal\{C\}\}\(L,u,\-p\)
18:returnsevere,

\|y\|−L∗\|y\|\-L^\{\*\},

p∗p^\{\*\}

The lexicographic choice prefers the longest explained terminal tail, then higher periodic agreement, then the shorter period\. The detector alone decides whether a rollout is severely repetitive\. Its returned onset is the structural effective\-length candidate\. We then search locally from that onset for the first3232\-token window whose OPD loss and teacher NLL fall below their absolute/relative thresholds; this refines the boundary but does not decide whether repetition exists\. If no loss\-confirmed boundary is found, the structural onset is retained\. A rollout contributes to the*clean truncation rate*only if it fillsHsH\_\{s\}and this detector does not mark it severely repetitive\.

## 9Conditional OPD gradient at teacher–student agreement

That a smooth divergence has zero gradient at its minimum is immediate; the content of this appendix is the*rate*at which the implemented per\-token gradient vanishes near teacher–student agreement, and that the two implementation details – top\-KK\+tail aggregation and clipped importance weighting – preserve this stationary point\. Condition on one sampled prefix\(x,y<t\)\(x,y\_\{<t\}\)and treat that discrete state as fixed during back\-propagation, as in the implemented on\-policy distillation update\. Letppbe the frozen teacher distribution andq=softmax⁡\(z\)q=\\operatorname\{softmax\}\(z\)the student distribution\. For the canonical forward\-KL token lossℓKL=KL​\(p∥q\)\\ell\_\{\\mathrm\{KL\}\}=\\mathrm\{KL\}\(p\\\|q\), the standard softmax\-cross\-entropy derivative gives

∂ℓKL∂zj=qj−pj,∇θℓKL=Jz,θ⊤​\(q−p\)\.\\frac\{\\partial\\ell\_\{\\mathrm\{KL\}\}\}\{\\partial z\_\{j\}\}=q\_\{j\}\-p\_\{j\},\\qquad\\nabla\_\{\\theta\}\\ell\_\{\\mathrm\{KL\}\}=J\_\{z,\\theta\}^\{\\top\}\(q\-p\)\.\(5\)The conditional gradient therefore vanishes as the student distribution approaches the teacher distribution\.

Our implementation uses generalized JSD rather than forward KL\. Form=α​p\+\(1−α\)​qm=\\alpha p\+\(1\-\\alpha\)q, the generalized Jensen–Shannon divergence is

Dα​\(p,q\)=α​∑ipi​log⁡pimi\+\(1−α\)​∑iqi​log⁡qimi\.D\_\{\\alpha\}\(p,q\)=\\alpha\\sum\_\{i\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{m\_\{i\}\}\+\(1\-\\alpha\)\\sum\_\{i\}q\_\{i\}\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}\.\(6\)Because∂mi/∂qi=1−α\\partial m\_\{i\}/\\partial q\_\{i\}=1\-\\alpha, differentiating both KL terms makes their non\-logarithmic terms cancel:

∂Dα∂qi\\displaystyle\\frac\{\\partial D\_\{\\alpha\}\}\{\\partial q\_\{i\}\}=−α​\(1−α\)​pimi\+\(1−α\)​\(log⁡qimi\+1−\(1−α\)​qimi\)\\displaystyle=\-\\frac\{\\alpha\(1\-\\alpha\)p\_\{i\}\}\{m\_\{i\}\}\+\(1\-\\alpha\)\\\!\\left\(\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}\+1\-\\frac\{\(1\-\\alpha\)q\_\{i\}\}\{m\_\{i\}\}\\right\)=\(1−α\)​log⁡qimi\.\\displaystyle=\(1\-\\alpha\)\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}\.\(7\)The softmax Jacobian is∂qi/∂zj=qi​\(𝟙​\[i=j\]−qj\)\\partial q\_\{i\}/\\partial z\_\{j\}=q\_\{i\}\(\\mathbb\{1\}\[i=j\]\-q\_\{j\}\)\. Applying the chain rule therefore gives

∂Dα∂zj\\displaystyle\\frac\{\\partial D\_\{\\alpha\}\}\{\\partial z\_\{j\}\}=\(1−α\)​∑ilog⁡qimi​qi​\(𝟙​\[i=j\]−qj\)\\displaystyle=\(1\-\\alpha\)\\sum\_\{i\}\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}q\_\{i\}\(\\mathbb\{1\}\[i=j\]\-q\_\{j\}\)=\(1−α\)​qj​\[log⁡qjmj−∑iqi​log⁡qimi\]\.\\displaystyle=\(1\-\\alpha\)q\_\{j\}\\left\[\\log\\frac\{q\_\{j\}\}\{m\_\{j\}\}\-\\sum\_\{i\}q\_\{i\}\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}\\right\]\.\(8\)Atq=pq=p, we also havem=pm=p, so every logarithm in Equation \([8](https://arxiv.org/html/2607.13124#S9.E8)\) is zero and the conditional logit gradient is exactly zero\. More locally, writingq=p\+δq=p\+\\deltawith∑iδi=0\\sum\_\{i\}\\delta\_\{i\}=0and assumingpi\>0p\_\{i\}\>0on the retained support,

log⁡qimi=α​δipi\+O​\(δi2/pi2\),∂Dα∂zj=α​\(1−α\)​δj\+O​\(‖δ‖2/mini⁡pi\),\\log\\frac\{q\_\{i\}\}\{m\_\{i\}\}=\\alpha\\frac\{\\delta\_\{i\}\}\{p\_\{i\}\}\+O\(\\delta\_\{i\}^\{2\}/p\_\{i\}^\{2\}\),\\qquad\\frac\{\\partial D\_\{\\alpha\}\}\{\\partial z\_\{j\}\}=\\alpha\(1\-\\alpha\)\\delta\_\{j\}\+O\(\\\|\\delta\\\|^\{2\}/\\min\_\{i\}p\_\{i\}\),\(9\)so the logit gradient vanishes linearly as the two distributions agree \(atα=0\.5\\alpha\{=\}0\.5, with slope1/41/4of the forward\-KL gradientqj−pj=δjq\_\{j\}\-p\_\{j\}=\\delta\_\{j\}\)\. For model parametersθ\\theta,∇θDα=Jz,θ⊤​∇zDα\\nabla\_\{\\theta\}D\_\{\\alpha\}=J\_\{z,\\theta\}^\{\\top\}\\nabla\_\{z\}D\_\{\\alpha\}; hence the parameter gradient also tends to zero whenever the local logit Jacobian is bounded\.

Our implementation evaluates Equation \([6](https://arxiv.org/html/2607.13124#S9.E6)\) after mapping the vocabulary distribution into the teacher top\-100100bins plus one aggregated tail bin\. This mapping is differentiable, and the analysis above applies verbatim to the binned distributions, so the gradient vanishes already when the*binned*distributions agree – a strictly weaker condition than full\-vocabulary agreement, since differences inside the tail bin are invisible to the loss\. This is the relevant condition for the loop states of Section[1](https://arxiv.org/html/2607.13124#S1): teacher and student need only agree at this coarse granularity for the update to carry no signal\. The clipped importance multiplier is bounded by construction; at exact agreement, both the divergence and its gradient are zero, so multiplying by this weight also preserves the stationary point\. This is a conditional, per\-token statement rather than a claim that every small scalar loss implies a small full\-objective gradient\. Empirically, the low JSD measures teacher–student agreement on loop states, while the separately reported low teacher NLL shows that the teacher itself assigns high probability to continuing the observed loop\.

## 10Evaluation details

GSM8K uses the full test split \(1,3191\{,\}319examples\) and exact/normalized answer matching\. MATH\-500 uses boxed\-answer extraction and normalization\. HumanEval \(164164examples\) and sanitized MBPP \(257257examples\) are evaluated with executionpass@11in a subprocess sandbox\. Alpaca, QA, Summarization, and MT\-Bench\[zheng2023mtbench\]use fixed EAGLE\-style prompt sets and are judged on a11–1010scale by GPT\-5\.5, which is neither the teacher nor a Qwen model\.

## 11Repetition versus pruning depth: the full sweep

Figure[2](https://arxiv.org/html/2607.13124#S1.F2)\(a\) shows the suffix\-repetition rate for0–1212removed layers; Figure[7](https://arxiv.org/html/2607.13124#S11.F7)extends the sweep to all3636layers of Qwen3\-4B\-Instruct, with the distinct\-22diversity of the generations on the right axis\. Three regimes emerge\. While the model remains a coherent generator \(k≤12k\\\!\\leq\\\!12\), repetition grows monotonically with removed depth and distinct\-22falls: the model retains enough structure to sustainnn\-gram loops, and looping is its dominant degeneration mode\. Beyondk≈13k\\\!\\approx\\\!13, outputs disintegrate into incoherent token sequences: distinct\-22rebounds toward that of random text andnn\-gram loops no longer form, so the measured repetition rate collapses even though the model is far more damaged\. At the extreme \(k≥35k\\\!\\geq\\\!35\), the near\-empty stack degenerates into trivial single\-token loops and the repetition rate saturates at100%100\\%\. The probe, generation settings, and repetition detector are identical across all points \(Section[4\.3](https://arxiv.org/html/2607.13124#S4.SS3)and Appendix[7](https://arxiv.org/html/2607.13124#S7)\)\.

![Refer to caption](https://arxiv.org/html/2607.13124v1/x9.png)Figure 7:Full pruning sweep on the192192\-prompt probe: suffix\-repetition rate \(left axis\) and distinct\-22diversity \(right axis\) versus the number of BI\-pruned layers\. Repetition is the characteristic failure mode only in the coherent regime \(k≤12k\\\!\\leq\\\!12\); deeper pruning collapses generation into incoherent text where loops no longer form\.#### Qualitative collapse regimes\.

Table[8](https://arxiv.org/html/2607.13124#S11.T8)shows short excerpts from the same probe\. The25%25\\%compression point is still structured enough to produce recognizable prefixes and repetitive suffixes; deeper compression often leaves the repetition metric by entering an incoherent mixed\-token regime instead\.

Table 8:Probe excerpts illustrating the transition from normal generation to repetition and then to incoherent collapse\. Green indicates usable content; red indicates repeated or incoherent content\.

## 12Multiple\-choice sanity check

Although the main evaluation targets free\-form generation, we also evaluate held\-out multiple\-choice recovery on ARC\-Challenge, HellaSwag, MMLU, and WinoGrande\. These benchmarks are not part of the recovery corpus\. Table[9](https://arxiv.org/html/2607.13124#S12.T9)reports Qwen3\-4B\-Instruct results under candidate log\-likelihood scoring over the answer options; SFT\-init ShortOPD is ShortOPD initialized from the one\-epoch SFT w/o KD checkpoint instead of the raw pruned student, trained with the otherwise identical one\-epoch recipe\.

The pattern is the mirror image of the main results, and we read it as further evidence that recognition and generation recover along partly independent axes\. Candidate log\-likelihood scoring is itself an off\-policy, teacher\-forced task: the model must assign high likelihood to externally written text it did not generate\. SFT and KD optimize exactly this objective on fixed sequences, so they recover it best \(74\.974\.9and74\.474\.4Avg\) while remaining far behind on generation \(Table[3](https://arxiv.org/html/2607.13124#S4.T3)\)\. ShortOPD optimizes the complementary objective: it reshapes the student’s distribution on its own rollouts and never trains likelihood on external text, so it dominates generation while its one\-epoch MC average sits lower \(67\.667\.6\) – though still2222points above the untrained pruned model and improving with exposure \(71\.671\.6at three epochs\)\. Neither direction of the recognition\-generation gap \(Section[1](https://arxiv.org/html/2607.13124#S1)\) certifies the other\. The two signals are complementary rather than conflicting: SFT\-init ShortOPD composes them and recovers both axes, matching the dense teacher’s MC average \(76\.876\.8vs\.76\.776\.7\)\.

Table 9:Held\-out multiple\-choice recovery for the25%25\\%\-pruned Qwen3\-4B\-Instruct, scored by candidate log\-likelihood over the answer options\. Scores are accuracies in percent\.