提示广度与展开刷新在策略蒸馏中的交互

# Prompt Breadth and Rollout RefreshInteract in On-Policy Distillation
Source: [https://arxiv.org/html/2609.25048](https://arxiv.org/html/2609.25048)
###### Abstract

How many prompts does on\-policy distillation \(OPD\) need, and how does the answer depend on the student policies that generate its training responses? We study these two controls jointly: prompt breadth and rollout refresh\. A3×33\\times 3mathematical\-reasoning experiment fixes14,08014\{,\}080trajectories and110110optimizer updates while varying the prompt bank and the number of response\-generating policy snapshots\. With ten snapshots, eight prompts reach24\.09%24\.09\\%average accuracy, close to24\.51%24\.51\\%for14,08014\{,\}080distinct prompts\. With responses frozen at the initial policy, however, increasing breadth lowers accuracy from21\.16%21\.16\\%to19\.05%19\.05\\%; under per\-update refresh, it raises accuracy from23\.61%23\.61\\%to25\.57%25\.57\\%\. The resulting interaction is4\.074\.07percentage points, with a95%95\\%question\-paired interval of\[2\.00,6\.28\]\[2\.00,6\.28\]\. Matched comparisons under two teachers reveal a second reversal: the periodic models have higher short\-budget accuracy and answer completion, but frozen\-response models overtake in average accuracy at a3232K output limit, using1\.71\.7–1\.8×1\.8\\timesas many response tokens\. These results show that prompt efficiency in OPD can depend on both refresh and inference budget\.

## 1Introduction

On\-policy distillation \(OPD\) trains a student on its own generated prefixes while a teacher supplies token\-level supervision\([Gu et al\., 2024](https://arxiv.org/html/2609.25048#bib.bib7);[Agarwal et al\., 2024](https://arxiv.org/html/2609.25048#bib.bib1);[Song & Zheng, 2026](https://arxiv.org/html/2609.25048#bib.bib13)\)\. Its training data are therefore policy\-dependent: a run chooses both the prompts that initiate generation and the student snapshots that generate the responses\. Prompt count describes only the first choice\.

This distinction matters whenever responses are reused\. A broad corpus can supply responses from an old student, while a small prompt bank can be revisited after every update\. Both settings can use the same numbers of trajectories and optimizer updates, yet expose the teacher to different student states\. Prior work studies prompt\-efficient reasoning, token\-efficient distillation, and freshness\-aware replay\([Ye et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib21);[Muennighoff et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib11);[Chen et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib4);[Arnal et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib3)\)\. Concurrent work finds substantial OPD gains from a single query and sustained learning on fixed response states\([Fu et al\., 2026b](https://arxiv.org/html/2609.25048#bib.bib6)\)\. We ask how prompt breadth and response refresh interact under matched trajectory and update counts, and how to interpret the resulting models at different inference budgets\.

We study two controls\. Prompt breadthNNis the number of distinct prompts in the training bank; rollout refreshMMis the number of student snapshots used to generate training responses\. Our central3×33\\times 3grid fixesC=14,080C=14\{,\}080trajectories andU=110U=110optimizer updates while varyingN∈\{8,48,Full\}N\\in\\\{8,48,\\mathrm\{Full\}\\\}andM∈\{1,10,110\}M\\in\\\{1,10,110\\\}\. This matches trajectory and update counts; token consumption remains a measured outcome because response lengths change\.

Figure 1:Prompt breadth and rollout refresh are separate controls\.\(a\) Student\-generated responses receive teacher supervision at student\-top\-16 candidates\. \(b\) Nine cells vary unique promptsNNand response\-generating snapshotsMM, with14,08014\{,\}080trajectories and110110optimizer updates each\. \(c\) Increasing breadth lowers Average Accuracy with frozen responses, changes it little under periodic refresh, and raises it under per\-update refresh\. Values are dataset\-macro averages over AIME24, AIME25, and AMC23: 143 questions, 32 samples per question, one training run per cell\.The response surface reveals a sign change in the value of breadth \(Figure[1](https://arxiv.org/html/2609.25048#S1.F1)\)\. Increasing the bank from eight prompts to Full changes Average Accuracy by−2\.10\-2\.10points with one frozen response collection, by\+0\.41\+0\.41with periodic refresh, and by\+1\.96\+1\.96with per\-update refresh\. The difference between the two endpoint breadth effects is4\.074\.07points, with a question\-paired95%95\\%interval of\[2\.00,6\.28\]\[2\.00,6\.28\]; each of the three benchmarks has the same interaction sign\. More unique prompts therefore do not uniformly improve OPD, even at the same trajectory and update counts\.

A second set of comparisons holds the prompt bank fixed at 48 and matches 4,800 trajectories and 40 updates under each of two teachers\. Periodic generation raises the fraction of questions solved among 32 samples by 12\.69 points with Qwen3\-4B\-Instruct and by 22\.57 points with JustRL\-DeepSeek\-1\.5B at a 16K limit\. The largest behavioral difference concerns whether the model produces a parseable answer within that limit\. Re\-scoring saved responses at larger prefix budgets reverses the Average Accuracy ranking under both teachers: frozen\-response models win at 32K, while using substantially more tokens\. Refresh thus changes the accuracy–inference\-cost trade\-off\. A model that delivers more correct answers within a short budget need not have the highest accuracy when given a longer one\.

Our contributions are:

- •A matched\-trajectory\-and\-update grid measures a breadth–refresh interaction: the effect of increasing prompt breadth changes sign\.
- •Two\-teacher comparisons document higher sampled question coverage and answer completion for the periodic models at a 16K limit\.
- •Budget curves reveal higher long\-budget accuracy from frozen\-response training at greater token cost\.

## 2Related Work

#### On\-policy distillation\.

OPD trains a student on states induced by its own generated responses while a teacher supplies token\-level supervision\([Gu et al\., 2024](https://arxiv.org/html/2609.25048#bib.bib7);[Agarwal et al\., 2024](https://arxiv.org/html/2609.25048#bib.bib1)\)\. It has become a common component of reasoning post\-training\([Yang et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib20);[Song & Zheng, 2026](https://arxiv.org/html/2609.25048#bib.bib13)\)\. Recent work studies when this supervision helps or fails, how the divergence and support affect optimization, and how to control unreliable or hard\-to\-learn token updates\([Armandpour et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib2);[Zhu et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib25);[Fu et al\., 2026a](https://arxiv.org/html/2609.25048#bib.bib5);[Xie et al\., 2026b](https://arxiv.org/html/2609.25048#bib.bib18);[Wang et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib16)\)\. Other methods shorten responses, relay teacher prefixes, or select positions and samples\([Liu et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib10);[Xie et al\., 2026a](https://arxiv.org/html/2609.25048#bib.bib17);[Zhou et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib24);[Zhang et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib23);[Xu et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib19);[Wang et al\., 2025a](https://arxiv.org/html/2609.25048#bib.bib14)\)\. Our study keeps the implemented token objective fixed and instead measures two properties of the data supplied to it: the breadth of the prompt bank and the number of student snapshots that generate the responses\.

#### Reasoning with small prompt sets\.

Supervised fine\-tuning and reinforcement learning can produce substantial reasoning gains from small curated datasets\([Ye et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib21);[Muennighoff et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib11);[Li et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib9);[Wang et al\., 2025b](https://arxiv.org/html/2609.25048#bib.bib15)\)\. These results motivate asking whether OPD also needs broad prompt coverage\. However, OPD adds a second source of variation: even with a fixed prompt bank, the state distribution changes when responses are regenerated from an updated student\. Our grid varies prompt breadth and response\-generating policy snapshots separately, revealing that the measured breadth effect changes with the refresh schedule\.

#### Concurrent one\-shot OPD\.

[Fu et al\. \(2026b\)](https://arxiv.org/html/2609.25048#bib.bib6)show that a single query can recover much of full\-data OPD’s gain and relate query diversity to the states visited during training\. Their fixed\-response experiments also find sustained learning when the state pool is held constant\. Our small\-bank results complement those observations\. We jointly vary prompt breadth and response\-generation schedules at matched trajectory and update counts, revealing a sign change in the breadth effect\. The two\-teacher budget analysis further shows that the ranking of frozen and periodically refreshed models depends on the inference limit\.

#### Freshness and replay\.

In on\-policy optimization, asynchronously generated or replayed trajectories can lag behind the learner, motivating importance correction, clipping, or freshness\-aware scheduling\([Yu et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib22);[Sheng et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib12)\)\. Experience replay can improve throughput and preserve diversity when lag is bounded\([Arnal et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib3)\)\. In OPD, f\-OPD explicitly models rollout and supervision drift and assigns sample\-level freshness scores\([Chen et al\., 2026](https://arxiv.org/html/2609.25048#bib.bib4)\)\. Our goal is complementary: we treat the number of response\-generating policy snapshots as an experimental axis and measure its interaction with prompt breadth\. TheM=1M=1condition is an intentionally extreme frozen\-response endpoint, not a claim that ordinary replay buffers are generally harmful\.

## 3Setup

#### Implemented OPD objective\.

Letπθ\\pi\_\{\\theta\}be the student,πT\\pi\_\{T\}the frozen teacher, andst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\)a prefix from a student response\. For each prefix, the implementation stores the set𝒞t\\mathcal\{C\}\_\{t\}of the student’s 16 highest\-probability tokens\. Define full\-vocabulary log probabilities

ℓt​jS=log⁡πold​\(j∣st\),ℓt​jT=log⁡πT​\(j∣st\),j∈𝒞t,\\ell^\{S\}\_\{tj\}=\\log\\pi\_\{\\mathrm\{old\}\}\(j\\mid s\_\{t\}\),\\qquad\\ell^\{T\}\_\{tj\}=\\log\\pi\_\{T\}\(j\\mid s\_\{t\}\),\\qquad j\\in\\mathcal\{C\}\_\{t\},and the student mass renormalized only for weighting,

p¯t​jS=exp⁡\(ℓt​jS\)∑k∈𝒞texp⁡\(ℓt​kS\)\.\\bar\{p\}^\{S\}\_\{tj\}=\\frac\{\\exp\(\\ell^\{S\}\_\{tj\}\)\}\{\\sum\_\{k\\in\\mathcal\{C\}\_\{t\}\}\\exp\(\\ell^\{S\}\_\{tk\}\)\}\.The detached token–candidate coefficient is

At​j=sg⁡\[p¯t​jS​\(ℓt​jT−ℓt​jS\)\]\.A\_\{tj\}=\\operatorname\{sg\}\\\!\\left\[\\bar\{p\}^\{S\}\_\{tj\}\\bigl\(\\ell^\{T\}\_\{tj\}\-\\ell^\{S\}\_\{tj\}\\bigr\)\\right\]\.Forrt​j​\(θ\)=πθ​\(j∣st\)/πold​\(j∣st\)r\_\{tj\}\(\\theta\)=\\pi\_\{\\theta\}\(j\\mid s\_\{t\}\)/\\pi\_\{\\mathrm\{old\}\}\(j\\mid s\_\{t\}\), define the dual\-clipped PPO term used by the code as

ϕ⁡\(A,r\)=\{max⁡\{−A​r,−A​clip​\(r,0\.8,1\.2\)\},A≥0,min⁡\{−3​A,max⁡\[−A​r,−A​clip​\(r,0\.8,1\.2\)\]\},A<0\.\\phi\(A,r\)=\\begin\{cases\}\\max\\\{\-Ar,\-A\\,\\mathrm\{clip\}\(r,0\.8,1\.2\)\\\},&A\\geq 0,\\\\ \\min\\\{\-3A,\\max\[\-Ar,\-A\\,\\mathrm\{clip\}\(r,0\.8,1\.2\)\]\\\},&A<0\.\\end\{cases\}For a microbatchℬ\\mathcal\{B\}, the actor loss is

ℒOPD​\(θ,ℬ\)=1∑t∈ℬmt​∑t∈ℬmt​∑j∈𝒞tϕ⁡\(At​j,rt​j​\(θ\)\),\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}\(\\theta;\\mathcal\{B\}\)=\\frac\{1\}\{\\sum\_\{t\\in\\mathcal\{B\}\}m\_\{t\}\}\\sum\_\{t\\in\\mathcal\{B\}\}m\_\{t\}\\sum\_\{j\\in\\mathcal\{C\}\_\{t\}\}\\phi\\\!\\left\(A\_\{tj\},r\_\{tj\}\(\\theta\)\\right\),\(1\)wheremtm\_\{t\}masks padding\. Microbatch losses are weighted by their sequence counts relative to the optimizer minibatch before gradient accumulation\. All runs use one PPO epoch and 128\-trajectory optimizer minibatches\. Candidate sets, coefficients, andπold\\pi\_\{\\mathrm\{old\}\}are computed before each scoring batch, which can span one or several optimizer updates\. This is a student\-top\-16, reverse\-KL\-derived surrogate: teacher log probabilities use the full vocabulary, while student weights are normalized on𝒞t\\mathcal\{C\}\_\{t\}\. The objective, optimizer, and candidate\-selection rule are shared across comparisons\.

#### Two controls of OPD data\.

Prompt breadthNNis the number of distinct prompts in the bank\. Rollout refreshMMis the number of student policy snapshots used to generate theCCtraining responses\. AtM=1M=1, all responses come fromθ0\\theta\_\{0\}; their text and prefixes remain fixed while student and teacher probabilities are recomputed before each scoring batch\. AtM=UM=U, each update receives responses from the current student\. Intermediate schedules collect a block of responses, consume it over several updates, and then regenerate from the new student\.

#### Matched trajectory and update counts\.

The central grid fixesC=14,080C=14\{,\}080trajectories andU=110U=110optimizer updates in every cell\. We varyN∈\{8,48,Full\}N\\in\\\{8,48,\\mathrm\{Full\}\\\}andM∈\{1,10,110\}M\\in\\\{1,10,110\\\}\. The eight\-prompt bank is a subset of the random 48\-prompt bank; at fixedCC, breadth and responses per prompt vary reciprocally\. AtM=10M=10, each phase collects 1,408 responses and performs 11 updates; atM=110M=110, each update collects 128 current\-policy responses\. MatchingCCandUUdoes not match token\-level computation because response lengths differ\. We therefore report trajectory/update budgets as the experimental control and treat generated and scored tokens as measured costs\.

#### Models, data, and optimization\.

The student is Qwen3\-0\.6B in non\-thinking mode and the main teacher is Qwen3\-4B\-Instruct\-2507; they share a tokenizer\([Yang et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib20)\)\. A matched refresh comparison replaces the teacher with JustRL\-DeepSeek\-1\.5B\([He et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib8)\)\. The source data are the English subset of DAPO\-Math\-17K\([Yu et al\., 2025](https://arxiv.org/html/2609.25048#bib.bib22)\)\. Filtering leaves 14,107 usable prompts; the Full runs consume 14,080 distinct prompts\. Training uses FP32, AdamW, learning rate10−610^\{\-6\}, 128 trajectories per update, one rollout per prompt occurrence, and a 16,384\-token response limit\. Appendix[A](https://arxiv.org/html/2609.25048#A1)gives the complete configuration\.

#### Evaluation\.

The primary suite contains AIME24, AIME25, and AMC23\. Each checkpoint generates 32 responses per question at temperature 1\.0, top\-pp1\.0, and a 16,384\-token limit\.*Average accuracy*is the response\-level correctness rate computed per question and then averaged within each dataset;*pass@32*is the fraction of questions with at least one correct response among the 32 stored samples\. We report the unweighted mean over the three datasets as macro3\. A wider eight\-dataset suite additionally includes AIME26, HMMT25\-Nov, HMMT26\-Feb, MATH\-500, and OlympiadBench\. The last two have four samples per question, so mixed\-suite results are called Average Accuracy rather than mean@32; coverage is reported only with an explicit commonkk\.

The answer matcher also returns a parseable\-format indicator\. We determine whether a response reaches the length cap by re\-tokenizing the stored text with the student tokenizer\. These attributes are distinct: a capped response may already contain a parseable answer\. We therefore report four mutually exclusive categories formed by parseability and whether the stored response is below the cap\. Appendix[B](https://arxiv.org/html/2609.25048#A2)defines all aggregations and uncertainty estimates\.

## 4The Breadth–Refresh Interaction

Every cell in the3×33\\times 3grid uses 14,080 trajectories and 110 optimizer updates\. Table[1](https://arxiv.org/html/2609.25048#S4.T1)reports final macro3 results; Appendix[D](https://arxiv.org/html/2609.25048#A4)gives the per\-dataset measurements\. We first examine how the value of prompt breadth changes across response\-generation schedules, then distinguish Average Accuracy from sampled question coverage\.

Table 1:Prompt breadth and rollout refresh interact under matched trajectory and update counts\.Each cell uses 14,080 student trajectories and 110 optimizer updates; rows vary unique promptsNNand columns vary the response\-generation schedule, summarized by its policy snapshotsMM\. Values are percentages on macro3: the unweighted mean over AIME24, AIME25, and AMC23 \(143 questions, 32 samples per question\)\. Below\-cap parseable means that the rule\-based matcher extracts an answer before 16,384 tokens, regardless of correctness\. Bold marks the largest observed value within each metric; every cell is one training run\.### 4\.1Increasing breadth has opposite effects at the two endpoints

With all responses generated by the initial student \(M=1M=1\), Average Accuracy falls from21\.16%21\.16\\%at eight prompts to20\.18%20\.18\\%at 48 and19\.05%19\.05\\%at Full\. Pass@32 follows the same ordering:46\.44%46\.44\\%,41\.59%41\.59\\%, and39\.68%39\.68\\%\. Under per\-update refresh \(M=110M=110\), the Average Accuracy ordering reverses:23\.61%23\.61\\%,24\.22%24\.22\\%, and25\.57%25\.57\\%\.

Leta⁡\(N,M\)a\(N,M\)denote macro3 Average Accuracy in percentage points\. The endpoint interaction is

I=\[a⁡\(Full,110\)−a⁡\(8,110\)\]−\[a⁡\(Full,1\)−a⁡\(8,1\)\]=4\.07​points\.I=\\bigl\[a\(\\mathrm\{Full\},110\)\-a\(8,110\)\\bigr\]\-\\bigl\[a\(\\mathrm\{Full\},1\)\-a\(8,1\)\\bigr\]=4\.07\\ \\text\{points\}\.\(2\)Its95%95\\%question\-paired bootstrap interval is\[2\.00,6\.28\]\[2\.00,6\.28\]\. The per\-benchmark interactions are\+4\.79\+4\.79,\+1\.88\+1\.88, and\+5\.53\+5\.53points on AIME24, AIME25, and AMC23\. The corresponding pass@32 interaction is6\.456\.45points, with interval\[0\.71,12\.69\]\[0\.71,12\.69\]\. These intervals compare the fixed trained models by resampling matched questions within each dataset \(Appendix[B](https://arxiv.org/html/2609.25048#A2)\)\.

The sign change is the central empirical result: prompt breadth has schedule\-dependent value\. Since total trajectories are fixed, a smaller bank also receives more responses per prompt\. The breadth axis therefore measures the allocation between distinct prompts and within\-prompt sampling depth\.

### 4\.2Periodic refresh yields similar mean accuracy across bank sizes

AtM=10M=10, Average Accuracy is24\.09%24\.09\\%,24\.40%24\.40\\%, and24\.51%24\.51\\%for eight, 48, and 14,080 distinct prompts\. A1,760×1\{,\}760\\timesbreadth change spans only0\.410\.41points\. Under per\-update refresh, the eight\-prompt bank is1\.961\.96points below Full\. The small\-bank result thus depends on which refresh schedule is used\.

For the three bank sizes, moving from one frozen collection to ten snapshots changes Average Accuracy by\+2\.94\+2\.94,\+4\.21\+4\.21, and\+5\.45\+5\.45points\. Moving from ten to 110 changes it by−0\.49\-0\.49,−0\.17\-0\.17, and\+1\.07\+1\.07points\. The periodic schedule captures most of the measured mean\-accuracy gain among these three tested schedules\.

Sampled question coverage remains more heterogeneous\. AtM=10M=10, pass@32 is52\.86%52\.86\\%,61\.45%61\.45\\%, and57\.71%57\.71\\%; atM=110M=110, it is58\.42%58\.42\\%,59\.22%59\.22\\%, and58\.11%58\.11\\%\. N48/M10 has the highest observed pass@32, whereas Full/M110 has the highest Average Accuracy\. A small difference in mean accuracy can therefore coexist with a larger difference in the fraction of questions reached by repeated sampling\.

### 4\.3The grid differs sharply in budget\-limited answer completion

Figure 2:Output categories across the nine\-cell grid at 16K\.Panels group prompt banks; rows showM=1,10,110M=1,10,110\. Each horizontal bar is the unweighted average of per\-dataset response proportions on AIME24, AIME25, and AMC23 \(4,576 responses per cell, 32 per question\)\. Parseability means that the answer matcher extracts an answer, regardless of correctness\. Under frozen responses, only 21\.2–24\.9% are parseable below the cap; periodic or per\-update refresh raises this proportion above 92%\.We divide responses by parseability and whether they reach the length cap\. Figure[2](https://arxiv.org/html/2609.25048#S4.F2)reports the four mutually exclusive categories, with each benchmark weighted equally\. UnderM=1M=1, only21\.2%21\.2\\%–24\.9%24\.9\\%are both parseable and below the cap\. Another48%48\\%–50%50\\%reach the cap without a parseable answer, and24%24\\%–27%27\\%fall below it without one\. UnderM∈\{10,110\}M\\in\\\{10,110\\\},92\.6%92\.6\\%–96\.8%96\.8\\%are parseable below the cap\.

The observed performance gap is accompanied by a large shift in answer completion within the evaluation budget\. The matched experiment below tests this behavior while keeping the prompt bank fixed, then asks what happens when the model is allowed to continue for longer\.

## 5Refresh Changes the Accuracy–Inference\-Cost Trade\-off

The grid links refresh to budget\-limited completion\. We now compare periodic and frozen responses on the same 48\-prompt bank under two teachers, then evaluate the resulting models at progressively larger inference budgets\.

### 5\.1Matched refresh comparisons under two teachers

Both arms use 4,800 trajectories, 40 optimizer updates, and the same objective, initialization, and 48 prompts\. The periodic arm generates 480 responses from the current student in each of ten phases\. The frozen arm generates all responses atθ0\\theta\_\{0\}\. Both consume ten batches of 480, recompute student and teacher scores before each batch, and perform four optimizer updates per batch\.

Table 2:Matched 48\-prompt comparisons at the native 16K limit\.Both arms use 4,800 trajectories and 40 updates\. Values are macro3 percentages; differences are percentage points\. Brackets give 95% question\-paired intervals for the Average Accuracy difference\. Parseability is independent of correctness\.Periodic generation raises pass@32 by 12\.69 points with the shared\-tokenizer Qwen3 teacher and by 22\.57 points with the cross\-tokenizer JustRL teacher \(Table[2](https://arxiv.org/html/2609.25048#S5.T2)\)\. Parseability rises by about 40 points in each comparison\. The Average Accuracy gains differ:\+2\.63\+2\.63points for Qwen3, whose interval includes zero, and\+6\.72\+6\.72for JustRL\. The two\-teacher comparison thus reproduces the direction of the short\-budget refresh effect, with the largest changes in sampled coverage and answer usability\.

### 5\.2Longer inference reverses the average\-accuracy ranking

For each matched arm, we evaluate one set of 32K generations on eight benchmarks\. We truncate the same responses at 4K, 8K, 16K, and 24K using the student tokenizer and apply the same answer matcher\. Each point measures the answer available in a stored prefix\. MATH\-500 and OlympiadBench use four samples per question; the other six benchmarks use 32\. Their unweighted aggregate is called Average Accuracy\.

Figure 3:The refresh advantage changes sign as the prefix budget grows\.Values are Periodic\-minus\-Frozen Average Accuracy on the eight benchmarks, using the same saved 32K generations at every prefix limit\. Positive values favor periodic refresh\. Error bars at 32K are pointwise 95% question\-paired intervals; the earlier points are aggregate estimates\. These prefix evaluations are separate from the native 16K samples in Table[2](https://arxiv.org/html/2609.25048#S5.T2)\.Periodic generation has higher aggregate Average Accuracy through 16K under both teachers \(Figure[3](https://arxiv.org/html/2609.25048#S5.F3)\)\. The measured ranking crosses between 16K and 24K with Qwen3\-4B\-Instruct, and between 24K and 32K with JustRL\. At 32K, Frozen leads by4\.394\.39points with Qwen3 \(95%95\\%interval\[3\.07,5\.75\]\[3\.07,5\.75\]\) and by2\.072\.07with JustRL \(\[0\.49,3\.67\]\[0\.49,3\.67\]\)\. Frozen is more accurate on all eight benchmarks with the Qwen3 teacher and on six of eight with JustRL, with one tie\. Appendix Table[9](https://arxiv.org/html/2609.25048#A4.T9)gives every benchmark and budget\.

### 5\.3The longer\-budget gains consume more tokens

Figure 4:Average Accuracy against realized response\-token cost\.Panels use \(a\) the Qwen3\-4B\-Instruct teacher and \(b\) the JustRL teacher on the same eight\-benchmark suite\. Points use prefix limits 4K, 8K, 16K, 24K, and 32K; the 16K and 32K limits are annotated\. At 32K, Frozen uses 1\.81 times the average response tokens of Periodic with the Qwen3 teacher and 1\.72 times with JustRL\.Equal maximum lengths do not imply equal token consumption \(Figure[4](https://arxiv.org/html/2609.25048#S5.F4)\)\. At the 32K limit, the Qwen3\-teacher Frozen model uses about 21,736 response tokens on average, versus 12,011 for Periodic\. The JustRL comparison is about 24,359 versus 14,153\. Frozen therefore consumes1\.81×1\.81\\timesand1\.72×1\.72\\timesas many tokens, respectively\. Appendix Table[7](https://arxiv.org/html/2609.25048#A4.T7)reports the costs at all five limits\.

Together, the two views expose the trade\-off: periodic refresh delivers higher accuracy within short budgets and finishes earlier; frozen\-response training yields models that use substantially longer continuations to reach higher long\-budget accuracy\. Reporting only a single evaluation limit would hide one side of this result\.

## 6Discussion

#### Prompt efficiency belongs to a training schedule\.

A prompt count alone leaves out how deeply each prompt is sampled and how often the response\-generating policy changes\. The N48/M10 cell illustrates a useful operating point: ten generation phases, 14,080 trajectories, and 110 updates yield 24\.40 Average Accuracy and 61\.45 pass@32\. Full/M110 instead achieves the highest Average Accuracy\. The appropriate allocation depends on which metric and resource the application values\.

#### Completion and accuracy must be evaluated at an explicit budget\.

Periodic models deliver more correct responses at short prefix budgets; frozen\-response models continue improving at longer budgets and overtake in Average Accuracy\. The four output categories locate the short\-budget difference in answer usability\. Conditional correctness among parseable responses addresses a different question because each arm selects a different subset\. Unconditional accuracy, sampled coverage, and actual token consumption together describe the deployment trade\-off\.

#### Changing responses changes the available supervision\.

On fixed model parameters, initial\-policy responses induce raw gradient norms 2\.53–2\.82 times those from current\-policy responses at the later checkpoints\. Their paired coordinate\-sketch cosines are 0\.127 and 0\.034 \(Appendix[E](https://arxiv.org/html/2609.25048#A5)\)\. Stale responses thus remain a substantial training signal after the student has moved\. Relating this diagnostic to completion behavior requires distinguishing response\-distribution effects from ordinary minibatch variation and from the optimizer’s transformation of raw gradients\.

#### Report both counts and schedules\.

An OPD data\-efficiency report should specify\(N,M,C,U\)\(N,M,C,U\), the trajectory and update allocation within each refresh phase, and the inference budget\. Generated tokens, teacher\-scored tokens, and wall\-clock time measure costs that trajectory counts alone cannot determine\. This separates savings in prompt acquisition, response generation, optimization, and inference\.

## 7Limitations

#### Experimental scope and uncertainty\.

The study uses one Qwen3\-0\.6B student and competition mathematics\. The nine\-cell grid uses one teacher and one nested pair of eight\- and 48\-prompt banks; the second teacher replicates the matched refresh comparison\. Each cell is one training run\. Question\-paired intervals quantify evaluation uncertainty for these trained models, rather than training\-seed variation\. The three tested schedules establish a discrete response surface, not a continuous refresh threshold\.

#### Interpretation and resources\.

At fixed trajectory count, breadth and responses per prompt vary together\. The observed inversion can reflect sampling depth, prompt identity, or their interaction\. The implemented schedules also determine how many updates consume each scored batch\. Token\-level compute and training time are not matched\. The output categories characterize completion under a cap, while conditioning on valid responses selects different subsets in the two arms\. The gradient probe measures raw norms and coordinate\-sketch directions without a fresh–fresh reference distribution; it does not identify the parameter\-update mechanism responsible for completion behavior\. The 32K budget curves assess saved prefixes of long generations, with remaining truncation at their largest budget\.

## 8Conclusion

In this study, the value of prompt breadth depends on the response\-generation schedule\. At matched trajectory and update counts, adding prompts lowers accuracy with responses frozen at the initial policy but raises it with per\-update refresh\. Periodic refresh makes a small prompt bank competitive in mean accuracy, while sampled question coverage varies across the grid\.

The inference budget changes the interpretation of these training results\. Across two teachers, periodic refresh improves short\-budget answer completion and sampled coverage; frozen\-response training yields higher 32K Average Accuracy at substantially greater token cost\. OPD data efficiency is therefore a joint property of prompt allocation, response refresh, and the budget available to the trained model\.

## Reproducibility Statement

Code and artifacts are available at[https://anonymous\.4open\.science/r/opd\-breadth\-refresh](https://anonymous.4open.science/r/opd-breadth-refresh)\. The artifact contains the training configurations, prompt\-bank definitions, analysis code, and result summaries used in the paper\. The student, teachers, and source dataset are publicly available\. Appendix[A](https://arxiv.org/html/2609.25048#A1)specifies the loss and training schedules; Appendix[B](https://arxiv.org/html/2609.25048#A2)defines evaluation and aggregation; Appendix[C](https://arxiv.org/html/2609.25048#A3)describes prompt\-bank construction\. The nine response\-surface cells are single runs\. Released validation generations permit reproducing the output\-category and inference\-budget analyses without retraining\.

## Ethics Statement

This work uses public models and competition\-mathematics problems and involves no human\-subject data collection\. Distillation can lower the cost of transferring both desirable and undesirable behavior from a larger model\. The present study focuses on experimental controls and mathematical reasoning and does not add capabilities targeted at harmful use\.

## AI Use Statement

Generative AI tools assisted with code inspection, figure generation, and language editing\. The authors verified the experimental code, reported values, citations, and final claims\.

## References

- Agarwal et al\. \(2024\)Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2306\.13649](https://arxiv.org/abs/2306.13649)\.
- Armandpour et al\. \(2026\)Mohammadreza Armandpour, Fatih Ilhan, David Harrison, Ajay Jaiswal, Duc N\. M\. Hoang, Fartash Faghri, Yizhe Zhang, Minsik Cho, and Mehrdad Farajtabar\.Unmasking on\-policy distillation: Where it helps, where it hurts, and why\.*arXiv preprint arXiv:2605\.10889*, 2026\.URL[https://arxiv\.org/abs/2605\.10889](https://arxiv.org/abs/2605.10889)\.
- Arnal et al\. \(2026\)Charles Arnal, Vivien Cabannes, Taco Cohen, Julia Kempe, and Remi Munos\.Efficient rl training for llms with experience replay\.*arXiv preprint arXiv:2604\.08706*, 2026\.URL[https://arxiv\.org/abs/2604\.08706](https://arxiv.org/abs/2604.08706)\.
- Chen et al\. \(2026\)Xianwei Chen, Shimin Zhang, and Jibin Wu\.f\-opd: Stabilizing long\-horizon on\-policy distillation with freshness\-aware control\.*arXiv preprint arXiv:2605\.17862*, 2026\.URL[https://arxiv\.org/abs/2605\.17862](https://arxiv.org/abs/2605.17862)\.
- Fu et al\. \(2026a\)Yuqian Fu, Haohuan Huang, Kaiwen Jiang, Jiacai Liu, Zhuo Jiang, Yuanheng Zhu, and Dongbin Zhao\.Revisiting on\-policy distillation: Empirical failure modes and simple fixes\.*arXiv preprint arXiv:2603\.25562*, 2026a\.URL[https://arxiv\.org/abs/2603\.25562](https://arxiv.org/abs/2603.25562)\.
- Fu et al\. \(2026b\)Zixuan Fu, Bingxiang He, Yuxin Zuo, Haohuan Huang, Jinqian Zhang, Ruhang Xiao, Cheng Qian, Qinyu Luo, Huan\-ang Gao, Yudong Wang, Zhiyuan Liu, Ning Ding, and Chaojun Xiao\.Rethinking on\-policy distillation of large language models ii: One training example\.*arXiv preprint arXiv:2609\.04172*, 2026b\.URL[https://arxiv\.org/abs/2609\.04172](https://arxiv.org/abs/2609.04172)\.
- Gu et al\. \(2024\)Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang\.Minillm: On\-policy distillation of large language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2306\.08543](https://arxiv.org/abs/2306.08543)\.
- He et al\. \(2025\)Bingxiang He, Zekai Qu, Zeyuan Liu, Yinghao Chen, Yuxin Zuo, Cheng Qian, Kaiyan Zhang, Weize Chen, Chaojun Xiao, Ganqu Cui, Ning Ding, and Zhiyuan Liu\.Justrl: Scaling a 1\.5b llm with a simple rl recipe\.*arXiv preprint arXiv:2512\.16649*, 2025\.URL[https://arxiv\.org/abs/2512\.16649](https://arxiv.org/abs/2512.16649)\.
- Li et al\. \(2025\)Xuefeng Li, Haoyang Zou, and Pengfei Liu\.Limr: Less is more for rl scaling\.*arXiv preprint arXiv:2502\.11886*, 2025\.URL[https://arxiv\.org/abs/2502\.11886](https://arxiv.org/abs/2502.11886)\.
- Liu et al\. \(2026\)Xiaogeng Liu, Xinyan Wang, Yingzi Ma, Yechao Zhang, and Chaowei Xiao\.When are teacher tokens reliable? position\-weighted on\-policy self\-distillation for reasoning\.*arXiv preprint arXiv:2605\.21606*, 2026\.URL[https://arxiv\.org/abs/2605\.21606](https://arxiv.org/abs/2605.21606)\.
- Muennighoff et al\. \(2025\)Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei\-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto\.s1: Simple test\-time scaling\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025\.
- Sheng et al\. \(2025\)Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu\.Hybridflow: A flexible and efficient rlhf framework\.In*Proceedings of the Twentieth European Conference on Computer Systems*, pp\. 1279–1297, 2025\.
- Song & Zheng \(2026\)Mingyang Song and Mao Zheng\.A survey of on\-policy distillation for large language models\.*arXiv preprint arXiv:2604\.00626*, 2026\.URL[https://arxiv\.org/abs/2604\.00626](https://arxiv.org/abs/2604.00626)\.
- Wang et al\. \(2025a\)Chao Wang, Tao Yang, Hongtao Tian, Yunsheng Shi, Qiyao Ma, Xiaotao Liu, Ting Yao, and Wenbo Ding\.Learning more with less: A dynamic dual\-level down\-sampling framework for efficient policy optimization\.*arXiv preprint arXiv:2509\.22115*, 2025a\.URL[https://arxiv\.org/abs/2509\.22115](https://arxiv.org/abs/2509.22115)\.
- Wang et al\. \(2025b\)Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen\.Reinforcement learning for reasoning in large language models with one training example\.*Advances in Neural Information Processing Systems*, 38, 2025b\.
- Wang et al\. \(2026\)Yuanyi Wang, Su Lu, Yanggan Gu, Pengkai Wang, Yifan Yang, Zhaoyi Yan, Congkai Xie, Jianmin Wu, and Hongxia Yang\.Not all disagreement is learnable: Token teachability in on\-policy distillation\.*arXiv preprint arXiv:2605\.26844*, 2026\.URL[https://arxiv\.org/abs/2605\.26844](https://arxiv.org/abs/2605.26844)\.
- Xie et al\. \(2026a\)Yan Xie, Sijie Zhu, Tiansheng Wen, Bo Chen, and Yifei Wang\.On the position bias of on\-policy distillation\.*arXiv preprint arXiv:2606\.22600*, 2026a\.URL[https://arxiv\.org/abs/2606\.22600](https://arxiv.org/abs/2606.22600)\.
- Xie et al\. \(2026b\)Zhengpeng Xie, Li Lyna Zhang, Zeke Xie, and Mao Yang\.Trust region policy distillation\.*arXiv preprint arXiv:2607\.04751*, 2026b\.URL[https://arxiv\.org/abs/2607\.04751](https://arxiv.org/abs/2607.04751)\.
- Xu et al\. \(2026\)Haolei Xu, Xiaowen Xu, Haiwen Hong, Zixuan Ni, Hongxing Li, Yiwen Qiu, Weiming Lu, and Yongliang Shen\.Pass the baton: Trajectory\-relayed on\-policy distillation\.*arXiv preprint arXiv:2607\.26057*, 2026\.URL[https://arxiv\.org/abs/2607\.26057](https://arxiv.org/abs/2607.26057)\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Ye et al\. \(2025\)Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu\.Limo: Less is more for reasoning\.*arXiv preprint arXiv:2502\.03387*, 2025\.URL[https://arxiv\.org/abs/2502\.03387](https://arxiv.org/abs/2502.03387)\.
- Yu et al\. \(2025\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.Dapo: An open\-source llm reinforcement learning system at scale\.*Advances in Neural Information Processing Systems*, 38, 2025\.
- Zhang et al\. \(2026\)Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D\. Lyng, Sanjit Singh Batra, and Robert E\. Tillman\.Fast and effective on\-policy distillation from reasoning prefixes\.*arXiv preprint arXiv:2602\.15260*, 2026\.URL[https://arxiv\.org/abs/2602\.15260](https://arxiv.org/abs/2602.15260)\.
- Zhou et al\. \(2026\)Ziheng Zhou, Jiaqi Li, Huacong Tang, Ying Nian Wu, and Demetri Terzopoulos\.Less is more: Early stopping rollout for on\-policy distillation\.*arXiv preprint arXiv:2605\.27028*, 2026\.URL[https://arxiv\.org/abs/2605\.27028](https://arxiv.org/abs/2605.27028)\.
- Zhu et al\. \(2026\)Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu\.The many faces of on\-policy distillation: Pitfalls, mechanisms, and fixes\.*arXiv preprint arXiv:2605\.11182*, 2026\.URL[https://arxiv\.org/abs/2605\.11182](https://arxiv.org/abs/2605.11182)\.

## Appendix ATraining Protocol

### A\.1Objective and update unit

All central experiments use the student\-top\-16 objective in Equation[1](https://arxiv.org/html/2609.25048#S3.E1):top\_k=16,top\_k\_strategy=only\_stu, andreward\_weight\_mode=student\_p\. Teacher log probabilities at these candidate IDs are taken from the teacher’s full\-vocabulary softmax\. Student weights are renormalized within the candidate set\. The resulting coefficients are detached before entering the PPO surrogate\. No task reward, verifier, selective\-token mask, entropy bonus, or auxiliary KL loss contributes to the actor objective\.

Each optimizer update consumes a PPO minibatch of 128 trajectories\. A trainer iteration loads and scores either one minibatch or a phase containing several minibatches, as specified below\. Dynamic microbatches accommodate long contexts; their token\-mean losses are weighted by their sequence count relative to the PPO minibatch before gradient accumulation\. Training uses eight FSDP ranks and one PPO epoch, so the main\-grid budgetU=110U=110means 110 AdamW steps\.

We fix data order and disable shuffling\. Appendix[C](https://arxiv.org/html/2609.25048#A3)describes the prompt\-bank construction\. Checkpoints are evaluated at the specified final step rather than selected post hoc by a validation benchmark\.

### A\.2Refresh schedules

For the central grid,C=14,080C=14\{,\}080andU=110U=110in every cell\.

- •M=1M=1: all 14,080 response texts are generated fromθ0\\theta\_\{0\}and cached\. They are partitioned into 110 consecutive update batches and each trajectory is consumed once\. At consumption time, the current student support and probabilities and the teacher probabilities are recomputed on the cached prefixes\.
- •M=10M=10: each of ten phases generates 1,408 responses from the current student snapshot, then partitions them into 11 update batches of 128\.
- •M=110M=110: every update first generates 128 responses from the current student\. This is the per\-update\-refresh condition\.

Student supports and OPD coefficients are recomputed before each trainer batch\. A phase with several optimizer minibatches retains that batch’s scoring snapshot for the PPO ratios\. Thus frozen responses refer to initial\-policy text, not to a permanently frozen set of OPD coefficients\.

For the matchedC=4,800C=4\{,\}800,U=40U=40comparisons in Section[5\.1](https://arxiv.org/html/2609.25048#S5.SS1), the periodic arm has ten phases\. Each phase generates 480 responses and consumes them in four updates of sizes128/128/128/96128/128/128/96\. The frozen arm generates all 4,800 responses atθ0\\theta\_\{0\}and consumes them in the same 40\-update partition\. These runs share prompts, initialization, objective, and update schedule within each teacher comparison\.

For the cross\-tokenizer teacher, student candidate strings are mapped to teacher token IDs before teacher scoring\. Candidates without a single\-token mapping are masked from the support\. Eight Qwen3 control tokens are unmappable under this rule; the semantic EOS mapping is retained\. This comparison simultaneously changes teacher model, training history, and tokenizer and is used only as a replication of the direction of the refresh effect\.

### A\.3Gradient diagnostic

The probe in Section[E](https://arxiv.org/html/2609.25048#A5)performs no optimizer step\. It uses 48 prompts and 128 trajectories\. Unlike main training, it sets the PPO minibatch size to eight, producing 16 raw minibatch gradients per condition\. Exact norms use the complete parameter gradient\. Directional measurements select the same 16 deterministic coordinates per parameter tensor and FSDP rank across all conditions, for 3,712 coordinates in total\.

## Appendix BMeasurement Details

### B\.1Evaluation suites

The primary suite \(macro3\) contains AIME24 \(30 questions\), AIME25 \(30\), and AMC23 \(83\), totaling 143 questions and 4,576 responses per checkpoint\. Each question has 32 independent samples\. Macro3 is the unweighted mean of the three per\-dataset metrics, rather than a question\-count\-weighted pool\.

The inference\-budget analysis uses an eight\-dataset suite that adds AIME26, HMMT25\-Nov, HMMT26\-Feb, MATH\-500, and OlympiadBench\. The first six datasets use 32 samples per question; MATH\-500 and OlympiadBench use four\. We aggregate only single\-sample Average Accuracy across all eight\. We do not assign a pass@32 label to this mixed\-sample aggregate\.

Main evaluation uses temperature 1\.0, top\-pp1\.0, non\-thinking mode, and a 16,384\-token response limit\. The budget analysis generates one 32K sample set per arm, truncates the stored responses at 4K, 8K, 16K, 24K, and 32K, and reruns the same answer matcher on every prefix\. Native 16K evaluation and the post\-hoc 16K prefix of a 32K generation are different Monte Carlo samples and need not have identical point estimates\.

### B\.2Metrics and response categories

For questioniiwithnin\_\{i\}samples andcic\_\{i\}correct samples, Average Accuracy isci/nic\_\{i\}/n\_\{i\}, averaged first over questions in a dataset and then over datasets\. For macro3, pass@32 is the fraction of questions withci\>0c\_\{i\}\>0among the 32 stored samples\. It is an observed finite\-sample coverage statistic, not a claim about nonzero support under unlimited sampling\.

Appendix Figure[6](https://arxiv.org/html/2609.25048#A4.F6)additionally reports the probability thatkkdraws with replacement from a question’s empirical 32\-sample distribution contain a correct response,

coverageiWR​\(k\)=1−\(1−ci32\)k\.\\mathrm\{coverage\}^\{\\mathrm\{WR\}\}\_\{i\}\(k\)=1\-\\left\(1\-\\frac\{c\_\{i\}\}\{32\}\\right\)^\{k\}\.This diagnostic is called empirical with\-replacement coverage in the text and figure metadata\. Atk=32k=32it differs from observed pass@32\.

The answer matcher returns both correctness and a parseability indicator\. Cap status is determined by retokenizing the stored response with the student tokenizer: length at least the nominal limit is “at cap\.” Parseability and cap status can overlap, so Figure[2](https://arxiv.org/html/2609.25048#S4.F2)uses four mutually exclusive categories: below\-cap parseable, below\-cap unparseable, at\-cap parseable, and at\-cap unparseable\. Correctness is computed separately within each category\. A response is called valid only as shorthand for below\-cap and parseable; it is not used to define the main accuracy metric\.

Scoring uses the same version ofttrl\_math\.compute\_scorefor every arm, checkpoint, and prefix budget\. The matcher processes the stored text as given, so a response that reaches the cap can still be parseable or correct if an accepted final answer occurs before the cutoff\.

### B\.3Uncertainty

Intervals in Table[2](https://arxiv.org/html/2609.25048#S5.T2)use 10,000 bootstrap draws\. Each draw resamples matched questions within each dataset and averages the three per\-dataset differences\. The endpoint interaction in Equation[2](https://arxiv.org/html/2609.25048#S4.E2)pairs questions across its four cells before forming the difference of breadth effects; it uses 20,000 draws with seed 20260906\. The 32K differences in Figure[3](https://arxiv.org/html/2609.25048#S5.F3)use 20,000 analogous paired draws across eight datasets, with seed 20260907\.

These percentile 95% intervals describe question\-level evaluation uncertainty for fixed trained models and their stored samples\. The budget\-curve intervals are pointwise and available at 32K; earlier prefixes retain their aggregate estimates without confidence bands\. Each training cell is one run\. Gradient uncertainties are standard errors over the 16 diagnostic minibatches unless stated otherwise\.

### B\.4Conditional analyses

Conditional correctness among below\-cap parseable responses uses pooled responses within each dataset and then takes the unweighted dataset macro average\. It is descriptive because the frozen and periodically refreshed arms select different response subsets\. A stricter matched\-valid analysis pairs responses by question and within\-question sample index and retains an index only when both arms are below\-cap and parseable\. This removes the response\-category mismatch for that subset, but still does not pair latent reasoning paths or identify a causal effect of completion\.

## Appendix CPrompt Banks

### C\.1Source corpus

All prompts come from DAPO\-Math\-17K \(English split\),14,11614\{,\}116raw rows\. After filtering to rows that fit the Qwen3\-0\.6B chat template under a1,0241\{,\}024prompt\-token limit,14,10714\{,\}107usable rows remain\. Runs labelled “Full” consume the first14,08014\{,\}080of these, matchingCCexactly\.

### C\.2Selection

The4848\-prompt random bank is drawn with seed20260829without reading rewards, difficulty labels, or any model output\. The88\-prompt bank is a strict subset of the4848\-prompt bank\. The difficulty\-selected4848\-prompt bank of Table[4](https://arxiv.org/html/2609.25048#A4.T4)is the union of two teacher–student frontier stages \(2525and3434prompts, overlap1111\) and is the only bank in the paper that consults model behavior\.

### C\.3Trajectories per prompt

AtC=14,080C=14\{,\}080: approximately1,7601\{,\}760trajectories per prompt atN=8N=8,293293atN=48N=48, and11atN=FullN=\\text\{Full\}\. AtC=4,800C=4\{,\}800andN=48N=48:100100per prompt\.

## Appendix DAdditional Results

### D\.1The matched trajectory/update grid in numbers

Table 3:Per\-dataset measurements for all nine prompt–refresh cells\(percent\)\. Average Accuracy is response\-level correctness\. Empirical WR coverage@32 is the plug\-in probability1−\(1−p^q\)321\-\(1\-\\hat\{p\}\_\{q\}\)^\{32\}averaged over questions, whereas observed pass@32 is the fraction of questions with at least one correct stored response\. Parseable is the fraction from which the matcher extracts an answer, regardless of correctness\. Macro3 is the unweighted mean over AIME24, AIME25, and AMC23\.\(a\) Average Accuracy

\(b\) Empirical WR coverage@32

\(c\) Observed pass@32

\(d\) Parseable responses

Figure 5:Most of the observed Average Accuracy change occurs between one and ten policy snapshots, while coverage remains heterogeneous\.A curve view of Table[1](https://arxiv.org/html/2609.25048#S4.T1);MMis placed at equal spacing rather than to scale\. For every bank size the step from one frozen collection to ten snapshots produces the largest measured Average Accuracy improvement, and an eleven\-fold further increase changes it by at most1\.071\.07points\. pass@32 does not share the pattern: it rises5\.565\.56points fromM=10M=10to110110atN=8N=8and spreads8\.588\.58points across banks atM=10M=10\. Each marker is one training run; no cross\-seed uncertainty is implied\.
### D\.2Exploratory anchor runs

Table 4:Anchor runs on macro3\.These exploratory configurations motivated the controlled grid\. The Impulse point changes both update count and learning rate and is not a causal test of whether one direction suffices\.Figure 6:Empirical coverage across exploratory anchor runs\.Curves draw with replacement from each question’s 32 stored samples and average equally across the three datasets\. The difficulty\-selected 48\-prompt run and Full occupy a similar region; the single tested Impulse configuration remains near Base\. These runs have the different training budgets listed in Table[4](https://arxiv.org/html/2609.25048#A4.T4)\.Table 5:macro3empirical with\-replacement coverage\. The gap between periodically refreshed and frozen\-response arms generally grows with the number of draws in this plug\-in statistic\.
### D\.3Mutually exclusive output categories

Table 6:Response categories on macro3\(percent of responses\)\. BP/BU denote below\-cap parseable/unparseable; AP/AU denote at\-cap parseable/unparseable\. The four columns are mutually exclusive and sum to 100 up to rounding\.
### D\.4Generation behavior across inference budgets

Table 7:Periodic\-minus\-frozen Average Accuracy differences \(percentage points\) and frozen/periodic response\-token ratios on the eight\-benchmark suite\. Each budget uses prefixes of the same 32K generations\. MATH\-500 and OlympiadBench have four samples per question; the other datasets have 32\. All aggregates weight datasets equally\.Table 8:macro8generation statistics behind Table[7](https://arxiv.org/html/2609.25048#A4.T7)\.Qwen3\-4B\-Instruct teacherJustRL teacherBudgetPeriodic tok\.Frozen tok\.ratioPeriodic tok\.Frozen tok\.ratio4K3,5383,8081\.08×1\.08\\times3,7553,9931\.06×1\.06\\times8K6,4837,2541\.12×1\.12\\times6,8437,7111\.13×1\.13\\times16K10,49113,3821\.28×1\.28\\times11,05314,3281\.30×1\.30\\times24K11,75118,3521\.56×1\.56\\times13,17119,9411\.51×1\.51\\times32K12,01121,7361\.81×1\.81\\times14,15324,3591\.72×1\.72\\times*Truncation rate / format rate \(%\)*4K77\.7 / 23\.488\.1 / 13\.283\.4 / 21\.093\.9 / 8\.316K29\.1 / 72\.768\.8 / 33\.836\.2 / 67\.974\.4 / 28\.332K1\.5 / 97\.431\.7 / 73\.98\.4 / 92\.845\.9 / 58\.0Table 9:Periodic\-minus\-frozen Average Accuracy by benchmark and inference budget\.Every entry is a percentage\-point difference computed by truncating and rescoring the same saved 32K generations\. AIME, AMC, and HMMT use 32 samples per question; MATH\-500 and OlympiadBench use four\. Positive values favor periodic response refresh; negative values favor frozen\-response replay\. The two teacher blocks are separate matched experiments\.
### D\.5Gradient geometry

Figure 7:Curve\-valued gradient geometry from the fixed\-parameter probe\.Each curve is computed from 16 diagnostic minibatch gradients in the shared 3,712\-coordinate sketch\. \(a\) Fresh sketches need 10–11 singular components to retain 90% of their energy, which is energy concentration but not strong low rank\. \(b\) Principal angles between the three fresh rank\-90 checkpoint subspaces are shown descriptively; no matched random\-subspace baseline is available\. Scalar norm and fresh/stale cosine measurements are reported exactly in Table[11](https://arxiv.org/html/2609.25048#A5.T11)rather than duplicated here\.Table 10:Principal angles between rank\-90 subspaces spanned by fresh diagnostic minibatch gradients\. These coordinate\-sketch values are descriptive: without a matched random\-subspace baseline they do not calibrate how much training\-induced rotation occurred, and the maximum principal cosine is not the cosine between the first singular vectors\.

## Appendix EA Fixed\-Parameter Gradient Diagnostic

The behavioral results show that frozen\-response training produces many fewer usable answers within a 16K budget\. We next ask a narrower mechanistic question: at the same current parameters, how different are OPD gradients computed from current\-policy responses and from responses generated by the initial policy? This diagnostic characterizes the training signals; it does not by itself establish that their difference causes the output behavior in Section[5](https://arxiv.org/html/2609.25048#S5)\.

#### Probe design\.

We update no parameters\. At checkpointsθ0\\theta\_\{0\},θ55\\theta\_\{55\}, andθ110\\theta\_\{110\}of the full\-bank run, we compute gradients on a held\-fixed bank of 48 prompts and 128 trajectories\. A*fresh*condition generates responses from the checkpoint being probed\. A*stale\-0*condition reuses responses generated atθ0\\theta\_\{0\}, while recomputing student top\-16 supports, student probabilities, teacher probabilities, and gradients at the current checkpoint\. The probe deliberately uses PPO minibatches of eight, yielding 16 diagnostic minibatch gradients per condition; the main training runs instead use one 128\-trajectory minibatch per update\. Full\-parameterL2L\_\{2\}norms are computed exactly\. Directional statistics use the same deterministic 3,712\-coordinate parameter sketch in every condition, and fresh/stale comparisons pair the same minibatch index\.

Table 11:Gradients on fixed model parameters\.Means and standard deviations are over 16 diagnostic minibatches\. Norms use all parameters; cosine uses a fixed coordinate sketch\. Parentheses after cosine give its standard error over paired minibatches\.
#### Stale and fresh responses induce different raw gradients\.

The fresh gradient norm decreases from19\.4319\.43at initialization to5\.425\.42atθ55\\theta\_\{55\}and5\.065\.06atθ110\\theta\_\{110\}\. On the latter two parameter snapshots, stale\-0 responses yield norms of13\.7213\.72and14\.2614\.26, respectively:2\.53×2\.53\\timesand2\.82×2\.82\\timesthe corresponding fresh\-response norms\. Their paired sketch cosines are0\.1270\.127and0\.0340\.034\. Thus a response collection can remain a strong source of gradient after the student policy has moved, while agreeing little with the gradient obtained from responses generated by the current student\.

These are raw loss gradients, not realized AdamW parameter updates\. The probe does not include optimizer moments, gradient clipping, or a fresh\-versus\-fresh null distribution\. High\-dimensional vectors can have small cosine even under benign variation, so we use the cosine values only as evidence that the two signals are not interchangeable, rather than as a calibrated notion of orthogonality\.

#### Spectrum definitions\.

Ifλi\\lambda\_\{i\}are the squared singular values of the 16\-row gradient\-sketch matrix, participation rank is\(∑iλi\)2/∑iλi2\(\\sum\_\{i\}\\lambda\_\{i\}\)^\{2\}/\\sum\_\{i\}\\lambda\_\{i\}^\{2\}\. Rank90 is the smallest number of leading components whose cumulativeλi\\lambda\_\{i\}mass reaches 90%\.

#### The sampled gradient set is not strongly low\-rank\.

Across conditions, the leading direction contains3232–40%40\\%of the sampled gradient energy, but retaining90%90\\%requires 10–11 of the 16 available singular components\. The participation rank is 5\.15–6\.95 and rises across the fresh checkpoints\. At this minibatch granularity, the measurements therefore do not support the hypothesis that a fixed, very low\-dimensional update subspace explains why small prompt banks can work\. Appendix[D](https://arxiv.org/html/2609.25048#A4)reports the remaining descriptive spectrum and subspace statistics\.

## Appendix FExploratory and Inconclusive Findings

#### The tested difficulty selector did not improve the 48\-prompt bank\.

With per\-update refresh and the same 14,080\-trajectory, 110\-update budget, the difficulty\-selected bank reaches 23\.98 Average Accuracy and 58\.51 pass@32; the random bank reaches 24\.22 and 59\.22\. This comparison evaluates one selector and one random bank, so it neither establishes equivalence nor implies that prompt identity is irrelevant\.

#### One amplified initial update remains inconclusive\.

The Impulse configuration accumulates 480 initial\-policy trajectories into one optimizer step at ten times the main learning rate\. It reaches 10\.43 Average Accuracy, compared with 8\.38 before training and 23\.56 for the 4,800\-trajectory, 40\-update periodic run\. Because trajectory count, optimizer steps, learning rate, and optimizer history all differ, this point shows only that the tested single\-update construction does not reproduce periodic training\.

#### Sampled gradients are not strongly low\-rank\.

Rank90 is 10–11 among 16 diagnostic minibatch gradients, and participation rank ranges from 5\.15 to 6\.95\. The leading component contains 32–40% of the sampled energy, but the remaining energy occupies most available directions\. This probe does not support a fixed very\-low\-rank explanation for prompt\-bank compression at the measured granularity\.

#### Subspace comparisons are descriptive\.

Stale/fresh paired sketch cosine is 0\.127 atθ55\\theta\_\{55\}and 0\.034 atθ110\\theta\_\{110\}, while stale projection residual on the fresh rank\-90 subspace is 0\.743 and 0\.727\. The probe has no fresh\-versus\-fresh null distribution and only 16 minibatches per condition\. These values therefore characterize the sampled coordinate sketches but do not define a calibrated staleness threshold or a monotone law with lag\.

#### Average Accuracy and sampled coverage do not share one saturation pattern\.

AtM=10M=10, Average Accuracy spans only 0\.41 points across prompt banks, whereas pass@32 spans 8\.58 points non\-monotonically\. FromM=10M=10toM=110M=110, pass@32 still rises by 5\.56 points forN=8N=8\. Small\-bank sufficiency in the main text is therefore stated for Average Accuracy at the tested periodic schedule, not as a universal claim about all sampling metrics\.

#### The breadth inversion has multiple compatible explanations\.

HoldingCCfixed makes prompt breadth and trajectories per prompt reciprocal:N=8N=8receives 1,760 trajectories per prompt, whereas Full receives about one\. The observed inversion atM=1M=1may reflect sampling depth, prompt identity, or their interaction\. The present experiment measures the joint response surface without separating these explanations\.