Scaffold-Mediated Post-Training: Co-Evolving Model Parameters and Procedural Scaffold Graphs

# Scaffold-Mediated Post-Training: Co-Evolving Model Parameters and Procedural Scaffold Graphs
Source: [https://arxiv.org/html/2608.05156](https://arxiv.org/html/2608.05156)
Fei Ding Yongkang Zhang Runhao Liu Alibaba Group &Yuhao Liao Zijian Zeng Huiming Yang Tsinghua University

###### Abstract

Post\-training of large language models optimizes only parameters, while inference\-time procedural scaffolds are typically designed independently of parameter training\. This disconnect makes it difficult to automatically acquire and internalize complex strategies\. We proposescaffold\-mediated post\-training: procedural scaffolds are organized into an evolvable graph structure that co\-evolves with model parameters through discovery, distillation, and dynamic recompilation\. We instantiate this paradigm as Skill Training\. On FeatureBench, automatically discovered skills improve the passed rate by 8\.1pp, and after progressive distillation the model still achieves a 27\.7% passed rate without any external scaffold \(distillation retention rate 85\.2%, defined as post\-distillation / with\-skill passed rate\), significantly outperforming standard SFT on the same data\.

Co\-evolutionScaffold GraphGtG\_\{t\}Data GenerationModelMt\+1M\_\{t\+1\}Recompilationskill nodes\+dep\. edgessubgraph→\\toexec→\\toVV\-verifybottom\-up distill\.\+KLremove→\\torewrite→\\tore\-verifyCC\-select·II\-assembleℒ\+β​DKL\\mathcal\{L\}\+\\beta D\_\{\\text\{KL\}\}retaink≥τ\\text\{retain\}\_\{k\}\\\!\\geq\\\!\\tau?Gt\+1G\_\{t\+1\}Seed DataFigure 1:Co\-evolution loop in scaffold\-mediated post\-training\.GtG\_\{t\}assembles task\-specific subgraphs and generatesVV\-verified training data; the model internalizes skill behaviors via progressive distillation; skills meeting the retention threshold are removed and dependent scaffolds are recompiled intoGt\+1G\_\{t\+1\}\.## 1Introduction

The parameter\-learning bottleneck\.Post\-training of large language models \(LLMs\) typically treats only model parameters as trainable\. However, the procedural strategies required by complex tasks \(how to decompose requirements, edit multiple files in topological order, and verify correctness after each step\) are difficult to learn from scarce training examples alone\(Yang et al\.,[2025b](https://arxiv.org/html/2608.05156#bib.bib14)\)\. Evaluations on FeatureBench\(Zhou et al\.,[2026](https://arxiv.org/html/2608.05156#bib.bib19)\)show that Claude Opus 4\.5 achieves a 76\.8% solve rate on SWE\-bench\(Jimenez et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib4)\)yet only a 45\.5% passed rate on FeatureBench, revealing deficiencies in procedural strategies\. FeatureBench focuses on end\-to\-end feature development \(involving multi\-file coordination, architectural decisions, and multi\-step verification workflows\), placing higher demands on procedural strategies\.

The limits of scaffold engineering\.Inference\-time scaffolds such as chain\-of\-thought\(Wei et al\.,[2022](https://arxiv.org/html/2608.05156#bib.bib10)\), ReAct\(Yao et al\.,[2023](https://arxiv.org/html/2608.05156#bib.bib15)\), and agentic workflows can substantially improve performance on complex tasks, but they are typically hand\-designed and bolted on at inference time; as scaffolds accumulate, inference becomes increasingly complex and slower\.

Scaffold\-mediated post\-training\.This paper proposes a new post\-training paradigm that bridges parameter learning and scaffold engineering:procedural scaffolds themselves become evolvable objects that co\-evolve with model parameters\. The model first solves tasks with the aid of an external procedural scaffold graph, which generates high\-quality supervised data\. The model then absorbs the scaffold’s behavior through distillation\. Once a scaffold has been internalized into parameters, the external graph undergoesdynamic recompilation: the internalized skill is removed from the graph, and higher\-level scaffolds that depend on it are automatically rewritten, closing the co\-evolution loop between parameter learning and scaffold learning\.

We instantiate this paradigm asSkill Training\. A skill is a node in the procedural scaffold graph, comprising metadata \(meta\), an activation condition \(CC\), an execution procedure \(PP\), a verification checkpoint \(VV\), a compositional interface \(II\), and anti\-patterns \(EE\)\. The framework operates in three stages \(Figure[1](https://arxiv.org/html/2608.05156#S0.F1)\):

1. 1\.Skill Discovery: Structured skills are automatically discovered from seed data to construct an initial procedural scaffold graph\.
2. 2\.Scaffold\-Graph\-Augmented Data Generation: For each task, a scaffold subgraph is assembled to guide the model through the task and generate verified training data\.
3. 3\.Progressive Distillation with Dynamic Recompilation: Starting from the most frequently invoked bottom\-level skills, progressive distillation internalizes skill behavior into model parameters\. Internalized skills are removed from the scaffold graph, and higher\-level scaffolds that depend on them are automatically rewritten and re\-verified\.

The main contributions of this paper are:

- •A new paradigm: We propose scaffold\-mediated post\-training, in which model parameters and an external procedural scaffold graph co\-evolve through discovery, assembly, distillation, and dynamic recompilation\.
- •A new representation: We formalize skills as nodes in a procedural scaffold graphG=\(S,D\)G=\(S,D\), each containing the six\-tuple\(meta,C,P,V,I,E\)\(\\textit\{meta\},C,P,V,I,E\)\. Explicit skill calls inPPinduce dependency edges in the graph,IIdefines typed labels for inter\-node data flow, andVVprovides quality\-filtering signals for distillation\.
- •A new mechanism: We propose dynamic recompilation: once a skill has been distilled and internalized into parameters, the scaffold graph automatically rewrites higher\-level scaffolds that depend on it, enabling co\-evolution between parameters and scaffolds\.
- •Empirical validation: On FeatureBench, automatically discovered skills raise the passed rate from 24\.4% to 32\.5%; after progressive distillation the rate reaches 31\.5% \(22 skills internalized\); pure internalization without external scaffolds still achieves 27\.7% \(retention rate 85\.2%\), all significantly outperforming standard SFT on the same data \(25\.5%\)\. Ablations show thatVVandPPcontribute 6\.9pp and 4\.2pp to the retention rate, respectively\.

Limitations remain: this work validates the framework only in code generation and uses self\-distillation from a single model without exploring teacher diversity\. The framework itself is domain\-agnostic, and extending it to other domains with executable verifiers, such as mathematical reasoning, is an important direction for future work \(Section[7](https://arxiv.org/html/2608.05156#S7)\)\.

## 2Related work

### 2\.1Parameter learning

SFT and RLHF compress capabilities directly into model parameters\. STaR\(Zelikman et al\.,[2022](https://arxiv.org/html/2608.05156#bib.bib16)\)and Quiet\-STaR\(Zelikman et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib17)\)achieve self\-improvement by bootstrapping reasoning chains\. Context Distillation\(Snell et al\.,[2022](https://arxiv.org/html/2608.05156#bib.bib6)\)distills rich contexts into weights\. These methods can update parameters but lack explicit, composable, and verifiable external procedural structures; their distillation targets are flat text or reasoning chains that lack explicit composable interfaces for programmatic cross\-task reuse\.

### 2\.2Scaffold learning

APE\(Zhou et al\.,[2023](https://arxiv.org/html/2608.05156#bib.bib20)\)and OPRO\(Yang et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib13)\)automatically optimize prompts; Promptbreeder\(Fernando et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib3)\)optimizes prompts via evolutionary algorithms, yet all produce flat text\. Voyager\(Wang et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib7)\)builds a composable skill library; ExpeL\(Zhao et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib18)\)extracts natural\-language rules; SkillRL\(Xia et al\.,[2026](https://arxiv.org/html/2608.05156#bib.bib11)\)jointly trains skills with RL; Skill\-It\!\(Chen et al\.,[2023](https://arxiv.org/html/2608.05156#bib.bib1)\)studies data dependencies among skills\. DSPy\(Khattab et al\.,[2024](https://arxiv.org/html/2608.05156#bib.bib5)\)treats prompts as composable modules; SAGE\(Wang et al\.,[2026](https://arxiv.org/html/2608.05156#bib.bib8)\)accumulates a skill library within an RL framework\. These methods construct external scaffolds but alllack bidirectional co\-evolution between parameters and the scaffold graph\.

### 2\.3Bridging the two: scaffold\-mediated post\-training

This paper connects parameter learning and scaffold learning: procedural scaffolds are first discovered and used to guide the model in solving tasks and generating training data \(the strength of scaffold learning\), then distilled and internalized into parameters \(the strength of parameter learning\), after which the scaffold graph undergoes dynamic recompilation\. This “learn→\\rightarrowinternalize→\\rightarrowrecompile” loop has not been realized in prior work\.

## 3Method: scaffold\-mediated post\-training

Engineering motivation\.In engineering practice, we built a system of 136\+ manually authored skills covering Test\-Driven Development \(TDD\), debugging, and code review workflows\. These skills closely match the formal definition in structure \(Appendix[D](https://arxiv.org/html/2608.05156#A4)\), but manual design is prohibitively expensive, directly motivating the automated training research below\.

This section first defines the procedural scaffold graph \(§3\.1\), then describes three stages: skill discovery \(§3\.2\)→\\rightarrowscaffold\-graph\-augmented data generation \(§3\.3\)→\\rightarrowprogressive distillation and dynamic recompilation \(§3\.4–3\.5\)\. The first two stages constitutescaffold learning; the third constitutesparameter learning and scaffold\-graph co\-evolution\(Figure[2](https://arxiv.org/html/2608.05156#S3.F2)\)\.

### 3\.1Procedural scaffold graph

Definition 1\(Skill\)\. A skillssis a node in the procedural scaffold graph, comprising the following core components:

s=\(meta,C,P,V,I,E\)s=\(\\textit\{meta\},C,P,V,I,E\)where:

- •meta\(metadata\): name, description, category tags, available tool list, and version number\.
- •CC\(activation condition\): task characteristics and triggering rules that determine when the skill is activated \(the LLM automatically evaluates the match against the task description and executes upon match\)\.
- •PP\(execution procedure\): structured workflow steps, categorized as eitherrigid\(strict step ordering with hard constraints at each step\) orflexible\(advisory patterns that permit discretionary step selection\)\.
- •VV\(verification checkpoint\): checkable conditions that must be satisfied after skill execution\. Following theevidence\-drivenprinciple,VVrequires executable evidence rather than model assertions\. Verification may comprise multiple layers \(build→\\rightarrowtype\-check→\\rightarrowtest→\\rightarrowsecurity→\\rightarrowchange audit\); in rigid skills these are executed sequentially with fail\-fast semantics \(Appendix[I](https://arxiv.org/html/2608.05156#A9)\)\.
- •I=\(I\.in,I\.out\)I=\(I\.\\text\{in\},I\.\\text\{out\}\)\(compositional interface\): standardized input/output type label sets, enabling multiple skills to be chained according to dependency relations\.
- •EE\(anti\-patterns\): common failure modes that help the model avoid known pitfalls\.

In practice, each skill is implemented as a Markdown file with YAML frontmatter; the design derives from iterative experience with over 136 manually authored skills\.

Definition 2\(Procedural Scaffold Graph\)\. A scaffold graphG=\(S,D\)G=\(S,D\), whereSSis the set of skill nodes andDDis the set of dependency edges induced by explicit skill calls inPP: ifPbP\_\{b\}explicitly invokessas\_\{a\}, an edge fromsas\_\{a\}tosbs\_\{b\}is added\.IIprovides typed labels \(e\.g\.,file\_list,test\_result\) for the data flowing along these edges at runtime\. Acyclicity is verified via depth\-first search \(DFS\) after graph construction: if a node reappears in the path stack, the cycle is identified and its chain reported, the offending last edge is discarded, and the check is retried\. The graph of 30 skills in our experiments is always a DAG\. Composite skills correspond to executing a scaffold subgraph: nodes are scheduled in topological order\.VVprovides verification feedback after each node executes\.

Multi\-round discovery produces hierarchical skill structures: higher\-level skills discovered in later rounds explicitly invoke lower\-level skills in theirPP, from which dependency edges naturally arise; Algorithm[1](https://arxiv.org/html/2608.05156#alg1)formalizes these call relationships via interface matching \(Ia\.out∩Ib\.in≠∅I\_\{a\}\.\\text\{out\}\\cap I\_\{b\}\.\\text\{in\}\\neq\\emptyset\)\.

Key distinction from prompts\.A conventional prompt is flat text; skills possess three structural properties: \(1\)Verifiability—VVenables automatic evaluation of execution outcomes, providing a filtering signal for distillation; \(2\)Composability—IIdefines standardized interfaces that allow multiple skills to be chained; \(3\)Conditional activation—CCautomates skill selection\. Ablation experiments \(Section[5\.3](https://arxiv.org/html/2608.05156#S5.SS3)\) quantify the contributions ofVVandII\.

In practice, skills are classified as eitherrigid\(enforcing step ordering and verification checkpoints\) orflexible\(advisory patterns permitting flexible selection\)\. Ablation experiments \(Section[5\.3](https://arxiv.org/html/2608.05156#S5.SS3)\) progressively removeVVandIIto verify the contribution of skill’s core structural features \(EEremoval causes only 0\.8pp drop, not significant\)\. Full ablation configurations and skill examples are provided in Appendix[D](https://arxiv.org/html/2608.05156#A4)\.

Stage 1: Skill DiscoverySeed Data\(ti,ri\)\(t\_\{i\},r\_\{i\}\)LLMExtractLLMClusterLLMRefineValidationEvalScaffold GraphG0G\_\{0\}score<θ\\text\{score\}<\\theta: iterative refineStage 2: Scaffold\-Graph\-Augmented Data GenerationScaffold GraphGGSelect\(CC\-match\)BuildSubgraphGjG\_\{j\}TopologicalExecutionVV\-Verify\+ Test𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}Verified Tracespassfail: retry or discardStage 3: Bottom\-up Progressive Distillation \+ Dynamic Recompilation𝒟k\\mathcal\{D\}\_\{k\}\+ Pretrain DataTeacherMM\+skillStudentM′M^\{\\prime\}onlyRetention≥τ\\geq\\tau?RecompileGt\+1G\_\{t\+1\}ℒ\+β​DKL\\mathcal\{L\}\+\\beta D\_\{\\text\{KL\}\}yesno: keep externalupdateGG, return to Stage 2Figure 2:Detailed three\-stage data flow\.Stage 1: iteratively discover skills from seed data and build the scaffold graphG0G\_\{0\}\.Stage 2: for each task, select nodes byCC\-matching, build subgraphGjG\_\{j\}, execute in topological order, andVV\-verify; only passing traces are retained\.Stage 3: bottom\-up progressive distillation \(loss includes KL constraint, mixed with pretrain data\); skills whose retention rate meets the threshold are removed from the graph and trigger dependency recompilation\. Blue arrows denote cross\-stage data flow; red dashed lines denote feedback/fallback paths\.
### 3\.2Stage 1: skill discovery

Input: A high\-quality seed dataset𝒟seed=\{\(ti,ri\)\}i=1N\\mathcal\{D\}\_\{\\text\{seed\}\}=\\\{\(t\_\{i\},r\_\{i\}\)\\\}\_\{i=1\}^\{N\}, wheretit\_\{i\}is a task description andrir\_\{i\}is a successful solution\.NNis typically 1000–3000\.

Algorithm[1](https://arxiv.org/html/2608.05156#alg1)\(Appendix[A](https://arxiv.org/html/2608.05156#A1)\) provides the complete procedure\. HereLLM\.Extract,LLM\.Cluster, andLLM\.Refineare all LLM calls \(prompt templates are given in Appendix[B](https://arxiv.org/html/2608.05156#A2)\)\.Evaluatecompares the passed rate on a validation set before and after skill injection\. The thresholdθ\\theta\(default 0\.05, i\.e\., a skill must yield≥\\geq5 pp improvement on the tasks that invoke it\) controls the minimum effectiveness requirement;ϵ\\epsilon\(default 0\.01\) controls the iteration convergence criterion\. Sensitivity analysis is provided in Appendix[C](https://arxiv.org/html/2608.05156#A3)\.

Relationship to human\-designed skills\. The skills produced by automatic discovery are structurally highly similar to manually designed ones \(Appendix[D](https://arxiv.org/html/2608.05156#A4)provides a component\-by\-component comparison\), yet the process is fully automated, demonstrating that LLMs can induce structured strategies from successful exemplars\.

### 3\.3Stage 2: scaffold\-graph\-augmented data generation

Objective: Leverage the scaffold graphG0G\_\{0\}to guide the model in solving additional tasks and generate high\-quality training data\.𝒯new\\mathcal\{T\}\_\{\\text\{new\}\}comprises 2000 task instances generated by the FeatureBench data pipeline on additional repositories; success is determined by execution\-based verification \(running the test suite\)\. Algorithm[2](https://arxiv.org/html/2608.05156#alg2)\(Appendix[A](https://arxiv.org/html/2608.05156#A1)\) provides the procedure\.

Subgraph construction and execution\.The LLM selects candidate nodes𝒮j\\mathcal\{S\}\_\{j\}byCC\-matching, extracts subgraphGjG\_\{j\}fromG0G\_\{0\}in topological order \(depth limited to 3 layers\)\. During execution, nodes are dispatched sequentially with data passed viaII; each node runsVV\-verification after execution \(one retry permitted; second failure discards the trajectory\)\. A final task\-level test suite is run; only fully verified trajectories are retained as training data\. Formal definitions are given in Appendix[J](https://arxiv.org/html/2608.05156#A10)\.

### 3\.4Stage 3: progressive distillation and dynamic recompilation

Objective: Internalize skill behaviors into model parameters and dynamically recompile the scaffold graph after internalization\. Algorithm[3](https://arxiv.org/html/2608.05156#alg3)\(Appendix[A](https://arxiv.org/html/2608.05156#A1)\) provides the complete procedure\.

Distillation objective\. We adopt a self\-distillation approach in which the teacher and student are different configurations of the same model: the teacher receives\[skill prompt⊕tj\]\[\\text\{skill prompt\}\\oplus t\_\{j\}\]and producesrjr\_\{j\}\(the full execution trajectory including intermediate reasoning and action steps\); the student receives only\[tj\]\[t\_\{j\}\]and is trained to matchrjr\_\{j\}\. The training loss isℒ\+β⋅DKL​\(πθ∥πref\)\\mathcal\{L\}\+\\beta\\cdot D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\), whereℒ\\mathcal\{L\}is the cross\-entropy loss and the KL divergence constraint prevents the parameters from deviating too far from the reference model\.

Distillation order\. Progressive distillation proceeds bottom\-up along the topological order of the scaffold graphGG: root nodes with no dependencies \(in\-degree 0\) are distilled first; within the same level, nodes are sorted in descending order of dependency count \(out\-degree\), prioritizing internalization of foundational capabilities with the largest impact\. This ordering is uniquely determined by the graph structure and requires no manual weight tuning\.

Implementation details\. We use full\-parameter fine\-tuning \(Appendix[K](https://arxiv.org/html/2608.05156#A11)\)\. The retention rate threshold isτ=0\.85\\tau=0\.85, and the recompilation fallback threshold isδ=0\.02\\delta=0\.02\. The reference policyπref\\pi\_\{\\text\{ref\}\}in the KL constraint is statically anchored to the original base modelM0M\_\{0\}and is not updated across distillation rounds\. Algorithm[3](https://arxiv.org/html/2608.05156#alg3)iterates per skill for clarity; in implementation, skills within the same topological level are merged into a single training round to avoid redundant gradient steps\.

Catastrophic forgetting mitigation\.Five mechanisms jointly mitigate catastrophic forgetting: \(1\)self\-distillation—teacher and student share the same base model, inherently small KL divergence reduces distribution shift; \(2\)bottom\-up implicit replay—lower\-level skill behaviors are repeatedly replayed in higher\-level training data; \(3\)pretraining data mixing—50% pretrain data mixed per round; \(4\)KL constraint\(β=0\.01\\beta=0\.01\); \(5\)per\-round retention check—skills falling belowτ\\tauare not internalized and are retained as external scaffolds\. Detailed analysis is provided in Appendix[E](https://arxiv.org/html/2608.05156#A5)\.

### 3\.5Dynamic recompilation

Dynamic recompilation enables co\-evolution of parameters and the scaffold graph\. After skillsks\_\{k\}is internalized, its node cannot simply be removed—higher\-level skills may depend on it\. Recompilation consists of six steps: \(1\)Node Removal: removesks\_\{k\}fromGG; \(2\)Procedure Rewriting: an LLM replaces explicit calls tosks\_\{k\}in the higher\-level scaffoldshs\_\{h\}’s procedurePhP\_\{h\}with implicit assumptions about the model’s internalized capability; \(3\)Bridge Regeneration: regenerate inter\-node data\-passing instructions; \(4\)Interface Update: updateIhI\_\{h\}; \(5\)Re\-verification: recompiled scaffolds must passVVverification on a validation set \(60s timeout per node\); \(6\)Fallback: if performance drops by\>δ\>\\delta, retainsks\_\{k\}as an external scaffold\. Recompilation depth is limited to 1 hop; each node is processed independently\.

Formally:Mt\+1=Distill​\(Mt,𝒟​\(st\)\)M\_\{t\+1\}=\\text\{Distill\}\(M\_\{t\},\\mathcal\{D\}\(s\_\{t\}\)\);Gt\+1=Recompile​\(Gt,st,Mt\+1\)G\_\{t\+1\}=\\text\{Recompile\}\(G\_\{t\},s\_\{t\},M\_\{t\+1\}\)\.

## 4Experimental setup

Our experiments aim to answer five research questions:RQ1: Can a scaffold graph be automatically learned from seed trajectories?RQ2: Can the scaffold graph improve task solving and data generation?RQ3: Can scaffold behaviors be internalized into model parameters?RQ4: Is the graph structure \(VVandII\) necessary?RQ5: Is dynamic recompilation effective?

### 4\.1Benchmarks and models

Data and Evaluation\.We use the open\-source data pipeline from FeatureBench\(Zhou et al\.,[2026](https://arxiv.org/html/2608.05156#bib.bib19)\)to generate training data\. Evaluation strictly follows the FeatureBench paper setup: we use its standard Full set \(200 tasks, 24 open\-source repositories, execution\-based verification\), with OpenHands\(Wang et al\.,[2025](https://arxiv.org/html/2608.05156#bib.bib9)\)as the base agent framework \(providing tool interaction and environment management\), and skills injected as an additional procedural knowledge layer on top\. We report Passed Rate \(the average proportion of fail\-to\-pass tests passed per task\)\. Training data is generated using the data pipeline on additional repositories with no overlap with the evaluation set\.

Models\.Qwen3\-Coder\-480B\-A35B\-Instruct\(Yang et al\.,[2025a](https://arxiv.org/html/2608.05156#bib.bib12)\)\(MoE, 35B activated parameters, full\-parameter fine\-tuning\) andDeepSeek\-V3\.2\(DeepSeek\-AI,[2025](https://arxiv.org/html/2608.05156#bib.bib2)\)\(MoE, 37B activated parameters, to verify cross\-model generality\)\.

### 4\.2Experimental configurations

Table 1:Experimental configurations\. Direct Fine\-tune trains on the same𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}via standard SFT in a single pass, isolating the contribution of structured training pathways\.
### 4\.3Seed data construction

We use the FeatureBench data pipeline to generate training instances on additional repositories, selecting 2000 seed tasks and 200 validation tasks \(for skill quality evaluation, independent of the FeatureBench evaluation set\)\. Selection criteria: involving multi\-file modifications, containing strategic steps, and covering diverse development patterns\. An additional 2000 tasks are sampled as𝒯new\\mathcal\{T\}\_\{\\text\{new\}\}for the data generation phase\. Seed data details are provided in Appendix[G](https://arxiv.org/html/2608.05156#A7)\.

### 4\.4Evaluation metrics

- •Passed Rate: the average proportion of fail\-to\-pass tests passed per task\. We adopt Passed Rate as the primary metric because skills provide incremental help on procedural strategies \(e\.g\., correctly executing 3 out of 5 steps\), and Passed Rate captures such partial improvements\. The strict Resolved Rate \(all tests must pass\) is reported in Appendix[L](https://arxiv.org/html/2608.05156#A12); trends are consistent\.
- •Distillation retention rate: Passed Rate of the distilled model without skill / Passed Rate of the pre\-distillation base model with skill \(i\.e\., the Auto Skills passed rate of 32\.5%\), measuring how much of the skill\-augmented capability is retained after distillation \(evaluation tasks are highly relevant to the skills\)\.
- •Skill transfer rate: the proportion of skills that yield a positive improvement on tasks outside the seed set\.

Following the FeatureBench evaluation protocol, all 200 evaluation tasks are used for assessment \(model training data comes entirely from independent seed and generated sets with no overlap with these 200 evaluation tasks\)\. Each configuration is run with 5 random seeds; we report mean±\\pmstandard deviation\.

## 5Experimental results

### 5\.1Main results \(RQ1–RQ3\)

Table[2](https://arxiv.org/html/2608.05156#S5.T2)presents the passed rate of each configuration on FeatureBench\.

Table 2:Passed rate \(%\) on FeatureBench Full \(200 tasks, OpenHands framework, 5 random seeds\)\.Key observations\.

\(1\)Effectiveness of auto\-discovery: Auto Skills \(32\.5%\) approach Human Skills \(35\.0%\) with a gap of only 2\.5pp, indicating that the model can automatically discover skills from 2,000 seed cases at near\-human quality \(achieving 93% of the human\-designed passed rate\)\.

\(2\)Value of scaffold\-augmented inference: Auto Skills \(32\.5%\) significantly outperform Baseline \(24\.4%\) by 8\.1pp, demonstrating that skill\-augmented inference yields a substantial absolute improvement\.

\(3\)Value of distillation internalization: Progressive Distilled \(31\.5%, with 8 external skills\) significantly outperforms Direct Fine\-tune \(25\.5%\) by 6\.0pp\. Even after completely removing external skills, the purely internalized model still reaches 27\.7% \(retention rate 85\.2%\), outperforming Direct Fine\-tune by 2\.2pp\. Note that both usethe same𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}\(allVV\-verified\); the gap arises purely from structured training pathways: per\-skill progressive distillation vs\. one\-pass standard SFT\.

\(4\) Trends are consistent across both models\.

### 5\.2Skill quality analysis

The automatic discovery process extracted approximately 120 candidate strategies from 2,000 seed cases, which were clustered and merged into 45 skill candidates; 30 of these passed validation\-set filtering \(θ=0\.05\\theta=0\.05\), converging in 5 iterations\.

Auto Skills cover 78% of test tasks \(Human Skills cover 85%\), with a false\-trigger rate of 12% \(Human Skills: 8%\)\. On out\-of\-seed tasks, 27 out of 30 skills yield a positive improvement \(skill transfer rate 90%\), indicating good cross\-task transferability\. A fine\-grained analysis stratified by task complexity is provided in Appendix[F](https://arxiv.org/html/2608.05156#A6)\.

### 5\.3Graph structure ablation \(RQ4\)

Table 3:Skill component ablation \(Qwen3\-Coder\)\. Pure internal\. rate = infer\. passed rate×\\timesretention rate\. All rows use the progressive distillation pipeline; Flat Prompt’s low retention reflects pipeline incompatibility with unstructured prompts\. OPRO\+Distill in Table[2](https://arxiv.org/html/2608.05156#S5.T2)uses standard context distillation\.VV’s dual role: removingVVdrops inference\-time passed rate by 2\.5pp and retention rate by 6\.9pp;VVprovides execution feedback during inference and data filtering during distillation\.PP’s indirect value: removing explicit calls inPPdrops passed rate by 1\.5pp; the dependency graph induced byPPenables skill composition and makes distillation targets more modular \(−P\-Pleaves no dependency edges; the model autonomously decides which skill to invoke, and all skills are merged into a single round of joint distillation\)\. Full degradation to conditional prompt \(−V−P\-V\-P\) differs from Flat Prompt by only 1\.5pp in inference\-time passed rate \(28\.5% vs 27\.0%\), confirming thatVVandPPare the core distinguishing features\.

### 5\.4Dynamic recompilation analysis \(RQ5\)

Table[4](https://arxiv.org/html/2608.05156#S5.T4)presents a scaffold graph evolution summary during progressive distillation\.

Table 4:Scaffold graph evolution summary\. Of 30 skills, 22 were successfully internalized, triggering 50 dependency recompilations \(96% successful\); the passed rate gradually decreased from 32\.5% to 31\.5% \(−\-1\.0pp\)\.Key observations\. \(1\)Recompilation success rate: After internalizing 22 skills, a total of 50 upstream scaffold recompilations were triggered, of which 48 succeeded and 2 fell back \(success rate 96%\), indicating that theIIinterface provides sufficient dependency information to support automatic rewriting\. \(2\)Controllable gradual performance degradation: As external scaffolds decrease from 30 to 8, the passed rate gradually drops from 32\.5% to 31\.5% \(−\-1\.0pp\), far less than the drop from directly removing all scaffolds \(−\-4\.8pp\)\. \(3\)Graph structure evolution: The initial graphG0G\_\{0\}has 30 nodes and 52 dependency edges; after distillation,GTG\_\{T\}has 8 nodes and 6 edges, representing significant structural simplification\. 8 skills are retained as external scaffolds: 6 due to retention rates belowτ\\tau, and 2 due to recompilation performance drop exceedingδ\\deltadespite meeting the retention threshold\.

### 5\.5Case studies

Representative cases \(detailed in Appendix[H](https://arxiv.org/html/2608.05156#A8)\): in a success case, a 5\-file modification task is completed via skill\-guided dependency analysis and stepwise verification, with the model retaining this behavior after distillation; a failure case reveals that skills primarily improve procedural capabilities but offer limited help for creative decision\-making\.

## 6Discussion

Capability transfer\.Scaffold\-mediated post\-training enables bidirectional capability transfer between two storage media: distillation transfers scaffold capabilities to parameters \(G0G\_\{0\}’s 30 nodes→\\toGTG\_\{T\}’s 8 nodes\), while dynamic recompilation feeds parameter state back into the graph structure\. Direct Fine\-tune \(25\.5%, same data, one\-pass SFT\) lacks structured training pathways; hybrid deployment \(i\.e\., Progressive Distilled with 22 internalized \+ 8 external skills, 31\.5%\) outperforms by 6\.0pp\. At inference, hybrid deployment provides\[8 external skill prompts⊕tj\]\[\\text\{8 external skill prompts\}\\oplus t\_\{j\}\]to the model; the instruction\-following capability for conditioning on these prompts is already present in the base model\.

Self\-distillation and confirmation bias\.Self\-distillation risks reinforcing the model’s own biases; three factors mitigate this: \(1\)VVis execution\-based verification \(build exit codes, test pass rates\), not model self\-assessment—removingVVdrops retention by 6\.9pp; \(2\) consistent trends across two different MoE models \(Qwen3\-Coder/DeepSeek\-V3\.2\); \(3\) comparison with Direct Fine\-tune \(27\.7% vs 25\.5% on Qwen3\-Coder, same data\) indicates that the structured training pathway provides information beyond the data alone\.

## 7Limitations

Domain and scale\.The current work is validated only in the code generation domain\. The framework is domain\-agnostic in principle—VVandIIcan be instantiated with domain\-specific verifiers and interfaces—but empirical validation in other domains remains future work\.

Teacher diversity\.Same\-model self\-distillation is used throughout; stronger or differently architected teachers may further improve quality and mitigate confirmation bias\.

Pretraining contamination\.The 24 evaluation repositories are likely in the base models’ pretraining corpora, but this affects all configurations equally; our conclusions rest on relative differences\.

## 8Broader impact

The performance bottleneck of current AI coding agents on complex tasks largely stems from the scarcity of procedural strategy data\. The scaffold\-mediated post\-training paradigm proposed here, by treating procedural scaffolds as evolvable objects and distilling them into model parameters, offers a new path to overcome this bottleneck\. Developers can train structured skills from high\-quality engineering practices \(such as TDD and code review workflows\), then use distillation to endow the model with these capabilities without external scaffolds\. A potential risk is that distillation may internalize unsafe coding patterns from the training data; deployment should be coupled with security audits\.

## 9Conclusion

We propose scaffold\-mediated post\-training, where parameters and scaffold graphs co\-evolve through multiple rounds of the discover→\\rightarrowinternalize→\\rightarrowrecompile loop—multi\-round discovery builds higher\-level scaffolds, internalization simplifies the graph structure \(Table[4](https://arxiv.org/html/2608.05156#S5.T4): 30→\\to8\), enabling the model to tackle increasingly complex tasks\. Experiments confirm that this mechanism consistently outperforms standard fine\-tuning on equivalent data\. Code will be released upon acceptance\.

## References

- Chen et al\. \(2023\)Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue WANG, Ce Zhang, Frederic Sala, and Christopher Ré\.Skill\-it\! a data\-driven skills framework for understanding and training language models\.In A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine, editors,*Advances in Neural Information Processing Systems*, volume 36, pages 36000–36040\. Curran Associates, Inc\., 2023\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/70b8505ac79e3e131756f793cd80eb8d\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/70b8505ac79e3e131756f793cd80eb8d-Paper-Conference.pdf)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AI\.Deepseek\-v3\.2: Pushing the frontier of open large language models, 2025\.
- Fernando et al\. \(2024\)Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel\.Promptbreeder: Self\-referential self\-improvement via prompt evolution\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 13481–13544\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/fernando24a\.html](https://proceedings.mlr.press/v235/fernando24a.html)\.
- Jimenez et al\. \(2024\)Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan\.SWE\-bench: Can language models resolve real\-world github issues?In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66)\.
- Khattab et al\. \(2024\)Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts\.Dspy: Compiling declarative language model calls into state\-of\-the\-art pipelines\.In B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun, editors,*International Conference on Learning Representations*, volume 2024, pages 54928–54958, 2024\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2024/file/f1cf02ce09757f57c3b93c0db83181e0\-Paper\-Conference\.pdf](https://proceedings.iclr.cc/paper_files/paper/2024/file/f1cf02ce09757f57c3b93c0db83181e0-Paper-Conference.pdf)\.
- Snell et al\. \(2022\)Charlie Snell, Dan Klein, and Ruiqi Zhong\.Learning by distilling context, 2022\.URL[https://arxiv\.org/abs/2209\.15189](https://arxiv.org/abs/2209.15189)\.
- Wang et al\. \(2024\)Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar\.Voyager: An open\-ended embodied agent with large language models\.*Transactions on Machine Learning Research*, 2024\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=ehfRiF0R3a](https://openreview.net/forum?id=ehfRiF0R3a)\.
- Wang et al\. \(2026\)Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong\.Reinforcement learning for self\-improving agent with skill library, 2026\.URL[https://arxiv\.org/abs/2512\.17102](https://arxiv.org/abs/2512.17102)\.
- Wang et al\. \(2025\)Xingyao Wang, Boxuan Li, Yufan Song, Frank F\. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H\. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig\.Openhands: An open platform for AI software developers as generalist agents\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=OJd3ayDDoF](https://openreview.net/forum?id=OJd3ayDDoF)\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou\.Chain\-of\-thought prompting elicits reasoning in large language models\.In S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh, editors,*Advances in Neural Information Processing Systems*, volume 35, pages 24824–24837\. Curran Associates, Inc\., 2022\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf)\.
- Xia et al\. \(2026\)Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, Zeyu Zheng, Cihang Xie, and Huaxiu Yao\.Skillrl: Evolving agents via recursive skill\-augmented reinforcement learning, 2026\.URL[https://arxiv\.org/abs/2602\.08234](https://arxiv.org/abs/2602.08234)\.
- Yang et al\. \(2025a\)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu\.Qwen3 technical report, 2025a\.URL[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\.
- Yang et al\. \(2024\)Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen\.Large language models as optimizers\.In B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun, editors,*International Conference on Learning Representations*, volume 2024, pages 12028–12068, 2024\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2024/file/3339f19c5fcee3ad74502947a32be9e6\-Paper\-Conference\.pdf](https://proceedings.iclr.cc/paper_files/paper/2024/file/3339f19c5fcee3ad74502947a32be9e6-Paper-Conference.pdf)\.
- Yang et al\. \(2025b\)John Yang, Kilian Lieret, Carlos E\. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang\.Swe\-smith: Scaling data for software engineering agents, 2025b\.URL[https://arxiv\.org/abs/2504\.21798](https://arxiv.org/abs/2504.21798)\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao\.React: Synergizing reasoning and acting in language models\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=WE\_vluYUL\-X](https://openreview.net/forum?id=WE_vluYUL-X)\.
- Zelikman et al\. \(2022\)Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman\.Star: Bootstrapping reasoning with reasoning\.In S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh, editors,*Advances in Neural Information Processing Systems*, volume 35, pages 15476–15488\. Curran Associates, Inc\., 2022\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/file/639a9a172c044fbb64175b5fad42e9a5\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2022/file/639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf)\.
- Zelikman et al\. \(2024\)Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Goodman\.Quiet\-STar: Language models can teach themselves to think before speaking\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=oRXPiSOGH9](https://openreview.net/forum?id=oRXPiSOGH9)\.
- Zhao et al\. \(2024\)Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong\-Jin Liu, and Gao Huang\.Expel: Llm agents are experiential learners\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 38\(17\):19632–19642, Mar\. 2024\.doi:10\.1609/aaai\.v38i17\.29936\.URL[https://ojs\.aaai\.org/index\.php/AAAI/article/view/29936](https://ojs.aaai.org/index.php/AAAI/article/view/29936)\.
- Zhou et al\. \(2026\)Qixing Zhou, JiaCheng Zhang, Haiyang Wang, Rui Hao, Jiahe Wang, Minghao Han, Yuxue Yang, Shuzhe Wu, Feiyang Pan, Lue Fan, Dandan Tu, and Zhaoxiang Zhang\.Featurebench: Benchmarking agentic coding for complex feature development\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=41xrZ3uGuI](https://openreview.net/forum?id=41xrZ3uGuI)\.
- Zhou et al\. \(2023\)Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba\.Large language models are human\-level prompt engineers\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=92gvk82DE\-](https://openreview.net/forum?id=92gvk82DE-)\.

## Appendix AAlgorithm Pseudocode

Algorithm 1Skill Discovery0:Seed dataset

𝒟seed\\mathcal\{D\}\_\{\\text\{seed\}\}, validation set

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}, maximum iteration rounds

TT, effectiveness threshold

θ\\theta, convergence threshold

ϵ\\epsilon
0:Base scaffold graph

G0=\(𝒮base,D\)G\_\{0\}=\(\\mathcal\{S\}\_\{\\text\{base\}\},D\)
1:

ℛ←∅\\mathcal\{R\}\\leftarrow\\emptyset\{Strategy candidate set\}

2:foreach

\(ti,ri\)∈𝒟seed\(t\_\{i\},r\_\{i\}\)\\in\\mathcal\{D\}\_\{\\text\{seed\}\}do

3:

ℛi←LLM\.Extract​\(ti,ri,“extract reusable strategies”\)\\mathcal\{R\}\_\{i\}\\leftarrow\\text\{LLM\.Extract\}\(t\_\{i\},r\_\{i\},\\text\{\`\`extract reusable strategies''\}\)
4:

ℛ←ℛ∪ℛi\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\cup\\mathcal\{R\}\_\{i\}
5:endfor

6:

𝒮cand←LLM\.Cluster​\(ℛ,“merge similar strategies into skill candidates”\)\\mathcal\{S\}\_\{\\text\{cand\}\}\\leftarrow\\text\{LLM\.Cluster\}\(\\mathcal\{R\},\\text\{\`\`merge similar strategies into skill candidates''\}\)
7:foreach

sk∈𝒮cands\_\{k\}\\in\\mathcal\{S\}\_\{\\text\{cand\}\}do

8:Format

sks\_\{k\}into complete skill structure

\(metak,Ck,Pk,Vk,Ik,Ek\)\(\\textit\{meta\}\_\{k\},C\_\{k\},P\_\{k\},V\_\{k\},I\_\{k\},E\_\{k\}\)
9:endfor

10:foreach

sk∈𝒮cands\_\{k\}\\in\\mathcal\{S\}\_\{\\text\{cand\}\}do

11:

scorek←Evaluate​\(sk,𝒟val\)\\text\{score\}\_\{k\}\\leftarrow\\text\{Evaluate\}\(s\_\{k\},\\mathcal\{D\}\_\{\\text\{val\}\}\)\{Initial evaluation\}

12:endfor

13:for

iter=1\\text\{iter\}=1to

TTdo

14:

prev\_scores​\[k\]←scorek,∀k\\text\{prev\\\_scores\}\[k\]\\leftarrow\\text\{score\}\_\{k\},\\;\\forall k\{Cache previous scores\}

15:foreach

sk∈𝒮cands\_\{k\}\\in\\mathcal\{S\}\_\{\\text\{cand\}\}do

16:if

scorek<θ\\text\{score\}\_\{k\}<\\thetathen

17:

sk←LLM\.Refine​\(sk,failure case analysis\)s\_\{k\}\\leftarrow\\text\{LLM\.Refine\}\(s\_\{k\},\\text\{failure case analysis\}\)
18:

scorek←Evaluate​\(sk,𝒟val\)\\text\{score\}\_\{k\}\\leftarrow\\text\{Evaluate\}\(s\_\{k\},\\mathcal\{D\}\_\{\\text\{val\}\}\)\{Re\-evaluate after refinement\}

19:endif

20:endfor

21:if

maxk⁡\|scorek−prev\_scores​\[k\]\|<ϵ\\max\_\{k\}\|\\text\{score\}\_\{k\}\-\\text\{prev\\\_scores\}\[k\]\|<\\epsilonthen

22:break

23:endif

24:endfor

25:

𝒮base←\{sk∣scorek≥θ\}\\mathcal\{S\}\_\{\\text\{base\}\}\\leftarrow\\\{s\_\{k\}\\mid\\text\{score\}\_\{k\}\\geq\\theta\\\}
26:

D←\{\(sa,sb\)∣Ia\.out∩Ib\.in≠∅\}D\\leftarrow\\\{\(s\_\{a\},s\_\{b\}\)\\mid I\_\{a\}\.\\text\{out\}\\cap I\_\{b\}\.\\text\{in\}\\neq\\emptyset\\\}
27:return

G0=\(𝒮base,D\)G\_\{0\}=\(\\mathcal\{S\}\_\{\\text\{base\}\},D\)

Algorithm 2Scaffold\-Graph\-Augmented Data Generation0:Base scaffold graph

G0=\(𝒮base,D\)G\_\{0\}=\(\\mathcal\{S\}\_\{\\text\{base\}\},D\), new task set

𝒯new\\mathcal\{T\}\_\{\\text\{new\}\}
0:Augmented dataset

𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}
1:

𝒟aug←∅\\mathcal\{D\}\_\{\\text\{aug\}\}\\leftarrow\\emptyset
2:foreach

tj∈𝒯newt\_\{j\}\\in\\mathcal\{T\}\_\{\\text\{new\}\}do

3:

𝒮j←Select​\(G0,tj\)\\mathcal\{S\}\_\{j\}\\leftarrow\\text\{Select\}\(G\_\{0\},t\_\{j\}\)\{Match relevant scaffold nodes by

CC\}

4:

Gj←BuildSubgraph​\(𝒮j,D,depth≤3\)G\_\{j\}\\leftarrow\\text\{BuildSubgraph\}\(\\mathcal\{S\}\_\{j\},D,\\text\{depth\}\\leq 3\)\{Expand downstream dependencies, depth

≤\\leq3\}

5:Inject skills from

GjG\_\{j\}in topological order into LLM context; LLM executes autonomously, passing data via

II; 120s timeout per execution step

6:

rj,tracej←LLM\.Solve​\(tj,Gj\)r\_\{j\},\\text\{trace\}\_\{j\}\\leftarrow\\text\{LLM\.Solve\}\(t\_\{j\},G\_\{j\}\)
7:ifAny node

VVverification failsthen

8:Retry the failed node once; discard this trace on second failure andcontinue

9:endif

10:

VGj←⋃s∈Nodes​\(Gj\)VsV\_\{G\_\{j\}\}\\leftarrow\\bigcup\_\{s\\in\\text\{Nodes\}\(G\_\{j\}\)\}V\_\{s\}\{Union of subgraph verification checkpoints\}

11:if

Verify​\(rj,VGj\)\\text\{Verify\}\(r\_\{j\},V\_\{G\_\{j\}\}\)task tests passthen

12:

𝒟aug←𝒟aug∪\{\(tj,tracej,rj\)\}\\mathcal\{D\}\_\{\\text\{aug\}\}\\leftarrow\\mathcal\{D\}\_\{\\text\{aug\}\}\\cup\\\{\(t\_\{j\},\\text\{trace\}\_\{j\},r\_\{j\}\)\\\}
13:endif

14:endfor

15:return

𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}

Algorithm 3Progressive Distillation with Scaffold Recompilation0:Augmented dataset

𝒟aug\\mathcal\{D\}\_\{\\text\{aug\}\}, pretraining data

𝒟pretrain\\mathcal\{D\}\_\{\\text\{pretrain\}\}, validation set

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}, scaffold graph

G0G\_\{0\}, model

MM, reference policy

πref\\pi\_\{\\text\{ref\}\}, retention threshold

τ\\tau, fallback threshold

δ\\delta
0:Distilled model

M′M^\{\\prime\}, recompiled scaffold graph

GTG\_\{T\}
1:

M′←copy​\(M\)M^\{\\prime\}\\leftarrow\\text\{copy\}\(M\);

G←G0G\\leftarrow G\_\{0\}
2:Sort nodes in

GGby topological order \(ties broken by descending out\-degree\) \{Bottom\-up priority\}

3:foreach skill

sks\_\{k\}in sorted orderdo

4:

𝒟k←\{\(tj,tracej,rj\)∈𝒟aug∣sk∈tracej\}\\mathcal\{D\}\_\{k\}\\leftarrow\\\{\(t\_\{j\},\\text\{trace\}\_\{j\},r\_\{j\}\)\\in\\mathcal\{D\}\_\{\\text\{aug\}\}\\mid s\_\{k\}\\in\\text\{trace\}\_\{j\}\\\}
5:

𝒟k←𝒟k∪Sample​\(𝒟pretrain,\|𝒟k\|\)\\mathcal\{D\}\_\{k\}\\leftarrow\\mathcal\{D\}\_\{k\}\\cup\\text\{Sample\}\(\\mathcal\{D\}\_\{\\text\{pretrain\}\},\|\\mathcal\{D\}\_\{k\}\|\)\{Mix in 50% pretraining data\}

6:

Gprev←copy​\(G\)G\_\{\\text\{prev\}\}\\leftarrow\\text\{copy\}\(G\)\{Save graph snapshot\}

7:Fine\-tune

M′M^\{\\prime\}on

𝒟k\\mathcal\{D\}\_\{k\}, minimizing

ℒ\+β⋅DKL​\(πθ∥πref\)\\mathcal\{L\}\+\\beta\\cdot D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)
8:

depsk←\{sh∈Nodes​\(G\)∣\(sk,sh\)∈D\}\\text\{deps\}\_\{k\}\\leftarrow\\\{s\_\{h\}\\in\\text\{Nodes\}\(G\)\\mid\(s\_\{k\},s\_\{h\}\)\\in D\\\}\{Record dependents before removal\}

9:Remove

sks\_\{k\}from

GG\{Node removal\}

10:foreach

sh∈depsks\_\{h\}\\in\\text\{deps\}\_\{k\}do

11:Rewrite

PhP\_\{h\}, regenerate bridge instructions, update

IhI\_\{h\}
12:endfor

13:ifrecompiled

GGfails

VVverification on

𝒟val\\mathcal\{D\}\_\{\\text\{val\}\}\(60s timeout per node\)then

14:

G←GprevG\\leftarrow G\_\{\\text\{prev\}\};continue\{Re\-verification failed; roll back\}

15:endif

16:

perf\_before←Evaluate​\(M′,Gprev,𝒟val\)\\text\{perf\\\_before\}\\leftarrow\\text\{Evaluate\}\(M^\{\\prime\},G\_\{\\text\{prev\}\},\\mathcal\{D\}\_\{\\text\{val\}\}\)\{

M′M^\{\\prime\}with old graph\}

17:

perf\_after←Evaluate​\(M′,G,𝒟val\)\\text\{perf\\\_after\}\\leftarrow\\text\{Evaluate\}\(M^\{\\prime\},G,\\mathcal\{D\}\_\{\\text\{val\}\}\)\{

M′M^\{\\prime\}with new graph\}

18:

drop←perf\_before−perf\_after\\text\{drop\}\\leftarrow\\text\{perf\\\_before\}\-\\text\{perf\\\_after\}
19:

retaink←Evaluate​\(M′,𝒟val,without​sk\)/Evaluate​\(M,𝒟val,with​sk\)\\text\{retain\}\_\{k\}\\leftarrow\\text\{Evaluate\}\(M^\{\\prime\},\\mathcal\{D\}\_\{\\text\{val\}\},\\text\{without \}s\_\{k\}\)/\\text\{Evaluate\}\(M,\\mathcal\{D\}\_\{\\text\{val\}\},\\text\{with \}s\_\{k\}\)
20:if

retaink≥τ\\text\{retain\}\_\{k\}\\geq\\taudrop≤δ\\text\{drop\}\\leq\\deltathen

21:Mark

sks\_\{k\}as “internalized”

22:else

23:

G←GprevG\\leftarrow G\_\{\\text\{prev\}\}\{Roll back graph state; weight updates retained\}

24:endif

25:endfor

26:return

M′M^\{\\prime\},

GG

## Appendix BSkill Discovery Prompt Templates

Below are the prompt templates used in each stage of Skill Discovery\.

### B\.1LLM\.Extract Prompt

> You are a strategy extraction expert\. Given a coding task and its successful solution, extract REUSABLE problem\-solving strategies as structured skills\. TASK: \{task\_description\} SUCCESSFUL SOLUTION: \{solution\} For each strategy, output a structured skill with:\- name: concise name describing the core idea\- C \(condition\): what task characteristics trigger this skill\- P \(procedure\): 3\-\-5 step execution workflow\- V \(verification\): checkpoints to confirm successful execution \(must be evidence\-based, not self\-reported\)\- I \(interface\): required inputs and produced outputs\- E \(anti\-patterns\): common mistakes this skill helps avoid Rules: strategies must be GENERAL \(not specific to this task\), ACTIONABLE \(each step is executable\), and VERIFIABLE \(V must reference observable evidence like test output or build status\)\.

### B\.2LLM\.Cluster Prompt

> Below is a list of candidate strategies extracted from multiple tasks\. Merge semantically similar or functionally equivalent strategies into unified skill candidates\. CANDIDATE LIST: \{strategy\_list\} For each merged skill, preserve the most general C \(condition\), union the P \(procedure\) steps, and ensure V \(verification\) covers all merged strategies\. Output the merged skill in the same six\-component format\. If two strategies have compatible I \(interface\), note this as a potential composition edge\.

### B\.3LLM\.Refine Prompt

The following template is used for structured skill refinement\.

> The following skill performed poorly on validation tasks\. Analyze the failure cases and refine the skill\. CURRENT SKILL: \{skill\_definition\} FAILURE CASES: \{failure\_cases\} For each failure, diagnose the root cause:\(a\) C too broad \-\-\- triggered on tasks where it should not apply\(b\) P incomplete \-\-\- missing critical execution steps\(c\) V insufficient \-\-\- failed to detect execution errors\(d\) I ambiguous \-\-\- insufficient data passed during composition Available operations on each component:KEEP: component is correct as\-isEDIT: rewrite component with improvementADD: add new sub\-steps or checkpoints Output the refined skill in full\(m​e​t​a,C,P,V,I,E\)\(meta,C,P,V,I,E\)format\.

### B\.4LLM\.Solve Prompt \(Stage 2\)

The following template is used for scaffold\-graph\-augmented task solving\.

> You are a coding agent solving a software engineering task with the help of structured skills\. TASK: \{task\_description\} ACTIVE SKILLS \(in topological execution order\):\{skill\_1: C, P, V, I, E\}\{skill\_2: C, P, V, I, E\}\.\.\. For each active skill, follow its P \(procedure\) step by step\. After completing each skill, run its V \(verification\) checkpoints and report evidence \(e\.g\., build exit code, test output\)\. Pass data between skills according to their I \(interface\) specifications\. If a V checkpoint fails, retry once with a corrected approach before moving to the next skill\. Output your solution as a code patch\.

### B\.5Procedure Rewriting Prompt \(Stage 3\)

The following template is used for procedure rewriting during dynamic recompilation\.

> Skill \{skill\_k\} has been internalized into model parameters and will be removed from the scaffold graph\. The following higher\-level skill depends on it and must be rewritten\. HIGHER\-LEVEL SKILL: \{skill\_h definition\} REMOVED SKILL: \{skill\_k description\} MODEL PERFORMANCE on skill\_k tasks after distillation: \{retention\_stats\} Rewrite skill\_h’s procedure P\_h: replace explicit calls to skill\_k with implicit assumptions that the model can now perform those steps natively\. Update the data flow instructions \(bridge\) to remove skill\_k as an intermediary\. Keep all V checkpoints unchanged\. Output the rewritten skill\_h in full\(m​e​t​a,C,P,V,I,E\)\(meta,C,P,V,I,E\)format\.

## Appendix CHyperparameter Sensitivity Analysis

### C\.1Skill Effectiveness Thresholdθ\\theta

Table 5:θ\\thetasensitivity analysis\. A higherθ\\thetaretains fewer but stronger skills, yielding higher distillation retention rate but lower coverage\.θ=0\.05\\theta=0\.05achieves the best balance between passed rate and distillation retention rate\. A threshold that is too low introduces low\-quality skills that add noise, while a threshold that is too high leads to insufficient coverage\.

### C\.2Retention Rate Thresholdτ\\tau

Table 6:τ\\tausensitivity analysis\. A higherτ\\tauimposes stricter internalization quality requirements, causing more skills to be retained as external scaffolds\.τ=0\.85\\tau=0\.85achieves a balance between model autonomy \(independence from external skills\) and performance\. In practice, it can be adjusted based on tolerance for external dependencies\.

### C\.3Distillation Order Comparison

Table 7:Distillation order sensitivity\. “\# Forgetting Events” refers to the number of times an already\-internalized skill’s retention rate falls belowτ\\tauafter a distillation round\.Topological bottom\-up order performs best \(retention rate 85\.2%, only 1 forgetting event\), because lower\-level skills are internalized first and then implicitly replayed during upper\-level training\. Removing pretrain mixing, bottom\-up still achieves 81\.8%, higher than random order with pretrain mixing, indicating that topological ordering is the primary contributing factor, with pretrain mixing adding 3\.4pp on top\. Top\-down performs worst \(79\.3%, 4 forgetting events\)\. Note: this comparison is conducted on a scaffold graph of 30 skills; the differences are expected to be more pronounced in larger graph structures\.

## Appendix DAdditional Skill Examples

### D\.1Automatically Trained Skill Examples

Skill 1: Requirement Decomposition and Implementation Planning\(flexible\)

meta: name=requirement\-decomposition, tags=\[planning, multi\-file\], tools=\[Read, Bash\]

CC: The task description contains multiple functional requirements or requires modifications to multiple modules\.

PP\(flexible\): \(1\) Decompose task requirements into independent sub\-requirements\. \(2\) Identify the files and modules involved for each sub\-requirement\. \(3\) Identify dependency relationships among sub\-requirements\. \(4\) Order the implementation plan by dependencies\. \(5\) Define acceptance criteria for each sub\-requirement\.

VV\(evidence\-driven\): \(a\) Cross\-check the original requirement list against the subtask list; count of uncovered items must be 0\. \(b\) Run topological sort on subtask dependencies; pass if acyclic\. \(c\) Each subtask includes at least one executable acceptance command \(e\.g\., a test case or build check\)\.

II: input = complete task description, output = ordered list of sub\-tasks \(each with file list and acceptance criteria\)\.

EE: Modifying all files at once without decomposing first—leading to missed dependencies and regression issues\.

Skill 2: Test\-Driven Development \(TDD\)\(rigid\)

meta: name=tdd\-workflow, tags=\[testing, quality\], tools=\[Read, Write, Edit, Bash\]

CC: The task requires adding or modifying functional code, and the project has testing infrastructure\.

PP\(rigid, enforced order\): \(1\) Analyze requirements and determine test strategy\. \(2\) Write failing tests covering normal and edge cases\. \(3\) Write minimal implementation to make tests pass\. \(4\) Refactor code while keeping tests green\. \(5\) Check test coverage≥\\geq80%\.

VV\(evidence\-driven\): \(a\) Test output from each stage serves as evidence\. \(b\) Existing tests are not broken\. \(c\) Test coverage for new functionality≥\\geq80%\.

II: input = functional requirements \+ project test framework info, output = test files \+ implementation files\.

EE: Writing implementation code directly without writing tests first—leading to missed edge cases and difficulty in verification\.

### D\.2Component\-by\-Component Comparison of Auto Skills and Human Skills

Table[8](https://arxiv.org/html/2608.05156#A4.T8)provides a component\-by\-component comparison of the automatically trained TDD skill and the human\-designed TDD skill\.

Table 8:Component\-by\-component comparison of automatically trained and human\-designed TDD skills\. All four components show high correspondence\.Key differences: \(1\) The human\-designedPPis more concise \(4 stages vs\. 5 steps\) and includes IDE\-specific operations \(mandatory git commit\); \(2\) The human\-designedVVemploys a stricter evidence\-driven strategy \(does not trust the model’s claim of completion, requires executable evidence\), while the automatically trainedVVis similar but does not enforce commits; \(3\) The human\-designedIIis embedded in a five\-tier architecture, while the automatically trainedIIis a standalone input/output specification\. The core structures are highly consistent, demonstrating that automatic training can arrive at skill structures similar to those designed manually\.

## Appendix ECatastrophic Forgetting Analysis

Table 9:Catastrophic forgetting summary during progressive distillation\. Of 30 skills, 22 were internalized with an average forgetting of only 1\.4pp\.The overall forgetting magnitude is small \(average 1\.4pp, maximum 3\.1pp\), attributable to five mechanisms: \(1\) self\-distillation introduces less distribution shift than standard SFT; \(2\) bottom\-up distillation causes lower\-level skills to be implicitly replayed in subsequent rounds; \(3\) mixing 50% pretraining data per round prevents distribution shift; \(4\) the KL divergence constraint \(β=0\.01\\beta=0\.01\) prevents parameters from deviating too far; \(5\) per\-round retention rate checks ensure that underperforming skills are retained as external scaffolds rather than forcibly internalized\. Lower\-level skills \(distilled first\) are repeatedly implicitly replayed in subsequent rounds, continuously reinforced, achieving the best internalization\.

## Appendix FFine\-Grained Analysis by Task Complexity

Table 10:Passed rate stratified by number of files involved\. Skills yield the largest improvement on medium\-complexity tasks\.Skills yield the most significant improvement on medium\-complexity tasks \(\+10\.0pp\)\. The smaller improvement on easy tasks \(\+7\.5pp\) is because the baseline already has a relatively high passed rate; the limited improvement on hard tasks \(\+6\.0pp\) is because even with skill guidance, highly complex tasks still exceed the current model’s capabilities\.

## Appendix GSeed Data Details

The 2000 seed tasks were batch\-generated using the FeatureBench data pipeline on additional repositories, sourced from open\-source projects including django, flask, requests, celery, tornado, etc\., withno overlapwith the 24 repositories in the FeatureBench evaluation set, avoiding data leakage\. Selection criteria: \(1\) involving modifications to at least 3 files; \(2\) containing identifiable strategic steps; \(3\) covering at least 8 development patterns\. 200 validation tasks were used for skill quality evaluation\. Evaluation was conducted on the FeatureBench standard Full set \(200 tasks, 24 repositories\), with no overlap with seed data or training data\.

## Appendix HCase analysis

Case 1: Success—multi\-file dependency modification\. A task involving 5 file modifications; the Baseline missed 2 dependency files, causing a build failure\. After skill injection, “requirements decomposition” planned subtasks and “multi\-file consistency check” modified and verified files in topological order, completing the task successfully\. After distillation, the model retained dependency analysis and stepwise verification behavior without skills, indicating the strategy was internalized\.

Case 2: Failure—creative decision\-making\. A task requiring the design of an entirely new data structure\. Skills helped with “how to do it” but offered limited assistance for the creative decision of “what to do,” revealing that skills primarily improve procedural capabilities rather than creative ones\.

Case 3: Configuration file coordination\. The task required modifying an application configuration and ensuring all modules depending on that configuration work correctly\. The Baseline model modified the main configuration file but missed two test environment configuration files\. After injecting the “multi\-file consistency check” skill, the model systematically searched for all files referencing the configuration, updating and verifying each one\.

Case 4: Skill composition effect\. A task requiring a new REST API\. The “requirement decomposition” skill split the task into 5 sub\-tasks \(routing, validation, logic, formatting, testing\), and the “TDD” skill handled the testing portion\. The two skills were automatically chained viaII: the output of requirement decomposition served as input to TDD\.

Case 5: Skill false activation\. A simple docstring update task was matched by the “requirement decomposition” skill \(because it involved docstrings in multiple files\)\. The skill guided the model through unnecessary dependency analysis, increasing execution time without affecting correctness\. Such false activations \(12% of tasks\) suggest room for improvement by raising theθ\\thetathreshold or refining theCCcomponent\.

## Appendix IVerification Mechanism Details

The design practice of 136\+ manually authored skills led to a six\-stage verification workflow:

1. 1\.BUILD: Build passes \(exit code 0\)
2. 2\.TYPES: Type checking passes with no errors
3. 3\.LINT: Code style checks pass
4. 4\.TEST: Tests pass \+ coverage≥\\geq80%
5. 5\.SECURITY: Security scan \(no sensitive information leakage\)
6. 6\.DIFF: Change audit \(no unexpected file modifications\)

Stages are executed in strict order, halting on first failure\. This layered verification provides high\-quality filtering signals for distillation: only traces that pass all verifications are retained as training data\.

Verification follows the evidence\-driven principle: the model’s claims of completion are not accepted; executable evidence is required \(build exit codes, test execution output, coverage numbers, git commit records\)\.

## Appendix JFormal Composition Mechanism

Given two skillssas\_\{a\}andsbs\_\{b\}, whenIa\.out∩Ib\.in≠∅I\_\{a\}\.\\text\{out\}\\cap I\_\{b\}\.\\text\{in\}\\neq\\emptyset, the composed skillsa​bs\_\{ab\}is defined as:

sa​b=\(metaa​b,Ca∧Cb,Pa⊕bridge⊕Pb,Va∪Vb,Ia​b,Ea∪Eb\)s\_\{ab\}=\(\\textit\{meta\}\_\{ab\},\\;C\_\{a\}\\wedge C\_\{b\},\\;P\_\{a\}\\oplus\\text\{bridge\}\\oplus P\_\{b\},\\;V\_\{a\}\\cup V\_\{b\},\\;I\_\{ab\},\\;E\_\{a\}\\cup E\_\{b\}\)whereIa​b\.in=Ia\.in∪\(Ib\.in∖Ia\.out\)I\_\{ab\}\.\\text\{in\}=I\_\{a\}\.\\text\{in\}\\cup\(I\_\{b\}\.\\text\{in\}\\setminus I\_\{a\}\.\\text\{out\}\),Ia​b\.out=Ib\.out∪\(Ia\.out∖Ib\.in\)I\_\{ab\}\.\\text\{out\}=I\_\{b\}\.\\text\{out\}\\cup\(I\_\{a\}\.\\text\{out\}\\setminus I\_\{b\}\.\\text\{in\}\),⊕\\oplusdenotes sequential concatenation, andbridgeis an automatically generated data\-passing instruction\. This definition preserves external inputs and unconsumed outputs\. If conflicting checkpoints exist inVa∪VbV\_\{a\}\\cup V\_\{b\}, verification is performed only on the final output, with intermediateVVserving as soft constraints\.

## Appendix KComputational Cost Details

Table 11:Computational cost by stage\. All experiments were conducted on 512×\\timesH100\-80GB\. Skill Discovery and Data Generation involve extensive LLM inference calls and execution\-based verification, constituting the primary computational bottleneck\. Distillation is standard fine\-tuning with cost comparable to conventional SFT\. Note that the total cost of this framework should not be directly compared with standard SFT—SFT uses pre\-existing data, whereas this framework includes the data production process \(Skill Discovery \+ Data Generation\), which is an additional but necessary investment\.Fine\-tuning hyperparameters\.Full\-parameter fine\-tuning uses the AdamW optimizer with learning rate1×10−51\\times 10^\{\-5\}, cosine decay schedule, warmup ratio 5%, effective batch size 256, weight decay 0\.01, 1 epoch per distillation round, and maximum sequence length 160K\. Both models use identical fine\-tuning hyperparameters\.

Inference hyperparameters\.Qwen3\-Coder\-480B\-A35B\-Instruct: temperature=0\.7, top\_p=0\.8, top\_k=20, repetition\_penalty=1\.05\. DeepSeek\-V3\.2: temperature=1\.0, top\_p=0\.95\. Both models are configured with a maximum context length of 160K\.

## Appendix LResolved Rate Supplementary Results

Table[12](https://arxiv.org/html/2608.05156#A12.T12)reports the strict Resolved Rate \(all fail\-to\-pass tests must pass for a task to count as resolved\)\. Resolved Rates on FeatureBench Full are generally low, because end\-to\-end feature development tasks typically involve multiple tests and the bar for passing all of them is high\. The relative trends across configurations are consistent with Passed Rate: Auto Skills outperform Baseline, Progressive Distilled outperforms Direct Fine\-tune, confirming the robustness of main\-text conclusions\.

Table 12:Strict Resolved Rate \(%\) on FeatureBench Full\. Trends are consistent with Passed Rate \(Table[2](https://arxiv.org/html/2608.05156#S5.T2)\)\.
## Appendix MGeneral capability preservation

To verify that progressive distillation does not degrade the base model’s general capabilities, we evaluate DeepSeek\-V3\.2 before and after distillation on multiple standard benchmarks\. Table[13](https://arxiv.org/html/2608.05156#A13.T13)shows that post\-distillation performance on general \(MMLU\-Pro\), scientific reasoning \(GPQA Diamond\), and mathematics \(AIME 2025\) remains essentially unchanged\.

Table 13:General capability comparison before and after distillation \(DeepSeek\-V3\.2\)\. All metrics fluctuate by≤\\leq0\.4pp, confirming no degradation\.
## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and introduction clearly state four contributions \(new paradigm, new representation, new mechanism, empirical validation\) and these are supported by experiments in Section 5\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: Section 7 discusses limitations including domain scope \(code generation only\), teacher diversity \(same\-model self\-distillation\), and provides concrete transfer directions\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[N/A\]
14. Justification: The paper is primarily empirical\. Formal definitions \(Definition 1, 2\) are provided for skill and scaffold graph, but no theorems or proofs are claimed\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: Section 4 provides evaluation benchmark \(FeatureBench\), model specifications, experimental configurations \(Table 2\), seed data construction, evaluation metrics, and statistical methodology \(5 random seeds\)\. Algorithms 1–3 in the appendix provide complete pseudocode\. Prompt templates are in Appendix B\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: The complete framework will be open\-sourced upon acceptance, as stated in the conclusion\. FeatureBench is an existing open\-source benchmark\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: Sections 3 and 4 specify all hyperparameters:θ=0\.05\\theta=0\.05\(Section 3\.2\),τ=0\.85\\tau=0\.85,δ=0\.02\\delta=0\.02,β=0\.01\\beta=0\.01\(Section 3\.4\), 5 random seeds, 200 evaluation tasks \(FeatureBench Full\), 2000 seed tasks, full parameter fine\-tuning \(Section 4\)\. Sensitivity analyses are in Appendix C\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: Table 2 reports mean±\\pmstandard deviation across 5 random seeds\. The factor of variability \(model inference randomness\) is stated in Section 4\.4\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: Section[4\.1](https://arxiv.org/html/2608.05156#S4.SS1)specifies model size \(35B and 37B activated parameters, full parameter fine\-tuning\)\. The cost appendix provides a breakdown by stage \(Discovery and Data Generation are the main bottlenecks; Distillation is standard fine\-tuning\), with hardware specified as 512×\\timesH100\-80GB\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The research uses publicly available benchmarks and open\-source models\. No human subjects or private data are involved\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[Yes\]
49. Justification: Section 8 \(Social Impact\) discusses the positive impact: the framework addresses the data scarcity bottleneck in complex coding tasks, provides a new path to break through performance plateaus, and lowers the technical barrier for complex programming by distilling engineering best practices into model parameters\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: The paper proposes a training methodology, not a pre\-trained model release\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: FeatureBench \(MIT License\), Qwen3\-Coder \(Apache 2\.0\), and DeepSeek\-V3\.2 \(MIT License\) are properly cited\. All are publicly available open\-source/open\-weight assets with permissive licenses\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://arxiv.org/html/2608.05156v1/paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: No new datasets or models are released at submission time\. The code will be open\-sourced upon acceptance\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: The paper does not involve crowdsourcing or human subjects research\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The paper does not involve human subjects research\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research?
78. Answer:\[Yes\]
79. Justification: This paper studies LLM post\-training to automatically acquire procedural strategies\. LLM calls are core to skill discovery, data generation, and dynamic recompilation \(Section 3\)\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.