An Information-Theoretic Criterion for Efficient Data Synthesis

# An Information-Theoretic Criterion for Efficient Data Synthesis
Source: [https://arxiv.org/html/2605.16379](https://arxiv.org/html/2605.16379)
###### Abstract

Synthetic data becomes crucial for large language model training, but its effectiveness is highly inconsistent\. We provide an information\-theoretic account of this inconsistency: synthetic data improves a model only when the generation\-training loop is information\-open, i\.e\., shaped by external signals \(verifiers, environments, or rubrics\) that inject task\-relevant information beyond the model’s current distribution\. When the loop is information\-closed \(relying on the model’s own outputs without such signals\), the data processing inequality ensures that task\-relevant information can only decrease, making collapse a predicted outcome\. Among information\-open pipelines, both efficiency and generalization hinge on the meta\-level of supervision: a coarser signal such as binary correctness treats all acceptable outputs as equivalent, so the behavior it teaches is not tied to any particular domain or surface form and generalizes naturally across tasks and domains\. These observations lead to a guiding thesis: learning preferentially converges to the most information\-efficient signal component available, which accelerates learning when that component is the intended one, but causes reward hacking when a spurious pattern happens to be simpler\.

Machine Learning, ICML, Synthetic Data, Information Theory, Data Processing Inequality

## 1Introduction

Scaling laws have driven steady progress in large language models \(LLMs\), with performance improving predictably as model size, compute, and training data grow\(Kaplanet al\.,[2020](https://arxiv.org/html/2605.16379#bib.bib22)\)\. This paradigm, however, rests on a premise that is becoming increasingly fragile: the continued availability of abundant, high\-quality real\-world text\. Recent analyses suggest that such data may be approaching practical limits\(Villaloboset al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib48)\)\. At the same time, as LLMs are pushed toward long\-horizon reasoning, multi\-step planning, and agentic interaction, the most effective forms of supervision increasingly rely on explicit constraints, feedback, and verification—going well beyond what generic text corpora can provide\. These two trends—data scarcity and the demand for richer supervision—have made synthetic data a central ingredient in modern training pipelines\.

Yet synthetic data remains a double\-edged sword\. On one hand, generated supervision has produced substantial gains: verifier\-guided synthesis reaches olympiad\-level geometry\(Trinhet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib46)\), and reinforcement learning from verifiable rewards has driven major reasoning breakthroughs\(Guoet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib16); Zhenget al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib55)\)\. On the other hand, iterative self\-training—where models are trained on their own outputs—often leads to distribution collapse and progressive capability degradation\(Shumailovet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib41); Alemohammadet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib2); Gerstgrasseret al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib14)\)\. Despite numerous empirical mitigation techniques, a principled account of when and why synthetic data helps or hurts is still lacking\. This motivates a concrete question:under what conditions does synthetic data yield sustained improvements, and when does it induce degradation?

We approach this question through an information\-theoretic lens, anchored in the data processing inequality \(DPI\)\. The DPI states a simple but powerful fact: no computation on observed data can create new information about the underlying source\. Applied to self\-training, the implication is immediate\. When a model trains on samples drawn from its own distribution and no additional task\-relevant signal enters the loop, the pipeline isinformation\-closed: each iteration can only preserve or lose information about the true task\. In practice, finite sampling and optimization errors make the loss strict, so the model’s knowledge of the task degrades over iterations\. Model collapse is thus not an anomaly—it is the predicted outcome of a closed information loop\.

This DPI argument, however, is too pessimistic on its own: it cannot explain why synthetic data so often works spectacularly well\. The resolution is that successful pipelines are not actually closed systems\. They incorporateexternal signals—verifiers, rubrics, environments, or fixed teacher models—that carry task\-relevant information independent of the current model state\. Once we account for these signals, the DPI argument sharpens rather than breaks: the pessimistic monotonicity holds only when such signals are absent\. When an external signal is present, the relevant information bound includes not just the training data but also what the signal contributes beyond it\. The question then shifts from “can synthetic data add information?” to “how much task\-relevant information does the external signal carry beyond what the model already knows?” Sustained improvement is possible precisely when this quantity remains positive—that is, when the loop staysinformation\-open\.

Given that a pipeline is information\-open, a natural follow\-up is:how efficientlydoes it inject information? We argue that a key determinant is what we call themeta\-levelof the supervision signal—the granularity at which it distinguishes outputs\. A low meta\-level signal specifies a particular target \(e\.g\., a single reference answer\); a high meta\-level signal specifies only a coarse property \(e\.g\., “is the answer correct?”\) and treats all outputs satisfying that property as interchangeable\. Because a high meta\-level signal collapses many surface\-level differences into a single judgment, each unit of supervision constrains a much larger region of the output space, making learning markedly more sample\-efficient\.

A natural implication, and a guiding thesis of this paper, is thatlearning preferentially converges to the most information\-efficient signal component availablein the training data\. This thesis cuts both ways: when the intended signal is the most efficient one, it explains why high meta\-level supervision produces rapid learning and strong cross\-domain generalization; when a spurious signal happens to operate at a coarser granularity than the intended one, the same dynamic produces reward hacking\.

#### Conflict of Interest Disclosure\.

The authors declare no financial conflicts of interest related to the work presented in this paper\.

## 2Case Studies on Synthetic Data

We begin by defining precisely what we mean by synthetic data and identifying three recurring empirical patterns that any theoretical account must explain\.

We usesyntheticin a mechanism\-based sense: supervision is synthetic if it is produced by an automated pipeline \(often an LLM generator plus tools\) and used for training\. Under this definition, “synthetic” doesnotmean “fake”: it only indicates that the supervision originates from an automated procedure rather than direct human annotation\.

In much of the literature, “synthetic data” is used roughly synonymously withmodel\-generated text\. Our definition is broader inwhatcounts as supervision \(not only instruction–response pairs but also preferences, rewards, verifier feedback, and retained trajectories\)\. Concretely, even when prompts \(and sometimes final answers\) come from real datasets, reinforcement learning with verifiable rewards \(RLVR\) can still generate intermediate trajectories that are retained only if they pass verifiable checks; those retained trajectories function assynthetic supervisionin our sense\. With this terminology, we summarize three recurring patterns below: closed\-loop self\-training \(failure\), hard verifiers \(success I\), and soft but stable references \(success II\)\.

#### Failure \(closed\-loop self\-training\)\.

When training primarily recycles the model’s own generationswithouta persistent external signal, the loop tends to contract the output distribution: tail patterns are under\-sampled and errors become self\-reinforcing\(Shumailovet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib41); Alemohammadet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib2)\)\. The effect is strongest when synthetic data progressivelyreplacesreal data, whereas mixing real data each iteration can mitigate collapse\(Gerstgrasseret al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib14)\)\. Synthetic\-heavy regimes can also distort scaling behavior by weakening long\-tail modeling\(Dohmatobet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib12)\)\.

#### Success I \(hard, verifiable signals\)\.

A high\-reliability path isgenerate\-and\-verify: candidates are filtered, ranked, or rewarded by an external procedure \(proof checker, compiler/tests, benchmark harness\), via rejection sampling, best\-of\-NN, or RLVR\(Guoet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib16); Duoet al\.,[2026](https://arxiv.org/html/2605.16379#bib.bib13)\)\. The headline outcome is*qualitative performance jumps anchored by checkable signals*: in math, verifier\-guided search and distillation can reach medal\-level performance \(e\.g\., IMO 2024 silver\-medal standard\)\(Hubertet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib20); AlphaProof and AlphaGeometry teams,[2024](https://arxiv.org/html/2605.16379#bib.bib3)\), and AlphaGeometry\-style constrained synthesis reaches gold\-medalist\-level geometry solving\(Trinhet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib46); Chervonyiet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib8)\)\. In formal proving, similar recipes scale to strong benchmark performance under Lean verification\(Renet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib37); Chenet al\.,[2025b](https://arxiv.org/html/2605.16379#bib.bib7),[a](https://arxiv.org/html/2605.16379#bib.bib6)\)\. In discovery loops, formal/executable targets let systems like FunSearch, LegoNE, and AlphaEvolve report algorithmic improvements that are*certified*by solvers or proof obligations \(including guarantees that surpass prior best known human\-designed ones\)\(Romera\-Paredeset al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib38); Liet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib26); Novikovet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib32)\)\.

In code and systems, execution provides the verifier: AlphaCode and OpenCodeInterpreter rely on tests/runtime feedback\(Liet al\.,[2022](https://arxiv.org/html/2605.16379#bib.bib25); Zhenget al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib55)\), and agentic pipelines evaluate artifacts in containerized environments\(DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib11); Xiaomi LLM\-Core Teamet al\.,[2026](https://arxiv.org/html/2605.16379#bib.bib49)\)\. The same template supports optimization loops in SAT solving and code evolution\(Sunet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib43),[2025](https://arxiv.org/html/2605.16379#bib.bib42)\)and extends to kernel performance engineering with correctness checks plus runtime selection\(Chenet al\.,[2025c](https://arxiv.org/html/2605.16379#bib.bib5); Ouyanget al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib34)\)\.

#### Success II \(soft but stable references\)\.

For objectives without crisp correctness predicates, stability comes fromholding the reference fixed: teacher models, preference/rubric constraints, constitutions, or evaluation harnesses that do not co\-move freely with the student\. Examples include distillation and preference\-based alignment\(Hintonet al\.,[2015](https://arxiv.org/html/2605.16379#bib.bib17); Gudibandeet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib15); Tunstallet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib47)\), on\-policy distillation with teacher\-side judging\(Xiaomi LLM\-Core Teamet al\.,[2026](https://arxiv.org/html/2605.16379#bib.bib49); Agarwalet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib1); Lu,[2025](https://arxiv.org/html/2605.16379#bib.bib29)\), fixed\-principle self\-improvement\(Baiet al\.,[2022](https://arxiv.org/html/2605.16379#bib.bib4)\), staged filtering pipelines\(Gudibandeet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib15)\), and iterative search under a fixed evaluator\(Yuanet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib51)\)\.

## 3A Theoretical Analysis through Data Processing Inequality

As is discussed in the case studies, some synthetic\-data pipelines behave like a closed system that only recycles the model’s own distribution, while others remain open because they are persistently shaped by signals not determined by the model itself\. This section makes that distinction precise using the data processing inequality \(DPI\), and then states a practical criterion for when synthetic data can be effective\.

Our goal is not to deny DPI, but to clarify the modeling boundary: DPI applies to aninformation\-closedMarkov abstraction\. Synthetic data can be effective only when the actual pipeline isnotinformation\-closed, i\.e\., when it contains additional variables \(external signals, verifiers, fixed references, environments\) that are correlated with the underlying sourceXXand therefore cannot be absorbed into a single fixed channel\.

### 3\.1Markov Chain Formalization

LetXXdenote the underlying source of structure we ultimately care about \(e\.g\., the task distribution that induces correctness\),DDdenote the finite observations used for learning \(e\.g\., a training dataset\), andZZdenote the learned model state \(e\.g\., parameters\)\.

We model training as a \(possibly randomized\) procedure that mapsDDtoZZ\. To keep the role of stochasticity explicit, introduce an auxiliary random variableRRthat aggregates all internal randomness used by the algorithm \(initialization, minibatch sampling, optimizer noise, decoding randomness, etc\.\)\. The training procedure can then be written as

Z=ftrain​\(D,R\)\.Z=f\_\{\\mathrm\{train\}\}\(D,R\)\.Theinformation\-closedassumption is that the procedure has no additional side information aboutXXbeyond what is already inDD; equivalently,

P​\(Z∣X,D\)=P​\(Z∣D\),P\(Z\\mid X,D\)=P\(Z\\mid D\),so thatX→D→ZX\\to D\\to Zforms a Markov chain\.

Under this assumption, DPI yields

I​\(X;Z\)≤I​\(X;D\),I\(X;Z\)\\leq I\(X;D\),whereI​\(⋅;⋅\)I\(\\cdot;\\cdot\)is Shannon mutual information\. A compact proof follows from the chain rule:

I​\(X;D,Z\)\\displaystyle I\(X;D,Z\)=I​\(X;D\)\+I​\(X;Z∣D\)\\displaystyle=I\(X;D\)\+I\(X;Z\\mid D\)=I​\(X;Z\)\+I​\(X;D∣Z\),\\displaystyle=I\(X;Z\)\+I\(X;D\\mid Z\),and Markovness impliesI​\(X;Z∣D\)=0I\(X;Z\\mid D\)=0; henceI​\(X;Z\)=I​\(X;D\)−I​\(X;D∣Z\)≤I​\(X;D\)I\(X;Z\)=I\(X;D\)\-I\(X;D\\mid Z\)\\leq I\(X;D\)\.

Finally, it is useful to separate “randomness” from “information\.” Internal randomnessRRcan change optimization dynamics and exploration, but it does not by itself increase information aboutXXunless it is coupled to some variable correlated withXX\. Formally, ifRRcarries no information aboutXXbeyondDD, thenI​\(X;D,R\)=I​\(X;D\)I\(X;D,R\)=I\(X;D\)and DPI applied toX→\(D,R\)→ZX\\to\(D,R\)\\to Zstill gives the same upper bound\.

### 3\.2Failure of Self\-Training Loops

Consider a purely self\-training loop\. At iterationtt, a model stateZtZ\_\{t\}generates synthetic supervisionDtsynD^\{\\mathrm\{syn\}\}\_\{t\}via a \(possibly stochastic\) generation procedurefgenf\_\{\\mathrm\{gen\}\}\(the “synthetic data” in the mechanism\-based sense of this paper\), and the next model state is obtained by training on that synthetic supervision:

Dtsyn∼fgen​\(Zt\),Zt\+1=ftrain​\(Dtsyn,Rt\)\.D^\{\\mathrm\{syn\}\}\_\{t\}\\sim f\_\{\\mathrm\{gen\}\}\(Z\_\{t\}\),\\qquad Z\_\{t\+1\}=f\_\{\\mathrm\{train\}\}\(D^\{\\mathrm\{syn\}\}\_\{t\},R\_\{t\}\)\.If the pipeline introduces no persistent external signal \(no fresh human data, no environment feedback, no fixed verifier/reference that is independent of the current model state\), then the iteration is information\-closed in the sense that all variables downstream ofZtZ\_\{t\}are generated fromZtZ\_\{t\}and internal randomness\. In that case we have the Markov chain

X→Zt→Dtsyn→Zt\+1\.X\\to Z\_\{t\}\\to D^\{\\mathrm\{syn\}\}\_\{t\}\\to Z\_\{t\+1\}\.
Applying DPI to this chain yields a monotonicity statement:

I​\(X;Zt\+1\)≤I​\(X;Zt\)for all​t\.I\(X;Z\_\{t\+1\}\)\\leq I\(X;Z\_\{t\}\)\\qquad\\text\{for all \}t\.This captures the core limitation of closed\-loop self\-training: when synthetic data is sampled from the model’s own distribution and then fed back without any independent signal, the loop cannot systematically increase the model’s information aboutXX\. In principle, the best possible outcome is information preservation; in practice, finite sampling, capacity limits, and approximation errors typically make the inequality strict, so performance degrades over iterations\.

As a concrete instance of this phenomenon beyond the well\-studied model collapse setting, we observed the following in ongoing work on instruction\-following evaluation\. A judge model was trained to assess whether LLM responses satisfy user\-specified rubrics \(e\.g\., “is the reply concise?”, “does the response follow the requested format?”\)\. In one training variant, the judge was asked toinferimplicit rubrics from the conversation context and then evaluate compliance—without any external ground\-truth rubrics to anchor its judgments\. When this judge was trained iteratively \(using its own previous outputs as training signal for the next iteration, each time in a fresh context\), its acceptance rate decreased monotonically across iterations: it became progressively more critical of outputs, eventually rejecting nearly everything\. The degradation was not because the evaluated outputs worsened—they were held constant—but because the judge’s internal criteria drifted without external correction\. Each iteration appeared locally reasonable \(the judge produced plausible critiques\), yet the cumulative effect was systematic quality collapse invisible from within the loop\. This is precisely the closed\-loop monotonicity predicted above: without a stable external anchor, the “signal” becomes a function of the model’s own evolving state\.

### 3\.3Information\-Openness and Effective Synthetic Data

The preceding limitation is not a special property of language models; it is a direct consequence of modeling the loop as information\-closed\. The key question is therefore not whether data is “synthetic,” but whether the pipeline isinformation\-openwith respect toXX\.

Modern successful pipelines typically introduceexternal signalsthat shape either the generation stage or the training stage\. Here an “external signal” means any random variableSSthat \(i\) influences which synthetic outputs are retained or amplified, and \(ii\) is not itself determined solely by the model’s current output distribution\. Examples include executable verifiers, environments, frozen judges or rubrics, and fixed references that do not freely co\-move with the current model\. Formally, we call a pipelineinformation\-openifI​\(X;S∣D\)\>0I\(X;S\\mid D\)\>0, i\.e\., the external signal carries information aboutXXbeyond what is already contained inDD\.

Once such anSSexists, treating the whole pipeline as a single fixed channelP​\(Z∣D\)P\(Z\\mid D\)becomes misleading: the learned model is no longer a function ofDDalone\. The correct information\-theoretic object is the augmented observation\(D,S\)\(D,S\)\. In this view, synthetic data can be effective only insofar asSSprovides task\-relevant constraints or feedback that are correlated withXXand remain stable enough to act as a signal across iterations\.

### 3\.4Refining DPI via External Signals

We now refine the DPI argument by accounting for external signals: we do not violate DPI; rather, the relevant observation is the augmented pair\(D,S\)\(D,S\)rather thanDDalone, so the pessimistic closed\-loop DPI monotonicity need not apply\.

LetSSdenote an external signal used by the pipeline, which is a random variable that may depend onXXandDD\. We model training as

Z=ftrain​\(D,S,R\)\.Z=f\_\{\\mathrm\{train\}\}\(D,S,R\)\.ThenX→\(D,S\)→ZX\\to\(D,S\)\\to Zis a Markov chain, and DPI gives

I​\(X;Z\)≤I​\(X;D,S\)\.I\(X;Z\)\\leq I\(X;D,S\)\.By the chain rule,

I​\(X;D,S\)=I​\(X;D\)\+I​\(X;S∣D\)\.I\(X;D,S\)=I\(X;D\)\+I\(X;S\\mid D\)\.This yields a simple criterion for effectiveness: a synthetic\-data pipeline can improve the model’s information aboutXXbeyond what is available inDDonly through the additional conditional mutual information termI​\(X;S∣D\)I\(X;S\\mid D\)\. Equivalently, synthetic data is effective when the external signal contributes nontrivial information aboutXXthat is not already implied byDD\.

For iterative synthetic training, the same idea produces a per\-iteration bound\. IfZt\+1Z\_\{t\+1\}is produced from\(Zt,S\)\(Z\_\{t\},S\)\(possibly via intermediate synthetic data\), thenX→\(Zt,S\)→Zt\+1X\\to\(Z\_\{t\},S\)\\to Z\_\{t\+1\}and

I​\(X;Zt\+1\)≤I​\(X;Zt,S\)=I​\(X;Zt\)\+I​\(X;S∣Zt\)\.I\(X;Z\_\{t\+1\}\)\\leq I\(X;Z\_\{t\},S\)=I\(X;Z\_\{t\}\)\+I\(X;S\\mid Z\_\{t\}\)\.This clarifies when an iteration can avoid closed\-loop monotonicity: withoutSS, the additional capacity termI​\(X;S∣Zt\)I\(X;S\\mid Z\_\{t\}\)vanishes and the bound reduces to the closed\-loop monotonicityI​\(X;Zt\+1\)≤I​\(X;Zt\)I\(X;Z\_\{t\+1\}\)\\leq I\(X;Z\_\{t\}\); with anSSthat remains meaningfully correlated withXXconditioned onZtZ\_\{t\}, the loop can, in principle, introduce new task\-relevant constraints and sustain improvement\.

This also clarifies the role of stochasticity\. Random sampling does not by itself inject information aboutXX—it merely generates a diverse pool of candidate outputs\. The external signal then acts as a filter on this pool, retaining or reinforcing candidates that happen to align with the task\. Neither component alone suffices: without randomness, the model has no diversity to select from; without the signal, diverse candidates cannot be distinguished\. Effective synthetic pipelines thus operate as an explore\-then\-select loop, where stochasticity provides exploration andSSprovides selection pressure\.

The per\-iteration bound is also consistent with the observation that thestructureof noise matters far more than its magnitude\. Unbiased noise \(random disagreements that are independent across samples\) cancels in expectation across gradient updates and does not systematically corrupt learning\. By contrast, systematic bias accumulates coherently; and if the signal co\-evolves with the student model \(e\.g\., a judge that drifts without calibration, as in[Section3\.2](https://arxiv.org/html/2605.16379#S3.SS2)\), it ceases to be genuinely external and the loop effectively re\-closes\. We return to this asymmetry with detailed empirical evidence in[Section4](https://arxiv.org/html/2605.16379#S4)\.

A striking engineering validation of this stability requirement comes from Anthropic’s generator\-evaluator harness for long\-running agentic applications\(Rajasekaran,[2026](https://arxiv.org/html/2605.16379#bib.bib36)\), which architecturally separates a generator agent \(that proposes solutions\) from a fixed evaluator agent \(that judges quality\)\. The evaluator is deliberately held constant and does not adapt to the generator’s outputs; it functions as a stable external signal in precisely the sense of[Section3\.3](https://arxiv.org/html/2605.16379#S3.SS3)\. This design yielded large gains over single\-agent approaches where the same model both generates and evaluates—exactly as the framework predicts: a co\-moving evaluator closes the loop and triggers degradation, while a fixed evaluator keeps the loop information\-open\.

### 3\.5How External Signals Enter Training

To make the external\-signal framework concrete, we compare three representative methods that form a natural progression in how much they rely on an external signal: supervised fine\-tuning \(SFT\), rejection\-sampling fine\-tuning \(RFT\), and reinforcement learning with verifiable rewards \(RLVR, often implemented with GRPO\-style updates\(Shaoet al\.,[2024](https://arxiv.org/html/2605.16379#bib.bib40)\)\)\.

In standard SFT, we optimize log\-likelihood on a fixed dataset:

maxθ⁡𝔼\(x,y\)∼qdata​\[log⁡πθ​\(y∣x\)\]\.\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{\(x,y\)\\sim q\_\{\\text\{data\}\}\}\\bigl\[\\log\\pi\_\{\\theta\}\(y\\mid x\)\\bigr\]\.Each training pair\(x,y⋆\)\(x,y^\{\\star\}\)contributes a gradientgSFT=∇θlog⁡πθ​\(y⋆∣x\)g\_\{\\text\{SFT\}\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y^\{\\star\}\\mid x\)that simply increases the probability of a provided target\. There is no external signal beyond the dataset itself\.

RFT introduces an external signal throughfiltering\. The current model proposes candidatesy∼πθt\(⋅∣x\)y\\sim\\pi\_\{\\theta\_\{t\}\}\(\\cdot\\mid x\), and an acceptance testa​\(x,y;S\)∈\{0,1\}a\(x,y;S\)\\in\\\{0,1\\\}\(e\.g\., a correctness check\) retains only those that pass\. SFT is then run on the retained samples, whose distributionπθtacc​\(y∣x,S\)∝πθt​\(y∣x\)​a​\(x,y;S\)\\pi^\{\\text\{acc\}\}\_\{\\theta\_\{t\}\}\(y\\mid x,S\)\\propto\\pi\_\{\\theta\_\{t\}\}\(y\\mid x\)\\,a\(x,y;S\)is jointly shaped by the model proposal and the signal\. The signal reshapeswhich samples become training data\.

RLVR introduces the external signal directly into thegradient\. A verifier returns a rewardr​\(x,y;S\)r\(x,y;S\)for model\-generated outputs, and the policy gradient

gRLVR​\(θ;x\)∝𝔼y∼πθ\(⋅∣x\)​\[A​\(x,y;S\)​∇θlog⁡πθ​\(y∣x\)\],g\_\{\\text\{RLVR\}\}\(\\theta;x\)\\propto\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[A\(x,y;S\)\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\\mid x\)\\right\],whereA​\(x,y;S\)≔r​\(x,y;S\)−b​\(x\)A\(x,y;S\)\\coloneqq r\(x,y;S\)\-b\(x\)is an advantage relative to a baseline, directly weights each sample’s contribution by the signal\. Outputs with above\-baseline reward get positive weight; below\-baseline outputs get negative weight\. Unlike SFT, RLVR does not specify any particular target output—it selectively reinforces or suppresses model\-generated behaviors based on the external signal alone\.

The SFT–RFT–RLVR progression above uses hard verifiable signals, but the same\(D,S\)\(D,S\)formalism applies whenSSis a soft reference—such as a fixed rubric evaluated by an LLM judge \([Section4](https://arxiv.org/html/2605.16379#S4)\) or a held\-out evaluator that does not co\-adapt with the student\. The critical requirement is not hardness of verification butstability:SSmust not freely co\-move withZtZ\_\{t\}, so thatI​\(X;S∣Zt\)I\(X;S\\mid Z\_\{t\}\)remains positive across iterations\.

## 4Information Injection Efficiency

The previous section establishes an information\-openness criterion: synthetic data can improve a model only when an external signal injects task\-relevant information that the model cannot generate on its own\. This criterion, while precise, is largely a formalization of an intuitive observation—closed loops degrade, open loops can improve\. The more substantive question is what happensamonginformation\-open pipelines: given that some external signal exists, why do different methods exhibit orders\-of\-magnitude differences in sample efficiency and generalization? Some pipelines require massive volumes of synthetic supervision to yield marginal gains, while others trigger broad, cross\-task behavioral changes with comparatively little data\. This section develops a quantitative framework that explains this variance and identifies the structural property of the supervision signal that determines efficiency\.

### 4\.1Meta\-Level of Information Injection

We defineinformation injection efficiencyas the fraction of supervision information that is directed at task\-relevant distinctions, rather than at irrelevant within\-class details\. Intuitively, a synthetic\-data step is efficient if it eliminates much of the model’s uncertainty about what it should output, rather than merely adding more surface\-form examples\. To connect this intuition to information theory with minimal notation, we fix a prompt random variableQQand an output random variableYYtaking values in an output space𝒴\\mathcal\{Y\}\(an entire model response\)\. A synthetic\-data pipeline produces a supervision signalSSabout the pair\(Q,Y\)\(Q,Y\)\(e\.g\., accept/reject, a score, a preference\)\. The uncertainty reduction caused by one such supervised interaction isI​\(Y;S∣Q\)I\(Y;S\\mid Q\)—the number of bits by whichSSreduces the model’s uncertainty over its outputs\.

The remaining question is what makes this quantity large or small\. The key idea ofmeta\-levelcan be stated informally: different supervision signals care about different aspects of an output\. Some signals care about a very specific surface form \(a unique target string\), while others only care about a coarse property \(e\.g\., “is it acceptable?”, “does it compile?”\)\. A higher meta\-level signalignoresmany superficial differences between outputs and only distinguishes a small number of behaviorally meaningful categories\. When supervision is higher meta\-level, each labeled example can constrain a larger portion of the output space because it rules out whole categories at once, instead of identifying a single preferred surface form\.

A simple quantitative example makes this concrete\. Fix a promptqqand suppose there areMMdifferent answers that are all equally acceptable under an external criterion \(e\.g\., multiple correct proofs, multiple valid patches\)\. Denote this acceptable set by𝒜⊆𝒴\\mathcal\{A\}\\subseteq\\mathcal\{Y\}with\|𝒜\|=M\|\\mathcal\{A\}\|=M\. Assume for clarity that the model’s uncertainty within𝒜\\mathcal\{A\}is roughly uniform\.111Uniformity is not necessary for the argument; it only makes the bound explicit\. In general the within\-class entropy is at mostlog⁡M\\log M\.

#### High meta\-level supervision \(class\-level signal\)\.

A high meta\-level pipeline provides a coarse signal that only cares about whether the output falls inside the acceptable set:

Shigh​\(q,y\)=𝟏​\{y∈𝒜\}\.S\_\{\\mathrm\{high\}\}\(q,y\)=\\mathbf\{1\}\\\{y\\in\\mathcal\{A\}\\\}\.This signal deliberately treats allMMacceptable answers as equivalent: it does not distinguish between different members of𝒜\\mathcal\{A\}\. The uncertainty reduction contributed by observing this signal is

I​\(Y;Shigh∣Q=q\)=H​\(Shigh∣Q=q\),I\(Y;S\_\{\\mathrm\{high\}\}\\mid Q\{=\}q\)=H\(S\_\{\\mathrm\{high\}\}\\mid Q\{=\}q\),becauseShighS\_\{\\mathrm\{high\}\}is a deterministic function ofYYgivenqq\. If the model is unsure whether it will land inside𝒜\\mathcal\{A\}\(accept\) or outside \(reject\), thenH​\(Shigh∣Q=q\)H\(S\_\{\\mathrm\{high\}\}\\mid Q\{=\}q\)can be close to one bit, meaning each feedback instance can remove close to one bit of uncertainty by collapsing the space into just two categories: acceptable vs\. unacceptable\. Crucially, this signal never spends any information budget telling the modelwhichacceptable answer to prefer; it only tells the model to move probability mass into the acceptable region\.

#### Low meta\-level supervision \(instance\-level identification\)\.

A low meta\-level pipeline instead provides aspecific targety⋆∈𝒜y^\{\\star\}\\in\\mathcal\{A\}and trains the model to reproduce that particular surface form \(e\.g\., a single canonical reference answer\):

Slow​\(q,y\)=𝟏​\{y=y⋆\}\.S\_\{\\mathrm\{low\}\}\(q,y\)=\\mathbf\{1\}\\\{y=y^\{\\star\}\\\}\.This signal is much finer: it distinguishes one specific acceptable output from the otherM−1M\-1acceptable outputs that are, from the external criterion’s perspective, equally good\. Under the uniform\-within\-𝒜\\mathcal\{A\}assumption, the probability that the model outputs exactlyy⋆y^\{\\star\}is about1/M1/M, hence the information gain of this indicator is approximately

H​\(Slow∣Q=q\)≈h2​\(1/M\),H\(S\_\{\\mathrm\{low\}\}\\mid Q\{=\}q\)\\approx h\_\{2\}\(1/M\),whereh2​\(x\)=−x​log⁡x−\(1−x\)​log⁡\(1−x\)h\_\{2\}\(x\)=\-x\\log x\-\(1\{\-\}x\)\\log\(1\{\-\}x\)is binary entropy\. For largeMM\(which is typical in practice\),h2​\(1/M\)h\_\{2\}\(1/M\)is very close to zero, so each observation ofSlowS\_\{\\mathrm\{low\}\}provides a very weak information signal about the model’s output space: almost alwaysSlow=0S\_\{\\mathrm\{low\}\}=0, which tells the learner little about what to do next\.

More importantly, reproducingy⋆y^\{\\star\}forces the learner to resolve an additional identification burden inside the acceptable set\. If the acceptable answers are all treated as different targets, then “being correct” is not enough; the learner must also identifywhichmember of the acceptable set is desired\. The maximum number of bits needed to identify a particular member amongMMacceptable ones islog⁡M\\log M\. This can be seen directly from the within\-class entropy: under uniformity,

H​\(Y∣Q=q,Y∈𝒜\)=log⁡M\.H\(Y\\mid Q\{=\}q,\\,Y\\in\\mathcal\{A\}\)=\\log M\.Instance\-level supervision implicitly attempts to drive this within\-class entropy toward zero by selecting a single representative, while high meta\-level supervision does not\. Therefore, whenMMis large, a low meta\-level objective can waste a substantial portion of supervision capacity on distinctions that are irrelevant to acceptability, while the high meta\-level objective concentrates only on the coarse distinction that matters\.

In verifiable domains such as mathematics, this is the key difference between SFT and RLVR\. SFT imitates a specific referencey⋆y^\{\\star\}, spending capacity on identifying which particular acceptable answer to produce—a distinction irrelevant to the task criterion\. RLVR uses verifiable correctness \(membership in𝒜\\mathcal\{A\}\), treating all correct solutions as equivalent and training the model to reachanycorrect answer\. This ties learning to an invariant criterion, letting the model discover and reuse any verifier\-satisfying strategy, which typically yields stronger out\-of\-distribution robustness\.

A direct validation of this prediction is JudgeRLVR\(Duoet al\.,[2026](https://arxiv.org/html/2605.16379#bib.bib13)\), which trains a judge model on mathematics using only binary correctness as supervision—the signal is purely whether the response answers the query correctly, without any domain\-specific rubric or reference solution\. Because this signal treats all correct solutions as equivalent and does not depend on the domain being mathematics, the trained judge transfers to all other domains \(code, logic, general reasoning\) without further supervision, outperforming judges trained with domain\-specific rewards\. The mechanism is exactly what the meta\-level analysis predicts: binary correctness is the coarsest task\-relevant signal, so every bit of learned judgment generalizes across domains rather than being spent on domain\-specific distinctions\.

In this sense, generalization is precisely the ability to correctly “forget” irrelevant differences—to learn at a higher meta\-level\.

### 4\.2Formalizing the Efficiency Gap

The intuitive comparison above can be generalized into a decomposition theorem that applies to any supervision signal and any task\-relevant distinction\.

###### Definition 4\.1\(Task\-Relevant Partition\)\.

Atask\-relevant partitionπ=\{C1,…,CK\}\\pi=\\\{C\_\{1\},\\dots,C\_\{K\}\\\}of𝒴\\mathcal\{Y\}is a partition such that the external evaluation criterion assigns the same judgment to all elements within the same block\. The inducedquotient variable\[Y\]π∈\{1,…,K\}\[Y\]\_\{\\pi\}\\in\\\{1,\\dots,K\\\}is defined by\[Y\]π=k\[Y\]\_\{\\pi\}=kwheneverY∈CkY\\in C\_\{k\}\. A supervision signalS=σ​\(Q,Y\)S=\\sigma\(Q,Y\)isπ\\pi\-measurableifSSdepends onYYonly through\[Y\]π\[Y\]\_\{\\pi\}\.

The accept/reject partition of the preceding example \(K=2K=2,C1=𝒜C\_\{1\}=\\mathcal\{A\},C2=𝒴∖𝒜C\_\{2\}=\\mathcal\{Y\}\\setminus\\mathcal\{A\}\) is a special case\. In general,π\\pican encode any evaluation structure: multi\-level rubric scores, partial\-credit grading, or tiered quality judgments\.

###### Theorem 4\.2\(Decomposition of Supervision Information\)\.

For any supervision signalSSand task\-relevant partitionπ\\pi,

I​\(Y;S∣Q\)=I​\(\[Y\]π;S∣Q\)⏟task\-relevant gain\+I​\(Y;S∣\[Y\]π,Q\)⏟within\-class gain\.I\(Y;\\,S\\mid Q\)\\;=\\;\\underbrace\{I\\bigl\(\[Y\]\_\{\\pi\};\\,S\\mid Q\\bigr\)\}\_\{\\textup\{task\-relevant gain\}\}\\;\+\\;\\underbrace\{I\\bigl\(Y;\\,S\\mid\[Y\]\_\{\\pi\},\\,Q\\bigr\)\}\_\{\\textup\{within\-class gain\}\}\.\(1\)AssumingI​\(Y;S∣Q\)\>0I\(Y;\\,S\\mid Q\)\>0, we define thetask\-relevant efficiencyηπ​\(S\):=I​\(\[Y\]π;S∣Q\)/I​\(Y;S∣Q\)\\eta\_\{\\pi\}\(S\):=I\(\[Y\]\_\{\\pi\};\\,S\\mid Q\)\\,/\\,I\(Y;\\,S\\mid Q\)\. Then:\(i\)ηπ​\(S\)∈\[0,1\]\\eta\_\{\\pi\}\(S\)\\in\[0,1\];\(ii\)ηπ​\(S\)=1\\eta\_\{\\pi\}\(S\)=1iffY⟂⟂S∣\[Y\]π,QY\\perp\\\!\\\!\\\!\\perp S\\mid\[Y\]\_\{\\pi\},Q;\(iii\)everyπ\\pi\-measurable signal satisfiesηπ​\(S\)=1\\eta\_\{\\pi\}\(S\)=1\.

###### Proof\.

Since\[Y\]π\[Y\]\_\{\\pi\}is a deterministic function ofYY, the chain rule givesI​\(Y;S∣Q\)=I​\(\[Y\]π;S∣Q\)\+I​\(Y;S∣\[Y\]π,Q\)I\(Y;S\\mid Q\)=I\(\[Y\]\_\{\\pi\};S\\mid Q\)\+I\(Y;S\\mid\[Y\]\_\{\\pi\},Q\)\. Both terms are nonnegative\. Part \(iii\): ifS=g​\(Q,\[Y\]π\)S=g\(Q,\[Y\]\_\{\\pi\}\), thenI​\(Y;S∣\[Y\]π,Q\)=0I\(Y;S\\mid\[Y\]\_\{\\pi\},Q\)=0, givingηπ=1\\eta\_\{\\pi\}=1\. ∎

The decomposition formalizes the intuition from the preceding example: the total information injected by any signal splits cleanly into a task\-relevant component \(resolving which equivalence class the output belongs to\) and a within\-class component \(resolving which specific element within that class\)\. The SFT\-vs\-RLVR comparison of[Section4\.1](https://arxiv.org/html/2605.16379#S4.SS1)is a direct instance: RLVR’s binary verifier isπ\\pi\-measurable \(ηπ=1\\eta\_\{\\pi\}=1\), while SFT’s instance\-level target spendslog⁡M\\log Madditional bits on within\-class identification—capacity that is irrelevant to the task criterion and grows without bound as the number of acceptable solutionsMMincreases\.

A natural implication, consistent with known simplicity biases in gradient\-based learning, is that learning will preferentially converge to whichever signal component is most information\-efficient to exploit\. When the training signal contains multiple learnable patterns at different granularities, the model does not choose among them based on task relevance—it converges to the pattern that provides the largest per\-sample gradient signal, which is the one operating at the coarsest partition \(highestη\\eta\)\. When the intended task signal happens to be the most efficient one, this is desirable: learning is fast, robust, and generalizes broadly\. But when a spurious signal operates at a coarser granularity than the intended one, the model will converge to the spurious pattern first—this is the information\-theoretic mechanism underlying reward hacking \(see[Section4\.4](https://arxiv.org/html/2605.16379#S4.SS4)for a detailed example\)\. This convergence claim is not a formal consequence of the decomposition alone; it additionally relies on the empirical observation that gradient\-based learners preferentially exploit patterns with larger per\-sample signal, which coarser partitions provide\.

The decomposition also reveals whyπ\\pi\-measurable signals are inherentlyrobust to noise\. Consider rubric\-based RL for instruction following, where an LLM judge evaluates whether responses satisfy a set of rubrics \(e\.g\., “Does the response use bullet\-point format?”, “Is the reply under 200 words?”\) and a sample receives reward 1 only if all rubrics are judged as satisfied\. To measure the judge’s intrinsic noise, we scored 1,000 \(prompt, response\) pairs 16 independent times each\. Sample\-level consistency—all 16 scorings producing identical verdicts on every rubric—was only 24%\. Yet RL training using this judge converged to strong out\-of\-distribution generalization\. The reason connects directly to the theorem: because the reward isπ\\pi\-measurable \(it only distinguishes “all rubrics satisfied” from “not all satisfied”\), the noise exists entirelywithineach partition block—different rubric items flip in different directions on different runs, but the between\-class signal \(satisfies\-all vs\. not\) remains directionally consistent across samples\. Random within\-class noise cancels in the gradient; the task\-relevant signal accumulates\. This is a second advantage of high meta\-level signals beyond efficiency: they are structurally immune to noise that lacks systematic directionality at the class level\.

### 4\.3Application: Why Diversity Beats Volume

The partition framework also explains a widely observed empirical pattern: data*diversity*often matters more than data*volume*for generalization\. A small but diverse dataset can outperform a much larger but narrow one\. The mechanism is straightforward once we think in terms of partition coverage\.

Given a task\-relevant partitionπ=\{C1,…,CK\}\\pi=\\\{C\_\{1\},\\dots,C\_\{K\}\\\}and aπ\\pi\-measurable signalSS, consider what happens when the same \(prompt, output\) pair is observed repeatedly\. SinceSSis a deterministic function of\(Q,\[Y\]π\)\(Q,\[Y\]\_\{\\pi\}\), repeated observations of the same pair contributeexactly zeroadditional information: lettingS\(1\),…,S\(ℓ\)S^\{\(1\)\},\\dots,S^\{\(\\ell\)\}denote signals from successive observations of the same pair,I​\(\[Y\]π;S\(ℓ\+1\)∣S\(1\),…,S\(ℓ\),Q=q\)=0I\(\[Y\]\_\{\\pi\};\\,S^\{\(\\ell\+1\)\}\\mid S^\{\(1\)\},\\dots,S^\{\(\\ell\)\},\\,Q\{=\}q\)=0whenever the same output is observed\. In contrast, anewprompt that exercises an uncoveredπ\\pi\-block contributes up toH​\(\[Y\]π∣Q\)H\(\[Y\]\_\{\\pi\}\\mid Q\)fresh bits of task\-relevant information\. The marginal value of a new prompt is therefore bounded below by the gap between the covered and uncovered partition blocks, while the marginal value of a repeated prompt is exactly zero\.

This explains why strong alignment has been achieved with only 1,000 diverse examples in prior work: the task\-relevant partition for alignment \(helpful vs\. harmful vs\. refusal\) has relatively few blocks, so full coverage requires diversity, not volume\. Similarly, scaling data\-constrained models encounters diminishing returns from repeated data: after the partition is well\-covered, additional samples contribute mostly within\-class information \(if the signal is fine\-grained\) or zero information \(if the same observation is repeated\)\. The practical implication is clear: when designing synthetic\-data pipelines, prompt diversity—covering as many distinctπ\\pi\-blocks as possible—should be prioritized over data volume on already\-covered regions\.

### 4\.4High\-Efficiency Information Injection in Practice

The meta\-level view suggests a concrete signature of high\-efficiency injection: the supervision signal should induce a constraint that transfers across prompts, domains, and even interaction formats\. The following examples illustrate how such transfer emerges from meta\-level signals\.

A first example is to inject a format that is essentially task\-agnostic\. DeepSeek\-R1\-style post\-training makes this explicit by rewardingformat compliancein addition to correctness, e\.g\., requiring the model to wrap intermediate reasoning and final answers inside tags such as<think\>\.\.\.</think\>\(Guoet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib16)\)\. Format compliance defines a global constraint on the output space that does not depend on any particular task distribution\. Once learned, it manifests as cross\-task transfer: the same “output interface” applies to math, coding, and natural\-language tasks alike\.

The second example is chain\-of\-thought \(CoT\) compression, where the injected signal is explicitly meta\-level\. In mastery\-gated CoT compression with sample\-level soft penalties, the feedback depends only on the length of current rollouts and is applied only after the model already reaches stable correctness\(Liet al\.,[2026](https://arxiv.org/html/2605.16379#bib.bib28)\)\. This signal is agnostic to what the problem is: it does not rely on task identity, domain\-specific structure, or even what the model is “doing,” as long as the answer remains correct\. The result is one\-to\-all\-domain generalization: training on math alone makes reasoning shorter in code, instruction following, and general QA, and can even reduce the number of turns in agentic settings, while preserving accuracy\.

The meta\-level framework also explains why certain training failures occur\. In ongoing work on training a judge model for math, code, and logic evaluation, we constructed training data by pairing positive examples \(correct solutions\) with negative examples \(incorrect solutions\)\. An inadvertent data\-construction choice assigned all positives from one model family \(Gemini, which tends to produce longer, more detailed outputs with a distinctive prose style\) and all negatives from another \(MiMo, which produces shorter, more terse outputs\)\. The intended signal wascorrectness; the spurious signal waslength and style\.

The judge quickly learned the spurious shortcut: rather than evaluating mathematical correctness, it learned to predict length and style\. The telltale sign was a divergence between two metrics—the judge’s training reward continued to rise \(it was getting better at distinguishing long from short outputs\), while its actual correctness accuracy on a balanced held\-out set plateaued \(it was not getting better at judging math\)\. Critically, the reward hacking was invisible from inspecting the model’s chain\-of\-thought—it still appeared to reason carefully about mathematical correctness, showing no obvious change in its reasoning patterns\. The failure was only detectable through indirect metrics \(validation\-set output length stopped increasing, indicating the model was no longer exploring\)\.

Re\-balancing the data sources so that length and style were decorrelated from correctness immediately eliminated the hacking and restored accuracy growth\. In terms of Theorem[4\.2](https://arxiv.org/html/2605.16379#S4.Thmtheorem2), length and style define a simpler, more easily exploitable pattern than mathematical correctness—the model can detect them from surface statistics without evaluating reasoning—so the spurious signal achieves higherη\\etafor thewrongpartition\. The model converges to it preferentially because it is more information\-efficient to learn\.

Finally, meta\-level information injection can happenwithoutupdating parameters, purely by placing the model in a fixed propose–evaluate–select loop\. SATLUTION\(Sunet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib42)\)targets efficient SAT solvers and LegoNE\(Liet al\.,[2025](https://arxiv.org/html/2605.16379#bib.bib26)\)aims to discover general algorithms for approximate Nash equilibria with better provable worst\-case guarantees\. Both systems apply LLMs as explorers and use external verifiers \(benchmarks, formal analyzers\) to filter and retain only high\-quality candidates\. The meta\-level signal comes from how the verifiercollapsesthe search space: many very different proposals are treated as interchangeable once they satisfy the same executable or formal checks\. Empirically, SATLUTION evolves solver variants that outperform the human\-designed winners of SAT Competition evaluations, and LegoNE discovers algorithms whose certified worst\-case guarantees surpass all previously known human\-designed ones\.

Taken together, these examples highlight a consistent pattern: strong generalization appears when supervision targets meta\-level signals—format compliance, verifier\-defined validity, stable process structure, or formally checkable subspaces—so that learning can “forget” irrelevant differences and operate on a simpler quotient space of behaviors\.

## 5Discussion

The guiding thesis of this paper—that learning preferentially converges to the most information\-efficient signal component available—reframes a phenomenon often treated as a training failure\. Reward hacking is conventionally understood as the model “exploiting” the reward instead of learning the intended behavior\. Under our framework, the model is not failing to learn; it is learning exactly what the training signal most efficiently teaches\. When a spurious pattern \(such as output length or stylistic cues\) happens to operate at a coarser granularity than the intended criterion \(such as correctness\), gradient\-based optimization converges to the spurious pattern first, because it is more information\-efficient to exploit\. This means reward hacking cannot be eliminated simply by training harder or longer—it requires either ensuring that the intended signal is genuinely the most information\-efficient component, or decorrelating spurious coarse\-grained patterns from the reward\.

The framework also identifies a concrete bottleneck for scaling synthetic data:verification capacity\. Information\-open loops require external signals, and the most efficient such signals are high meta\-level verifiers—binary correctness checks, executable tests, formal proof obligations\. Accordingly, progress is fastest in domains where such verifiers are readily available: mathematics, code, and formal reasoning\. In domains where reliable verification is difficult—open\-ended generation, nuanced safety judgments, creative tasks—the loop either remains closed or relies on soft references \(LLM judges, human rubrics\) that are noisier and may co\-drift with the model\. The practical implication is that expanding the frontier of what can be reliably verified may matter more for capability gains than scaling data or compute alone\.

This perspective echoes Sutton’sBitter Lesson\(Sutton,[2019](https://arxiv.org/html/2605.16379#bib.bib44)\): general methods that leverage computation ultimately outperform methods that build in human knowledge\. In our framing, instance\-level supervision—hard\-coding particular solutions, formats, or trajectories—is the synthetic\-data analogue of “building knowledge in\.” Meta\-level signals, by contrast, specify only what must hold \(correctness, format compliance, constraint satisfaction\) and let computation discover how to achieve it—precisely Sutton’s prescription to invest in methods thatfind and capturestructure rather than in hand\-crafted structure itself\(Sutton,[2019](https://arxiv.org/html/2605.16379#bib.bib44)\)\.

Our framework is primarily qualitative: it identifies structural conditions for effective synthetic data but does not predict, for instance, the sample complexity of a given pipeline\. The convergence thesis is supported by our case studies and by known simplicity biases in gradient\-based learning, but it is not a formal consequence of the DPI decomposition alone\. Finally, the framework takes the external signal as given; analyzing how verifier errors propagate through iterative training, and how to design verifiers that remain reliable as models improve, are important open directions\.

## Acknowledgements

This work is supported by the Natural Science Foundation of China \(Grant No\. 62572010\)\.

## Impact Statement

This paper presents work whose goal is to advance the field of general artificial intelligence\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. Bachem \(2024\)On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Alemohammad, J\. Casco\-Rodriguez, L\. Luzi, A\. I\. Humayun, H\. Babaei, D\. LeJeune, A\. Siahkoohi, and R\. G\. Baraniuk \(2024\)Self\-Consuming Generative Models Go MAD\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=ShjMHfmPs0)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px1.p1.1)\.
- AlphaProof and AlphaGeometry teams \(2024\)AI achieves silver\-medal standard solving International Mathematical Olympiad problems\.Note:\(accessed 2026\-01\-28\)[https://deepmind\.google/blog/ai\-solves\-imo\-problems\-at\-silver\-medal\-level/](https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/)External Links:[Link](https://deepmind.google/blog/ai-solves-imo-problems-at-silver-medal-level/)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon, C\. Chen, C\. Olsson, C\. Olah, D\. Hernandez, D\. Drain, D\. Ganguli, D\. Li, E\. Tran\-Johnson, E\. Perez,et al\.\(2022\)Constitutional AI: Harmlessness from AI Feedback\.arXiv\.External Links:2212\.08073,[Document](https://dx.doi.org/10.48550/arXiv.2212.08073),[Link](http://arxiv.org/abs/2212.08073)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Chen, W\. Chen, J\. Du, J\. Hu, Z\. Jiang, A\. Jie, X\. Jin, X\. Jin, C\. Li, W\. Shi, Z\. Wang, M\. Wang, C\. Wei, S\. Wei, H\. Xin, F\. Yang, W\. Gao, Z\. Yuan, T\. Zhan,et al\.\(2025a\)Seed\-Prover 1\.5: Mastering Undergraduate\-Level Theorem Proving via Learning from Experience\.arXiv\.External Links:2512\.17260,[Document](https://dx.doi.org/10.48550/arXiv.2512.17260),[Link](http://arxiv.org/abs/2512.17260)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Chen, J\. Gu, L\. Huang, W\. Huang, Z\. Jiang, A\. Jie, X\. Jin, X\. Jin, C\. Li, K\. Ma, C\. Ren, J\. Shen, W\. Shi, T\. Sun, H\. Sun, J\. Wang, S\. Wang, Z\. Wang, C\. Wei,et al\.\(2025b\)Seed\-Prover: Deep and Broad Reasoning for Automated Theorem Proving\.arXiv\.External Links:2507\.23726,[Document](https://dx.doi.org/10.48550/arXiv.2507.23726),[Link](http://arxiv.org/abs/2507.23726)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Chen, B\. Xu, and K\. Devleker \(2025c\)Automating GPU Kernel Generation with DeepSeek\-R1 and Inference Time Scaling\.Note:\(accessed 2026\-01\-28\)[https://developer\.nvidia\.com/blog/automating\-gpu\-kernel\-generation\-with\-deepseek\-r1\-and\-inference\-time\-scaling/](https://developer.nvidia.com/blog/automating-gpu-kernel-generation-with-deepseek-r1-and-inference-time-scaling/)External Links:[Link](https://developer.nvidia.com/blog/automating-gpu-kernel-generation-with-deepseek-r1-and-inference-time-scaling/)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.
- Y\. Chervonyi, T\. H\. Trinh, M\. Olšák, X\. Yang, H\. Nguyen, M\. Menegali, J\. Jung, J\. Kim, V\. Verma, Q\. V\. Le, and T\. Luong \(2025\)Gold\-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2\.arXiv\.External Links:2502\.03544,[Document](https://dx.doi.org/10.48550/arXiv.2502.03544),[Link](http://arxiv.org/abs/2502.03544)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- DeepSeek\-AI, A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Lu, C\. Zhao, C\. Deng, C\. Xu, C\. Ruan, D\. Dai, D\. Guo, D\. Yang,et al\.\(2025\)DeepSeek\-V3\.2: Pushing the Frontier of Open Large Language Models\.arXiv\.External Links:2512\.02556,[Document](https://dx.doi.org/10.48550/arXiv.2512.02556),[Link](http://arxiv.org/abs/2512.02556)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.
- E\. Dohmatob, Y\. Feng, P\. Yang, F\. Charton, and J\. Kempe \(2024\)A Tale of Tails: Model Collapse as a Change of Scaling Laws\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=KVvku47shW)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Duo, H\. Li, H\. Zhang, Y\. Wang, S\. Li, and L\. Zhao \(2026\)JudgeRLVR: Judge First, Generate Second for Efficient Reasoning\.arXiv\.External Links:2601\.08468,[Document](https://dx.doi.org/10.48550/arXiv.2601.08468),[Link](http://arxiv.org/abs/2601.08468)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2605.16379#S4.SS1.SSS0.Px2.p4.1)\.
- M\. Gerstgrasser, R\. Schaeffer, A\. Dey, R\. Rafailov, T\. Korbak, H\. Sleight, R\. Agrawal, J\. Hughes, D\. B\. Pai, A\. Gromov, D\. Roberts, D\. Yang, D\. L\. Donoho, and S\. Koyejo \(2024\)Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=5B2K4LRgmz)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Gudibande, E\. Wallace, C\. Snell, X\. Geng, H\. Liu, P\. Abbeel, S\. Levine, and D\. Song \(2024\)The False Promise of Imitating Proprietary Language Models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=Kz3yckpCN5)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li, Z\. Gao, A\. Liu,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-025-09422-z),[Link](https://www.nature.com/articles/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2605.16379#S4.SS4.p2.1)\.
- G\. Hinton, O\. Vinyals, and J\. Dean \(2015\)Distilling the Knowledge in a Neural Network\.arXiv\.External Links:1503\.02531,[Document](https://dx.doi.org/10.48550/arXiv.1503.02531),[Link](http://arxiv.org/abs/1503.02531)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Hubert, R\. Mehta, L\. Sartran, M\. Z\. Horváth, G\. Žužić, E\. Wieser, A\. Huang, J\. Schrittwieser, Y\. Schroecker, H\. Masoom, O\. Bertolli, T\. Zahavy, A\. Mandhane, J\. Yung, I\. Beloshapka, B\. Ibarz, V\. Veeriah, L\. Yu, O\. Nash,et al\.\(2025\)Olympiad\-level formal mathematical reasoning with reinforcement learning\.Nature,pp\. 1–3\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-025-09833-y),[Link](https://www.nature.com/articles/s41586-025-09833-y)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling Laws for Neural Language Models\.arXiv\.External Links:2001\.08361,[Document](https://dx.doi.org/10.48550/arXiv.2001.08361),[Link](http://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p1.1)\.
- H\. Li, J\. Duo, B\. Gao, H\. Zhang, S\. Li, X\. Deng, and L\. Zhao \(2026\)Reinforcement Learning for Chain of Thought Compression with One\-Domain\-to\-All Generalization\.arXiv\.External Links:2601\.06052,[Document](https://dx.doi.org/10.48550/arXiv.2601.06052),[Link](http://arxiv.org/abs/2601.06052)Cited by:[§4\.4](https://arxiv.org/html/2605.16379#S4.SS4.p3.1)\.
- H\. Li, D\. Li, and X\. Deng \(2025\)Discovering Expert\-Level Nash Equilibrium Algorithms with Large Language Models\.arXiv\.External Links:2508\.11874,[Document](https://dx.doi.org/10.48550/arXiv.2508.11874),[Link](http://arxiv.org/abs/2508.11874)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1),[§4\.4](https://arxiv.org/html/2605.16379#S4.SS4.p7.1)\.
- Y\. Li, D\. Choi, J\. Chung, N\. Kushman, J\. Schrittwieser, R\. Leblond, T\. Eccles, J\. Keeling, F\. Gimeno, A\. Dal Lago, T\. Hubert, P\. Choy, C\. De Masson d’Autume, I\. Babuschkin, X\. Chen, P\. Huang, J\. Welbl, S\. Gowal, A\. Cherepanov,et al\.\(2022\)Competition\-level code generation with AlphaCode\.Science378\(6624\),pp\. 1092–1097\.External Links:ISSN 0036\-8075, 1095\-9203,[Document](https://dx.doi.org/10.1126/science.abq1158),[Link](https://www.science.org/doi/10.1126/science.abq1158)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.
- K\. Lu \(2025\)On\-Policy Distillation\.Note:\(accessed 2026\-01\-28\)[https://thinkingmachines\.ai/blog/on\-policy\-distillation/](https://thinkingmachines.ai/blog/on-policy-distillation/)External Links:[Document](https://dx.doi.org/10.64434/tml.20251026),[Link](https://thinkingmachines.ai/blog/on-policy-distillation/)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Novikov, N\. Vũ, M\. Eisenberger, E\. Dupont, P\. Huang, A\. Z\. Wagner, S\. Shirobokov, B\. Kozlovskii, F\. J\. R\. Ruiz, A\. Mehrabian, M\. P\. Kumar, A\. See, S\. Chaudhuri, G\. Holland, A\. Davies, S\. Nowozin, P\. Kohli, and M\. Balog \(2025\)AlphaEvolve: A coding agent for scientific and algorithmic discovery\.arXiv\.External Links:2506\.13131,[Document](https://dx.doi.org/10.48550/arXiv.2506.13131),[Link](http://arxiv.org/abs/2506.13131)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Ouyang, S\. Guo, S\. Arora, A\. L\. Zhang, W\. Hu, C\. Re, and A\. Mirhoseini \(2025\)KernelBench: Can LLMs Write Efficient GPU Kernels?\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=yeoN1iQT1x)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.
- P\. Rajasekaran \(2026\)Harness design for long\-running application development\.Note:\(accessed 2026\-05\-09\)[https://www\.anthropic\.com/engineering/harness\-design\-long\-running\-apps](https://www.anthropic.com/engineering/harness-design-long-running-apps)External Links:[Link](https://www.anthropic.com/engineering/harness-design-long-running-apps)Cited by:[§3\.4](https://arxiv.org/html/2605.16379#S3.SS4.p6.1)\.
- Z\. Z\. Ren, Z\. Shao, J\. Song, H\. Xin, H\. Wang, W\. Zhao, L\. Zhang, Z\. Fu, Q\. Zhu, D\. Yang, Z\. F\. Wu, Z\. Gou, S\. Ma, H\. Tang, Y\. Liu, W\. Gao, D\. Guo, and C\. Ruan \(2025\)DeepSeek\-Prover\-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition\.arXiv\.External Links:2504\.21801,[Document](https://dx.doi.org/10.48550/arXiv.2504.21801),[Link](http://arxiv.org/abs/2504.21801)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Romera\-Paredes, M\. Barekatain, A\. Novikov, M\. Balog, M\. P\. Kumar, E\. Dupont, F\. J\. R\. Ruiz, J\. S\. Ellenberg, P\. Wang, O\. Fawzi, P\. Kohli, and A\. Fawzi \(2024\)Mathematical discoveries from program search with large language models\.Nature625\(7995\),pp\. 468–475\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-023-06924-6),[Link](https://www.nature.com/articles/s41586-023-06924-6)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXiv\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300),[Link](http://arxiv.org/abs/2402.03300)Cited by:[§3\.5](https://arxiv.org/html/2605.16379#S3.SS5.p1.1)\.
- I\. Shumailov, Z\. Shumaylov, Y\. Zhao, N\. Papernot, R\. Anderson, and Y\. Gal \(2024\)AI models collapse when trained on recursively generated data\.Nature631\(8022\),pp\. 755–759\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-024-07566-y),[Link](https://www.nature.com/articles/s41586-024-07566-y)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Sun, F\. Ye, Z\. Chen, K\. Wei, and S\. Cai \(2025\)Automatically discovering heuristics in a complex SAT solver with large language models\.arXiv\.External Links:2507\.22876,[Document](https://dx.doi.org/10.48550/arXiv.2507.22876),[Link](http://arxiv.org/abs/2507.22876)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1),[§4\.4](https://arxiv.org/html/2605.16379#S4.SS4.p7.1)\.
- Y\. Sun, F\. Ye, X\. Zhang, S\. Huang, B\. Zhang, K\. Wei, and S\. Cai \(2024\)AutoSAT: Automatically Optimize SAT Solvers via Large Language Models\.arXiv\.External Links:2402\.10705,[Document](https://dx.doi.org/10.48550/arXiv.2402.10705),[Link](http://arxiv.org/abs/2402.10705)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.
- R\. Sutton \(2019\)The bitter lesson\.Incomplete Ideas \(blog\)13\(1\),pp\. 38\.External Links:[Link](https://heartyhaven.github.io/files/bitter_lesson.pdf)Cited by:[§5](https://arxiv.org/html/2605.16379#S5.p3.1)\.
- T\. H\. Trinh, Y\. Wu, Q\. V\. Le, H\. He, and T\. Luong \(2024\)Solving olympiad geometry without human demonstrations\.Nature625\(7995\),pp\. 476–482\.External Links:ISSN 1476\-4687,[Document](https://dx.doi.org/10.1038/s41586-023-06747-5),[Link](https://www.nature.com/articles/s41586-023-06747-5)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Tunstall, E\. E\. Beeching, N\. Lambert, N\. Rajani, K\. Rasul, Y\. Belkada, S\. Huang, L\. V\. Werra, C\. Fourrier, N\. Habib, N\. Sarrazin, O\. Sanseviero, A\. M\. Rush, and T\. Wolf \(2024\)Zephyr: Direct Distillation of LM Alignment\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=aKkAwZB6JV)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- P\. Villalobos, A\. Ho, J\. Sevilla, T\. Besiroglu, L\. Heim, and M\. Hobbhahn \(2024\)Position: Will we run out of data? Limits of LLM scaling based on human\-generated data\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 49523–49544\.External Links:[Link](https://proceedings.mlr.press/v235/villalobos24a.html)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p1.1)\.
- Xiaomi LLM\-Core Team, B\. Xiao, B\. Xia, B\. Yang, B\. Gao, B\. Shen, C\. Zhang, C\. He, C\. Lou, F\. Luo, G\. Wang, G\. Xie, H\. Zhang, H\. Lv, H\. Li, H\. Chen, H\. Xu, H\. Zhang, H\. Liu,et al\.\(2026\)MiMo\-V2\-Flash Technical Report\.arXiv\.External Links:2601\.02780,[Document](https://dx.doi.org/10.48550/arXiv.2601.02780),[Link](http://arxiv.org/abs/2601.02780)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Yuan, R\. Y\. Pang, K\. Cho, X\. Li, S\. Sukhbaatar, J\. Xu, and J\. E\. Weston \(2024\)Self\-Rewarding Language Models\.InForty\-first International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=0NphYCmgua)Cited by:[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px3.p1.1)\.
- T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. Yue \(2024\)OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12834–12859\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762),[Link](https://aclanthology.org/2024.findings-acl.762/)Cited by:[§1](https://arxiv.org/html/2605.16379#S1.p2.1),[§2](https://arxiv.org/html/2605.16379#S2.SS0.SSS0.Px2.p2.1)\.