The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning

# The Chase Is the Curriculum, the Capture Anchors the Credit:Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning
Source: [https://arxiv.org/html/2608.21871](https://arxiv.org/html/2608.21871)
###### Abstract

Reinforcement learning with verifiable rewards has become the dominant recipe for improving large language model reasoning, yet it presumes large human\-curated task collections\. Zero\-data self\-play removes this dependency, but existing methods vet learnability only by probing candidates and rejecting post hoc, never learning where along an environment’s difficulty axis to place a task, and credit the solver with sparse terminal rewards alone\. We recast zero\-data self\-play as a*pursuit\-evasion game*: inLURE, an LLM*evader*positions tasks along each environment’s difficulty axis to stay one step ahead of a planner\-executor*pursuer*that hunts it down through verifiable interaction\. The evader is trained on a*capture\-frontier*reward that peaks when the solver captures it on exactly half of its rollouts, turning barely catchable into a learned positioning strategy rather than a hand\-tuned rejection band\. The pursuer earns*capture\-anchored dense process credit*, in which monotone verifier progress is group\-normalized jointly with the terminal capture under a round\-anchored KL that keeps the co\-evolution stable\. Across three verifiable reasoning environments and three backbone families,LUREoutperforms advanced baselines under unified/specialist settings, while the unified model attains stronger aggregate OOD zero\-shot accuracy than all trained baselines across nine held\-out benchmarks from three task families\.

## Introduction

Large language models are increasingly deployed on multi\-step verifiable reasoning tasks, including multi\-hop question answering\([6](https://arxiv.org/html/2608.21871#bib.bib9)\), instruction following\([34](https://arxiv.org/html/2608.21871#bib.bib7)\), and logic\-grid deduction\([14](https://arxiv.org/html/2608.21871#bib.bib8)\), where success requires planning and executing several intermediate steps\. The prevailing route to such abilities is reinforcement learning with verifiable rewards \(RLVR\), which optimizes the policy directly against programmatic checkers rather than a learned reward model\([20](https://arxiv.org/html/2608.21871#bib.bib1);[8](https://arxiv.org/html/2608.21871#bib.bib2);[12](https://arxiv.org/html/2608.21871#bib.bib33)\)\. Its effectiveness, however, is bounded by the supply of human\-curated tasks with gold verification, which is expensive to build and quickly saturated by strong models\([4](https://arxiv.org/html/2608.21871#bib.bib34)\)\. Self\-evolving training promises to lift this ceiling: the model generates its own curriculum and improves from verifiable feedback with*zero*external data\([11](https://arxiv.org/html/2608.21871#bib.bib13);[33](https://arxiv.org/html/2608.21871#bib.bib15)\)\.

![Refer to caption](https://arxiv.org/html/2608.21871v1/int_figure.png)Figure 1:Zero\-data self\-play paradigms\.*\(a\)*Pseudo\-label self\-play uses majority\-vote labels\([11](https://arxiv.org/html/2608.21871#bib.bib13)\), and*\(b\)*code\-executor self\-play uses program execution\([33](https://arxiv.org/html/2608.21871#bib.bib15)\)\. Both author tasks free\-form, vetted post hoc under outcome\-only supervision\.*\(c\)*LUREadapts the challenger toward solver’s capture frontier and gives the pursuer dense verifier\-based credit\.Existing zero\-data self\-play, whether grounded in majority\-vote pseudo\-labels\([11](https://arxiv.org/html/2608.21871#bib.bib13)\)or a code executor\([33](https://arxiv.org/html/2608.21871#bib.bib15)\)\(Fig\.[1](https://arxiv.org/html/2608.21871#Sx1.F1)a,b\), gives its curriculum no explicit difficulty axis: a proposer authors candidate tasks in text space, and learnability is enforced only*post hoc*, by probing each candidate with solver rollouts and rejecting those whose empirical solve rate falls outside a fixed band\([11](https://arxiv.org/html/2608.21871#bib.bib13)\)\. Task placement therefore stays implicit in the authored text, and the hand\-set band edges never adapt as the solver’s competence changes\. When that proposal is mismatched with the solver, rejection sampling degenerates into paying rollout cost only to discard almost everything\.

The same non\-adaptive curriculum leaves the solver under\-informed\. A multi\-step episode returns a single terminal reward, so the trajectory is credited as one block, and under group\-relative policy optimization the gradient vanishes whenever every rollout of a task fails\([17](https://arxiv.org/html/2608.21871#bib.bib16)\), precisely the case for hard, self\-generated tasks\. Densifying this signal with a process reward model reintroduces the external supervision that self\-play is meant to eliminate\([13](https://arxiv.org/html/2608.21871#bib.bib3);[27](https://arxiv.org/html/2608.21871#bib.bib4)\), and the round\-by\-round drift of the task distribution makes optimization non\-stationary and prone to collapse\. What is missing is a loop that learns from the verifier alone: a challenger trained to place tasks at the solver’s moving frontier, and dense credit derived, without annotation, from the environment’s own progress signal, stabilized across co\-evolution\.

To address these challenges, this paper formulates zero\-data self\-play as a*pursuit\-evasion game*\(Fig\.[1](https://arxiv.org/html/2608.21871#Sx1.F1)c\), reminiscent of Scotland Yard and of cops\-and\-robbers games on graphs\([1](https://arxiv.org/html/2608.21871#bib.bib17)\), in which an LLM challenger \(the evader\) learns where to*position*tasks and a planner\-executor solver \(the pursuer\) learns to hunt them down, co\-evolving on verifiable natural\-language reasoning environments\. The evader is optimized with GRPO on a*capture\-frontier*reward that peaks when the solver captures a task on half of its rollouts, minus a difficulty\-signature repetition penalty, so “stay exactly one step ahead of the pursuer” becomes a positioning strategy the challenger*learns*, rather than a hand\-tuned rejection band applied after blind sampling\. In turn, the pursuer is optimized with*capture\-anchored dense process credit*, in which each executor step earns the monotone verifier\-progress it adds, with zero\-progress steps paying a small redundancy cost, jointly group\-normalized with the terminal capture, under a round\-anchored KL applied to both players so that neither escapes the other’s reach within a round\. Instantiated over three heterogeneous environments with per\-role weights shared across all of them, this chase turns verifier feedback into both players’ training signal without any human data\. We call the methodLURE: the evader acts as a learned lure, holding each task barely catchable so that it pulls the solver up to its own frontier rather than fleeing beyond reach\. Our contributions are fourfold:

- •We formulate zero\-data self\-play as a pursuit\-evasion game and, to our knowledge, are the first to*co\-train*an LLM difficulty\-selection challenger with a multi\-turn planner\-executor solver on verifiable NL reasoning\.
- •We propose a capture\-frontier challenger objective that turns the solver’s group capture statistics into a training signal for difficulty positioning, replacing post\-hoc rejection with a learned curriculum policy\.
- •We introduce a capture\-anchored dense process credit from the environment verifier alone, with round\-anchored KL stabilization applied to both players\.
- •Across three verifiable reasoning environments and backbone families,LUREoutperforms advanced baselines under unified/specialist settings, attaining stronger aggregate OOD zero\-shot accuracy across nine benchmarks\.

## Related Work

### Verifiable Multi\-Step Reasoning\.

Multi\-step reasoning has advanced by making intermediate computation explicit and verifiable\. Chain\-of\-thought exposes reasoning paths\([29](https://arxiv.org/html/2608.21871#bib.bib32)\), while step\-level scoring improves both training and inference beyond final\-answer supervision\([13](https://arxiv.org/html/2608.21871#bib.bib3)\)\. Reinforcement learning with verifiable rewards turns programmatic checkers into optimization targets, with group\-relative policy optimization now standard for reasoning LLMs\([20](https://arxiv.org/html/2608.21871#bib.bib1);[8](https://arxiv.org/html/2608.21871#bib.bib2);[21](https://arxiv.org/html/2608.21871#bib.bib10)\)\. For multi\-step tasks, agents interleave decomposition, grounded action, and self\-correction\([31](https://arxiv.org/html/2608.21871#bib.bib5);[23](https://arxiv.org/html/2608.21871#bib.bib6)\), while process reward models densify terminal rewards but typically require step annotations that zero\-data methods seek to avoid\([27](https://arxiv.org/html/2608.21871#bib.bib4)\)\. These advances also motivate programmatically verifiable benchmarks for instruction following\([34](https://arxiv.org/html/2608.21871#bib.bib7)\), logic\-grid deduction\([14](https://arxiv.org/html/2608.21871#bib.bib8)\), and multi\-hop relational question answering\([6](https://arxiv.org/html/2608.21871#bib.bib9)\)\. This line of work improves*solving*but leaves task generation open\.

### LLM Self\-Evolving\.

Self\-evolving training lets models generate their own curricula by bootstrapping supervision through synthesized instructions\([28](https://arxiv.org/html/2608.21871#bib.bib11)\)or outcome\-filtered rationales\([32](https://arxiv.org/html/2608.21871#bib.bib12)\), or by framing training as a game between a proposer and a solver\([15](https://arxiv.org/html/2608.21871#bib.bib14);[33](https://arxiv.org/html/2608.21871#bib.bib15)\)\. Most closely related, R\-Zero co\-evolves an LLM challenger that authors problem text, keeping candidates only when solve rates fall within a fixed hand\-set band\([11](https://arxiv.org/html/2608.21871#bib.bib13)\), which costs a probe of solver rollouts per candidate\. Separately, outcome\-only group\-relative training provides no learning signal when an entire group fails, known as the zero\-reward barrier\([17](https://arxiv.org/html/2608.21871#bib.bib16)\)\. Beyond language models, adaptive task generation at the learner’s frontier appears in asymmetric self\-play\([25](https://arxiv.org/html/2608.21871#bib.bib28)\), goal generators targeting intermediate success rates\([5](https://arxiv.org/html/2608.21871#bib.bib29)\), and regret\-based environment design\([3](https://arxiv.org/html/2608.21871#bib.bib30)\)\. Our setting instead considers natural\-language tasks generated by an LLM proposer and supervised solely by a programmatic verifier\. Building on the pursuit\-evasion framing\([1](https://arxiv.org/html/2608.21871#bib.bib17)\),LUREtrains the evader with a capture\-frontier reward and the pursuer with dense verifier progress, using the same programmatic signal for both roles\.

## Methodology

![Refer to caption](https://arxiv.org/html/2608.21871v1/overview.png)Figure 2:Overview of LURE\.*Tier 1:*the evader selects a difficultyddwith a penalty for repeated signatures, the generator instantiates the task, the planner\-executor pursuer runsGGmulti\-turn rollouts, and the verifier returns captureRRand stepwise progressϕ\\phi\.*Tier 2a:*the evader optimizesmin⁡\(p,1−p\)\\min\(p,1\{\-\}p\), peaking at the capture frontierp=1/2p=1/2\.*Tier 2b:*the pursuer receives task\-wise group\-normalized credit fromRRandΔ​ϕ\\Delta\\phi\.*Tier 3:*a round\-anchored KL stabilizes all three policies\.### Problem Formulation\.

We consider a family of verifiable language reasoning environmentsℰ\\mathcal\{E\}\. Each environmenteeprovides a programmatic task generatorge​\(d,ω\)g\_\{e\}\(d,\\omega\)indexed by a difficulty parameterd∈\[0,1\]d\\in\[0,1\], a multi\-turn interaction protocol, and a verifier that emits two signals per trajectoryτ\\tau: a binary terminal*capture*R⁡\(τ\)∈\{0,1\}R\(\\tau\)\\in\\\{0,1\\\}, which fires only when the task is solved in full, and a running progressϕt∈\[0,1\]\\phi\_\{t\}\\in\[0,1\], the best verified progress so far, withϕ=1\\phi=1at capture\. We writeΔ​ϕt\\Delta\\phi\_\{t\}for the nonnegative increment in best verified progress\. We cast self\-play as a two\-player pursuit\-evasion interaction: the*evader*πC\\pi\_\{C\}selects where on each environment’s difficulty axis to place the next tasks, which the generator instantiates, and the*pursuer*πS=\(πplan,πexec\)\\pi\_\{S\}=\(\\pi\_\{\\text\{plan\}\},\\pi\_\{\\text\{exec\}\}\)is a planner\-executor pair that hunts each task down, with empirical capture rate on taskxxoverGGrollouts

p\(x\)=1G∑g=1G\[R\(τg\)=1\]\.p\(x\)\\;=\\;\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}\\mathds\{1\}\\\!\\left\[R\(\\tau\_\{g\}\)=1\\right\]\.\(1\)The two players optimize a coupled pair of objectives, stated as a pair rather than a minimax and carrying no equilibrium or convergence claim:

maxπC\\displaystyle\\max\_\{\\pi\_\{C\}\}\\;\\;𝔼x​\[min⁡\(p⁡\(x\),1−p⁡\(x\)\)−ρ⁡\(x\)\],\\displaystyle\\mathbb\{E\}\_\{x\}\\\!\\left\[\\min\\bigl\(p\(x\),\\,1\-p\(x\)\\bigr\)\-\\rho\(x\)\\right\],maxπS\\displaystyle\\max\_\{\\pi\_\{S\}\}\\;\\;𝔼x∼πC𝔼τ\[R\(τ\)\+λ∑t\(Δϕt−cr​e​d1\[Δϕt=0\]\)\]\.\\displaystyle\\mathbb\{E\}\_\{x\\sim\\pi\_\{C\}\}\\,\\mathbb\{E\}\_\{\\tau\}\\\!\\left\[R\(\\tau\)\+\\lambda\\sum\_\{t\}\\bigl\(\\Delta\\phi\_\{t\}\-c\_\{red\}\\,\\mathds\{1\}\[\\Delta\\phi\_\{t\}\{=\}0\]\\bigr\)\\right\]\.\(2\)Both objectives are functions of the verifier alone, with no human\-curated tasks, labels, or reward models\.

### Overview\.

Fig\.[2](https://arxiv.org/html/2608.21871#Sx3.F2)shows the workflow ofLURE\. A single round of the chase couples three co\-evolving policies, the evader, the planner, and the executor, instantiated as three separate models\. Each round opens by freezing a snapshot of all three, and those snapshots generate every sample the round collects\. First, the evader emitsKKcandidate tasks per environment, each placed at a target difficulty, and the well\-formed ones become the round’s curriculum\. Second, the pursuer attacks each task withGGplan\-execute rollouts while the verifier records, per trajectory, the terminal captureRRand the per\-step progress\{ϕt\}\\\{\\phi\_\{t\}\\\}\. Third, capture statistics reposition the evader near the pursuer’s frontier, capture and progress increments credit the executor, and the progress its plans realize credits the planner\. Finally, each policy takes a single GRPO update on this round’s data\. Because every trajectory was generated by the round\-start snapshots, the round is simultaneous\-move, and each player faces the others’ updates only in the next round\.

### Frontier\-Seeking Evader\.

The evader learns*where to run*rather than what the tasks contain, repositioning along the difficulty axis against the pursuer’s measured competence\. Each emitted taskxxreceives the capture\-frontier reward

rC​\(x\)=\{min⁡\(p⁡\(x\),1−p⁡\(x\)\)−ρ⁡\(x\),x∈𝒱,−1−ρ⁡\(x\),x∉𝒱,r\_\{C\}\(x\)=\\begin\{cases\}\\min\\bigl\(p\(x\),\\,1\-p\(x\)\\bigr\)\-\\rho\(x\),&x\\in\\mathcal\{V\},\\\\\[2\.0pt\] \-1\-\\rho\(x\),&x\\notin\\mathcal\{V\},\\end\{cases\}\(3\)where𝒱\\mathcal\{V\}is the set of well\-formed emissions the pursuer attacked, and the repetition penaltyρ⁡\(x\)=\(ns​i​g​\(x\)−1\)/N\\rho\(x\)=\(n\_\{sig\}\(x\)\-1\)/Ngrows with the number of the round’sNNemissions that sharexx’s signature, so the evader cannot collapse onto one signature\. Signatures are coarse and per\-environment: hop count and population for multi\-hop question answering, constraint count for instruction following, grid dimensions for logic\-grid deduction\. The reward peaks atp=12p=\\tfrac\{1\}\{2\}, so the evader’s optimum is to remain*barely catchable*, neither escaping \(p=0p\{=\}0\) nor trivially caught \(p=1p\{=\}1\)\. This is also where the capture signal is most informative, since binary capture gives the terminal reward a within\-group variance ofp⁡\(1−p\)p\(1\-p\), and a task off the frontier returns near\-identical outcomes across its group and a vanishing group\-normalized advantage\. Rejection filtering, by contrast, discards finished candidates but never steers where the next ones are placed\. Evader advantages are group\-normalized over the samples sharing a prompt slot\. The generation prompt also carries an initial difficulty hint and a per\-difficulty summary of pursuer competence, so this scaffolding shapes the frontier behavior alongside the reward\.

### Dense\-Credit Pursuer\.

Terminal capture is sparse: on hard self\-generated tasks, whole rollout groups fail and a purely terminal group\-relative signal vanishes\. The pursuer’s dense credit therefore comes from the verifier’s own step progress\. For each executor turnttof trajectoryτ\\tauwe form a single raw scalar

rt=R\(τ\)\[t=Tτ\]\+λ\(Δϕt−cr​e​d\[Δϕt=0\]\),r\_\{t\}\\;=\\;R\(\\tau\)\\,\\mathds\{1\}\\\!\\left\[t=T\_\{\\tau\}\\right\]\\;\+\\;\\lambda\\left\(\\Delta\\phi\_\{t\}\-c\_\{red\}\\,\\mathds\{1\}\\\!\\left\[\\Delta\\phi\_\{t\}=0\\right\]\\right\),\(4\)and standardize these scalars jointly over the group𝒢⁡\(x\)\\mathcal\{G\}\(x\)of all executor turns across theGGrollouts ofxx:

At=z𝒢⁡\(x\)​\(rt\)\.A\_\{t\}\\;=\\;z\_\{\\mathcal\{G\}\(x\)\}\\\!\\left\(r\_\{t\}\\right\)\.\(5\)Three design choices matter\. First, the terminal reward is credited once, on the last executor turn, since replicating it across turns would bias the group statistics toward long trajectories\. Second, the zero\-progress cost is charged only whenΔ​ϕt=0\\Delta\\phi\_\{t\}=0: an unconditional constant would be exactly cancelled by the shift invariance of group\-normalization, so the indicator separates stalling from progressing whenever a group contains both\. Third, becauseϕt\\phi\_\{t\}is best\-so\-far,Δ​ϕt≥0\\Delta\\phi\_\{t\}\\geq 0and∑tΔ​ϕt\\sum\_\{t\}\\Delta\\phi\_\{t\}telescopes to the trajectory’s peak verified progress, so a regression receives zero dense credit rather than a penalty\. When an entire group fails, the terminal term is identically zero and the dense term still carries gradient wherever step progress varies within the group\. The planner receives its own credit: each rollout is scored by the stage potentialsΦi\\Phi\_\{i\}of the executor segments it governed,

rplan​\(τ\)=∑iΦi,r^\{\\mathrm\{plan\}\}\(\\tau\)=\\sum\_\{i\}\\Phi\_\{i\},\(6\)group\-normalized over the same task group with no terminal term, so a plan is judged by how far its execution advanced rather than by whether capture occurred\.

### Stabilized Co\-Evolution\.

The chase is non\-stationary by construction\. The task distribution the pursuer faces, and the pursuer the evader probes, both shift from round to round, so an update regularized toward a stale reference is pulled away from the distribution that produced its data\. We therefore formulate the round as a proximal co\-evolution step: the evader, planner, and executor each take a single GRPO update under a KL penalty to their own round\-start snapshot\. That snapshot acts as behavior policy, KL reference, and proximal center, aligning the regularizer with the distribution that generated the samples and limiting within\-round drift\. Without it we observed format and mode collapse\. Re\-anchoring at the next round preserves that control while letting the joint policy trajectory adapt across rounds\.

### Optimization Objective\.

All three roles are trained with the same clipped GRPO objective\([20](https://arxiv.org/html/2608.21871#bib.bib1);[19](https://arxiv.org/html/2608.21871#bib.bib27)\)and round\-anchored KL, differing only in their advantages\. Letπ¯r\\bar\{\\pi\}\_\{r\}denote the round\-start snapshot of rolerrandϱr=πθr​\(y∣s\)/π¯r​\(y∣s\)\\varrho\_\{r\}=\\pi\_\{\\theta\_\{r\}\}\(y\\mid s\)/\\bar\{\\pi\}\_\{r\}\(y\\mid s\)the token\-level ratio to it\. Each role maximizes the objective below:

𝒥r​\(θr\)=\\displaystyle\\mathcal\{J\}\_\{r\}\(\\theta\_\{r\}\)=𝔼⁡\[min⁡\(ϱr​Ar,clip⁡\(ϱr,1−ϵ,1\+ϵ\)​Ar\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\min\\bigl\(\\varrho\_\{r\}A^\{r\},\\,\\mathrm\{clip\}\(\\varrho\_\{r\},1\{\-\}\\epsilon,1\{\+\}\\epsilon\)\\,A^\{r\}\\bigr\)\\right\]\(7\)−βKL\(πθr∥π¯r\),\\displaystyle\-\\beta\\,\\mathrm\{KL\}\\bigl\(\\pi\_\{\\theta\_\{r\}\}\\,\\\|\\,\\bar\{\\pi\}\_\{r\}\\bigr\),forr∈\{C,plan,exec\}r\\in\\\{C,\\mathrm\{plan\},\\mathrm\{exec\}\\\}\. Each role’s advantage is its group\-normalized reward:AC=zslot​\(rC\)A^\{C\}=z\_\{\\text\{slot\}\}\(r\_\{C\}\)for the evader,Atexec=z𝒢⁡\(x\)​\(rt\)A^\{\\mathrm\{exec\}\}\_\{t\}=z\_\{\\mathcal\{G\}\(x\)\}\(r\_\{t\}\)for the executor, andAτplan=z𝒢⁡\(x\)​\(rplan​\(τ\)\)A^\{\\mathrm\{plan\}\}\_\{\\tau\}=z\_\{\\mathcal\{G\}\(x\)\}\\bigl\(r^\{\\mathrm\{plan\}\}\(\\tau\)\\bigr\)for the planner, with every token carrying the advantage of its turn or rollout\.Alg\.[1](https://arxiv.org/html/2608.21871#alg1)shows the workflow\.

Algorithm 1One round ofLURE\.Alg\. 2inApp\. Awrites out every step\.0:policies

πC,πplan,πexec\\pi\_\{C\},\\pi\_\{\\text\{plan\}\},\\pi\_\{\\text\{exec\}\}, environments

ℰ\\mathcal\{E\}, candidates

KK, rollouts

GG
0:updated policies

πC,πplan,πexec\\pi\_\{C\},\\pi\_\{\\text\{plan\}\},\\pi\_\{\\text\{exec\}\}
1:round\-start snapshots

π¯C,π¯plan,π¯exec←πC,πplan,πexec\\bar\{\\pi\}\_\{C\},\\bar\{\\pi\}\_\{\\text\{plan\}\},\\bar\{\\pi\}\_\{\\text\{exec\}\}\\leftarrow\\pi\_\{C\},\\pi\_\{\\text\{plan\}\},\\pi\_\{\\text\{exec\}\}
2:buffers

𝒟C,𝒟plan,𝒟exec←∅\\mathcal\{D\}\_\{C\},\\mathcal\{D\}\_\{\\text\{plan\}\},\\mathcal\{D\}\_\{\\text\{exec\}\}\\leftarrow\\emptyset, task pool

𝒳←∅\\mathcal\{X\}\\leftarrow\\emptyset
3:foreach environment

e∈ℰe\\in\\mathcal\{E\}do

4:

𝒳←𝒳∪\{valid tasks from​K​of​π¯C​through​ge\}\\mathcal\{X\}\\leftarrow\\mathcal\{X\}\\cup\\\{\\text\{valid tasks from \}K\\text\{ of \}\\bar\{\\pi\}\_\{C\}\\text\{ through \}g\_\{e\}\\\}
5:endfor

6:foreach task

x∈𝒳x\\in\\mathcal\{X\}do

7:roll out

GGtrajectories with

\(π¯plan,π¯exec\)\(\\bar\{\\pi\}\_\{\\text\{plan\}\},\\bar\{\\pi\}\_\{\\text\{exec\}\}\), scored by the verifier

8:

p\(x\)←1G∑g=1G𝟙\[R\(τg\)=1\]p\(x\)\\leftarrow\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}\\mathds\{1\}\[R\(\\tau\_\{g\}\)\{=\}1\]
9:add the evader sample for

xxto

𝒟C\\mathcal\{D\}\_\{C\}\(Eq\.[3](https://arxiv.org/html/2608.21871#Sx3.E3)\)

10:add the executor turns to

𝒟exec\\mathcal\{D\}\_\{\\text\{exec\}\}\(Eq\.[4](https://arxiv.org/html/2608.21871#Sx3.E4)\)

11:add the planner turns to

𝒟plan\\mathcal\{D\}\_\{\\text\{plan\}\}\(Eq\.[6](https://arxiv.org/html/2608.21871#Sx3.E6)\)

12:endfor

13:update

πC\\pi\_\{C\}by GRPO on

𝒟C\\mathcal\{D\}\_\{C\}with KL anchor

π¯C\\bar\{\\pi\}\_\{C\}
14:update

πplan\\pi\_\{\\text\{plan\}\}by GRPO on

𝒟plan\\mathcal\{D\}\_\{\\text\{plan\}\}with KL anchor

π¯plan\\bar\{\\pi\}\_\{\\text\{plan\}\}
15:update

πexec\\pi\_\{\\text\{exec\}\}by GRPO on

𝒟exec\\mathcal\{D\}\_\{\\text\{exec\}\}with KL anchor

π¯exec\\bar\{\\pi\}\_\{\\text\{exec\}\}
16:return

πC,πplan,πexec\\pi\_\{C\},\\pi\_\{\\text\{plan\}\},\\pi\_\{\\text\{exec\}\}

Table 1:Main results \(%\) on PhantomWiki, IFEval, and ZebraLogic across two settings\.Bestandsecond bestper backbone\.
### Theoretical Analysis\.

Three exact properties of the design, proved inApp\. D: the evader’s optimum sits precisely where the pursuer’s learning signal peaks, the repetition penalty makes curriculum collapse impossible, and the dense credit can never outrank capture\.

###### Theorem 1\(Frontier signal\)\.

Letppbe a task’s capture probability andA1,…,AGA\_\{1\},\\ldots,A\_\{G\}the group\-normalized advantages of itsG≥2G\\geq 2binary capture outcomes\. Then∑gAg2=\(G−1\)𝟙\[s\(R\)\>0\]\\sum\_\{g\}A\_\{g\}^\{2\}=\(G\-1\)\\mathds\{1\}\[s\(R\)\>0\]pointwise, withs⁡\(R\)s\(R\)the sample standard deviation of the outcomes, so𝔼⁡\[∑gAg2\]=\(G−1\)​\(1−pG−\(1−p\)G\)\\mathbb\{E\}\[\\sum\_\{g\}A\_\{g\}^\{2\}\]=\(G\-1\)\(1\-p^\{G\}\-\(1\-p\)^\{G\}\), uniquely maximized atp=1/2p=1/2, which is also the unique maximizer ofrCr\_\{C\}\.

###### Proposition 2\(No signature collapse\)\.

In the continuous relaxation the maximizer of Eq\.[3](https://arxiv.org/html/2608.21871#Sx3.E3)over signature distributions is unique, withms⋆=1/k\+\(fs−f¯\)/2m^\{\\star\}\_\{s\}=1/k\+\(f\_\{s\}\-\\bar\{f\}\)/2on its supportS⋆S^\{\\star\}of sizekk\. Hencek≥2k\\geq 2and\|ms⋆−ms′⋆\|≤1/4\|m^\{\\star\}\_\{s\}\-m^\{\\star\}\_\{s^\{\\prime\}\}\|\\leq 1/4onS⋆S^\{\\star\}\.

###### Proposition 3\(Capture dominance\)\.

Best\-so\-far progress gives∑tΔ​ϕt=ϕTτ\\sum\_\{t\}\\Delta\\phi\_\{t\}=\\phi\_\{T\_\{\\tau\}\}, so Eq\.[4](https://arxiv.org/html/2608.21871#Sx3.E4)totalsR⁡\(τ\)\+λ​ϕTτ−λ​cr​e​d​Z​\(τ\)R\(\\tau\)\+\\lambda\\phi\_\{T\_\{\\tau\}\}\-\\lambda c\_\{red\}Z\(\\tau\)over a trajectory, withZZits stalled\-turn count\. Ifλ​cr​e​d​\(Tmax−1\)<1\\lambda c\_\{red\}\(T\_\{\\max\}\-1\)<1, every capturing trajectory strictly outranks every non\-capturing one\.

## Experiments

### Implementation Details\.

We used Qwen2\.5\-7B\-Instruct, Llama\-3\.1\-8B\-Instruct, or Gemma\-2\-9B\-it as our backbone models\. All models were trained with verl\-GRPO\([21](https://arxiv.org/html/2608.21871#bib.bib10);[20](https://arxiv.org/html/2608.21871#bib.bib1)\)under FSDP\-2 on 16×\\timesRTX 6000 Pro Blackwell9696GB GPUs forR=8R=8rounds, using no external task data\. Each round emits4848candidate tasks per environment \(88prompt slots×6\\times\\,6samples\) and attacks every surviving task with a group ofG=8G=8pursuer rollouts\.LUREsets the dense\-credit weightλ=0\.25\\lambda=0\.25and zero\-progress costcr​e​d=0\.05c\_\{red\}=0\.05, and runs the stabilized recipe on all three policies: round\-anchored KL withβ=0\.1\\beta=0\.1, learning rate5×10−65\\times 10^\{\-6\}, and a single PPO epoch\. Baselines use the optimizer settings from their own papers\. The held\-out protocol, covering prompts, verifiers, and decoding, is identical across methods, with full configurations, role prompts, and baseline constructions inApp\. A\.

### Benchmarks & Metrics\.

To evaluate multi\-step verifiable reasoning, the setting the curriculum targets, we use three environments that carry their own programmatic verifier and need no human labels:PhantomWiki\([6](https://arxiv.org/html/2608.21871#bib.bib9)\)for relational inference,IFEval\([34](https://arxiv.org/html/2608.21871#bib.bib7)\)for instruction following, andZebraLogic\([14](https://arxiv.org/html/2608.21871#bib.bib8)\)for logical deduction\. Each doubles as a benchmark and is measured on a held\-out split\. PhantomWiki is multi\-hop kinship question answering, scored as chain\-verified exact match onn=120n=120, where every hop must resolve and the first error ends the chain\. IFEval is scored as per\-prompt accuracy on its541541prompts under strict evaluation\. ZebraLogic is scored on the official set of10001000puzzles, as puzzle accuracy, which demands every cell of the grid, and as cell accuracy, with every metric formalized inApp\. B\.

### Main Results\.

We compareLUREwith*Base*\(zero\-shot\),*GRPO*trained on curated tasks,*GRPO\-Zero*trained on a self\-generated pool frozen after the first round, a non\-adaptive curriculum, and*R\-Zero*\([11](https://arxiv.org/html/2608.21871#bib.bib13)\)\.Table[1](https://arxiv.org/html/2608.21871#Sx3.T1)reports results across three backbone families\. Our proposedLUREoutperforms all trained baselines in all three environments\. On PhantomWiki, it exceeds R\-Zero by15\.015\.0points and the frozen\-pool control by an even larger margin\. Since the frozen\-pool control also improves over Base, self\-generated data explains part of the gain, while the residual advantage over post\-hoc band rejection reflects the contribution of adaptive task placement\. On IFEval with the Qwen backbone, Base remains the strongest overall entry, and no self\-play method surpasses the corrected zero\-shot baseline\. On ZebraLogic,LUREachieves smaller leads on both metrics\. The larger separation reported previously resulted from gold\-answer leakage rather than a genuine method difference\.

Figure 3:Hyperparameter sensitivity on Qwen2\.5\-7B unified model\. Labels give each point’s relative drop from its curve’s best\.
### Ablation Study\.

Table[2](https://arxiv.org/html/2608.21871#Sx4.T2)removes one component at a time, with each row’s exact configuration specified inApp\. C, and every removal lowers the mean, so each contributes to the full system\. The magnitudes group by role: the three credit terms are the three smallest degradations, while every curriculum and stabilization ablation costs more than any of them, and the learned challenger is the largest at−10\.6\-10\.6\. The asymmetry follows from what each part controls, since the challenger decides which tasks carry a usable gradient at all, whereas the credit terms only reshape a gradient it has already supplied\. Stabilization patterns with the curriculum rather than with the credit terms, which is what a mechanism protecting the loop from its own non\-stationarity should do\. Since the components interact through one shared verifier signal, these are contributions within the full system rather than independent effects\.

ConfigurationPhantomWikiIFEvalZebraLogicMeanΔ\\DeltaFull65\.8\\mathbf\{65\.8\}61\.2\\mathbf\{61\.2\}16\.916\.948\.0\\mathbf\{48\.0\}—Curriculum*w/o*learned challenger40\.040\.058\.058\.014\.114\.137\.437\.4−10\.6\\mathbf\{\-10\.6\}*w/o*capture\-frontier reward53\.353\.358\.058\.015\.415\.442\.242\.2−5\.8\-5\.8*w/o*repetition penalty55\.855\.856\.956\.914\.614\.642\.442\.4−5\.6\-5\.6Pursuer credit*w/o*dense credit57\.557\.557\.957\.915\.115\.143\.543\.5−4\.5\-4\.5*w/o*zero\-progress cost59\.259\.258\.258\.215\.815\.844\.444\.4−3\.6\-3\.6*w/o*planner credit61\.761\.759\.559\.516\.216\.245\.845\.8−2\.2\-2\.2Stabilization*w/o*round\-anchored KL55\.055\.056\.756\.713\.413\.441\.741\.7−6\.3\-6\.3*w/o*stabilized recipe52\.552\.555\.555\.512\.912\.940\.340\.3−7\.7\-7\.7

Table 2:Ablation on the Qwen2\.5\-7B\-Instruct unified model\.Δ\\Deltais the change in mean against the full system\.
### Hyperparameter Sensitivity\.

Fig\.[3](https://arxiv.org/html/2608.21871#Sx4.F3)evaluates the six method\-specific hyperparameters: the dense\-credit weightλ\\lambda, zero\-progress costcredc\_\{\\mathrm\{red\}\}, round\-anchored KL coefficientβ\\beta, rollout group sizeGG, candidate budgetKK, and the repetition\-penalty scale, the coefficient onρ⁡\(x\)\\rho\(x\)that Eq\.[3](https://arxiv.org/html/2608.21871#Sx3.E3)fixes to one\. Performance is most sensitive toβ\\betaandGG\. Removing the round\-anchored KL substantially reduces accuracy, while an excessively largeβ\\betaalso degrades performance, indicating a trade\-off between stabilizing the co\-evolving policies and preserving sufficient update flexibility\. ReducingGGsimilarly causes a marked decline\. Because the capture rate is estimated fromGGrollouts, a small group yields a coarse estimate of the capture frontier and provides less reliable supervision for task selection\. The remaining four parameters are comparatively stable around the selected settings, although extreme values can still reduce performance\.

### Evaluation Integrity\.

Table[3](https://arxiv.org/html/2608.21871#Sx4.T3)examines whether the apparent baseline advantage on ZebraLogic reflects stronger reasoning or evaluation leakage\. Panel \(a\) disentangles access to the gold grid from the executor grammar\. The reference condition reproduces the original result, while removing the gold grid reduces performance by6262points and changing the grammar moves it by at most5\.75\.7\. The contrast locates the apparent advantage in prompt leakage rather than in the executor format or stronger reasoning\. Panel \(b\) applies the corrected evaluation protocol to PhantomWiki, where all trained methods improve and our method obtains the largest gain, further widening its margin\. Because the leakage originates from a role\-specific prompt rather than the benchmark answer key, reliable multi\-role self\-play evaluation requires auditing all role\-conditioned prompts\.App\. Blists the redacted fields\.

Gold*Kept*Gold*Removed*Δ\\Delta\(a\) ZebraLogic \(n=300n\{=\}300Subset\), R\-Zero Round\-5 CheckpointOld Grammar68\.068\.06\.06\.0−62\.0\-62\.0New Grammar62\.362\.36\.76\.7−55\.6\-55\.6\(b\) PhantomWiki, Same Leak, Every MethodBase25\.825\.825\.025\.0−0\.8\-0\.8GRPO\-Zero35\.835\.837\.537\.5\+1\.7\+1\.7R\-Zero49\.249\.250\.850\.8\+1\.6\+1\.6LURE \(Ours\)60\.860\.865\.8\\mathbf\{65\.8\}\+5\.0\\mathbf\{\+5\.0\}

Table 3:Evaluation integrity \(%\)\. Both panels contrast the same two conditions on the same tasks\.Figure 4:Anatomy of a chase on PhantomWiki per round\.
### Curriculum Dynamics\.

Fig\.[4](https://arxiv.org/html/2608.21871#Sx4.F4)shows howLUREadapts task difficulty to the evolving pursuer on PhantomWiki\. At round 1, the capture rate is well below the target frontier ofp=0\.5p=0\.5, indicating that the emitted tasks are initially too difficult\.LUREsubsequently lowers the mean difficulty, bringing the capture rate toward the frontier while held\-out success increases from approximately27%27\\%to65\.8%65\.8\\%\. R\-Zero instead maintains an approximately fixed emitted difficulty and achieves lower final capture and held\-out performance\. Thus,LUREmaintains tasks near a fixed relative difficulty with respect to the current pursuer, rather than increasing difficulty according to a predefined schedule\.

### Curriculum Cost\.

Table[4](https://arxiv.org/html/2608.21871#Sx4.T4)compares the rollout costs of post\-hoc filtering and adaptive task placement\. Both methods emit144144candidates per round\. R\-Zero spends11521152probe rollouts to retain3636tasks and discards the remaining108108before training, whereasLUREeliminates probing and trains on all144144tasks\. Consequently,LUREtrains on four times as many tasks at one\-fifth the rollout cost per task, while reducing the total per\-round cost from14401440to11521152rollouts\.

Table 4:Curriculum cost\. Per\-round rollouts from candidate generation to training\.
### Scaling Behavior\.

Fig\.[5](https://arxiv.org/html/2608.21871#Sx4.F5)evaluates whetherLURE’s advantage persists as the backbone scales from77B to7272B based on Qwen2\.5\-Instruct series\. The three environments exhibit distinct scaling patterns\. On PhantomWiki, all methods show diminishing gains at larger scales, whileLUREmaintains a substantial margin over the trained baselines\. On IFEval, the methods converge toward a similar performance range: Base performs best at77B, whereasLUREsurpasses it from1414B onward\. On ZebraLogic,LURE’s margin increases

Figure 5:Scaling behavior\(%\) across Qwen2\.5\-Instruct backbones from77B to7272B\. Filled markers are measured and hollow markers are projections of the power\-law fit inApp\. C, whose exponentα\\alphais shown forLURE\.
### OOD Generalization\.

We further evaluate OOD generalization of the unified model on nine benchmarks across competition mathematics \(MATH500\([10](https://arxiv.org/html/2608.21871#bib.bib21);[13](https://arxiv.org/html/2608.21871#bib.bib3)\),AIME 2024,AIME 2025\([2](https://arxiv.org/html/2608.21871#bib.bib31)\), andOlympiadBench\([9](https://arxiv.org/html/2608.21871#bib.bib22)\)\), rule\-based and general reasoning \(KOR\-Bench\([16](https://arxiv.org/html/2608.21871#bib.bib23)\),LiveBench\-Reasoning, andLiveBench\-IF\([30](https://arxiv.org/html/2608.21871#bib.bib24)\)\), and relational or spatial inference \(CLUTRR\([24](https://arxiv.org/html/2608.21871#bib.bib25)\)andStepGame\([22](https://arxiv.org/html/2608.21871#bib.bib26)\)\)\.Table[5](https://arxiv.org/html/2608.21871#Sx4.T5)reports results under a raw single\-shot prompt without the planner or executor role, isolating the capabilities retained by the trained backbone, with the per\-benchmark protocol inApp\. B\. No method shows consistent gains over Base across individual benchmarks, indicating that the in\-domain improvements do not transfer uniformly\. At the aggregate level, however,LUREis the only trained method to improve over Base, increasing the overall mean by0\.60\.6points, whereas GRPO and GRPO\-Zero decrease by0\.10\.1and1\.01\.0points and R\-Zero matches Base\. OurLUREalso achieves the highest aggregate score in all three benchmark families, indicating stronger OOD retention despite modest and benchmark\-dependent gains\.

Table 5:OOD generalization \(%\) with a raw single\-shot prompt and no planner or executor role, on the Qwen2\.5\-7B\-Instruct unified model\.Bestandsecond bestper row\.

## Conclusion

We proposeLURE, which recasts zero\-data self\-play as a pursuit\-evasion game between a challenger that places tasks and a planner\-executor solver that hunts them down\. The challenger is trained on a capture\-frontier reward that peaks at a capture rate of one half, while the solver receives capture\-anchored dense process credit under a round\-anchored KL that holds both stable\. Across three verifiable reasoning environments and backbone families,LUREoutperforms baselines under unified and specialist settings and attains stronger aggregate OOD zero\-shot accuracy across nine benchmarks, though the gains remain modest and benchmark\-dependent\.

## References

- Bonato \(2011\)A\. BonatoThe game of cops and robbers on graphs\.American Mathematical Soc\.\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p4.1),[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Dekonincket al\.\(2026\)J\. Dekoninck, N\. Jovanović, T\. Gehrunger, K\. Rögnvaldsson, I\. Petrov, C\. Sun, and M\. VechevBeyond benchmarks: matharena as an evaluation platform for mathematics with llms\.arXiv preprint arXiv:2605\.00674\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Denniset al\.\(2020\)M\. Dennis, N\. Jaques, E\. Vinitsky, A\. Bayen, S\. Russell, A\. Critch, and S\. LevineEmergent complexity and zero\-shot transfer via unsupervised environment design\.Advances in neural information processing systems33,pp\. 13049–13061\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Fanget al\.\(2026\)W\. Fang, S\. Liu, Y\. Zhou, K\. Zhang, T\. Zheng, K\. Chen, M\. Song, and D\. TaoSerl: self\-play reinforcement learning for large language models with limited data\.Advances in Neural Information Processing Systems38,pp\. 103706–103738\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1)\.
- Florensaet al\.\(2017\)C\. Florensa, D\. Held, X\. Geng, and P\. AbbeelAutomatic goal generation for reinforcement learning agents\.arXiv preprint arXiv:1705\.06366\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Gonget al\.\(2025\)A\. Gong, C\. Wan, K\. Stankevičiūtė, A\. Kabra, R\. Thesmar, J\. Lee, J\. Klenke, C\. P\. Gomes, and K\. Q\. WeinbergerPhantomwiki: on\-demand datasets for reasoning and retrieval evaluation\.InICML 2025 Workshop on Long\-Context Foundation Models,Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[Benchmarks & Metrics\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.10.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Huanget al\.\(2025\)C\. Huang, W\. Yu, X\. Wang, H\. Zhang, Z\. Li, R\. Li, J\. Huang, H\. Mi, and D\. YuR\-zero: self\-evolving reasoning llm from zero data\.arXiv preprint arXiv:2508\.05004\.Cited by:[Figure 1](https://arxiv.org/html/2608.21871#Sx1.F1),[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.21871#Sx1.p2.1),[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.14.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.20.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.8.1),[Main Results\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px3.p1.1)\.
- Lambertet al\.\(2024\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 39578–39601\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p3.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Linet al\.\(2025\)B\. Y\. Lin, R\. Le Bras, K\. Richardson, A\. Sabharwal, R\. Poovendran, P\. Clark, and Y\. ChoiZebralogic: on the scaling limits of llms for logical reasoning\.InForty\-second International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[Benchmarks & Metrics\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2025\)B\. Liu, L\. Guertler, S\. Yu, Z\. Liu, P\. Qi, D\. Balcells, M\. Liu, C\. Tan, W\. Shi, M\. Lin,et al\.Spiral: self\-play on zero\-sum games incentivizes reasoning via multi\-agent multi\-turn reinforcement learning\.arXiv preprint arXiv:2506\.24119\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Maet al\.\(2025\)K\. Ma, X\. Du, Y\. Wang, H\. Zhang, Z\. Wen, X\. Qu, J\. Yang, J\. Liu, M\. Liu, X\. Yue,et al\.Kor\-bench: benchmarking language models on knowledge\-orthogonal reasoning tasks\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 80062–80161\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Prakash and Buvanesh \(2025\)J\. Prakash and A\. BuvaneshWhat can you do when you have zero rewards during rl?\.arXiv preprint arXiv:2510\.03971\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p3.1),[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Qwenet al\.\(2025\)Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.4.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[Optimization Objective\.](https://arxiv.org/html/2608.21871#Sx3.SS0.SSS0.Px6.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[Optimization Objective\.](https://arxiv.org/html/2608.21871#Sx3.SS0.SSS0.Px6.p1.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.12.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.18.1),[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.6.1),[Implementation Details\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px1.p1.1)\.
- Shenget al\.\(2024\)G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. WuHybridflow: a flexible and efficient rlhf framework\.arXiv preprint arXiv:2409\.19256\.Cited by:[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[Implementation Details\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px1.p1.1)\.
- Shiet al\.\(2022\)Z\. Shi, Q\. Zhang, and A\. LipaniStepgame: a new benchmark for robust multi\-hop spatial reasoning in texts\.InProceedings of the AAAI conference on artificial intelligence,Vol\.36,pp\. 11321–11329\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.Advances in neural information processing systems36,pp\. 8634–8652\.Cited by:[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1)\.
- Sinhaet al\.\(2019\)K\. Sinha, S\. Sodhani, J\. Dong, J\. Pineau, and W\. L\. HamiltonCLUTRR: a diagnostic benchmark for inductive reasoning from text\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 4506–4515\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Sukhbaataret al\.\(2017\)S\. Sukhbaatar, Z\. Lin, I\. Kostrikov, G\. Synnaeve, A\. Szlam, and R\. FergusIntrinsic motivation and automatic curricula via asymmetric self\-play\.arXiv preprint arXiv:1703\.05407\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Teamet al\.\(2024\)G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[Table 1](https://arxiv.org/html/2608.21871#Sx3.T1.1.1.16.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. SuiMath\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p3.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1)\.
- Whiteet al\.\(2024\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Dey,et al\.Livebench: a challenging, contamination\-limited llm benchmark\.arXiv preprint arXiv:2406\.19314\.Cited by:[OOD Generalization\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px10.p1.1)\.
- Yaoet al\.\(2022\)S\. Yao, J\. Zhao, D\. Yu, I\. Shafran, K\. R\. Narasimhan, and Y\. CaoReact: synergizing reasoning and acting in language models\.InNeurIPS 2022 Foundation Models for Decision Making Workshop,Cited by:[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. GoodmanStar: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026\)A\. Zhao, Y\. Wu, T\. Wu, Q\. Xu, Y\. Yue, M\. Lin, S\. Wang, Q\. Wu, Z\. Zheng, and G\. HuangAbsolute zero: reinforced self\-play reasoning with zero data\.Advances in Neural Information Processing Systems38,pp\. 105816–105879\.Cited by:[Figure 1](https://arxiv.org/html/2608.21871#Sx1.F1),[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.21871#Sx1.p2.1),[LLM Self\-Evolving\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[Introduction](https://arxiv.org/html/2608.21871#Sx1.p1.1),[Verifiable Multi\-Step Reasoning\.](https://arxiv.org/html/2608.21871#Sx2.SS0.SSS0.Px1.p1.1),[Benchmarks & Metrics\.](https://arxiv.org/html/2608.21871#Sx4.SS0.SSS0.Px2.p1.1)\.