Reasoning doesn’t have to mean longer chains of thought.

PDR = draft in parallel → distill into a compact workspace → refine, and shift the Pareto frontier.

https://t.co/4Sca6dFu4Q https://t.co/AHwyg53xd7

Rethinking Thinking Tokens: LLMs as Improvement Operators

Source: https://arxiv.org/html/2510.01123 1]Meta Superintelligence Labs 2]University College London 3]Mila, University of Montreal 4]Anthropic 5]Princeton University\contribution[*]Work done at Meta

Aniket DidolkarSuchin GururanganJohn QuanRuan SilvaRuslan SalakhutdinovManzil ZaheerSanjeev AroraAnirudh Goyal[[[[[

Abstract

Reasoning training incentivizes LLMs to produce long chains of thought (long CoT), which among other things, allows them to explore solution strategies with self-checking. This results in higher accuracy, but inflates context length, token/compute cost, and answer latency. We ask:Can current models leverage their metacognition to provide other combinations on this Pareto frontier, e.g., better accuracy with lower context length and/or latency?Abstractly, we view the model as animprovement operatoron its own “thoughts” with a continuum of possible strategies. We study an inference familyParallel-Distill-Refine (\PDR), which performs the following: (i) generate diverse drafts in parallel; (ii)distillthem into a bounded, textual workspace; and (iii)refineconditioned on this workspace, producing an output that seeds the next round. Importantly, context length (hence compute cost) is controllable via degree of parallelism, and is no longer conflated with the total number of generated tokens. We report\PDRinstantiations of current models that give better accuracy than long CoT while incurring lower latency. Setting degree of parallelism to11yields a subcaseSequential Refinement (\SR)(iteratively improve a single candidate answer) which provides performance superior to long CoT (at the cost of higher latency). Success of such model orchestrations raises the question whether further training could shift the Pareto frontier. To this end, we train an8​B8Bthinking model with Reinforcement Learning (RL) to make it consistent with\PDRas the inference method. On math tasks with verifiable answers, iterative pipelines surpass single-pass baselines at matched sequential budgets, with\PDRdelivering the largest gains (+11% on AIME 2024 and +9% on AIME 2025).

1Introduction

Figure 1:AIME 2024: Accuracy vs. sequential budgetBseqB_{\mathrm{seq}}. We compare Long CoT,\SR, and\PDR; the dashed curve is an oracle upper bound (Oracle-\PDR) that perfectly transmits any correct solutions under the same budgets to the compact workspace.Scaling language models to solve harder problems has increasingly relied on eliciting explicit reasoning traces (“thinking tokens”) at inference time(Wei et al.,2022; Jaech et al.,2024; Guo et al.,2025). While longer traces often correlate with accuracy, they entangle reasoning depth with sequence length and inherit long-context failure modes(Ghosal et al.,2025). In parallel, the field is gravitating towardsself-improvement: systems that refine their own outputs viaself-directed operations(critique, revision, debate, sample-and-select) without expert supervision(Gou et al.,2023; Du et al.,2023b; Irving et al.,2018; Yao et al.,2023; Pan et al.,2025; Zhang et al.,2025).

Stepping back from the rich body of work, one begins to see LLM inference as a malleable concept; instead of a single “reasoning trace” one encounters choices to be made from a larger pool: generate fresh answers; critique/revise/debate/summarize generated answers; create an updated answer. With this choice comes an unexplored Pareto-frontier:

What is the best possible task accuracy achievable after fixing constraints on the inference process, e.g.: (i) total tokens across all generations, (ii) max depth of the generation chain (“latency”), (iii) total context length, and (iv) total compute (which depends on all of the above in complicated and system-dependent ways).

Figure 2:(a)Parallel-Distill-Refine (\PDR). In roundrr, the model generatesMrM_{r}parallel drafts, then distills them into a compact workspace using one of the schemes in (b); the refined state seeds the next round. (b) Distillation schemes used to build the workspace (e.g., global summary, shared top-kk, per-sample top-kk, random-kk). (c) Three inference regimes. Top-Long chain-of-thought: a single, long trace. Middle-Sequential Refinement (SR): one draft updated over short rounds. Bottom-\PDR: each round spawnsMrM_{r}drafts, distills into a workspace, and refines. The example shows a 3-round configurationM=(8,4,1)M=(8,4,1)(configuration is a hyperparameter, and any other choice is possible). Across panels, the per-callsequential budgetBseqB_{\mathrm{seq}}(latency proxy) is held fixed, while\PDRincreasestotal computeBtotalB_{\mathrm{total}}via parallelism without increasing per-call context.The confounding factor is that iteration alone does not guarantee progress. Simply asking the model to “try again” risks forgetting useful partial results and repeating earlier mistakes. Naïvely appending all prior attempts to the context recreates long-context failures and scales cost with the number of rounds. Current models suffer from anchoring biases (see Figure6,8) as well as forgetfulness. A viable scheme needs a compact state that (i) carries forward salient facts and intermediate results, (ii) flags disagreements and open subgoals, and (iii) remains bounded so each generation (and overall context-size) stays short.

This paper studies inference strategies that generate many tokens with a compact context size. Instead of long chains of thought, inference has phases that generate solutions within the allowed context/token budget and then write a bounded, round-wise summary/report (e.g., listing agreements, contradictions, intermediate results, and open subgoals). The next phase starts with only this summary and uses available workspace for fresh generations (which benefit from accumulated wisdom in the summary). Iterating this process can generate long “thinking” albeit with a bounded context size.111Such LLMs fall within a traditional framework ofrandomized space-bounded computation. Computational complexity theoryArora & Barak (2007)shows it is capable of surprisingly powerful reasoning, such as determining connectivity of much larger graphs that cannot even fit the working memory; see SectionC.

We study two inference instantiations: (i)Sequential refinement (\SR), where a single artifact (solution, proof, program) is iteratively improved for a fixed number of steps; and (ii)Parallel-Distill-Refine (\PDR)(round-wise workspace), where each round samplesMMdrafts in parallel, distills them into a bounded summary for the next round, and continues. The workspace is not persistent across rounds; it is freshly synthesized for each round. A central challenge is information synthesis: compressing salient facts and intermediate results; flagging uncertainty; and retiring stale information. Its effectiveness hinges on four meta-skills:verification(detect and localize errors via self-judging and cross-candidate checks),refinement(use feedback/context to improve the artifact),compression(retain only past history via bounded summaries rather than replay), anddiversification(exploratory variation to avoid consensus collapse).

Learning to improve short-context iteration.It is also of interest to teach the model a policy that effectively leverages this improvement operator. Standard RL training for reasoning models typically optimizes a single, long chain-of-thought conditioned on the prompt, with reward on the final answer(Shao et al.,2024; Guo et al.,2025).\PDR, by contrast, comprises multiple short iterations that read a bounded summary, write a refinement, and re-synthesize a fresh summary. This creates a train-test mismatch in the information flow (short updates vs. one long trace). To make sure training is consistent with deployment, we optimize an objective that unrolls the operator itself during training: sampleMMshort drafts, distill them into a compact summary, and condition on the prompt plus that summary to produce a refined attempt. We use verifiable rewards to supervise the end-to-end computation. This objective narrows the train–test gap.

Results.On math tasks, iterative pipelines surpass single-pass baselines at matched sequential budgets; with shallow\PDRdelivering the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025). Making training consistent with inference, via an operator-consistent RL objective that optimizes the same read/write interface used at test time yields further improvements (e.g.,∼5%\sim 5\%on AIME 2024 and AIME 2025 when mixing standard and operator RL). These findings suggest that iteration with short contexts and compact summaries can substitute for long traces while holding latency fixed.

2Background & Related Work

Overview.This section situates our work among several threads that seek to scale test-time reasoning: long-trace chains of thought and self-consistency; self-improvement and debate; structured search (trees/graphs of thoughts); multi-trace selection and aggregation; turning compute into supervision; memory/summarization; adaptive multi-turn RL; and learning-to-search/planning. We view these through a unified lens:inference as a round-wise improvement operator under explicit budgets; holding the per-callsequentialbudgetBseq\mathrm{B}_{\text{seq}}fixed while varyingtotalcomputeBtotal\mathrm{B}_{\text{total}}. We do not claim the micro-primitives themselves are new: parallel sampling, aggregation and selection(Wang et al.,2023; Fu et al.,2025; Zhao et al.,2025), sequential revision, critique-revise-verify and debate(Madaan et al.,2023; Gou et al.,2023; Shinn et al.,2023; Du et al.,2023b), structured search(Yao et al.,2023; Besta et al.,2024), and summarization/memory(Wu et al.,2025; Yang et al.,2024)are all well explored. Our contribution is to (i) formalize these pieces within a single round-wise operator (\SR,\PDR), (ii) analyze compute-normalized depth via shallow parallel rounds and distillation at matchedBseqB_{\text{seq}}while varyingBtotalB_{\text{total}}, and (iii) make training consistent with deployment viaoperator-consistent RL. The text below organizes prior work along these axes and clarifies similarities and differences.

Test-time reasoning with long traces.Eliciting step-by-step “chains of thought” improves accuracy on multi-step tasks(Wei et al.,2022). Recent “reasoning” models (e.g., OpenAIo1, Deepseekr1) explicitly trade more test-time thinking for better results, increasing tokens and latency(Jaech et al.,2024; Guo et al.,2025). Sampling multiple traces and aggregating answers (self-consistency) further boosts performance but scales cost with the number of samples(Wang et al.,2023). Our approach targets a complementary design point: keep each call short while letting evidence accumulate across rounds via a bounded, re-synthesized summary.

Self-improvement.A growing line of work lets models critique and refine their own outputs:Self-Refinealternates self-feedback and revision(Madaan et al.,2023);Reflexionmaintains textual memory to guide subsequent attempts(Shinn et al.,2023);CRITICverifies with tools and revises accordingly(Gou et al.,2023). Multi-agent debate improves factuality and reasoning via argumentation and adversarial checking(Irving et al.,2018; Du et al.,2023b). Our operator shares the self-improvement spirit but constrains per-call context by using a round-wise, re-synthesized compact stateC(r)C^{(r)}instead of replaying full histories.Compute as Teacher(CaT) synthesizes a single reference from parallel rollouts and optimizes the policy toward it, converting extra inference compute into reference-free supervision. Our focus differs: we use parallelism within\PDRto explore and distill at inference time, and analyze compute explicitly under fixedBseq\mathrm{B}_{\text{seq}}while varyingBtotal\mathrm{B}_{\text{total}}.

Multi-trace selection and aggregation.Confidence-aware test-time scaling generates multiple traces in parallel and filters or re-weights them with model-internal confidence, improving the accuracy–compute trade-off without extra training(Fu et al.,2025). Another line trains anaggregatorto select or combine solutions using RL rather than majority vote or reward-model ranking(Zhao et al.,2025). In contrast, we cast inference itself asround-wise operators(\SR,\PDR), with aggregation as one of the meta-cognitive abilities necessary to improve the performance, and propose\PDRRL to reduce the train-inference mismatch.Zheng et al. (2025)proposeParallel-R1, an RL framework that instills parallel thinking through a progressive curriculum: supervised fine-tuning on easier prompts to bootstrap the behavior, followed by RL on harder problems. On math benchmarks (MATH, AMC23, AIME), Parallel-R1 improves over a sequential-thinking RL baseline and shows that parallel thinking functions as a mid-training exploration scaffold before aiding verification in later stages.

Structured search beyond single chains.Prompting schemes structure exploration explicitly:Tree of Thoughts(ToT) expands and evaluates branches of reasoning(Yao et al.,2023);Graph of Thoughtsgeneralizes to arbitrary thought graphs(Besta et al.,2024);Least-to-Mostdecomposes problems into subproblems solved in sequence(Zhou et al.,2022). These methods typically grow tokens with breadth/depth or rely on long contexts to carry intermediate state. In contrast,\PDRconcentrates exploration within a round, then distills to a boundedC(r)C^{(r)}, preventing unbounded context growth.

Multi-agent debate and compressed debate.Debate-style methods have multiple LLM “agents” propose answers and iteratively read and critique one another, often improving robustness(Du et al.,2023a). OurParallel-Distill-Refine (\PDR)can be seen as acompresseddebate: treat a round’s diverse drafts as agent outputs, but instead of replaying full transcripts, distill them into a boundedC(r)C^{(r)}that conditions the next round. This preserves cross-agent scrutiny while controlling per-call context and bothBseq\mathrm{B}_{\text{seq}}andBtotal\mathrm{B}_{\text{total}}.

Compact summaries vs. persistent memory.Agent systems often add external memory or retrieval to persist context across sessions/tasks; thought buffers and memory-augmented agents exemplify this direction (e.g., Buffer-of-Thoughts)(Yang et al.,2024). We instead usenon-persistent, round-wise summaries. This choice keeps prompts short and reduces long-context failure modes(Liu et al.,2023). Concurrently,Wu et al. (2025)introduce ReSum, a web-agent paradigm that periodically summarizes growing interaction histories into compact states and pairs this with ReSum-GRPO so agents learn summary-conditioned reasoning. Our work differs in focusing on multiple inference-time operators instead of summary (\SR,\PDR), alongside operator-consistent RL aligned to the round-wise interface.Didolkar et al. (2025)introduceMetacognitive Reuse, extracting recurring reasoning into concise, named behaviors that reduce token usage and can be distilled via SFT. This is complementary to our within-instance\PDR: rather than compressing cross-instance procedures, we parallelize and distill drafts per instance.

Adaptive multi-turn training.Unary-feedback multi-turn RL (“try again”) trains models to revise across rounds improving multi-turn accuracy with minimal supervision(Liu et al.,2025a).Exploratory Iteration(ExIt) leverages the recurrent structure of self-improvement by performing multi-step refinement at test time while training emphasizes the most informative single-step iterations(Jiang et al.,2025).\SRis similar to “try again” iterative interface.

RL with on-policy tree search.Hou et al. (2025)introduceTreeRL, which integrates on-policy tree search into RL for LLM reasoning, improving exploration and providing dense, process-level rewards(Xie et al.,2024)compared to independent chain sampling with outcome-only supervision. TreeRL thus exemplifies atraining-time, search-augmentedimprovement operator. In contrast, our operator-consistent RL trains in the same iterative interface used at inference: we unroll a single round (akin to a shallow tree expansion) but optimize with outcome supervision without deep tree search. A promising direction is to incorporate process-level rewards into this operator-consistent setting and study their impact on the test-time performance of round-wise operators (\SR,\PDR).

3LLMs as Improvement Operators

3.1Problem setting and notation

We consider tasksxx(e.g., a math problem) and aim to produce a high-quality final artifactsfinals_{\text{final}}(solution, proof, or program) under a given token budget. Letℳθ\mathcal{M}_{\theta}denote a (frozen or trainable) LLM used as animprovement operator. Given a current artifactsts_{t}(single generation or set of generations) and a compact textual workspaceCtC_{t}, the model proposes a refinement:

st+1←ℳθ​(x,st,Ct).s_{t+1}\leftarrow\mathcal{M}_{\theta}(x,s_{t},C_{t}).(1)The workspaceCtC_{t}is a bounded summary (|Ct|≤κ|C_{t}|\!\leq\!\kappatokens) meant to capture agreements, contradictions, intermediate results, and open subgoals.

Read-write-compress cycle.

Each step (i) reads the current workspaceCtC_{t}, (ii) writes a refined artifactst+1s_{t+1}viaℳθ\mathcal{M}_{\theta}, and (iii) compresses back into a bounded workspace for the next step using a synthesis operator𝒟\mathcal{D}:

Ct+1←𝒟​(x,st+1),|Ct+1|≤κ.C_{t+1}\leftarrow\mathcal{D}(x,s_{t+1}),\qquad|C_{t+1}|\leq\kappa.(2)

Token budgets.

We evaluate every method under two budgets:

Bseq=∑c∈𝒫(inc+outc)\displaystyle\mathrm{B}_{\text{seq}}=\sum_{c\in\mathcal{P}}\bigl(\mathrm{in}_{c}+\mathrm{out}_{c}\bigr)(latency proxy; tokens along the accepted path),\displaystyle\text{(latency proxy; tokens along the accepted path)},(3)Btotal=∑c=1C(inc+outc)\displaystyle\mathrm{B}_{\text{total}}=\sum_{c=1}^{C}\bigl(\mathrm{in}_{c}+\mathrm{out}_{c}\bigr)(compute/cost proxy; all calls, including discarded branches).\displaystyle\text{(compute/cost proxy; all calls, including discarded branches)}.(4) Herec=1,…,Cc=1,\dots,Cindexes all model calls (prompts, candidate generations, and distillation/summary updates);inc\mathrm{in}_{c}andoutc\mathrm{out}_{c}are the input and output tokens for callcc; and𝒫⊆{1,…,C}\mathcal{P}\subseteq\{1,\dots,C\}is the final accepted path. We report accuracy as a function of both axes and match baselines per axis (e.g., equalBseq\mathrm{B}_{\text{seq}}for latency-controlled comparisons).

3.2Operator Instantiations

We study two short-context iterative refinement pipelines.

3.2.1Sequential refinement (\SR; depth over a single candidate).

We setCt≡∅C_{t}\equiv\varnothingfor allttand iteratively improve a single artifact forRRrounds:

st+1←ℳθ​(x,st,∅),t=0,…,R−1,sfinal=sR.s_{t+1}\leftarrow\mathcal{M}_{\theta}(x,s_{t},\varnothing),\quad t=0,\dots,R-1,\qquad s_{\text{final}}=s_{R}.(5) SR with compact workspace.In\SR, no explicit workspace is provided. We also evaluate a variant that inserts an error analysis step between rounds: rather than directly refining the previous answer, the model first identifies and explains flaws in the current solution, then generates a revised solution. These notes act as a transient, local workspace at each round.

3.2.2Parallel-Distill-Refine (\PDR; round-wise workspace)

We do not maintain a persistent memory. Instead, for roundsr=1,…,Rr=1,\dots,R; we sampleMrM_{r}drafts (Parallel) conditioned on the current bounded summary, then re-synthesize (Distill) a fresh bounded summary for the next round:

(Parallel)S(r)={si(r)←ℳθ​(x,C(r−1))}i=1Mr,C(0)=∅,\displaystyle\text{(Parallel)}\quad S^{(r)}=\bigl\{\,s^{(r)}_{i}\leftarrow\mathcal{M}_{\theta}(x,C^{(r-1)})\,\bigr\}_{i=1}^{M_{r}},\qquad C^{(0)}=\varnothing,(6)(Distill)C(r)←𝒟​(x,S(r)),|C(r)|≤κ.\displaystyle\text{(Distill)}\quad C^{(r)}\leftarrow\mathcal{D}\bigl(x,S^{(r)}\bigr),\qquad|C^{(r)}|\leq\kappa.(7)We enforce single generation in last roundMR=1M_{R}=1; which is returned assfinals_{\text{final}}. The summary is round-wise and non-persistent: earlier text is not replayed, preventing growth in per-call context.

Why a round-wise summary?Replay of all prior attempts scales linearly with steps and reintroduces long-context failure modes. Re-synthesizingC(r)C^{(r)}from the current drafts keeps the memorybounded(|C(r)|≤κ\lvert C^{(r)}\rvert\!\leq\!\kappa) and focuses each round on the most recent and informative evidence.

Constructing the compact summaryC(r)C^{(r)}.We consider several practical instantiations of the distillation operator𝒟\mathcal{D}, all obeying|C(r)|≤κ|C^{(r)}|\!\leq\!\kappa:

• •Global summary:Produce a single sharedC(r)C^{(r)}that captures agreements, contradictions, derived facts, unresolved subgoals, and next actions. This emphasizes verification and comparison while retiring stale or contradicted information.
• •Extractive top-kkevidence (shared):Instead of free-form text, select thekksolutions fromS(r)S^{(r)}as the workspace itself, trading compression for higher fidelity to the best evidence.
• •Random-kk/ bootstrapped workspaces:For the next round, construct multiple small workspaces by randomly samplingkksolutions per generation. This injects diversity and mitigates premature consensus while keeping each workspace small.

Budgets.Tokens used forParallel,Distill, andRefinecontribute toBtotal\mathrm{B}_{\text{total}}. The reported latencyBseq\mathrm{B}_{\text{seq}}only counts the tokens on the acceptedgenerate→distill→refine\text{generate}\!\rightarrow\!\text{distill}\!\rightarrow\!\text{refine}path for the final output.

3.3Operator-Consistent Training

The previous sections treatℳθ\mathcal{M}_{\theta}as frozen and rely purely on prompting/orchestration. We now make sure training is consistent with deployment/inference by optimizing the model under the same short-context, iterative interface used at test time.

Motivation.Most RL for reasoning LLMs optimizes a single, long chain-of-thought trajectory. If inference instead runs multiple short passes with a compact workspaceCC, this creates a train-test mismatch. We remove this mismatch by mixing two training modes: (i) standard long-trace optimization, and (ii)operator rolloutsthat execute the generate→\rightarrowdistill→\rightarrowrefine interface under short contexts.

Base Algorithm.For the baseline RL, we use the CISPO objective from Minimax-M1(Li et al.,2025). For a given promptxx, the generatorπ(⋅∣θold)\pi(\cdot\mid\theta_{\text{old}})generatesGGrollouts{oi=1G}\{o_{i=1}^{G}\}using the old policyθold\theta_{\text{old}}. Automated checkers like sympy(Meurer et al.,2017)or math-verify222https://github.com/huggingface/Math-Verifyare used to assign scalar rewardsrir_{i}(±1\pm 1) to each of the rollouts. CISPO combines the group-normalized advantage from GRPO(Shao et al.,2024)with REINFORCE(Williams,1992)to achieve the following objective:

𝒥CISPO​(θ)=𝔼x∼𝒟,{oi}i=1G∼π(⋅∣x;θold)​[1∑i=1G|oi|​∑i=1G∑t=1|oi|sg​(ri,t​(θ))​Ai​log⁡(π​(oi,t|x,oi,<t;θ))]\mathcal{J}_{\text{CISPO}}(\theta)=\underset{x\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi(\cdot\mid x;\theta_{\text{old}})}{\mathbb{E}}\left[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\text{sg}(r_{i,t}(\theta))A_{i}\log(\pi(o_{i,t}|x,o_{i,<t};\theta))\right](8)whereAi=ri−mean​({r}j=1G)std​({r}j=1G)A_{i}=\frac{r_{i}-\text{mean}(\{r\}_{j=1}^{G})}{\text{std}(\{r\}_{j=1}^{G})}is the advantage,sgis the stop-gradient operation, andri,t​(θ)r_{i,t}(\theta)is computed using the asymmetric clipping fromYu et al. (2025)as follows:

ri,t=clip​(π​(oi∣x,oi,<t;θ)π​(oi∣x,oi,<t;θold),1−ϵ−,1+ϵ+)r_{i,t}=\text{clip}\left(\frac{\pi(o_{i}\mid x,o_{i,<t};\theta)}{\pi(o_{i}\mid x,o_{i,<t};\theta_{\text{old}})},1-\epsilon^{-},1+\epsilon^{+}\right)(9) whereπ​(oi∣x,oi,<t;θ)π​(oi∣x,oi,<t;θold)\frac{\pi(o_{i}\mid x,o_{i,<t};\theta)}{\pi(o_{i}\mid x,o_{i,<t};\theta_{\text{old}})}is the importance-sampling (IS) weight. Additionally, we add an SFT loss (negative log-likelihood) similar toSeed et al. (2025)on rollouts which lead to positive rewards. The final training objective becomes:

𝒥​(θ)=𝒥CISPO​(θ)+α⋅𝒥SFT​(θ)\mathcal{J}(\theta)=\mathcal{J}_{\text{CISPO}}(\theta)+\alpha\cdot\mathcal{J}_{\text{SFT}}(\theta)(10) whereα\alphais set to a small value like0.10.1in our experiments. The addition of this SFT loss boosts the utilization of positive rollouts and enforces better verification behavior in model training.

Data mixture.At each update, draw a mini-batchℬ={xi}i=1N\mathcal{B}=\{x_{i}\}_{i=1}^{N}and split it evenly into two sub-batchesℬtrace\mathcal{B}_{\mathrm{trace}}andℬop\mathcal{B}_{\mathrm{op}}with|ℬtrace|=⌊N/2⌋|\mathcal{B}_{\mathrm{trace}}|=\lfloor N/2\rfloorand|ℬop|=⌈N/2⌉|\mathcal{B}_{\mathrm{op}}|=\lceil N/2\rceil. We train onℬtrace\mathcal{B}_{\mathrm{trace}}with a standard long-trace objective𝒥trace​(θ)\mathcal{J}_{\mathrm{trace}}(\theta), and onℬop\mathcal{B}_{\mathrm{op}}withoperator rolloutsunder short context, yielding𝒥op​(θ)\mathcal{J}_{\mathrm{op}}(\theta). The per-step objective is the average of the two:

𝒥train​(θ)=12​𝒥traceℬtrace​(θ)+12​𝒥opℬop​(θ),\mathcal{J}_{\text{train}}(\theta)\;=\;\tfrac{1}{2}\,\mathcal{J}_{\mathrm{trace}}^{\mathcal{B}_{\mathrm{trace}}}(\theta)\;+\;\tfrac{1}{2}\,\mathcal{J}_{\mathrm{op}}^{\mathcal{B}_{\mathrm{op}}}(\theta),(11)where𝒥traceℬtrace\mathcal{J}_{\mathrm{trace}}^{\mathcal{B}_{\mathrm{trace}}}and𝒥opℬop\mathcal{J}_{\mathrm{op}}^{\mathcal{B}_{\mathrm{op}}}denote the losses computed on their respective sub-batches. Other ratios are possible; we use a 1:1 split in our experiments.

Mode A: Standard long-trace optimization.

Givenxx, sample a single, long trajectorys1:T∼ℳθ​(x)s_{1:T}\sim\mathcal{M}_{\theta}(x)and optimize a conventional RL verifiable signal (e.g., a rule based verifiable reward for math problems). This preserves the model’s ability to reason in extended traces when available.

Mode B: Operator rollouts under short context.

We roll out the same interface used at test time but with one round for stability and cost.

(i) Parallel-Distill-Refine (\PDR; one-round rollout).

1. 1.GenerateMMparallel generations (reasoning traces, solutions) conditioned on an empty summary: S={si←ℳθ​(x,C(0)=∅)}i=1M.S=\{\,s_{i}\leftarrow\mathcal{M}_{\theta}(x,C^{(0)}{=}\varnothing)\,\}_{i=1}^{M}.
2. 2.Distill to a bounded, round-wise summaryCC.
3. 3.Refine a single candidate conditioned onCC:s~←ℳθ​(x,sj,C).\tilde{s}\leftarrow\mathcal{M}_{\theta}(x,s_{j},C).

Why one round during training?Rolling out a single\PDRround (withMMearly drafts, distillation toCC, and a single refinement) captures the key interface while controllingBtotal\mathrm{B}_{\text{total}}and stabilizing RL. At inference we can run multiple rounds (R>1R>1) using the same operator.

Our datamix preserves competence on long traces while teaching the model to reason across short iterations.\PDRis emulated by one-round of parallel→\rightarrowdistill→\rightarrowrefine rollout where the model observes(x,C)(x,C)and is optimized with a verifiable reward on the final solution trace.

4Experiments

Figure 3:AIME 2024: Iterative improvement beats single-pass long-CoT at matched sequential budgets.Thexx-axis reportsBseqB_{\mathrm{seq}}: the thinking tokens consumed along the accepted path of the iterative chain, plus any distilled summary that conditions the next step. Tokens spent on unused parallel proposals are excluded, soBseqB_{\mathrm{seq}}serves as a latency proxy. At comparableBseqB_{\mathrm{seq}}, both\SRand\PDRoutperform the single-pass long CoT baseline, with\PDRyielding the largest gains by converting additional total compute (via parallelism) into accuracy without increasing per-call context.In this section, we compare the Sequential refinement (\SR) and Parallel-Distill-Refine (\PDR) operators against long chain-of-thought baselines under a budget-aware protocol. We measure accuracy with symbolic verifiers like sympy(Meurer et al.,2017)and math-verify333https://github.com/huggingface/Math-Verify. Additionally, we report the results as functions of both the sequential budgetBseq\mathrm{B}_{\text{seq}}(latency proxy along the accepted path) and the total budgetBtotal\mathrm{B}_{\text{total}}(all tokens across calls).

Figure 4:Token Budgets comparison:We plot all the different configurations for Long CoT,\SRand\PDRoperators for bothBseqB_{\text{seq}}andBtotalB_{\text{total}}token budgets forgemini-2.5-flash. ForBseqB_{\text{seq}},\PDRforms the Pareto-frontier and gives consistent gains over Long CoT and\SR. However, forBtotalB_{\text{total}},\SRforms the pareto-frontier because there are no parallel drafts involved so no generations are discarded.We try to answer the following four research questions through our experiments:

• •RQ1:Can short-context iterations outperform long traces by comparing {\SR,\PDR} to long-trace CoT at matchedBseq\mathrm{B}_{\text{seq}}andBtotal\mathrm{B}_{\text{total}}.
• •RQ2:Figuring out the best distillation strategy for producingC(r)C^{(r)}by comparing three𝒟\mathcal{D}variants: global summary, extractive top-kk, and random-kkbootstraps.
• •RQ3:Identifying the effect of the verification ability of a given model on the final performance.
• •RQ4:Whether operator-consistent training shifts the Pareto-Frontier. We compare a operator-consistent + standard RL with standard single-trace RL (Sec.3.3).

4.1Experiments to understand\SRand\PDR

Setup.We evaluate\SRand\PDRas inference-time operators on math problems. Given a promptxx, the model produces a thinking trace and a final solution. The thinking spans, delimited by<think>…</think>are stripped out and only the self-contained solutions are used to build the inputs for subsequent rounds. We evaluate on AIME 2024 and AIME 2025(AoPS,2025)and report the accuracy computed over 16 independent generations -mean​@​16\text{mean}@16.

Models and inference budgets.We evaluateo3-mini(“medium” reasoning effort)(OpenAI,2025)andgemini-2.5-flash(Comanici et al.,2025). Forgemini-2.5-flash, we vary the thinking budget from 8,192 to 24,576 tokens (its maximum), and reserve an additional 2,048 tokens for the final solution. Becauseo3-minidoes not expose a separate thinking budget, we vary its maximum generation length from 10,240 to 26,624 tokens to match the same total allowance (assuming 8,192–24,576 thinking tokens plus 2,048 solution tokens). Both operators (\SRand\PDR) are compared at matched per-call sequential budgetsBseqB_{\mathrm{seq}}(latency proxy) while allowing different total token budgetsBtotalB_{\mathrm{total}}via parallelism. All runs use temperature=1.0=1.0andtop-p=1.0=1.0.

RQ1: Do short-context iterations beat long traces at matched latency?

Sequential Refinement (\SR).For the\SRoperator, we runo3-miniandgemini-2.5-flashfor thinking budgetsB∈{8192,16384,24576}B\in\{8192,\,16384,\,24576\}and refinement roundsr∈{1,…,6}r\in\{1,\ldots,6\}. The prompt template is given in §B.1.

Figure 5:AIME 2024: Long CoT,\SR, and\PDRat thinking budget of 24576.Thexx-axis reportsBseqB_{\mathrm{seq}}: the thinking tokens consumed along the accepted path of the iterative chain, plus any distilled summary that conditions the next step. Tokens spent on unused parallel proposals are excluded, soBseqB_{\mathrm{seq}}serves as a latency proxy. At a token matched budget of442​k442ktokens,\SRhas a score of 90.4 butBseqB_{\mathrm{seq}}of442​k442k, whereas\PDRhas a score of 90.6 butBseqB_{\mathrm{seq}}of172​k172ktokens.Table 1:\SRoperator variants: Instead of just asking the model to refine the solution, we also ask the model to find and analyze errors in the solution followed by the correct solution. Error analysis followed by solution generation leads to better performance foro3-minibut not forgemini-2.5-flash.ModelBenchmarkThinking Budget\SR\SR-Errorgemini-2.5-flashAIME 20242457688.7587.71gemini-2.5-flashAIME 20252457678.7579.17gpt-o3-miniAIME 20242457680.8382.08gpt-o3-miniAIME 20252457673.1377.92SR with a local workspace.We also evaluate a variant of\SRthat inserts a brief, local workspace between refinements: the model first performserror analysis: identifying and explaining flaws in the current solution, and then generates a revised solution conditioned on these notes. All other settings (prompts, budgets) match standard\SRfor a fair comparison. As shown inTable˜1, this augmentation is effective foro3-minibut not forgemini-2.5-flash.

Parallel-Distill-Refine (\PDR).We evaluate\PDRunder a fixed thinking budgetBBusing three settings. These settings differ by number of rounds, number of parallel generations in each round, and selecting thekkcandidate solutions to carry forward via textual workspace:g=[8],k=4g=[8],\,k{=}4;g=[16,8],k=4g=[16,8],\,k{=}4; andg=[16,8,4],k=2g=[16,8,4],\,k{=}2. Hereg=[g1,…,gr]g=[g_{1},\ldots,g_{r}]specifies the number of parallel generations in each round, andk≤mind⁡grk\leq\min_{d}g_{r}is the number of candidates forwarded to the next round. For distillation (i.e., selecting thekkcandidates to carry forward), we compare: Random-kk(uniform sampling per instance); Top-kk(model-graded) where the same base model assigns a quality score to each candidate and we keep the topkkper instance (we report both a shared rubric and a per-instance grading prompt); and global-summary that aggregates all candidates using a summarization prompt. Refinement, selection, and summarization prompts are detailed in §B.2.

Token-matched baselines.InFigure˜5we sweep the depth of\SRto find the sequential budgetBseqB_{\mathrm{seq}}at which it matches\PDR. Holding the total token budget fixed atBtotal=442​kB_{\mathrm{total}}=442\text{k}(for\SR,Btotal=BseqB_{\mathrm{total}}=B_{\mathrm{seq}}since there is no parallelism),\PDRattains the target accuracy withBseq=172​kB_{\mathrm{seq}}=172\text{k}. To reach the same accuracy,\SRis run for 17 rounds, consumingBseq≈442​kB_{\mathrm{seq}}\approx 442\text{k}. Thus, at equalBtotalB_{\mathrm{total}},\PDRachieves the same accuracy with a2.57×2.57\timessmaller sequential budget by converting parallel compute into accuracy without lengthening per-call context.

Figure 6:AIME 2024: Anchoring bias due to++ve and−-ve examples:With\PDRwe compare three selection policies for the summary: Random-kk, Oracle-Incorrect (allkkcandidates are incorrect), and Oracle-Correct (allkkcandidates are correct), evaluated on bothgemini-2.5-flashando3-mini. Across all thinking budgets, admitting only incorrect candidates into the summary yields a pronounced drop in accuracy, whereas admitting only correct candidates improves over the Random-kkbaseline. The degradation under Oracle-Incorrect is larger foro3-minithan forgemini-2.5-flash, indicating weaker self-verification ino3-mini.Results.Figures˜3and9report accuracy on AIME 2024 and AIME 2025 under the same effective token budgetsBseq\mathrm{B}_{\text{seq}}. We observe consistent gains when moving from long chain-of-thought to\SR, which continue when moving from\SRto\PDR. Foro3-miniat an effective budget of49​k49\text{k}tokens with a per-call thinking budget of16​k16\text{k}, accuracy improves from76.976.9(Long CoT) to81.581.5(\SR) and86.786.7(\PDR), an absolute improvement of+9.8+9.8percentage points over Long CoT.gemini-2.5-flashshows smaller deltas from\SRto\PDRthano3-mini, suggesting stronger intrinsic self-verification ingemini-2.5-flash. AIME 2025 exhibits similar trends.

RQ2: Which distillation (i.e., summarization) strategy works best?

Table˜2studies the distillation operator𝒟\mathcal{D}in\PDRunder a (fixed number of rounds, number of generations in each round) settingg=[16,8,4]g=[16,8,4]withk=2k=2candidates per round. Across datasets and base models,per-sample top-kkandglobal-summaryselection consistently outperformshared top-kkandrandom-kk, and the margin widens as the thinking budgetBBincreases. The main exception is AIME 2025 witho3-mini, where global summary outperforms the alternatives. We hypothesize thato3-mini’s summarization is particularly effective at capturing cues from both correct and incorrect drafts, and these cues, when distilled, lead to stronger subsequent refinements.

Table 2:Effect of distillation operator𝒟\mathcal{D}:We compare the effect on final performance by changing the distillation operator𝒟\mathcal{D}. Each table column reports accuracies on AIME 2024 / AIME 2025. We compare four choices: (i)Global summary: aggregate all candidates and synthesizes a single compact summary; (ii)Per-sample top-kk: each downstream branch selects its own top-kkcandidates as the summary; (iii)Shared top-kk: a single set of top-kkcandidates is shared as the summary across generations for next round; (iv)Random-kk: each generation for next round receiveskkcandidates sampled uniformly at random for the summary. Overall, global summary and per-sample top-kktend to perform best, with gains more pronounced at higher thinking budgets. For o3-mini on AIME 2025, global summary yields the largest improvement, suggesting strong summarization ability in o3-mini. We usek=2k=2for these experiments.Budgetgemini-2.5-flashgpt-o3-miniGlobalPS top-kkShared top-kkRandom-kkGlobalPS top-kkShared top-kkRandom-kk819283.13 / 66.8883.75 / 71.8884.17 / 70.2183.33 / 66.6786.04 / 82.9284.79 / 76.0485.00 / 76.6782.50 / 71.251638486.46 / 84.3886.88 / 83.9686.46 / 83.7586.25 / 80.6386.46 / 84.7985.42 / 74.5885.83 / 76.8883.13 / 71.8824576\cellcolorblue!2088.75 / 87.71\cellcolorblue!2090.63 / 85.0087.71 / 85.4288.13 / 79.58\cellcolorblue!2085.42 / 83.5485.21 / 77.9285.00 / 75.4282.29 / 72.08RQ3: How does the verification abilities effect the inference time performance ?

Oracle\PDRanalysis.To understand the role of model verification within\PDR, we intervene on the set of candidates admitted to the summary at each round. We use a three-round\PDRwith number of generations in each round as[16,8,4][16,8,4]and top-kkselection (k=2k=2), and compare:(i)Random-kk:choosekkcandidates uniformly at random from the previous depth;(ii)Oracle (Correct):admit only correct candidates to the compact summary when available;(iii)Oracle (Incorrect):admit only incorrect candidates.

FromFigures˜6and8, we observe that injecting incorrect candidates (Oracle (Incorrect)) causes large drops for all models. The degradation is substantially larger foro3-minithan forgemini-2.5-flash, suggesting stronger self-verification and recovery in the latter. The same trend holds across AIME 2024 and AIME 2025.

We further provide a detailed analysis on the mechanics of\PDRand self-verification requirements to improve downstream performance inAppendix˜E.

4.2Operator-consistent RL Training

RQ4: Does operator-consistent training move the Pareto Frontier?

Building on the above, we present an operator consistent RL training strategy. This also addresses a train-test gap where models are not explicitly trained to perform\PDR.

Training setup.We train an 8B dense model similar to Llama-3 style architecture(Dubey et al.,2024). For warm-start supervised fine-tuning (SFT), we use GPT-OSS 120B(Agarwal et al.,2025)to generate synthetic traces for math and code prompts sampled from Polaris-53K(An et al.,2025)and DeepCoder(Luo et al.,2025), respectively. We run SFT for 8B tokens (∼\sim4 epochs). For RL training, we use the Polaris-53K dataset. Both SFT and RL datasets are decontaminated against AIME 2024/2025(AoPS,2025)and MATH-500(Hendrycks et al.,2021). Further details and hyper-parameters are detailed inSection˜B.3.

Baseline RLAs described inSection˜3.3, we use the CISPO objective for RL post-training(Li et al.,2025). We setϵ−=0.0\epsilon^{-}=0.0andϵ+=5.0\epsilon^{+}=5.0, and remove “zero-variance” prompts from a given batch(Seed et al.,2025). We use forced interruptions(Hong et al.,2025; Yang et al.,2025)to control generation length from exploding after a thinking budget of16,38416,384tokens. We additionally keep a buffer of20482048tokens for the final solution, thus keeping a maximum generation length of1843218432.3232generations are sampled per prompt with a batch size of3232, resulting in a global batch size of10241024generations per gradient step. Following(Liu et al.,2025b), we use a mini-batch size of256256and perform44gradient updates per rollout step.

Table 3:Operator RL results:Comparison of RL training objectives on AIME 2024/2025 at matched sequential budgetBseq=65,536B_{\mathrm{seq}}=65{,}536tokens using a dense 8B model. Mixing standard RL with operator-consistent RL (Op-RL) yields consistent gains for iterative inference operators such as\PDRwhile preserving performance on the Long CoT baseline. Op-RL can also be applied as a continual RL to the existing baseline RL checkpoint.ModelAIME 2024AIME 2025Long CoT\PDRLong CoT\PDR8B SFT Policy47.5062.9235.0047.508B Baseline RL67.5075.8359.5865.838B\PDRRL69.5879.1757.5067.508B Continual\PDRRL70.0080.8361.2570.42Operator-consistent RL with\PDR.For training, we use the\PDRoperator with configuration (4 parallel generations, 1 round)g=[4];k=2g=[4];k=2, and use the training objective described inEquation˜11and make two changes to the baseline RL method above: (i) increasing the input prompt length from20482048tokens to1024010240to allow for the compact workspace to be a part of the input, and (ii) mixing the standard RL and operator RL batches in the dataloader, keeping all other design choices the same. This setup allows to scale inference compute within the RL training.

Results.Table˜3summarizes the main results. The resulting model from each RL objective is evaluated for Long CoT generation and\PDR.\PDRRL improves over the baseline by+3.34+3.34points on AIME 2024 and+1.67+1.67points on AIME 2025. With continual updates starting from a baseline RL checkpoint, additional\PDRRL yields larger gains of+5.00+5.00and+4.59+4.59percentage points on AIME 2024 and AIME 2025, respectively. Additionally, we also observe marginal gains on Long CoT generations with\PDRRL training. These results indicate that training with operator-consistent RL objectives reduces the mismatch between training and deployment, converting extra compute into accuracy without increasing the per-call sequential budget.

5Conclusion

In this paper, we initiate the exploration of a broader design space around “long CoT.” We study two operators in this design space,\SRand\PDRwhich give better accuracy compared to standard long CoT, while offering the benefit of smaller context size. Empirically, compact-memory iteration outperforms long-trace baselines at matchedBseq\mathrm{B}_{\text{seq}}.\PDRyields the largest gains (e.g., +11% on AIME 2024 and +9% on AIME 2025), showing that evidence accumulation via bounded summaries can substitute for long reasoning traces while holding latency fixed. Beyond inference orchestration, making sure that training is consistent with inference using anoperator-consistent RLobjective further improves performance (e.g.,∼5%\sim 5\%on AIME 2024 and AIME 2025), suggesting that models can learn the meta-skills that make iteration effective. Iterative reasoning improves when diversity, verification, and refinement become reliably good; by measuring and training these micro-skills directly, we can accelerate the gains of improvement operators under fixed latency budgets.

Promising future directions include learning to improve the synthesis operator𝒟\mathcal{D}(trainable summaries), adaptive round and fan-out schedules conditioned on uncertainty (adaptivet​o​p−ktop-k), budget-aware controllers for allocating test-time compute, and tighter integration with verifiers and tool use. We also see value in scaling studies and cross-domain evaluations (reasoning, coding, and planning) to map when short-context iteration most benefits accuracy and latency.

6Acknowledgments

This work includes contributions from S.G. during his time at Meta. We also thank Xuewei Wang, Devvrit Khatri, and Alan Schelten for helpful discussions, and Jenya Lee and Abhinav Jauhri for infrastructure and compute support.

References

• Agarwal et al. (2025)Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al.gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925, 2025.
• An et al. (2025)Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong.Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025.URLhttps://hkunlp.github.io/blog/2025/Polaris.
• Anthony et al. (2017)Thomas Anthony, Zheng Tian, and David Barber.Thinking fast and slow with deep learning and tree search.InNeurIPS, 2017.
• AoPS (2025)AoPS.Aime problem set 1983-2025, 2025.URLhttps://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions.
• Arora & Barak (2007)Sanjeev Arora and Boaz Barak.Computational Complexity.Cambridge University Press, 2007.
• Baars (2005)Bernard J Baars.Global workspace theory of consciousness: toward a cognitive neuroscience of human experience.Progress in brain research, 150:45–53, 2005.
• Besta et al. (2024)Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al.Graph of thoughts: Solving elaborate problems with large language models.InProceedings of the AAAI conference on artificial intelligence, volume 38, pp. 17682–17690, 2024.
• Comanici et al. (2025)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025.
• Daumé III & Langford (2006)Hal Daumé III and John Langford.Search-based structured prediction.InICML, 2006.
• Didolkar et al. (2025)Aniket Didolkar, Nicolas Ballas, Sanjeev Arora, and Anirudh Goyal.Metacognitive reuse: Turning recurring llm reasoning into concise behaviors.arXiv preprint arXiv:2509.13237, 2025.
• Du et al. (2023a)Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch.Improving factuality and reasoning in language models through multiagent debate.arXiv preprint arXiv:2305.14325, 2023a.
• Du et al. (2023b)Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.Improving factuality and reasoning in language models through multiagent debate.InForty-first International Conference on Machine Learning, 2023b.
• Dubey et al. (2024)Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al.The llama 3 herd of models.arXiv e-prints, pp. arXiv–2407, 2024.
• Fu et al. (2025)Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao.Deep think with confidence.arXiv preprint arXiv:2508.15260, 2025.
• Ghosal et al. (2025)Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Yifu Lu, Mengdi Wang, Dinesh Manocha, Furong Huang, Mohammad Ghavamzadeh, and Amrit Singh Bedi.Does thinking more always help? understanding test-time scaling in reasoning models.arXiv preprint arXiv:2506.04210, 2025.
• Gou et al. (2023)Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen.Critic: Large language models can self-correct with tool-interactive critiquing.arXiv preprint arXiv:2305.11738, 2023.
• Goyal et al. (2022)Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio.Coordination among neural modules through a shared global workspace.InInternational Conference on Learning Representations (ICLR), 2022.arXiv:2103.01197.
• Guo et al. (2025)Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025.
• Hendrycks et al. (2021)Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874, 2021.
• Hong et al. (2025)Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, et al.Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.arXiv e-prints, pp. arXiv–2507, 2025.
• Hou et al. (2025)Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong.Treerl: Llm reinforcement learning with on-policy tree search.arXiv preprint arXiv:2506.11902, 2025.URLhttps://arxiv.org/abs/2506.11902.
• Irving et al. (2018)Geoffrey Irving, Paul Christiano, and Dario Amodei.Ai safety via debate.arXiv preprint arXiv:1805.00899, 2018.
• Jaech et al. (2024)Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al.Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024.
• Jiang et al. (2025)Minqi Jiang, Andrei Lupu, and Yoram Bachrach.Bootstrapping task spaces for self-improvement.arXiv preprint arXiv:2509.04575, 2025.
• Li et al. (2025)Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, et al.Minimax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025.
• Liu et al. (2025a)Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li.A simple “try again” can elicit multi-turn llm reasoning.arXiv preprint arXiv:2507.14295, 2025a.10.48550/arXiv.2507.14295.
• Liu et al. (2025b)Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong.Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.arXiv preprint arXiv:2505.24864, 2025b.
• Liu et al. (2023)Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.Lost in the middle: How language models use long contexts.arXiv preprint arXiv:2307.03172, 2023.
• Loshchilov & Hutter (2017)Ilya Loshchilov and Frank Hutter.Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017.
• Luo et al. (2025)Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Erran Li Li, Raluca Ada Popa, and Ion Stoica.Deepcoder: A fully open-source 14b coder at o3-mini level, 2025.URLhttps://www.together.ai/blog/deepcoder.Notion Blog.
• Madaan et al. (2023)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems, 36:46534–46594, 2023.
• Meurer et al. (2017)Aaron Meurer, Christopher P Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K Moore, Sartaj Singh, et al.Sympy: symbolic computing in python.PeerJ Computer Science, 3:e103, 2017.
• OpenAI (2025)Team OpenAI.Introducing openai o3 and o4-mini.2025.URLhttps://openai.com/index/introducing-o3-and-o4-mini/.
• Pan et al. (2025)Jiayi Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr.Learning adaptive parallel reasoning with language models.arXiv preprint arXiv:2504.15466, 2025.
• Ross et al. (2011)Stéphane Ross, Geoffrey Gordon, and J. Andrew Bagnell.A reduction of imitation learning and structured prediction to no-regret online learning.InAISTATS, 2011.
• Schrittwieser et al. (2020)Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, et al.Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588:604–609, 2020.
• Seed et al. (2025)ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al.Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning.arXiv preprint arXiv:2504.13914, 2025.
• Shanahan (2006)Murray Shanahan.A cognitive architecture that combines internal simulation with a global workspace.Consciousness and cognition, 15(2):433–449, 2006.
• Shao et al. (2024)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al.Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024.
• Shinn et al. (2023)Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning, 2023.URL https://arxiv. org/abs/2303.11366, 1, 2023.
• Silver et al. (2016)David Silver, Aja Huang, Chris J. Maddison, et al.Mastering the game of go with deep neural networks and tree search.Nature, 529(7587):484–489, 2016.
• Silver et al. (2017)David Silver, Julian Schrittwieser, Karen Simonyan, et al.Mastering chess and shogi by self-play with a general reinforcement learning algorithm.arXiv:1712.01815, 2017.
• Wang et al. (2024)Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkun.Reasoning in token economies: budget-aware evaluation of llm reasoning strategies.arXiv preprint arXiv:2406.06461, 2024.
• Wang et al. (2023)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.Self-consistency improves chain of thought reasoning in language models, 2023.URLhttps://arxiv.org/abs/2203.11171.
• Wei et al. (2022)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022.
• Williams (1992)Ronald J Williams.Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992.
• Wu et al. (2025)Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin, Zhongwang Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou.Resum: Unlocking long-horizon search intelligence via context summarization.arXiv preprint arXiv:2509.13313, 2025.
• Xie et al. (2024)Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh.Monte carlo tree search boosts reasoning via iterative preference learning.arXiv preprint arXiv:2405.00451, 2024.
• Yang et al. (2025)An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025.
• Yang et al. (2024)Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui.Buffer of thoughts: Thought-augmented reasoning with large language models.Advances in Neural Information Processing Systems, 37:113519–113544, 2024.
• Yao et al. (2023)Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan.Tree of thoughts: Deliberate problem solving with large language models.Advances in neural information processing systems, 36:11809–11822, 2023.
• Yu et al. (2025)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476, 2025.
• Zhang et al. (2025)Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune.Darwin godel machine: Open-ended evolution of self-improving agents.arXiv preprint arXiv:2505.22954, 2025.
• Zhao et al. (2025)Wenting Zhao, Pranjal Aggarwal, Swarnadeep Saha, Asli Celikyilmaz, Jason Weston, and Ilia Kulikov.The majority is not always right: Rl training for solution aggregation.arXiv preprint arXiv:2509.06870, 2025.
• Zheng et al. (2025)Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu.Parallel-r1: Towards parallel thinking via reinforcement learning.arXiv preprint arXiv:2509.07980, 2025.10.48550/arXiv.2509.07980.
• Zhou et al. (2022)Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al.Least-to-most prompting enables complex reasoning in large language models.arXiv preprint arXiv:2205.10625, 2022.

Appendix AExtended Related Work

Budget-aware evaluation and test-time compute.Recent work argues for comparing methods at matched compute budgets and reporting token usage, and not just accuracy(Wang et al.,2024). Our protocol reports sequential budgetBseq\mathrm{B}_{\text{seq}}(latency proxy along the accepted path) and total budgetBtotal\mathrm{B}_{\text{total}}(all tokens, including discarded branches), enabling apples-to-apples comparisons among single-pass, long-trace, sampling-heavy, and iterative pipelines.

MCTS, Learning to search and amortizing search.AlphaGo, AlphaZero, and MuZero couple a learned policy/value with an expensive test-time search (e.g., MCTS) that serves as an improvement operator; the outputs of search are then distilled back into the network, amortizing future search cost(Silver et al.,2016,2017; Schrittwieser et al.,2020).Expert Iterationformalizes this loop as policy improvement via planning followed by supervised or RL updates toward the planner’s targets(Anthony et al.,2017). Earlier “learning to search” work in structured prediction similarly alternates local rollouts with policy updates (e.g., SEARN, DAgger)(Daumé III & Langford,2006; Ross et al.,2011). Our setting is analogous in spirit but distinct in mechanics: we operate in thetextual reasoningregime with round-wise operators (\SR,\PDR) that keep the per-call sequential budgetBseq\mathrm{B}_{\text{seq}}small, optionally raising total computeBtotal\mathrm{B}_{\text{total}}via parallel drafts. Operator-consistent RL then amortizes this improvement procedure into the model weights.

Global workspace and modular coordination.Our compact, round-wise summaryC(r)C^{(r)}is conceptually related to theshared global workspaceproposed byGoyal et al. (2022), which enables coordination among neural modules through a small communication bottleneck (inspired by Global Workspace Theory(Baars,2005; Shanahan,2006)). In contrast, our workspace is textual, re-synthesized at each round rather than persisted, and used as an inference-time operator. Thus, we borrow the coordination intuition while avoiding long-context replay and architectural changes.

Appendix BPrompts

B.1Sequential Refinement

Refinement Prompt⬇Solvethefollowingmathproblemefficientlyandclearly.Pleasereasonstepbystep,andputyourfinalanswerwithin\\\\boxed\{answer\}.Where[answer]isjustthefinalnumberorexpressionthatsolvestheproblem.Problem:{{problem}}Hereisanexamplecandidateresponsewrappedinanglebrackets:<model_response></model_response>Treattheresponseasunverified;andcomeupwithabetteranswerwithoutstartingfromscratch.

B.2Parallel-Distill-Refine

Refinement Prompt (Non-summary)⬇Solvethefollowingmathproblemefficientlyandclearly.Pleasereasonstepbystep,andputyourfinalanswerwithin\\\\boxed\{answer\}.Where[answer]isjustthefinalnumberorexpressionthatsolvestheproblem.Problem:{{problem}}Herearesomecandidateresponses,eachwrappedinanglebrackets:<model_response_1></model_response_1>...<model_response_k></model_response_k>Treattheseresponsesasunverified;andusetheseresponsestocomeupwithabetteranswerwithoutstartingfromscratch.

Refinement Prompt (Summary)⬇Solvethefollowingmathproblemefficientlyandclearly.Pleasereasonstepbystep,andputyourfinalanswerwithin\\\\boxed\{answer\}.Where[answer]isjustthefinalnumberorexpressionthatsolvestheproblem.Problem:{{problem}}Hereisasummaryofthereasoningtracesbyafewothersolvers:<summary></summary>Treatthesummaryasunverified;andusethesummaryascontexttocomeupwithananswer.

B.3Training Setup

We run a small SFT on the pre-trained 8B base model using a batch size of 2M tokens, max sequence length of 32768, and a learning rate of2×10−52\times 10^{-5}using the AdamW optimizer(Loshchilov & Hutter,2017)on 32 H100 GPU nodes for approximately 4 epochs and 8B tokens in total. For RL, we use a constant learning rate of5×10−75\times 10^{-7}, AdamW optimizer(Loshchilov & Hutter,2017)withϵ=10−15\epsilon=10^{-15}, weight decay of 0.01, and a linear warmup of 100 steps. We use 80 Nvidia H200 GPUs for the baseline RL run with a 64/16 generators/trainers split and 288 H200 GPUs for\PDRand continual\PDRRL with a 256/16 generators/trainers split to parallelize inference during rollout generation. We run all RL training for 800 steps. All evaluations are done with a temperature andtop-pvalue of 1.0.

Appendix CComplexity of Space-bounded computation

Our work focused on language models that emit reasoning traces that are longer than the context size. We sketch similarity to the settingspace-bounded computationwhich is formally studied in computational complexity theory. Since LLMs have probabilistic output (unless if temperature is set to0) the fixed-context LLM considered in the paper is most similar to randomized space-bounded machine.

Figure 7:Space-bounded Turing Machine [Figure fromComputational Complexityby Arora and Barak, 2007]. The input has sizeNNand the machine has read-only capability for the input. A special “tape head” can be moved over the input to read bits from it. The amount of working memory (read/write/erase) for actual computation has sizeS​(N)S(N)whereS​(N)≥log⁡NS(N)\geq\log N.The most interesting result about randomized space-bounded machines is that if the input contains a graph ofNNvertices, then the randomized machine can determine connectivity of theNN-vertex graph even though it only has𝒪​(log⁡N)\mathcal{O}(\log N)space.

Furthermore, imagine that the graph of sizeNNis a knowledge-graph whose local structure is known to the space-bounded machine. Specifically, given vertex indicesi,ji,jthe space-bounded machine is able to determine whether edge{i,j}\{i,j\}exists in the graph. Then the machine does not need access to the full graph tape at all! It can do a random walk through the graph “in its mind” to determine connectivity. This is the closest setting to ours, whereby seemingly complex reasoning-connectivity of anNN-node graph can be carried out in less than𝒪​(N)\mathcal{O}(N)space.

Appendix DAdditional Results

D.1Oracle

Similar toFigure˜6, we observe that having incorrect solutions in the context workspace can heavily degrade performance, and this effect is more noticeable foro3-mini.

Figure 8:AIME 2025: Anchoring bias due to++ve and−-ve examples:With\PDRwe compare three selection policies for the summary: Random-kk, Oracle-Incorrect (allkkcandidates are incorrect), and Oracle-Correct (allkkcandidates are correct), evaluated on bothgemini-2.5-flashando3-mini. Across all thinking budgets, admitting only incorrect candidates into the summary yields a pronounced drop in accuracy, whereas admitting only correct candidates improves over the Random-kkbaseline. The degradation under Oracle-Incorrect is markedly larger foro3-minithan forgemini-2.5-flash, indicating weaker self-verification ino3-mini.

Figure 9:AIME 2025: Iterative improvement beats single-pass long-CoT at matched sequential budgets.Thexx-axis reportsBseqB_{\mathrm{seq}}: the thinking tokens consumed along the accepted path of the iterative chain, plus any distilled summary that conditions the next step. Tokens spent on unused parallel proposals are excluded, soBseqB_{\mathrm{seq}}serves as a latency proxy. At comparableBseqB_{\mathrm{seq}}, both\SRand\PDRoutperform the single-pass long CoT baseline, with\PDRyielding the largest gains by converting additional total compute (via parallelism) into accuracy without increasing per-call context.

D.2\SRand\PDRoperators

AIME 2025 results using the two iterative improvement operators\SRand\PDRare presented inFigure˜9. For sequential token budget of49​k49k, the performance ono3-miniimproves from 73.5 for Long CoT to 77.1 using\SRoperator and further to 82.9 using the\PDRoperator.

Additionally, we show two\SRvariant results inTable˜1, where error analysis followed by solution generation leads to improvements ono3-miniwithout any affect on the sequential token budgetBseqB_{\text{seq}}.

Figure 10:Token Budgets comparison:We plot all the different configurations for Long CoT,\SRand\PDRoperators for bothBseqB_{\text{seq}}andBtotalB_{\text{total}}token budgets foro3-mini. For bothBseqB_{\text{seq}}andBtotalB_{\text{total}},\PDRforms the pareto-frontier and gives consistent gains over Long CoT and\SR.

Appendix EMechanics of Improvement operator: Source of accuracy gain

Under the default\PDRsetting,gemini-2.5-flashmisses 4 AIME 2024 questions. We probe how additional parallel compute affects the performance on these four AIME questions. We compare a 4-round schedule (less compute) to a 5-round, wider schedule (more compute). Accuracy (fraction correct over 16 seeds) changes as follows:Q1:​0.4375→0.625\text{Q1: }0.4375\rightarrow 0.625(gain),Q2:​0.0625→0\text{Q2: }0.0625\rightarrow 0(drop),Q3:​0.1875→0.1875\text{Q3: }0.1875\rightarrow 0.1875(no change),Q4:​0→0\text{Q4: }0\rightarrow 0(no change). In the high-compute setting (first round widthM1=32M_{1}{=}32), number of correct drafts among 32 is: Q1:3/323/32, Q2:0/320/32, Q3:3/323/32, Q4:0/320/32. This breakdown clarifies how\PDRcan (or cannot) improve with additional rounds:

Table 4:PDR hard-case analysis forgemini-2.5-flash.We compare a 4-round schedule (less compute:1616generations in round 1) to a 5-round, wider schedule (more compute:3232generations in round 1). Accuracies are fraction correct over 16 seeds. “Round-1 hits” counts how many of the 32 first-round drafts are already correct ((more compute setting).Round-1 hitsAccuracy (over 16 seeds)Δ\DeltaInterpretationQuestion(correct/32)Less computeMore compute(More−-Less)Q13/323/327/167/16(0.4375)10/1610/16(0.6250)+3/16+3/16(+0.1875)GainQ20/320/321/161/16(0.0625)0/160/16(0.0000)−1/16-1/16(−0.0625-0.0625)DropQ33/323/323/163/16(0.1875)3/163/16(0.1875)0FlatQ40/320/320/160/16(0.0000)0/160/16(0.0000)0Flat

1. 1.When a round-1 draft is correct,\PDRimproves if two things happen: (i) the correct evidence is carried into the summaryC(1)C^{(1)}(i.e., high recall in the distillation operator𝒟\mathcal{D}); and (ii) the refine step uses that evidence to update the answer. If𝒟\mathcal{D}drops or down-weights the signal amid conflicting drafts, later rounds cannot exploit it. The Q1 gain from 4→5 rounds suggests both steps succeeded; the flat Q3 curve despite3/323/32correct drafts points to a verification/refinement gap.
2. 2.Verification among many distractors (Q1/Q3).Even when correct drafts are present, round-1 mixes a small number of correct drafts with many incorrect ones (here,33vs.2929). The summary must surface signals that distinguish correctness. This is where\PDR’s distill step should act as a verifier-aware aggregator.
3. 3.Recovery when no draft is correct (Q2/Q4).If round 1 has0/320/32correct, the summary still needs to extract useful structure (partial progress, contradictions, eliminated avenues) that increases the chance of success in round 2+. The refine step should then expand diversity informed by these cues. The mild regression on Q2 with more compute is consistent with summary drift: the distillation over-weights a wrong pattern, and subsequent rounds reinforce it. The no-change on Q4 may suggest either a capability ceiling or the case that even more generations are required in round 1.

Overall, the analysis here points to some gaps in the core skills required to carry out\PDR- verification, refinement, and diversity. Improving the model along each of these skills will not only improve\PDRbut also any situation where a multiplicative combination of these skills is required.

Noam Brown (@polynoamial): More test-time compute leads to greater intelligence. But as we push ttc from seconds to weeks, latency becomes a bottleneck.

GPT-5.6 Sol Ultra scales parallel ttc. The time taken to generate a proof to a 50-year-old problem drops from perhaps a whole day to a single hour.