If you still think AI agents can’t do real research, this paper will end that argument. Researchers from Google and Meta built a framework where Claude Code proposes its own algorithms for making LLMs reason better, then tests them, then refines them based on what failed. No human in the loop after the environment is set up. In 5 rounds the agent discovered a controller with 4 coordinated mechanisms working together. EMA momentum stopping. Coupled width-depth control. Alignment-aware depth allocation. Conservative branch abandonment. The paper says directly: “a level of coordinated complexity that would be difficult to arrive at through manual intuition alone.” That’s a polite way of saying the agent built something a human probably wouldn’t have. The cost of the entire discovery was $39.90. The cost of one researcher’s coffee budget just outperformed years of hand-tuned work. Paper is from Google and Meta. Read it here: https://arxiv.org/abs/2605.08083

Agentic Discovery for Test-Time Scaling

Source: https://arxiv.org/html/2605.08083

LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling

Tong Zheng1, Haolin Liu2, Chengsong Huang3, Huiwen Bao, Sheng Zhang1, Rui Liu1, Runpeng Dai4, Ruibo Chen1, Chenxi Liu1, Tianyi Xiong1, Xidong Wu5, Hongming Zhang6Heng Huang1 1UMD,2UVA,3WUSTL,4UNC,5Google,6Meta

Abstract

Test-time scaling (TTS) has become an effective approach for improving large language model performance by allocating additional computation during inference. However, existing TTS strategies are largely hand-crafted: researchers manually design reasoning patterns and tune heuristics by intuition, leaving much of the computation-allocation space unexplored. We propose an environment-driven framework,AutoTTS, that changes what researchers design: from individual TTS heuristics to environments where TTS strategies can be discovered automatically. The key to AutoTTS lies in environment construction: the discovery environment must make the control space tractable and provide cheap, frequent feedback for TTS search. As a concrete instantiation, we formulate width–depth TTS as controller synthesis over pre-collected reasoning trajectories and probe signals, where controllers decide when to branch, continue, probe, prune, or stop and can be evaluated cheaply without repeated LLM calls. We further introduce beta parameterization to make the search tractable and fine-grained execution trace feedback to improve discovery efficiency by helping the agent diagnose why a TTS program fails. Experiments on mathematical reasoning benchmarks show that the discovered strategies improve the overall accuracy–cost tradeoff over strong manually designed baselines. The discovered strategies generalize to held-out benchmarks and model scales, while the entire discovery costs only$39.9and160 minutes. Our data, and code will be open-source athttps://github.com/zhengkid/AutoTTS.

Figure 1:Overview of our Auto-TTS framework.Unlike the traditional workflow of manually designing TTS strategies, Auto-TTS shifts the human role from directly hand-crafting branching, pruning, and stopping heuristics to constructing environments by defining states, actions, feedback, and objectives. Given the constructed environment, an explorer LLM iteratively proposes candidate controllers, evaluates them in the offline replay environment, receives feedback from scaling curves and execution traces, and uses the accumulated history to refine future proposals. The right panel shows an example evaluation on Qwen-1.7B and AIME25, where the discovered controller improves the accuracy–cost Pareto frontier over hand-crafted baselines with an affordable one-time search cost.## 1Introduction

Figure 2:Existing TTS algorithms as special cases of the width–depth control space. Each algorithm traces a distinct path:SC@64wang2022selfoccupies a fixed full-budget corner;ASCaggarwal2023letandESCli2024escapeadapt only along the width axis at max depth;Answer Consistencyliu2025answeradapts only along the depth axis on a single chain;ST-BoNwang2025samplingexpands wide, prunes to one branch, then deepens;Parallel-Probezheng2026parallelstarts wide and progressively prunes while deepening.Test-time scaling (TTS)snell2024scaling;brown2024large;muennighoff2025s1has emerged as a powerful paradigm for improving large language model performance by allocating additional computation during inference. However, performance depends not just on the amount of computation used, but on how it is allocatedsnell2024scalingand existing strategies for doing so are largely hand-crafted: researchers manually hypothesize heuristics for when to branch, deepen, probe, prune, or stop reasoning trajectorieszhao2025majority;wen2025parathinker;wang2026not;zheng2026parallel;tu2025deepprune;zhang2025alphaone, implement them, and tune thresholds by intuition.

Looking back at the development of TTS strategies reveals a valuable perspective. Although existing methods differ substantially in form, many of them can be interpreted as manually specified policies within some underlying computation-allocation space. A simple example is the width–depth space, where width denotes how many reasoning branches are explored and depth denotes how far each branch is developed, as illustrated in Figure2. Under this view, several representative methods correspond to different trajectories through the space: some expand width by sampling more reasoning brancheswang2022self;aggarwal2023let;li2024escape; some increase depth by extending reasoning trajectoriesmuennighoff2025s1;zhang2025alphaone; and others introduce adaptive stopping, pruning, or selection rules to move through the space more selectivelyfu2025deep;tu2025deepprune;liu2025answer;zheng2026parallel. Notably, this perspective is not intended to reduce all TTS algorithms to a two-dimensional abstraction, as many methods involve richer structures such as tree searchyao2023tree;inoue2025wideror verifier-guided refinementsnell2024scaling;wang2024math;luo2024improve. Rather, the case of width–depth space reveals that many TTS strategies can be seen as hand-designed special cases within a structured control space.

This perspective suggests a fundamental reframing of the problem.In this work, we proposeAutoTTS, an environment-driven paradigm for automatic TTS strategy discovery (Figure1). Instead of hand-crafting individual branching, pruning, and stopping heuristics, AutoTTS shifts the human role to constructing discovery environments, where humans define the control space through states, actions, feedback, and objectives, and agents search within this space for effective allocation policies.

As a proof-of-concept instantiation, we formulate width–depth test-time scaling (Figure2) as controller synthesis in an offline replay environment. For each problem, we pre-collect reasoning trajectories and intermediate probe signals, allowing a controller to replay decisions over when to branch, continue, probe, prune, or stop. The controller observes active branches, their depths, revealed probe outputs, and the remaining budget, and is evaluated by the resulting accuracy–cost tradeoff. Since evaluation reuses pre-collected trajectories, candidate controllers can be assessed cheaply and deterministically without repeatedly invoking the base LLM. However, effective discovery faces two additional challenges: automatically discovered controllers tend to introduce excessive hyperparameters that makes the search space large and difficult to navigate within a limited number of rounds, and scalar accuracy–cost feedback alone is not sufficient to tell the explorer why a controller fails.

To make controller search tractable, we introduce beta parameterization, where each controller exposes only one scalar trade-off parameterβ\betaand derives all internal hyperparameters deterministically from it, reducing overfitting to the search set. Additionally, we address the feedback issue by logging execution traces that expose how each controller allocates computation over time, enabling the explorer to diagnose failure modes and propose targeted improvements.

Experiments on mathematical reasoning benchmarks show that AutoTTS discovers controllers that improve the accuracy–cost Pareto frontier over strong hand-crafted baselines. The discovered controllers generalize from the search benchmark to held-out benchmarks and across model scales, while the entire discovery process remains affordable due to fixed replay.These results suggest that environment-driven discovery offers a scalable and reusable alternative to manually designing TTS strategies, and that the right place to invest human effort is in environment design, not strategy design.

2Test-Time Scaling as Algorithmic Search

We consider adaptive test-time algorithms that allocate inference budget across multiple reasoning branches. For each questionqq, a controller may create, extend, probe, and prune branches before aggregating the explored prefixes into a final answer, covering strategies such as best-of-NN, self-consistency, early stopping, and adaptive branching. Each branchiiproduces prefixeszi,1,zi,2,…z_{i,1},z_{i,2},\ldots, wherezi,kz_{i,k}is obtained afterkkfixed-length generation intervals. Each prefix induces an intermediate answerωi,k\omega_{i,k}(i.e., the answer that would be produced from the current prefix), which is observed only if the controller explicitly takes a probing action at that branch.

Since we consider a unit generation as an interval with a fixed token length, we measure computation in units of intervals. At decision steptt, letmt∈ℤ≥0m_{t}\in\mathbb{Z}_{\geq 0}denote the number of branches instantiated so far, and use the convention[mt]={1,…,mt}[m_{t}]=\{1,\ldots,m_{t}\}, with[0]=∅[0]=\emptyset. The state isst=(q,mt,It,ℓt,Zt,Ωt)s_{t}=(q,m_{t},I_{t},\ell_{t},Z_{t},\Omega_{t}), whereq∈𝒬q\in\mathcal{Q}is the question,It⊆[mt]I_{t}\subseteq[m_{t}]is the set of currently active branches,ℓt=(ℓt,i)i∈[mt]\ell_{t}=(\ell_{t,i})_{i\in[m_{t}]}records the current depth of every instantiated branch,Zt=(Zt,i)i∈[mt]Z_{t}=(Z_{t,i})_{i\in[m_{t}]}records the generated prefixes, andΩt\Omega_{t}is the set of probe feedback revealed so far. For every instantiated branchi∈[mt]i\in[m_{t}],ℓt,i≥1\ell_{t,i}\geq 1denotes how many fixed-length generation intervals have been generated on that branch, andZt,i=(zi,1,…,zi,ℓt,i)Z_{t,i}=(z_{i,1},\ldots,z_{i,\ell_{t,i}})contains the prefixes generated on branchiiup to its current depth. For a pruned branchi∈[mt]∖Iti\in[m_{t}]\setminus I_{t},ℓt,i\ell_{t,i}andZt,iZ_{t,i}record the depth and generated prefixes at which it was pruned. The revealed feedback set satisfiesΩt⊆{(i,k,ωi,k):i∈[mt],1≤k≤ℓt,i}\Omega_{t}\subseteq\{(i,k,\omega_{i,k}):i\in[m_{t}],\ 1\leq k\leq\ell_{t,i}\}. Unrevealed probe outputs are not part of the controller state. The computation cost of a state isCost​(st)=∑i=1mtℓt,i+κprobe​|Ωt|\mathrm{Cost}(s_{t})=\sum_{i=1}^{m_{t}}\ell_{t,i}+\kappa_{\mathrm{probe}}|\Omega_{t}|, whereκprobe≥0\kappa_{\mathrm{probe}}\geq 0is the cost of reading one probe signal. In settings where probing is treated as free relative to generation, we setκprobe=0\kappa_{\mathrm{probe}}=0.

Givenst=(q,mt,It,ℓt,Zt,Ωt)s_{t}=(q,m_{t},I_{t},\ell_{t},Z_{t},\Omega_{t}), the admissible action set is𝒜​(st)={BRANCH}∪{CONTINUE​(i):i∈It}∪{PROBE​(i):i∈It,∄​ω​s.t.(i,ℓt,i,ω)∈Ωt}∪{PRUNE​(i):i∈It}∪{ANSWER}\mathcal{A}(s_{t})=\{\texttt{BRANCH}\}\cup\{\texttt{CONTINUE}(i):i\in I_{t}\}\cup\{\texttt{PROBE}(i):i\in I_{t},\ \nexists\omega\ \mathrm{s.t.}\ (i,\ell_{t,i},\omega)\in\Omega_{t}\}\cup\{\texttt{PRUNE}(i):i\in I_{t}\}\cup\{\texttt{ANSWER}\}. HereBRANCHcreates a new branchmt+1m_{t}+1and advances it from the question to the end of the first interval.CONTINUE​(i)\texttt{CONTINUE}(i)advances branchiiby one fixed-length generation interval.PROBE​(i)\texttt{PROBE}(i)reveals the current probe signalωi,ℓt,i\omega_{i,\ell_{t,i}}without advancing the branch.PRUNE​(i)\texttt{PRUNE}(i)removes branchiifrom the active set, while keeping its current information.ANSWERterminates inference and invokes the aggregation rule. An aggregation ruleAgg\operatorname{Agg}takes a state as input and outputs the final answer.

Formally, the initial state iss0=(q,0,∅,∅,∅,∅)s_{0}=(q,0,\emptyset,\emptyset,\emptyset,\emptyset). Ifat=BRANCHa_{t}=\texttt{BRANCH}, a new branchmt+1m_{t}+1is instantiated and advanced to the first probe point, producing prefixzmt+1,1z_{m_{t}+1,1}. Thenmt+1=mt+1m_{t+1}=m_{t}+1,It+1=It∪{mt+1}I_{t+1}=I_{t}\cup\{m_{t}+1\},ℓt+1=(ℓt,1)\ell_{t+1}=(\ell_{t},1),Zt+1=(Zt,(zmt+1,1))Z_{t+1}=(Z_{t},(z_{m_{t}+1,1})), andΩt+1=Ωt\Omega_{t+1}=\Omega_{t}. Ifat=CONTINUE​(i)a_{t}=\texttt{CONTINUE}(i), branchiiis advanced by one interval, producing prefixzi,ℓt,i+1z_{i,\ell_{t,i}+1}. Thenmt+1=mtm_{t+1}=m_{t},It+1=ItI_{t+1}=I_{t},Ωt+1=Ωt\Omega_{t+1}=\Omega_{t},ℓt+1,j=ℓt,j+𝟏​{j=i}\ell_{t+1,j}=\ell_{t,j}+\mathbf{1}\{j=i\}for allj∈[mt]j\in[m_{t}],Zt+1,i=(Zt,i,zi,ℓt,i+1)Z_{t+1,i}=(Z_{t,i},z_{i,\ell_{t,i}+1}), andZt+1,j=Zt,jZ_{t+1,j}=Z_{t,j}for allj≠ij\neq i. Ifat=PROBE​(i)a_{t}=\texttt{PROBE}(i), thenmt+1=mtm_{t+1}=m_{t},It+1=ItI_{t+1}=I_{t},ℓt+1=ℓt\ell_{t+1}=\ell_{t},Zt+1=ZtZ_{t+1}=Z_{t}, andΩt+1=Ωt∪{(i,ℓt,i,ωi,ℓt,i)}\Omega_{t+1}=\Omega_{t}\cup\{(i,\ell_{t,i},\omega_{i,\ell_{t,i}})\}. Ifat=PRUNE​(i)a_{t}=\texttt{PRUNE}(i), thenmt+1=mtm_{t+1}=m_{t},It+1=It∖{i}I_{t+1}=I_{t}\setminus\{i\},ℓt+1=ℓt\ell_{t+1}=\ell_{t},Zt+1=ZtZ_{t+1}=Z_{t}, andΩt+1=Ωt\Omega_{t+1}=\Omega_{t}. IfaT=ANSWERa_{T}=\texttt{ANSWER}at timeTT, the episode terminates and the final answer is produced byy^=Agg⁡(sT)\hat{y}=\operatorname{Agg}(s_{T}).

Our goal is to find a code-defined policyπ\pithat maps a state and a hyperparameterβ\betato a distribution over admissible atomic actionsπ(⋅∣s,β)∈Δ(𝒜(s))\pi(\cdot\mid s,\beta)\in\Delta(\mathcal{A}(s)). Here,β\betaspecifies the tunable hyperparameters of the algorithm. We also allow each controller to include its own terminal aggregation ruleAggπ,β\operatorname{Agg}_{\pi,\beta}. For a questionqq, letτ=(s0,a0,s1,a1,…,sT)∼Pπ,β(⋅∣q)\tau=(s_{0},a_{0},s_{1},a_{1},\ldots,s_{T})\sim P_{\pi,\beta}(\cdot\mid q)denote the execution trajectory induced by running(π,β)(\pi,\beta)from the initial states0=(q,0,∅,∅,∅)s_{0}=(q,0,\emptyset,\emptyset,\emptyset)untilANSWERis selected at timeTT. The trajectory distributionPπ,β(⋅∣q)P_{\pi,\beta}(\cdot\mid q)includes the randomness from the policy, the generation process, and any stochastic probe signals. At the terminal statesTs_{T}, the final answer and computation cost arey^π,β​(τ)=Aggπ,β⁡(sT),C​(τ)=Cost⁡(sT).\hat{y}_{\pi,\beta}(\tau)=\operatorname{Agg}_{\pi,\beta}(s_{T}),C(\tau)=\operatorname{Cost}(s_{T}).For a task distribution𝒟\mathcal{D}over question-answer pairs(q,y)(q,y), we choose(π,β)(\pi,\beta)to maximize accuracy while controlling computation cost. With trade-off parameterγ\gamma, the objective is

max(π,β)⁡𝔼(q,y)∼𝒟,τ∼Pπ,β(⋅∣q)​[𝟏​{y^π,β​(τ)=y}−γ​C​(τ)].\displaystyle\max_{(\pi,\beta)}\mathbb{E}_{(q,y)\sim\mathcal{D},\ \tau\sim P_{\pi,\beta}(\cdot\mid q)}\left[\mathbf{1}\{\hat{y}_{\pi,\beta}(\tau)=y\}-\gamma C(\tau)\right]. The discovery loop searches over code-defined controllers. Each candidate is run on every question to obtain an execution trajectoryτ\tau; we compare its final answer with the ground truth and record its computation cost. The resulting execution histories are stored in memory and used to guide subsequent rounds of policy search. Finally, we output the code-defined policy with the best accuracy–cost trade-off.

3AutoTTS: Environment-Driven Discovery

We instantiateAutoTTSas a concrete discovery pipeline for the objective in Section2. The key challenge is making the search over code-defined policiesπ(⋅∣s,β)\pi(\cdot\mid s,\beta)tractable: evaluating a candidate policy online requires generating token intervalzi,kz_{i,k}and probing answerωi,k\omega_{i,k}on demand, which is prohibitively expensive at search time. We address this challenge through three complementary design choices: anoffline replay environmentthat eliminates repeated LLM calls during evaluation (Section3.1),beta parameterizationthat prevents overfitting to the search set (Section3.3), andexecution trace feedbackthat enables the agent to diagnose failure modes rather than relying on scalar outcomes alone (Section3.2).

3.1Replay Environment Construction

The central challenge in evaluating candidate controllers online is that each evaluation requires invoking the base LLM to generate reasoning trajectories on demand, which is prohibitively expensive at search time. To address this, we construct an offline replay environment that moves all LLM calls prior to the discovery process, making controller evaluation deterministic and cheap.

Offline data collection.

Following the data collection protocol of Parallel-Probezheng2026parallel, for each questionq∈𝒬q\in\mathcal{Q}, we pre-collectNNindependent reasoning trajectories from the base LLM, each segmented into fixed-length intervals ofΔ\Deltatokens. This directly instantiates the branch prefixeszi,1,zi,2,…z_{i,1},z_{i,2},\ldotsand probe signalsωi,1,ωi,2,…\omega_{i,1},\omega_{i,2},\ldotsintroduced in Section2, with all data stored offline before any discovery begins. Each controller decision is then executed against this pre-collected data rather than invoking the LLM, making repeated controller evaluation affordable.

Evaluation via offline replay.

To evaluate each controller for a givenβ\beta, we run it on each question in𝒬search\mathcal{Q}_{\mathrm{search}}: at each statests_{t}it selects an action from𝒜​(st)\mathcal{A}(s_{t})and advances untilAnsweraction is taken. Because all branch prefixes and probe signals are pre-collected offline, each action reads deterministically from this stored data rather than invoking the LLM—for instance, aProbeaction on branchiiat depthkksimply retrieves the pre-collected signalωi,k\omega_{i,k}at zero generation cost. This makes the entireβ\betasweep affordable without any additional LLM calls.

3.2Discovery Loop

We partition𝒬\mathcal{Q}into a search set𝒬search\mathcal{Q}_{\mathrm{search}}and a held-out evaluation set𝒬eval\mathcal{Q}_{\mathrm{eval}}.AutoTTSdiscovers an effective controller through a multi-round loop: each round, an explorer LLM (Claude Code) reads the accumulated historyℋ\mathcal{H}and proposes an improved controller by directly editing the code; the controller is then evaluated on𝒬search\mathcal{Q}_{\mathrm{search}}and the results are appended toℋ\mathcal{H}.

Agent-driven proposal.

The explorer readsℋ\mathcal{H}—which stores all previously proposed controller implementations, their accuracy–cost outcomes, and execution traces—and is prompted to analyse what went wrong in prior proposals, and propose a new controller that improves accuracy while reducing token usage (full prompt in AppendixLABEL:app:prompt).

History Design.

While scalar outcomes such as accuracy and token usage provide a coarse signal of whether a proposed controller is good enough, they reveal little aboutwhya controller fails. To address this limitation, we augment the history with the full decision-making trajectories that the controller executed during the replay environment. For each round, we sweep across multiple betas and record the resulting scaling curve as the scalar component; the trajectory component then supplies fine-grained behavioral evidence, enabling the agent to diagnose failure modes and propose a more targeted controller in the next round. This design is consistent with the finding inlee2026metathat fine-grained execution feedback improves agentic discovery for harness engineering.

Controller selection.

AfterRRrounds, we select the controller andβ\betavalue that achieves the highest accuracy on𝒬search\mathcal{Q}_{\mathrm{search}}.

3.3Beta Parameterization for Tractable Search

In our preliminary experiments, we empirically find that agents tend to propose TTS controllers with a large number of hyperparameters, up to 10. With only five discovery rounds, navigating this high-dimensional space causes the agent to collapse onto extreme solutions—such as overly aggressive pruning thresholds—that happen to minimize token cost on the search set but fail to represent robust allocation strategies.

To mitigate this, we proposebeta parameterization: each controller must expose only a single hyperparameterβ\betaand implement a map function fromβ\betato all internal hyperparameters. We further require this map to be monotone, such that largerβ\betacorresponds to larger token budget. This collapses the search space to a one-dimensional sweep and prevents the agent from discovering sharp, search-set-specific thresholds. Notably, the map function is produced directly by the coding agent.

4Experimental Setup

Experimental Protocol.

All experiments use offline replay environments, each built from a specific (model, benchmark) pair across four Qwen3 models (0.6B, 1.7B, 4B, 8B)yang2025qwen3. Following Parallel-Probezheng2026parallel, we pre-sample 128 reasoning trajectories per (model, problem) pair at temperature 0.7 with a probing interval of 500 tokens to construct the replay matrix. To reduce variance, each controller is evaluated 64 times independently by randomly sampling a subset of trajectories from the pre-sampled pool of 128, and the results are averaged. For discovery, we use AIME24 as𝒬search\mathcal{Q}_{\text{search}}and constructℰsearch\mathcal{E}_{\text{search}}as the union of AIME24 environments across all four models. The discovery loop runs for five rounds with Claude Code as the agent, and the final controller is selected as the one achieving the highest accuracy onℰsearch\mathcal{E}_{\text{search}}. The discovered controller (AppendixLABEL:app:controller) is fixed and evaluated on held-out environments from AIME25 and HMMT25, which are never used during discovery or selection.

Baselines.

To demonstrate the effectiveness of our proposed discovery framework, we compare the discovered algorithm with several representative handcrafted test time scaling methods. 1)Self-Consistency (SC@64)wang2022self:A vanilla parallel reasoning approach that first samples 64 reasoning trajectories and performs majority voting to obtain the final answer; 2)ASCaggarwal2023let:A parallel sampling approach that samples trajectories one by one and stop until reaching a pre-defined threshold. We follow the original setting with threshold 0.95; 3)ESCli2024escape:A chunk-based hybrid approach that generates trajectories in parallel and terminates early when answer stability is detected within a sliding window. We use a chunk size of 8 and 4)Parallel-Probezheng2026parallel:A recent efficient parallel reasoning approach that leverages cross-branch information to dynamically decide when to stop reasoning, prune unpromising branches, or continue computation.

Metrics.

We report both task accuracy and tokens that measure the total number of tokens consumed across all used branches.

5Results and Analysis

5.1Main Results

Table 1:Accuracy and total tokens. AIME24 (search) is used for controller discovery; AIME25 and HMMT25 are held-out.