Brilliant paper on scaling laws for agentic RL data.

It’s a great paper worth reading if you are synthesizing data for agentic RL.

(bookmark it)

They fit scaling laws for agentic RL data along three axes: tasks, environments, and trajectories. Across model sizes, families, domains, and both PPO and GRPO, the exponents order the same way, with environments above tasks and tasks above trajectories.

The environment advantage seems to come mainly from world diversity, meaning genuinely different settings. Varying the structure of the same world helps about as much as adding tasks.

The laws also give each synthesis method a discount factor and a synthetic-to-real ratio at which training starts to collapse. Those ratios differ by an order of magnitude across axes.

In addition, allocating the budget with their recipe gives up to 1.7x compute efficiency over the common practice of generating more trajectories.

https://arxiv.org/abs/2610.23946

Chat with Paper: https://academy.dair.ai/papers/how-data-scales-in-agentic-reinforcement-learning-laws-and-synthesis-strategies-2610.23946…

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

Source: https://arxiv.org/html/2610.23946 Yankai ChenAffiliation:MBZUAIAffiliation:McGill UniversityXiaokun ZhangAffiliation:CityUHKChangjiang HanAffiliation:MBZUAIYe YuanAffiliation:McGill UniversityChong LiAffiliation:Columbia UniversityXue LiuAffiliation:MBZUAIAffiliation:McGill University

Abstract

Pretraining scaling laws treat training data as a one-dimensional quantity: a token count. Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generatetasks,environments, andtrajectoriesindependently and at very different costs, turning data into a structured design space. This raises a question pretraining never had to answer: given a synthesis budget,which axisshould be scaled? We address this question end-to-end with three tightly coupled contributions. First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains. Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment>>task>>trajectory. When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity. Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method’s efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes. Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale. The empirical results demonstrate that following the recipe yields up to1.7×1.7\timescompute efficiency over the dominant “scale-the-trajectories” practice.

1Introduction

Existing compute-optimal scaling laws for language model training typically reduce the training data distribution to a single scalar, the number of tokens, as exemplified by the Chinchilla scaling law[16]and its predecessor framework[19]. Recent agentic reinforcement learning (RL) works[13,30,20]inherited this single-axis abstraction, plotting performance against rollout tokens, prompt count, or training steps as if they were interchangeable. But agentic data is fundamentally different. A single “data point” simultaneously encodes a task specification, an environment with its API surface and dynamics, a sampled trajectory of model actions and tool feedback, and a reward signal, each generated by a different pipeline at a different cost. Synthesis pipelines amplify this heterogeneity: environment generators[50,4,14], task generators[36,59], and trajectory generators[10,2]act on different axes and saturate at different rates. The scalar abstraction has quietly become wrong.

This shift even changes what the central question is. With modern synthesis tooling, agentic data is effectively unbounded in supply, so the practitioner’s problem is no longer “how much more data should I collect,” but “where should I spend my synthesis budget?” Figure1makes this concrete: two training runs on Qwen3-8B with identical total tokens and FLOPs, differing only in how the budget is split across (task, environment, trajectory), reach success rates that differ by 14 percentage points. A single-axis power law cannot fit both, which means the bottleneck has moved from quantity to allocation. However, allocation in current agentic RL pipelines is largely decided by convention or convenience, not by any quantitative account of what each axis is worth.

A principled answer to “where to synthesize” requires three conceptual moves, each enabling the next: (i)decomposethe data axis into independent design dimensions and measure each independently; (ii)pricesynthesis methods by how efficiently they convert compute into “real-equivalent” data on each dimension; (iii)solvebudget allocation as a constrained optimization under the fitted laws and discounts. The order is not optional: without per-axis exponents there is no common unit in which to price synthesis, and without prices there is no objective for the optimization to minimize.

Concretely, we argue that in agentic RL, synthesis turns the data axis from a scalar into a design space, and the right scaling laws turn that design space into a solvable optimization problem. We instantiate this argument with three contributions, each enabling the next:

• •*Decomposing the design space (Section4).*We separate agentic data into three cardinality axes, task, environment, and trajectory, plus a reward-density control variable, and fit independent scaling exponents on each axis. Across Qwen3-4B/8B/14B, GRPO and PPO, five domains, and two further model families (twenty one settings in total), we observe a robust ordering: environmentαE>\alpha_{\mathrm{E}}>taskαT>\alpha_{\mathrm{T}}>trajectoryατ\alpha_{\tau}; resolving the environment axis into world and environment-structure diversity sharpens it to worldαW>\alpha_{\mathrm{W}}>environment structureαV≈\alpha_{\mathrm{V}}\approxtaskαT>\alpha_{\mathrm{T}}>trajectoryατ\alpha_{\tau}.
• •*Pricing synthesis along each axis (Section5).*Using the fitted laws as a common ruler, we measure how efficiently existing synthesis methods convert compute into real-equivalent data, capturing this as a per-axis discountκ∈[0,1]\kappa\in[0,1]and identifying critical synthetic-to-real ratiosr⋆r^{\star}at which collapse begins. We findrE⋆≈9.4≫rT⋆≈3.4≫rτ⋆≈1.4r^{\star}_{\mathrm{E}}\approx 9.4\gg r^{\star}_{\mathrm{T}}\approx 3.4\gg r^{\star}_{\tau}\approx 1.4, an ordering we link mechanistically to how grounded each axis is to external truth.
• •*Solving the allocation problem (Section6).*With laws and discounts in hand, “what should I synthesize” becomes a constrained optimization with a tractable solution. We derive a compute-optimal recipe that specifies how each data-axis budgetDDand model sizeNNshould scale with the total compute budgetCC:DE⋆∝C0.34D_{\mathrm{E}}^{\star}\propto C^{0.34},DT⋆∝C0.27D_{\mathrm{T}}^{\star}\propto C^{0.27},Dτ⋆∝C0.13D_{\tau}^{\star}\propto C^{0.13},N⋆∝C0.26N^{\star}\propto C^{0.26}, and validate the predicted allocation at a held-out compute scale. Whether the budget is allocated to generating data or building new worlds, trajectory resampling ranks last.

Together, these contributions reframe agentic data scaling: the bottleneck is not data quantity but data allocation across a structured design space.

Figure 1:Same compute, very different outcomes.Two GRPO runs on Qwen3-8B with identical total rollout tokens and gradient FLOPs, differing only in how the data budget is split across the task, environment, and trajectory axes and the reward control.Left:task success rate on AppWorld, ALFWorld, WebShop, and a held-out math suite.Right:the corresponding allocation pies.

2Related Work

Data scaling laws.In LLM pretraining, Kaplan et al.[19]and Hoffmann et al.[16]established power-law relationships between loss, parameters, and tokens, giving rise to compute-optimal training; subsequent work has refined these laws under inference constraints[38], robustness checks[34], and parameter-counting interpretations[39]. In RL for LLMs, DeepSeek-R1[13]demonstrated that pure RL with verifiable rewards scales reasoning capability, ScaleRL[20]characterized RL compute scaling under a fixed recipe, Mai et al.[30]reported a ZeroTIR-specific “Agent RL Scaling Law”, and Tan et al.[47]provided an empirical study of RL post-training scaling. In both settings, data enters as a single axis: tokens in pretraining, and rollout tokens or unique prompts in RL. Rather than adding a new global axis, we decompose the data axis itself into task, environment, and trajectory axes, and find that this decomposition is necessary to predict allocation outcomes.

Agentic data synthesis.Recent work synthesizes individual axes of agentic data: environments[50,4,14,56], tasks[36,59,48], and trajectories[10,2,57,22]. A growing body of work has documented synthesis-induced quality degradation, or model collapse, in pretraining[46,41,9,7,54]. Existing methods scale a chosen axis in isolation; we provide a framework that prices them on a common scale, finding their collapse ratios differ by an order of magnitude.

Curriculum and unsupervised environment design.GenEnv[14], self-evolving curricula[3], and difficulty-aligned task generation[36,56]address a question our framework deliberately leaves open:in what ordershould data along a given axis be presented? Our decomposition operates one level up, asking how much budget each axis deserves; curriculum operates within an axis, deciding which slice to sample at each training step. The two are complementary: our recipe specifies the per-axis budget, and curriculum methods can then schedule consumption of that budget.

3Preliminaries and Setup

3.1Agentic RL and the data tensor

A policyπθ\pi_{\theta}acts in an environment instancee∈𝒮Ee\in\mathcal{S}_{E}on a taskt∈𝒮Tt\in\mathcal{S}_{T}, producing a trajectoryτ=(o0,a0,o1,a1,…,oH)\tau=(o_{0},a_{0},o_{1},a_{1},\ldots,o_{H}), whereaia_{i}are tool or action calls andoio_{i}are observations, including environment feedback. A reward functionρ\rhoassignsR⁡(τ,t,e)=ρ⁡(τ,t,e)R(\tau,t,e)=\rho(\tau,t,e). We define the training corpus as𝒟=(𝒮T,𝒮E,{(ti,ei,τi)}i=1n,ρ)\mathcal{D}=\bigl(\mathcal{S}_{T},\mathcal{S}_{E},\{(t_{i},e_{i},\tau_{i})\}_{i=1}^{n},\rho\bigr), where𝒮T\mathcal{S}_{T}is the finite set of training tasks,𝒮E\mathcal{S}_{E}is the finite set of training environment instances, and each trajectoryτi\tau_{i}is associated with a task-environment pair(ti,ei)(t_{i},e_{i}). For later use, letnt,e=|{i:ti=t,ei=e}|,Ω={(t,e):nt,e>0}n_{t,e}=\bigl|\{\,i:t_{i}=t,\ e_{i}=e\,\}\bigr|,\Omega=\{(t,e):n_{t,e}>0\}, wherent,en_{t,e}is the number of trajectories collected for cell(t,e)(t,e)andΩ\Omegais the set of covered task-environment cells.

The data tensor induces three cardinality axes and one control variable.(1) Task axisDTD_{\mathrm{T}}:the number of unique training task specifications,DT=|𝒮T|D_{\mathrm{T}}=|\mathcal{S}_{T}|. Task specifications are deduplicated by sentence-embedding cosine distance>0.15>0.15; five alternative rules, including embedding-free ones, keepαT\alpha_{\mathrm{T}}within[0.238,0.261][0.238,0.261](AppendixG).(2) Environment axisDED_{\mathrm{E}}:the number of distinct training environment instances,DE=|𝒮E|D_{\mathrm{E}}=|\mathcal{S}_{E}|. An environment-structure instance is identified by its structural signature, a canonical hash over the typed tool/API surface, the state schema, the transition semantics, and the verifier interface; content never changes it, so a procedural seed creates a new instance only if it changes the signature. The signature is a deduplication criterion, not a claim that two instances with distinct signatures constitute distinct worlds; variation in the underlying world, meaning a distinct repository, corpus, app, or site, is reported separately as world diversityDWD_{\mathrm{W}}(Table1; AppendixB).(3) Trajectory axisDτD_{\tau}:the average number of rollouts per covered task-environment cell,Dτ=1|Ω|​∑(t,e)∈Ωnt,eD_{\tau}=\frac{1}{|\Omega|}\sum_{(t,e)\in\Omega}n_{t,e}. IncreasingDτD_{\tau}corresponds to repeated sampling within already covered task-environment cells, which is the standard practice in GRPO[43]. In the balanced full-factorial case, where every(t,e)∈𝒮T×𝒮E(t,e)\in\mathcal{S}_{T}\times\mathcal{S}_{E}has the same number of rollouts, this reduces toDτ=nDT​DED_{\tau}=\frac{n}{D_{\mathrm{T}}D_{\mathrm{E}}}.(4) Reward control variableDRD_{\mathrm{R}}:the information density of the reward signal, used as a categorical sparse/dense control rather than a cardinality axis: sparse rewards provide only an outcome-level signal, whereas dense rewards provide at least three verifiable intermediate signals per trajectory (AppendixH).

Table 1:Three levels of environment variation(Section4.4). Only L1 and L2 create new instances.

3.2Experimental testbeds

To stress-test the law’s domain invariance, we use four domains: (a)Math+code (TIR): Python interpreter as tool, training on synthetic math problems following the ZeroTIR recipe[23,30], evaluated on AIME24/25[28], MATH500[15], AMC23[29]. (b)Code agents (SWE): SWE-Gym[33]-style execution environments, evaluated on SWE-bench Verified[18]. (c)Web agents: WebArena-Lite[58,51]with the BrowserGym[6]interface. (d)UI/tool agents: AppWorld[48]and ALFWorld[45]. A fifth domain, (e)search agents(Wikipedia retrieval plus a paginated web-search API; Bamboogle[35]and a multi-hop split), serves as a robustness check. Evaluation is world-held-out on SWE and AppWorld but task-held-out on web (AppendixA). For axes that require synthesis, we use AWM[50]for environments, AutoPlay[36]/SCA[59]for tasks, and standard policy-rollout for trajectories.

Models and training.We use Qwen3-4B / 8B / 14B base checkpoints[52]as the size sweep, with Qwen3-8B as the default, plus Llama-3.1-8B[11]and Qwen2.5-7B[53].GRPO[43](default; group size 8, no critic) andPPO[40](with a separate critic of equal size; details in AppendixA) are employed as the agentic reinforcement learning algorithms. All experiments use the verl framework[44]with VerlTool[17]for multi-tool rollout. We log full per-axis sweeps; all reported numbers are means over 3 seeds with bootstrap 95% confidence intervals (CIs).

Loss-equivalent metric.Because RL with verifiable rewards has no smooth “loss,” we useL≡−log⁡psuccessL\equiv-\log p_{\mathrm{success}}wherepsuccessp_{\mathrm{success}}is task success rate on a held-out evaluation set, following[20,47]. This makes our laws directly comparable to pretraining laws and allows clean log-linear fits (AppendixF).

4Decomposing the Design Space: Per-Axis Scaling Laws

Before deciding where to synthesize data, we first quantify the marginal scaling behavior of each axis. This section implements the first of the three steps outlined in Section1: we decompose agentic data into its three cardinality axes and fit an independent scaling exponent for each.

Functional form.To make the three cardinality axes comparable, we need a functional form in which each axis contributes its own scaling term and its own exponent. Following the additive power-law tradition of pretraining scaling laws[16,19], we posit:

L⁡(DT,DE,Dτ)−L∞=A⋅DT−αT+B⋅DE−αE+C⋅Dτ−ατ+I⁡(DT,DE,Dτ),L(D_{\mathrm{T}},D_{\mathrm{E}},D_{\tau})-L_{\infty}\;=\;A\cdot D_{\mathrm{T}}^{-\alpha_{\mathrm{T}}}+B\cdot D_{\mathrm{E}}^{-\alpha_{\mathrm{E}}}+C\cdot D_{\tau}^{-\alpha_{\tau}}\;+\;I(D_{\mathrm{T}},D_{\mathrm{E}},D_{\tau}),(1)whereL∞L_{\infty}is the irreducible loss andIIcaptures cross-axis interactions that we measure separately in Section4.2. The independent exponentsαi\alpha_{i}are the loss-reduction-per-log-unit of each axis, a unit-normalized currency we use throughout the paper to compare and price synthesis methods. We call such fitted per-axis scaling relations laws for brevity; we claim the ordering of their exponents, not their values.

4.1Independent-axis exponents

Protocol.For each axis, we hold the other two at values fixed in advance by a calibration sweep (doubling an axis changes success by less than 2 points),(DE,DT,Dτ)=(256,10,240,8)(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})=(256,10{,}240,8), reused unchanged in every cell (AppendixA), and sweep the target axis over 6 nested grid points spanning32×32\times, extended to 8 points as a stability check. Each sweep is repeated for Qwen3-4B/8B/14B in the math+code domain. The sweeps match training signal: everyDED_{\mathrm{E}}andDTD_{\mathrm{T}}grid point uses the same 256,000 task-environment cells (8 rollouts each) and 1,000 updates, so rollout tokens drift by under4%4\%and per-environment repetition even falls32×32\timesalongDED_{\mathrm{E}}; onlyDτD_{\tau}scales tokens (32×32\times), and equalizing tokens or FLOPs lowersατ\alpha_{\tau}to0.070.07to0.080.08while movingαE\alpha_{\mathrm{E}}andαT\alpha_{\mathrm{T}}by at most0.010.01(AppendixD.1). We fitL=K​D−α+L∞L=KD^{-\alpha}+L_{\infty}by nonlinear weighted least squares, profilingL∞L_{\infty}over a grid and solving the inner log-linear problem in closed form (Algorithm1).

Figure 2:Per-axis scaling laws under controlled sweeps.Each panel sweeps one axis with the other two held at saturation; symbols are mean±\pmSE over 3 seeds, lines are weighted log-log fits. Across all three model sizes, environment scaling is the steepest, trajectory scaling the flattest. The exponent orderingαE>αT>ατ\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}holds with non-overlapping 95% CIs at every model size.Result.Figure2shows clean power-law fits (R2>0.97R^{2}>0.97on every panel) with the following fitted exponents (Qwen3-8B, math+code; full table in AppendixD):

αE=0.41±0.03,αT=0.26±0.02,ατ=0.11±0.02.\alpha_{\mathrm{E}}=0.41\pm 0.03,\quad\alpha_{\mathrm{T}}=0.26\pm 0.02,\quad\alpha_{\tau}=0.11\pm 0.02.(2)The orderingαE>αT>ατ\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}holds with non-overlapping CIs at every model size we tested. It also holds without any fit, as local log-log slopes never overlap across axes (49 of 49 comparisons; Mann-Whitney[31]p=2.9×10−4p=2.9\times 10^{-4}), and paired bootstrap differences areαE−αT=0.148\alpha_{\mathrm{E}}-\alpha_{\mathrm{T}}=0.148[0.109,0.187][0.109,0.187]andαT−ατ=0.150\alpha_{\mathrm{T}}-\alpha_{\tau}=0.150[0.116,0.184][0.116,0.184](AppendixD).

Interpretation.Environment diversity is the highest-leverage axis because each new environment forces the policy to generalize across different state spaces, action vocabularies, and tool-feedback dynamics, which are structural variation forms that cannot be obtained from new tasks within a fixed environment; Section4.4locates most of this leverage in world diversity. Task diversity is intermediate: more tasks expose new goal structures, but the underlying interaction grammar is fixed. Trajectory resampling has the lowest exponent because additional rollouts on already-covered (task, env) pairs primarily reduce variance within an already-explored region of policy space; once a sample is informative enough to estimate advantage well, further rollouts give diminishing returns[43,55].

4.2Cross-axis interactions

Figure 3:Cross-axis interaction.Heatmap of held-out success on Qwen3-8B (math+code) over a(DE,DT)(D_{\mathrm{E}},D_{\mathrm{T}})grid at fixedDτ=8D_{\tau}=8. Iso-cost contours (dashed) reveal a Pareto ridge: at lowDED_{\mathrm{E}}, scalingDTD_{\mathrm{T}}has near-zero return. Environment diversity is a prerequisite for task diversity to matter.Pure independence (Eq.1withI≡0I\!\equiv\!0) is a first-order approximation: tasks are bound to compatible environments (9 to 41 per task on average; AppendixB), andIIcaptures the residual coupling. Figure3shows that at lowDED_{\mathrm{E}}, scalingDTD_{\mathrm{T}}yields almost no gain, because the surface bends sharply rather than being separable. Fitting a multiplicative correctionI=−γ⋅(DEDT)−βI=-\gamma\cdot(D_{\mathrm{E}}\,D_{\mathrm{T}})^{-\beta}improvesR2R^{2}from0.930.93to0.980.98withγ=0.18,β=0.21\gamma=0.18,\beta=0.21. The substantive scientific takeaway is the prerequisite structure: environment diversity gates the value of task diversity. Practically, this means that pipelines that pour task synthesis into a small environment pool (e.g., many tasks inτ\tau-bench[1]or its small-scale variants) will see early saturation regardless of how many tasks they generate.

4.3Robustness across model sizes, algorithms, domains, and model families

Table 2:Fitted exponents across model sizes, algorithms, and domains.Numbers are mean±\pmbootstrap 95% CI half-width over 3 seeds. Note the orderingαE>αT>ατ\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}holds in all 18 settings.†No separable world axis:αE\alpha_{\mathrm{E}}is a structure exponent (Section4.4).Table2reports the fitted exponents; across all 18 (model×\timesalgo×\timesdomain) cells we measured, the orderingαE>αT>ατ\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}holds with non-overlapping CIs. Absolute exponents shift modestly: web/UI domains have largerαE\alpha_{\mathrm{E}}(more diverse tools amplify the value of environment generalization), and PPO/GRPO yield similar exponents (within 1 SE), suggesting the ordering is an algorithm-invariant property of agentic data, not an artifact of GRPO’s variance-reduction mechanics.

Other families and a search domain.Reduced 4-point sweeps preserve the ordering on Llama-3.1-8B(0.36,0.23,0.09)(0.36,0.23,0.09), Qwen2.5-7B(0.38,0.24,0.10)(0.38,0.24,0.10), and the search domain(0.42,0.25,0.12)(0.42,0.25,0.12): it holds in all 21 settings while absolute values shift by up to0.050.05, so the constants are regime-specific (AppendixD).

Reward control variable.Switching from sparse (outcome only) to dense (process-shaped) rewards[25,42]on the same data corpus yields a1.4×1.4\timesto2.1×2.1\timeseffective multiplier onDτD_{\tau}but does not changeαE\alpha_{\mathrm{E}}orαT\alpha_{\mathrm{T}}(AppendixH). We therefore treatDRD_{\mathrm{R}}as a control variable, claim no comparable reward scaling law, and fold its multiplier into an effective trajectory count in Section6.

4.4Resolving the environment axis: world versus structure

Outside math+code, our pools grow the number of distinct worldsWW(repositories, apps, sites) and of structural variants per worldVVtogether,DE=W×VD_{\mathrm{E}}=W\times V; the 128 SWE instances, for example, span 11 repositories. Math+code has a single world (a Python tool server), so itsαE\alpha_{\mathrm{E}}is a pure structure exponent (†in Table2). HoldingDED_{\mathrm{E}}, compute, and training signal fixed and moving budget fromVVtoWW, from(W,V)=(1,64)(W,V)=(1,64)to(8,8)(8,8)((4,16)(4,16)on web), raises held-out-world success at every step in SWE, UI/tool, and web, with non-overlapping CIs (e.g., from18.018.0to26.1%26.1\%on SWE; AppendixC). A joint fit givesαW=0.44±0.04\alpha_{\mathrm{W}}=0.44\pm 0.04andαV=0.28±0.03\alpha_{\mathrm{V}}=0.28\pm 0.03(a repository-only SWE sweep givesαW=0.46±0.05\alpha_{\mathrm{W}}=0.46\pm 0.05), so wherever the two separate,

αW​(0.44)>αV​(0.28)≈αT​(0.24​to​0.27)>ατ​(0.11).\alpha_{\mathrm{W}}\,(0.44)\;>\;\alpha_{\mathrm{V}}\,(0.28)\;\approx\;\alpha_{\mathrm{T}}\,(0.24\text{ to }0.27)\;>\;\alpha_{\tau}\,(0.11).World diversity is the steepest axis we measured, while structure and task diversity are not separable at our resolution: environment diversity beats task diversity through its world component. The blendedαE\alpha_{\mathrm{E}}is the composition-weighted mixture ofαW\alpha_{\mathrm{W}}andαV\alpha_{\mathrm{V}}(on SWE,0.87×0.44+0.13×0.28≈0.420.87\times 0.44+0.13\times 0.28\approx 0.42against the measured0.39±0.030.39\pm 0.03), and without a world axis, structure carries the whole generalization burden (αV=0.41\alpha_{\mathrm{V}}=0.41on math+code). Content-only variation is not environment diversity (seeds at a fixed signature give0.07±0.020.07\pm 0.02), and structural diversity keeps paying beyond factor-level coverage up to an inventory ceiling of 384 signatures (AppendixC).

5Pricing Synthesis: A Per-Axis Discount Factor

The per-axis exponents fitted in Section4provide us with a ruler for how data scales. With that ruler in hand, we can finally compare heterogeneous synthesis methods on a common scale, asking how efficiently each one converts compute into real-equivalent data along the axis it targets.

Table 3:Synthesis taxonomy along the data axes.Rows: representative synthesis methods, all run through one common harness (AppendixI). Columns: which axis each primarily scales, the effective discountκ\kappameasured atr=Ds/Dr=1r=D_{s}/D_{r}=1(equal real and synthetic), the axis-specific collapse ratior⋆r^{\star}, and rough cost per “unit” (1k tasks, 1 environment, or 1 trajectory) on Qwen3-8B. The estimator of Section5.2givesr⋆=9.4r^{\star}=9.4,3.43.4,1.41.4for AWM, AutoPlay, SWiRL.Table3maps a representative cross-section of agentic synthesis methods to the axes they primarily scale. This taxonomy alone organizes a fragmented field by giving every method a shared coordinate system; theκ\kappaandr⋆r^{\star}columns (defined in Sections5.1and5.2) make the comparison quantitative. Reduced to a common generate, filter, and train core, axis membership still explains 97% of the variance ofκ\kappa(ANOVAF⁡(2,5)=76.6F(2,5)=76.6,p<0.001p<0.001; AppendixI).

5.1The synthesis discount factorκ\kappa

LetDrD_{r}andDsD_{s}denote real and synthetic data on a target axis. We posit

L⁡(Dr,Ds)=f⁡(Dr+κ⁡(r)⋅Ds),r≡Ds/Dr,L(D_{r},D_{s})\;=\;f\!\big(D_{r}+\kappa(r)\cdot D_{s}\big),\qquad r\equiv D_{s}/D_{r},(3)whereffis the fitted single-axis law andκ⁡(r)∈[0,1]\kappa(r)\in[0,1]is a real-data-equivalent yield. Crucially,κ\kappais meaningful precisely because Section4provides a conversion rate: an exponentα\alphatells how loss responds to data, so a synthesis method’s effective contribution can be measured in real-data units. Without the per-axis exponent, “synthetic data is X% as good” would have no unit.

Estimation.For each method and axis, we hold totalD=Dr+DsD=D_{r}+D_{s}constant and sweepr∈{0,0.25,1,4,16}r\in\{0,0.25,1,4,16\}, fittingκ\kappafrom the resulting performance curve via Eq.3; collapse curves use a denser grid. Errors are bootstrapped over 3 training seeds.

Figure 4:Synthesis collapse, per axis.xx-axis: synthetic-to-real ratior=Ds/Drr=D_{s}/D_{r}(log scale).yy-axis: agent success rate on a held-out evaluation, normalized to ther=0r=0(real-only) value of the same seed group. Vertical dashed lines mark fittedr⋆r^{\star}, the ratio at which the curve drops 5% below the real-only baseline. Environment synthesis remains net-positive up tor≈9r{\approx}9, while trajectory synthesis collapses byr≈1.4r{\approx}1.4.

5.2Per-axis collapse points

We define the critical collapse ratior⋆r^{\star}as the ratio at which normalized success, past its peak, falls 5% below the real-only baseline, estimated as the0.950.95crossing of a monotone logistic fit inlog⁡r\log ron a 13-point ratio grid (AppendixJ). Figure4plots normalized success vs.rrfor the three axes (using AWM, AutoPlay, and SWiRL as representatives). Notably, we find

rE⋆≈9.4≫rT⋆≈3.4≫rτ⋆≈1.4,r^{\star}_{\mathrm{E}}\approx 9.4\;\gg\;r^{\star}_{\mathrm{T}}\approx 3.4\;\gg\;r^{\star}_{\tau}\approx 1.4,(4)an order-of-magnitude spread across axes that mirrors the exponent ordering on the same testbed and survives alternative collapse criteria (AppendixJ).

Mechanism: groundedness of the synthesis target.The ordering aligns with how strongly each axis is grounded by external truth.Environmentssynthesized as executable code are checked at every rollout step by an interpreter, providing a hard constraint that bounds distributional drift; this is consistent with the design principles of AWM[50]and DeepSeek-V3.2’s environment pipeline[26].Tasksare anchored less directly. Synthetic tasks are grounded only by whether their reward is verifiable, leaving room for distributional drift in problem complexity.Trajectoriessynthesized by sampling from the policy itself[10]are grounded only by the model’s own distribution, reproducing and amplifying its biases; this is the agentic analog of the recursive collapse documented in pretraining[46,41,9].

The synthesis-allocation paradox.Most current synthesis pipelines spend the largest share of compute on trajectory generation[2,57]. However, this exhibits both the lowest scaling exponent (ατ\alpha_{\tau}) and the lowest collapse threshold (rτ⋆r^{\star}_{\tau}). Our results suggest moving budget away from trajectory resampling and toward environment and task diversity; the next section makes this quantitative.

6Solving Allocation: A Compute-Optimal Recipe

With per-axis laws (Section4) and per-axis synthesis discounts (Section5), the question of “what should I synthesize” is no longer a heuristic call: it becomes a constrained optimization with a tractable solution. In this section, we plug the fitted exponents and discounts into a cost-aware Lagrangian to derive a compute-optimal recipe, and then validate the recipe at a held-out scale.

6.1The optimization problem

Cost model.We adopt a linear, axis-additive cost model, in which the bilinear rollout cost in model size and trajectory count is linearized at the optimum (AppendixL):

Φcompute​(N,DE,DT,Dτ)=cEκE​DE+cTκT​DT+cτκτ​Dτ+cN​N,\Phi_{\mathrm{compute}}(N,D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})\;=\;\frac{c_{\mathrm{E}}}{\kappa_{\mathrm{E}}}\,D_{\mathrm{E}}\,+\,\frac{c_{\mathrm{T}}}{\kappa_{\mathrm{T}}}\,D_{\mathrm{T}}\,+\,\frac{c_{\tau}}{\kappa_{\tau}}\,D_{\tau}\,+\,c_{N}\,N,(5)wherecic_{i}is the per-unit synthesis cost on axisii(calibrated from the USD/unit column of Table3and our infrastructure’s measured FLOP-per-step) andκi\kappa_{i}is the per-axis synthesis discount of Section5. The1/κi1/\kappa_{i}weighting is the natural correction: an axis whose synthetic data is onlyκi\kappa_{i}as effective as real data is, in budget-units,1/κi1/\kappa_{i}as expensive per real-equivalent unit. We measurecE=×10−2c_{\mathrm{E}}{=}3.0\!\times\!10^{-2},cT=×10−3c_{\mathrm{T}}{=}1.0\!\times\!10^{-3},cτ=×10−3c_{\tau}{=}3.0\!\times\!10^{-3},cN=×10−1c_{N}{=}1.5\!\times\!10^{-1}in units ofC0C_{0}(AppendixL).

The constrained optimization.Substituting the fitted law (Eq.1) and Eq.5into the budget-constrained loss-minimization problem, we obtain:

minN,DE,DT,Dτ⁡KE​DE−αE+KT​DT−αT+Kτ​Dτ−ατ+KN​N−αN+L∞s.t.Φcompute≤C,\min_{N,\,D_{\mathrm{E}},\,D_{\mathrm{T}},\,D_{\tau}}\;\;K_{\mathrm{E}}D_{\mathrm{E}}^{-\alpha_{\mathrm{E}}}+K_{\mathrm{T}}D_{\mathrm{T}}^{-\alpha_{\mathrm{T}}}+K_{\tau}D_{\tau}^{-\alpha_{\tau}}+K_{N}N^{-\alpha_{N}}+L_{\infty}\quad\text{s.t.}\quad\Phi_{\mathrm{compute}}\;\leq\;C,(6)whereαN\alpha_{N}is the model-size exponent fitted by extending Section4’s sweep across the 4B/8B/14B series (AppendixEreportsαN=±0.04\alpha_{N}{=}0.30\!\pm\!0.04).

Lagrangian solution.Formingℒ=(Eq.6)+λ⋅(Φcompute−C)\mathcal{L}=(\text{Eq.~\ref{eq:opt}})+\lambda\!\cdot\!(\Phi_{\mathrm{compute}}-C)and applying KKT yields, for each design variableDiD_{i}, we have:

αi​Ki​Di−αi−1ci/κi=λ⟹Di⋆∝(κi​αi​Kici)1αi+1⋅λ−1αi+1,\frac{\alpha_{i}K_{i}\,D_{i}^{-\alpha_{i}-1}}{c_{i}/\kappa_{i}}\;=\;\lambda\quad\Longrightarrow\quad D_{i}^{\star}\;\propto\;\left(\frac{\kappa_{i}\,\alpha_{i}\,K_{i}}{c_{i}}\right)^{\!\!\frac{1}{\alpha_{i}+1}}\!\cdot\,\lambda^{-\frac{1}{\alpha_{i}+1}},(7)and analogously forN⋆N^{\star}. The Lagrange multiplierλ\lambdais then determined by the budget constraint, and substituting it back yields a separate power-law dependence onCCfor each axis.

6.2Fitted recipe and IsoFLOP analysis

Figure 5:Compute-optimal allocation derived from the decomposed law.(a)The Lagrangian-optimal values of the four design variables(N,DE,DT,Dτ)(N,D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})scale as power laws in total computeCC(Eq.8). Lines are the predicted slopes; points are the empirical optima found by fine-grid search at six IsoFLOP levels with 3 seeds each.(b)Held-out task success rate vs. compute under three allocation strategies, all sharing the same budgetCCat every point: our recipe (Eq.8, solid blue), a Chinchilla-transferred baseline that treatsD=DE+DT+DτD{=}D_{\mathrm{E}}{+}D_{\mathrm{T}}{+}D_{\tau}as a scalar (dashed orange), and the dominant “naive” practice of pouring budget into trajectory rollouts (dotted grey). Smooth curves are predictions of Eq.1under each recipe’s allocation; markers with error bars are 3-seed measurements at the IsoFLOP grid points; the black star is the held-out validation run on Qwen3-14B atC=32​C0C{=}32C_{0}.Setup.We sweep total compute over six logarithmically-spaced points,C∈{1,2,4,8,16,32}×C0C\in\{1,2,4,8,16,32\}\times C_{0}, whereC0≈×1018C_{0}\approx 2.4\!\times\!10^{18}FLOPs is a base unit calibrated against a single Qwen3-4B GRPO run on the math+code testbed. For eachCC, we (i) solve Eq.6numerically using the fitted(Ki,αi,κi,ci)(K_{i},\alpha_{i},\kappa_{i},c_{i})from Sections4and5, and (ii) verify the prediction empirically by training at the predicted optimum and at four perturbations±30%\pm 30\%along each axis. Three seeds per configuration.

The recipe.Fitting power laws to the Lagrangian-optimal values acrossCC(Fig.5a) gives:

DE⋆∝C0.34,DT⋆∝C0.27,Dτ⋆∝C0.13,N⋆∝C0.26.\boxed{\;\;D_{\mathrm{E}}^{\star}\propto C^{0.34},\;\;D_{\mathrm{T}}^{\star}\propto C^{0.27},\;\;D_{\tau}^{\star}\propto C^{0.13},\;\;N^{\star}\propto C^{0.26}.\;\;}(8)The four exponents sum to approximately one (0.34+0.27+0.13+0.26=1.000.34+0.27+0.13+0.26=1.00), as they do in every domain (AppendixM). Two qualitative observations carry over directly from Sections4and5:(i)The data-axis exponents are ordered identically to the loss-side exponents,αE>αT>ατ\alpha_{\mathrm{E}}{>}\alpha_{\mathrm{T}}{>}\alpha_{\tau}. The optimization preserves the per-axis ordering, i.e., an axis with a steeper loss curve and a higher synthesis discount earns a larger share of the budget.(ii)Compared with Chinchilla’sD∝C0.5D\propto C^{0.5}, our combined “data exponent”0.34+0.27+0.13=0.740.34+0.27+0.13=0.74is larger, with model size correspondingly smaller (0.260.26vs. Chinchilla’s0.50.5). This is consistent with Section4’s finding that agentic RL benefits more from data variety than from raw model scale, a regime shift relative to next-token pretraining.

Comparison to baselines.Figure5b compares the recipe to two natural alternatives:Naive: place essentially all budget onDτD_{\tau}(the dominant practice in current open-source agentic RL pipelines[2,57]), withNN,DED_{\mathrm{E}},DTD_{\mathrm{T}}pinned to small constants;Chinchilla-transferred: treatD=DE+DT+DτD=D_{\mathrm{E}}+D_{\mathrm{T}}+D_{\tau}as a scalar, allocateD∝C0.5D\propto C^{0.5}split equally across the three axes, andN∝C0.5N\propto C^{0.5}following Hoffmann et al.[16]. All three recipes consume identical total compute at everyCC(we rescale each shape to satisfy Eq.5with equality). At smallCC(≤4​C0\leq 4C_{0}) the three recipes are within∼1\sim 1point of each other, so recipe choice barely matters when budgets are tight. The recipes diverge asCCgrows: atC=32​C0C{=}32C_{0}, our recipe reaches51.1%51.1\%predicted (51.3%51.3\%measured), Chinchilla-transferred reaches47.3%47.3\%, and naive reaches42.8%42.8\%, an absolute gap of8.38.3points at the largest scale we measured.

6.3Held-out validation

To test whether the recipe extrapolates beyond the regime it was fitted on, we predict the optimal allocation at a compute scale and model size held out from the fit, and verify it with a training run.

Protocol.We fit Eq.8using only the four smaller compute levelsC∈{1,2,4,8}×C0C\in\{1,2,4,8\}\times C_{0}on Qwen3-4B and Qwen3-8B, then predict the optimal allocation atCtest=32​C0C_{\mathrm{test}}=32C_{0}on Qwen3-14B, a held-out point in both compute scale and model size. The prediction yields(DE,DT,Dτ,N)=(500,9,800,30,14​B)(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau},N)=(500,\,9{,}800,\,30,\,14\mathrm{B})atCtestC_{\mathrm{test}}. We then run a single Qwen3-14B GRPO training job at this allocation, repeated over 3 seeds, on the math+code testbed. The predictedDE=500D_{\mathrm{E}}=500exceeds the 384 signatures our generator inventory can realize (Section4.4), so the validation pool adds 116 world-structure pairs in which an existing signature is instantiated on a different world; only 384 of its 500 units are structurally unique.

Results and analysis.The predicted held-out success rate from Eq.1is51.7%±1.6%51.7\%\pm 1.6\%; the measured value is51.3%±1.4%51.3\%\pm 1.4\%over 3 seeds, well within the prediction interval (Fig.5b, black star). Running the same training budget under the naive recipe yields42.9%±1.3%42.9\%\pm 1.3\%, and under Chinchilla-transferred47.3%±1.5%47.3\%\pm 1.5\%. Our recipe wins by4.04.0to8.48.4absolute points, corresponding to a1.4×1.4\timesto1.7×1.7\timescompute-efficiency gain: the Chinchilla-transferred allocation would need≈1.4​Ctest\approx 1.4C_{\mathrm{test}}of compute to reach our measured success rate, while our recipe matches the naive allocation’s success rate with only≈0.6​Ctest\approx 0.6C_{\mathrm{test}}. The recipe is not over-fit to our testbed or cost model: its exponents shift by less than±0.04\pm 0.04when re-derived on SWE and web data (AppendixM), by less than0.050.05under±50%\pm 50\%cost perturbations (AppendixN), and by at most0.060.06(0.7 points of predicted success atC=32​C0C=32C_{0}) when the measured horizon, latency, failure, and utilization costs are added, always toward tasks and trajectories (AppendixL). Under SWE coefficients, however, the environment and task exponents become indistinguishable (0.280.28vs.0.290.29), so we do not claim thatDED_{\mathrm{E}}dominates everywhere; it loses its top rank only above roughly3.4×3.4\timesour highest measured fully loaded cost per environment (11.8 USD-equivalent on SWE; 2.9 to 11.8 across four domains and two clusters).

6.4Two allocation regimes: synthesis versus world acquisition

Eq.8allocatesDED_{\mathrm{E}}as our pools grow it; pricing worlds and structures separately (Section4.4) gives two regimes.Within a synthesis budget, the choice is between structural variants and tasks: where a world axis separates, their exponents are not separable (αV=0.28\alpha_{\mathrm{V}}=0.28vs.αT=0.24\alpha_{\mathrm{T}}=0.24to0.270.27) but unit costs differ 23-fold (4.10 vs. 0.18 USD), so task synthesis is the better purchase per dollar and structural variation pays mainly until the world pool’s structural space is covered, after which the inventory ceiling binds (in math+code, whereαV=0.41>αT\alpha_{\mathrm{V}}=0.41>\alpha_{\mathrm{T}}, Eq.8already prices this).For procuring new worlds,αW\alpha_{\mathrm{W}}is the steepest exponent, and acquisition ranks first per dollar until the fully loaded cost per world exceeds about3.4×3.4\timesour highest per-environment cost, roughly 40 USD-equivalent. A repository with a working harness and verifier plausibly sits above that line, so practitioners should rerun this calculation with their own costs. In both regimes, trajectory resampling remains last, which is the central practical claim of this paper.

7Conclusions and Future Work

We have argued and shown that agentic RL data is not a scalar but a structured design space whose axes carry distinct scaling exponents and distinct synthesis discounts. Decomposing the space yields a common ruler on which heterogeneous synthesis methods can be priced, and the priced laws in turn turn budget allocation into a constrained optimization. The robust ordering of per-axis exponents, the order-of-magnitude spread of synthesis collapse points, and the validated compute-optimal recipe at held-out scale together suggest that “where to spend” is a more productive question than “how much to collect” in the agentic regime. Several directions can be further explored: (i) extending the law with a continuous, fully orthogonal treatment of the reward axis and richer interaction terms; (ii) verifying the recipe at frontier model scale; and (iii) coupling our axis-level decomposition with curriculum methods that schedule within an axis. We hope the framework catalyzes a shift from pursuing data quantity to allocating across a structured design space.

Acknowledgments and Disclosure of Funding

The research presented in this paper was exclusively supported by the MBZUAI Base Fund. The authors declare no competing interests.

References

• [1]V. Barres, H. Dong, S. Ray, X. Si, and K. Narasimhan(2025)τ2\tau^{2}-Bench: evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982.Cited by:§4.2.
• [2]G. Chen, C. Sun, C. Fu, Q. Wang, Z. Huang, C. Wei, G. Chen, F. Fang, A. Argha, B. Zhao,et al.(2026)Beyond quantity: trajectory diversity scaling for code agents.arXiv preprint arXiv:2602.03219.Cited by:§1,§2,§5.2,Table 3,§6.2.
• [3]X. Chen, J. Lu, M. Kim, D. Zhang, J. Tang, A. Piché, N. Gontier, Y. Bengio, and E. Kamalloo(2025)Self-evolving curriculum for llm reasoning.arXiv preprint arXiv:2505.14970.Cited by:Appendix O,§2.
• [4]Z. Chen, Z. Zhao, K. Zhang, B. Liu, Q. Qi, Y. Wu, T. Kalluri, S. Cao, Y. Xiong, H. Tong,et al.(2025)Scaling agent learning via experience synthesis.arXiv preprint arXiv:2511.03773.Cited by:§1,§2,Table 3.
• [5]J. Cohen(1960)A coefficient of agreement for nominal scales.Educational and psychological measurement20(1),pp. 37–46.Cited by:Appendix G.
• [6]T. L. S. de Chezelles, M. Gasse, A. Lacoste, M. Caccia, A. Drouin, L. Boisvert, M. Thakkar, T. Marty, R. Assouel, S. O. Shayegan,et al.(2025)The browsergym ecosystem for web agent research.Transactions on Machine Learning Research.Cited by:§3.2.
• [7]Y. Feng, E. Dohmatob, P. Yang, F. Charton, and J. Kempe(2025)Beyond model collapse: scaling up with synthesized data requires verification.InThe Thirteenth International Conference on Learning Representations,Cited by:§2.
• [8]J. L. Fleiss(1971)Measuring nominal scale agreement among many raters..Psychological bulletin76(5),pp. 378.Cited by:Appendix G.
• [9]M. Gerstgrasser, R. Schaeffer, A. Dey, R. Rafailov, T. Korbak, H. Sleight, R. Agrawal, J. Hughes, D. B. Pai, A. Gromov,et al.(2024)Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data.InFirst Conference on Language Modeling,Cited by:§2,§5.2.
• [10]A. Goldie, A. Mirhoseini, H. Zhou, I. Cai, and C. D. Manning(2025)Synthetic data generation & multi-step rl for reasoning & tool use.arXiv preprint arXiv:2504.04736.Cited by:§1,§2,§5.2,Table 3.
• [11]A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan,et al.(2024)The llama 3 herd of models.arXiv preprint arXiv:2407.21783.Cited by:Appendix A,§3.2.
• [12]A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola(2012)A kernel two-sample test.The journal of machine learning research13,pp. 723–773.Cited by:Appendix J.
• [13]D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi,et al.(2025)Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948.Cited by:§1,§2.
• [14]J. Guo, L. Yang, P. Chen, Q. Xiao, Y. Wang, X. Juan, J. Qiu, K. Shen, and M. Wang(2025)GenEnv: difficulty-aligned co-evolution between llm agents and environment simulators.arXiv preprint arXiv:2512.19682.Cited by:§1,§2,§2,Table 3.
• [15]D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. SteinhardtMeasuring mathematical problem solving with the math dataset.InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2),Cited by:Appendix A,§3.2.
• [16]J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. Casas, L. A. Hendricks, J. Welbl, A. Clark,et al.(2022)Training compute-optimal large language models.arXiv preprint arXiv:2203.1555610.Cited by:§1,§2,§4,§6.2.
• [17]D. Jiang, Y. Lu, Z. Li, Z. Lyu, P. Nie, H. Wang, A. Su, H. Chen, K. Zou, C. Du,et al.(2025)Verltool: towards holistic agentic reinforcement learning with tool use.arXiv preprint arXiv:2509.01055.Cited by:Appendix A,§3.2.
• [18]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan(2024)SWE-bench: can language models resolve real-world github issues?.InThe Twelfth International Conference on Learning Representations,Cited by:Appendix A,§3.2.
• [19]J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei(2020)Scaling laws for neural language models.arXiv preprint arXiv:2001.08361.Cited by:§1,§2,§4.
• [20]D. Khatri, L. Madaan, R. Tiwari, R. Bansal, S. S. Duvvuri, M. Zaheer, I. S. Dhillon, D. Brandfonbrener, and R. Agarwal(2025)The art of scaling reinforcement learning compute for llms.arXiv preprint arXiv:2510.13786.Cited by:Appendix F,§1,§2,§3.2.
• [21]K. Levenberg(1944)A method for the solution of certain non-linear problems in least squares.Quarterly of applied mathematics2(2),pp. 164–168.Cited by:§D.2.
• [22]X. Li, W. Jiao, J. Jin, G. Dong, J. Jin, Y. Wang, H. Wang, Y. Zhu, J. Wen, Y. Lu,et al.(2026)Deepagent: a general reasoning agent with scalable toolsets.InProceedings of the ACM Web Conference 2026,pp. 2219–2230.Cited by:§2.
• [23]X. Li, H. Zou, and P. Liu(2025)Torl: scaling tool-integrated rl.arXiv preprint arXiv:2503.23383.Cited by:Appendix A,§3.2.
• [24]Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang(2023)Towards general text embeddings with multi-stage contrastive learning.arXiv preprint arXiv:2308.03281.Cited by:Appendix G.
• [25]H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe(2023)Let’s verify step by step.InThe twelfth international conference on learning representations,Cited by:Appendix H,§4.3.
• [26]A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong,et al.(2025)Deepseek-v3. 2: pushing the frontier of open large language models.arXiv preprint arXiv:2512.02556.Cited by:§5.2.
• [27]B. Liu, L. Guertler, S. Yu, Z. Liu, P. Qi, D. Balcells, M. Liu, C. Tan, W. Shi, M. Lin,et al.(2025)Spiral: self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning.arXiv preprint arXiv:2506.24119.Cited by:Table 3.
• [28]MAAAmerican Invitational Mathematics Examination (AIME).Note:Mathematics Competition SeriesExternal Links:LinkCited by:Appendix A,§3.2.
• [29]MAAAmerican Mathematics Competitions (AMC 10/12).Note:Mathematics Competition SeriesExternal Links:LinkCited by:Appendix A,§3.2.
• [30]X. Mai, H. Xu, Z. Li, W. Wang, J. Hu, Y. Zhang, W. Zhang,et al.(2025)Agent rl scaling law: agent rl with spontaneous code execution for mathematical problem solving.arXiv preprint arXiv:2505.07773.Cited by:Appendix A,§1,§2,§3.2.
• [31]H. B. Mann and D. R. Whitney(1947)On a test of whether one of two random variables is stochastically larger than the other.The annals of mathematical statistics,pp. 50–60.Cited by:§D.3,§4.1.
• [32]D. W. Marquardt(1963)An algorithm for least-squares estimation of nonlinear parameters.Journal of the society for Industrial and Applied Mathematics11(2),pp. 431–441.Cited by:§D.2.
• [33]J. Pan, X. Wang, G. Neubig, N. Jaitly, H. Ji, A. Suhr, and Y. Zhang(2025)Training software engineering agents and verifiers with swe-gym.InForty-second International Conference on Machine Learning,Cited by:Appendix A,§3.2.
• [34]T. Pearce and J. Song(2024)Reconciling kaplan and chinchilla scaling laws.arXiv preprint arXiv:2406.12907.Cited by:§2.
• [35]O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis(2023)Measuring and narrowing the compositionality gap in language models.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp. 5687–5711.Cited by:Appendix A,§3.2.
• [36]R. Ramrakhya, A. Szot, O. Attia, Y. Yang, A. Nguyen, B. Mazoure, Z. Gan, H. Agrawal, and A. Toshev(2025)Scaling synthetic task generation for agents via exploration.arXiv preprint arXiv:2509.25047.Cited by:Appendix A,§1,§2,§2,§3.2,Table 3.
• [37]S. Robertson and H. Zaragoza(2009)The probabilistic relevance framework: bm25 and beyond.Foundations and trends® in information retrieval4(1-2),pp. 1–174.Cited by:Appendix A.
• [38]N. Sardana, J. Portes, S. Doubov, and J. Frankle(2023)Beyond chinchilla-optimal: accounting for inference in language model scaling laws.arXiv preprint arXiv:2401.00448.Cited by:§2.
• [39]R. Schaeffer, N. Levi, A. Kirsch, T. Guenais, B. Miranda, E. Obbad, and S. Koyejo(2025)Evaluating the robustness of chinchilla compute-optimal scaling.arXiv preprint arXiv:2509.23963.Cited by:§2.
• [40]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov(2017)Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347.Cited by:§3.2.
• [41]M. E. A. Seddik, S. Chen, S. Hayou, P. Youssef, and M. A. DEBBAH(2024)How bad is training on synthetic data? a statistical analysis of language model collapse.InFirst Conference on Language Modeling,Cited by:§2,§5.2.
• [42]A. Setlur, C. Nagpal, A. Fisch, X. Geng, J. Eisenstein, R. Agarwal, A. Agarwal, J. Berant, and A. Kumar(2024)Rewarding progress: scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146.Cited by:§4.3.
• [43]Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu,et al.(2024)Deepseekmath: pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300.Cited by:§3.1,§3.2,§4.1.
• [44]G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu(2025)Hybridflow: a flexible and efficient rlhf framework.InProceedings of the Twentieth European Conference on Computer Systems,pp. 1279–1297.Cited by:Appendix A,§3.2.
• [45]M. Shridhar, X. Yuan, M. Cote, Y. Bisk, A. Trischler, and M. Hausknecht(2021)ALFWorld: aligning text and embodied environments for interactive learning.InInternational Conference on Learning Representations,Cited by:Appendix A,§3.2.
• [46]I. Shumailov, Z. Shumaylov, Y. Zhao, Y. Gal, N. Papernot, and R. Anderson(2023)The curse of recursion: training on generated data makes models forget.arXiv preprint arXiv:2305.17493.Cited by:§2,§5.2.
• [47]Z. Tan, H. Geng, X. Yu, M. Zhang, G. Wan, Y. Zhou, Q. He, X. Xue, H. Zhou, Y. Fan,et al.(2025)Scaling behaviors of llm reinforcement learning post-training: an empirical study in mathematical reasoning.arXiv preprint arXiv:2509.25300.Cited by:Appendix F,§2,§3.2.
• [48]H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian(2024)Appworld: a controllable world of apps and people for benchmarking interactive coding agents.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp. 16022–16076.Cited by:Appendix A,§2,§3.2.
• [49]L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei(2024)Improving text embeddings with large language models.InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: long papers),pp. 11897–11916.Cited by:Appendix G.
• [50]Z. Wang, C. Xu, B. Liu, Y. Wang, S. Han, Z. Yao, H. Yao, and Y. He(2026)Agent world model: infinity synthetic environments for agentic reinforcement learning.arXiv preprint arXiv:2602.10090.Cited by:Appendix A,§1,§2,§3.2,§5.2,Table 3.
• [51]Z. Wei, W. Yao, Y. Liu, W. Zhang, Q. Lu, L. Qiu, C. Yu, P. Xu, C. Zhang, B. Yin,et al.(2025)Webagent-r1: training web agents via end-to-end multi-turn reinforcement learning.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp. 7920–7939.Cited by:§3.2.
• [52]A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:Appendix A,Appendix E,Appendix I,§3.2.
• [53]A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei,et al.(2024)Qwen2.5 technical report.arXiv preprint arXiv:2412.15115.Cited by:Appendix A,§3.2.
• [54]B. Yi, Q. Liu, Y. Cheng, and H. Xu(2025)Escaping model collapse via synthetic data verification: near-term improvements and long-term convergence.arXiv preprint arXiv:2510.16657.Cited by:§2.
• [55]Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu,et al.(2025)Dapo: an open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476.Cited by:§4.1.
• [56]Z. Zeng, H. Ivison, Y. Wang, L. Yuan, S. S. Li, Z. Ye, S. Li, J. He, R. Zhou, T. Chen,et al.(2025)Rlve: scaling up reinforcement learning for language models with adaptive verifiable environments.arXiv preprint arXiv:2511.07317.Cited by:§2,§2,Table 3.
• [57]H. Zhang, X. Liu, B. Lv, X. Sun, B. Jing, I. L. Iong, Z. Hou, Z. Qi, H. Lai, Y. Xu,et al.(2025)Agentrl: scaling agentic reinforcement learning with a multi-turn, multi-task framework.arXiv preprint arXiv:2510.04206.Cited by:§2,§5.2,§6.2.
• [58]S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried,et al.(2024)WebArena: a realistic web environment for building autonomous agents.InThe Twelfth International Conference on Learning Representations,Cited by:Appendix A,§3.2.
• [59]Y. Zhou, S. Levine, J. Weston, X. Li, and S. Sukhbaatar(2025)Self-challenging language model agents.arXiv preprint arXiv:2506.01716.Cited by:Appendix A,§1,§2,§3.2,Table 3.

Contents

1. 1Introduction
2. 2Related Work
3. 3Preliminaries and Setup1. 3.1Agentic RL and the data tensor 2. 3.2Experimental testbeds
4. 4Decomposing the Design Space: Per-Axis Scaling Laws1. 4.1Independent-axis exponents 2. 4.2Cross-axis interactions 3. 4.3Robustness across model sizes, algorithms, domains, and model families 4. 4.4Resolving the environment axis: world versus structure
5. 5Pricing Synthesis: A Per-Axis Discount Factor1. 5.1The synthesis discount factorκ\kappa 2. 5.2Per-axis collapse points
6. 6Solving Allocation: A Compute-Optimal Recipe1. 6.1The optimization problem 2. 6.2Fitted recipe and IsoFLOP analysis 3. 6.3Held-out validation 4. 6.4Two allocation regimes: synthesis versus world acquisition
7. 7Conclusions and Future Work
8. References
9. AExperimental Setup
10. BEnvironment Construction and Statistics
11. CWorld and Structure Diversity
12. DSweep Protocol and Fitting Methodology1. D.1Training-signal accounting 2. D.2Fitting methodology 3. D.3Stability of the ordering
13. EModel-Size Exponent Fit
14. FLoss-Equivalent Metric
15. GTask Individuation: Threshold, Definition, and Human Validation
16. HReward Density as a Control Variable
17. ISynthesis Discount Estimation and Common Harness
18. JCollapse-Ratio Estimation
19. KLagrangian Solution to The Allocation Problem
20. LCost-Coefficient Calibration and Refined Cost Models
21. MCross-Domain Robustness of The Recipe
22. NCost-Coefficient Sensitivity
23. ONegative Results and Failed Fits
24. PReproducibility
25. QCompute Disclosure
26. RLimitation Analysis
27. SEthics Statement
28. TBroader Impacts
29. UDeclaration of LLM usage

Appendix AExperimental Setup

We introduce our experimental setup from multiple aspects as follows.

Models.All experiments use Qwen3 base checkpoints[52]at 4B, 8B, and 14B parameters. We use the base (non-instruct) variants throughout to isolate the effect of RL training data; preliminary experiments with instruct variants gave qualitatively identical scaling-law exponents (within±0.02\pm 0.02) but higher absolute success rates due to instruction-following priors. For the model-family check of Section4.3, we additionally run reduced 4-point sweeps on Llama-3.1-8B[11]and Qwen2.5-7B[53]with the hyperparameters below, without retuning (AppendixD).

Hyperparameters.For GRPO we use group sizeG=8G=8, learning rate×10−61\!\times\!10^{-6}with linear warmup over the first 10% of steps and constant thereafter, KL coefficientβ=0.001\beta=0.001on a frozen reference policy initialized from the base model, clip ratioϵ=0.2\epsilon=0.2, and global batch size 256 prompts. Maximum sequence length is 8192 tokens for math+code and 16384 for SWE/web/UI. We train each configuration for 1000 gradient steps unless otherwise specified, which empirically reaches the success-rate plateau on every domain we tested.

PPO.PPO uses the same settings plus a critic. The critic is a separate copy of the same base checkpoint as the policy, with an equal parameter count and no shared trunk. Its value head is one linear layer on the final hidden state, with zero-initialized weights and a bias set to the mean return of a 200-rollout warm-up, followed by 20 critic-only steps. Critic and policy receive identical token sequences, including tool observations; values are predicted per token, but only policy-generated tokens receive value targets, and tool-observation tokens are masked from the value loss. We use a discount factor of1.01.0, GAEλ=0.95\lambda=0.95, value-loss coefficient0.50.5, value clipping at0.20.2, critic learning rate×10−61\!\times\!10^{-6}, and one critic epoch per policy epoch. Rewards are not whitened: the reward is binary (0 or 1) with a−0.1-0.1format penalty, advantages are batch-normalized after GAE, and the KL coefficient to the frozen base-model reference isβ=0.001\beta=0.001. At convergence, the critic’s explained variance is0.61±0.040.61\pm 0.04and its TD residual is stationary over the last 200 steps.

Infrastructure.Training uses the verl framework[44]with VerlTool[17]for multi-tool rollout. Rollouts are run on vLLM 0.6.3 with prefix caching; gradient updates use FSDP with mixed-precision bf16 weights and fp32 optimiser states. All experiments run on H100 80GB GPUs, with node sizes ranging from 8 GPUs (4B/8B) to 32 GPUs (14B).

Datasets per domain.Math+code (TIR):training prompts are 12k synthetic math problems following the ZeroTIR recipe[23,30]; evaluation on AIME24/25[28](30 problems each), MATH500[15], and AMC23[29](40 problems).SWE:training on SWE-Gym[33](2.4k issues across 11 repos); evaluation on the 500 instances of SWE-bench Verified[18]. The 12 repositories behind SWE-bench Verified do not overlap with the 11 training repositories, so SWE evaluation is world-held-out.Web:training on WebArena-Lite[58]sub-environments (5 sites×\times200 tasks); evaluation on a held-out partition of 240 tasks. These tasks reuse the same five sites, so we report web results as task-held-out rather than site-held-out.UI/tool:training on AppWorld[48]“train” split (250 tasks×\times9 apps) and ALFWorld[45]; evaluation on the AppWorld test partition, whose challenge split contains 417 tasks that each require at least one app designated unseen at training time.Search:a retrieval tool over a Wikipedia dump plus a paginated web-search API; evaluation on Bamboogle[35]and a held-out multi-hop split. Environment instances differ in the retrieval backend (BM25[37], a dense retriever, or the paginated web-search API), in snippet grammar and rate-limit semantics, and in whether a page-fetch tool exists. This domain is used for the robustness check of Section4.3on Qwen3-8B. For axes that require synthesis we use AWM[50]for environments, AutoPlay[36]/ SCA[59]for tasks, and standard policy-rollout for trajectories.

Held-axis values.For each per-axis sweep, the two non-swept axes are held at values fixed by one calibration sweep on Qwen3-8B math+code. The sweep was run before any exponent was fitted, under a criterion fixed in advance: doubling an axis changes absolute held-out success by less than 2 points. Table4reports it in full so that the choice can be checked. The selected values(DE,DT,Dτ)=(256,10,240,8)(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})=(256,10{,}240,8)satisfy the criterion (doubling them changes success by0.30.3,0.30.3, and0.40.4points) and were reused unchanged across every cell of Table2, every cross-domain refit, and every sensitivity analysis, with no per-cell retuning (AppendixP). To check that the conclusions do not hinge on this choice, Table5re-runs all three sweeps with the held axes at a quarter, a half, and double these values. The ordering is preserved in all four settings, and the drift at a quarter is the one predicted by the gating result of Section4.2: with a small environment pool, every axis is worth less.

Table 4:Calibration sweep that fixed the held-axis values (Qwen3-8B, math+code). It was run before any exponent was fitted, under the pre-specified criterion of AppendixA.Table 5:Fitted exponents (Qwen3-8B, math+code) when the held axes are set to a quarter, a half, or double the calibrated values.

Appendix BEnvironment Construction and Statistics

Structural signature.An environment instance consists of four components: the tool/API surface with typed signatures, the state schema, the transition semantics (how a call mutates state, the observation grammar, the error taxonomy, and statefulness or non-determinism), and the verifier interface. Two instances are the same environment if and only if their structural signature, a canonical hash over these four components, matches. Content such as problem text, seeds, or initial states never changes the signature, so a procedural seed creates a new instance only when it changes the signature. AWM and GenEnv sample tool composition and schema, so most generator seeds do change it; seeds that change only content are collapsed. The signature is a deduplication criterion and does not claim that instances with distinct signatures are distinct worlds; AppendixCtreats world diversity separately.

Per-domain instantiation.Inmath+code (TIR), an environment is the Python tool server. Instances differ, for example, in a statelesspython_execversus a persistent REPL in which variables survive across turns, a one-tool server versus a three-tool server (python_exec,sympy_solve,unit_check) that requires piping between tools, a whitelist that raisesImportErroronnumpy, or a 200-character stdout cap that requires pagination. A new math problem is a task. InSWE, an environment is a configuration of container image, dependency graph, build and test harness, and agent toolset. Instances differ in the test runner and its output grammar (pytest, unittest, or tox; JUnit XML or raw stderr), which changes the feedback the policy must parse; in the patch interface (line-range edit, whole-file write, or diff apply), which changes the action vocabulary; and in whether a repository-search tool exists. A new GitHub issue against the same repository and harness is a task, not an environment.WebandUI/toolenvironments are sites and MCP-style servers with distinct action spaces, entity schemas and business logic, authentication and permission models, pagination semantics, and error taxonomies. Thesearchdomain varies the retrieval backend and its result grammar (AppendixA).

Pool statistics.Table6characterizes each domain’s pool at its largest grid point. Instances fall into four levels: S3 if the API set differs, S2 if the API set matches but the state schema differs, S1 if both match but the transition semantics differ, and S0 if the signature is identical, in which case the instance is collapsed and not counted. Transition divergence is the fraction of canonical length-3 call prefixes that yield a different observation type across instances sharing a tool name; it is 0 for S0 pairs by construction, so it measures dynamics rather than content. No counted instance differs only in seed or content. In math+code, for example, 2,048 raw generator outputs collapse to the 256 counted instances, of which 94 differ in API set, 88 in state schema with a matching API set, and 74 only in transition semantics.

Table 6:Environment pools at the largest grid point of each domain. The Jaccard columns are mean pairwise Jaccard distances between tool sets and between state schemas; S3, S2, and S1 count instances whose API set, state schema (with matching API set), or only transition semantics differ; the last column shows raw generator outputs collapsed to signature-distinct instances.Signature-restricted and content-only pools.Table7refitsαE\alpha_{\mathrm{E}}(Qwen3-8B, math+code) on pools restricted by signature level, and on a placebo sweep that holds the signature fixed and scales only procedural seeds from 8 to 256. Restricting the pool to instances that differ in API set, schema, or transitions leavesαE\alpha_{\mathrm{E}}between0.390.39and0.420.42, whereas content-only variation yields0.07±0.020.07\pm 0.02: less than an extra rollout (ατ=0.11\alpha_{\tau}=0.11) and about one sixth of structural variation.

Table 7:Environment exponent under alternative pool constructions (Qwen3-8B, math+code).Task-environment binding.Tasks and environments are not orthogonal, and Eq.1does not assume they are. Task specifications are authored against an abstract goal schema and bound to an environment through a compatibility check, so one task can be instantiated in many environments. TheDED_{\mathrm{E}}sweep re-binds the same 10,240 task specifications to different environment pools, holding the number of distinct tasks consumed constant at every grid point. The mean number of environments to which a task can be bound is 41 (math+code), 18 (SWE), 9 (web), and 27 (UI/tool), and the realizable fraction of the task×\timesenvironment grid is 78%, 54%, 41%, and 63%, respectively. The residual coupling is what the interaction term of Section4.2quantifies (γ=0.18\gamma=0.18,β=0.21\beta=0.21); its content, that environment diversity gates task diversity, is a result rather than a nuisance.

Appendix CWorld and Structure Diversity

Three levels and pool composition.Table1distinguishes content (L0), environment structure (L1), and world (L2); content variation is never counted as environment diversity. Our pools contain different amounts of world variation. Math+code has a single world, a Python tool server, so all of its instances are L1 variants and itsαE\alpha_{\mathrm{E}}should be read as an environment-structure exponent. The 128 SWE instances are built over 11 real repositories with roughly 12 structural variants each, soDE=128D_{\mathrm{E}}=128counts structures rather than worlds; what the definition does fix is the boundary that a new issue against an unchanged repository and harness is a task. Web instances span the 5 WebArena sites. Outside math+code, instances differ in the state itself and not only in wrappers around one state: through distinct entity schemas and business logic, authentication and permission models, pagination and error taxonomies, and, for search, retrieval backends with different snippet grammars and rate-limit semantics.

World-structure swap.At fixedDE=W×VD_{\mathrm{E}}=W\times Vand fixed compute, we sweep the split between the number of distinct worldsWWand the number of structural variants per worldVV, evaluating on held-out worlds (SWE-bench Verified for SWE and the unseen-app split for UI/tool). SWE uses repositories as worlds with(W,V)∈{(1,64),(2,32),(4,16),(8,8)}(W,V)\in\{(1,64),(2,32),(4,16),(8,8)\}, whereWWruns to 8 rather than 11 to keep the grid dyadic; UI/tool uses apps (8 of the 9 in AppWorld) on the same grid; and web uses its 5 sites with the reduced grid{(1,64),(2,32),(4,16)}\{(1,64),(2,32),(4,16)\}. To reachV=64V=64within a single world, we expanded the per-world variant pool with AWM beyond the roughly 12 variants per repository of the Table2pool, with signature deduplication on the same criterion.WWis capped by the number of real worlds we can obtain rather than by budget, and we do not extrapolate past it. Trajectories, gradient updates, distinct task specifications, and normalized FLOPs are matched across grid points by the construction of AppendixD.1, so the comparison is iso-compute and iso-signal.

Table 8:Held-out-world success (%) as budget moves from structural variants per world (VV) to worlds (WW) at fixedDE=W×VD_{\mathrm{E}}=W\times Vand fixed compute (3 seeds, bootstrap 95% CI half-widths). Column headers give(W,V)(W,V). Web has five sites, so its grid stops atW=4W=4(n/a). The last column gives the reductionΔ​L\Delta LofL=−log⁡psuccessL=-\log p_{\mathrm{success}}at each successive doubling ofWW.Results.We give the fit-free reading first. At every grid transition in all three domains, moving budget fromVVtoWWat fixedDED_{\mathrm{E}}improves held-out-world success, with no overlapping intervals within any domain, and in the loss-equivalent metric the improvements decelerate over successive doublings ofWW(Table8). No functional form is needed for this statement. A joint fit over these points, withL∞L_{\infty}fixed at the value of AppendixDand one scale constant per domain, givesαW=0.44±0.04\alpha_{\mathrm{W}}=0.44\pm 0.04andαV=0.28±0.03\alpha_{\mathrm{V}}=0.28\pm 0.03; we report this interval as the fit’s own and do not claim that a two-exponent fit over so few points supports fine distinctions. A repository-only sweep on SWE at fixed harness,W∈{1,2,4,8,11}W\in\{1,2,4,8,11\}, independently givesαW=0.46±0.05\alpha_{\mathrm{W}}=0.46\pm 0.05; it spans a single order of magnitude, and its width reflects that limitation. The substitution rate at the(4,16)(4,16)operating point, defined as the factor by whichVVmust be multiplied to match the held-out-world gain from doublingWW, is approximately3.03.0; it is a local rate rather than a global constant.

Mixture check.On a pool that growsWWandVVtogether, which is what AWM and GenEnv do when run as intended, the effective slope inlog⁡DE\log D_{\mathrm{E}}is the composition-weighted mixture ofαW\alpha_{\mathrm{W}}andαV\alpha_{\mathrm{V}}. In SWE, the sweep of Table2spansWWby about11×11\timesbutVVby only about1.5×1.5\times, so roughly0.870.87of the log-increment is world growth, which predicts a blended exponent of0.87×0.44+0.13×0.28≈0.420.87\times 0.44+0.13\times 0.28\approx 0.42against the measured0.39±0.030.39\pm 0.03. In math+code the world weight is zero, so the blend reduces toαV\alpha_{\mathrm{V}}itself, which is why that domain reads0.410.41. The structural return is therefore domain-dependent, from0.280.28pooled over SWE, UI/tool, and web to0.410.41on math+code: where no world axis is available to absorb it, structural variation carries the whole generalization burden. For a practitioner who scales such a pool, the blendedαE\alpha_{\mathrm{E}}remains the right number, but it is never the return on world diversity alone.

Transfer to unseen worlds.Because SWE-bench Verified and AppWorld’s challenge split are world-held-out (AppendixA), the structural gains we report on SWE and AppWorld already measure generalization to worlds that were never acquired for training. Structural diversity thus yields measurable transfer even where a pool is world-poor relative to its count.

Factor inventory and coverage.One may ask whether effective structural diversity is merely the number of factor levels in our generators. After canonicalizing each domain’s generator into four domain-specific structural-factor slots, the observed level vector is(4,4,3,8)(4,4,3,8)in each of SWE, UI/tool, and web: execution or interaction mode (4 levels), tool/API or action-surface family (4), observation or verifier-output grammar (3), and state, dependency, or error-semantics regime (8). The realizable inventory is therefore4×4×3×8=3844\times 4\times 3\times 8=384signature-distinct structural combinations per domain, and the median normalized factor-space Hamming distance among sampled instances is0.750.75. A level-isolating construction first covers every observed level atDE=16D_{\mathrm{E}}=16(pool A), and a pair-complete construction first covers every observed level pair atDE=48D_{\mathrm{E}}=48(pool B); both coverage points lie well below the sampled pool C (|C|=256|C|=256) and the top sweep valueDE=384D_{\mathrm{E}}=384. Pool A is a level-isolating construction rather than a minimal strength-1 covering array, which would need only as many instances as the largest slot has levels; we use it because it also supports a matched-count control. If effective diversity were the number of levels, the curve would saturate at level coverage, or at pair coverage in the stronger pairwise form. Neither occurs. The local log-log slopes overDE=4D_{\mathrm{E}}=4to384384are0.44,0.42,0.41,0.40,0.38,0.370.44,0.42,0.41,0.40,0.38,0.37, and0.360.36, and the three segments lying entirely past pair coverage still run at0.380.38,0.370.37, and0.360.36. Regressing held-out success on the logarithm of levels covered andlog⁡DE\log D_{\mathrm{E}}jointly, the partial contribution oflog⁡DE\log D_{\mathrm{E}}given level coverage isΔ​R2=0.18\Delta R^{2}=0.18(p=0.004p=0.004). Directly, held-out success is27.827.8,31.631.6, and34.5%34.5\%for pools A, B, and C: A trails C by6.76.7points despite being level-complete, and B trails C by2.92.9points despite being pair-complete. Finally, replacingDED_{\mathrm{E}}by effective counts that combinatorics cannot inflate, namely the exponential of the factor-combination entropy and the participation ratio of the signature-distance kernel, givesαE=0.37±0.03\alpha_{\mathrm{E}}=0.37\pm 0.03and0.36±0.030.36\pm 0.03, respectively, both preserving the ordering againstαT\alpha_{\mathrm{T}}andατ\alpha_{\tau}. What the axis buys beyond level coverage is therefore higher-order combinations of factors.

Ceiling of the structure axis.The realizable combination count of the inventory is finite, and our top grid point approaches it, so the L1 axis is not indefinitely scalable by combinatorics: past the ceiling, more structural diversity requires designing new factors rather than spending more compute. This is a real limit on the L1 recommendation, and it is the strongest argument for pricing the L2 axis, which has no such ceiling (Section6.4). It also interacts with the held-out validation of Section6.3: the predicted allocation atCtest=32​C0C_{\mathrm{test}}=32C_{0}specifiesDE=500D_{\mathrm{E}}=500, which exceeds the 384 signatures realizable under the configured inventory. The validation pool therefore used all 384 signature-distinct L1 combinations plus 116 world-structure pairs in which an L1 signature is instantiated on a different world, and only 384 of its 500 units are structurally unique. The ceiling is a property of the configured generator inventory, not of environment structure in general.

Appendix DSweep Protocol and Fitting Methodology

Per-axis sweep protocol.For each axis we collect 6 nested grid points spanning32×32\times, with 3 random seeds per point (different seeds for both rollout and gradient noise):DE∈{8,16,32,64,128,256}D_{\mathrm{E}}\in\{8,16,32,64,128,256\},DT∈{320,640,1280,2560,5120,10240}D_{\mathrm{T}}\in\{320,640,1280,2560,5120,10240\}, andDτ∈{1,2,4,8,16,32}D_{\tau}\in\{1,2,4,8,16,32\}on Qwen3-8B math+code. The stability extension addsDE∈{4,384}D_{\mathrm{E}}\in\{4,384\},DT∈{160,20480}D_{\mathrm{T}}\in\{160,20480\}, andDτ∈{3,48}D_{\tau}\in\{3,48\}(54 new runs), so that every sweep has 8 points spanning48×48\timesto128×128\times. The other two axes are held at the calibrated values of AppendixA. Each (axis, scale) cell consumes between 0.5 and 20 H100-GPU-hours depending on the axis sweep; total cost of the per-axis sweeps is reported in AppendixQ.

Nesting and subset sensitivity.Pools are nested across adjacent grid points,E8⊂E16⊂⋯⊂E256E_{8}\subset E_{16}\subset\cdots\subset E_{256}andT320⊂⋯⊂T10240T_{320}\subset\cdots\subset T_{10240}, and theDτD_{\tau}sweep reuses the same cell pool. As a stronger check against differences in the sampled difficulty or quality of each subset, drawing 3 disjoint random subsets per grid point shifts the fitted exponents by at most0.010.01(αE\alpha_{\mathrm{E}}from0.410.41to0.400.40;αT\alpha_{\mathrm{T}}from0.260.26to0.270.27), with a per-point success spread of at most0.80.8points. Base-policy pass@8 is also flat along the sweeps (0.1860.186to0.1990.199acrossDED_{\mathrm{E}}and0.1810.181to0.2030.203acrossDTD_{\mathrm{T}}).

D.1Training-signal accounting

A cell draw is one task-environment pair rolled outGGtimes. Every grid point of theDED_{\mathrm{E}}andDTD_{\mathrm{T}}sweeps uses|Ω|=256,000|\Omega|=256{,}000cells, each drawn once, with 256 draws per step, 1,000 updates, andDτ=G=8D_{\tau}=G=8. IncreasingDED_{\mathrm{E}}therefore redistributes the same cells across more environments (Table9): trajectories, distinct task specifications, and updates are constant, rollout tokens drift by less than 4%, and normalized FLOPs stay between0.980.98and1.021.02. AlongDTD_{\mathrm{T}}, rollout tokens drift by2.1%2.1\%at the same constant trajectory and update counts, while theDτD_{\tau}sweep scales rollout tokens32×32\timesat a constant number of updates. Per-environment repetition falls32×32\timesalongDED_{\mathrm{E}}(32,000 visits per environment atDE=8D_{\mathrm{E}}=8against 1,000 atDE=256D_{\mathrm{E}}=256), so the steepest exponent is obtained with the least per-environment signal.

Table10refits the exponents under stricter budget conventions. Under an exact rollout-token cap and under iso-FLOP matching,αE\alpha_{\mathrm{E}}andαT\alpha_{\mathrm{T}}move by at most0.010.01whileατ\alpha_{\tau}falls, so correcting the token difference widens the gap between trajectories and the other two axes rather than closing it.

Table 9:Training-signal accounting along theDED_{\mathrm{E}}sweep (Qwen3-8B, math+code) at four grid points spanning its full range. Tasks are distinct task specifications; tokens are totals per run; FLOPs are normalized to the mean over the sweep.Table 10:Fitted exponents (Qwen3-8B, math+code) under three training-budget conventions.

D.2Fitting methodology

Functional form and estimator.We fit

L⁡(D)=K​D−α+L∞,L(D)\;=\;K\,D^{-\alpha}+L_{\infty},(9)with parameters(K,α,L∞)(K,\alpha,L_{\infty}), by nonlinear weighted least squares. BecauseL∞L_{\infty}enters nonlinearly, we solve the problem by profilingL∞L_{\infty}out (Algorithm1): for each candidateL∞L_{\infty}on a 400-point grid over[−0.5,1.0][-0.5,1.0], the inner problemlog⁡(L−L∞)=log⁡K−α​log⁡D\log(L-L_{\infty})=\log K-\alpha\log Dis linear in(log⁡K,α)(\log K,\alpha)and is solved in closed form by weighted least squares on log-loss residuals, with weightswi=1/σ^i2w_{i}=1/\hat{\sigma}_{i}^{2}, whereσ^i\hat{\sigma}_{i}is the sample standard error at grid pointiiover 3 seeds; we keep the candidate with the smallest weighted residual. Section4.1and this paragraph describe the same, single estimator. As an independent check, a Levenberg-Marquardt fit[21,32]of Eq.9in raw space gives(αE,αT,ατ)=(0.415,0.264,0.107)(\alpha_{\mathrm{E}},\alpha_{\mathrm{T}},\alpha_{\tau})=(0.415,0.264,0.107), within0.0060.006of the reported values. On synthetic data with known ground truth, the estimator recoversα\alphato within±0.015\pm 0.015across the relevant range, and on synthetic data generated with our exact design it recovers the correct ordering in 99.8% of replicas.

Algorithm 1Profiled weighted least squares forL⁡(D)=K​D−α+L∞L(D)=KD^{-\alpha}+L_{\infty}1:grid points

(Di,L¯i,σ^i)(D_{i},\bar{L}_{i},\hat{\sigma}_{i})for

i=1,…,ni=1,\ldots,n; a grid

Λ\Lambdaof 400 candidate values on

[−0.5,1.0][-0.5,1.0] 2:foreach

ℓ∈Λ\ell\in\Lambdawith

ℓ<mini⁡L¯i\ell<\min_{i}\bar{L}_{i}do

3:

xi←log⁡Dix_{i}\leftarrow\log D_{i},

yi←log⁡(L¯i−ℓ)y_{i}\leftarrow\log(\bar{L}_{i}-\ell),

wi←1/σ^i2w_{i}\leftarrow 1/\hat{\sigma}_{i}^{2} 4:

(aℓ,αℓ)←arg⁡min⁡∑i(a,α)⁡wi​(yi−a+α​xi)2(a_{\ell},\alpha_{\ell})\leftarrow\arg\min_{(a,\alpha)}\sum_{i}w_{i}\,(y_{i}-a+\alpha x_{i})^{2}⊳\trianglerightclosed form

5:

RSS⁡(ℓ)←∑iwi​(yi−aℓ+αℓ​xi)2\mathrm{RSS}(\ell)\leftarrow\sum_{i}w_{i}\,(y_{i}-a_{\ell}+\alpha_{\ell}x_{i})^{2} 6:endfor

7:

ℓ^←arg⁡minℓ⁡RSS⁡(ℓ)\hat{\ell}\leftarrow\arg\min_{\ell}\mathrm{RSS}(\ell) 8:return

K^=exp⁡(aℓ^)\hat{K}=\exp(a_{\hat{\ell}}),

α^=αℓ^\hat{\alpha}=\alpha_{\hat{\ell}},

L^∞=ℓ^\hat{L}_{\infty}=\hat{\ell}

Confidence intervals.95% CIs are computed by paired bootstrap: for each of 1000 resamples we sample 6 grid points with replacement and 3 seeds with replacement, refit, and record(K,α,L∞)(K,\alpha,L_{\infty}). The CI forα\alphais the range from the 2.5th to the 97.5th percentile of the bootstrap distribution. We chose the paired bootstrap over the delta method because the latter underestimates uncertainty in small-sample regimes (n=18n\!=\!18per fit).

Goodness-of-fit diagnostics.For all per-axis fits reported in Table2, the weightedR2R^{2}exceeds0.970.97. Residuals show no systematic pattern withDD(Durbin-Watson statistic in[1.7,2.3][1.7,2.3]across all 18 fits, consistent with no autocorrelation). We also tested an alternative two-term formL=K1​D−α1+K2​D−α2+L∞L=K_{1}D^{-\alpha_{1}}+K_{2}D^{-\alpha_{2}}+L_{\infty}; the BIC favours the single-term form (Eq.9) at every scale we measured, indicating the data does not support a two-regime fit.

D.3Stability of the ordering

Eight-point grids.Table11compares the 6-point and 8-point fits for Qwen3-8B on math+code. Adding two points per axis moves each exponent by at most0.010.01, and dropping any single grid point moves it by at most0.0150.015.

Table 11:Six-point versus eight-point fits (Qwen3-8B, math+code). The last column is the largest change inα\alphawhen any single grid point is left out.Fit-free local slopes.A highR2R^{2}obtained with few points and three parameters is weak evidence on its own, so we also report the ordering without any fit. Table12lists the local log-log slopes between adjacent points of the 8-point grids,sj=−[logL(Dj+1)−logL(Dj)]/[logDj+1−logDj]s_{j}=-[\log L(D_{j+1})-\log L(D_{j})]/[\log D_{j+1}-\log D_{j}], which involve no functional form, noL∞L_{\infty}, and no estimator. The three ranges do not overlap (0.360.36to0.440.44,0.220.22to0.290.29, and0.080.08to0.140.14), so every environment segment is steeper than every task segment, which is in turn steeper than every trajectory segment: 49 of 49 pairwise comparisons agree for each adjacent pair of axes (one-sided Mann-Whitney test[31],p=2.9×10−4p=2.9\times 10^{-4}).

Table 12:Fit-free local log-log slopessjs_{j}between adjacent grid points of the 8-point grids (Qwen3-8B, math+code).Fewer free parameters.A joint fit with a singleL∞L_{\infty}shared across the three sweeps uses 7 parameters against 54 observations (47 residual degrees of freedom) and returns(αE,αT,ατ)=(0.40,0.26,0.11)(\alpha_{\mathrm{E}},\alpha_{\mathrm{T}},\alpha_{\tau})=(0.40,0.26,0.11). FixingL∞L_{\infty}at00,0.100.10, and0.150.15instead gives 2-parameter fits of(0.36,0.23,0.10)(0.36,0.23,0.10),(0.41,0.26,0.11)(0.41,0.26,0.11), and(0.44,0.28,0.12)(0.44,0.28,0.12). The ordering holds throughout.

Differences rather than CI overlap.Non-overlapping CIs are a conservative but indirect criterion; the appropriate test is on the differences. A paired bootstrap over seeds and grid points givesαE−αT=0.148\alpha_{\mathrm{E}}-\alpha_{\mathrm{T}}=0.148with 95% interval[0.109,0.187][0.109,0.187]andαT−ατ=0.150\alpha_{\mathrm{T}}-\alpha_{\tau}=0.150with 95% interval[0.116,0.184][0.116,0.184], both withp<0.001p<0.001.

Other model families and a search domain.Table13reports reduced 4-point sweeps on two further model families and on the search-agent domain, using the hyperparameters of AppendixAwithout retuning. The ordering holds in all three; absolute exponents on Llama-3.1-8B sit0.020.02to0.050.05below Qwen3, which is why we claim the ordering rather than the constants.

Table 13:Fitted exponents on further model families and a new domain (reduced 4-point sweeps).

Appendix EModel-Size Exponent Fit

The main paper’s law (Eq.1) is fitted with(DE,DT,Dτ)(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})at fixedN=8​BN=8\mathrm{B}. To incorporate model size into Section6’s allocation problem, we additionally fit the marginalNN-axis exponent.

Protocol.Holding(DE,DT,Dτ)(D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau})at saturation values for each model size, we sweepN∈{0.6,1.7,4,8,14,32}N\in\{0.6,1.7,4,8,14,32\}B (using Qwen3-0.6B/1.7B/4B/8B/14B and Qwen3-32B[52]as the largest point) and fitL=KN​N−αN+L∞L=K_{N}\,N^{-\alpha_{N}}+L_{\infty}following the same protocol as AppendixD.

Result.The fitted exponent isαN=0.30±0.04\alpha_{N}=0.30\pm 0.04(95% CI), withKN=0.85K_{N}=0.85andL∞L_{\infty}jointly fitted to the same value as the data-axis fits (L∞=0.10±0.03L_{\infty}=0.10\pm 0.03). ThisαN\alpha_{N}is materially smaller than Chinchilla’s data-axis exponent (∼0.5\sim\!0.5), consistent with the observation in Section6.2that agentic RL benefits more from data variety than from raw model scale in the regime we measured.

Range of validity.We caution that theNN-sweep covers only0.60.6B to3232B; extrapolation to frontier scales (≥70\geq 70B) is unverified and is listed as a limitation in Section7and AppendixR.

Appendix FLoss-Equivalent Metric

Definition.Standard pretraining scaling laws fit cross-entropy loss against data and parameters. RL with verifiable rewards has no smooth loss; we use

L≡−log⁡(max⁡(psuccess,p0)),p0=10−2.L\;\equiv\;-\log\!\bigl(\max(p_{\mathrm{success}},p_{0})\bigr),\quad p_{0}=10^{-2}.(10)The floorp0p_{0}prevents divergence at zero success and corresponds to the empirical lower bound observed in our smallest-scale runs.

Why this works.The success ratepsuccessp_{\mathrm{success}}saturates at11, producing a non-power-law plateau in raw success-rate plots. The transformation toL=−log⁡pL=-\log punfolds this plateau into a power-law decay matching pretraining-style scaling, allowing clean log-log fits. This is equivalent to the “effective error” transformation used in[20,47].

Comparison to alternatives.We tested three alternative metrics on the per-axis fits: (i) rawpsuccessp_{\mathrm{success}}, (ii) logitlogit⁡(psuccess)=log⁡p1−p\mathrm{logit}(p_{\mathrm{success}})=\log\frac{p}{1-p}, and (iii) Brier score on per-task success. For (i), naive power-law fits saturate poorly (residuals>>5% even atR2=0.85R^{2}\!=\!0.85). For (ii), the fit is comparable in goodness (R2=0.96R^{2}\!=\!0.96vs. our0.970.97+) but the resulting exponents differ from pretraining-styleα\alphaby a constant factor, complicating cross-paper comparison. For (iii), the metric is too noisy for a 3-seed regime. We thus standardize on−log⁡psuccess-\log p_{\mathrm{success}}throughout.

Appendix GTask Individuation: Threshold, Definition, and Human Validation

Threshold.Task countsDTD_{\mathrm{T}}are deduplicated by sentence-embedding cosine distance>θ>\theta(Section3.1); we ablateθ∈{0.05,0.10,0.15,0.20,0.30}\theta\in\{0.05,0.10,0.15,0.20,0.30\}. Embeddings are computed with the gte-Qwen2-1.5B encoder[24]. Across this range, the fittedαT\alpha_{\mathrm{T}}varies between0.2450.245and0.2720.272on the math+code testbed at 8B (Table14), with the chosen default (θ=0.15\theta=0.15) at0.261±0.020.261\pm 0.02. The qualitative orderingαE>αT>ατ\alpha_{\mathrm{E}}>\alpha_{\mathrm{T}}>\alpha_{\tau}is preserved across all five thresholds. At lowθ\theta(e.g., 0.05), near-duplicates inflateDTD_{\mathrm{T}}but contribute little gradient information, slightly biasing the fittedα\alphadownward. At highθ\theta(e.g., 0.30), the deduplication is so aggressive that we remove genuinely distinct tasks, slightly biasing it upward. The plateau atθ∈[0.10,0.20]\theta\in[0.10,0.20]is robust, justifying our default choice.

Table 14:Ablation of the task-deduplication thresholdθ\theta. EffectiveDTD_{\mathrm{T}}is the count of unique tasks remaining after deduplication at each threshold; fittedαT\alpha_{\mathrm{T}}is robust across the range.Definition.The threshold ablation tests numerical sensitivity but not the definition itself, which is the sharper question: semantically different instructions can require nearly identical skills, while superficially similar ones can require different reasoning or tool-use patterns. We therefore refitαT\alpha_{\mathrm{T}}under five alternative individuation rules, two of them embedding-free (Table15): the raw count without deduplication; the default rule with a different encoder (e5-mistral-7b[49]); a skill signature that deduplicates by the tool-call skeleton (tool types plus argument types) of a reference solution; a verifier signature that deduplicates by the structure of the verification program; and a human-validated pool. All six values lie in[0.238,0.261][0.238,0.261], strictly betweenατ\alpha_{\tau}andαE\alpha_{\mathrm{E}}under every rule.

Table 15:Fitted task exponent under alternative definitions of a distinct task (Qwen3-8B, math+code). EffectiveDTD_{\mathrm{T}}is the number of distinct tasks atDr​a​w=10,240D_{raw}=10{,}240.Human validation.We sampled 500 task pairs stratified by embedding distance, and three annotators, volunteers from MBZUAI who participated without compensation, labeled each pair for whether it requires a materially different skill or tool-use pattern (Fleiss’ kappa0.680.68[8]). Agreement with the majority label is 87.2% for the skill signature (Cohen’s kappa0.740.74[5]), 84.6% for the verifier signature (kappa0.690.69), 80.4% for the embedding rule (kappa0.610.61, with 11.2% false merges), and 68.0% for the raw count (kappa0.360.36). The embedding rule is substantial but imperfect and the skill signature is better, so we use the skill-signature refit as the primary robustness check and keep the embedding rule as the cheap operational default. The ordering does not depend on which rule is used.

Appendix HReward Density as a Control Variable

The main paper treatsDRD_{\mathrm{R}}as a control variable with two categories (sparse outcome reward and dense process reward) and does not claim a reward scaling law comparable to those of the three cardinality axes. Here we report its marginal sensitivity under a continuous treatment via the number of process-reward signals per trajectory.

Protocol.We instantiate process rewards using a Qwen3-8B-based PRM trained following[25]on a held-out math+code corpus. For each trajectory, we extractk∈{0,1,3,8}k\in\{0,1,3,8\}process-reward signals (placed at uniformly spaced positions across the trajectory length) and append them as additional return-to-go terms.k=0k=0recovers sparse outcome reward;k=8k=8provides dense step-level supervision.

Marginal sensitivity to reward density.FittingL=KR​(k+1)−αR+L∞L=K_{R}(k+1)^{-\alpha_{R}}+L_{\infty}on the math+code testbed at 8B yields

αR=0.18±0.03,KR=0.41,L∞=0.11.\alpha_{R}=0.18\pm 0.03,\quad K_{R}=0.41,\quad L_{\infty}=0.11.(11)We reportαR\alpha_{R}as the marginal sensitivity of the loss to reward density on this testbed, not as a scaling exponent on par with those of the cardinality axes; numerically, it falls between the trajectory (ατ≈0.11\alpha_{\tau}\approx 0.11) and task (αT≈0.26\alpha_{\mathrm{T}}\approx 0.26) exponents.

Implications.Reward density and trajectory count interact. Holding the total reward-signal budgetk⋅Dτk\cdot D_{\tau}fixed, we observe approximate substitutability: doublingkkat halfDτD_{\tau}produces equivalent loss within±0.5\pm 0.5pp on the same compute. This justifies treatingDRD_{\mathrm{R}}as an effective multiplier onDτD_{\tau}in Section6.2’s allocation, rather than as an independent axis. We leave a fully orthogonal treatment of reward density to future work.

Appendix ISynthesis Discount Estimation and Common Harness

Estimator.For each (synthesis method, axis) pair we hold totalD=Dr+DsD=D_{r}+D_{s}constant and sweep the ratior=Ds/Dr∈{0,0.25,1,4,16}r=D_{s}/D_{r}\in\{0,0.25,1,4,16\}, fittingκ⁡(r=1)\kappa(r=1)from

L⁡(Dr,Ds)=f⁡(Dr+κ​Ds),L(D_{r},D_{s})\;=\;f\!\bigl(D_{r}+\kappa\,D_{s}\bigr),(12)whereffis the fitted single-axis law of Section4. Sinceffis monotone, this gives a closed-form estimator forκ\kappaat eachrr:

κ^​(r)=f−1​(L^​(Dr,Ds))−DrDs.\hat{\kappa}(r)\;=\;\frac{f^{-1}\!\bigl(\hat{L}(D_{r},D_{s})\bigr)-D_{r}}{D_{s}}.(13)We reportκ^​(r=1)\hat{\kappa}(r=1)in Table3as the headline number. The collapse curves of Figure4use a denser ratio grid (AppendixJ).

Errors.Bootstrap CIs forκ\kappapropagate uncertainty in bothL^\hat{L}and the parameters offf(AppendixD). Reported half-widths are 95% bootstrap CIs over 1000 paired resamples.

Common harness.Table3uses one common harness rather than numbers reported in the original papers. All nine methods use math+code with Qwen3-8B and the same 3 seeds; the same real pool of 10,240 tasks and 256 environments; the same generator (Qwen3-32B[52], replacing each method’s original generator), with 8 generation calls per unit and at most 4k output tokens; identical executability and verifier filtering and deduplication thresholds; the same outcome verifier; identical downstream training (1,000 steps, batch 256,G=8G=8, and the same held-out evaluation); and the same ratio grid andκ\kappaestimator.

Common-core ablation.Because the selected methods also differ in maturity and implementation quality, we strip method-specific components (e.g., AWM’s repair loop, AutoPlay’s exploration bonus, and DreamGym’s replay buffer), leaving a common generate, filter, and train core (Table16). Across the eight single-axis methods, axis membership explains 97% of the variance of common-coreκ\kappa(one-way ANOVA,F⁡(2,5)=76.6F(2,5)=76.6,p<0.001p<0.001), with within-axis spread at most0.070.07against0.300.30between axes, so the axis effects are unlikely to be method artifacts. The exponents of Section4are measured independently on axis-controlled pools; Table3only suppliesκi\kappa_{i}to Section6. Becauseκi\kappa_{i}enters the allocation problem only through the ratioci/κic_{i}/\kappa_{i}, the cost perturbations of AppendixNalso cover the corresponding misestimates ofκi\kappa_{i}.

Table 16:Discount and collapse ratio of each single-axis method as reported in Table3and after reduction to a common generate, filter, and train core.

Appendix JCollapse-Ratio Estimation

Definition and grids.We definer⋆r^{\star}as the ratio at which normalized held-out success, past its peak, falls 5% below the real-only value atr=0r=0. Two ratio grids are used and are kept separate:κ\kappais estimated onr∈{0,0.25,1,4,16}r\in\{0,0.25,1,4,16\}(AppendixI), while the collapse curves of Figure4user∈{0,0.1,0.25,0.5,1,2,3,4,6,8,10,16,25}r\in\{0,0.1,0.25,0.5,1,2,3,4,6,8,10,16,25\}, and every curve is normalized to the real-only value of the same seed group.

Estimator.We fit a monotone four-parameter logistic inlog⁡r\log rto the post-peak branch of each curve, weighted by1/σ^21/\hat{\sigma}^{2}, and taker⋆r^{\star}as its analytic crossing of0.950.95; confidence intervals use the paired bootstrap of AppendixD. This givesrE⋆=9.4r^{\star}_{\mathrm{E}}=9.4,rT⋆=3.4r^{\star}_{\mathrm{T}}=3.4, andrτ⋆=1.4r^{\star}_{\tau}=1.4for AWM, AutoPlay, and SWiRL. Relative to Table3, only AutoPlay moves (3.4 against 3.1, within its CI). Table17shows that the ordering does not depend on the criterion used to define collapse.

Table 17:Collapse ratio of the three representative synthesis methods (Qwen3-8B, math+code) under alternative definitions of collapse.Mechanism: groundedness of synthesis.Section5.2attributes the orderingrE⋆≫rT⋆≫rτ⋆r^{\star}_{E}\gg r^{\star}_{T}\gg r^{\star}_{\tau}to how strongly each axis is grounded by external truth. We additionally quantify this: across the three synthesis methods we measured the distributional drift between synthetic and real samples by the maximum mean discrepancy (MMD)[12]of state-action embeddings against the reference real-data distribution. MMD scales withrrasMMD⁡(r)∝r0.45\mathrm{MMD}(r)\propto r^{0.45}for environment synthesis,r0.62r^{0.62}for task synthesis, andr0.91r^{0.91}for trajectory synthesis on Qwen3-8B. The exponents inversely trackr⋆r^{\star}: faster drift implies earlier collapse, a quantitative confirmation of the groundedness story. Replicating the measurement on Llama-3.1-8B gives drift exponents of0.470.47,0.640.64, and0.880.88, so the mechanism that predicts the ordering transfers across model families.

Appendix KLagrangian Solution to The Allocation Problem

Setup.The constrained optimization (Eq.6) is

min⁡∑i∈𝒜N,DE,DT,Dτ⁡Ki​Di−αi+KN​N−αN+L∞s.t.∑i∈𝒜ciκi​Di+cN​N=C,\min_{N,D_{\mathrm{E}},D_{\mathrm{T}},D_{\tau}}\;\;\sum_{i\in\mathcal{A}}K_{i}\,D_{i}^{-\alpha_{i}}+K_{N}\,N^{-\alpha_{N}}+L_{\infty}\quad\text{s.t.}\quad\sum_{i\in\mathcal{A}}\frac{c_{i}}{\kappa_{i}}D_{i}+c_{N}N=C,(14)where𝒜={E,T,τ}\mathcal{A}=\{\mathrm{E},\mathrm{T},\tau\}and the constraint is taken with equality at the optimum (since the objective is strictly decreasing in eachDiD_{i}).

KKT conditions.Form the Lagrangianℒ=(Eq.14)+λ⋅[∑iciκi​Di+cN​N−C]\mathcal{L}=(\text{Eq.~\ref{eq:app-opt}})+\lambda\cdot\bigl[\sum_{i}\tfrac{c_{i}}{\kappa_{i}}D_{i}+c_{N}N-C\bigr]. Setting∂ℒ/∂Di=0\partial\mathcal{L}/\partial D_{i}=0for each axis:

−αi​Ki​Di−αi−1+λ​ciκi=0⟹Di⋆=(αi​Ki​κiλ​ci)1αi+1.-\alpha_{i}K_{i}\,D_{i}^{-\alpha_{i}-1}+\lambda\frac{c_{i}}{\kappa_{i}}=0\quad\Longrightarrow\quad D_{i}^{\star}\;=\;\left(\frac{\alpha_{i}K_{i}\,\kappa_{i}}{\lambda\,c_{i}}\right)^{\!\!\frac{1}{\alpha_{i}+1}}.(15)Analogously forNN:

N⋆=(αN​KNλ​cN)1αN+1.N^{\star}\;=\;\left(\frac{\alpha_{N}K_{N}}{\lambda\,c_{N}}\right)^{\!\!\frac{1}{\alpha_{N}+1}}.(16) Closed-form scaling inCC.Substituting Eqs.15and16into the budget equation gives, after collecting powers ofλ\lambda:

∑i∈𝒜ciκi​(αi​Ki​κiλ​ci)1αi+1+cN​(αN​KNλ​cN)1αN+1=C.\sum_{i\in\mathcal{A}}\frac{c_{i}}{\kappa_{i}}\left(\frac{\alpha_{i}K_{i}\kappa_{i}}{\lambda c_{i}}\right)^{\!\!\frac{1}{\alpha_{i}+1}}+c_{N}\left(\frac{\alpha_{N}K_{N}}{\lambda c_{N}}\right)^{\!\!\frac{1}{\alpha_{N}+1}}\;=\;C.(17)For each axis the exponent ofλ\lambdais−1/(αi+1)-1/(\alpha_{i}+1), so to leading orderλ∝C−ξ\lambda\propto C^{-\xi}for some effective exponentξ\xidetermined by the dominant axis. Substituting back gives

Di⋆∝Cξ/(αi+1),N⋆∝Cξ/(αN+1).D_{i}^{\star}\;\propto\;C^{\,\xi/(\alpha_{i}+1)},\qquad N^{\star}\;\propto\;C^{\,\xi/(\alpha_{N}+1)}.(18) Numerical solution.With our fitted values (αE=0.41\alpha_{\mathrm{E}}\!=\!0.41,αT=0.26\alpha_{\mathrm{T}}\!=\!0.26,ατ=0.11\alpha_{\tau}\!=\!0.11,αN=0.30\alpha_{N}\!=\!0.30;KE=3.39K_{E}\!=\!3.39,KT=0.53K_{T}\!=\!0.53,Kτ=0.38K_{\tau}\!=\!0.38,KN=0.85K_{N}\!=\!0.85;κE=0.71\kappa_{E}\!=\!0.71,κT=0.54\kappa_{T}\!=\!0.54,κτ=0.34\kappa_{\tau}\!=\!0.34;cEc_{E},cTc_{T},cτc_{\tau},cNc_{N}from AppendixL), we solve the system numerically by binary search onλ\lambdato satisfy the budget constraint. Repeating acrossC∈{1,2,4,8,16,32}​C0C\in\{1,2,4,8,16,32\}\,C_{0}and fitting power-laws to the resultingDi⋆​(C)D_{i}^{\star}(C)yields the recipe in Eq.8:DE⋆∝C0.34D_{\mathrm{E}}^{\star}\propto C^{0.34},DT⋆∝C0.27D_{\mathrm{T}}^{\star}\propto C^{0.27},Dτ⋆∝C0.13D_{\tau}^{\star}\propto C^{0.13},N⋆∝C0.26N^{\star}\propto C^{0.26}. The four exponents sum to approximately one (0.34+0.27+0.13+0.26=1.000.34+0.27+0.13+0.26=1.00).

Appendix LCost-Coefficient Calibration and Refined Cost Models

Decomposition.The cost coefficients(cE,cT,cτ,cN)(c_{E},c_{T},c_{\tau},c_{N})in Eq.5aggregate two physically distinct cost sources:(i) synthesis cost(one-time generation of environments, tasks, or trajectories), measured in API/compute USD per unit; and(ii) training cost(rollout + gradient FLOPs), measured in H100-GPU-hours per unit. We normalize both into a common base unitC0C_{0}.

Calibration toC0C_{0}.We defineC0=2.4×1018C_{0}=2.4\times 10^{18}FLOPs as the cost of one Qwen3-4B GRPO training run on the math+code testbed for 1000 gradient steps, which corresponds approximately to55H100-GPU-hours at the throughput we measured on our cluster (vLLM 0.6.3 + FSDP). Synthesis costs are converted via 1 H100-hour≈\approx$2.50 (a representative cloud rate at the time of writing); the resulting normalized coefficients are:

cE=×10−2,cT=×10−3,cτ=×10−3,cN=×10−1.c_{E}=3.0\!\times\!10^{-2},\quad c_{T}=1.0\!\times\!10^{-3},\quad c_{\tau}=3.0\!\times\!10^{-3},\quad c_{N}=1.5\!\times\!10^{-1}.(19)The relative magnitudes carry the meaning, not the absolute USD values:cE/cT≈30c_{E}/c_{T}\approx 30reflects that environment synthesis (4.10 USD/env) is much costlier per unit than task synthesis (0.18 USD/task), andcN/cτ≈50c_{N}/c_{\tau}\approx 50reflects that doubling model size is much more expensive than doubling trajectory count when the rest is held fixed.

Multiplicative compute term.A more accurate cost model accounts for the fact that rollout and gradient FLOPs scale jointly withDτD_{\tau}andNN:

Ctrain​(Dτ,N)=(crollout⋅H+cupdate)⋅N⋅Dτ,C_{\mathrm{train}}(D_{\tau},N)\;=\;\bigl(c_{\mathrm{rollout}}\cdot H+c_{\mathrm{update}}\bigr)\cdot N\cdot D_{\tau},(20)whereHHis the average rollout length. We verified that absorbing this bilinear term into an effectivecτ⋅N+cNc_{\tau}\cdot N+c_{N}(linearised at the optimum) gives KKT optima that differ from Eq.8by less than 0.02 in every exponent acrossC∈[1,32]​C0C\in[1,32]\,C_{0}, justifying the simpler additive form in the main text.

Measured cost drivers.Rollout cost also depends on horizon length, tool latency, environment execution, verifier cost, failed rollouts, and infrastructure utilization. We instrumented the training stack and report these drivers per domain in Table18.

Table 18:Measured cost drivers per domain: average horizonHH(steps), rollout tokens per trajectory, environment latency per call, verifier time per trajectory, failed-rollout fractionff, and accelerator utilizationη\eta.Refined cost models.Table19re-derives the recipe adding one factor at a time: the bilinear rollout termc⋅N⋅Dτc\cdot N\cdot D_{\tau}of Eq.20, tool and environment execution latency plus verifier cost, failed-rollout overhead1/(1−f)1/(1-f), and batching and utilizationη\eta. Every refinement moves budget from environments toward tasks and trajectories, but the largest shift in any exponent is0.060.06, which changes predicted held-out success atC=32​C0C=32C_{0}by0.70.7points against the measured8.38.3-point gap over the naive allocation. The qualitative conclusion is therefore not near its flipping point, although the precise exponents are stack-dependent. In the SWE worst case (18-step horizons, a 41 s verifier, and 19% failed rollouts), the environment and task exponents become statistically indistinguishable, so we do not claim thatDED_{\mathrm{E}}strictly dominates everywhere.

Table 19:Recipe exponents under progressively richer cost models (math+code coefficients unless stated otherwise).Cost modelDED_{\mathrm{E}}exp.DTD_{\mathrm{T}}exp.DτD_{\tau}exp.NNexp.Linear-additive (Eq.5)0.340.340.270.270.130.130.260.26++bilinear rollout termc⋅N⋅Dτc\cdot N\cdot D_{\tau}0.330.330.270.270.140.140.260.26++tool and environment latency, verifier cost0.310.310.280.280.150.150.260.26++failed-rollout overhead1/(1−f)1/(1-f)0.300.300.280.280.160.160.260.26++batching and utilizationη\eta0.300.300.280.280.160.160.260.26Full model, SWE coefficients (worst case forDED_{\mathrm{E}})0.280.280.290.290.180.180.250.25Full model, web coefficients0.330.330.270.270.140.140.260.26Break-even condition.The environment axis loses its top rank only when the fully loaded cost per environment unit exceeds roughly3.4×3.4\timesour highest measured value (11.8 USD-equivalent per environment, on SWE) at unchanged trajectory cost. Across four domains and two clusters (H100 with vLLM and FSDP, and an A100 stack), the measured fully loaded cost ranges from 2.9 to 11.8 USD-equivalent per environment, so practitioners can check their own infrastructure against this number rather than trusting ours. Section6.4applies the same break-even to the acquisition of new worlds.

Appendix MCross-Domain Robustness of The Recipe

We re-derive the recipe (Eq.8) on each of the four domains independently to verify generality.

Table 20:Recipe exponents per domain. The qualitative orderingDE⋆>DT⋆>Dτ⋆D_{\mathrm{E}}^{\star}>D_{\mathrm{T}}^{\star}>D_{\tau}^{\star}is preserved in every domain, and absolute exponents shift by less than±0.04\pm 0.04from the math+code reference.As Table20shows, the web and UI/tool domains have slightly largerDED_{\mathrm{E}}exponents and smallerDτD_{\tau}exponents than math+code, consistent with these domains’ richer state-space structure (more API surface to generalize across). The SWE domain is an outlier in the opposite direction: itsDED_{\mathrm{E}}exponent is the smallest because the code-execution sandbox provides relatively limited environmental variation between tasks. Despite these shifts, the qualitative ordering and the approximate normalization∑iexpi≈1\sum_{i}\mathrm{exp}_{i}\approx 1are preserved everywhere. Under the richer cost models of AppendixL, however, the SWE environment and task exponents become indistinguishable.

Appendix NCost-Coefficient Sensitivity

To verify the recipe is not over-fit to our specific cost calibration (AppendixL), we perturb each coefficient±50%\pm 50\%and re-solve the Lagrangian.

Table 21:Recipe-exponent sensitivity to±50%\pm 50\%perturbations of cost coefficients (math+code, 8B). All shifts are<0.05<0.05on every axis.The largest sensitivity is tocEc_{E}(the environment-synthesis cost), which makes sense intuitively:cEc_{E}sets the price of the highest-leverage axis, and small changes propagate. Even so, all perturbations stay within±0.05\pm 0.05of the default exponents, and the qualitative ordering is preserved in every row of Table21. Because eachκi\kappa_{i}enters the allocation problem only throughci/κic_{i}/\kappa_{i}, these perturbations also cover the corresponding misestimates of the synthesis discounts. We conclude the recipe is robust to reasonable mis-specification of cost coefficients; AppendixLcomplements these perturbations with structurally richer cost models.

Appendix ONegative Results and Failed Fits

For transparency we report alternatives we tried that did not work, in the spirit of strengthening the paper’s positive claims.

Multiplicative interaction term.We testedL=KE​DE−αE⋅KT​DT−αT⋅Kτ​Dτ−ατ+L∞L\!=\!K_{E}D_{\mathrm{E}}^{-\alpha_{\mathrm{E}}}\!\cdot\!K_{T}D_{\mathrm{T}}^{-\alpha_{\mathrm{T}}}\!\cdot\!K_{\tau}D_{\tau}^{-\alpha_{\tau}}\!+\!L_{\infty}(fully multiplicative form). Goodness-of-fit was strictly worse than the additive form (R2R^{2}drops from0.9780.978to0.9510.951on the math+code fit), and the resulting individual exponents were not robust across model sizes (±0.08\pm 0.08shift between 4B and 14B). We retain the additive form.

Unified data axis.A reasonable null hypothesis is that the three cardinality axes can be collapsed into a single “effective data” axisDeff=wE​DE+wT​DT+wτ​DτD_{\mathrm{eff}}=w_{E}D_{\mathrm{E}}+w_{T}D_{\mathrm{T}}+w_{\tau}D_{\tau}with weightswiw_{i}fitted jointly. Best-fit weights are(wE,wT,wτ)=(4.2,1.0,0.04)(w_{E},w_{T},w_{\tau})=(4.2,1.0,0.04), but the resulting one-axis fit hasR2=0.83R^{2}=0.83vs.0.9780.978for the three-axis decomposition, and the weights themselves are unstable across domains. We conclude the axes are genuinely independent and not a relabeling of one underlying quantity.

Direct success-rate fit (no log transform).We tried fitting power laws directly inpsuccessp_{\mathrm{success}}space rather than the loss-equivalentLL(AppendixF). Fits saturate poorly near the success-rate ceiling, with residuals exceeding 5% at the high-success grid points and drivingR2R^{2}below0.850.85. The log transform is necessary.

Curriculum at fixed allocation.We tested whether scheduling difficulty (curriculum) within a fixed allocation could substitute for axis decomposition. A simple difficulty-easy-to-hard curriculum[3]on the naive allocation gives a1.21.2pp improvement atC=32C=32but does not approach our recipe’s8.38.3pp gain over naive. Curriculum is a useful but orthogonal improvement.

Appendix PReproducibility

Code and data release.We release, athttps://github.com/BokwaiHo/AgentRL_Scaling: (i) the verl-compatible training scripts for all per-axis sweeps; (ii) the calibrated(Ki,αi,κi,ci)(K_{i},\alpha_{i},\kappa_{i},c_{i})fitting code, including the profiled estimator of Algorithm1; (iii) the synthesised env/task corpora used (subject to licensing of upstream data); (iv) plotting code reproducing every figure in the paper; (v) the per-grid-point environment signature manifests and the structural-factor inventory of AppendixC; and (vi) the held-axis calibration script together with its pre-specified criterion (AppendixA).

Determinism.All RL runs use deterministic torch settings (torch.use_deterministic_algorithms(True)) and fixed seeds. Run-to-run variance reported in tables is genuinely the seed effect, not stochastic data ordering.

Hyperparameter sweep budget.We did not tune RL hyperparameters per scaling-law fit. The hyperparameters (AppendixA) were fixed once based on a held-out 8B/math+code calibration and used unchanged across all cells of Table2, all 4 cross-domain re-fits, the additional model families and search domain, and all sensitivity analyses. This is critical for the validity of the scaling-law fits: had we re-tuned hyperparameters per cell, the fittedαi\alpha_{i}would conflate data-axis effects with hyperparameter effects.

Appendix QCompute Disclosure

The total experimental compute for this paper is approximately61,00061{,}000H100-GPU-hours: approximately48,00048{,}000for the experiments of the original study and approximately13,00013{,}000for the additional experiments added in this version. The breakdown of the former is:

• •Section4per-axis sweeps and robustness (Table2):∼\sim32,00032{,}000GPU-hours (18 cells×\times6 grid points×\times3 seeds, with median per-run cost varying by domain).
• •Section5synthesis-discount and collapse experiments (Fig.4, Table3):∼\sim10,00010{,}000GPU-hours.
• •Section6IsoFLOP sweeps and held-out validation:∼\sim5,0005{,}000GPU-hours, dominated by the single Qwen3-14B held-out run atC=32​C0C=32C_{0}.
• •Failed runs, hyperparameter calibration, and pilots:∼\sim1,0001{,}000GPU-hours.

The synthesis-side compute (env/task generation via AWM, AutoPlay, etc.) used an additional∼\sim3,5003{,}500H100-GPU-hours plus approximately $2,800 in API costs for the closed-source generators we used as baselines. The additional experiments, reported in AppendicesBtoD,G,I,J, andL(the eight-point grid extension, the training-signal conventions, the signature-restricted, placebo, and world-structure sweeps, the coverage pools, the further model families and search domain, the common-core synthesis ablation, and the refined cost models), ran on the same infrastructure and used approximately13,00013{,}000H100-GPU-hours in total.

Appendix RLimitation Analysis

This work has several limitations. First, although we evaluate the proposed axis-decomposed scaling framework across multiple model sizes, algorithms, domains, and model families, the experiments remain limited to the specific model families, benchmark environments, and compute scales studied in this paper. In particular, the model-size sweep covers open-weight models in the sub-frontier regime, and the held-out validation is performed at a single larger scale. Llama-3.1-8B and Qwen2.5-7B preserve the ordering of the exponents while shifting their absolute values by up to0.050.05, and the fitted exponents and compute-optimal allocation recipe may change at frontier scale, under different pretraining distributions, or for agents with substantially different tool-use and reasoning priors. Therefore, the ordering is our claim, and the numerical exponents reported in this paper should be viewed as empirical estimates for the measured regime rather than universal constants.

Second, our decomposition of agentic data into task, environment, and trajectory axes plus a reward-density control variable is a useful abstraction, but it is not perfectly orthogonal. In practice, tasks and environments often co-vary: a new environment may induce new task types, and a new task distribution may require different environment structure; our sweeps re-bind a fixed task pool to compatible environments (AppendixB), but only part of the task×\timesenvironment grid is realizable. Similarly, reward density can interact with trajectory diversity because denser rewards may change which rollouts are informative. We model some cross-axis interactions, especially between task and environment diversity, but richer higher-order interactions may exist. Future work should study more expressive interaction terms and determine when the additive power-law approximation breaks down.

Third, the resolution of the environment axis into world and structure diversity rests on limited world counts. World diversity is swept over only 1 to 8 real worlds on SWE and UI/tool and 1 to 4 on web (1 to 11 repositories in the SWE-only sweep), soαW\alpha_{\mathrm{W}}spans a single order of magnitude. The structure axis has a finite realizable ceiling under our factor inventory. The structural exponentαV\alpha_{\mathrm{V}}varies across domains from0.280.28to0.410.41, and we do not offer a single value. We do not claim that structure and world variation are interchangeable, and we do not claim that any of the fitted relations continue past the measured ranges.

Fourth, the loss-equivalent metric used in this paper is based on held-out task success. This makes scaling-law fitting practical for verifiable-reward RL, but it does not capture every property relevant to agent quality. For example, two agents with similar success rates may differ in robustness, calibration, latency, safety, interpretability, or the severity of their failures. The metric also depends on the coverage and difficulty distribution of the held-out evaluation set. As a result, improvements predicted by the fitted law should not be interpreted as comprehensive improvements in all downstream deployment settings.

Fifth, our treatment of synthetic data quality is necessarily approximate. The synthesis discount factorκ\kappaand collapse ratior⋆r^{\star}provide a common way to compare methods, but they compress many dimensions of quality into a small number of scalar quantities. Synthetic environments, tasks, and trajectories may differ in realism, diversity, difficulty, distributional coverage, and susceptibility to model-generated artifacts. These factors may vary across domains and over training time. In addition, the measured discount for a synthesis method depends on the base model, generator, verifier, and filtering pipeline used in the experiment; we fix these through a common harness (AppendixI), but different synthesis pipelines may yield different effective discounts even when targeting the same axis.

Sixth, the compute-allocation recipe relies on a simplified cost model. We assume an approximately linear, axis-additive cost structure with per-axis cost coefficients and synthesis discounts. Real training systems may have non-linear bottlenecks due to rollout latency, tool execution, environment simulation, distributed training overhead, memory constraints, data storage, or human review costs. Adding the main measured drivers changes recipe exponents by at most0.060.06but makes the environment and task exponents indistinguishable under SWE coefficients (AppendixL). The recipe is therefore best understood as a principled first-order allocation rule rather than an exact prescription for every infrastructure setting. Practitioners should recalibrate the cost coefficients and fitted exponents under their own hardware, software stack, and data-generation pipeline before using the recipe operationally.

Seventh, our experiments focus on benchmark-style domains with controlled evaluation and relatively clear reward signals. Many real-world agentic tasks involve ambiguous objectives, partial observability, long-horizon dependencies, dynamic user preferences, and non-stationary environments. In such settings, defining task success and reward density is more difficult, and synthetic data may introduce artifacts that are harder to detect. Extending the framework to open-ended real-world applications will require stronger evaluation protocols, safety checks, and methods for measuring distribution shift.

Finally, this work does not provide a complete safety or deployment framework for agentic systems. The proposed scaling laws can make agentic RL training more efficient, but they do not guarantee that the resulting agents are safe, fair, private, secure, or aligned with user intent. These concerns require separate evaluations and safeguards. Future work should combine axis-level data scaling with robustness testing, adversarial evaluation, privacy analysis, fairness auditing, and deployment-time monitoring.

Appendix SEthics Statement

This work is an empirical study of data scaling and synthesis allocation in agentic reinforcement learning. The experiments are conducted on benchmark-style math, code, web, UI/tool, and search environments, using scripted training pipelines and verifiable or process-based reward signals. We do not intentionally collect or use private personal data, and the synthetic tasks, environments, and trajectories studied in this paper are intended for research evaluation rather than for deployment in real user-facing decision systems. To reduce privacy risk, experimental logs should be stored securely, inspected for unintended sensitive content where applicable, and released only after appropriate filtering or anonymization.

The work involves synthetic data generation along multiple axes, including tasks, environments, and trajectories. Synthetic data can amplify biases, encode unrealistic assumptions, or produce artifacts that affect downstream model behavior. Our measurements focus on aggregate scaling behavior and task success, and they should not be interpreted as evidence that a trained agent is safe, fair, or reliable in deployment. Any use of the proposed allocation recipe in real applications should include additional evaluations for robustness, bias, privacy leakage, security, and subgroup-level performance.

Because agentic systems can interact with tools, websites, code execution environments, or simulated applications, there are potential dual-use risks. More efficient agentic RL training could improve beneficial automation, but it could also lower the cost of training agents for harmful activities such as phishing, spam, vulnerability discovery, unauthorized data collection, or manipulation of online services. Our experiments should be run in sandboxed environments with controlled tool access, and released artifacts should avoid enabling direct misuse. We encourage practitioners to pair these techniques with access controls, abuse monitoring, rate limits, permission boundaries, and adversarial safety evaluations.

We also consider fairness and labor-related issues. Apart from a small annotation of task-pair similarity by three volunteer annotators from MBZUAI who participated without compensation (AppendixG), which labels properties of synthetic tasks rather than of the annotators themselves, the paper does not rely on new human-subject experiments or sensitive demographic annotations. If future extensions use human annotators, user interaction traces, or real-world application data, they should follow informed-consent procedures, provide appropriate compensation, minimize exposure to harmful content, and comply with applicable institutional and legal requirements. Similarly, if agentic systems trained using these methods are deployed in socially consequential domains, humans should retain meaningful oversight and affected users should have mechanisms for appeal or correction.

Overall, this paper provides a framework for measuring and allocating agentic RL data, not a guarantee of responsible deployment. The ethical use of this work depends on the context in which the resulting agents are trained and applied. We recommend that downstream users treat the proposed scaling laws and synthesis strategies as one component of a broader responsible-development process that includes privacy protection, security review, fairness auditing, misuse prevention, and transparent reporting of data sources and evaluation limitations.

Appendix TBroader Impacts

This work studies how data scales in agentic reinforcement learning and proposes a framework for allocating synthesis budget across tasks, environments, trajectories, and reward density. The potential positive societal impact is that more efficient data allocation can reduce the compute and financial cost of training capable agentic systems. By identifying which forms of data provide the highest marginal return, the proposed framework may help researchers and practitioners train useful agents with fewer resources, improving accessibility for academic labs and smaller organizations. More efficient agentic RL could also accelerate progress on beneficial applications such as software engineering assistance, scientific problem solving, education, accessibility tools, and automation of repetitive digital workflows. In addition, the explicit decomposition of agentic data may improve transparency: rather than treating training data as an undifferentiated token budget, our framework encourages practitioners to document what kinds of tasks, environments, trajectories, and reward signals are used.

At the same time, the same efficiency gains could enable harmful or unintended uses. More capable and cheaper-to-train agents may be misused for automated disinformation campaigns, generation of fake online profiles, scalable phishing or social engineering, spam, scraping, surveillance, or other forms of cyber abuse. Because our results suggest that environment and task synthesis can be especially high-leverage, a malicious actor could use similar methods to generate many realistic interaction settings for training agents to operate across websites, applications, or tool interfaces. The work therefore should not be interpreted as an endorsement of unrestricted deployment of agentic systems. Practical use of these techniques should be accompanied by safety evaluations, access controls, monitoring for abuse, and domain-specific restrictions where agent actions can affect real users or systems.

There are also fairness and representational concerns. Synthetic tasks and environments may encode the assumptions, biases, or omissions of the systems and datasets used to generate them. If agentic RL systems trained under such distributions are deployed in settings involving hiring, lending, education, healthcare, policing, or access to public services, they could make or support decisions that unfairly affect particular demographic or socioeconomic groups. Our scaling laws measure aggregate task success and do not by themselves guarantee equitable performance across subpopulations, languages, regions, or accessibility needs. Any downstream deployment should therefore include disaggregated evaluation, bias auditing, human oversight, and clear limits on the decisions agents are allowed to make autonomously.

Privacy is another important consideration. Agentic environments often involve realistic user data, application states, logs, tool outputs, or simulated personal information. Even when data is synthetic, it may be derived from real data sources or may resemble sensitive user records. Care is required to avoid memorization, leakage, or reconstruction of private information through generated tasks, environments, trajectories, or reward traces. Practitioners applying this framework should use data minimization, anonymization where appropriate, secure storage of trajectories and logs, and privacy reviews for any environment that models real users or organizations.

Finally, security risks arise because agentic RL systems interact with tools and external environments. Training agents to generalize across diverse environments may improve their usefulness, but it may also improve their ability to discover vulnerabilities, bypass interface constraints, or execute unintended actions. Synthetic environments should therefore be sandboxed, tool permissions should be least-privilege, and evaluations should include adversarial and misuse-oriented testing. We view our contribution as a measurement and allocation framework, not as a complete safety framework. Future work should integrate the proposed data-scaling perspective with stronger safeguards for alignment, robustness, privacy, security, and responsible deployment.

Appendix UDeclaration of LLM usage

Large language models are a central object of study and an operational component of this work. Our experiments train and evaluate different-family base models under agentic reinforcement learning, using LLM policies to generate multi-step trajectories involving tool calls, environment feedback, and verifiable rewards. In addition, several data-synthesis procedures studied in the paper operate along LLM-agentic data axes: environment synthesis, task synthesis, and policy-rollout-based trajectory generation. These uses are part of the scientific methodology rather than merely writing, editing, or formatting assistance.

All reported scaling-law fits, synthesis-discount estimates, collapse-ratio measurements, allocation recipes, and held-out validations are computed from logged experimental runs and scripted analyses. LLM outputs used during synthesis or rollout are treated as experimental data and are evaluated through the verification and reward mechanisms described in the paper. The authors remain responsible for the design of the experiments, interpretation of results, and all claims made in the paper. Any use of general-purpose LLM tools for language polishing or formatting did not affect the core methodology, experimental results, or scientific conclusions.

NeurIPS Paper Checklist

1. 1.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:[Yes]
4. Justification: Please see Section1.
5. Guidelines: - •The answer[N/A]means that the abstract and introduction do not include the claims made in the paper. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A[No]or[N/A]answer to this question will not be perceived well by the reviewers. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
6. 2.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:[Yes]
9. Justification: Please see AppendixR.
10. Guidelines: - •The answer[N/A]means that the paper has no limitation while the answer[No]means that the paper has limitations, but those are not discussed in the paper. - •The authors are encouraged to create a separate “Limitations” section in their paper. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. - •The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. - •The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
11. 3.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
13. Answer:[Yes]
14. Justification: Please see Section6and AppendixK.
15. Guidelines: - •The answer[N/A]means that the paper does not include theoretical results. - •All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. - •All assumptions should be clearly stated or referenced in the statement of any theorems. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. - •Theorems and Lemmas that the proof relies upon should be properly referenced.
16. 4.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
18. Answer:[Yes]
19. Justification: Please see Section3.2, AppendixAand AppendixP.
20. Guidelines: - •The answer[N/A]means that the paper does not include experiments. - •If the paper includes experiments, a[No]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. - •Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example 1. (a)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 2. (b)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. 3. (c)If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 4. (d)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
21. 5.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:[Yes]
24. Guidelines: - •The answer[N/A]means that paper does not include experiments requiring code. - • - •While we encourage the release of code and data, we understand that this might not be possible, so[No]is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). - •The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. - •At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). - •Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
25. 6.Experimental setting/details
26. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
27. Answer:[Yes]
28. Justification: Please see Section3.2and AppendixA.
29. Guidelines: - •The answer[N/A]means that the paper does not include experiments. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. - •The full details can be provided either with the code, in appendix, or as supplemental material.
30. 7.Experiment statistical significance
31. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
32. Answer:[Yes]
33. Justification: Please see Section3.2, Section4, and Section5.
34. Guidelines: - •The answer[N/A]means that the paper does not include experiments. - •The authors should answer[Yes]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. - •The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). - •The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) - •The assumptions made should be given (e.g., Normally distributed errors). - •It should be clear whether the error bar is the standard deviation or the standard error of the mean. - •It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
35. 8.Experiments compute resources
36. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
37. Answer:[Yes]
38. Justification: Please see AppendixAand AppendixQ.
39. Guidelines: - •The answer[N/A]means that the paper does not include experiments. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
40. 9.Code of ethics
41. Answer:[Yes]
42. Justification: Please see AppendixS.
43. Guidelines: - •The answer[N/A]means that the authors have not reviewed the NeurIPS Code of Ethics. - •If the authors answer[No], they should explain the special circumstances that require a deviation from the Code of Ethics. - •The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
44. 10.Broader impacts
45. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
46. Answer:[Yes]
47. Justification: Please see AppendixT.
48. Guidelines: - •The answer[N/A]means that there is no societal impact of the work performed. - •If the authors answer[N/A]or[No], they should explain why their work has no societal impact or why the paper does not address societal impact. - •Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
49. 11.Safeguards
50. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
51. Answer:[N/A]
52. Justification: This paper poses no such risks.
53. Guidelines: - •The answer[N/A]means that the paper poses no such risks. - •Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. - •Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
54. 12.Licenses for existing assets
55. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
56. Answer:[Yes]
57. Justification: Please see Section3and AppendixA.
58. Guidelines: - •The answer[N/A]means that the paper does not use existing assets. - •The authors should cite the original paper that produced the code package or dataset. - •The authors should state which version of the asset is used and, if possible, include a URL. - •The name of the license (e.g., CC-BY 4.0) should be included for each asset. - •For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. - •If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets,paperswithcode.com/datasetshas curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. - •For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators.
59. 13.New assets
60. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
61. Answer:[Yes]
62. Guidelines: - •The answer[N/A]means that the paper does not release new assets. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. - •The paper should discuss whether and how consent was obtained from people whose asset is used. - •At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
63. 14.Crowdsourcing and research with human subjects
64. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
65. Answer:[N/A]
66. Justification: The paper involves neither crowdsourcing nor research with human subjects. Its only human input is a small annotation of 500 synthetic task pairs by three volunteer annotators from MBZUAI, who participated without compensation and labeled properties of the tasks rather than of themselves; the labeling question and the agreement statistics are reported in AppendixG.
67. Guidelines: - •The answer[N/A]means that the paper does not involve crowdsourcing nor research with human subjects. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
68. 15.Institutional review board (IRB) approvals or equivalent for research with human subjects
69. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
70. Answer:[N/A]
71. Justification: This paper does not involve crowdsourcing nor research with human subjects.
72. Guidelines: - •The answer[N/A]means that the paper does not involve crowdsourcing nor research with human subjects. - •Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. - •For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
73. 16.Declaration of LLM usage
74. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and doesnotimpact the core methodology, scientific rigor, or originality of the research, declaration is not required.
75. Answer:[Yes]
76. Justification: Please see AppendixU.
77. Guidelines: - •The answer[N/A]means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.