On Training in Imagination

# On Training in Imagination
Source: [https://arxiv.org/html/2605.06732](https://arxiv.org/html/2605.06732)
Nadav Timor Weizmann Institute of Science &Ravid Shwartz\-Ziv New York University Meta FAIR &Micah Goldblum Columbia University &Yann LeCun New York University AMI Labs &David Harel Weizmann Institute of Science

###### Abstract

State\-of\-the\-art model\-based reinforcement learning methods train policies on imagined rollouts\. These rollouts are trajectories generated by a learned dynamics model and are scored by a learned reward model, but without querying the true environment during policy updates\. We study this training paradigm by quantifying how errors in learned dynamics and reward models affect returns and policy optimization\. First, we extend the analysis ofAsadiet al\.\([2018b](https://arxiv.org/html/2605.06732#bib.bib2)\)to MDPs with learned reward models, and derive the optimal sample allocation—the ratio of dynamics samples to reward samples that minimizes a bound on return error under power\-law scaling assumptions\. We identify lower Lipschitz constants of the learned dynamics, reward, and policy as a representation desideratum that tightens this bound, and we connect this perspective to the temporal\-straightening objective ofWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\)\. Second, we examine how policy optimization withreinforcetolerates noisy rewards, which are often cheaper to obtain\. We show that zero\-mean reward noise leaves the gradient estimator unbiased and adds at most a variance term that decreases with the number of rollouts\. This introduces a practical tradeoff: given a fixed budget, should one buy more rollouts with cheaper but noisier rewards, or fewer rollouts with more expensive but less noisy rewards? We reduce this choice to a one\-dimensional optimization problem and characterize the optimum\.

## 1Introduction

In*training in imagination*, the policy is trained on rollouts generated by a learned dynamics model and scored by a learned reward model, with no environment interaction during the policy update step itself\. Recent state\-of\-the\-art instantiations include Dreamer 3\(Hafneret al\.,[2025a](https://arxiv.org/html/2605.06732#bib.bib3)\), trained across diverse control tasks with a single configuration, and Dreamer 4\(Hafneret al\.,[2025b](https://arxiv.org/html/2605.06732#bib.bib4)\), which extends the paradigm to long\-horizon offline control\.Schrittwieseret al\.\([2020](https://arxiv.org/html/2605.06732#bib.bib17)\)earlier instantiate a closely related paradigm in board games and Atari, learning dynamics, reward, and value jointly\.

These recent results provide strong empirical evidence that training in imagination can be effective on challenging control tasks\. Existing simulation\-lemma\-style bounds\(Kearns and Singh,[2002](https://arxiv.org/html/2605.06732#bib.bib20); Asadiet al\.,[2018b](https://arxiv.org/html/2605.06732#bib.bib2)\), however, do not assign independently controllable coefficients to dynamics\-model and reward\-model error, nor do they say how a sample budget should be split between dynamics samples and the typically more expensive reward samples \(e\.g\., human preference labels in reinforcement learning from human feedback, or expert evaluation in robotics\)\. The optimal trade\-off between the two has not been theoretically characterized\.

Four questions about this paradigm remain open\. The first is error attribution: how much of the return gap comes from dynamics\-model error versus reward\-model error? The second is representation properties: what properties of the learned representations and the maps acting on them tighten the return\-error bound? The third is budget allocation: given a fixed sample budget, how should it be split between dynamics transitions and reward annotations? The fourth is reward fidelity: how doesreinforcetolerate noisy or biased reward annotations, and when is it preferable to buy many cheap noisy annotations rather than fewer accurate ones?

#### Our contribution\.

This paper treats the learned reward model as a separate source of error with its own sample budget, distinct from the learned dynamics, and quantifies the resulting attribution, allocation, and noise\-tolerance trade\-offs under Lipschitz and power\-law assumptions\.

1. 1\.Error attribution\.[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)extendsAsadiet al\.\([2018b](https://arxiv.org/html/2605.06732#bib.bib2)\)by replacing the assumed ground\-truth reward with a learned reward model, and gives a Lipschitz\-based decomposition of the return gap with separable, independently controllable dynamics\-error and reward\-error coefficients\.
2. 2\.Representation desiderata\.[Corollary˜1](https://arxiv.org/html/2605.06732#Thmcorollary1)shows that the dynamics\-error coefficient in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)is monotone non\-decreasing in the Lipschitz constantsLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}of the learned dynamics, reward, and policy, identifying lower Lipschitz constants of the learned models as a representation desideratum\.[Proposition˜1](https://arxiv.org/html/2605.06732#Thmproposition1)couples this perspective to the temporal\-straightening objective ofWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\), upper\-bounding its curvature loss by a function of the Lipschitz constant of the latent velocity map\.
3. 3\.Budget allocation\.[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1), under power\-law error scaling for the dynamics and reward errors, gives a closed\-form expression for the optimal ratio of dynamics samples to reward samples in terms of the power\-law exponents, the per\-sample costs, and the Lipschitz coefficient inherited from[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\.
4. 4\.Reward fidelity\.[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)shows that the multi\-trajectoryreinforceestimator under additive zero\-mean reward noise is unbiased with bounded variance inflation;[Corollary˜2](https://arxiv.org/html/2605.06732#Thmcorollary2)reduces the optimal\-fidelity allocation problem to a one\-dimensional minimization in the per\-rollout annotation cost; and[Proposition˜2](https://arxiv.org/html/2605.06732#Thmproposition2)formalizes systematic reward bias as a gradient bias that trajectory averaging cannot remove\.

Empirical evaluations of the assumptions and predictions of these results appear in[Sections˜3\.1](https://arxiv.org/html/2605.06732#S3.SS1),[4\.1](https://arxiv.org/html/2605.06732#S4.SS1)and[4\.2](https://arxiv.org/html/2605.06732#S4.SS2)\.

#### Notation\.

We writeℳ=\(𝒮,𝒜,f,r,γ\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},f,r,\\gamma\)for a Markov decision process \(mdp\) with state space𝒮⊆ℝds\\mathcal\{S\}\\subseteq\\mathbb\{R\}^\{d\_\{s\}\}, action space𝒜⊆ℝda\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\_\{a\}\}, deterministic dynamicsff, rewardrr, and discountγ∈\[0,1\)\\gamma\\in\[0,1\)\. Starting from an initial states0s\_\{0\}, a policyπ\\pigenerates a trajectory byat=π​\(st\)a\_\{t\}=\\pi\(s\_\{t\}\)andst\+1=f​\(st,at\)s\_\{t\+1\}=f\(s\_\{t\},a\_\{t\}\)\. We writeJ​\(π,ℳ\):=∑t=0∞γt​r​\(st,at\)J\(\\pi,\\mathcal\{M\}\):=\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)for the discounted return ofπ\\piinℳ\\mathcal\{M\}\. Hats denote estimated quantities\. In particular,f^\\hat\{f\}is the learned dynamics,r^\\hat\{r\}the learned reward, andℳ^=\(𝒮,𝒜,f^,r^,γ\)\\hat\{\\mathcal\{M\}\}=\(\\mathcal\{S\},\\mathcal\{A\},\\hat\{f\},\\hat\{r\},\\gamma\)is themdpobtained by replacingffandrrwithf^\\hat\{f\}andr^\\hat\{r\}\.εdyn:=sups,a‖f^​\(s,a\)−f​\(s,a\)‖\\varepsilon\_\{\\mathrm\{dyn\}\}:=\\sup\_\{s,a\}\\\|\\hat\{f\}\(s,a\)\-f\(s,a\)\\\|andεrew:=sups,a\|r^​\(s,a\)−r​\(s,a\)\|\\varepsilon\_\{\\mathrm\{rew\}\}:=\\sup\_\{s,a\}\|\\hat\{r\}\(s,a\)\-r\(s,a\)\|are the worst\-case model errors\. Throughout,∥⋅∥\\\|\\cdot\\\|denotes the Euclidean norm\.

## 2Related work

Training a policy on rollouts from a learned model of the environment dates back to the Dyna architecture ofSutton \([1990](https://arxiv.org/html/2605.06732#bib.bib6)\), which interleaves real\-environment transitions with updates on imagined transitions drawn from a learned dynamics model\. Model\-based policy optimization\(Janneret al\.,[2019](https://arxiv.org/html/2605.06732#bib.bib5)\)uses short imagined rollouts from an ensemble dynamics model to augment policy updates\. More recently,Hafneret al\.\([2025a](https://arxiv.org/html/2605.06732#bib.bib3)\)train across diverse control tasks with a single learned world model and reward predictor, andHafneret al\.\([2025b](https://arxiv.org/html/2605.06732#bib.bib4)\)train agents inside scalable learned world models, calling this process “training in imagination”\. These works treat the learned dynamics model and the learned reward predictor as a single coupled object and tune it empirically;[appendix˜A](https://arxiv.org/html/2605.06732#A1)reviews the broader latent\-world\-model lineage\. Unlike the training\-in\-imagination lineage ofHafneret al\.\([2025b](https://arxiv.org/html/2605.06732#bib.bib4)\), this paper decomposes return error into separate, independently controllable dynamics\-model and reward\-model terms \([Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\), derives a closed\-form split of a single sample budget between dynamics transitions and reward annotations \([Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)\), and characterizes the policy gradient inside this paradigm under both zero\-mean noise and bias in the learned reward \([Theorems˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)and[2](https://arxiv.org/html/2605.06732#Thmproposition2)\)\.

Simulation return\-error bounds have a long history in reinforcement learning theory, beginning with the simulation lemma ofKearns and Singh \([2002](https://arxiv.org/html/2605.06732#bib.bib20)\), which bounds the value gap between a true and approximate Markov decision process in terms of one\-step transition and reward errors\. Closest to our setting,Asadiet al\.\([2018b](https://arxiv.org/html/2605.06732#bib.bib2)\)bound multi\-step prediction error in model\-based reinforcement learning under Lipschitz assumptions on the dynamics and policy, but they assume access to the ground\-truth reward, so reward\-model error never enters their bound\.[Appendix˜A](https://arxiv.org/html/2605.06732#A1)surveys subsequent refinements of these bounds and value\-aware model learning\. UnlikeAsadiet al\.\([2018b](https://arxiv.org/html/2605.06732#bib.bib2)\),[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)carries a learned reward model through the analysis and produces an explicit reward\-error term alongside the dynamics\-error term with independently controllable coefficients, which is the structural ingredient that makes a budget split between dynamics and reward data well\-posed\.

Allocating sample budget between dynamics samples and reward samples sits at the intersection of reward\-aware data collection and neural scaling laws\. Reward\-free exploration separates a reward\-agnostic data\-collection phase from later reward\-conditioned planning, with sample\-complexity guarantees that hold uniformly over downstream reward functions\(Jinet al\.,[2020](https://arxiv.org/html/2605.06732#bib.bib29)\)\. Neural scaling laws fit power\-law decays of loss in data and parameters\(Kaplanet al\.,[2020](https://arxiv.org/html/2605.06732#bib.bib25)\)and, in the compute\-optimal regime, split a single training budget between model parameters and tokens\(Hoffmannet al\.,[2022](https://arxiv.org/html/2605.06732#bib.bib26)\)\.[Appendix˜A](https://arxiv.org/html/2605.06732#A1)discusses related work on active observation, simulation budget allocation, and scaling laws specific to reinforcement learning and world\-model pre\-training\. UnlikeHoffmannet al\.\([2022](https://arxiv.org/html/2605.06732#bib.bib26)\),[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)splits a single sample budget between two heterogeneous data streams—dynamics transitions and reward annotations—whose errors obey separately fitted power\-law exponents, and yields a closed\-form ratio in those exponents, the unit costs, and the Lipschitz coefficient inherited from[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\.

Policy\-gradient analysis under noisy rewards begins with thereinforceestimator ofWilliams \([1992](https://arxiv.org/html/2605.06732#bib.bib1)\)\. A line of work studies robustness to reward corruption:Zhanget al\.\([2021](https://arxiv.org/html/2605.06732#bib.bib34)\)analyze adversarial corruption in which anε\\varepsilon\-fraction of episodes have their rewards or transitions arbitrarily modified and develop estimators with provable robustness guarantees\.Caiet al\.\([2025](https://arxiv.org/html/2605.06732#bib.bib35)\)treat asymmetric verifier noise with false positives and false negatives in reinforcement learning with verifiable rewards\. A parallel literature documents Goodhart\-style overoptimization of learned reward models, in which the gap between proxy and gold rewards grows with optimization budget\(Gaoet al\.,[2023](https://arxiv.org/html/2605.06732#bib.bib39)\), a systematic failure mode rather than zero\-mean reward noise\.[Appendix˜A](https://arxiv.org/html/2605.06732#A1)reviews the policy\-gradient and variance\-reduction lineage and the broader reward\-modeling literature\. Unlike the adversarial setting ofZhanget al\.\([2021](https://arxiv.org/html/2605.06732#bib.bib34)\),[Theorems˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)and[2](https://arxiv.org/html/2605.06732#Thmcorollary2)treat zero\-mean i\.i\.d\. reward noise as a per\-rollout fidelity cost, which reduces annotation\-budget allocation to a one\-dimensional problem\.[Proposition˜2](https://arxiv.org/html/2605.06732#Thmproposition2)addresses reward bias separately, showing that any non\-zero reward\-bias gradient survives trajectory averaging\.

The choice of latent representation, and the regularity of the maps acting on it, governs how reliably an imagined rollout tracks reality\.LeCun and others \([2022](https://arxiv.org/html/2605.06732#bib.bib12)\)advocates joint\-embedding predictive architectures, in which prediction takes place in a learned latent space rather than at the level of raw observations\.Wanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\)propose a temporal\-straightening loss that encourages consecutive latent differences along a rollout to be parallel, so that long\-horizon predictions in latent space follow a near\-linear trajectory\. On the regularization side,Miyatoet al\.\([2018](https://arxiv.org/html/2605.06732#bib.bib41)\)introduce spectral normalization, which controls the Lipschitz constant of a neural network by normalizing the spectral norm of each weight matrix;[appendix˜A](https://arxiv.org/html/2605.06732#A1)discusses related joint\-embedding instantiations and Lipschitz\-regularization mechanisms\. These representation\-learning and Lipschitz\-regularization proposals are motivated by stability or representational quality, and their connection to long\-horizon policy value is left implicit\. UnlikeWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\)andMiyatoet al\.\([2018](https://arxiv.org/html/2605.06732#bib.bib41)\),[Corollaries˜1](https://arxiv.org/html/2605.06732#Thmcorollary1)and[1](https://arxiv.org/html/2605.06732#Thmproposition1)couple these representation\-learning desiderata to an explicit return\-error coefficient, showing that the dynamics\-error coefficient is monotone in the Lipschitz constants of the learned dynamics, reward, and policy and that the temporal\-straightening loss is upper\-bounded by a function of the latent\-velocity\-map Lipschitz constant\.

## 3Properties of representations for training in imagination

What makes a representation of the system useful for training in imagination?LeCun and others \([2022](https://arxiv.org/html/2605.06732#bib.bib12)\)hypothesized that designing the dynamics, reward, and policy models to operate on latent statesztz\_\{t\}that capture a higher\-level representation of the system—in place of raw observationssts\_\{t\}and past actionsata\_\{t\}—may improve prediction and planning across different time horizons\. However, what properties such a representation should have has remained an open question\.[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)and[Corollary˜1](https://arxiv.org/html/2605.06732#Thmcorollary1)give one answer to this open question: representations that lower the Lipschitz constants of the learned models tighten our bound on return error\.

Similar toAsadiet al\.\([2018b](https://arxiv.org/html/2605.06732#bib.bib2)\), we assume that the dynamicsff, rewardrr, and policyπ\\pisatisfy the following Lipschitz conditions:

- •ffisLfL\_\{f\}\-Lipschitz:‖f​\(s,a\)−f​\(s′,a′\)‖≤Lf​\(‖s−s′‖\+‖a−a′‖\)\\\|f\(s,a\)\-f\(s^\{\\prime\},a^\{\\prime\}\)\\\|\\leq L\_\{f\}\(\\\|s\-s^\{\\prime\}\\\|\+\\\|a\-a^\{\\prime\}\\\|\)for alls,s′,a,a′s,s^\{\\prime\},a,a^\{\\prime\}\.
- •rrisLrL\_\{r\}\-Lipschitz:\|r​\(s,a\)−r​\(s′,a′\)\|≤Lr​\(‖s−s′‖\+‖a−a′‖\)\|r\(s,a\)\-r\(s^\{\\prime\},a^\{\\prime\}\)\|\\leq L\_\{r\}\(\\\|s\-s^\{\\prime\}\\\|\+\\\|a\-a^\{\\prime\}\\\|\)for alls,s′,a,a′s,s^\{\\prime\},a,a^\{\\prime\}\.
- •π\\piisLπL\_\{\\pi\}\-Lipschitz:‖π​\(s\)−π​\(s′\)‖≤Lπ​‖s−s′‖\\\|\\pi\(s\)\-\\pi\(s^\{\\prime\}\)\\\|\\leq L\_\{\\pi\}\\\|s\-s^\{\\prime\}\\\|for alls,s′s,s^\{\\prime\}\.

###### Lemma 1\(Simulation error decomposition\)\.

Letℳ,ℳ^\\mathcal\{M\},\\hat\{\\mathcal\{M\}\}bemdps sharing\(𝒮,𝒜,γ\)\(\\mathcal\{S\},\\mathcal\{A\},\\gamma\)with deterministic dynamicsf,f^f,\\hat\{f\}and rewardsr,r^r,\\hat\{r\}\. Assumeγ​Lf​\(1\+Lπ\)<1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)<1\. Then for anyLπL\_\{\\pi\}\-Lipschitz policyπ\\pi,

\|J​\(π,ℳ\)−J​\(π,ℳ^\)\|≤11−γ​εrew\+γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​εdyn\.\\left\|J\(\\pi,\\mathcal\{M\}\)\-J\(\\pi,\\hat\{\\mathcal\{M\}\}\)\\right\|\\leq\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}\+\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}\.\(1\)

###### Proof\.

See[Section˜B\.1](https://arxiv.org/html/2605.06732#A2.SS1)\. ∎

###### Corollary 1\(Lipschitz constants control the dynamics\-error coefficient\)\.

Under the hypothesisγ​Lf​\(1\+Lπ\)<1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)<1, the coefficientγ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}ofεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)is non\-decreasing in each ofLfL\_\{f\},LrL\_\{r\}, andLπL\_\{\\pi\}\. Hence, at fixedεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}andεrew\\varepsilon\_\{\\mathrm\{rew\}\}, lowering any ofLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}tightens the bound in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)on return error\.

[Corollary˜1](https://arxiv.org/html/2605.06732#Thmcorollary1)formalizes this: representations that lower the Lipschitz constants of the learned modelsLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}tighten the bound in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)on return error at fixedεdyn,εrew\\varepsilon\_\{\\mathrm\{dyn\}\},\\varepsilon\_\{\\mathrm\{rew\}\}\.

A simple implementation may define the latent statezzas an encoding of the current observationz=ϕ​\(s\)z=\\phi\(s\)\. The Lipschitz constants are then defined by comparing the outputs of the learned models at nearby latent states, rather than at nearby raw observations\. For example, the Lipschitz constant of the dynamics model is then‖f​\(ϕ​\(s\),a\)−f​\(ϕ​\(s′\),a′\)‖≤Lf​\(‖ϕ​\(s\)−ϕ​\(s′\)‖\+‖a−a′‖\)\\\|f\(\\phi\(s\),a\)\-f\(\\phi\(s^\{\\prime\}\),a^\{\\prime\}\)\\\|\\leq L\_\{f\}\\bigl\(\\\|\\phi\(s\)\-\\phi\(s^\{\\prime\}\)\\\|\+\\\|a\-a^\{\\prime\}\\\|\\bigr\)\. The reward and policy maps are handled analogously, with absolute value replacing the output norm for scalar rewards\. Higher\-order representations are also possible: latent statesztz\_\{t\}may encode the full trajectory\(s0,a0,…,st,at−1\)\(s\_\{0\},a\_\{0\},\\ldots,s\_\{t\},a\_\{t\-1\}\), states or activations of other learned models, and so on\.

This Lipschitz perspective also connects to the temporal straightening objective ofWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\), which compares consecutive latent differences generated by the dynamics model and encourages these differences to point in the same direction\. To make the connection explicit, let𝒵\\mathcal\{Z\}denote the latent state space and writezt=ϕ​\(st\)z\_\{t\}=\\phi\(s\_\{t\}\)for the latent state associated with observationsts\_\{t\}\. FollowingWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\), we writefffor the learned dynamics model on𝒵\\mathcal\{Z\}, so thatzt\+1=f​\(zt\)z\_\{t\+1\}=f\(z\_\{t\}\)along a latent rollout\. Thus, in this discussion of temporal straightening,ffno longer denotes the observation\-level dynamics on𝒮\\mathcal\{S\}\.

###### Definition 1\(temporal straightening curvature loss, adapted fromWanget al\.\([2026](https://arxiv.org/html/2605.06732#bib.bib11)\)\)\.

Define the latent velocity mapv​\(z\):=f​\(z\)−zv\(z\):=f\(z\)\-z\. For a latent rolloutzt\+1=f​\(zt\)z\_\{t\+1\}=f\(z\_\{t\}\)and anyttsuch thatv​\(zt\)≠0v\(z\_\{t\}\)\\neq 0andv​\(zt\+1\)≠0v\(z\_\{t\+1\}\)\\neq 0, define

ℒcurv​\(t\):=1−v​\(zt\)⊤​v​\(zt\+1\)‖v​\(zt\)‖​‖v​\(zt\+1\)‖\.\\mathcal\{L\}\_\{\\mathrm\{curv\}\}\(t\):=1\-\\frac\{v\(z\_\{t\}\)^\{\\top\}v\(z\_\{t\+1\}\)\}\{\\\|v\(z\_\{t\}\)\\\|\\\|v\(z\_\{t\+1\}\)\\\|\}\.

On observed transitions,v​\(ϕ​\(st\)\)v\(\\phi\(s\_\{t\}\)\)approximatesϕ​\(st\+1\)−ϕ​\(st\)\\phi\(s\_\{t\+1\}\)\-\\phi\(s\_\{t\}\), so changes invvalong a latent rollout measure how smoothly the latent state moves along the trajectory\. The temporal straightening objective maximizes the cosine similarity betweenv​\(zt\)v\(z\_\{t\}\)andv​\(zt\+1\)v\(z\_\{t\+1\}\), or equivalently minimizes the curvature loss in[Definition˜1](https://arxiv.org/html/2605.06732#Thmdefinition1)\. The relevant Lipschitz quantity is not the Lipschitz constant offfitself, but the Lipschitz constant of the latent velocity mapvv\.

###### Proposition 1\(temporal straightening from a Lipschitz latent velocity map\)\.

Letzt\+1=f​\(zt\)z\_\{t\+1\}=f\(z\_\{t\}\)be a latent rollout generated by the dynamics model, and definev​\(z\):=f​\(z\)−zv\(z\):=f\(z\)\-z\. Assumevvisε\\varepsilon\-Lipschitz on the latent states visited by the rollout, with0<ε<10<\\varepsilon<1\. For anyttsuch thatv​\(zt\)≠0v\(z\_\{t\}\)\\neq 0andv​\(zt\+1\)≠0v\(z\_\{t\+1\}\)\\neq 0, the temporal straightening curvature loss from[Definition˜1](https://arxiv.org/html/2605.06732#Thmdefinition1)satisfies

ℒcurv​\(t\)≤ε22​\(1−ε\)\.\\mathcal\{L\}\_\{\\mathrm\{curv\}\}\(t\)\\leq\\frac\{\\varepsilon^\{2\}\}\{2\(1\-\\varepsilon\)\}\.\(2\)

###### Proof\.

See[Section˜B\.2](https://arxiv.org/html/2605.06732#A2.SS2)\. ∎

[Proposition˜1](https://arxiv.org/html/2605.06732#Thmproposition1)shows that making the latent velocity map slowly varying makes consecutive latent differences nearly parallel, which directly lowers the temporal straightening loss\. Thus, minimizing the Lipschitz constant ofvvminimizes the upper bound in[Equation˜2](https://arxiv.org/html/2605.06732#S3.E2)\.

Two caveats apply\. First, a representation that lowers the Lipschitz constants might also increaseεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}\. The bound in[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)only tightens if the decrease in the dynamics\-error coefficient outweighs the increase inεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}\. Second, that bound assumesγ​Lf​\(1\+Lπ\)<1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)<1, and the behavior whenγ​Lf​\(1\+Lπ\)≥1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\\geq 1remains an open question for future work\.

### 3\.1Numerical illustration of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)

We empirically test the inequality in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)\. For each test configuration, define the realized return gapLHS:=\|J​\(π,ℳ\)−J​\(π,ℳ^\)\|\\mathrm\{LHS\}:=\|J\(\\pi,\\mathcal\{M\}\)\-J\(\\pi,\\hat\{\\mathcal\{M\}\}\)\|and the bound’s right\-hand sideRHS:=\(1−γ\)−1​εrew\+γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​εdyn\\mathrm\{RHS\}:=\(1\-\\gamma\)^\{\-1\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}\+\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}, evaluated with the configuration’s analytical Lipschitz constantsLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}, discount factorγ\\gamma, and realized per\-step errorsεdyn,εrew\\varepsilon\_\{\\mathrm\{dyn\}\},\\varepsilon\_\{\\mathrm\{rew\}\}\. We report the ratioR:=LHS/RHSR:=\\mathrm\{LHS\}/\\mathrm\{RHS\}\. The bound holds wheneverR≤1R\\leq 1, and smallerRRindicates a looser bound\. Each configuration consists of anmdp, anLπL\_\{\\pi\}\-Lipschitz evaluation policy, and a perturbed model pair with realized per\-step errorsεdyn,εrew\\varepsilon\_\{\\mathrm\{dyn\}\},\\varepsilon\_\{\\mathrm\{rew\}\}\. We test on two benchmarks\. The synthetic benchmark uses globally Lipschitzffandrr, so the hypotheses of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)hold by construction\. The Linear–Quadratic–Gaussian \(LQG\) benchmark uses a quadratic reward that is only locally Lipschitz, testing[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)on a bounded operating domain\.[Section˜B\.4](https://arxiv.org/html/2605.06732#A2.SS4)describes the per\-configuration construction\. Acrossn=525n=525configurations \(150150synthetic,375375LQG\) the bound holds on every configuration\. The per\-benchmark medians areRsynthmed=0\.0035R^\{\\mathrm\{med\}\}\_\{\\mathrm\{synth\}\}=0\.0035andRLQGmed=0\.034R^\{\\mathrm\{med\}\}\_\{\\mathrm\{LQG\}\}=0\.034, the pooled median is0\.0150\.015, and the per\-benchmark maxima areRsynthmax=0\.9995R^\{\\max\}\_\{\\mathrm\{synth\}\}=0\.9995andRLQGmax=0\.999R^\{\\max\}\_\{\\mathrm\{LQG\}\}=0\.999\. For per\-benchmark empirical CDF \(ECDF\) shapes and full implementation details, see[Section˜B\.4](https://arxiv.org/html/2605.06732#A2.SS4)\. Every configuration satisfies the bound, but the typical bound is loose: at the median it overshoots by1/RLQGmed≈29×1/R^\{\\mathrm\{med\}\}\_\{\\mathrm\{LQG\}\}\\approx 29\\timeson LQG and1/Rsynthmed≈286×1/R^\{\\mathrm\{med\}\}\_\{\\mathrm\{synth\}\}\\approx 286\\timeson the synthetic benchmark \(factor≈65×\\approx 65\\timesat the pooled median\)\. Both benchmarks use deterministic dynamics; extending the calibration to stochastic dynamics or unbounded operating domains would require revisiting the assumptions of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)and is left to future work\.

## 4The optimal sample allocation to minimize return error

We distinguish between two types of samples: dynamics\-transition samples and reward samples\. A dynamics\-transition sample\(s,a,f​\(s,a\)\)\(s,a,f\(s,a\)\)consists of a statess, an actionaa, and the resulting next statef​\(s,a\)f\(s,a\)\. A reward sample\(s,a,r​\(s,a\)\)\(s,a,r\(s,a\)\)consists of a state, an action, and the resulting rewardr​\(s,a\)r\(s,a\)\. LetNdynN\_\{\\mathrm\{dyn\}\}andNrewN\_\{\\mathrm\{rew\}\}denote the numbers of dynamics\-transition and reward samples, and letcdync\_\{\\mathrm\{dyn\}\}andcrewc\_\{\\mathrm\{rew\}\}denote their per\-sample costs\. These costs may reflect any incurred costs, including environment interaction, annotation, and training on the samples\. We reuse the symbolsεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}andεrew\\varepsilon\_\{\\mathrm\{rew\}\}to denote the error levels achievable when the sample counts areNdynN\_\{\\mathrm\{dyn\}\}andNrewN\_\{\\mathrm\{rew\}\}, according to standard power laws

εdyn​\(Ndyn\)=Ad⋅Ndyn−α,εrew​\(Nrew\)=Ar⋅Nrew−β\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\)=A\_\{d\}\\cdot N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\},\\qquad\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\)=A\_\{r\}\\cdot N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\(3\)whereα,β\>0\\alpha,\\beta\>0are exponents fit from data, andAd,ArA\_\{d\},A\_\{r\}are constants\. For each budgetB=cdyn​Ndyn\+crew​NrewB=c\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\+c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}, let\(Ndyn∗,Nrew∗\)\(N\_\{\\mathrm\{dyn\}\}^\{\*\},N\_\{\\mathrm\{rew\}\}^\{\*\}\)denote a minimizer of the bound in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1), and letεdyn∗:=εdyn​\(Ndyn∗\)\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}:=\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}^\{\*\}\)andεrew∗:=εrew​\(Nrew∗\)\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}:=\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}^\{\*\}\)be the dynamics and reward errors at those minimizing sample counts\. We study the optimal ratio of dynamics samples to reward samples,Ndyn∗Nrew∗\\frac\{N\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{N\_\{\\mathrm\{rew\}\}^\{\*\}\}, asB→∞B\\to\\infty\.

###### Theorem 1\(Optimal dynamics\-to\-reward sample ratio\)\.

If[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3)holds with exponentsα,β\>0\\alpha,\\beta\>0and the bound of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)holds, then the minimizing sample counts\(Ndyn∗,Nrew∗\)\(N\_\{\\mathrm\{dyn\}\}^\{\*\},N\_\{\\mathrm\{rew\}\}^\{\*\}\)under the constraintcdyn​Ndyn\+crew​Nrew=Bc\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\+c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}=Bsatisfy

Ndyn∗Nrew∗:=αβ⋅γ​Lr​\(1\+Lπ\)1−γ​Lf​\(1\+Lπ\)⋅crewcdyn⋅εdyn∗εrew∗\.\\frac\{N\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{N\_\{\\mathrm\{rew\}\}^\{\*\}\}:=\\frac\{\\alpha\}\{\\beta\}\\cdot\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\}\\cdot\\frac\{c\_\{\\mathrm\{rew\}\}\}\{c\_\{\\mathrm\{dyn\}\}\}\\cdot\\frac\{\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}\}\.\(4\)

###### Proof\.

See[Section˜B\.3](https://arxiv.org/html/2605.06732#A2.SS3)\. ∎

[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)gives the optimal sample ratioNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}in terms of the error ratioεdyn∗/εrew∗\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}/\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}, and shows that these two ratios are proportional\. The multiplier in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)uses the global Lipschitz constantsLf,LrL\_\{f\},L\_\{r\}assumed by[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1), so it is an upper bound rather than an equality\.[Section˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2)measures, on the same configurations, how much smaller the realized value\-function sensitivities are than these global constants, and how close the resulting prediction ofNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}becomes\. For practitioners who design systems that train policies in imagination, changes in sample costs or planning horizon affect the optimal sample ratio as follows\.

#### Sample costs\.

Costs enter throughcdyn​Ndyn\+crew​Nrew=Bc\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\+c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}=Band the fitted error curves in[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3); in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4), they appear ascrew/cdync\_\{\\mathrm\{rew\}\}/c\_\{\\mathrm\{dyn\}\}\. Loweringcrewc\_\{\\mathrm\{rew\}\}lowers this factor, soNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}decreases and the allocation shifts toward reward samples\. Loweringcdync\_\{\\mathrm\{dyn\}\}raises the factor, soNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}increases and the allocation shifts toward dynamics\-transition samples\. However, since the optimal sample ratio also includesεdyn∗/εrew∗\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}/\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}, these statements give the direction of the cost effect rather than a complete allocation rule by themselves\.

#### Planning horizon\.

The discounting effect is the familiar one: smallerγ\\gammagives less weight to later rewards\. Asγ\\gammadecreases,[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)becomes tighter, and the dynamics multiplier in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4),γ​Lr​\(1\+Lπ\)/\(1−γ​Lf​\(1\+Lπ\)\)\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)/\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\), decreases\. The optimal allocation therefore shifts toward reward samples, as expected for a shorter\-horizon objective in which dynamics errors have fewer discounted steps to affect future rewards\. A similar effect arises from lowering the Lipschitz constants of the learned models\. By[Corollary˜1](https://arxiv.org/html/2605.06732#Thmcorollary1), lowering any ofLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}tightens[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)and decreases the dynamics multiplier in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)\. The optimal allocation therefore shifts toward reward samples, since dynamics errors then compound less strongly along rollouts\.

### 4\.1Numerical experiments: scaling of dynamics and reward errors

We estimate how fast each model learns with more data and whether the dynamics and reward models learn at the same rate\. To this end, we conduct an experiment that tests the power\-law scaling assumptions in[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3)and estimates the exponentsα\\alphaandβ\\betafor a particular architecture and environment\. In this experiment, we estimate the dynamics and reward errors as a function of the number of training samplesNdynN\_\{\\mathrm\{dyn\}\}andNrewN\_\{\\mathrm\{rew\}\}, respectively, and fit the standard power\-law scaling laws from[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3)\.[Figure˜1](https://arxiv.org/html/2605.06732#S4.F1)shows the fitted scaling laws, which are consistent with the power\-law assumptions in[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)\. It measuresεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}andεrew\\varepsilon\_\{\\mathrm\{rew\}\}on a fixed held\-out set𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}of transitions\(s,a,s′,r\)\(s,a,s^\{\\prime\},r\)in a synthetic continuous\-control environment whose dynamics and reward function are defined by frozen, randomly\-initialized 2\-layer ReLU MLPs, and whose studentsf^\\hat\{f\}andr^\\hat\{r\}use the same architecture as the teachers\. Specifically,εdyn​\(Ndyn\):=1\|𝒟val\|​∑\(s,a,s′,r\)∈𝒟val‖f^​\(s,a\)−s′‖2\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\):=\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\mathrm\{val\}\}\|\}\\sum\_\{\(s,a,s^\{\\prime\},r\)\\in\\mathcal\{D\}\_\{\\mathrm\{val\}\}\}\\\|\\hat\{f\}\(s,a\)\-s^\{\\prime\}\\\|^\{2\}andεrew​\(Nrew\):=1\|𝒟val\|​∑\(s,a,s′,r\)∈𝒟val\(r^​\(s,a\)−r\)2\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\):=\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\mathrm\{val\}\}\|\}\\sum\_\{\(s,a,s^\{\\prime\},r\)\\in\\mathcal\{D\}\_\{\\mathrm\{val\}\}\}\(\\hat\{r\}\(s,a\)\-r\)^\{2\}, wheref^\\hat\{f\}andr^\\hat\{r\}are trained onNdynN\_\{\\mathrm\{dyn\}\}dynamics\-transition samples andNrewN\_\{\\mathrm\{rew\}\}reward samples drawn independently of𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}\. We useεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}andεrew\\varepsilon\_\{\\mathrm\{rew\}\}as practical surrogates for the sup\-norm errors in[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\. The fitted laws areεdyn​\(Ndyn\)=0\.34​Ndyn−0\.11\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\)=0\.34\\,N\_\{\\mathrm\{dyn\}\}^\{\-0\.11\}withR2=0\.954R^\{2\}=0\.954andεrew​\(Nrew\)=90\.4​Nrew−0\.96\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\)=90\.4\\,N\_\{\\mathrm\{rew\}\}^\{\-0\.96\}withR2=0\.997R^\{2\}=0\.997\. For bootstrap standard errors, 95% bootstrap confidence intervals, and full implementation details, see[Section˜C\.2](https://arxiv.org/html/2605.06732#A3.SS2)\. The exponent ratio0\.96/0\.11≈90\.96/0\.11\\approx 9is the central empirical observation: reward error decays nearly an order of magnitude faster per decade of training data than dynamics error\. This ratio of exponents is consistent withr^\\hat\{r\}predicting a scalar whilef^\\hat\{f\}predicts adsd\_\{s\}\-dimensional next state, and its exact value depends on the dimensions and architectures used here\.

![Refer to caption](https://arxiv.org/html/2605.06732v1/x1.png)Figure 1:Dynamics and reward errors obey distinct power\-law scaling laws\. Reward error decays roughly9×9\\timesfaster per decade of training data than dynamics error \(≈0\.960\.11\\approx\\frac\{0\.96\}\{0\.11\}\)\.[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)uses this ratio to characterize the optimal fraction of transitions that should carry a reward annotation\.
### 4\.2Empirical evaluation of[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)

[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)predictsNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}as the product of two factors: the proportionalityNdyn∗/Nrew∗∝εdyn∗/εrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}\\propto\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}/\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}and a multiplier that depends onγ\\gamma,LfL\_\{f\},LrL\_\{r\},LπL\_\{\\pi\},crew/cdync\_\{\\mathrm\{rew\}\}/c\_\{\\mathrm\{dyn\}\}, andα/β\\alpha/\\beta\. We test the two factors separately: whether the proportionality holds when the multiplier is replaced by realized value\-function sensitivities, and how loose the multiplier is when instantiated with the global Lipschitz constants assumed by[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\. Define the log\-ratio residualℓ:=log⁡\(Ndyn∗/Nrew∗\)−log⁡\(εdyn∗/εrew∗\)\\ell:=\\log\(N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}\)\-\\log\(\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}/\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}\), so\|ℓ\|≤log⁡3\|\\ell\|\\leq\\log 3means predicted and realized ratios agree within a factor of33\. The realized value\-function sensitivitiesSf,SrS\_\{f\},S\_\{r\}and the per\-configuration construction are defined in[Section˜B\.5](https://arxiv.org/html/2605.06732#A2.SS5)\. Replacing the global constantsLf,LrL\_\{f\},L\_\{r\}with the realized sensitivitiesSf,SrS\_\{f\},S\_\{r\}recovers the predicted ratio\. We test this recovery on configurations whose value functionVπV^\{\\pi\}has a known parametric form: linear,tanh\\tanh,sin\\sin, and a separate quadratic\-value control group that isolates the looseness from usingsup\\sup\-norm overestimates in place of the realized sensitivities\. The linear configurations achieve median\|ℓ\|=0\|\\ell\|=0and Spearman rank correlationρ=1\\rho=1between predicted and realized ratios; this case is an algebraic consistency check rather than an empirical test, since a linearVπV^\{\\pi\}forces the realized sensitivities to match the analytical ratio\. The empirical content comes from the nonlinear configurations:tanh\\tanhandsin\\singive median\|ℓ\|=0\.054\|\\ell\|=0\.054\. On the control group, substitutingsup\\sup\-norm overestimates for the realized sensitivities inflates the median residual by roughly an order of magnitude\.[Figure˜2](https://arxiv.org/html/2605.06732#S4.F2)plots predicted against realized sample ratiosNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}for each group\. Instantiating the multiplier with the analytical global Lipschitz constants of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1), by contrast, overshoots realized ratios by about three orders of magnitude\. On the LQG configurations every configuration’sℓ\\ellis positive, with medianℓ=7\.585\\ell=7\.585\(a factorexp⁡\(7\.585\)≈1968\\exp\(7\.585\)\\approx 1968between predicted and realized ratios\)\.[Figure˜4](https://arxiv.org/html/2605.06732#A2.F4)shows the per\-configuration distribution ofℓ\\ell\. The contraction regime, sample sizes, statistical methodology, and a separate ratio isolating the dynamics\-coefficient contribution to the multiplier are reported in[Section˜B\.5](https://arxiv.org/html/2605.06732#A2.SS5)\.

![Refer to caption](https://arxiv.org/html/2605.06732v1/x2.png)Figure 2:Predicted and realized sample ratiosNdyn∗/Nrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}agree when global Lipschitz constants are replaced by realized value\-function sensitivitiesSf,SrS\_\{f\},S\_\{r\}\. The linear,tanh\\tanh, andsin\\singroups in panel \(a\) concentrate on the diagonaly=xy=x, while thesup\\sup\-norm control in panel \(b\) lies systematically below it—using the globalsup\\sup\-norm in place of the realized sensitivities reintroduces the looseness shown in[Figure˜4](https://arxiv.org/html/2605.06732#A2.F4)\. See[Section˜B\.5](https://arxiv.org/html/2605.06732#A2.SS5)for plotting and methodology details\.The proportionalityNdyn∗/Nrew∗∝εdyn∗/εrew∗N\_\{\\mathrm\{dyn\}\}^\{\*\}/N\_\{\\mathrm\{rew\}\}^\{\*\}\\propto\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}/\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)therefore carries the predictive content, while the multiplier built from global Lipschitz constants is loose at the order\-of\-magnitude scale\. Recovering the predicted ratio requires evaluatingVπV^\{\\pi\}at perturbed models \(the construction in[Section˜B\.5](https://arxiv.org/html/2605.06732#A2.SS5)\), which may be impractical at scale\.

## 5Cost\-effective learning from noisy rewards

The reward\-noise analysis in this section is independent of the allocation results in[Section˜4](https://arxiv.org/html/2605.06732#S4): the unbiasedness statement forreinforceunder zero\-mean reward noise applies to any stationary MDP satisfying the standard policy\-gradient assumptions, regardless of how dynamics and reward samples are allocated\.

We study policy optimization under noisy rewards through the classicalreinforceestimator introduced byWilliams \([1992](https://arxiv.org/html/2605.06732#bib.bib1)\)\. In this section letπθ\\pi\_\{\\theta\}be a differentiable policy, fix a finite horizonHH, and fix a discount factorγ∈\[0,1\)\\gamma\\in\[0,1\)\. DefineJH​\(π,ℳ\):=𝔼​\[∑t=0H−1γt​r​\(st,at\)\]J\_\{H\}\(\\pi,\\mathcal\{M\}\):=\\mathbb\{E\}\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\], where the expectation is over trajectories generated by executingπ\\piinℳ\\mathcal\{M\}\. Define the discounted cumulative reward from timettonward asGt:=∑t′=tH−1γt′−t​r​\(st′,at′\)G\_\{t\}:=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}r\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\), so thatG0G\_\{0\}is the discounted return of the full trajectory andJH​\(π,ℳ\)=𝔼​\[G0\]J\_\{H\}\(\\pi,\\mathcal\{M\}\)=\\mathbb\{E\}\[G\_\{0\}\]\. Define the finite\-horizon policy gradient bygH:=∇θJH​\(πθ,ℳ\)g\_\{H\}:=\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\mathcal\{M\}\)\. Define the noisy counterpart ofGtG\_\{t\}asG^t:=∑t′=tH−1γt′−t​r^t′\\hat\{G\}\_\{t\}:=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\hat\{r\}\_\{t^\{\\prime\}\}, wherer^t\\hat\{r\}\_\{t\}denotes the observed reward at timett\. Thereinforceestimator ofgHg\_\{H\}computed from a single trajectory isg^\(1\):=∑t=0H−1∇θlog⁡πθ​\(at∣st\)​G^t\\hat\{g\}^\{\(1\)\}:=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,\\hat\{G\}\_\{t\}\. To estimategHg\_\{H\}, sampleK≥1K\\geq 1independent trajectories by executingπθ\\pi\_\{\\theta\}inℳ\\mathcal\{M\}\. For eachk∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}, letg^\(k\)\\hat\{g\}^\{\(k\)\}denote thereinforceestimator computed on thekkth trajectory, and defineg^:=1K​∑k=1Kg^\(k\)\\hat\{g\}:=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\hat\{g\}^\{\(k\)\}\. The estimatorsg^\(1\),…,g^\(K\)\\hat\{g\}^\{\(1\)\},\\ldots,\\hat\{g\}^\{\(K\)\},g^\\hat\{g\}, and their noise\-free counterparts are vector\-valued\. To measure their dispersion with a single scalar, we use the natural generalization of scalar variance obtained by adding coordinate variances: forz=\(z1,…,zd\)z=\(z\_\{1\},\\ldots,z\_\{d\}\), writeVar​\[z\]:=∑i=1dVar​\[zi\]\\mathrm\{Var\}\[z\]:=\\sum\_\{i=1\}^\{d\}\\mathrm\{Var\}\[z\_\{i\}\]\.

###### Theorem 2\(Finite\-horizonreinforceunder noisy rewards\)\.

AssumeWH2:=𝔼\[max0≤t≤H−1∥∇θlogπθ\(at∣st\)∥2\]<∞W\_\{H\}^\{2\}:=\\mathbb\{E\}\\bigl\[\\max\_\{0\\leq t\\leq H\-1\}\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\\|^\{2\}\\bigr\]<\\infty\. Suppose the rewards are observed with additive noise,r^t=r​\(st,at\)\+ηt\\hat\{r\}\_\{t\}=r\(s\_\{t\},a\_\{t\}\)\+\\eta\_\{t\}, where the noise variablesηt\\eta\_\{t\}are i\.i\.d\. with𝔼​\[ηt\]=0\\mathbb\{E\}\[\\eta\_\{t\}\]=0andVar​\[ηt\]=ση2<∞\\mathrm\{Var\}\[\\eta\_\{t\}\]=\\sigma\_\{\\eta\}^\{2\}<\\infty, and eachηt\\eta\_\{t\}is independent of the state\-action history\(\(s0,a0\),…,\(st,at\)\)\(\(s\_\{0\},a\_\{0\}\),\\ldots,\(s\_\{t\},a\_\{t\}\)\)\. Theng^\\hat\{g\}satisfies

𝔼​\[g^\]=gH,Var​\[g^\]≤Var​\[g^\]η≡0\+ση2​H​WH2K​\(1−γ\)2\.\\mathbb\{E\}\[\\hat\{g\}\]=g\_\{H\},\\qquad\\mathrm\{Var\}\[\\hat\{g\}\]\\leq\\mathrm\{Var\}\[\\hat\{g\}\]\_\{\\eta\\equiv 0\}\+\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{K\(1\-\\gamma\)^\{2\}\}\.\(5\)whereVar​\[g^\]η≡0\\mathrm\{Var\}\[\\hat\{g\}\]\_\{\\eta\\equiv 0\}denotes the variance of the same estimator when the rewards are noise\-free\.

###### Proof\.

See[Section˜B\.6](https://arxiv.org/html/2605.06732#A2.SS6)\. ∎

In many settings, practitioners can pay more per reward annotation to obtain less noisy rewards, for example by averaging multiple annotators or using a more careful annotation pipeline\. We use[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)to ask how a fixed budget for reward annotations should be split between fidelity \(lower noise per annotation\) and quantity \(more rollouts at higher noise\)\.

Letc\>0c\>0denote the per\-rollout cost of acquiring reward annotations along that rollout, and defineση2​\(c\):=Var​\[ηt∣per\-rollout annotation cost equals​c\]∈\[0,∞\)\\sigma\_\{\\eta\}^\{2\}\(c\):=\\mathrm\{Var\}\[\\eta\_\{t\}\\mid\\text\{per\-rollout annotation cost equals \}c\]\\in\[0,\\infty\)as the variance of the reward noiseηt\\eta\_\{t\}from[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)when reward annotations are acquired at per\-rollout costcc\. We assume thatση2:\(0,∞\)→\[0,∞\)\\sigma\_\{\\eta\}^\{2\}\\colon\(0,\\infty\)\\to\[0,\\infty\)is measurable\. Given a budgetB\>0B\>0for reward annotations, the number of independent rollouts that fit in the budget at fidelityccisK=B/cK=B/c\.

###### Corollary 2\(Optimal fidelity for noise\-induced variance\)\.

DefineΦ​\(c\):=c​ση2​\(c\)\\Phi\(c\):=c\\,\\sigma\_\{\\eta\}^\{2\}\(c\)\. Under the assumptions of[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)withVar​\[ηt\]=ση2​\(c\)\\mathrm\{Var\}\[\\eta\_\{t\}\]=\\sigma\_\{\\eta\}^\{2\}\(c\)andK=B/cK=B/c, the upper bound on the noise\-induced excess variance from[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)equalsΦ​\(c\)​H​WH2/\(B​\(1−γ\)2\)\\Phi\(c\)\\,H\\,W\_\{H\}^\{2\}/\(B\\,\(1\-\\gamma\)^\{2\}\)\. Consequently, anyc∗∈arg​minc\>0⁡Φ​\(c\)c^\{\*\}\\in\\operatorname\*\{arg\\,min\}\_\{c\>0\}\\Phi\(c\)minimizes this upper bound overc\>0c\>0\.

###### Proof\.

SubstitutingK=B/cK=B/cinto the upper bound from[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)givesση2​\(c\)​H​WH2/\(K​\(1−γ\)2\)=Φ​\(c\)​H​WH2/\(B​\(1−γ\)2\)\\sigma\_\{\\eta\}^\{2\}\(c\)\\,H\\,W\_\{H\}^\{2\}/\(K\\,\(1\-\\gamma\)^\{2\}\)=\\Phi\(c\)\\,H\\,W\_\{H\}^\{2\}/\(B\\,\(1\-\\gamma\)^\{2\}\)\. The prefactorH​WH2/\(B​\(1−γ\)2\)H\\,W\_\{H\}^\{2\}/\(B\\,\(1\-\\gamma\)^\{2\}\)is non\-negative and does not depend oncc, so the upper bound andΦ​\(c\)\\Phi\(c\)share their minimizers overc\>0c\>0\. ∎

[Corollary˜2](https://arxiv.org/html/2605.06732#Thmcorollary2)reduces the choice of fidelity for reward annotations to minimizingΦ​\(c\)\\Phi\(c\)\. We illustrate the consequences with three examples of how the noise varianceση2​\(c\)\\sigma\_\{\\eta\}^\{2\}\(c\)depends on the costcc, summarized in[Figure˜5](https://arxiv.org/html/2605.06732#A2.F5)\.

#### Power\-law fidelity\.

Supposeση2​\(c\)=A​c−p\\sigma\_\{\\eta\}^\{2\}\(c\)=A\\,c^\{\-p\}for constantsA,p\>0A,p\>0, so that each multiplicative increase in cost yields a fixed multiplicative reduction in noise variance\. ThenΦ​\(c\)=A​c1−p\\Phi\(c\)=A\\,c^\{1\-p\}, which is strictly decreasing inccwhenp\>1p\>1, strictly increasing inccwhenp<1p<1, and constant whenp=1p=1\. The exponentp=1p=1therefore separates two regimes: whenp\>1p\>1, the bound is minimized by paying for the highest\-fidelity annotations available; whenp<1p<1, the bound is minimized by paying as little as possible per annotation and using the saved budget for additional rollouts\.

#### Bounded fidelity\.

Supposeση2​\(c\)=σ02​\(1−c/cmax\)\\sigma\_\{\\eta\}^\{2\}\(c\)=\\sigma\_\{0\}^\{2\}\\,\(1\-c/c\_\{\\max\}\)on\(0,cmax\]\(0,c\_\{\\max\}\]for constantsσ02,cmax\>0\\sigma\_\{0\}^\{2\},c\_\{\\max\}\>0, modeling a setting in which annotations have varianceσ02\\sigma\_\{0\}^\{2\}asc→0c\\to 0and become noise\-free at the finite costcmaxc\_\{\\max\}\. ThenΦ​\(c\)=σ02​c​\(1−c/cmax\)\\Phi\(c\)=\\sigma\_\{0\}^\{2\}\\,c\\,\(1\-c/c\_\{\\max\}\)is a downward parabola inccthat is maximized atc=cmax/2c=c\_\{\\max\}/2, attains the value0atc=cmaxc=c\_\{\\max\}, and approaches0asc→0c\\to 0\. Both extremes of the cost range therefore minimize the bound, and intermediate fidelities are strictly worse\.

#### Irreducible noise floor\.

Supposeση2​\(c\)=σfloor2\+A/c\\sigma\_\{\\eta\}^\{2\}\(c\)=\\sigma\_\{\\mathrm\{floor\}\}^\{2\}\+A/cfor constantsσfloor2,A\>0\\sigma\_\{\\mathrm\{floor\}\}^\{2\},A\>0, modeling a setting in which no level of spending can drive the noise variance belowσfloor2\\sigma\_\{\\mathrm\{floor\}\}^\{2\}\. ThenΦ​\(c\)=σfloor2​c\+A\\Phi\(c\)=\\sigma\_\{\\mathrm\{floor\}\}^\{2\}\\,c\+Ais strictly increasing incc, so the bound is minimized byc→0c\\to 0: when reward noise has a floor that money cannot remove, the optimal allocation buys the cheapest annotations and relies on the1/K1/Kfactor in[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)to reduce variance\.

Together, these cases show that the shape ofΦ​\(c\)\\Phi\(c\)determines whether the budget should be spent on fidelity, on quantity, or on one of the two extremes\.

The preceding results assume the reward noise is zero\-mean\. We note that this assumption is essential: averaging over more trajectories cannot remove a systematic reward bias\.

###### Proposition 2\(Finite\-horizonreinforceunder biased rewards\)\.

Letb:𝒮×𝒜→ℝb:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}be a reward bias function, definer~​\(s,a\):=r​\(s,a\)\+b​\(s,a\)\\tilde\{r\}\(s,a\):=r\(s,a\)\+b\(s,a\), and defineℳ~:=\(𝒮,𝒜,f,r~,γ\)\\tilde\{\\mathcal\{M\}\}:=\(\\mathcal\{S\},\\mathcal\{A\},f,\\tilde\{r\},\\gamma\)\. For trajectories sampled by executingπθ\\pi\_\{\\theta\}under the true dynamicsff, letg~\(1\),…,g~\(K\)\\tilde\{g\}^\{\(1\)\},\\ldots,\\tilde\{g\}^\{\(K\)\}be independentreinforceestimators computed using the biased rewardsr~​\(st,at\)\\tilde\{r\}\(s\_\{t\},a\_\{t\}\), and defineg~:=1K​∑k=1Kg~\(k\)\\tilde\{g\}:=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\tilde\{g\}^\{\(k\)\}\. DefineBH​\(θ\):=𝔼​\[∑t=0H−1γt​b​\(st,at\)\]B\_\{H\}\(\\theta\):=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}b\(s\_\{t\},a\_\{t\}\)\\right\], where the expectation is over trajectories generated by executingπθ\\pi\_\{\\theta\}under the true dynamicsff\. Then

𝔼​\[g~\]=∇θJH​\(πθ,ℳ~\)=gH\+∇θBH​\(θ\)\.\\mathbb\{E\}\[\\tilde\{g\}\]=\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\tilde\{\\mathcal\{M\}\}\)=g\_\{H\}\+\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\.\(6\)IfVar​\[g~\(1\)\]<∞\\mathrm\{Var\}\[\\tilde\{g\}^\{\(1\)\}\]<\\infty, then

𝔼​\[‖g~−gH‖2\]=1K​Var​\[g~\(1\)\]\+‖∇θBH​\(θ\)‖2\.\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-g\_\{H\}\\\|^\{2\}\\right\]=\\frac\{1\}\{K\}\\mathrm\{Var\}\[\\tilde\{g\}^\{\(1\)\}\]\+\\\|\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\\\|^\{2\}\.\(7\)Consequently, when∇θBH​\(θ\)≠0\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\\neq 0, averaging over more trajectories reduces the variance term but does not remove the bias as an estimator ofgHg\_\{H\}\.

###### Proof\.

See[Section˜B\.8](https://arxiv.org/html/2605.06732#A2.SS8)\. ∎

## 6Discussion

Although the dynamicsf:𝒮×𝒜→𝒮f:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathcal\{S\}and rewardr:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}depend only on the current state and action, this does not require the underlying environment to be memoryless: any dependence on past states and actions can be absorbed into𝒮\\mathcal\{S\}by taking the state to encode a summary of the past, e\.g\. via recurrent neural networks\(Rumelhartet al\.,[1986](https://arxiv.org/html/2605.06732#bib.bib7); Elman,[1990](https://arxiv.org/html/2605.06732#bib.bib8); Hochreiter and Schmidhuber,[1997](https://arxiv.org/html/2605.06732#bib.bib9); Henaffet al\.,[2016](https://arxiv.org/html/2605.06732#bib.bib10)\)\.

## References

- K\. Asadi, E\. Cater, D\. Misra, and M\. L\. Littman \(2018a\)Equivalence between wasserstein and value\-aware loss for model\-based reinforcement learning\.arXiv preprint arXiv:1806\.01265\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1)\.
- K\. Asadi, D\. Misra, and M\. Littman \(2018b\)Lipschitz continuity in model\-based reinforcement learning\.InInternational conference on machine learning,pp\. 264–273\.Cited by:[item 1](https://arxiv.org/html/2605.06732#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2605.06732#S1.p2.1),[§2](https://arxiv.org/html/2605.06732#S2.p2.1),[§3](https://arxiv.org/html/2605.06732#S3.p2.3)\.
- M\. Assran, Q\. Duval, I\. Misra, P\. Bojanowski, P\. Vincent, M\. Rabbat, Y\. LeCun, and N\. Ballas \(2023\)Self\-supervised learning from images with a joint\-embedding predictive architecture\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15619–15629\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px5.p1.1)\.
- C\. Bellinger, R\. Coles, M\. Crowley, and I\. Tamblyn \(2020\)Active measure reinforcement learning for observation cost minimization\.arXiv preprint arXiv:2005\.12697\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1)\.
- X\. Cai, W\. Wang, F\. Liu, T\. Liu, G\. Niu, and M\. Sugiyama \(2025\)Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers\.arXiv preprint arXiv:2510\.00915\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p4.1)\.
- P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. Amodei \(2017\)Deep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1)\.
- M\. Deisenroth and C\. E\. Rasmussen \(2011\)PILCO: a model\-based and data\-efficient approach to policy search\.InProceedings of the 28th International Conference on machine learning \(ICML\-11\),pp\. 465–472\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px1.p1.1)\.
- J\. L\. Elman \(1990\)Finding structure in time\.Cognitive science14\(2\),pp\. 179–211\.Cited by:[§6](https://arxiv.org/html/2605.06732#S6.p1.3)\.
- A\. Farahmand, A\. Barreto, and D\. Nikovski \(2017\)Value\-aware loss function for model\-based reinforcement learning\.InArtificial Intelligence and Statistics,pp\. 1486–1494\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,pp\. 10835–10866\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p4.1)\.
- H\. Gouk, E\. Frank, B\. Pfahringer, and M\. J\. Cree \(2021\)Regularisation of neural networks by enforcing lipschitz continuity\.Machine Learning110\(2\),pp\. 393–416\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px5.p1.1)\.
- E\. Greensmith, P\. L\. Bartlett, and J\. Baxter \(2004\)Variance reduction techniques for gradient estimates in reinforcement learning\.Journal of Machine Learning Research5\(Nov\),pp\. 1471–1530\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1)\.
- D\. Ha and J\. Schmidhuber \(2018\)World models\.arXiv preprint arXiv:1803\.101222\(3\),pp\. 440\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px1.p1.1)\.
- D\. Hafner, T\. Lillicrap, J\. Ba, and M\. Norouzi \(2020\)Dream to control: learning behaviors by latent imagination\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=S1lOTC4tDS)Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px1.p1.1)\.
- D\. Hafner, J\. Pasukonis, J\. Ba, and T\. Lillicrap \(2025a\)Mastering diverse control tasks through world models\.Nature640\(8059\),pp\. 647–653\.Cited by:[§1](https://arxiv.org/html/2605.06732#S1.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p1.1)\.
- D\. Hafner, W\. Yan, and T\. Lillicrap \(2025b\)Training agents inside of scalable world models\.arXiv preprint arXiv:2509\.24527\.Cited by:[§1](https://arxiv.org/html/2605.06732#S1.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p1.1)\.
- H\. Han and H\. Yang \(2026\)Non\-uniform noise\-to\-signal ratio in the reinforce policy\-gradient estimator\.arXiv preprint arXiv:2602\.01460\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1)\.
- N\. A\. Hansen, H\. Su, and X\. Wang \(2022\)Temporal difference learning for model predictive control\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 8387–8406\.External Links:[Link](https://proceedings.mlr.press/v162/hansen22a.html)Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px1.p1.1)\.
- M\. Henaff, A\. Szlam, and Y\. LeCun \(2016\)Recurrent orthogonal networks and long\-memory tasks\.InInternational Conference on Machine Learning,pp\. 2034–2042\.Cited by:[§6](https://arxiv.org/html/2605.06732#S6.p1.3)\.
- J\. Hilton, J\. Tang, and J\. Schulman \(2023\)Scaling laws for single\-agent reinforcement learning\.arXiv preprint arXiv:2301\.13442\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1)\.
- S\. Hochreiter and J\. Schmidhuber \(1997\)Long short\-term memory\.Neural computation9\(8\),pp\. 1735–1780\.Cited by:[§6](https://arxiv.org/html/2605.06732#S6.p1.3)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.1555610\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p3.1)\.
- X\. Huang and S\. H\. Choi \(2022\)An efficient simulation\-based policy improvement with optimal computing budget allocation based on accumulated samples\.Electronics11\(7\),pp\. 1141\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1)\.
- M\. Janner, J\. Fu, M\. Zhang, and S\. Levine \(2019\)When to trust your model: model\-based policy optimization\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p1.1)\.
- C\. Jin, A\. Krishnamurthy, M\. Simchowitz, and T\. Yu \(2020\)Reward\-free exploration for reinforcement learning\.InInternational Conference on Machine Learning,pp\. 4870–4879\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p3.1)\.
- S\. Kakade and J\. Langford \(2002\)Approximately optimal approximate reinforcement learning\.InProceedings of the International Conference on Machine Learning \(ICML\),pp\. 267–274\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1),[§B\.5](https://arxiv.org/html/2605.06732#A2.SS5.SSS0.Px4.p8.1)\.
- J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei \(2020\)Scaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p3.1)\.
- M\. Kearns and S\. Singh \(2002\)Near\-optimal reinforcement learning in polynomial time\.Machine learning49\(2\),pp\. 209–232\.Cited by:[§1](https://arxiv.org/html/2605.06732#S1.p2.1),[§2](https://arxiv.org/html/2605.06732#S2.p2.1)\.
- Y\. LeCunet al\.\(2022\)A path towards autonomous machine intelligence version 0\.9\. 2, 2022\-06\-27\.Open Review62\(1\),pp\. 1–62\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p5.1),[§3](https://arxiv.org/html/2605.06732#S3.p1.3)\.
- S\. Lobel and R\. Parr \(2024\)An optimal tightness bound for the simulation lemma\.InReinforcement Learning Conference,External Links:[Link](https://openreview.net/forum?id=RcoIAfiM5g)Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1)\.
- T\. Miyato, T\. Kataoka, M\. Koyama, and Y\. Yoshida \(2018\)Spectral normalization for generative adversarial networks\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p5.1)\.
- R\. Munos \(2003\)Error bounds for approximate policy iteration\.Proceedings of the International Conference on Machine Learning \(ICML\),pp\. 560–567\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1),[§B\.5](https://arxiv.org/html/2605.06732#A2.SS5.SSS0.Px4.p8.1)\.
- T\. Pearce, T\. Rashid, D\. Bignell, R\. Georgescu, S\. Devlin, and K\. Hofmann \(2025\)Scaling laws for pre\-training agents and world models\.InInternational Conference on Machine Learning,pp\. 48542–48562\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px3.p1.1)\.
- D\. E\. Rumelhart, G\. E\. Hinton, and R\. J\. Williams \(1986\)Learning representations by back\-propagating errors\.nature323\(6088\),pp\. 533–536\.Cited by:[§6](https://arxiv.org/html/2605.06732#S6.p1.3)\.
- J\. Schrittwieser, I\. Antonoglou, T\. Hubert, K\. Simonyan, L\. Sifre, S\. Schmitt, A\. Guez, E\. Lockhart, D\. Hassabis, T\. Graepel,et al\.\(2020\)Mastering atari, go, chess and shogi by planning with a learned model\.Nature588\(7839\),pp\. 604–609\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2605.06732#S1.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. Christiano \(2020\)Learning to summarize with human feedback\.Advances in neural information processing systems33,pp\. 3008–3021\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1)\.
- R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour \(1999\)Policy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1)\.
- R\. S\. Sutton \(1990\)Integrated architectures for learning, planning, and reacting based on approximating dynamic programming\.InMachine learning proceedings 1990,pp\. 216–224\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p1.1)\.
- E\. Talvitie \(2018\)Learning the reward function for a misspecified model\.InInternational Conference on Machine Learning,pp\. 4838–4847\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px2.p1.1)\.
- Y\. Wang, O\. Bounou, G\. Zhou, R\. Balestriero, T\. G\. Rudner, Y\. LeCun, and M\. Ren \(2026\)Temporal straightening for latent planning\.arXiv preprint arXiv:2603\.12231\.Cited by:[item 2](https://arxiv.org/html/2605.06732#S1.I1.i2.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p5.1),[§3](https://arxiv.org/html/2605.06732#S3.p5.8),[Definition 1](https://arxiv.org/html/2605.06732#Thmdefinition1)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[Appendix A](https://arxiv.org/html/2605.06732#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2605.06732#S2.p4.1),[§5](https://arxiv.org/html/2605.06732#S5.p2.29)\.
- X\. Zhang, Y\. Chen, X\. Zhu, and W\. Sun \(2021\)Robust policy gradient against strong data corruption\.InInternational Conference on Machine Learning,pp\. 12391–12401\.Cited by:[§2](https://arxiv.org/html/2605.06732#S2.p4.1)\.

## Appendix AAdditional related work

This appendix collects related works that inform the setting of[Section˜2](https://arxiv.org/html/2605.06732#S2)but are not load\-bearing for the novelty claims of[Lemmas˜1](https://arxiv.org/html/2605.06732#Thmlemma1),[1](https://arxiv.org/html/2605.06732#Thmtheorem1),[2](https://arxiv.org/html/2605.06732#Thmtheorem2),[2](https://arxiv.org/html/2605.06732#Thmcorollary2),[2](https://arxiv.org/html/2605.06732#Thmproposition2),[1](https://arxiv.org/html/2605.06732#Thmcorollary1)and[1](https://arxiv.org/html/2605.06732#Thmproposition1)\.

#### Latent\-state world models\.

Beyond the works cited in[Section˜2](https://arxiv.org/html/2605.06732#S2), several lines of work shape the modern training\-in\-imagination paradigm\.Deisenroth and Rasmussen \[[2011](https://arxiv.org/html/2605.06732#bib.bib18)\]fit a Gaussian\-process dynamics model and propagate uncertainty through imagined trajectories under a known reward\.Ha and Schmidhuber \[[2018](https://arxiv.org/html/2605.06732#bib.bib15)\]learn a latent dynamics model from pixels and train a policy primarily inside this learned model\. Dreamer\[Hafneret al\.,[2020](https://arxiv.org/html/2605.06732#bib.bib16)\]learns a latent\-space dynamics model together with a learned reward predictor and optimizes the policy entirely on imagined latent rollouts\.Schrittwieseret al\.\[[2020](https://arxiv.org/html/2605.06732#bib.bib17)\]learn a value\-equivalent latent model whose dynamics, reward, and value predictions are trained jointly to support planning, andHansenet al\.\[[2022](https://arxiv.org/html/2605.06732#bib.bib19)\]combine a learned latent dynamics and reward model with short\-horizon planning and a learned terminal value\.

#### Simulation\-style bounds and value\-aware model learning\.

The simulation\-lemma family extends in several directions\.Kakade and Langford \[[2002](https://arxiv.org/html/2605.06732#bib.bib14)\]express the difference in expected discounted return between two policies as an expectation of single\-step advantages under one of them, which underpins conservative policy iteration\.Munos \[[2003](https://arxiv.org/html/2605.06732#bib.bib13)\]givesLpL\_\{p\}\-style error\-propagation bounds for approximate policy iteration, andLobel and Parr \[[2024](https://arxiv.org/html/2605.06732#bib.bib24)\]recently establish optimal tightness for the simulation lemma\. A parallel line of work asks whether the model loss should target value prediction rather than raw transition accuracy: value\-aware model learning\[Farahmandet al\.,[2017](https://arxiv.org/html/2605.06732#bib.bib21)\]replaces the next\-state likelihood objective with a loss measuring the worst\-case discrepancy between the true and learned dynamics on expected values over a class of value functions,Asadiet al\.\[[2018a](https://arxiv.org/html/2605.06732#bib.bib22)\]show that, restricted to a 1\-Lipschitz value\-function class, this loss is equivalent to the Wasserstein distance between the dynamics, andTalvitie \[[2018](https://arxiv.org/html/2605.06732#bib.bib23)\]addresses how to learn the reward function on states drawn from a misspecified dynamics model\.

#### Active observation and simulation budget allocation\.

Active\-measure reinforcement learning chooses when to pay for an observation under explicit observation costs\[Bellingeret al\.,[2020](https://arxiv.org/html/2605.06732#bib.bib30)\]\. In the simulation\-optimization literature, optimal computing budget allocation divides a fixed simulation budget across candidate policies to maximize the probability of selecting the best one\[Huang and Choi,[2022](https://arxiv.org/html/2605.06732#bib.bib31)\]\. Power\-law fits analogous to those ofKaplanet al\.\[[2020](https://arxiv.org/html/2605.06732#bib.bib25)\]andHoffmannet al\.\[[2022](https://arxiv.org/html/2605.06732#bib.bib26)\]have since been reported for single\-agent reinforcement learning and for pre\-training of agents and world models\[Hiltonet al\.,[2023](https://arxiv.org/html/2605.06732#bib.bib27), Pearceet al\.,[2025](https://arxiv.org/html/2605.06732#bib.bib28)\]\.

#### Policy\-gradient theory and reward modeling\.

BeyondWilliams \[[1992](https://arxiv.org/html/2605.06732#bib.bib1)\], the policy\-gradient theorem ofSuttonet al\.\[[1999](https://arxiv.org/html/2605.06732#bib.bib32)\]and the variance\-reduction analysis ofGreensmithet al\.\[[2004](https://arxiv.org/html/2605.06732#bib.bib33)\]provide the standard analytic toolkit invoked by[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)\.Han and Yang \[[2026](https://arxiv.org/html/2605.06732#bib.bib36)\]characterize how thereinforcenoise\-to\-signal ratio varies non\-uniformly across the parameter landscape\. Reinforcement learning from human feedback trains reward models from preferences\[Christianoet al\.,[2017](https://arxiv.org/html/2605.06732#bib.bib37), Stiennonet al\.,[2020](https://arxiv.org/html/2605.06732#bib.bib38)\], the empirical setting in whichGaoet al\.\[[2023](https://arxiv.org/html/2605.06732#bib.bib39)\]document Goodhart\-style overoptimization\.

#### Representation learning and Lipschitz regularization\.

Assranet al\.\[[2023](https://arxiv.org/html/2605.06732#bib.bib40)\]provide a concrete image instantiation of joint\-embedding predictive architectures, andGouket al\.\[[2021](https://arxiv.org/html/2605.06732#bib.bib42)\]propose projection\-based mechanisms for enforcing Lipschitz continuity during training, with upper bounds applicable to multiplepp\-norms beyond the spectral norm\.

## Appendix BFull proofs

### B\.1Proof of Lemma[1](https://arxiv.org/html/2605.06732#Thmlemma1)

See[1](https://arxiv.org/html/2605.06732#Thmlemma1)

###### Proof\.

Fix anLπL\_\{\\pi\}\-Lipschitz policyπ\\piand a shared initial states0=s^0s\_\{0\}=\\hat\{s\}\_\{0\}\. Let\{st\}\\\{s\_\{t\}\\\}denote the state trajectory generated by executingπ\\piunder the true dynamicsff, and\{s^t\}\\\{\\hat\{s\}\_\{t\}\\\}the trajectory underf^\\hat\{f\}, both starting froms0s\_\{0\}\. The actions differ because the policy is evaluated at different states:

at=π​\(st\),a^t=π​\(s^t\)\.\\displaystyle a\_\{t\}=\\pi\(s\_\{t\}\),\\qquad\\hat\{a\}\_\{t\}=\\pi\(\\hat\{s\}\_\{t\}\)\.
At each steptt:

r​\(st,at\)−r^​\(s^t,a^t\)\\displaystyle r\(s\_\{t\},a\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)=\[r​\(st,at\)−r​\(s^t,a^t\)\]\+\[r​\(s^t,a^t\)−r^​\(s^t,a^t\)\]\.\\displaystyle=\\bigl\[r\(s\_\{t\},a\_\{t\}\)\-r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\bigr\]\+\\bigl\[r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\bigr\]\.\(8\)
LetLcomp:=Lf​\(1\+Lπ\)L\_\{\\mathrm\{comp\}\}:=L\_\{f\}\(1\+L\_\{\\pi\}\)\. By induction:

‖st−s^t‖≤εdyn​∑k=0t−1Lcompk\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\\leq\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\}
*Base case*\(t=0t=0\):s0=s^0s\_\{0\}=\\hat\{s\}\_\{0\}, so‖s0−s^0‖=0\\\|s\_\{0\}\-\\hat\{s\}\_\{0\}\\\|=0\.

*Inductive step*: Assume‖st−s^t‖≤εdyn​∑k=0t−1Lcompk\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\\leq\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\}\.

‖st\+1−s^t\+1‖\\displaystyle\\\|s\_\{t\+1\}\-\\hat\{s\}\_\{t\+1\}\\\|=‖f​\(st,at\)−f^​\(s^t,a^t\)‖\\displaystyle=\\\|f\(s\_\{t\},a\_\{t\}\)\-\\hat\{f\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\\|≤‖f​\(st,at\)−f​\(s^t,a^t\)‖\+‖f​\(s^t,a^t\)−f^​\(s^t,a^t\)‖\\displaystyle\\leq\\\|f\(s\_\{t\},a\_\{t\}\)\-f\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\\|\+\\\|f\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\-\\hat\{f\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\\|≤Lf​\(‖st−s^t‖\+‖at−a^t‖\)\+εdyn\\displaystyle\\leq L\_\{f\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\\|a\_\{t\}\-\\hat\{a\}\_\{t\}\\\|\\bigr\)\+\\varepsilon\_\{\\mathrm\{dyn\}\}=Lf​\(‖st−s^t‖\+‖π​\(st\)−π​\(s^t\)‖\)\+εdyn\\displaystyle=L\_\{f\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\\|\\pi\(s\_\{t\}\)\-\\pi\(\\hat\{s\}\_\{t\}\)\\\|\\bigr\)\+\\varepsilon\_\{\\mathrm\{dyn\}\}≤Lf​\(‖st−s^t‖\+Lπ​‖st−s^t‖\)\+εdyn\\displaystyle\\leq L\_\{f\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+L\_\{\\pi\}\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\\bigr\)\+\\varepsilon\_\{\\mathrm\{dyn\}\}≤Lf​\(1\+Lπ\)​‖st−s^t‖\+εdyn\\displaystyle\\leq L\_\{f\}\(1\+L\_\{\\pi\}\)\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\varepsilon\_\{\\mathrm\{dyn\}\}=Lcomp​‖st−s^t‖\+εdyn\\displaystyle=L\_\{\\mathrm\{comp\}\}\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\varepsilon\_\{\\mathrm\{dyn\}\}≤Lcomp​\(εdyn​∑k=0t−1Lcompk\)\+εdyn\\displaystyle\\leq L\_\{\\mathrm\{comp\}\}\\left\(\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\}\\right\)\+\\varepsilon\_\{\\mathrm\{dyn\}\}=εdyn​∑k=0t−1Lcompk\+1\+εdyn\\displaystyle=\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\+1\}\+\\varepsilon\_\{\\mathrm\{dyn\}\}=εdyn​∑k=0t−1Lcompk\+1\+εdyn​Lcomp0\\displaystyle=\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\+1\}\+\\varepsilon\_\{\\mathrm\{dyn\}\}L\_\{\\mathrm\{comp\}\}^\{0\}=εdyn​∑j=1tLcompj\+εdyn​Lcomp0\\displaystyle=\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{j=1\}^\{t\}L\_\{\\mathrm\{comp\}\}^\{j\}\+\\varepsilon\_\{\\mathrm\{dyn\}\}L\_\{\\mathrm\{comp\}\}^\{0\}=εdyn​∑k=0tLcompk\.\\displaystyle=\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\}L\_\{\\mathrm\{comp\}\}^\{k\}\.
*Error from using the learned reward function:*\|r​\(s^t,a^t\)−r^​\(s^t,a^t\)\|≤εrew\|r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\|\\leq\\varepsilon\_\{\\mathrm\{rew\}\}, by the definition ofεrew\\varepsilon\_\{\\mathrm\{rew\}\}\.

*Error from evaluating the true reward at different state\-action pairs:*

\|r​\(st,at\)−r​\(s^t,a^t\)\|\\displaystyle\|r\(s\_\{t\},a\_\{t\}\)\-r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\|≤Lr​\(‖st−s^t‖\+‖at−a^t‖\)\\displaystyle\\leq L\_\{r\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\\|a\_\{t\}\-\\hat\{a\}\_\{t\}\\\|\\bigr\)=Lr​\(‖st−s^t‖\+‖π​\(st\)−π​\(s^t\)‖\)\\displaystyle=L\_\{r\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+\\\|\\pi\(s\_\{t\}\)\-\\pi\(\\hat\{s\}\_\{t\}\)\\\|\\bigr\)≤Lr​\(‖st−s^t‖\+Lπ​‖st−s^t‖\)\\displaystyle\\leq L\_\{r\}\\bigl\(\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\+L\_\{\\pi\}\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|\\bigr\)=Lr​\(1\+Lπ\)​‖st−s^t‖\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\\|s\_\{t\}\-\\hat\{s\}\_\{t\}\\\|≤Lr​\(1\+Lπ\)​εdyn​∑k=0t−1Lcompk\.\\displaystyle\\leq L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\}\.
Using the definition of discounted return and[Equation˜8](https://arxiv.org/html/2605.06732#A2.E8),

\|J​\(π,ℳ\)−J​\(π,ℳ^\)\|\\displaystyle\|J\(\\pi,\\mathcal\{M\}\)\-J\(\\pi,\\hat\{\\mathcal\{M\}\}\)\|=\|∑t=0∞γt​\(r​\(st,at\)−r^​\(s^t,a^t\)\)\|\\displaystyle=\\left\|\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\bigl\(r\(s\_\{t\},a\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\bigr\)\\right\|≤∑t=0∞γt​\|r​\(st,at\)−r^​\(s^t,a^t\)\|\\displaystyle\\leq\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\|r\(s\_\{t\},a\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\right\|=∑t=0∞γt​\|\(r​\(st,at\)−r​\(s^t,a^t\)\)\+\(r​\(s^t,a^t\)−r^​\(s^t,a^t\)\)\|\\displaystyle=\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\|\\bigl\(r\(s\_\{t\},a\_\{t\}\)\-r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\bigr\)\+\\bigl\(r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\bigr\)\\right\|≤∑t=0∞γt​\|r​\(st,at\)−r​\(s^t,a^t\)\|\+∑t=0∞γt​\|r​\(s^t,a^t\)−r^​\(s^t,a^t\)\|\\displaystyle\\leq\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\|r\(s\_\{t\},a\_\{t\}\)\-r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\right\|\+\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\|r\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\-\\hat\{r\}\(\\hat\{s\}\_\{t\},\\hat\{a\}\_\{t\}\)\\right\|≤Lr​\(1\+Lπ\)​εdyn​∑t=0∞γt​\(∑k=0t−1Lcompk\)\+εrew​∑t=0∞γt\\displaystyle\\leq L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\(\\sum\_\{k=0\}^\{t\-1\}L\_\{\\mathrm\{comp\}\}^\{k\}\\right\)\+\\varepsilon\_\{\\mathrm\{rew\}\}\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}=Lr​\(1\+Lπ\)​εdyn​∑t=0∞∑k=0t−1γt​Lcompk\+11−γ​εrew\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{t=0\}^\{\\infty\}\\sum\_\{k=0\}^\{t\-1\}\\gamma^\{t\}L\_\{\\mathrm\{comp\}\}^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=Lr​\(1\+Lπ\)​εdyn​∑k=0∞∑t=0k≤t−1∞γt​Lcompk\+11−γ​εrew\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{\\infty\}\\sum\_\{\\begin\{subarray\}\{c\}t=0\\\\ k\\leq t\-1\\end\{subarray\}\}^\{\\infty\}\\gamma^\{t\}L\_\{\\mathrm\{comp\}\}^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=Lr​\(1\+Lπ\)​εdyn​∑k=0∞∑t=k\+1∞γt​Lcompk\+11−γ​εrew\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{\\infty\}\\sum\_\{t=k\+1\}^\{\\infty\}\\gamma^\{t\}L\_\{\\mathrm\{comp\}\}^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=Lr​\(1\+Lπ\)​εdyn​∑k=0∞∑j=0∞γk\+1\+j​Lcompk\+11−γ​εrew\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{\\infty\}\\sum\_\{j=0\}^\{\\infty\}\\gamma^\{k\+1\+j\}L\_\{\\mathrm\{comp\}\}^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=Lr​\(1\+Lπ\)​εdyn​∑k=0∞\(γk\+1​∑j=0∞γj\)​Lcompk\+11−γ​εrew\\displaystyle=L\_\{r\}\(1\+L\_\{\\pi\}\)\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{\\infty\}\\left\(\\gamma^\{k\+1\}\\sum\_\{j=0\}^\{\\infty\}\\gamma^\{j\}\\right\)L\_\{\\mathrm\{comp\}\}^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=γ​Lr​\(1\+Lπ\)1−γ​εdyn​∑k=0∞\(γ​Lcomp\)k\+11−γ​εrew\\displaystyle=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}\\sum\_\{k=0\}^\{\\infty\}\(\\gamma L\_\{\\mathrm\{comp\}\}\)^\{k\}\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}Becauseγ​Lcomp<1\\gamma L\_\{\\mathrm\{comp\}\}<1, the geometric series converges\.=γ​Lr​\(1\+Lπ\)1−γ​εdyn​\(11−γ​Lcomp\)\+11−γ​εrew\\displaystyle=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}\\left\(\\frac\{1\}\{1\-\\gamma L\_\{\\mathrm\{comp\}\}\}\\right\)\+\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}=11−γ​εrew\+γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​εdyn\\displaystyle=\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}\+\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}∎

### B\.2Proof of[Proposition˜1](https://arxiv.org/html/2605.06732#Thmproposition1)

See[1](https://arxiv.org/html/2605.06732#Thmproposition1)

###### Proof\.

Fixttsuch thatv​\(zt\)≠0v\(z\_\{t\}\)\\neq 0andv​\(zt\+1\)≠0v\(z\_\{t\+1\}\)\\neq 0, and set

a:=v​\(zt\),b:=v​\(zt\+1\)\.a:=v\(z\_\{t\}\),\\qquad b:=v\(z\_\{t\+1\}\)\.Along the rollout,a=zt\+1−zta=z\_\{t\+1\}\-z\_\{t\}andb=zt\+2−zt\+1b=z\_\{t\+2\}\-z\_\{t\+1\}\. Using theε\\varepsilon\-Lipschitz condition forvvgives

‖b−a‖\\displaystyle\\\|b\-a\\\|=‖v​\(zt\+1\)−v​\(zt\)‖\\displaystyle=\\\|v\(z\_\{t\+1\}\)\-v\(z\_\{t\}\)\\\|≤ε​‖zt\+1−zt‖\\displaystyle\\leq\\varepsilon\\\|z\_\{t\+1\}\-z\_\{t\}\\\|=ε​‖a‖\.\\displaystyle=\\varepsilon\\\|a\\\|\.
By the triangle inequality,

‖b‖\\displaystyle\\\|b\\\|=‖a\+\(b−a\)‖\\displaystyle=\\\|a\+\(b\-a\)\\\|≥‖a‖−‖b−a‖\\displaystyle\\geq\\\|a\\\|\-\\\|b\-a\\\|≥\(1−ε\)​‖a‖\.\\displaystyle\\geq\(1\-\\varepsilon\)\\\|a\\\|\.Define

C:=a⊤​b‖a‖​‖b‖\.C:=\\frac\{a^\{\\top\}b\}\{\\\|a\\\|\\\|b\\\|\}\.Then

‖b−a‖2\\displaystyle\\\|b\-a\\\|^\{2\}=‖a‖2\+‖b‖2−2​a⊤​b\\displaystyle=\\\|a\\\|^\{2\}\+\\\|b\\\|^\{2\}\-2a^\{\\top\}b=‖a‖2\+‖b‖2−2​C​‖a‖​‖b‖\\displaystyle=\\\|a\\\|^\{2\}\+\\\|b\\\|^\{2\}\-2C\\\|a\\\|\\\|b\\\|=\(‖a‖−‖b‖\)2\+2​‖a‖​‖b‖​\(1−C\)\.\\displaystyle=\(\\\|a\\\|\-\\\|b\\\|\)^\{2\}\+2\\\|a\\\|\\\|b\\\|\(1\-C\)\.Rearranging and using\(‖a‖−‖b‖\)2≥0\(\\\|a\\\|\-\\\|b\\\|\)^\{2\}\\geq 0yields

1−C\\displaystyle 1\-C=‖b−a‖2−\(‖a‖−‖b‖\)22​‖a‖​‖b‖\\displaystyle=\\frac\{\\\|b\-a\\\|^\{2\}\-\(\\\|a\\\|\-\\\|b\\\|\)^\{2\}\}\{2\\\|a\\\|\\\|b\\\|\}≤‖b−a‖22​‖a‖​‖b‖\.\\displaystyle\\leq\\frac\{\\\|b\-a\\\|^\{2\}\}\{2\\\|a\\\|\\\|b\\\|\}\.Substituting‖b−a‖≤ε​‖a‖\\\|b\-a\\\|\\leq\\varepsilon\\\|a\\\|and‖b‖≥\(1−ε\)​‖a‖\\\|b\\\|\\geq\(1\-\\varepsilon\)\\\|a\\\|gives

1−C\\displaystyle 1\-C≤ε2​‖a‖22​‖a‖​\(1−ε\)​‖a‖\\displaystyle\\leq\\frac\{\\varepsilon^\{2\}\\\|a\\\|^\{2\}\}\{2\\\|a\\\|\(1\-\\varepsilon\)\\\|a\\\|\}=ε22​\(1−ε\)\.\\displaystyle=\\frac\{\\varepsilon^\{2\}\}\{2\(1\-\\varepsilon\)\}\.Since1−C=ℒcurv​\(t\)1\-C=\\mathcal\{L\}\_\{\\mathrm\{curv\}\}\(t\)by definition, this proves the curvature\-loss bound\.

The functionε↦ε2/\(2​\(1−ε\)\)\\varepsilon\\mapsto\\varepsilon^\{2\}/\(2\(1\-\\varepsilon\)\)is increasing on\(0,1\)\(0,1\)because

dd​ε​ε22​\(1−ε\)=ε​\(2−ε\)2​\(1−ε\)2\>0\.\\frac\{d\}\{d\\varepsilon\}\\frac\{\\varepsilon^\{2\}\}\{2\(1\-\\varepsilon\)\}=\\frac\{\\varepsilon\(2\-\\varepsilon\)\}\{2\(1\-\\varepsilon\)^\{2\}\}\>0\.Therefore lowering the Lipschitz constant ofvvlowers the upper bound whenever the Lipschitz constant remains in\(0,1\)\(0,1\)\. ∎

### B\.3Proof of[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)

See[1](https://arxiv.org/html/2605.06732#Thmtheorem1)

###### Proof\.

For sample counts\(Ndyn,Nrew\)\(N\_\{\\mathrm\{dyn\}\},N\_\{\\mathrm\{rew\}\}\),[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3)gives

εrew​\(Nrew\)=Ar​Nrew−β,εdyn​\(Ndyn\)=Ad​Ndyn−α\.\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\)=A\_\{r\}N\_\{\\mathrm\{rew\}\}^\{\-\\beta\},\\qquad\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\)=A\_\{d\}N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\.Substituting these identities into the upper bound from[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)gives

Ar1−γ​Nrew−β\+γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​Ad​Ndyn−α\.\\frac\{A\_\{r\}\}\{1\-\\gamma\}\\,N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\+\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,A\_\{d\}\\,N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\.Define the sample\-count\-independent coefficients by

Cr:=Ar1−γ,Cd:=γ​Lr​\(1\+Lπ\)​Ad\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\),C\_\{r\}:=\\frac\{A\_\{r\}\}\{1\-\\gamma\},\\qquad C\_\{d\}:=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)A\_\{d\}\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\},so the upper bound after substitution becomes

ℒbound​\(Ndyn,Nrew\):=Cr​Nrew−β\+Cd​Ndyn−α\.\\mathcal\{L\}\_\{\\mathrm\{bound\}\}\(N\_\{\\mathrm\{dyn\}\},N\_\{\\mathrm\{rew\}\}\):=C\_\{r\}N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\+C\_\{d\}N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\.For the fixed budgetBB, the feasible sample counts are exactly the pairs\(Ndyn,Nrew\)\(N\_\{\\mathrm\{dyn\}\},N\_\{\\mathrm\{rew\}\}\)satisfyingcdyn​Ndyn\+crew​Nrew=Bc\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\+c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}=B\. Therefore minimizing the upper bound from[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)over all feasible sample counts is equivalent to minimizingℒbound​\(Ndyn,Nrew\)\\mathcal\{L\}\_\{\\mathrm\{bound\}\}\(N\_\{\\mathrm\{dyn\}\},N\_\{\\mathrm\{rew\}\}\)subject to the same budget constraint\.

Becauseα,β\>0\\alpha,\\beta\>0, the objective decreases as either sample count increases, so the minimizer satisfiesNdyn∗\>0N\_\{\\mathrm\{dyn\}\}^\{\*\}\>0andNrew∗\>0N\_\{\\mathrm\{rew\}\}^\{\*\}\>0\. Define the Lagrangian

Λ​\(Ndyn,Nrew,λ\):=Cd​Ndyn−α\+Cr​Nrew−β\+λ​\(cdyn​Ndyn\+crew​Nrew−B\)\.\\Lambda\(N\_\{\\mathrm\{dyn\}\},N\_\{\\mathrm\{rew\}\},\\lambda\):=C\_\{d\}N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\+C\_\{r\}N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\+\\lambda\(c\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\+c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}\-B\)\.Because the minimizing counts satisfyNdyn∗\>0N\_\{\\mathrm\{dyn\}\}^\{\*\}\>0andNrew∗\>0N\_\{\\mathrm\{rew\}\}^\{\*\}\>0, setting the partial derivatives ofΛ\\Lambdawith respect toNdynN\_\{\\mathrm\{dyn\}\}andNrewN\_\{\\mathrm\{rew\}\}equal to zero gives

α​Cd​Ndyn−α−1\\displaystyle\\alpha\\,C\_\{d\}\\,N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\-1\}=λ​cdyn,\\displaystyle=\\lambda\\,c\_\{\\mathrm\{dyn\}\},β​Cr​Nrew−β−1\\displaystyle\\beta\\,C\_\{r\}\\,N\_\{\\mathrm\{rew\}\}^\{\-\\beta\-1\}=λ​crew\.\\displaystyle=\\lambda\\,c\_\{\\mathrm\{rew\}\}\.
Multiply the equation forNdynN\_\{\\mathrm\{dyn\}\}byNdynN\_\{\\mathrm\{dyn\}\}and the equation forNrewN\_\{\\mathrm\{rew\}\}byNrewN\_\{\\mathrm\{rew\}\}:

α​Cd​Ndyn−α\\displaystyle\\alpha\\,C\_\{d\}\\,N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}=λ​cdyn​Ndyn,\\displaystyle=\\lambda\\,c\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\},β​Cr​Nrew−β\\displaystyle\\beta\\,C\_\{r\}\\,N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}=λ​crew​Nrew\.\\displaystyle=\\lambda\\,c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}\.Now divide the equation forNdynN\_\{\\mathrm\{dyn\}\}by the equation forNrewN\_\{\\mathrm\{rew\}\}:

α​Cd​Ndyn−αβ​Cr​Nrew−β\\displaystyle\\frac\{\\alpha\\,C\_\{d\}\\,N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\}\{\\beta\\,C\_\{r\}\\,N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\}=cdyn​Ndyncrew​Nrew\.\\displaystyle=\\frac\{c\_\{\\mathrm\{dyn\}\}N\_\{\\mathrm\{dyn\}\}\}\{c\_\{\\mathrm\{rew\}\}N\_\{\\mathrm\{rew\}\}\}\.Rearranging this identity gives

NdynNrew=αβ⋅crewcdyn⋅Cd​Ndyn−αCr​Nrew−β\.\\frac\{N\_\{\\mathrm\{dyn\}\}\}\{N\_\{\\mathrm\{rew\}\}\}=\\frac\{\\alpha\}\{\\beta\}\\cdot\\frac\{c\_\{\\mathrm\{rew\}\}\}\{c\_\{\\mathrm\{dyn\}\}\}\\cdot\\frac\{C\_\{d\}N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}\}\{C\_\{r\}N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}\}\.\(9\)Evaluate[Equation˜9](https://arxiv.org/html/2605.06732#A2.E9)at the minimizing counts\(Ndyn∗,Nrew∗\)\(N\_\{\\mathrm\{dyn\}\}^\{\*\},N\_\{\\mathrm\{rew\}\}^\{\*\}\)\. Using the definition ofCdC\_\{d\}together with[Equation˜3](https://arxiv.org/html/2605.06732#S4.E3), we obtain

Cd​\(Ndyn∗\)−α\\displaystyle C\_\{d\}\(N\_\{\\mathrm\{dyn\}\}^\{\*\}\)^\{\-\\alpha\}=γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​Ad​\(Ndyn∗\)−α\\displaystyle=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,A\_\{d\}\(N\_\{\\mathrm\{dyn\}\}^\{\*\}\)^\{\-\\alpha\}=γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)​εdyn∗,\\displaystyle=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\,\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\},Cr​\(Nrew∗\)−β\\displaystyle C\_\{r\}\(N\_\{\\mathrm\{rew\}\}^\{\*\}\)^\{\-\\beta\}=Ar1−γ​\(Nrew∗\)−β\\displaystyle=\\frac\{A\_\{r\}\}\{1\-\\gamma\}\(N\_\{\\mathrm\{rew\}\}^\{\*\}\)^\{\-\\beta\}=11−γ​εrew∗\.\\displaystyle=\\frac\{1\}\{1\-\\gamma\}\\,\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}\.Substituting these two identities yields

Ndyn∗Nrew∗\\displaystyle\\frac\{N\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{N\_\{\\mathrm\{rew\}\}^\{\*\}\}=αβ⋅\(1−γ\)​γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)⋅crewcdyn⋅εdyn∗εrew∗\\displaystyle=\\frac\{\\alpha\}\{\\beta\}\\cdot\(1\-\\gamma\)\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}\\cdot\\frac\{c\_\{\\mathrm\{rew\}\}\}\{c\_\{\\mathrm\{dyn\}\}\}\\cdot\\frac\{\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}\}=αβ⋅γ​Lr​\(1\+Lπ\)1−γ​Lf​\(1\+Lπ\)⋅crewcdyn⋅εdyn∗εrew∗\.\\displaystyle=\\frac\{\\alpha\}\{\\beta\}\\cdot\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\}\\cdot\\frac\{c\_\{\\mathrm\{rew\}\}\}\{c\_\{\\mathrm\{dyn\}\}\}\\cdot\\frac\{\\varepsilon\_\{\\mathrm\{dyn\}\}^\{\*\}\}\{\\varepsilon\_\{\\mathrm\{rew\}\}^\{\*\}\}\.
Becausex↦x−αx\\mapsto x^\{\-\\alpha\}andx↦x−βx\\mapsto x^\{\-\\beta\}are convex on\(0,∞\)\(0,\\infty\)forα,β\>0\\alpha,\\beta\>0, the objective is convex on the feasible set\. Therefore the feasible point satisfying the derivative equations forNdynN\_\{\\mathrm\{dyn\}\}andNrewN\_\{\\mathrm\{rew\}\}is the unique minimizer\. ∎

### B\.4Experiment details: empirical calibration of the decomposed bound

This appendix collects the protocol and per\-configuration construction for the calibration experiment in[Section˜3\.1](https://arxiv.org/html/2605.06732#S3.SS1), whose main\-text result is summarized in[Figure˜3](https://arxiv.org/html/2605.06732#A2.F3)\.

![Refer to caption](https://arxiv.org/html/2605.06732v1/x3.png)Figure 3:The bound of[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)holds across alln=525n=525test configurations on both benchmarks\. The empirical CDF ofR:=LHS/RHSR:=\\mathrm\{LHS\}/\\mathrm\{RHS\}stays at or belowR=1R=1on every configuration, supporting the use of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)as a hypothesis of[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1); full per\-benchmark statistics are in this appendix\.Each configuration is a tuple\(f,r,π,f^,r^\)\(f,r,\\pi,\\hat\{f\},\\hat\{r\}\), where\(f,r\)\(f,r\)is anmdpinstance,π\\piis anLπL\_\{\\pi\}\-Lipschitz evaluation policy, and\(f^,r^\)\(\\hat\{f\},\\hat\{r\}\)is the perturbed model pair\. The realized per\-step errorsεdyn,εrew\\varepsilon\_\{\\mathrm\{dyn\}\},\\varepsilon\_\{\\mathrm\{rew\}\}are computed from the\(f,f^\)\(f,\\hat\{f\}\)and\(r,r^\)\(r,\\hat\{r\}\)pair using the same definitions that the main\-text formulas use\.

The synthetic benchmark containsn=150n=150configurations in whichffandrrare globally Lipschitz; on this benchmarkLf,Lr,LπL\_\{f\},L\_\{r\},L\_\{\\pi\}inRHS\\mathrm\{RHS\}of[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1)are computed analytically as the global Lipschitz constants of the configuration’s maps\. The LQG benchmark containsn=375n=375configurations in which the quadratic reward is Lipschitz only on a bounded operating domain \(a fixed compact subset of state\-action space within which the configuration’s rollouts are confined\); on this benchmarkLrL\_\{r\}is the Lipschitz constant ofrrrestricted to that domain\.

The empirical CDFs ofRRin[Figure˜3](https://arxiv.org/html/2605.06732#A2.F3)have qualitatively different shapes on the two benchmarks\. The LQG ECDF climbs steadily acrossR∈\[10−3,1\]R\\in\[10^\{\-3\},1\]\. The synthetic ECDF rises sharply through its low\-RRmass, stays flat at value≈0\.66\\approx 0\.66acrossR∈\[10−2,5×10−1\]R\\in\[10^\{\-2\},\\,5\\times 10^\{\-1\}\], and then climbs towardR=1R=1on the remaining≈25%\\approx 25\\%of configurations\. The pooled medianR=0\.015R=0\.015is therefore dominated by the LQG arm\. The pool maximum isR=0\.9995R=0\.9995on a synthetic configuration, and the LQG arm peaks atR=0\.999R=0\.999\.

Two annotations on[Figure˜3](https://arxiv.org/html/2605.06732#A2.F3)should be read as follows:max R = 0\.999 \(LQG\)is the LQG\-arm maximum \(the pool maximum is the slightly larger synthetic valueR=0\.9995R=0\.9995\), and the legend labeltoydenotes the synthetic benchmark\.

We report only the per\-benchmark medians, the pooled median, and the per\-benchmark maxima; no bootstrap confidence intervals are used because the relevant claim is the universalR≤1R\\leq 1, which is verified directly on every configuration\.

### B\.5Experiment details: empirical evaluation of the optimal sample\-ratio formula

This appendix collects the per\-configuration construction, definitions, statistical methodology, and numerical results for the allocation\-evaluation experiment in[Section˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2), whose main\-text findings are summarized in[Figures˜2](https://arxiv.org/html/2605.06732#S4.F2)and[4](https://arxiv.org/html/2605.06732#A2.F4)\.

The configurations used in[Figures˜2](https://arxiv.org/html/2605.06732#S4.F2)and[4](https://arxiv.org/html/2605.06732#A2.F4)are partitioned into four groups, each generated by a Cartesian product over a small set of axes; per\-configuration values are recorded in the committed CSVs and we list the grids verbatim below\. The interpretation of the axessigma\_ratio,cost\_ratio,λ\\lambda, andtheta\_f0is deferred to the open item at the end of this appendix\.

#### Linear\-value configurations \(n=30n=30, panel \(a\) of[Figure˜2](https://arxiv.org/html/2605.06732#S4.F2)\)\.

The Cartesian product is\(Lf,λ\)∈\{\(0\.5,0\.5\),\(2\.0,2\.0\)\}\(L\_\{f\},\\lambda\)\\in\\\{\(0\.5,0\.5\),\\;\(2\.0,2\.0\)\\\}\(withLf=λL\_\{f\}=\\lambdaenforced\) crossed withsigma\_ratio∈\{0\.1,0\.3,1\.0,3\.0,10\.0\}\\texttt\{sigma\\\_ratio\}\\in\\\{0\.1,0\.3,1\.0,3\.0,10\.0\\\}andcost\_ratio∈\{0\.1,1\.0,10\.0\}\\texttt\{cost\\\_ratio\}\\in\\\{0\.1,1\.0,10\.0\\\}, for2×5×3=302\\times 5\\times 3=30configurations\. The reward Lipschitz constant is fixed atLr=1L\_\{r\}=1\.

#### tanh\\tanh\-value andsin\\sin\-value configurations \(n=9n=9each, panel \(a\) of[Figure˜2](https://arxiv.org/html/2605.06732#S4.F2)\)\.

For each of these two value\-function families, the Cartesian product issigma\_ratio∈\{0\.3,1\.0,3\.0\}\\texttt\{sigma\\\_ratio\}\\in\\\{0\.3,1\.0,3\.0\\\}crossed withcost\_ratio∈\{0\.1,1\.0,10\.0\}\\texttt\{cost\\\_ratio\}\\in\\\{0\.1,1\.0,10\.0\\\}, withLf=Lr=λ=1L\_\{f\}=L\_\{r\}=\\lambda=1fixed\.

#### Quadratic\-valuesup\\sup\-norm\-control configurations \(n=9n=9, panel \(b\) of[Figure˜2](https://arxiv.org/html/2605.06732#S4.F2)\)\.

The Cartesian product is the samesigma\_ratio×cost\_ratio\\texttt\{sigma\\\_ratio\}\\times\\texttt\{cost\\\_ratio\}grid of size99, withλ=1\\lambda=1andtheta\_f0=0\.5\\texttt\{theta\\\_f0\}=0\.5fixed\. Each configuration is evaluated twice: once withLflocal=1L\_\{f\}^\{\\mathrm\{local\}\}=1\(the realized\-sensitivity calculation, plotted as filled circles in panel \(b\)\) and once withLfsup=2L\_\{f\}^\{\\sup\}=2\(thesup\\sup\-norm control, plotted as open squares\)\.

#### LQG configurations \(n=30n=30,[Figure˜4](https://arxiv.org/html/2605.06732#A2.F4)\)\.

Configurations are indexed byseed∈\{0,1,…,29\}\\texttt\{seed\}\\in\\\{0,1,\\ldots,29\\\}and parameterized so that, atγ=0\.8\\gamma=0\.8, the contraction quantityγ​Lf​\(1\+Lπ\)\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)lies in\[0\.224,0\.228\]\[0\.224,0\.228\]\(margin1−γ​Lf​\(1\+Lπ\)∈\[0\.772,0\.776\]1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\\in\[0\.772,0\.776\]\)\. The realized constants per seed land inLf∈\[0\.280,0\.285\]L\_\{f\}\\in\[0\.280,0\.285\],Lπ∈\[1\.6×10−4,1\.0×10−3\]L\_\{\\pi\}\\in\[1\.6\\times 10^\{\-4\},1\.0\\times 10^\{\-3\}\],Lr∈\[0\.82,1\.49\]L\_\{r\}\\in\[0\.82,1\.49\]\.

For each seed we fit per\-configuration power\-law scaling lawsεdyn​\(N\)=Ad⋅N−αd\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\)=A\_\{d\}\\cdot N^\{\-\\alpha\_\{d\}\}andεrew​\(N\)=Ar⋅N−αr\\varepsilon\_\{\\mathrm\{rew\}\}\(N\)=A\_\{r\}\\cdot N^\{\-\\alpha\_\{r\}\}, withAd∈\[1\.0×10−3,1\.7×10−3\]A\_\{d\}\\in\[1\.0\\times 10^\{\-3\},1\.7\\times 10^\{\-3\}\],Ar∈\[2\.2×10−3,4\.0×10−3\]A\_\{r\}\\in\[2\.2\\times 10^\{\-3\},4\.0\\times 10^\{\-3\}\],αd∈\[0\.97,1\.05\]\\alpha\_\{d\}\\in\[0\.97,1\.05\],αr∈\[0\.99,1\.08\]\\alpha\_\{r\}\\in\[0\.99,1\.08\],Rd2∈\[0\.995,0\.9997\]R^\{2\}\_\{d\}\\in\[0\.995,0\.9997\], andRr2∈\[0\.996,0\.9998\]R^\{2\}\_\{r\}\\in\[0\.996,0\.9998\]\(recorded per\-configuration in theA\_d, A\_r, alpha\_d, alpha\_r, R2\_d, R2\_rcolumns oflqg\_per\_instance\.csv\)\. These per\-seed exponents are distinct from the globalα,β\\alpha,\\betafit in[Section˜4\.1](https://arxiv.org/html/2605.06732#S4.SS1)\.

Theboundary\_excludedcolumn reports that0/300/30configurations were filtered out as boundary cases \(the filter would exclude any configuration whose realized contractionγ​Lf​\(1\+Lπ\)\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)approached11, which would invalidate the hypothesis of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1); none did\)\.

We define the quantities used in this experiment\. LetVπ​\(s;f,r\)V^\{\\pi\}\(s;f,r\)denote the discounted return of a fixed evaluation policyπ\\piwhen the dynamics map isffand the reward map isrr\. For each test configuration, letΔ​f:=f^−f\\Delta f:=\\hat\{f\}\-fandΔ​r:=r^−r\\Delta r:=\\hat\{r\}\-rdenote the realized model perturbations, and fix a step sizeh\>0h\>0\. Define the realized value sensitivities by the one\-sided finite\-difference quotients

Sf​\(s,a\)\\displaystyle S\_\{f\}\(s,a\):=\|Vπ​\(s;f\+h​Δ​f,r\)−Vπ​\(s;f,r\)\|h​‖Δ​f‖,\\displaystyle\\;:=\\;\\frac\{\|V^\{\\pi\}\(s;f\+h\\,\\Delta f,r\)\-V^\{\\pi\}\(s;f,r\)\|\}\{h\\,\\\|\\Delta f\\\|\},Sr​\(s,a\)\\displaystyle S\_\{r\}\(s,a\):=\|Vπ​\(s;f,r\+h​Δ​r\)−Vπ​\(s;f,r\)\|h​\|Δ​r\|,\\displaystyle\\;:=\\;\\frac\{\|V^\{\\pi\}\(s;f,r\+h\\,\\Delta r\)\-V^\{\\pi\}\(s;f,r\)\|\}\{h\\,\|\\Delta r\|\},where∥⋅∥\\\|\\cdot\\\|is the Euclidean norm; ash→0h\\to 0,SfS\_\{f\}andSrS\_\{r\}approach the directional derivatives∂fVπ\\partial\_\{f\}V^\{\\pi\}and∂rVπ\\partial\_\{r\}V^\{\\pi\}, which provide the underlying intuition for these realized sensitivities\. The global Lipschitz constant of the dynamics map in the spectral norm is any

Lf≥sup\(s,a\)≠\(s′,a′\)‖f​\(s,a\)−f​\(s′,a′\)‖‖s−s′‖\+‖a−a′‖,L\_\{f\}\\;\\geq\\;\\sup\_\{\(s,a\)\\neq\(s^\{\\prime\},a^\{\\prime\}\)\}\\;\\frac\{\\\|f\(s,a\)\-f\(s^\{\\prime\},a^\{\\prime\}\)\\\|\}\{\\\|s\-s^\{\\prime\}\\\|\+\\\|a\-a^\{\\prime\}\\\|\},andLrL\_\{r\}is defined analogously, on a bounded operating domain when the reward is only locally Lipschitz\. LetKLip:=γ​Lr​\(1\+Lπ\)\(1−γ\)​\(1−γ​Lf​\(1\+Lπ\)\)K\_\{\\mathrm\{Lip\}\}:=\\frac\{\\gamma L\_\{r\}\(1\+L\_\{\\pi\}\)\}\{\(1\-\\gamma\)\(1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\)\}denote the dynamics coefficient appearing in[Equation˜1](https://arxiv.org/html/2605.06732#S3.E1), instantiated with the analytical global Lipschitz constantsLf,LrL\_\{f\},L\_\{r\}, and letK′K^\{\\prime\}denote the same coefficient withLfL\_\{f\}in the contraction factor1−γ​Lf​\(1\+Lπ\)1\-\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)replaced by the realized value\-function sensitivitySfS\_\{f\}\. The log\-ratio residualℓ\\ellis defined in[Section˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2)\.

![Refer to caption](https://arxiv.org/html/2605.06732v1/x4.png)Figure 4:The multiplier in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)instantiated with the global Lipschitz constants of[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)overshoots realized ratios by about three orders of magnitude on LQG\. Each ofn=30n=30LQG configurations atγ=0\.8\\gamma=0\.8lies above the dashed horizontal lineℓ=log⁡3\\ell=\\log 3\(predicted and realized ratios within a factor of33\); the solid horizontal line marks the medianℓ=7\.585\\ell=7\.585\.We summarize each group of configurations by the across\-configuration median of\|ℓ\|\|\\ell\|\(orℓ\\ellwhen overshoot direction is informative\);[Figure˜4](https://arxiv.org/html/2605.06732#A2.F4)jitters the vertical position of each LQG marker for visibility\. To check that overshoot in the global\-Lipschitz multiplier comparison is not consistent with chance, we run a one\-sided sign test against the null that overprediction and underprediction are equally likely; with30/3030/30configurations overshooting, the test givesp=0\.530≈9\.31×10−10p=0\.5^\{30\}\\approx 9\.31\\times 10^\{\-10\}\.

The headline per\-group medians are reported in[Section˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2); here we record the additional numbers underlying thesup\\sup\-norm\-control comparison\. On the99quadratic\-value configurations, the median residual rises from\|ℓrealized\|=0\.074\|\\ell\_\{\\mathrm\{realized\}\}\|=0\.074to\|ℓsup\|=0\.684\|\\ell\_\{\\sup\}\|=0\.684when realized sensitivities are replaced by theirsup\\sup\-norm overestimates, a factor of≈9\.25×\\approx 9\.25\\times\. The maximum\|ℓ\|\|\\ell\|on the linear\-value configurations is also0, complementing the median value of0already reported in main text\.

A separate statistic isolates the dynamics\-coefficient slack itself\. On the samen=30n=30LQG configurations \(committed per\-configuration as theK\_primecolumn oflqg\_per\_instance\.csv\), the multiplier ratioKLip/K′K\_\{\\mathrm\{Lip\}\}/K^\{\\prime\}has median2\.372\.37and maximum3\.233\.23\(minimum1\.711\.71\)\. This is a different quantity from the residualℓ\\ell: the factor≈1968\\approx 1968figure isexp⁡\(7\.585\)\\exp\(7\.585\)on the residual itself, whileKLip/K′K\_\{\\mathrm\{Lip\}\}/K^\{\\prime\}isolates the contribution of replacing the globalLfL\_\{f\}with the realizedSfS\_\{f\}in the dynamics\-side coefficient alone\.

Worst\-case bounds in approximate dynamic programming, such as the simulation\-lemma bounds on policy\-evaluation and policy\-improvement error stated in terms of model error, are classically conservative when instantiated with global Lipschitz or sup\-norm constants\[Munos,[2003](https://arxiv.org/html/2605.06732#bib.bib13), Kakade and Langford,[2002](https://arxiv.org/html/2605.06732#bib.bib14)\]\. The pattern observed here matches that classical observation: the bound in[Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)certifies a sufficient condition, while the proportionality in[Equation˜4](https://arxiv.org/html/2605.06732#S4.E4)carries the predictive content for sample allocation\.

### B\.6Proof of[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2)

See[2](https://arxiv.org/html/2605.06732#Thmtheorem2)

###### Proof\.

A trajectoryτ=\(\(s0,a0\),…,\(sH−1,aH−1\)\)\\tau=\(\(s\_\{0\},a\_\{0\}\),\\ldots,\(s\_\{H\-1\},a\_\{H\-1\}\)\)is sampled by executingπθ\\pi\_\{\\theta\}inℳ\\mathcal\{M\}\. LetGtG\_\{t\},G^t\\hat\{G\}\_\{t\}, andg^\(1\)\\hat\{g\}^\{\(1\)\}be as defined in[Section˜5](https://arxiv.org/html/2605.06732#S5)\. Define the correspondingreinforceestimator when the rewards are noise\-free asg\(1\):=∑t=0H−1∇θlog⁡πθ​\(at∣st\)​Gtg^\{\(1\)\}:=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,G\_\{t\}, so𝔼τ​\[g\(1\)\]=gH\\mathbb\{E\}\_\{\\tau\}\[g^\{\(1\)\}\]=g\_\{H\}\.

LetNt:=∑t′=tH−1γt′−t​ηt′N\_\{t\}:=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\eta\_\{t^\{\\prime\}\}, so thatG^t=Gt\+Nt\\hat\{G\}\_\{t\}=G\_\{t\}\+N\_\{t\}\. Letδ\(1\):=∑t=0H−1∇θlog⁡πθ​\(at∣st\)​Nt\\delta^\{\(1\)\}:=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,N\_\{t\}, so thatg^\(1\)=g\(1\)\+δ\(1\)\\hat\{g\}^\{\(1\)\}=g^\{\(1\)\}\+\\delta^\{\(1\)\}\. Fix the sampled trajectoryτ\\tauand take conditional expectation over the reward noise\. Because eachηt′\\eta\_\{t^\{\\prime\}\}is independent of the state\-action history and has mean zero,

𝔼​\[Nt∣τ\]\\displaystyle\\mathbb\{E\}\[N\_\{t\}\\mid\\tau\]=∑t′=tH−1γt′−t​𝔼​\[ηt′∣τ\]\\displaystyle=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\,\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\\mid\\tau\]=∑t′=tH−1γt′−t​𝔼​\[ηt′\]\\displaystyle=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\,\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\]=0\.\\displaystyle=0\.Therefore

𝔼​\[δ\(1\)∣τ\]\\displaystyle\\mathbb\{E\}\[\\delta^\{\(1\)\}\\mid\\tau\]=∑t=0H−1∇θlog⁡πθ​\(at∣st\)​𝔼​\[Nt∣τ\]\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,\\mathbb\{E\}\[N\_\{t\}\\mid\\tau\]\(10\)=0,\\displaystyle=0,and hence

𝔼​\[g^\(1\)∣τ\]=g\(1\)\.\\mathbb\{E\}\[\\hat\{g\}^\{\(1\)\}\\mid\\tau\]=g^\{\(1\)\}\.Therefore𝔼​\[g^\(1\)\]=𝔼​\[g\(1\)\]=gH\\mathbb\{E\}\[\\hat\{g\}^\{\(1\)\}\]=\\mathbb\{E\}\[g^\{\(1\)\}\]=g\_\{H\}, and averaging overKKtrajectories gives𝔼​\[g^\]=gH\\mathbb\{E\}\[\\hat\{g\}\]=g\_\{H\}\.

Fix the sampled trajectoryτ\\tauand define

wt:=∇θlog⁡πθ​\(at∣st\)\.w\_\{t\}:=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\.Then

δ\(1\)\\displaystyle\\delta^\{\(1\)\}=∑t=0H−1wt​∑t′=tH−1γt′−t​ηt′\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}w\_\{t\}\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\eta\_\{t^\{\\prime\}\}=∑t=0H−1∑t′=tH−1γt′−t​wt​ηt′\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}w\_\{t\}\\eta\_\{t^\{\\prime\}\}=∑t′=0H−1∑t=0t≤t′H−1γt′−t​wt​ηt′\\displaystyle=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{\\begin\{subarray\}\{c\}t=0\\\\ t\\leq t^\{\\prime\}\\end\{subarray\}\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}w\_\{t\}\\eta\_\{t^\{\\prime\}\}=∑t′=0H−1∑t=0t′γt′−t​wt​ηt′\.\\displaystyle=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{t=0\}^\{t^\{\\prime\}\}\\gamma^\{t^\{\\prime\}\-t\}w\_\{t\}\\eta\_\{t^\{\\prime\}\}\.Define

vt′:=∑t=0t′γt′−t​wt\.v\_\{t^\{\\prime\}\}:=\\sum\_\{t=0\}^\{t^\{\\prime\}\}\\gamma^\{t^\{\\prime\}\-t\}w\_\{t\}\.Then

δ\(1\)=∑t′=0H−1vt′​ηt′\.\\delta^\{\(1\)\}=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}v\_\{t^\{\\prime\}\}\\eta\_\{t^\{\\prime\}\}\.Using[Equation˜10](https://arxiv.org/html/2605.06732#A2.E10),

Var​\[δ\(1\)∣τ\]\\displaystyle\\mathrm\{Var\}\[\\delta^\{\(1\)\}\\mid\\tau\]=∑i=1dVar​\[δi\(1\)∣τ\]\\displaystyle=\\sum\_\{i=1\}^\{d\}\\mathrm\{Var\}\[\\delta^\{\(1\)\}\_\{i\}\\mid\\tau\]=∑i=1d𝔼\[\(δi\(1\)−𝔼\[δi\(1\)∣τ\]\)2\|τ\]\\displaystyle=\\sum\_\{i=1\}^\{d\}\\mathbb\{E\}\\\!\\left\[\\left\(\\delta^\{\(1\)\}\_\{i\}\-\\mathbb\{E\}\[\\delta^\{\(1\)\}\_\{i\}\\mid\\tau\]\\right\)^\{2\}\\middle\|\\tau\\right\]=𝔼\[∥δ\(1\)−𝔼\[δ\(1\)∣τ\]∥2∣τ\]\\displaystyle=\\mathbb\{E\}\\bigl\[\\\|\\delta^\{\(1\)\}\-\\mathbb\{E\}\[\\delta^\{\(1\)\}\\mid\\tau\]\\\|^\{2\}\\mid\\tau\\bigr\]=𝔼​\[‖δ\(1\)‖2∣τ\]\\displaystyle=\\mathbb\{E\}\\bigl\[\\\|\\delta^\{\(1\)\}\\\|^\{2\}\\mid\\tau\\bigr\]=𝔼\[∥∑t′=0H−1vt′ηt′∥2\|τ\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\left\\\|\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}v\_\{t^\{\\prime\}\}\\eta\_\{t^\{\\prime\}\}\\right\\\|^\{2\}\\middle\|\\tau\\right\]=𝔼\[⟨∑t′=0H−1vt′ηt′,∑u=0H−1vuηu⟩\|τ\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\left\\langle\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}v\_\{t^\{\\prime\}\}\\eta\_\{t^\{\\prime\}\},\\sum\_\{u=0\}^\{H\-1\}v\_\{u\}\\eta\_\{u\}\\right\\rangle\\middle\|\\tau\\right\]=𝔼\[∑t′=0H−1∑u=0H−1⟨vt′ηt′,vuηu⟩\|τ\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{u=0\}^\{H\-1\}\\left\\langle v\_\{t^\{\\prime\}\}\\eta\_\{t^\{\\prime\}\},v\_\{u\}\\eta\_\{u\}\\right\\rangle\\middle\|\\tau\\right\]=𝔼\[∑t′=0H−1∑u=0H−1ηt′ηu⟨vt′,vu⟩\|τ\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{u=0\}^\{H\-1\}\\eta\_\{t^\{\\prime\}\}\\eta\_\{u\}\\langle v\_\{t^\{\\prime\}\},v\_\{u\}\\rangle\\middle\|\\tau\\right\]=∑t′=0H−1∑u=0H−1𝔼​\[ηt′​ηu∣τ\]​⟨vt′,vu⟩\\displaystyle=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{u=0\}^\{H\-1\}\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\\eta\_\{u\}\\mid\\tau\]\\,\\langle v\_\{t^\{\\prime\}\},v\_\{u\}\\rangle=∑t′=0H−1𝔼​\[ηt′2∣τ\]​⟨vt′,vt′⟩\+∑t′=0H−1∑u=0u≠t′H−1𝔼​\[ηt′​ηu∣τ\]​⟨vt′,vu⟩\\displaystyle=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}^\{2\}\\mid\\tau\]\\,\\langle v\_\{t^\{\\prime\}\},v\_\{t^\{\\prime\}\}\\rangle\+\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sum\_\{\\begin\{subarray\}\{c\}u=0\\\\ u\\neq t^\{\\prime\}\\end\{subarray\}\}^\{H\-1\}\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\\eta\_\{u\}\\mid\\tau\]\\,\\langle v\_\{t^\{\\prime\}\},v\_\{u\}\\rangleFort′≠ut^\{\\prime\}\\neq u, independence of the reward noise fromτ\\tau, together with independence across time and zero mean, gives𝔼​\[ηt′​ηu∣τ\]=𝔼​\[ηt′​ηu\]=𝔼​\[ηt′\]​𝔼​\[ηu\]=0\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\\eta\_\{u\}\\mid\\tau\]=\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\\eta\_\{u\}\]=\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}\]\\mathbb\{E\}\[\\eta\_\{u\}\]=0, while𝔼​\[ηt′2∣τ\]=𝔼​\[ηt′2\]=ση2\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}^\{2\}\\mid\\tau\]=\\mathbb\{E\}\[\\eta\_\{t^\{\\prime\}\}^\{2\}\]=\\sigma\_\{\\eta\}^\{2\}\. Therefore=∑t′=0H−1ση2​‖vt′‖2\\displaystyle=\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\sigma\_\{\\eta\}^\{2\}\\\|v\_\{t^\{\\prime\}\}\\\|^\{2\}=ση2​∑t′=0H−1‖vt′‖2\.\\displaystyle=\\sigma\_\{\\eta\}^\{2\}\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\\|v\_\{t^\{\\prime\}\}\\\|^\{2\}\.Next,

‖vt′‖\\displaystyle\\\|v\_\{t^\{\\prime\}\}\\\|=‖∑t=0t′γt′−t​wt‖\\displaystyle=\\left\\\|\\sum\_\{t=0\}^\{t^\{\\prime\}\}\\gamma^\{t^\{\\prime\}\-t\}w\_\{t\}\\right\\\|≤∑t=0t′γt′−t​‖wt‖\\displaystyle\\leq\\sum\_\{t=0\}^\{t^\{\\prime\}\}\\gamma^\{t^\{\\prime\}\-t\}\\\|w\_\{t\}\\\|≤\(∑t=0t′γt′−t\)​max0≤t≤H−1⁡‖wt‖\\displaystyle\\leq\\left\(\\sum\_\{t=0\}^\{t^\{\\prime\}\}\\gamma^\{t^\{\\prime\}\-t\}\\right\)\\max\_\{0\\leq t\\leq H\-1\}\\\|w\_\{t\}\\\|≤11−γ​max0≤t≤H−1⁡‖wt‖\.\\displaystyle\\leq\\frac\{1\}\{1\-\\gamma\}\\max\_\{0\\leq t\\leq H\-1\}\\\|w\_\{t\}\\\|\.Substituting the bound on‖vt′‖\\\|v\_\{t^\{\\prime\}\}\\\|into the conditional variance gives

Var​\[δ\(1\)∣τ\]\\displaystyle\\mathrm\{Var\}\[\\delta^\{\(1\)\}\\mid\\tau\]≤ση2​∑t′=0H−1\(11−γ​max0≤t≤H−1⁡‖wt‖\)2\\displaystyle\\leq\\sigma\_\{\\eta\}^\{2\}\\sum\_\{t^\{\\prime\}=0\}^\{H\-1\}\\left\(\\frac\{1\}\{1\-\\gamma\}\\max\_\{0\\leq t\\leq H\-1\}\\\|w\_\{t\}\\\|\\right\)^\{2\}=ση2​H\(1−γ\)2​max0≤t≤H−1⁡‖wt‖2\.\\displaystyle=\\frac\{\\sigma\_\{\\eta\}^\{2\}H\}\{\(1\-\\gamma\)^\{2\}\}\\max\_\{0\\leq t\\leq H\-1\}\\\|w\_\{t\}\\\|^\{2\}\.Taking expectations overτ\\tauand using the definitions ofwtw\_\{t\}andWH2W\_\{H\}^\{2\}yields

𝔼​\[Var​\[δ\(1\)\|τ\]\]\\displaystyle\\mathbb\{E\}\[\\mathrm\{Var\}\[\\delta^\{\(1\)\}\|\\tau\]\]≤ση2​H\(1−γ\)2​𝔼​\[max0≤t≤H−1⁡‖wt‖2\]\\displaystyle\\leq\\frac\{\\sigma\_\{\\eta\}^\{2\}H\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\\\!\\left\[\\max\_\{0\\leq t\\leq H\-1\}\\\|w\_\{t\}\\\|^\{2\}\\right\]=ση2​H\(1−γ\)2𝔼\[max0≤t≤H−1∥∇θlogπθ\(at∣st\)∥2\]\\displaystyle=\\frac\{\\sigma\_\{\\eta\}^\{2\}H\}\{\(1\-\\gamma\)^\{2\}\}\\mathbb\{E\}\\\!\\left\[\\max\_\{0\\leq t\\leq H\-1\}\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\\|^\{2\}\\right\]=ση2​H​WH2\(1−γ\)2\.\\displaystyle=\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\.Now apply the law of total variance to each coordinate and sum over coordinates\. Here the conditional variance is over the reward noise given a fixed trajectoryτ\\tau:

Var​\[g^\(1\)\]\\displaystyle\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\]=Var​\(𝔼​\[g^\(1\)∣τ\]\)\+𝔼​\[Var​\[g^\(1\)∣τ\]\]\\displaystyle=\\mathrm\{Var\}\\\!\\bigl\(\\mathbb\{E\}\[\\hat\{g\}^\{\(1\)\}\\mid\\tau\]\\bigr\)\+\\mathbb\{E\}\\bigl\[\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\\mid\\tau\]\\bigr\]=Var​\[g\(1\)\]\+𝔼​\[Var​\[g\(1\)\+δ\(1\)∣τ\]\]\\displaystyle=\\mathrm\{Var\}\[g^\{\(1\)\}\]\+\\mathbb\{E\}\\bigl\[\\mathrm\{Var\}\[g^\{\(1\)\}\+\\delta^\{\(1\)\}\\mid\\tau\]\\bigr\]Becauseg\(1\)g^\{\(1\)\}depends only onτ\\tau, conditioning onτ\\taumakesg\(1\)g^\{\(1\)\}deterministic\. HenceVar​\[g\(1\)\+δ\(1\)∣τ\]=Var​\[δ\(1\)∣τ\]\\mathrm\{Var\}\[g^\{\(1\)\}\+\\delta^\{\(1\)\}\\mid\\tau\]=\\mathrm\{Var\}\[\\delta^\{\(1\)\}\\mid\\tau\]almost surely, so=Var​\[g\(1\)\]\+𝔼​\[Var​\[δ\(1\)∣τ\]\]\\displaystyle=\\mathrm\{Var\}\[g^\{\(1\)\}\]\+\\mathbb\{E\}\[\\mathrm\{Var\}\[\\delta^\{\(1\)\}\\mid\\tau\]\]≤Var​\[g\(1\)\]\+ση2​H​WH2\(1−γ\)2\.\\displaystyle\\leq\\mathrm\{Var\}\[g^\{\(1\)\}\]\+\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\.\(11\)
Since the trajectories and their reward noises are sampled independently acrosskk, the estimatorsg^\(1\),…,g^\(K\)\\hat\{g\}^\{\(1\)\},\\ldots,\\hat\{g\}^\{\(K\)\}are independent\. Each estimator has meangHg\_\{H\}\. Writeg^i\\hat\{g\}\_\{i\}and\(gH\)i\(g\_\{H\}\)\_\{i\}for theiith coordinates ofg^\\hat\{g\}andgHg\_\{H\}\. Then

Var​\[g^\]\\displaystyle\\mathrm\{Var\}\[\\hat\{g\}\]=∑i=1dVar​\[g^i\]\\displaystyle=\\sum\_\{i=1\}^\{d\}\\mathrm\{Var\}\[\\hat\{g\}\_\{i\}\]=∑i=1d𝔼​\[\(g^i−𝔼​\[g^i\]\)2\]\\displaystyle=\\sum\_\{i=1\}^\{d\}\\mathbb\{E\}\\\!\\left\[\(\\hat\{g\}\_\{i\}\-\\mathbb\{E\}\[\\hat\{g\}\_\{i\}\]\)^\{2\}\\right\]=∑i=1d𝔼​\[\(g^i−\(gH\)i\)2\]\\displaystyle=\\sum\_\{i=1\}^\{d\}\\mathbb\{E\}\\\!\\left\[\(\\hat\{g\}\_\{i\}\-\(g\_\{H\}\)\_\{i\}\)^\{2\}\\right\]=𝔼​\[∑i=1d\(g^i−\(gH\)i\)2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{i=1\}^\{d\}\(\\hat\{g\}\_\{i\}\-\(g\_\{H\}\)\_\{i\}\)^\{2\}\\right\]=𝔼​\[‖g^−gH‖2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|\\hat\{g\}\-g\_\{H\}\\\|^\{2\}\\right\]=𝔼​\[‖1K​∑k=1Kg^\(k\)−gH‖2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\left\\\|\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\hat\{g\}^\{\(k\)\}\-g\_\{H\}\\right\\\|^\{2\}\\right\]=1K2​𝔼​\[‖∑k=1K\(g^\(k\)−gH\)‖2\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\mathbb\{E\}\\\!\\left\[\\left\\\|\\sum\_\{k=1\}^\{K\}\(\\hat\{g\}^\{\(k\)\}\-g\_\{H\}\)\\right\\\|^\{2\}\\right\]=1K2​𝔼​\[⟨∑k=1K\(g^\(k\)−gH\),∑ℓ=1K\(g^\(ℓ\)−gH\)⟩\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\mathbb\{E\}\\\!\\left\[\\left\\langle\\sum\_\{k=1\}^\{K\}\(\\hat\{g\}^\{\(k\)\}\-g\_\{H\}\),\\sum\_\{\\ell=1\}^\{K\}\(\\hat\{g\}^\{\(\\ell\)\}\-g\_\{H\}\)\\right\\rangle\\right\]=1K2​∑k=1K∑ℓ=1K𝔼​\[⟨g^\(k\)−gH,g^\(ℓ\)−gH⟩\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{\\ell=1\}^\{K\}\\mathbb\{E\}\\\!\\left\[\\left\\langle\\hat\{g\}^\{\(k\)\}\-g\_\{H\},\\hat\{g\}^\{\(\\ell\)\}\-g\_\{H\}\\right\\rangle\\right\]Fork≠ℓk\\neq\\ell, independence gives𝔼​\[⟨g^\(k\)−gH,g^\(ℓ\)−gH⟩\]=⟨𝔼​\[g^\(k\)−gH\],𝔼​\[g^\(ℓ\)−gH\]⟩=0\\mathbb\{E\}\[\\langle\\hat\{g\}^\{\(k\)\}\-g\_\{H\},\\hat\{g\}^\{\(\\ell\)\}\-g\_\{H\}\\rangle\]=\\langle\\mathbb\{E\}\[\\hat\{g\}^\{\(k\)\}\-g\_\{H\}\],\\mathbb\{E\}\[\\hat\{g\}^\{\(\\ell\)\}\-g\_\{H\}\]\\rangle=0\. Therefore only the terms withk=ℓk=\\ellremain, so=1K2​∑k=1K𝔼​\[⟨g^\(k\)−gH,g^\(k\)−gH⟩\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\!\\left\[\\left\\langle\\hat\{g\}^\{\(k\)\}\-g\_\{H\},\\hat\{g\}^\{\(k\)\}\-g\_\{H\}\\right\\rangle\\right\]=1K2​∑k=1K𝔼​\[∑i=1d\(g^i\(k\)−\(gH\)i\)2\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\!\\left\[\\sum\_\{i=1\}^\{d\}\(\\hat\{g\}^\{\(k\)\}\_\{i\}\-\(g\_\{H\}\)\_\{i\}\)^\{2\}\\right\]=1K2​∑k=1K∑i=1d𝔼​\[\(g^i\(k\)−\(gH\)i\)2\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{d\}\\mathbb\{E\}\\\!\\left\[\(\\hat\{g\}^\{\(k\)\}\_\{i\}\-\(g\_\{H\}\)\_\{i\}\)^\{2\}\\right\]Since each coordinate ofg^\(k\)\\hat\{g\}^\{\(k\)\}has mean the corresponding coordinate ofgHg\_\{H\},=1K2​∑k=1K∑i=1dVar​\[g^i\(k\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{d\}\\mathrm\{Var\}\[\\hat\{g\}^\{\(k\)\}\_\{i\}\]By the definition ofVar\\mathrm\{Var\}for vector\-valued estimators,=1K2​∑k=1KVar​\[g^\(k\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathrm\{Var\}\[\\hat\{g\}^\{\(k\)\}\]The estimatorsg^\(1\),…,g^\(K\)\\hat\{g\}^\{\(1\)\},\\ldots,\\hat\{g\}^\{\(K\)\}have the same distribution, so each term in the sum equalsVar​\[g^\(1\)\]\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\]:=1K2​∑k=1KVar​\[g^\(1\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\]=1K​Var​\[g^\(1\)\]\.\\displaystyle=\\frac\{1\}\{K\}\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\]\.\(12\)When the rewards are noise\-free, the averaged estimatorg^\\hat\{g\}equalsg^=1K​∑k=1Kg\(k\)\\hat\{g\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g^\{\(k\)\}\. Repeating the expansion above gives

Var​\[g^\]η≡0\\displaystyle\\mathrm\{Var\}\[\\hat\{g\}\]\_\{\\eta\\equiv 0\}=Var​\[1K​∑k=1Kg\(k\)\]\\displaystyle=\\mathrm\{Var\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}g^\{\(k\)\}\\right\]=1K2​∑k=1K∑ℓ=1K𝔼​\[⟨g\(k\)−gH,g\(ℓ\)−gH⟩\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{\\ell=1\}^\{K\}\\mathbb\{E\}\\\!\\left\[\\left\\langle g^\{\(k\)\}\-g\_\{H\},g^\{\(\\ell\)\}\-g\_\{H\}\\right\\rangle\\right\]Fork≠ℓk\\neq\\ell, independence gives𝔼​\[⟨g\(k\)−gH,g\(ℓ\)−gH⟩\]=⟨𝔼​\[g\(k\)−gH\],𝔼​\[g\(ℓ\)−gH\]⟩=0\\mathbb\{E\}\[\\langle g^\{\(k\)\}\-g\_\{H\},g^\{\(\\ell\)\}\-g\_\{H\}\\rangle\]=\\langle\\mathbb\{E\}\[g^\{\(k\)\}\-g\_\{H\}\],\\mathbb\{E\}\[g^\{\(\\ell\)\}\-g\_\{H\}\]\\rangle=0, so=1K2​∑k=1K𝔼​\[‖g\(k\)−gH‖2\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\!\\left\[\\\|g^\{\(k\)\}\-g\_\{H\}\\\|^\{2\}\\right\]=1K2​∑k=1KVar​\[g\(k\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathrm\{Var\}\[g^\{\(k\)\}\]=1K2​∑k=1KVar​\[g\(1\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathrm\{Var\}\[g^\{\(1\)\}\]=1K​Var​\[g\(1\)\]\.\\displaystyle=\\frac\{1\}\{K\}\\mathrm\{Var\}\[g^\{\(1\)\}\]\.\(13\)Combining[Equation˜12](https://arxiv.org/html/2605.06732#A2.E12)with[Equation˜11](https://arxiv.org/html/2605.06732#A2.E11)gives

Var​\[g^\]\\displaystyle\\mathrm\{Var\}\[\\hat\{g\}\]=1K​Var​\[g^\(1\)\]\\displaystyle=\\frac\{1\}\{K\}\\mathrm\{Var\}\[\\hat\{g\}^\{\(1\)\}\]≤1K​\(Var​\[g\(1\)\]\+ση2​H​WH2\(1−γ\)2\)\\displaystyle\\leq\\frac\{1\}\{K\}\\left\(\\mathrm\{Var\}\[g^\{\(1\)\}\]\+\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{\(1\-\\gamma\)^\{2\}\}\\right\)=1K​Var​\[g\(1\)\]\+ση2​H​WH2K​\(1−γ\)2\\displaystyle=\\frac\{1\}\{K\}\\mathrm\{Var\}\[g^\{\(1\)\}\]\+\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{K\(1\-\\gamma\)^\{2\}\}=Var​\[g^\]η≡0\+ση2​H​WH2K​\(1−γ\)2,\\displaystyle=\\mathrm\{Var\}\[\\hat\{g\}\]\_\{\\eta\\equiv 0\}\+\\frac\{\\sigma\_\{\\eta\}^\{2\}HW\_\{H\}^\{2\}\}\{K\(1\-\\gamma\)^\{2\}\},where the last equality uses[Equation˜13](https://arxiv.org/html/2605.06732#A2.E13)\. ∎

### B\.7Visualization of the three fidelity\-cost regimes from[Corollary˜2](https://arxiv.org/html/2605.06732#Thmcorollary2)

[Figure˜5](https://arxiv.org/html/2605.06732#A2.F5)plotsση2​\(c\)\\sigma\_\{\\eta\}^\{2\}\(c\)andΦ​\(c\)=c​ση2​\(c\)\\Phi\(c\)=c\\,\\sigma\_\{\\eta\}^\{2\}\(c\)for the three regimes analyzed in[Section˜5](https://arxiv.org/html/2605.06732#S5)\.

![Refer to caption](https://arxiv.org/html/2605.06732v1/x5.png)\(a\)Power\-law fidelity\.
![Refer to caption](https://arxiv.org/html/2605.06732v1/x6.png)\(b\)Bounded fidelity\.
![Refer to caption](https://arxiv.org/html/2605.06732v1/x7.png)\(c\)Irreducible noise floor\.

Figure 5:Three examples of how the noise varianceση2​\(c\)\\sigma\_\{\\eta\}^\{2\}\(c\)depends on the costcc, and the correspondingΦ​\(c\)=c​ση2​\(c\)\\Phi\(c\)=c\\,\\sigma\_\{\\eta\}^\{2\}\(c\)from[Corollary˜2](https://arxiv.org/html/2605.06732#Thmcorollary2)\.
### B\.8Proof of[Proposition˜2](https://arxiv.org/html/2605.06732#Thmproposition2)

See[2](https://arxiv.org/html/2605.06732#Thmproposition2)

###### Proof\.

For one trajectoryτ=\(\(s0,a0\),…,\(sH−1,aH−1\)\)\\tau=\(\(s\_\{0\},a\_\{0\}\),\\ldots,\(s\_\{H\-1\},a\_\{H\-1\}\)\)sampled by executingπθ\\pi\_\{\\theta\}under the true dynamicsff, define the biased finite\-horizon return from timettonward by

G~t:=∑t′=tH−1γt′−t​r~​\(st′,at′\)\.\\tilde\{G\}\_\{t\}:=\\sum\_\{t^\{\\prime\}=t\}^\{H\-1\}\\gamma^\{t^\{\\prime\}\-t\}\\tilde\{r\}\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\.The single\-trajectory estimator using biased rewards is

g~\(1\):=∑t=0H−1∇θlog⁡πθ​\(at∣st\)​G~t\.\\tilde\{g\}^\{\(1\)\}:=\\sum\_\{t=0\}^\{H\-1\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,\\tilde\{G\}\_\{t\}\.
The estimatorg~\(1\)\\tilde\{g\}^\{\(1\)\}is the finite\-horizonreinforceestimator for themdpℳ~\\tilde\{\\mathcal\{M\}\}, which has the true dynamicsffand the biased rewardr~\\tilde\{r\}\. Therefore the finite\-horizon policy\-gradient identity gives

𝔼​\[g~\(1\)\]=∇θJH​\(πθ,ℳ~\)\.\\mathbb\{E\}\[\\tilde\{g\}^\{\(1\)\}\]=\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\tilde\{\\mathcal\{M\}\}\)\.Averaging overKKindependent trajectories does not change the mean:

𝔼​\[g~\]\\displaystyle\\mathbb\{E\}\[\\tilde\{g\}\]=𝔼​\[1K​∑k=1Kg~\(k\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\tilde\{g\}^\{\(k\)\}\\right\]=1K​∑k=1K𝔼​\[g~\(k\)\]\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\[\\tilde\{g\}^\{\(k\)\}\]=∇θJH​\(πθ,ℳ~\)\.\\displaystyle=\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\tilde\{\\mathcal\{M\}\}\)\.By the definitions ofr~\\tilde\{r\},JHJ\_\{H\}, andBHB\_\{H\},

JH​\(πθ,ℳ~\)\\displaystyle J\_\{H\}\(\\pi\_\{\\theta\},\\tilde\{\\mathcal\{M\}\}\)=𝔼​\[∑t=0H−1γt​r~​\(st,at\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\tilde\{r\}\(s\_\{t\},a\_\{t\}\)\\right\]=𝔼​\[∑t=0H−1γt​\(r​\(st,at\)\+b​\(st,at\)\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\(r\(s\_\{t\},a\_\{t\}\)\+b\(s\_\{t\},a\_\{t\}\)\)\\right\]=𝔼​\[∑t=0H−1γt​r​\(st,at\)\]\+𝔼​\[∑t=0H−1γt​b​\(st,at\)\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\]\+\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}b\(s\_\{t\},a\_\{t\}\)\\right\]=JH​\(πθ,ℳ\)\+BH​\(θ\)\.\\displaystyle=J\_\{H\}\(\\pi\_\{\\theta\},\\mathcal\{M\}\)\+B\_\{H\}\(\\theta\)\.Taking gradients with respect toθ\\thetagives

∇θJH​\(πθ,ℳ~\)\\displaystyle\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\tilde\{\\mathcal\{M\}\}\)=∇θJH​\(πθ,ℳ\)\+∇θBH​\(θ\)\\displaystyle=\\nabla\_\{\\theta\}J\_\{H\}\(\\pi\_\{\\theta\},\\mathcal\{M\}\)\+\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)=gH\+∇θBH​\(θ\)\.\\displaystyle=g\_\{H\}\+\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\.Combining the preceding displays proves[Equation˜6](https://arxiv.org/html/2605.06732#S5.E6)\.

Let

μb:=𝔼​\[g~\]−gH\.\\mu\_\{b\}:=\\mathbb\{E\}\[\\tilde\{g\}\]\-g\_\{H\}\.By[Equation˜6](https://arxiv.org/html/2605.06732#S5.E6),μb=∇θBH​\(θ\)\\mu\_\{b\}=\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\. We use the identity‖x\+y‖2=‖x‖2\+2​⟨x,y⟩\+‖y‖2\\\|x\+y\\\|^\{2\}=\\\|x\\\|^\{2\}\+2\\langle x,y\\rangle\+\\\|y\\\|^\{2\}withx=g~−𝔼​\[g~\]x=\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\]andy=μby=\\mu\_\{b\}\. We also use linearity of expectation and the fact thatμb\\mu\_\{b\}is a fixed vector onceθ\\thetais fixed, so𝔼​\[‖μb‖2\]=‖μb‖2\\mathbb\{E\}\[\\\|\\mu\_\{b\}\\\|^\{2\}\]=\\\|\\mu\_\{b\}\\\|^\{2\}\. Then

𝔼​\[‖g~−gH‖2\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-g\_\{H\}\\\|^\{2\}\\right\]=𝔼​\[‖g~−𝔼​\[g~\]\+𝔼​\[g~\]−gH‖2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\]\+\\mathbb\{E\}\[\\tilde\{g\}\]\-g\_\{H\}\\\|^\{2\}\\right\]=𝔼​\[‖g~−𝔼​\[g~\]\+μb‖2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\]\+\\mu\_\{b\}\\\|^\{2\}\\right\]=𝔼​\[‖g~−𝔼​\[g~\]‖2\+2​⟨g~−𝔼​\[g~\],μb⟩\+‖μb‖2\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\]\\\|^\{2\}\+2\\left\\langle\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\],\\mu\_\{b\}\\right\\rangle\+\\\|\\mu\_\{b\}\\\|^\{2\}\\right\]=𝔼​\[‖g~−𝔼​\[g~\]‖2\]\+2​𝔼​\[⟨g~−𝔼​\[g~\],μb⟩\]\+‖μb‖2\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\\|\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\]\\\|^\{2\}\\right\]\+2\\,\\mathbb\{E\}\\\!\\left\[\\left\\langle\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\],\\mu\_\{b\}\\right\\rangle\\right\]\+\\\|\\mu\_\{b\}\\\|^\{2\}=Var​\[g~\]\+‖μb‖2\\displaystyle=\\mathrm\{Var\}\[\\tilde\{g\}\]\+\\\|\\mu\_\{b\}\\\|^\{2\}because𝔼​\[⟨g~−𝔼​\[g~\],μb⟩\]=⟨𝔼​\[g~\]−𝔼​\[g~\],μb⟩=0\\mathbb\{E\}\[\\langle\\tilde\{g\}\-\\mathbb\{E\}\[\\tilde\{g\}\],\\mu\_\{b\}\\rangle\]=\\langle\\mathbb\{E\}\[\\tilde\{g\}\]\-\\mathbb\{E\}\[\\tilde\{g\}\],\\mu\_\{b\}\\rangle=0\. Since the estimatorsg~\(1\),…,g~\(K\)\\tilde\{g\}^\{\(1\)\},\\ldots,\\tilde\{g\}^\{\(K\)\}are independent and identically distributed,

Var​\[g~\]\\displaystyle\\mathrm\{Var\}\[\\tilde\{g\}\]=Var​\[1K​∑k=1Kg~\(k\)\]\\displaystyle=\\mathrm\{Var\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\tilde\{g\}^\{\(k\)\}\\right\]=1K2​∑k=1KVar​\[g~\(k\)\]\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathrm\{Var\}\[\\tilde\{g\}^\{\(k\)\}\]=1K​Var​\[g~\(1\)\]\.\\displaystyle=\\frac\{1\}\{K\}\\mathrm\{Var\}\[\\tilde\{g\}^\{\(1\)\}\]\.Substitutingμb=∇θBH​\(θ\)\\mu\_\{b\}=\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)gives[Equation˜7](https://arxiv.org/html/2605.06732#S5.E7)\. If∇θBH​\(θ\)≠0\\nabla\_\{\\theta\}B\_\{H\}\(\\theta\)\\neq 0, the second term in[Equation˜7](https://arxiv.org/html/2605.06732#S5.E7)is positive and does not depend onKK\. ∎

## Appendix CExperimental details

### C\.1Synthetic teacher environment

The environment is a synthetic continuous\-control environment whose dynamics and reward function are defined by frozen, randomly\-initialized neural networks\. Decoupling the environment from any particular physics simulator gives exact control over dynamics complexity, reward fidelity, and partial observability, which enables the controlled scaling experiment of[Section˜4\.1](https://arxiv.org/html/2605.06732#S4.SS1)\.

The teacher consists of two 2\-layer ReLU MLPs\. The dynamics teacherfdyn:ℝds\+da→ℝdsf\_\{\\mathrm\{dyn\}\}:\\mathbb\{R\}^\{d\_\{s\}\+d\_\{a\}\}\\to\\mathbb\{R\}^\{d\_\{s\}\}maps\(st,at\)\(s\_\{t\},a\_\{t\}\)to the next state viast\+1=tanh⁡\(fdyn​\(\[st,at\]\)\)s\_\{t\+1\}=\\tanh\(f\_\{\\mathrm\{dyn\}\}\(\[s\_\{t\},a\_\{t\}\]\)\)\. The reward teacherfrew:ℝds\+da→ℝf\_\{\\mathrm\{rew\}\}:\\mathbb\{R\}^\{d\_\{s\}\+d\_\{a\}\}\\to\\mathbb\{R\}produces the scalar rewardrt=frew​\(\[st,at\]\)/dhr\_\{t\}=f\_\{\\mathrm\{rew\}\}\(\[s\_\{t\},a\_\{t\}\]\)/\\sqrt\{d\_\{h\}\}, wheredhd\_\{h\}is the teacher hidden width\. All teacher weights are drawn from𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)using a fixed seed and never updated, so the same seed yields the identicalmdp\. Thetanh\\tanhactivation in dynamics constrains the state to\[−1,1\]ds\[\-1,1\]^\{d\_\{s\}\}, and the1/dh1/\\sqrt\{d\_\{h\}\}reward scaling ensures comparable reward magnitude across teacher widths\. Default dimensions:ds=12d\_\{s\}=12,da=4d\_\{a\}=4,dh=64d\_\{h\}=64, episode lengthT=500T=500\.

### C\.2Experiment details: power\-law scaling of dynamics and reward error

This appendix collects the protocol and engineering details for the error–sample\-size scaling experiment in[Section˜4\.1](https://arxiv.org/html/2605.06732#S4.SS1), whose main\-text result is summarized in[Figure˜1](https://arxiv.org/html/2605.06732#S4.F1)\.

We use the teacher of[Section˜C\.1](https://arxiv.org/html/2605.06732#A3.SS1), which keeps the experiment inside the regime where the assumptions of[Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)hold\. Training transitions are drawn i\.i\.d\. from the teacher’s induced state distribution by rolling out a smooth reference policy under the frozen teacher and subsampling\(s,a,s′,r\)\(s,a,s^\{\\prime\},r\)tuples; the released code records the exact rollout and subsampling procedure\.

Power\-law exponents are fit on the seven anchorsN∈\{2,000,5,000,10,000,20,000,50,000,100,000,200,000\}N\\in\\\{2\{,\}000,\\;5\{,\}000,\\;10\{,\}000,\\;20\{,\}000,\\;50\{,\}000,\\;100\{,\}000,\\;200\{,\}000\\\}, each with 100 independent seeds\. A singleN=500,000N=500\{,\}000run is preserved in the released dataset for future analyses but is excluded from both the figure and the fit, because only one seed was completed at that anchor\.

For each\(N,seed\)\(N,\\text\{seed\}\)we train two independent students that mirror the teacher architecture: a dynamics headfdynθ:ℝds\+da→ℝdsf\_\{\\mathrm\{dyn\}\}^\{\\theta\}:\\mathbb\{R\}^\{d\_\{s\}\+d\_\{a\}\}\\to\\mathbb\{R\}^\{d\_\{s\}\}and a reward headfrewϕ:ℝds\+da→ℝf\_\{\\mathrm\{rew\}\}^\{\\phi\}:\\mathbb\{R\}^\{d\_\{s\}\+d\_\{a\}\}\\to\\mathbb\{R\}, each implemented as a 2\-layer ReLU MLP with hidden widthdh=64d\_\{h\}=64matching the teacher \(no output activation, no shared trunk, no weight tying, no input/target normalization, default PyTorchnn\.Linearinitialization\)\. Both heads are optimized independently with Adam at learning rate10−310^\{\-3\}\(PyTorch defaults forβ1,β2,ϵ\\beta\_\{1\},\\beta\_\{2\},\\epsilon; no weight decay\) and a mini\-batch size of256256drawn from the sameNN\-sample training pool with reshuffling each epoch\. Training proceeds for a fixed schedule of200200epochs with no learning\-rate schedule and no early stopping; the held\-out validation MSE is recorded every epoch but is not used to gate the optimizer, and the reportedεdyn​\(Ndyn\)\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\)andεrew​\(Nrew\)\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\)are the final\-epoch validation MSEs of the two heads, computed undertorch\.no\_grad\. The training objective is the per\-head mean\-squared error,ℒdyn​\(θ\)=1\|ℬ\|​∑\(s,a,s′\)∈ℬ∥fdynθ​\(\[s,a\]\)−s′∥22\\mathcal\{L\}\_\{\\mathrm\{dyn\}\}\(\\theta\)=\\tfrac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{\(s,a,s^\{\\prime\}\)\\in\\mathcal\{B\}\}\\lVert f\_\{\\mathrm\{dyn\}\}^\{\\theta\}\(\[s,a\]\)\-s^\{\\prime\}\\rVert\_\{2\}^\{2\}andℒrew​\(ϕ\)=1\|ℬ\|​∑\(s,a,r\)∈ℬ\(frewϕ​\(\[s,a\]\)−r\)2\\mathcal\{L\}\_\{\\mathrm\{rew\}\}\(\\phi\)=\\tfrac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{\(s,a,r\)\\in\\mathcal\{B\}\}\(f\_\{\\mathrm\{rew\}\}^\{\\phi\}\(\[s,a\]\)\-r\)^\{2\}, evaluated on each mini\-batchℬ\\mathcal\{B\}\. At200200training epochs, and for every anchorN≥2,000N\\geq 2\{,\}000, the training MSE is fully saturated, so the final\-epoch values used by the fit coincide with the converged held\-out loss to within seed noise\.

Each anchor evaluates against a fixed held\-out set of5,0005\{,\}000transitions, drawn independently of the training pool\. Holding the validation set size constant acrossNNkeeps the evaluation noise floor independent of the training pool size, so anyNN\-dependence of the measured MSE reflects student generalization rather than a changing estimator variance\.

Per anchor, we report the across\-seed mean MSE in the figure with±1\\pm 1standard error of the mean \(SEM\) error bars; this across\-seed SEM is distinct from the bootstrap standard errors reported below for the fitted parameters\. The power lawsεdyn​\(Ndyn\)=Ad⋅Ndyn−α\\varepsilon\_\{\\mathrm\{dyn\}\}\(N\_\{\\mathrm\{dyn\}\}\)=A\_\{d\}\\cdot N\_\{\\mathrm\{dyn\}\}^\{\-\\alpha\}andεrew​\(Nrew\)=Ar⋅Nrew−β\\varepsilon\_\{\\mathrm\{rew\}\}\(N\_\{\\mathrm\{rew\}\}\)=A\_\{r\}\\cdot N\_\{\\mathrm\{rew\}\}^\{\-\\beta\}are fit by ordinary log–log linear regression on the per\-anchor means, whereα\\alphais the dynamics exponent andβ\\betais the reward exponent reported in[Section˜4\.1](https://arxiv.org/html/2605.06732#S4.SS1)\. Uncertainty on\(Ad,Ar,α,β\)\(A\_\{d\},A\_\{r\},\\alpha,\\beta\)is obtained by a stratified bootstrap with1,0001\{,\}000resamples that draws seeds with replacement*within*each anchor before refitting; we report the bootstrap standard error and the2\.52\.5/97\.597\.5percentile interval as the 95% confidence interval\. The resulting bootstrap standard errors areAd±0\.04A\_\{d\}\\pm 0\.04,α±0\.01\\alpha\\pm 0\.01,Ar±13\.3A\_\{r\}\\pm 13\.3, andβ±0\.02\\beta\\pm 0\.02, and the 95% bootstrap confidence intervals areAd∈\[0\.28,0\.42\]A\_\{d\}\\in\[0\.28,\\,0\.42\],α∈\[0\.09,0\.13\]\\alpha\\in\[0\.09,\\,0\.13\],Ar∈\[68\.1,121\]A\_\{r\}\\in\[68\.1,\\,121\], andβ∈\[0\.93,0\.99\]\\beta\\in\[0\.93,\\,0\.99\]\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: The abstract and the “Our contribution” list in the introduction state three contributions—error attribution \([Lemma˜1](https://arxiv.org/html/2605.06732#Thmlemma1)\), budget allocation \([Theorem˜1](https://arxiv.org/html/2605.06732#Thmtheorem1)\), and reward fidelity \([Theorems˜2](https://arxiv.org/html/2605.06732#Thmtheorem2),[2](https://arxiv.org/html/2605.06732#Thmcorollary2)and[2](https://arxiv.org/html/2605.06732#Thmproposition2)\)—each of which is formally proved in the appendix and empirically calibrated in[Sections˜3\.1](https://arxiv.org/html/2605.06732#S3.SS1),[4\.1](https://arxiv.org/html/2605.06732#S4.SS1)and[4\.2](https://arxiv.org/html/2605.06732#S4.SS2)\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: Limitations are discussed inline rather than in a dedicated section: the two caveats following[Proposition˜1](https://arxiv.org/html/2605.06732#Thmproposition1)\(a representation that lowers Lipschitz constants may raiseεdyn\\varepsilon\_\{\\mathrm\{dyn\}\}, and the bound assumesγ​Lf​\(1\+Lπ\)<1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)<1\); the deterministic\-dynamics restriction noted in[Section˜3\.1](https://arxiv.org/html/2605.06732#S3.SS1); and the order\-of\-magnitude looseness of the global\-Lipschitz multiplier shown in[Section˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: Each lemma, theorem, corollary, and proposition states its assumptions explicitly—Lipschitz conditions onf,r,πf,r,\\piand the contractionγ​Lf​\(1\+Lπ\)<1\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)<1for[Lemmas˜1](https://arxiv.org/html/2605.06732#Thmlemma1)and[1](https://arxiv.org/html/2605.06732#Thmtheorem1); the moment conditionWH2<∞W\_\{H\}^\{2\}<\\inftytogether with zero\-mean i\.i\.d\. reward noise for[Theorem˜2](https://arxiv.org/html/2605.06732#Thmtheorem2); the analogous setup for[Proposition˜2](https://arxiv.org/html/2605.06732#Thmproposition2); and anε\\varepsilon\-Lipschitz latent velocity map for[Proposition˜1](https://arxiv.org/html/2605.06732#Thmproposition1)—and full proofs are given in the appendix\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification:[Sections˜C\.1](https://arxiv.org/html/2605.06732#A3.SS1),[C\.2](https://arxiv.org/html/2605.06732#A3.SS2),[B\.4](https://arxiv.org/html/2605.06732#A2.SS4)and[B\.5](https://arxiv.org/html/2605.06732#A2.SS5)document the synthetic teacher MLPs \(architecture, dimensions, frozen\-seed initialization\), the training protocol \(optimizer, learning rate, batch size, epochs, seeds per anchor, evaluation set size\), the configuration grids and contraction regimes for the LQG and synthetic benchmarks, and the per\-configuration definitions used in each experiment\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[No\]
24. Justification: Code and data are not released with this submission; the experiments are described in sufficient detail in the appendix to allow independent reimplementation, and we plan to release code and the per\-configuration data upon publication\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification:[Section˜C\.2](https://arxiv.org/html/2605.06732#A3.SS2)specifies the seven training\-set anchors,100100seeds per anchor, the held\-out validation set of5,0005\{,\}000transitions, and the optimizer \(Adam, learning rate10−310^\{\-3\}, batch size256256,200200epochs, no learning\-rate schedule, no early stopping\);[Sections˜B\.4](https://arxiv.org/html/2605.06732#A2.SS4)and[B\.5](https://arxiv.org/html/2605.06732#A2.SS5)specify the configuration grids, the contraction regimeγ​Lf​\(1\+Lπ\)∈\[0\.224,0\.228\]\\gamma L\_\{f\}\(1\+L\_\{\\pi\}\)\\in\[0\.224,0\.228\]for the LQG arm, and the per\-configuration value\-sensitivity construction\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification:[Sections˜4\.1](https://arxiv.org/html/2605.06732#S4.SS1)and[C\.2](https://arxiv.org/html/2605.06732#A3.SS2)report±1\\pm 1across\-seed SEM error bars on per\-anchor validation MSEs and a1,0001\{,\}000\-resample stratified bootstrap \(resampling seeds within each anchor\) for standard errors and95%95\\%confidence intervals on the fitted power\-law parameters;[Sections˜4\.2](https://arxiv.org/html/2605.06732#S4.SS2)and[B\.5](https://arxiv.org/html/2605.06732#A2.SS5)report a one\-sided sign test \(p≈9\.31×10−10p\\approx 9\.31\\times 10^\{\-10\}for the LQG overshoot\) and Spearman rank correlation alongside per\-group medians\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: All experiments train small 2\-layer MLPs \([Section˜C\.1](https://arxiv.org/html/2605.06732#A3.SS1)\) or evaluate closed\-form expressions on LQG and synthetic configurations, so no GPU is required; total compute across the reported experiments is approximately10,00010\{,\}000CPU\-hours on commodity x86 hardware\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: The work consists of theoretical analysis of model\-based reinforcement learning together with synthetic numerical experiments on internally defined MLP teachers and LQG configurations; it involves no human subjects, no scraped or third\-party datasets, and no deployed system, and we have reviewed and conform to the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[N/A\]
49. Justification: The paper is foundational research on the sample complexity and gradient bias of model\-based reinforcement learning under learned reward models, with no direct path to societal impact\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: The paper releases no pretrained models, generators, or scraped datasets, so no misuse safeguards are required\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[N/A\]
59. Justification: The experiments do not reuse any third\-party datasets or pretrained models; they train small MLPs on a synthetic environment defined inside the paper and evaluate closed\-form expressions on LQG configurations\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://arxiv.org/html/2605.06732v1/paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: No new assets are released with this submission\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: The work does not involve crowdsourcing or research with human subjects\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: The work does not involve research with human subjects, so no IRB approval is required\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: LLMs were used only for writing, editing, and formatting; they were not part of the methodology, theorems, or experiments in this paper\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.