ArcFlow: Training Flow Policies for Multiple Solver Budgets

# ArcFlow: Training Flow Policies for Multiple Solver Budgets
Source: [https://arxiv.org/html/2610.28788](https://arxiv.org/html/2610.28788)
Hong YuAffiliation:University of Massachusetts Amherst University of Massachusetts Lowell

###### Abstract

Flow\-matching policies adjust inference cost through the number of solver steps, but reinforcement learning at a single fixed solver budget can hide degradation at cheaper deployment budgets and even erase useful one\-step behavior present in a pretrained policy\. We introduceAction–Ratio–Clip Flow \(ArcFlow\), a deployment\-aware algorithm for training one flow policy across a specified range of solver budgets\.ArcFlowcombines action\-aligned supervision, budget exposure, and worst\-budget checkpoint selection\. For reconstruction objectives, its bridge aggregates temporal probes into one surrogate ratio and one PPO\-style clip per action, with a boundary residual exactly equal to deployed one\-step reconstruction error\. Cycling exposes the shared field to the deployment range, and selection optimizes its performance floor\. For native velocity objectives, multi\-budget endpoint supervision applies the same principle without replacing the native update\. Both realizations preserve the architecture and deployment\-time cost\. Across DM Control, manipulation, and locomotion,ArcFlowattains the highest mean one\-step return among five flow\-policy baselines on 25 of 30 tasks under each method’s reported training settings\. Component studies identify complementary improvements from boundary supervision, the fixed\-grid bridge, budget exposure, and deployment selection\. These results establish deployment\-budget\-aware training as an effective route to strong low\-step control with one policy serving multiple inference costs\.111Code can be accessed at:[https://anonymous\.4open\.science/r/arcflow\-868A](https://anonymous.4open.science/r/arcflow-868A)

## 1Introduction

Flow matching represents expressive action distributions by learning a velocity field that transports noise to actions\([Lipman et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib1);[Liu et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib2)\)\. Its solver controls inference cost: more steps refine integration, while fewer steps reduce network evaluations, helping robotic policies meet latency budgets\. Consistency and shortcut models enable one\- and few\-step generation\([Song et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib12);[Frans et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib13)\), creating an opportunity to retain expressive policies at low deployment cost\.

Reinforcement learning \(RL\) improves policies through interaction\. FPO and FPO\+\+ derive policy\-gradient surrogates from flow\-matching losses\([McAllister et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib4);[Yi et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib9)\); Diffusion Policy Policy Optimization \(DPPO\) applies policy gradients to a stochastic diffusion denoising chain\([Ren et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib10)\); ReinFlow adds learnable exploration noise to flow policies\([Zhang et al\., 2025b](https://arxiv.org/html/2610.28788#bib.bib5)\); and PolicyFlow approximates importance ratios from velocity\-field changes along interpolation paths\([Yang et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib8)\)\. At deployment, solver steps determine latency and may change the action of the same trained field\. High return at the training budget gives no assurance of high one\-step return; RL can even erode low\-step behavior already present in the prior\. This gap motivates us to train and select one field across its deployment range \(Figure[1](https://arxiv.org/html/2610.28788#S1.F1)\)\.

![Refer to caption](https://arxiv.org/html/2610.28788v1/fig1_schematic.png)Figure 1:ArcFlowfor multi\-budget deployment\.Left: the solver budgetKKcontrols deployment cost\. Right: the bridge\-based control realization uses fresh noise and the observed action; at the boundary, the residual norm equals one\-step reconstruction error\. Probes are averaged before one surrogate ratio and PPO clip per action\. Rollout budgets cycle, and checkpoint selection maximizes the worst validation return overK∈\{1,4,16\}K\\in\\\{1,4,16\\\}\.Two properties of the training objective and solver discretization help explain this behavior\. First, reconstruction\-based objectives may provide limited supervision at the deployment boundary\. In FPO\([McAllister et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib4)\), for example, the noise\-reconstruction formulation weights velocity error by\(1−t\)2\(1\-t\)^\{2\}, so the boundary queried by a one\-step solver receives zero weight\. Direct velocity supervision instead provides a boundary\-aligned signal\. This observation complements prior work on time\-dependent loss weighting in generative modeling\([Choi et al\., 2022](https://arxiv.org/html/2610.28788#bib.bib22);[Hang et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib23)\)and highlights the importance of aligning the weighting scheme with the deployed action map\. Second, because different solver budgets induce different action maps, updates based on rollouts from a single budget need not improve all budgets uniformly\. We study this effect in the ReinFlow\([Zhang et al\., 2025b](https://arxiv.org/html/2610.28788#bib.bib5)\)fine\-tuning pipeline, where native velocity supervision is already available\.

We introduceAction–Ratio–Clip Flow \(ArcFlow\), an RL algorithm for one flow field across a specified solver\-budget range\. Its control realization uses a fixed\-grid bridge with an exact one\-step action\-reconstruction residual\. It aggregates temporal probes into one energy\-based surrogate ratio and one PPO\-style clip per action; the ratio is not the executed policy’s likelihood ratio\. For native velocity objectives, multi\-budget endpoint losses and an anchored max–min update extend the same deployment principle without replacing the original objective\. Both realizations retain one field and the original deployment architecture\.

Our contributions are:

- •A diagnosis tied to deployment\.We expose one\-step degradation hidden by training\-budget evaluation and identify how the supervision target suppresses the relevant boundary signal\.
- •An algorithm for multiple solver budgets\.We connect an exact action\-reconstruction identity to action\-aligned supervision, budget exposure, and worst\-budget checkpoint selection\.
- •Evidence across objectives and domains\.A controlled common\-stack benchmark and a released manipulation pipeline test the algorithm, with component studies and deployment\-cost accounting identifying the sources of improvement\. Across DM Control, manipulation, and locomotion,ArcFlowleads five flow\-policy baselines in mean one\-step return on 25 of 30 tasks including cross\-regime manipulation comparisons\.

## 2Problem formulation

Letssdenote an environment state,z∼p0z\\sim p\_\{0\}source noise, andvθ​\(x,t∣s\)v\_\{\\theta\}\(x,t\\mid s\)a velocity field with reverse\-time conventiont:1→0t:1\\to 0\. A numerical solverSKS\_\{K\}withKKsteps induces the policya=FθSK​\(z,s\)a=F^\{S\_\{K\}\}\_\{\\theta\}\(z,s\)\. The environment observes the resulting action; the solver’s internal queries do not constitute environment transitions\. WriteJK​\(θ\)J\_\{K\}\(\\theta\)for the expected episodic return of this policy\. Our objective is to learn one field that remains useful across a specified set of deployment budgets, without supplyingKKas an input to the field\.

Flow\-policy algorithms use several tractable training signals, including flow\-matching residuals, velocity\-based surrogates, and likelihoods of stochastic internal transitions\. These signals differ in their treatment of solver time\. An improvement in a training surrogate, or inJKtrainJ\_\{K\_\{\\rm train\}\}, does not in itself imply an improvement inJKJ\_\{K\}for a different budget\. We distinguish three outcomes\.*Numerical failure*is a non\-finite update or evaluation\.*Cross\-budget degradation*is a lower return at one deployment budget than another\.*Temporal degradation*is a fall in return during training at a fixed budget\. The census calls a task a*one\-step collapse*when its specified one\-to\-sixteen\-step return statistic falls below one half\. This threshold is descriptive and depends on the seed pool and aggregation rule; it is not a synonym for temporal degradation\. We report the graded return ratio alongside the threshold count\.

## 3Solver\-budget mismatch

![Refer to caption](https://arxiv.org/html/2610.28788v1/fig_census.png)Figure 2:ArcFlowstrengthens one\-step deployment across the task census and solver budgets\.\(a\) Within\-task normalized one\-step return on all 30 retained tasks from the registered selection results; dots mark a one\-to\-sixteen\-step mean\-return ratio below0\.50\.5\. The right band shows nine Playground tasks on the display regimes of Table[1](https://arxiv.org/html/2610.28788#S4.T1)\. \(b\) Mean normalized return versus solver steps over all 30 tasks for every method: 21 held\-out DM Control tasks and 9 Playground tasks on their registered display regimes\.#### Cross\-budget performance\.

Figure[2](https://arxiv.org/html/2610.28788#S3.F2)compares all six flow methods on the same 30\-task census\. In panel \(a\),ArcFlowrecords no one\-step collapses while maintaining strong one\-step returns across the DM Control and Playground tasks\. The marked collapse counts are0/300/30forArcFlow,2/302/30for FPO,4/304/30for FPO\+\+,0/300/30for DPPO,3/303/30for ReinFlow, and0/300/30for PolicyFlow\. Panel \(b\) shows thatArcFlowhas the highest mean normalized return at every evaluated solver budget\. Its aggregate return increases from one to two steps and then remains nearly constant through sixteen steps, whereas the other methods plateau at lower levels; the separation is largest at one\-step deployment\.

#### Boundary suppression\.

For a straight bridgext=\(1−t\)​a\+t​εx\_\{t\}=\(1\-t\)a\+t\\varepsilonand target velocityu=ε−au=\\varepsilon\-a, the reference implementation predicts noise asε^θ=xt\+\(1−t\)​vθ​\(xt,t∣s\)\\hat\{\\varepsilon\}\_\{\\theta\}=x\_\{t\}\+\(1\-t\)v\_\{\\theta\}\(x\_\{t\},t\\mid s\)\. Thus

‖ε^θ−ε‖2,d2=\(1−t\)2​‖vθ​\(xt,t∣s\)−u‖2,d2,\\\|\\hat\{\\varepsilon\}\_\{\\theta\}\-\\varepsilon\\\|\_\{2,d\}^\{2\}=\(1\-t\)^\{2\}\\\|v\_\{\\theta\}\(x\_\{t\},t\\mid s\)\-u\\\|\_\{2,d\}^\{2\},\(1\)where‖r‖2,d2=d−1​∑iri2\\\|r\\\|\_\{2,d\}^\{2\}=d^\{\-1\}\\sum\_\{i\}r\_\{i\}^\{2\}\. Both the loss and its actor gradient vanish att=1t=1, the only node queried by a one\-step reverse Euler solver\. Nearby supervision is also quadratically down\-weighted\. Velocity supervision removes this factor\. The ablations in Section[5\.4](https://arxiv.org/html/2610.28788#S5.SS4)attributes most of the one\-step recovery to this change\. The released ReinFlow study complements this target\-level diagnosis by testing deployment\-aware training in a pipeline that already supervises velocity \(Table[5\.2](https://arxiv.org/html/2610.28788#S5.SS2.SSS0.Px2)\)\.

#### Collapse versus absolute performance\.

The collapse count and the mean return capture different aspects of solver\-budget behavior\. DPPO and PolicyFlow have no marked collapses in Figure[2](https://arxiv.org/html/2610.28788#S3.F2)\(a\), yet their mean returns remain well belowArcFlowat every displayed budget in panel \(b\)\. Conversely, a method can achieve a higher aggregate return while still containing tasks with severe one\-to\-sixteen\-step degradation\. Evaluating low\-step return together with within\-policy retention is therefore necessary: the absence of collapse alone does not imply an effective low\-latency policy\.

## 4TheArcFlowalgorithm

Figure[1](https://arxiv.org/html/2610.28788#S1.F1)previewsArcFlow: its left panel contrasts the budget\-dependent rollout action with a reference endpoint; its right panel connects a collected action to fresh noise on a fixed\-grid bridge\. The boundary residual measures one\-step reconstruction error, while temporal probes yield one clipped surrogate weight per action\. Rollout budgets cycle, and worst\-budget validation selects the checkpoint\. The deployment target is

𝒥⁡\(θ\)=𝔼K∼p​\[JK​\(θ\)\]\.\\mathcal\{J\}\(\\theta\)=\\mathbb\{E\}\_\{K\\sim p\}\[J\_\{K\}\(\\theta\)\]\.\(2\)Algorithm[1](https://arxiv.org/html/2610.28788#algorithm1)specifies this control loop\. The native\-objective image\-policy variant instead uses multi\-budget endpoint supervision\.

### 4\.1Budget\-cycled rollouts

Forz∼𝒩⁡\(0,I\)z\\sim\\mathcal\{N\}\(0,I\), a reverse Euler rollout produces

x0=z,xj\+1=xj−h​vθ​\(xj,tj∣s\),tj=1−j/K,h=1/K,a=xK\.x\_\{0\}=z,\\qquad x\_\{j\+1\}=x\_\{j\}\-hv\_\{\\theta\}\(x\_\{j\},t\_\{j\}\\mid s\),\\qquad t\_\{j\}=1\-j/K,\\quad h=1/K,\\quad a=x\_\{K\}\.\(3\)Training cycles through𝒮=\{1,2,3,4,6,8,12,16\}\\mathcal\{S\}=\\\{1,2,3,4,6,8,12,16\\\}in increasing order, using oneKKfor every rollout in an outer iteration\. This gives equal exposure to each supported budget over a cycle\. Parameters evolve across the cycle, and the field is not conditioned onKK\.

### 4\.2Boundary\-aware bridge

For each collected tuple\(s,a,A^\)\(s,a,\\widehat\{A\}\), draw fresh noisez~∼𝒩⁡\(0,I\)\\widetilde\{z\}\\sim\\mathcal\{N\}\(0,I\)and define

x~t=\(1−t\)​a\+t​z~,u=z~−a\.\\widetilde\{x\}\_\{t\}=\(1\-t\)a\+t\\widetilde\{z\},\\qquad u=\\widetilde\{z\}\-a\.\(4\)The bridge is an analytic reference path to the observed action, not the trajectory that generated it\. On the fixed reference gridG4=\{1,3/4,1/2,1/4\}G\_\{4\}=\\\{1,3/4,1/2,1/4\\\}, samplejjuniformly and score

ℓj​\(θ,s,a,z~\)=h​‖vθ​\(x~tj,tj∣s\)−u‖2,d2,h=1/4\.\\ell\_\{j\}\(\\theta;s,a,\\widetilde\{z\}\)=h\\\|v\_\{\\theta\}\(\\widetilde\{x\}\_\{t\_\{j\}\},t\_\{j\}\\mid s\)\-u\\\|\_\{2,d\}^\{2\},\\qquad h=1/4\.\(5\)The mean sampled energy estimatesLθ=𝔼z~,j​\[ℓj\]L\_\{\\theta\}=\\mathbb\{E\}\_\{\\widetilde\{z\},j\}\[\\ell\_\{j\}\]\. For fixedz~\\widetilde\{z\}, this is14​∑j=03ℓj\\tfrac\{1\}\{4\}\\sum\_\{j=0\}^\{3\}\\ell\_\{j\}, not the unnormalized sum of grid residuals\. The factor matters because it sets the scale of the exponential weight below\. Every collecting solver uses the same grid and scale\.

###### Proposition 1\(Exact one\-step reconstruction\)\.

For any fixed\(s,a,z~\)\(s,a,\\widetilde\{z\}\), the boundary residual satisfies

h−1​ℓ0​\(θ\)=‖z~−vθ​\(z~,1∣s\)−a‖2,d2=‖FθS1​\(z~,s\)−a‖2,d2\.h^\{\-1\}\\ell\_\{0\}\(\\theta\)=\\\|\\widetilde\{z\}\-v\_\{\\theta\}\(\\widetilde\{z\},1\\mid s\)\-a\\\|\_\{2,d\}^\{2\}=\\\|F^\{S\_\{1\}\}\_\{\\theta\}\(\\widetilde\{z\},s\)\-a\\\|\_\{2,d\}^\{2\}\.

The identity follows by substitutingt0=1t\_\{0\}=1into Eqs\.[3](https://arxiv.org/html/2610.28788#S4.E3)–[5](https://arxiv.org/html/2610.28788#S4.E5)\. It links one quarter of the sampled\-node distribution to the reconstruction error of a one\-step action\. The appendix extends this connection to endpoint error bounds under regularity assumptions \(Propositions[3](https://arxiv.org/html/2610.28788#Thmproposition3)and[5](https://arxiv.org/html/2610.28788#Thmproposition5)\)\. The resulting design separates boundary supervision from the fixed\-grid bridge used to score actions at every rollout budget\.

#### Action\-level surrogate\.

Letθ¯\\bar\{\\theta\}be the rollout parameters\. CacheMMcommon noise–node probes for the old and new policies and form

Δ^θ=1M​∑m=1M\[ℓjm​\(θ¯\)−ℓjm​\(θ\)\],w^θ=exp⁡\{clip⁡\(Δ^θ,−cE,cE\)\}\.\\widehat\{\\Delta\}\_\{\\theta\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\[\\ell\_\{j\_\{m\}\}\(\\bar\{\\theta\}\)\-\\ell\_\{j\_\{m\}\}\(\\theta\)\],\\qquad\\widehat\{w\}\_\{\\theta\}=\\exp\\\{\\operatorname\{clip\}\(\\widehat\{\\Delta\}\_\{\\theta\},\-c\_\{E\},c\_\{E\}\)\\\}\.\(6\)We maximize a PPO\-style clipped surrogate\([Schulman et al\., 2017](https://arxiv.org/html/2610.28788#bib.bib3)\),

𝒥Arc​\(θ\)=𝔼⁡\[min⁡\(w^θ​A^,clip⁡\(w^θ,1−ϵ,1\+ϵ\)​A^\)\]\.\\mathcal\{J\}\_\{\\rm Arc\}\(\\theta\)=\\mathbb\{E\}\\left\[\\min\\left\(\\widehat\{w\}\_\{\\theta\}\\widehat\{A\},\\operatorname\{clip\}\(\\widehat\{w\}\_\{\\theta\},1\-\\epsilon,1\+\\epsilon\)\\widehat\{A\}\\right\)\\right\]\.\(7\)The probes are averaged before clipping, so each environment action receives one weight and one advantage\. The weight is an energy\-based surrogate, not the density ratio of the executed solver policy\. Since common probes giveΔ^θ¯=0\\widehat\{\\Delta\}\_\{\\bar\{\\theta\}\}=0, its local gradient is

∇θ𝒥Arc\|θ¯=−𝔼⁡\[A^​∇θL^θ​\(s,a\)\]\|θ¯\.\\left\.\\nabla\_\{\\theta\}\\mathcal\{J\}\_\{\\rm Arc\}\\right\|\_\{\\bar\{\\theta\}\}=\-\\left\.\\mathbb\{E\}\[\\widehat\{A\}\\nabla\_\{\\theta\}\\widehat\{L\}\_\{\\theta\}\(s,a\)\]\\right\|\_\{\\bar\{\\theta\}\}\.\(8\)Positive advantages therefore decrease reconstruction energy locally\. Appendix Proposition[4](https://arxiv.org/html/2610.28788#Thmproposition4)relates this expression to an auxiliary normalized exponential\-tilt policy under its stated centering assumptions; that auxiliary policy need not equal the deployed flow policy\.

Algorithm 1ArcFlow: bridge\-based control realization- 1:Require:parametersθ\\theta, support𝒮\\mathcal\{S\}, reference gridG4G\_\{4\}, validation bankℬval\\mathcal\{B\}\_\{\\rm val\}
- 2:Evaluateθ\\thetaat𝒦=\{1,4,16\}\\mathcal\{K\}=\\\{1,4,16\\\}onℬval\\mathcal\{B\}\_\{\\rm val\}\.
- 3:θ⋆←θ\\theta^\{\\star\}\\leftarrow\\theta;s⋆←minK∈𝒦⁡J^K​\(θ,ℬval\)s^\{\\star\}\\leftarrow\\min\_\{K\\in\\mathcal\{K\}\}\\widehat\{J\}\_\{K\}\(\\theta;\\mathcal\{B\}\_\{\\rm val\}\)
- 4:foreach training cycledo
- 5:forK∈𝒮=\{1,2,3,4,6,8,12,16\}K\\in\\mathcal\{S\}=\\\{1,2,3,4,6,8,12,16\\\}in orderdo
- 6:CollectKK\-step rollouts and estimate advantages\.
- 7:CacheMMnoise–node probes per action and old bridge energies\.
- 8:Apply Eqs\.[6](https://arxiv.org/html/2610.28788#S4.E6)–[7](https://arxiv.org/html/2610.28788#S4.E7)with those probes; update the critic with the backbone rule\.
- 9:Evaluate the current policy at𝒦\\mathcal\{K\}onℬval\\mathcal\{B\}\_\{\\rm val\}\.
- 10:s←minK∈𝒦⁡J^K​\(θ,ℬval\)s\\leftarrow\\min\_\{K\\in\\mathcal\{K\}\}\\widehat\{J\}\_\{K\}\(\\theta;\\mathcal\{B\}\_\{\\rm val\}\)
- 11:ifs\>s⋆s\>s^\{\\star\}then\(θ⋆,s⋆\)←\(θ,s\)\(\\theta^\{\\star\},s^\{\\star\}\)\\leftarrow\(\\theta,s\)
- 12:returnθ⋆\\theta^\{\\star\}\.

Algorithm[1](https://arxiv.org/html/2610.28788#algorithm1)instantiates the right\-hand sequence of Figure[1](https://arxiv.org/html/2610.28788#S1.F1): cycleKKfor rollout collection, score actions on fresh fixed\-grid probes, and apply one clipped update per action\. After each cycle it retains the checkpoint with the highest minimum validation return over𝒦=\{1,4,16\}\\mathcal\{K\}=\\\{1,4,16\\\}\. The rollout path and reference bridge have distinct roles: the former chooses the action that receives an environment advantage, whereas the latter scores that action against fresh noise\. ChangingKKtherefore changes which actions enter the update without changing the grid or scale of their bridge energies\. Locally, positive advantages lower the energy of collected actions and negative advantages raise it; the cached probes make the old and current energies comparable during each actor update\. At every rollout budget, uniform sampling onG4G\_\{4\}assigns probability1/41/4tot=1t=1\. Thus the exact one\-step residual in Proposition[1](https://arxiv.org/html/2610.28788#Thmproposition1)contributes to the expected actor update even when rollouts take many steps\. Averaging theMMprobes before exponentiation gives each collected action one weight, aligned with its one advantage, rather than separate weights for individual bridge nodes\.

### 4\.3Training and selection

The reference gridG4G\_\{4\}and its1/41/4residual scale remain fixed as the rollout budget changes\. Actor updates reuse cached probes, and the critic follows the backbone rule\. Algorithm[1](https://arxiv.org/html/2610.28788#algorithm1)evaluates the initial field and each cycle\-end checkpoint at𝒦=\{1,4,16\}\\mathcal\{K\}=\\\{1,4,16\\\}, then selects

θ⋆∈arg⁡maxθ∈𝒞​minK∈𝒦​J^Kval​\(θ\),\\theta^\{\\star\}\\in\\arg\\max\_\{\\theta\\in\\mathcal\{C\}\}\\min\_\{K\\in\\mathcal\{K\}\}\\widehat\{J\}^\{\\rm val\}\_\{K\}\(\\theta\),\(9\)where𝒞\\mathcal\{C\}is the set of evaluated checkpoints\. Taking the minimum prevents a high return at one validation budget from compensating for a low return at another\. The selectedθ⋆\\theta^\{\\star\}is one shared field: deployment changes the solver budget, not the checkpoint or the policy parameters\. Training and selection have different roles: Eq\.[7](https://arxiv.org/html/2610.28788#S4.E7)updates the field through the advantage\-weighted bridge surrogate, whereas Eq\.[9](https://arxiv.org/html/2610.28788#S4.E9)ranks the resulting checkpoints by environment return\. No gradient is taken through the validation returns\.

#### Native\-objective realization\.

The bridge surrogate is unnecessary when the native learner already provides velocity supervision\. In the released ReinFlow pipeline,ArcFlowretains the native actor objective as an anchor, collects atK=4K=4, and supervises deterministic endpoint views at𝒱=\{1,2,4,8,16\}\\mathcal\{V\}=\\\{1,2,4,8,16\\\}against stored rollout actions, weighted by normalized advantage midranks within each minibatch\. Letg0g\_\{0\}be the native ascent direction andgKg\_\{K\}the negative gradient of the endpoint loss at budgetKK; writeu0,uKu\_\{0\},u\_\{K\}for their unit directions\. The update preserves the native gradient norm while combining its direction equally with a max–min direction across views:

p⋆∈arg⁡max‖p‖≤1,⟨p,u0⟩≥0​minK∈𝒱​⟨p,uK⟩,d=‖g0‖​u0\+p⋆‖u0\+p⋆‖\.p^\{\\star\}\\in\\arg\\max\_\{\\\|p\\\|\\leq 1,\\ \\langle p,u\_\{0\}\\rangle\\geq 0\}\\min\_\{K\\in\\mathcal\{V\}\}\\langle p,u\_\{K\}\\rangle,\\qquad d=\\\|g\_\{0\}\\\|\\frac\{u\_\{0\}\+p^\{\\star\}\}\{\\\|u\_\{0\}\+p^\{\\star\}\\\|\}\.\(10\)The native AdamW optimizer proposes a step alongdd\. Backtracking accepts a finite step only if endpoint losses stay within a small tolerance of their pre\-step values and common\-noise endpoints remain close across budgets\. This check reuses training samples and makes no extra reward queries\. Appendix[B\.1](https://arxiv.org/html/2610.28788#A2.SS1)gives its thresholds, clock views, fallback, and optimizer\-state handling\. The separate cycling baseline changes only the collecting solver; both realizations deploy one field\.

Table 1:Cross\-domain evaluation at three deployment budgets\.Deployed episodic return forArcFlowand five flow\-policy baselines on 21 DM Control, three manipulation, and six locomotion tasks\. Bold marks the highest displayed mean for each task and budget\.

## 5Experiments

### 5\.1Experimental setup

#### Common\-stack benchmark\.

We instantiate all methods in a control stack derived from the released FPO Playground configuration\. It covers 21 DM Control tasks\([Tassa et al\., 2018](https://arxiv.org/html/2610.28788#bib.bib25)\), 64 outer iterations, and62\.962\.9million interactions; fixed\-budget arms collect at 4 steps\. We compare FPO\([McAllister et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib4)\), FPO\+\+\([Yi et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib9)\), DPPO\([Ren et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib10)\), ReinFlow\([Zhang et al\., 2025b](https://arxiv.org/html/2610.28788#bib.bib5)\), and PolicyFlow\([Yang et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib8)\)\. All arms share the normalizer, architecture, initialization, critic, and training\-interaction budget\. The DPPO row applies its denoising\-MDP policy\-gradient objective to the shared flow actor, not the original diffusion actor; Appendix[D](https://arxiv.org/html/2610.28788#A4)specifies the adaptation\. Raw returns use each configuration’s registered seeds and display regime; the Panda comparisons are cross\-regime\. Figure[2](https://arxiv.org/html/2610.28788#S3.F2)min–max normalizes each task over six methods, five budgets, and a Gaussian reference\. “Collapse” meansR1/R16<0\.5R\_\{1\}/R\_\{16\}<0\.5\. The separatecanstudy uses five matched training seeds\.

### 5\.2Multi\-budget performance

#### Broad low\-budget gains\.

Table[1](https://arxiv.org/html/2610.28788#S4.T1)reports the deployed return of all six methods at three deployment budgets on 30 tasks\. AtK=1K=1,ArcFlowhas the highest displayed mean on 25 of 30 tasks under the registered display regimes: 18 of 21 DM Control tasks, all 3 manipulation tasks, and 4 of six locomotion tasks\. Across the 25 wins, the strongest baseline is FPO on 11 tasks, PolicyFlow on six, ReinFlow on four, and FPO\+\+ and DPPO on two each\. The gains are also often substantial\. Using the displayed means, 12 of the 25 wins improve on the strongest baseline by at least10%10\\%, and nine improve by at least20%20\\%\. The largest multiplicative gains occur on HopperStand, whereArcFlowreaches361361versus115115for the strongest baseline; HopperHop, with60\.460\.4versus22\.822\.8; and PandaOpenCabinet, with618618versus225225\. The advantage is not limited to these outliers: at one step,ArcFlowalso obtains956956versus772772on CartpoleBalance,849849versus672672on CheetahRun,554554versus457457on FingerSpin, and10\.510\.5versus7\.547\.54on Op3Joystick\. Thus the lowest\-cost setting yields both broad task coverage and pronounced gains on several of the more difficult control and manipulation problems\.

Figure 3:One\-step learning dynamics across 30 tasks\.Mean raw episodic return atK=1K=1, ordered as in Table[1](https://arxiv.org/html/2610.28788#S4.T1)\. Training progress is the percentage of each method’s registered interaction budget\. Curves use the registered seeds\. Axis ranges are task\-specific; compare within panels\.
#### Domain and budget coverage\.

The manipulation block provides the clearest uniform result:ArcFlowhas the highest displayed mean at all three budgets on PandaPickCube, PandaPickCubeOrient, and PandaOpenCabinet\. In DM Control, it is best at all three budgets on eight tasks, including BallInCup, HopperHop, HopperStand, PendulumSwingup, PointMass, ReacherEasy, Swimmer6, and WalkerStand; it also leads all three budgets on Op3Joystick in locomotion\. Across the full table,ArcFlowis best on 13 tasks atK=4K=4and 14 atK=16K=16, and on 12 tasks at every reported budget\. Task\-level leaders are therefore more mixed at higher solver counts than at one step, but the table shows thatArcFlowcombines a broad low\-budget advantage with strong performance as additional computation becomes available\. These task\-level counts identify where each method leads; Figure[2](https://arxiv.org/html/2610.28788#S3.F2)\(b\) instead shows how each trained field performs as the deployment budget changes, averaged over the 30 tasks\.ArcFlowhas the highest mean normalized return at every evaluated solver count, with the largest separation at one step\. Its mean rises from one to two steps and then remains nearly constant through sixteen steps\. Figure[2](https://arxiv.org/html/2610.28788#S3.F2)\(a\) likewise records no one\-step collapse forArcFlowacross the 30\-task census\. Although the strongest high\-budget method varies on the remaining tasks, the aggregate curve stays above all baselines\. Thus the one\-step gains do not come at the cost of losing the policy’s higher\-budget capability\.

Table 2:Boundary supervision, bridge training, and worst\-KKselection improve deployment robustness\.Point\-estimate means on a 25\-task component pool\.K=1K=1and WorstKKevaluate normalized deployment return, with WorstKKsummarizing the weakest budget over𝒦\\mathcal\{K\}\. AUC is the mean four\-step training\-curve area, min–max normalized within each task over these rows\. Bold: column maximum\.

Table 3:ArcFlowrestores full\-range manipulation\.Meancansuccess \(%\) over five matched training seeds\. BC is the behavior\-cloned prior; “1\-step” is the specialist\. Worst and meanKKsummarize the five budget means\. Bold: row maximum\.

### 5\.3Training dynamics

#### Early emergence of low\-step gains\.

Figure[3](https://arxiv.org/html/2610.28788#S5.F3)shows that the deployed advantage often emerges before training ends\. On CheetahRun, FingerSpin, HopperHop, HopperStand, the Walker tasks, and the manipulation tasks,ArcFlowseparates from most baselines by roughly mid\-training\. Its Hopper returns rise steadily while competing curves remain near the floor or plateau lower, so Table[1](https://arxiv.org/html/2610.28788#S4.T1)’s differences are not created solely by the deployed endpoints\.

#### Stable and late\-stage improvement\.

On BallInCup, PointMass, ReacherEasy, and ReacherHard,ArcFlowrises early and remains stable; PendulumSwingup, Swimmer6, H1JoystickGait, and Op3Joystick keep improving after mid\-training\. On CartpoleBalance, PandaOpenCabinet, and H1InplaceGait, the selected rightmost marker rises above the preceding chronological trajectory\. The deployed result thus combines early acquisition, continued gains, and checkpoint selection\.

### 5\.4Control component analysis

Table[5\.2](https://arxiv.org/html/2610.28788#S5.SS2.SSS0.Px2)isolates the contribution of each component on the separate 25\-task component\-study pool\. Replacing noise reconstruction with directuu\-supervision raises normalized one\-step return from0\.3500\.350to0\.5340\.534and worst\-budget return from0\.3340\.334to0\.5080\.508, confirming that restoring the boundary signal accounts for a substantial part of the recovery\. This direct repair alone reduces training AUC to0\.3330\.333\. Introducing the fixed\-scale bridge atK=4K=4raises one\-step and worst\-budget return further to0\.6120\.612and0\.5930\.593, while producing the highest training AUC,0\.7000\.700\. This row shows the combined benefit of the fixed\-grid bridge formulation and its fixed scale\.

Budget exposure provides the next improvement\. Cycling over\{1,4,16\}\\\{1,4,16\\\}raises the worst\-budget return to0\.6350\.635, and the broader support𝒮\\mathcal\{S\}reaches0\.6680\.668at one step with training AUC0\.6610\.661\. Worst\-budget checkpoint selection completes the algorithm: the fullArcFlowrow reaches0\.7110\.711at one step and0\.6850\.685at its weakest evaluated budget, the strongest deployment values in the table\. The ladder shows that the components play distinct roles: boundary supervision repairs low\-step execution, the fixed\-grid bridge improves the training curve, cycling supplies budget coverage, and selection converts that coverage into a stronger deployment floor\.

### 5\.5Native\-pipeline validation

Table[5\.2](https://arxiv.org/html/2610.28788#S5.SS2.SSS0.Px2)tests whether the same deployment\-budget principle transfers beyond the reconstruction\-based control realization\. The released ReinFlow ShortCut pipeline starts from a behavior\-cloned image policy, uses native velocity supervision, and fine\-tunes a four\-step actor\. It tests the principle beyond FPO’s reconstruction target and compares one multi\-budget policy with a one\-step specialist and rollout\-budget cycling\. The released behavior\-cloned prior retains moderate performance across solver budgets, but official ReinFlow fine\-tuning raises four\-step success to92\.592\.5% while reducing one\-step success to1\.51\.5%\. The specialist reverses this tradeoff: it reaches98\.098\.0% at one step but has a worst\-budget score of only74\.674\.6%\. Cycling performs best at two and four steps, yet falls to78\.378\.3% at its weakest budget\. In contrast,ArcFlowremains between91\.191\.1% and97\.197\.1% across all five budgets, giving the highest worst\-budget score \(91\.191\.1%\) and mean score \(94\.794\.7%\)\. Thus, its advantage is the high performance floor obtained with one field across the deployment range\.

## 6Related work

Generative transport and control\.Diffusion models learn reverse transport from noise to data\([Sohl\-Dickstein et al\., 2015](https://arxiv.org/html/2610.28788#bib.bib29);[Ho et al\., 2020](https://arxiv.org/html/2610.28788#bib.bib30);[Song et al\., 2021b](https://arxiv.org/html/2610.28788#bib.bib31)\), while deterministic samplers and neural ODEs expose an accuracy–computation tradeoff\([Song et al\., 2021a](https://arxiv.org/html/2610.28788#bib.bib32);[Chen et al\., 2018](https://arxiv.org/html/2610.28788#bib.bib33)\)\. Flow matching and rectified flow learn velocity fields directly\([Lipman et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib1);[Liu et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib2)\); transport couplings and multisample matching straighten paths, and rectified flows scale to high\-resolution generation\([Tong et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib34);[Pooladian et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib35);[Esser et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib36)\)\. In control, generative models support planning and return conditioning\([Janner et al\., 2022](https://arxiv.org/html/2610.28788#bib.bib37);[Ajay et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib38)\), plus visuomotor imitation across 2D, goal\-conditioned, and 3D inputs\([Chi et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib26);[Reuss et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib39);[Ze et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib40)\)\. AdaFlow adapts integration to action variance and FlowPolicy learns one\-step consistency\([Hu et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib17);[Zhang et al\., 2025a](https://arxiv.org/html/2610.28788#bib.bib20)\)\. This work instead trains one flow field with RL for a declared set of deployment budgets\.

Reinforcement learning of generative policies\.Trust\-region and clipped policy objectives motivate many modern policy updates\([Schulman et al\., 2015](https://arxiv.org/html/2610.28788#bib.bib49);[Schulman et al\., 2017](https://arxiv.org/html/2610.28788#bib.bib3)\)\. DDPO casts denoising as an MDP, while DRaFT differentiates rewards through the sampler\([Black et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib42);[Clark et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib43)\)\. For control, DPPO applies policy gradients to diffusion\-policy denoising transitions and QVPO uses a value\-weighted update\([Ren et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib10);[Ding et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib44)\)\. ReinFlow makes flow transitions stochastic; FPO and FPO\+\+ form PPO\-style ratios from flow losses; PolicyFlow and Flow\-GRPO derive velocity\- or transition\-based objectives\([Zhang et al\., 2025b](https://arxiv.org/html/2610.28788#bib.bib5);[McAllister et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib4);[Yi et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib9);[Yang et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib8);[Liu et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib6)\)\. Offline and critic\-based methods include Diffusion\-QL, EDP, DTQL, BDPO, Q\-score matching, Flow Q\-Learning, and SAC Flow\([Wang et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib27);[Kang et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib45);[Chen et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib47);[Gao et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib46);[Psenka et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib28);[Park et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib16);[Zhang et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib7)\)\.ArcFlowfocuses on the return floor of one policy across solver budgets\.

Few\-step and one\-step policies\.DDIM and DPM\-Solver accelerate sampling numerically\([Song et al\., 2021a](https://arxiv.org/html/2610.28788#bib.bib32);[Lu et al\., 2022](https://arxiv.org/html/2610.28788#bib.bib15)\); progressive distillation, consistency models, and InstaFlow learn fast generators\([Salimans and Ho, 2022](https://arxiv.org/html/2610.28788#bib.bib11);[Song et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib12);[Liu et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib48)\)\. Shortcut models and MeanFlow train directly over variable or one\-step generation\([Frans et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib13);[Geng et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib14)\), and Consistency Policy distills a visuomotor policy\([Prasad et al\., 2024](https://arxiv.org/html/2610.28788#bib.bib41)\)\. One\-step flow policy mirror descent, FLAME, and MFPO bring this objective to RL\([Chen et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib18);[Li et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib19);[Dong et al\., 2026](https://arxiv.org/html/2610.28788#bib.bib21)\)\. These methods optimize a designated short sampler or fast policy;ArcFlowpreserves one RL\-trained field and measures it at every declaredK∈𝒦K\\in\\mathcal\{K\}\.

Training\-time weighting and deployment budgets\.P2, Min\-SNR, and EDM reweight diffusion times\([Choi et al\., 2022](https://arxiv.org/html/2610.28788#bib.bib22);[Hang et al\., 2023](https://arxiv.org/html/2610.28788#bib.bib23);[Karras et al\., 2022](https://arxiv.org/html/2610.28788#bib.bib24)\)\. Our problematic weight instead follows algebraically from noise reconstruction and is zero at the one\-step boundary\. Boundary supervision restores that signal, solver cycling exposes the intended budgets, and worst\-budget selection makes their return floor an explicit selection criterion\.

## 7Conclusion

ArcFlowaligns flow\-policy learning with its deployment range through boundary supervision, solver\-budget exposure, and worst\-budget selection\. Across three control domains it improves one\-step performance while retaining competitive higher\-budget returns; the native ReinFlow study extends the principle to a released image\-policy pipeline and five budgets\. Deployment retains one architecture and pays only for the selected solver budget\. Component studies further separate the gains from action\-aligned supervision, budget exposure, and selection\. These results support training and evaluating generative policies over their full deployment range\.

### AI use statement

Generative AI tools were used to help develop and critique the conceptual framework and hypotheses; formulate mathematical claims and proofs; design synthetic environments and experimental protocols; implement, test, and edit software; interpret results; search literature; prepare figures and tables; and draft and edit the manuscript\. AI\-generated outputs were not treated as empirical evidence: reported values come from versioned artifacts, citations are checked against primary sources, and assumptions and proofs are stated in the paper\. Generative AI was not used to generate or clean experimental datasets or to translate source material; qualitative or thematic data analysis was not applicable\. We reviewed the AI\-assisted work and take responsibility for the final text, claims, code, and artifacts\.

### Ethics statement

This work studies reinforcement learning for simulated control and does not involve human participants, personal data, or physical\-robot experiments\. Like other reinforcement\-learning methods,ArcFlowmay amplify errors or unintended incentives in a specified reward function; improved optimization does not by itself provide safety guarantees\. Applying the method to physical or safety\-critical systems would therefore require independently validated objectives, operational constraints, monitoring, and fail\-safe mechanisms\. We report task\-level outcomes and the limits of the present evaluation to support accurate interpretation of the results\.

### Reproducibility statement

Code and artifacts are available in the anonymous repository at[https://anonymous\.4open\.science/r/arcflow\-868A](https://anonymous.4open.science/r/arcflow-868A)\. The repository contains theArcFlowimplementation, experiment configurations\. The main text and appendices specify the algorithms, solver budgets, checkpoint\-selection rule, evaluation conventions, and matched five\-seed manipulation protocol; the repository provides the experiment configurations\.

## References

- Ajayet al\.\(2023\)A\. Ajay, Y\. Du, A\. Gupta, J\. B\. Tenenbaum, T\. S\. Jaakkola, and P\. AgrawalIs conditional generative modeling all you need for decision\-making?\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=sP1fo2K9DFG)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Blacket al\.\(2024\)K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. LevineTraining diffusion models with reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/14f75513f0f1ca01de1e826b52e6b840-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Chenet al\.\(2018\)R\. T\. Q\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. K\. DuvenaudNeural ordinary differential equations\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper/2018/hash/69386f6bb1dfed68692a24c8686939b9-Abstract.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Chenet al\.\(2025\)T\. Chen, H\. Ma, N\. Li, K\. Wang, and B\. DaiOne\-step flow policy mirror descent\.arXiv preprint arXiv:2507\.23675\.External Links:[Link](https://arxiv.org/abs/2507.23675)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Chenet al\.\(2024\)T\. Chen, Z\. Wang, and M\. ZhouDiffusion policies creating a trust region for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/59a48c111f97f2174709ea9ed8e920d1-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Chiet al\.\(2023\)C\. Chi, S\. Feng, Y\. Du, Z\. Xu, E\. Cousineau, B\. Burchfiel, and S\. SongDiffusion policy: visuomotor policy learning via action diffusion\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2023.XIX.026),[Link](https://roboticsproceedings.org/rss19/p026.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Choiet al\.\(2022\)J\. Choi, J\. Lee, C\. Shin, S\. Kim, H\. Kim, and S\. YoonPerception prioritized training of diffusion models\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11472–11481\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2022/html/Choi_Perception_Prioritized_Training_of_Diffusion_Models_CVPR_2022_paper.html)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p3.1),[§6](https://arxiv.org/html/2610.28788#S6.p4.1)\.
- Clarket al\.\(2024\)K\. Clark, P\. Vicol, K\. Swersky, and D\. J\. FleetDirectly fine\-tuning diffusion models on differentiable rewards\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/13e8be77982beb73d7ed0bbf122f9f3c-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Dinget al\.\(2024\)S\. Ding, K\. Hu, Z\. Zhang, K\. Ren, W\. Zhang, J\. Yu, J\. Wang, and Y\. ShiDiffusion\-based reinforcement learning via Q\-weighted variational policy optimization\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://openreview.net/forum?id=UWUUVKtKeu)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Donget al\.\(2026\)X\. Dong, X\. S\. Zhang, and J\. ChengMean flow policy optimization\.arXiv preprint arXiv:2604\.14698\.External Links:[Link](https://arxiv.org/abs/2604.14698)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Esseret al\.\(2024\)P\. Esser, S\. Kulal, A\. Blattmann, R\. Entezari, J\. Müller, H\. Saini, Y\. Levi, D\. Lorenz, A\. Sauer, F\. Boesel, D\. Podell, T\. Dockhorn, Z\. English, and R\. RombachScaling rectified flow transformers for high\-resolution image synthesis\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 12606–12633\.External Links:[Link](https://proceedings.mlr.press/v235/esser24a.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Franset al\.\(2025\)K\. Frans, D\. Hafner, S\. Levine, and P\. AbbeelOne step diffusion via shortcut models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2410.12557)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Gaoet al\.\(2025\)C\. Gao, C\. Wu, M\. Cao, C\. Xiao, Y\. Yu, and Z\. ZhangBehavior\-regularized diffusion policy optimization for offline reinforcement learning\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267\.External Links:[Link](https://openreview.net/forum?id=pUCYJ9JJuZ)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Genget al\.\(2025\)Z\. Geng, M\. Deng, X\. Bai, J\. Z\. Kolter, and K\. HeMean flows for one\-step generative modeling\.arXiv preprint arXiv:2505\.13447\.External Links:[Link](https://arxiv.org/abs/2505.13447)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Hanget al\.\(2023\)T\. Hang, S\. Gu, C\. Li, J\. Bao, D\. Chen, H\. Hu, X\. Geng, and B\. GuoEfficient diffusion training via min\-SNR weighting strategy\.InIEEE/CVF International Conference on Computer Vision,pp\. 7441–7451\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2023/html/Hang_Efficient_Diffusion_Training_via_Min-SNR_Weighting_Strategy_ICCV_2023_paper.html)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p3.1),[§6](https://arxiv.org/html/2610.28788#S6.p4.1)\.
- Hoet al\.\(2020\)J\. Ho, A\. Jain, and P\. AbbeelDenoising diffusion probabilistic models\.InAdvances in Neural Information Processing Systems,Vol\.33\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Huet al\.\(2024\)X\. Hu, B\. Liu, X\. Liu, and Q\. LiuAdaFlow: imitation learning with variance\-adaptive flow\-based policies\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/fa9cfdb49f7de3dee54007f84c0745b1-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Janneret al\.\(2022\)M\. Janner, Y\. Du, J\. Tenenbaum, and S\. LevinePlanning with diffusion for flexible behavior synthesis\.InProceedings of the 39th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.162,pp\. 9902–9915\.External Links:[Link](https://proceedings.mlr.press/v162/janner22a.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Kanget al\.\(2023\)B\. Kang, X\. Ma, C\. Du, T\. Pang, and S\. YanEfficient diffusion policies for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/d45e0bfb5a39477d56b55c0824200008-Abstract.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Karraset al\.\(2022\)T\. Karras, M\. Aittala, T\. Aila, and S\. LaineElucidating the design space of diffusion\-based generative models\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/a98846e9d9cc01cfb87eb694d946ce6b-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p4.1)\.
- Liet al\.\(2026\)Z\. Li, Y\. Wang, H\. Wang, Z\. Li, and Z\. ZuoBoosting maximum entropy reinforcement learning via one\-step flow matching\.arXiv preprint arXiv:2602\.01606\.External Links:[Link](https://arxiv.org/abs/2602.01606)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Lipmanet al\.\(2023\)Y\. Lipman, R\. T\. Q\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. LeFlow matching for generative modeling\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2210.02747)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. OuyangFlow\-GRPO: training flow matching models via online RL\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/3a10c46572628d58cb44fb705f25cbbf-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Liuet al\.\(2023\)X\. Liu, C\. Gong, and Q\. LiuFlow straight and fast: learning to generate and transfer data with rectified flow\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2209.03003)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Liuet al\.\(2024\)X\. Liu, X\. Zhang, J\. Ma, J\. Peng, and Q\. LiuInstaFlow: one step is enough for high\-quality diffusion\-based text\-to\-image generation\.InInternational Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/hash/4dc37a7bc61057252ce043fa3b83aac2-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Luet al\.\(2022\)C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. ZhuDPM\-Solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps\.InAdvances in Neural Information Processing Systems,Vol\.35\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/260a14acce2a89dad36adc8eefe7c59e-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- McAllisteret al\.\(2025\)D\. McAllister, S\. Ge, B\. Yi, C\. M\. Kim, E\. Weber, H\. Choi, H\. Feng, and A\. KanazawaFlow matching policy gradients\.arXiv preprint arXiv:2507\.21053\.External Links:[Link](https://arxiv.org/abs/2507.21053)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p2.1),[§1](https://arxiv.org/html/2610.28788#S1.p3.1),[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Parket al\.\(2025\)S\. Park, Q\. Li, and S\. LevineFlow q\-learning\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 48104–48127\.External Links:[Link](https://proceedings.mlr.press/v267/park25f.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Pooladianet al\.\(2023\)A\. Pooladian, H\. Ben\-Hamu, C\. Domingo\-Enrich, B\. Amos, Y\. Lipman, and R\. T\. Q\. ChenMultisample flow matching: straightening flows with minibatch couplings\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 28100–28127\.External Links:[Link](https://proceedings.mlr.press/v202/pooladian23a.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Prasadet al\.\(2024\)A\. Prasad, K\. Lin, J\. Wu, L\. Zhou, and J\. BohgConsistency policy: accelerated visuomotor policies via consistency distillation\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2024.XX.071),[Link](https://roboticsproceedings.org/rss20/p071.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Psenkaet al\.\(2024\)M\. Psenka, A\. Escontrela, P\. Abbeel, and Y\. MaLearning a diffusion model policy from rewards via Q\-score matching\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 41163–41182\.External Links:[Link](https://proceedings.mlr.press/v235/psenka24a.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Renet al\.\(2025\)A\. Z\. Ren, J\. Lidard, L\. Ankile, A\. Simeonov, P\. Agrawal, A\. Majumdar, B\. Burchfiel, H\. Dai, and M\. SimchowitzDiffusion policy policy optimization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mEpqHvbD2h)Cited by:[Appendix D](https://arxiv.org/html/2610.28788#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.28788#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Reusset al\.\(2023\)M\. Reuss, M\. Li, X\. Jia, and R\. LioutikovGoal\-conditioned imitation learning using score\-based diffusion policies\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2023.XIX.028),[Link](https://roboticsproceedings.org/rss19/p028.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Salimans and Ho \(2022\)T\. Salimans and J\. HoProgressive distillation for fast sampling of diffusion models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2202.00512)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Schulmanet al\.\(2015\)J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. MoritzTrust region policy optimization\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37,pp\. 1889–1897\.External Links:[Link](https://proceedings.mlr.press/v37/schulman15.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.External Links:[Link](https://arxiv.org/abs/1707.06347)Cited by:[§4\.2](https://arxiv.org/html/2610.28788#S4.SS2.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Sohl\-Dicksteinet al\.\(2015\)J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. GanguliDeep unsupervised learning using nonequilibrium thermodynamics\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37,pp\. 2256–2265\.External Links:[Link](https://proceedings.mlr.press/v37/sohl-dickstein15.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Songet al\.\(2021a\)J\. Song, C\. Meng, and S\. ErmonDenoising diffusion implicit models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=St1giarCHLP)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Songet al\.\(2023\)Y\. Song, P\. Dhariwal, M\. Chen, and I\. SutskeverConsistency models\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 32211–32252\.External Links:[Link](https://proceedings.mlr.press/v202/song23a.html)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p3.1)\.
- Songet al\.\(2021b\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PxTIG12RRHS)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Tassaet al\.\(2018\)Y\. Tassa, Y\. Doron, A\. Muldal, T\. Erez, Y\. Li, D\. de Las Casas, D\. Budden, A\. Abdolmaleki, J\. Merel, A\. Lefrancq, T\. Lillicrap, and M\. RiedmillerDeepMind control suite\.arXiv preprint arXiv:1801\.00690\.External Links:[Link](https://arxiv.org/abs/1801.00690)Cited by:[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1)\.
- Tonget al\.\(2024\)A\. Tong, K\. Fatras, N\. Malkin, G\. Huguet, Y\. Zhang, J\. Rector\-Brooks, G\. Wolf, and Y\. BengioImproving and generalizing flow\-based generative models with minibatch optimal transport\.Transactions on Machine Learning Research,pp\. 1–34\.External Links:[Link](https://openreview.net/forum?id=CD9Snc73AW)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Wanget al\.\(2023\)Z\. Wang, J\. J\. Hunt, and M\. ZhouDiffusion policies as an expressive policy class for offline reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=AHvFDPi-FA)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Yanget al\.\(2026\)S\. Yang, B\. Liu, and H\. ChenPolicyFlow: policy optimization with continuous normalizing flow in reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YETCQLcKtn)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Yiet al\.\(2026\)B\. Yi, H\. Choi, H\. G\. Singh, X\. Huang, T\. E\. Truong, C\. Sferrazza, Y\. Ma, R\. Duan, P\. Abbeel, G\. Shi, K\. Liu, and A\. KanazawaFlow policy gradients for robot control\.arXiv preprint arXiv:2602\.02481\.External Links:[Link](https://arxiv.org/abs/2602.02481)Cited by:[§1](https://arxiv.org/html/2610.28788#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Zeet al\.\(2024\)Y\. Ze, G\. Zhang, K\. Zhang, C\. Hu, M\. Wang, and H\. Xu3D diffusion policy: generalizable visuomotor policy learning via simple 3D representations\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2024.XX.067),[Link](https://roboticsproceedings.org/rss20/p067.html)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Zhanget al\.\(2025a\)Q\. Zhang, Z\. Liu, H\. Fan, G\. Liu, B\. Zeng, and S\. LiuFlowPolicy: enabling fast and robust 3d flow\-based policy via consistency flow matching for robot manipulation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 14754–14762\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v39i14.33617),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/33617)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p1.1)\.
- Zhanget al\.\(2025b\)T\. Zhang, C\. Yu, S\. Su, and Y\. WangReinFlow: fine\-tuning flow matching policy with online reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/98d6f928497d9eaad9f320fb2db3040d-Abstract-Conference.html)Cited by:[Appendix D](https://arxiv.org/html/2610.28788#A4.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.28788#S1.p2.1),[§1](https://arxiv.org/html/2610.28788#S1.p3.1),[§5\.1](https://arxiv.org/html/2610.28788#S5.SS1.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, S\. Yu, T\. Zhang, M\. Guang, H\. Hui, K\. Long, Y\. Wang, C\. Yu, and W\. DingSAC Flow: sample\-efficient reinforcement learning of flow\-based policies via velocity\-reparameterized sequential modeling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=zZvWj4JrYj)Cited by:[§6](https://arxiv.org/html/2610.28788#S6.p2.1)\.

## Appendix ALimitations and scope

#### Empirical scope\.

Our evidence is limited to simulated control\. The common\-stack study covers DM Control, simulated Panda manipulation, and locomotion tasks, while the native image\-policy study examines one manipulation task\. These experiments do not establish robustness to real\-robot sensing and actuation, safety constraints, distribution shift, or substantially larger policy and action spaces\. The native study also implements the deployment\-budget principle with endpoint supervision rather than the bridge objective, so it supports transfer of the principle but is not an independent test of the same update rule\.

#### Comparison and statistical scope\.

Most baselines in the common\-stack benchmark are controlled adaptations to a shared flow backbone, critic, and interaction budget; they are not reproductions of each method’s native architecture, pretraining, or published benchmark\.

#### Training and selection cost\.

ArcFlowleaves deployment architecture and per\-step solver cost unchanged, but training is not free: the bridge realization evaluates cached probes, cycles through several rollout budgets, and repeatedly measures validation return atK∈\{1,4,16\}K\\in\\\{1,4,16\\\}\. In the reported control protocol, validation adds2\.75%2\.75\\%to the environment\-interaction count, in addition to neural\-network compute from the probes and multi\-budget evaluations\. Moreover, selection protects only the budgets it evaluates and gives no guarantee for unseen solver counts\.

#### Objective and theory\.

The bridge weight is an energy\-based surrogate, not the likelihood ratio of the executed policy, and PPO\-style clipping therefore has no exact trust\-region interpretation\. The appendix bounds finite\-solver reconstruction error for a deterministic Euler field under Lipschitz assumptions; it does not prove monotone improvement in environment return, and it does not directly cover the clipped ShortCut operator used in the native pipeline\. The fixed four\-node bridge, the chosen budget sets, and the associated thresholds are empirical design choices\. Whether the same choices remain effective with adaptive or higher\-order solvers, continuous budget ranges, longer horizons, or other generative\-policy parameterizations remains open\.

## Appendix BImplementation details

Algorithm[1](https://arxiv.org/html/2610.28788#algorithm1)and the complete bridge\-based control realization appear in Section[4\.3](https://arxiv.org/html/2610.28788#S4.SS3)\. This supplement gives the mathematical details, baseline scope, energy\-scale diagnostic, and exact native\-anchored manipulation update\.

### B\.1Native\-objective manipulation update

The configuration collects with the released ReinFlow ShortCut actor at four steps\. Its extra supervision evaluates the same actor at𝒱=\{1,2,4,8,16\}\\mathcal\{V\}=\\\{1,2,4,8,16\\\}using the stored rollout noise and actions\. ShortCut retains its native time and duration inputs; the reverse\-time field in the bridge realization has only time\. The following specifies the actual manipulation update, including its distinction from the bridge loss of Eq\.[7](https://arxiv.org/html/2610.28788#S4.E7)\.

#### Endpoint views and advantage weights\.

For each captured minibatchBBof\(si,zi,ai,A^i\)\(s\_\{i\},z\_\{i\},a\_\{i\},\\widehat\{A\}\_\{i\}\),ziz\_\{i\}andaia\_\{i\}are the initial and final stored chain states\. LetEK,α​\(θ,z,s\)E\_\{K,\\alpha\}\(\\theta;z,s\)be the released ShortCut field integrated ontj=\(j/K\)αt\_\{j\}=\(j/K\)^\{\\alpha\}, using its native forward\-time convention, durationtj\+1−tjt\_\{j\+1\}\-t\_\{j\}, and action clipping after every step\. The trained views useα=1\\alpha=1\. Within each minibatch, define normalized empirical midrank weights

ri=∑j∈B𝟏\{A^j<A^i\}\+12\(1\+∑j∈B𝟏\{A^j=A^i\}\),wi=ri/∑j∈Brj\.r\_\{i\}=\\sum\_\{j\\in B\}\\mathbf\{1\}\\\{\\widehat\{A\}\_\{j\}<\\widehat\{A\}\_\{i\}\\\}\+\\tfrac\{1\}\{2\}\\left\(1\+\\sum\_\{j\\in B\}\\mathbf\{1\}\\\{\\widehat\{A\}\_\{j\}=\\widehat\{A\}\_\{i\}\\\}\\right\),\\qquad w\_\{i\}=r\_\{i\}\\Big/\\sum\_\{j\\in B\}r\_\{j\}\.\(11\)The endpoint objective is

LKend​\(θ\)=∑B∑i∈Bwi​‖EK,1​\(θ,zi,si\)−stopgrad⁡\(ai\)‖2,d2\.L\_\{K\}^\{\\rm end\}\(\\theta\)=\\sum\_\{B\}\\sum\_\{i\\in B\}w\_\{i\}\\\|E\_\{K,1\}\(\\theta;z\_\{i\},s\_\{i\}\)\-\\operatorname\{stopgrad\}\(a\_\{i\}\)\\\|\_\{2,d\}^\{2\}\.\(12\)The loss sums the normalized minibatch losses\. All1515accumulation minibatches oncancontribute to every view gradient\. This matches the native update’s accumulation window\. Rank weights favor high\-advantage actions and handle ties with their average rank; they are not signed policy\-gradient coefficients\.

#### Native anchor and direction\.

Letg0g\_\{0\}be the negative of the accumulated native actor gradient \(policy loss plus the release’s entropy and behavior\-cloning terms\), andgK=−∇LKendg\_\{K\}=\-\\nabla L\_\{K\}^\{\\rm end\}\. Normalize these to unit vectorsu0,uKu\_\{0\},u\_\{K\}\. The small view\-space solve is

p⋆∈arg⁡max‖p‖≤1,⟨p,u0⟩≥0​minK∈𝒱​⟨p,uK⟩\.p^\{\\star\}\\in\\arg\\max\_\{\\\|p\\\|\\leq 1,\\ \\langle p,u\_\{0\}\\rangle\\geq 0\}\\min\_\{K\\in\\mathcal\{V\}\}\\langle p,u\_\{K\}\\rangle\.\(13\)It is implemented through the simplex\-plus\-ray dualminβ∈Δ,λ≥0⁡‖∑KβK​uK\+λ​u0‖\\min\_\{\\beta\\in\\Delta,\\lambda\\geq 0\}\\\|\\sum\_\{K\}\\beta\_\{K\}u\_\{K\}\+\\lambda u\_\{0\}\\\|, using a float64 Gram matrix and active\-set enumeration\. The applied direction uses a fixed equal blend,

d=‖g0‖​12​u0\+12​p⋆‖12​u0\+12​p⋆‖\.d=\\\|g\_\{0\}\\\|\\frac\{\\tfrac\{1\}\{2\}u\_\{0\}\+\\tfrac\{1\}\{2\}p^\{\\star\}\}\{\\\|\\tfrac\{1\}\{2\}u\_\{0\}\+\\tfrac\{1\}\{2\}p^\{\\star\}\\\|\}\.\(14\)A degenerate max–min solution falls back tod=g0d=g\_\{0\}; zero or non\-finite input gradients are rejected by the implementation\. The norm floor is10−1210^\{\-12\}, feasibility tolerance10−810^\{\-8\}, and cone\-apex tolerance the square root of the input dtype’s machine epsilon\. The equal blend is an explicit native anchor, applied after the max–min solve\.

#### Step acceptance\.

Replace the actor’s gradient by−d\-dand call the native AdamW step to obtain a candidate parameter stateθc\\theta\_\{c\}and optimizer state\. Tryθf=θ0\+f⁡\(θc−θ0\)\\theta\_\{f\}=\\theta\_\{0\}\+f\(\\theta\_\{c\}\-\\theta\_\{0\}\)in descending orderf∈\{1,1/2,1/4,1/8,0\}f\\in\\\{1,1/2,1/4,1/8,0\\\}\. Accept the first finite candidate for which everyLKendL\_\{K\}^\{\\rm end\}increases by at most10−710^\{\-7\}from its pre\-step value on the captured minibatches and the maximum old\-to\-candidate common\-noise endpoint RMSE is at most0\.25\+10−70\.25\+10^\{\-7\}\. The displacement check uses the first3232stored noise–state pairs and all1515views𝒱×\{α=1,2,4\}\\mathcal\{V\}\\times\\\{\\alpha=1,2,4\\\}\. The view losses use only the native\-clock views; the clock\-warped views constrain displacement\.

Actor optimizer moments interpolate between the old and candidate moments with the accepted fractionff\. A positive fraction retains the candidate step counter;f=0f=0restores the old actor state completely\. The critic’s native update is unchanged\. Acceptance reuses training advantages and samples and makes no additional reward queries; it is a finite\-step surrogate check, not an independent return certificate\. The separate cycling control uses the native update and cyclesK∈\{1,2,4\}K\\in\\\{1,2,4\\\}at iteration boundaries, with fixed\-KKvalidation\.

## Appendix CMathematical basis

The statements below concern the deterministic reverse Euler field in Section[4](https://arxiv.org/html/2610.28788#S4)\. They identify the supervised deployment boundary and bound action reconstruction under explicit regularity assumptions\. They do not assert monotone environment return or transfer directly to the clipped ShortCut operator in Appendix[B\.1](https://arxiv.org/html/2610.28788#A2.SS1)\.

### C\.1Boundary non\-coercivity

###### Proposition 2\(Boundary non\-coercivity\)\.

Fix a scalar velocity errorccand letbδ​\(t\)=c​max⁡\{0,\(t−\(1−δ\)\)/δ\}b\_\{\\delta\}\(t\)=c\\max\\\{0,\(t\-\(1\-\\delta\)\)/\\delta\\\}\. This perturbation is continuous, has one\-step boundary errorbδ​\(1\)2=c2b\_\{\\delta\}\(1\)^\{2\}=c^\{2\}, and has continuous noise\-reconstruction risk∫01\(1−t\)2​bδ​\(t\)2​𝑑t=c2​δ3/30\\int\_\{0\}^\{1\}\(1\-t\)^\{2\}b\_\{\\delta\}\(t\)^\{2\}\\,dt=c^\{2\}\\delta^\{3\}/30\. Thus arbitrarily small continuous surrogate risk does not control the deployed one\-step endpoint error, even over continuous fields\.

*Proof\.*Substituteq=\(t−\(1−δ\)\)/δq=\(t\-\(1\-\\delta\)\)/\\deltaon\[1−δ,1\]\[1\-\\delta,1\]:∫01\(1−t\)2​bδ​\(t\)2​𝑑t=c2​δ3​∫01q2​\(1−q\)2​𝑑q=c2​δ3/30\\int\_\{0\}^\{1\}\(1\-t\)^\{2\}b\_\{\\delta\}\(t\)^\{2\}\\,dt=c^\{2\}\\delta^\{3\}\\int\_\{0\}^\{1\}q^\{2\}\(1\-q\)^\{2\}\\,dq=c^\{2\}\\delta^\{3\}/30, whilebδ​\(1\)2=c2b\_\{\\delta\}\(1\)^\{2\}=c^\{2\}\. ∎

### C\.2Finite\-solver endpoint bound

###### Proposition 3\(Finite\-solver endpoint bound\)\.

Withxj⋆x^\{\\star\}\_\{j\}the bridge nodes,x^j\\widehat\{x\}\_\{j\}the Euler trajectory from the same source,rj=vθ​\(xj⋆,tj∣s\)−ur\_\{j\}=v\_\{\\theta\}\(x^\{\\star\}\_\{j\},t\_\{j\}\\mid s\)\-u, andvθv\_\{\\theta\}LL\-Lipschitz in its state argument uniformly over the grid times,

‖x^K−a‖≤h​∑j=0K−1\(1\+h​L\)K−1−j​‖rj‖,‖x^K−a‖2≤h2​\[∑j\(1\+h​L\)2​\(K−1−j\)\]​\[∑j‖rj‖2\]\.\\\|\\widehat\{x\}\_\{K\}\-a\\\|\\leq h\\sum\_\{j=0\}^\{K\-1\}\(1\+hL\)^\{K\-1\-j\}\\\|r\_\{j\}\\\|,\\quad\\\|\\widehat\{x\}\_\{K\}\-a\\\|^\{2\}\\leq h^\{2\}\\\!\\left\[\\sum\_\{j\}\(1\+hL\)^\{2\(K\-1\-j\)\}\\right\]\\left\[\\sum\_\{j\}\\\|r\_\{j\}\\\|^\{2\}\\right\]\.\(15\)

Letej=x^j−xj⋆e\_\{j\}=\\widehat\{x\}\_\{j\}\-x\_\{j\}^\{\\star\},h=1/Kh=1/K, andrj=vθ​\(xj⋆,tj∣s\)−ur\_\{j\}=v\_\{\\theta\}\(x\_\{j\}^\{\\star\},t\_\{j\}\\mid s\)\-u\. Lipschitz continuity gives

‖ej\+1‖≤\(1\+h​L\)​‖ej‖\+h​‖rj‖\.\\\|e\_\{j\+1\}\\\|\\leq\(1\+hL\)\\\|e\_\{j\}\\\|\+h\\\|r\_\{j\}\\\|\.\(16\)Withe0=0e\_\{0\}=0, induction yields

‖eK‖≤h​∑j=0K−1\(1\+h​L\)K−1−j​‖rj‖\.\\\|e\_\{K\}\\\|\\leq h\\sum\_\{j=0\}^\{K\-1\}\(1\+hL\)^\{K\-1\-j\}\\\|r\_\{j\}\\\|\.\(17\)Cauchy–Schwarz gives the squared form\. ∎

### C\.3One\-step identity and local gradient

On the fixed reference gridG4G\_\{4\},h=1/4h=1/4,t0=1t\_\{0\}=1, and the boundary query isx~t0=z~\\widetilde\{x\}\_\{t\_\{0\}\}=\\widetilde\{z\}\. Substitutingu=z~−au=\\widetilde\{z\}\-agives

h−1​ℓ0​\(θ\)\\displaystyle h^\{\-1\}\\ell\_\{0\}\(\\theta\)=‖vθ​\(z~,1∣s\)−\(z~−a\)‖2,d2\\displaystyle=\\\|v\_\{\\theta\}\(\\widetilde\{z\},1\\mid s\)\-\(\\widetilde\{z\}\-a\)\\\|\_\{2,d\}^\{2\}\(18\)=‖z~−vθ​\(z~,1∣s\)−a‖2,d2\.\\displaystyle=\\\|\\widetilde\{z\}\-v\_\{\\theta\}\(\\widetilde\{z\},1\\mid s\)\-a\\\|\_\{2,d\}^\{2\}\.\(19\)The deployed one\-step reverse\-Euler endpoint from the same source isFθEuler1​\(z~,s\)=z~−vθ​\(z~,1∣s\)F^\{\\mathrm\{Euler\}\_\{1\}\}\_\{\\theta\}\(\\widetilde\{z\},s\)=\\widetilde\{z\}\-v\_\{\\theta\}\(\\widetilde\{z\},1\\mid s\), proving Proposition[1](https://arxiv.org/html/2610.28788#Thmproposition1)\.

WriteL^θ=M−1​∑mℓm​\(θ\)\\widehat\{L\}\_\{\\theta\}=M^\{\-1\}\\sum\_\{m\}\\ell\_\{m\}\(\\theta\),Δ^θ=L^θ¯−L^θ\\widehat\{\\Delta\}\_\{\\theta\}=\\widehat\{L\}\_\{\\bar\{\\theta\}\}\-\\widehat\{L\}\_\{\\theta\}, andw^θ=exp⁡\{clip⁡\(Δ^θ,−cE,cE\)\}\\widehat\{w\}\_\{\\theta\}=\\exp\\\{\\operatorname\{clip\}\(\\widehat\{\\Delta\}\_\{\\theta\},\-c\_\{E\},c\_\{E\}\)\\\}\. Atθ=θ¯\\theta=\\bar\{\\theta\},Δ^=0\\widehat\{\\Delta\}=0,w^=1\\widehat\{w\}=1, and, because the inner cap is locally inactive,∇w^=−∇L^\\nabla\\widehat\{w\}=\-\\nabla\\widehat\{L\}\. Away from the outer clip kink, the actor objective therefore has derivative−A^∇L^\-\\widehat\{A\}\\nabla\\widehat\{L\}\. Taking expectation establishes Eq\.[8](https://arxiv.org/html/2610.28788#S4.E8)\. This derivation requires common probes: independent old and new probes would make the sampled energy difference nonzero even at identical parameters\.

###### Proposition 4\(Normalized local\-policy interpretation\)\.

LetLθ​\(s,a\)=𝔼z~,j​\[ℓj​\(θ\)\]L\_\{\\theta\}\(s,a\)=\\mathbb\{E\}\_\{\\widetilde\{z\},j\}\[\\ell\_\{j\}\(\\theta\)\]and define the normalized local policy

qθ​\(a∣s\)=πθ¯​\(a∣s\)​exp⁡\{Lθ¯​\(s,a\)−Lθ​\(s,a\)\}Zθ​\(s\),Zθ​\(s\)=𝔼a∼πθ¯​exp⁡\{Lθ¯−Lθ\}\.q\_\{\\theta\}\(a\\mid s\)=\\frac\{\\pi\_\{\\bar\{\\theta\}\}\(a\\mid s\)\\exp\\\{L\_\{\\bar\{\\theta\}\}\(s,a\)\-L\_\{\\theta\}\(s,a\)\\\}\}\{Z\_\{\\theta\}\(s\)\},\\quad Z\_\{\\theta\}\(s\)=\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\bar\{\\theta\}\}\}\\exp\\\{L\_\{\\bar\{\\theta\}\}\-L\_\{\\theta\}\\\}\.\(20\)Thenqθ¯=πθ¯q\_\{\\bar\{\\theta\}\}=\\pi\_\{\\bar\{\\theta\}\}and, for any integrable,θ\\theta\-independentA⁡\(s,a\)A\(s,a\),

∇θ𝔼a∼qθ​\[A\]\|θ¯=−𝔼a∼πθ¯​\[\(A−𝔼⁡\[A∣s\]\)​∇θLθ\]θ¯,\\left\.\\nabla\_\{\\theta\}\\mathbb\{E\}\_\{a\\sim q\_\{\\theta\}\}\[A\]\\right\|\_\{\\bar\{\\theta\}\}=\-\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\bar\{\\theta\}\}\}\\left\[\(A\-\\mathbb\{E\}\[A\\mid s\]\)\\nabla\_\{\\theta\}L\_\{\\theta\}\\right\]\_\{\\bar\{\\theta\}\},\(21\)so with a conditionally centered advantage, Eq\.[8](https://arxiv.org/html/2610.28788#S4.E8)is the exact first\-order policy gradient of a normalized exponential tilt of the behavior policy\. The finite clipped weight in Eq\.[6](https://arxiv.org/html/2610.28788#S4.E6)inherits no exact likelihood\-ratio or trust\-region interpretation\.

For Proposition[4](https://arxiv.org/html/2610.28788#Thmproposition4), differentiate

log⁡qθ​\(a∣s\)=log⁡πθ¯​\(a∣s\)\+Lθ¯​\(s,a\)−Lθ​\(s,a\)−log⁡Zθ​\(s\)\.\\log q\_\{\\theta\}\(a\\mid s\)=\\log\\pi\_\{\\bar\{\\theta\}\}\(a\\mid s\)\+L\_\{\\bar\{\\theta\}\}\(s,a\)\-L\_\{\\theta\}\(s,a\)\-\\log Z\_\{\\theta\}\(s\)\.\(22\)Atθ¯\\bar\{\\theta\},q=πθ¯q=\\pi\_\{\\bar\{\\theta\}\}and∇log⁡Z=−𝔼πθ¯​\[∇L\]\\nabla\\log Z=\-\\mathbb\{E\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\[\\nabla L\]\. Hence

∇𝔼qθ​\[A\]\|θ¯\\displaystyle\\left\.\\nabla\\mathbb\{E\}\_\{q\_\{\\theta\}\}\[A\]\\right\|\_\{\\bar\{\\theta\}\}=𝔼πθ¯​\[A⁡\{−∇L\+𝔼πθ¯​\[∇L∣s\]\}\]\\displaystyle=\\mathbb\{E\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\\left\[A\\\{\-\\nabla L\+\\mathbb\{E\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\[\\nabla L\\mid s\]\\\}\\right\]\(23\)=−𝔼πθ¯\[\(A−𝔼\[A∣s\]\)∇L\],\\displaystyle=\-\\mathbb\{E\}\_\{\\pi\_\{\\bar\{\\theta\}\}\}\\left\[\(A\-\\mathbb\{E\}\[A\\mid s\]\)\\nabla L\\right\],\(24\)which proves Eq\.[21](https://arxiv.org/html/2610.28788#A3.E21)\.

#### Energy scale\.

Forℓ=w​D\\ell=wD, writeΔ^D\\widehat\{\\Delta\}\_\{D\}for the probed old\-minus\-new residual difference\. The local update is proportional to−w𝔼\[A^∇D\]\-w\\,\\mathbb\{E\}\[\\widehat\{A\}\\nabla D\], while the uncapped likelihood\-like weight isexp⁡\(w​Δ^D\)\\exp\(w\\widehat\{\\Delta\}\_\{D\}\)\. Thus changingwwchanges both the local gradient scale and when the numerical ratio reaches the PPO clipping thresholds\. A fixedw=1/4w=1/4keeps that scale independent of the collecting budget\. The matched\-scale controls isolate this choice in Appendix[E\.1](https://arxiv.org/html/2610.28788#A5.SS1)\.

### C\.4Cross\-budget bound

ForGK=\{1−j/K:j=0,…,K−1\}G\_\{K\}=\\\{1\-j/K:j=0,\\ldots,K\-1\\\}andr⁡\(t\)=vθ​\(xt⋆,t∣s\)−ur\(t\)=v\_\{\\theta\}\(x\_\{t\}^\{\\star\},t\\mid s\)\-u, define

RK=\(1K​∑t∈GK‖r⁡\(t\)‖2\)1/2,δK,K′=maxt′∈GK′⁡mint∈GK​\|t′−t\|\.R\_\{K\}=\\left\(\\frac\{1\}\{K\}\\sum\_\{t\\in G\_\{K\}\}\\\|r\(t\)\\\|^\{2\}\\right\)^\{1/2\},\\qquad\\delta\_\{K,K^\{\\prime\}\}=\\max\_\{t^\{\\prime\}\\in G\_\{K^\{\\prime\}\}\}\\min\_\{t\\in G\_\{K\}\}\|t^\{\\prime\}\-t\|\.\(25\)
###### Proposition 5\(Cross\-budget endpoint control\)\.

Assume the hypotheses of Proposition[3](https://arxiv.org/html/2610.28788#Thmproposition3)for aK′K^\{\\prime\}\-step deployment and additionally thatr⁡\(t\)r\(t\)isHH\-Lipschitz along the analytic bridge\. Then, withh′=1/K′h^\{\\prime\}=1/K^\{\\prime\},

‖x^K′−a‖≤CL,K′​\(K​RK\+H​δK,K′\),CL,K′=h′​∑j=0K′−1\(1\+h′​L\)K′−1−j\.\\\|\\widehat\{x\}\_\{K^\{\\prime\}\}\-a\\\|\\leq C\_\{L,K^\{\\prime\}\}\\bigl\(\\sqrt\{K\}R\_\{K\}\+H\\delta\_\{K,K^\{\\prime\}\}\\bigr\),\\quad C\_\{L,K^\{\\prime\}\}=h^\{\\prime\}\\sum\_\{j=0\}^\{K^\{\\prime\}\-1\}\(1\+h^\{\\prime\}L\)^\{K^\{\\prime\}\-1\-j\}\.\(26\)Hereδ4,1=0\\delta\_\{4,1\}=0andδ4,16=3/16\\delta\_\{4,16\}=3/16\.

For eacht′∈GK′t^\{\\prime\}\\in G\_\{K^\{\\prime\}\}, choose a nearestt∈GKt\\in G\_\{K\}\. The definition ofRKR\_\{K\}implies‖r⁡\(t\)‖≤K​RK\\\|r\(t\)\\\|\\leq\\sqrt\{K\}R\_\{K\}, while time regularity gives‖r⁡\(t′\)−r⁡\(t\)‖≤H​δK,K′\\\|r\(t^\{\\prime\}\)\-r\(t\)\\\|\\leq H\\delta\_\{K,K^\{\\prime\}\}\. Thus every deployment\-grid residual is at mostK​RK\+H​δK,K′\\sqrt\{K\}R\_\{K\}\+H\\delta\_\{K,K^\{\\prime\}\}\. Substitution into Eq\.[15](https://arxiv.org/html/2610.28788#A3.E15), withh′=1/K′h^\{\\prime\}=1/K^\{\\prime\}, proves Proposition[5](https://arxiv.org/html/2610.28788#Thmproposition5)\. LargeHHor a shifted reference bridge makes the bound loose, and it is not a reward theorem\. Throughout we keep Euler fixed: the endpoint results assert no invariance to changes of flow geometry, and state which residuals control one specified finite map\.

## Appendix DExperimental protocol and baselines

#### Control training and checkpoint protocol\.

The control run uses6464outer iterations, or eight passes through𝒮\\mathcal\{S\}, with eight cached bridge probes per action\. It evaluates the initialization and each cycle\-end checkpoint at𝒦=\{1,4,16\}\\mathcal\{K\}=\\\{1,4,16\\\}\. At each checkpoint and budget,6464evaluation episodes split into3232for selection and3232for disjoint reporting\. The selection half applies Eq\.[9](https://arxiv.org/html/2610.28788#S4.E9); exact ties choose the earliest checkpoint\. The report half cannot affect selection and supplies the selected checkpoint’s reported return\. On tasks with1,0001\{,\}000\-step episodes, the nine evaluations add1,728,0001\{,\}728\{,\}000environment steps, or2\.75%2\.75\\%of the62,914,56062\{,\}914\{,\}560\-interaction training budget\. This is an empirical deployment rule, not a formal robustness certificate\.

#### Common\-stack comparison\.

FPO is the control backbone\. FPO\+\+, DPPO, ReinFlow, and PolicyFlow are adaptations to the same architecture, initialization, critic, and interaction budget\. Their official releases use other tasks and training regimes\. DPPO originally trains a diffusion policy: it treats stochastic denoising transitions as an inner MDP and applies policy gradients to their likelihood ratios\([Ren et al\., 2025](https://arxiv.org/html/2610.28788#bib.bib10)\)\. ReinFlow compares against that diffusion\-policy implementation using separately pretrained actors\([Zhang et al\., 2025b](https://arxiv.org/html/2610.28788#bib.bib5)\)\. Our row labeled DPPO instead retains the shared flow velocity field and applies the denoising\-MDP objective to Gaussian\-noised Euler transitions,zj\+1\|zj,o∼𝒩⁡\(zj\+Δ​tj​vθ​\(zj,tj,o\),σj2​I\)z\_\{j\+1\}\\mid z\_\{j\},o\\sim\\mathcal\{N\}\(z\_\{j\}\+\\Delta t\_\{j\}v\_\{\\theta\}\(z\_\{j\},t\_\{j\},o\),\\sigma\_\{j\}^\{2\}I\); the PPO ratios use the corresponding transition log densities\. Thus it tests the DPPO update on a flow backbone, rather than reproducing the original diffusion actor or its published results\. The DM Control comparison starts every arm from a shared random initialization, whereas the original DPPO and ReinFlow studies fine\-tune behavior priors\. The native image\-based ReinFlow study is therefore reported separately\. The following distinctions define the baseline rows, rather than implying a reproduction of each paper’s native benchmark results\. Table[4](https://arxiv.org/html/2610.28788#A4.T4)summarizes these adaptations\.

Table 4:Implementation scope of the shared\-backbone baselines\.

## Appendix EAdditional controls

Table[5\.2](https://arxiv.org/html/2610.28788#S5.SS2.SSS0.Px2)reports the current component ladder\. This appendix adds the collecting\-grid isolation used to distinguish energy scale from node placement\.

### E\.1Energy scale and node placement

The collecting\-grid controls further separate the time nodes from their weight\. Using weight1/K1/Kmakes the energy scale depend on the collecting solver; holding it at1/41/4removes that dependence\.

Figure 4:A fixed energy scale stabilizes the numerical weight across collecting budgets\.Clip fraction, log\-weight spread, and bridge energy versus collectingKK; medians and quartiles over the selection\-pool iterations\. The collecting\-grid1/K1/Kweight changes with budget; both fixed\-1/41/4controls keep the energy scale comparable\.Figure[4](https://arxiv.org/html/2610.28788#A5.F4)shows that weight1/K1/Kmakes the bridge energy fall as the collecting budget increases, whereas the fixed\-1/41/4variants keep bridge energy and log\-weight spread on comparable scales\. This is a scale diagnostic; it does not by itself compare episodic return\.