Reinforcement Learning via Fixed-point Flow Matching

# Reinforcement Learning via Fixed-point Flow Matching
Source: [https://arxiv.org/html/2610.12848](https://arxiv.org/html/2610.12848)
Onur CelikGerhard NeumannAffiliation:Karlsruhe Institute of Technology \(KIT\)Affiliation:Autonomous Learning Robots Lab \(ALR\)

###### Abstract

In maximum entropy and behavior\-constrained reinforcement learning \(RL\), the optimal policy is given by a Boltzmann distribution\. Generating samples from this distribution can be challenging, as it is often complex and multimodal\. While flow policies trained with flow matching provide a powerful framework for representing such distributions, they traditionally rely on access to training data, which is generally unavailable in online RL\. To address this limitation, we propose fixed\-point flow matching \(FFM\), a data\-free procedure for training and fine\-tuning flow policies in RL\. FFM leverages the target flow identity to formulate the actor update as a fixed\-point iteration, utilizing a flow\-matching\-style regression objective where the critic’s action gradients serve as regression targets\. We show that the optimal policy is a fixed point of this iteration\. Through extensive experiments, we demonstrate that FFM effectively fine\-tunes pretrained flow policies in the offline\-to\-online setting and enables efficient and accurate training of flow policies from scratch in online RL\. These results establish fixed\-point flow matching as a principled approach for learning expressive flow\-based policies without requiring training data\.

## 1Introduction

Across a range of reinforcement learning \(RL\) formulations, the improved policy is available in closed form as a Boltzmann distribution of the current critic\. In maximum entropy RL,π⋆​\(x∣s\)∝exp⁡\(Q⁡\(s,x\)/λ\)\\pi^\{\\star\}\(x\\mid s\)\\propto\\exp\(Q\(s,x\)/\\lambda\)is optimal for the policy improvement step of soft policy iteration\([Ziebart et al\., 2008](https://arxiv.org/html/2610.12848#bib.bib104);[Haarnoja et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib38);[Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39)\); in KL\-regularized and behavior\-constrained RL, which includes offline\-to\-online fine\-tuning, the same expression appears tilted by a reference policy\([Peters et al\., 2010](https://arxiv.org/html/2610.12848#bib.bib68);[Abdolmaleki et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib2);[Peng et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib67);[Nair et al\., 2020](https://arxiv.org/html/2610.12848#bib.bib62)\)\. This target distribution can be complex and multimodal whenever several distinct action choices are viable, but the classical actor–critic recipe projects it onto a Gaussian due to the simplicity of the resulting optimization problem and the tractability of the policy’s log\-likelihood\([Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39)\)\. Flow and diffusion models\([Lipman et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib52);[Liu et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib55);[Albergo & Vanden\-Eijnden, 2023](https://arxiv.org/html/2610.12848#bib.bib7);[Ho et al\., 2020](https://arxiv.org/html/2610.12848#bib.bib42);[Song et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib84)\)remove this representational restriction and have become the policy class of choice wherever a dataset is available\([Chi et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib16);[Reuss et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib73);[Janner et al\., 2022](https://arxiv.org/html/2610.12848#bib.bib44);[Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\)\. Yet, the stable training objectives that are used to train these models are not directly usable for online RL due to a lack of samples fromπ⋆\\pi^\{\\star\}\. Existing methods for training flow and diffusion models in the on\-policy RL setting often require backpropagating the critic through the integrator\([Wang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib93);[Ding & Jin, 2024](https://arxiv.org/html/2610.12848#bib.bib21);[Wang et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib92);[Zhang et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib101);[Celik et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib11);[Le et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib47)\), whose memory cost grows with every integration step and quickly becomes prohibitive, or rely on importance\-weighted regression\([Ding et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib19);[Ma et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib59);[Dong et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib25)\), whose exponential critic weights have high variance and lead to unstable training updates\. Moreover, methods for fine\-tuning flow models typically use stochastic optimal control, which requires converting the flow into a diffusion model in order to use the adjoint method\([Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib23);[Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49);[Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\)\. The adjoint method stores each sampled trajectory and integrates an adjoint state backward along it, leading to complex and expensive update schemes\.

1Generate2Label3InterpolateX0∼𝒩⁡\(0,σ02​I\)X\_\{0\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{0\}^\{2\}I\)X1∼πθ\(⋅\|s\)X\_\{1\}\\sim\\pi\_\{\\theta\}\(\\cdot\\,\|\\,s\)vθv\_\{\\theta\}d​Xt=vθ​\(Xt,t\|s\)​d​t\\mathrm\{d\}X\_\{t\}=v\_\{\\theta\}\(X\_\{t\},t\\,\|\\,s\)\\,\\mathrm\{d\}tQϕ​\(s,⋅\)Q\_\{\\phi\}\(s,\\cdot\)∇xQϕ​\(s,X1\)/λ\\nabla\_\{x\}Q\_\{\\phi\}\(s,X\_\{1\}\)/\\lambdaε\\varepsilonX1X\_\{1\}XtX\_\{t\}Xt=α⁡\(t\)​X1\+β⁡\(t\)​εX\_\{t\}=\\alpha\(t\)\\,X\_\{1\}\+\\beta\(t\)\\,\\varepsilonX1X\_\{1\}∇xQϕ\\nabla\_\{x\}Q\_\{\\phi\}4Regressminθ⁡𝔼​‖vθ​\(Xt,t\|s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)λ​α​\(t\)​∇xQϕ​\(s,X1\)‖2\\min\_\{\\theta\}\\ \\mathbb\{E\}\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\,\|\\,s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\lambda\\alpha\(t\)\}\\nabla\_\{x\}Q\_\{\\phi\}\(s,X\_\{1\}\)\\big\\\|^\{2\}

Figure 1:The FFM policy\-improvement step\.FFM repeats:\(1\)Clean actionsX1X\_\{1\}are generated from the current policy by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)\(forward only, no gradients\); the grey curves are sampling trajectories\.\(2\)Critic action gradients are required for the FFM regression target of[Equation12](https://arxiv.org/html/2610.12848#S4.E12)\.\(3\)Fresh noiseε\\varepsilonand the clean actionX1X\_\{1\}are used to sample a noised actionXtX\_\{t\}via[Equation3](https://arxiv.org/html/2610.12848#S3.E3)\.\(4\)A flow\-matching\-style regression pullsvθv\_\{\\theta\}toward the FFM target\. The optimal policyπ⋆∝eQ/λ\\pi^\{\\star\}\\propto e^\{Q/\\lambda\}is a fixed point of this loop\.In this work we ask whether a flow policy can be trained against a Boltzmann target with the same simple regression that makes flow matching stable, but without samples from the target\. Our starting point is the*target flow identity*: for a Boltzmann target, the velocity field that generates the noised target is, up to a rescaling of the noisy action, a conditional average of the critic’s action gradients∇xQ​\(s,x\)\\nabla\_\{x\}Q\(s,x\)at clean actions\. It replaces the data\-derived regression target of flow matching by a critic\-derived one, and the resulting objective,*target flow matching*, has the same minimizer as flow matching while reading its targets off the critic instead of a dataset\. Since the objective still averages over samples of the target, we evaluate it with samples of the current policy and iterate\. The resulting*fixed\-point flow matching*\(FFM\) update, illustrated in[Figure1](https://arxiv.org/html/2610.12848#S1.F1), generates actions with the current flow, labels them with critic gradients, noises them, and regresses the velocity field onto the labels\. We show that the optimal policy is a fixed point of this iteration\. The whole procedure is conceptually simple, easy to implement and requires neither backpropagation through the integrator nor importance weights or conversion into a diffusion model\.

##### Contributions\.

- •The target flow identity and target flow matching\([Section4](https://arxiv.org/html/2610.12848#S4)\)\. We derive an identity that expresses the velocity field of a Boltzmann target as a conditional expectation of the gradient of its log\-density, and turn it into target flow matching, an alternative training objective to flow matching that shares its population minimizer but regresses onto the gradient of the target rather than onto the displacement of the data\.
- •Fixed\-point flow matching\([Section4](https://arxiv.org/html/2610.12848#S4)\)\. We propose a fixed\-point iteration that evaluates the target flow matching objective with samples of the current policy instead of the optimal one, and prove that the optimal policy is a fixed point of this iteration \(Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)\)\.
- •Extensions\([Section5](https://arxiv.org/html/2610.12848#S5)\)\. We further introduce several extensions for flow policies: a trust\-region loss in velocity space, a loss for behavior\-constrained fine\-tuning that reads the reference score off the pretrained flow, and correlated exploration through temporally correlated prior draws\.
- •Two RL instantiations\([Section5](https://arxiv.org/html/2610.12848#S5)\)\. We instantiate FFM for maximum entropy RL and for offline\-to\-online fine\-tuning\. We validate both across5050offline\-to\-online tasks and2424online tasks \([Section6](https://arxiv.org/html/2610.12848#S6)\), where FFM is competitive with or better than the strongest available baselines\.

## 2Related work

##### Reinforcement learning with diffusion and flow policies\.

Diffusion and flow models entered RL through imitation learning and offline RL, where a dataset makes their standard training objectives applicable and the critic enters through weighting, guidance, or gradients through the sampling chain\([Janner et al\., 2022](https://arxiv.org/html/2610.12848#bib.bib44);[Chi et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib16);[Wang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib93);[Chen et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib12);[Hansen\-Estruch et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib40);[Lu et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib56);[Kang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib45);[Chen et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib13);[Ding & Jin, 2024](https://arxiv.org/html/2610.12848#bib.bib21);[Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\)\. In online RL there are no samples of the target policy, and existing methods differ in how the critic drives the actor update\. One line backpropagates the critic value through the sampling chain\([Wang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib93);[Ding & Jin, 2024](https://arxiv.org/html/2610.12848#bib.bib21);[Wang et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib92);[Zhang et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib101)\), which is memory\-intensive and unstable for many integration steps\. A second line uses critic\-weighted regression, interpreting the denoising loss as a variational bound or as a mirror\-descent step\([Ding et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib19);[Ma et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib59);[Dong et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib25)\), and a third estimates policy gradients through the denoising process, typically under a trust region\([Ren et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib72);[McAllister et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib60);[Zhang et al\., 2025a](https://arxiv.org/html/2610.12848#bib.bib100);[Le et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib47);[Lv et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib58)\)\. Closest to FFM are methods that use the action gradient of the critic: DIPO\([Yang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib97)\)clones actions improved by gradient ascent onQQ; QSM\([Psenka et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib70)\)regresses the score onto∇xQ\\nabla\_\{x\}Qevaluated at the noisy action, a plug\-in approximation of the target score identity that is exact only at the clean end; and RFM\([Li et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib51)\)estimates the exact posterior\-mean target by self\-normalized importance sampling, using Langevin–Stein control variates to interpolate betweenQQ\-weighted noise averages andQQ\-weighted averages of∇xQ\\nabla\_\{x\}Q, which allows it to train flow policies as well\. Maximum entropy RL with generative policies additionally requires an entropy estimate: DACER\([Wang et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib92)\)fits a Gaussian mixture, DIME\([Celik et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib11)\)and MaxEntDP\([Dong et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib25)\)use a lower bound and numerical integration, respectively,[Sanokowski & Patil \(2025\)](https://arxiv.org/html/2610.12848#bib.bib76)augment the MDP with the denoising steps, FLAG\([Kim et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib46)\)uses importance sampling in the latent space of the flow, and AMDP\([Thilges et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib85)\)amortizes the entropy\-regularized update with adjoint matching\. FFM obtains the entropy from the divergence of the velocity field it trains, using the instantaneous change of variables\.

##### Fine\-tuning and offline\-to\-online RL\.

Reward fine\-tuning of pretrained diffusion and flow models has been cast as policy gradient over the denoising chain\([Fan et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib31);[Black et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib8);[Liu et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib54)\)and as entropy\-regularized stochastic optimal control\([Uehara et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib86);[Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib23)\), with the KL divergence to the pretrained model as regularizer;[Uehara et al\. \(2025\)](https://arxiv.org/html/2610.12848#bib.bib87)review the corresponding inference\-time methods\. The same behavior\-tilted target governs offline\-to\-online RL with flow policies\. FQL\([Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\)distills the flow into a one\-step policy that is trained by reparameterized critic gradients, DSRL\([Wagenmaker et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib91)\)keeps the flow fixed and learns a policy over its latent noise, ReinFlow and FINO\([Zhang et al\., 2025a](https://arxiv.org/html/2610.12848#bib.bib100);[Shin et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib81)\)inject noise into the flow to obtain tractable likelihoods and exploration, and Q\-Flow\([Doo et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib28)\)propagates terminal values to intermediate states of the flow\. QAM\([Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49)\)imports adjoint matching: the pretrained flow is converted into a memoryless SDE and critic gradients are transported backward along sampled trajectories by the lean adjoint ODE, and TRQAM\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\)stabilizes it with a path\-space trust region\. Concurrent work regresses velocity corrections onto critic gradients\([Wang et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib94);[Shi et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib79);[Ziakas et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib103)\)or steers a fixed flow policy at test time\([Zhou et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib102);[Ruan et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib75)\)\. FFM fine\-tunes the pretrained flow directly in velocity space, requires neither SDE conversion nor adjoint simulation, and its behavior\-tilted target has the KL\-regularized optimum as a fixed point \([AppendixE](https://arxiv.org/html/2610.12848#A5)\)\.

Beyond the RL literature, FFM shares the fixed\-point structure of the matching objectives used to train diffusion and flow models from unnormalized densities, and of adjoint matching;[AppendixC](https://arxiv.org/html/2610.12848#A3)discusses these methods, together with further critic\-driven policy updates\.

## 3Preliminaries

##### Reinforcement learning\.

We consider a Markov decision process with statess∈𝒮s\\in\\mathcal\{S\}, continuous actionsx∈𝒜⊆ℝdx\\in\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{d\}, transitionsP⁡\(s′∣s,x\)P\(s^\{\\prime\}\\mid s,x\), rewardr⁡\(s,x\)r\(s,x\), and discountγ∈\[0,1\)\\gamma\\in\[0,1\)\. The state\-action value function of the current policyπ\\piis the expected discounted returnQπ\(s,x\)=𝔼π\[∑k≥0γkr\(sk,xk\)∣s0=s,x0=x\]Q^\{\\pi\}\(s,x\)=\\mathbb\{E\}\_\{\\pi\}\\big\[\\sum\_\{k\\geq 0\}\\gamma^\{k\}r\(s\_\{k\},x\_\{k\}\)\\mid s\_\{0\}=s,\\,x\_\{0\}=x\\big\]when taking actionxxin statessand followingπ\\pithereafter\. Maximizing this return alone drives the policy towards a deterministic optimum, which often suffers from premature convergence\. The two settings we consider therefore both regularize the policy improvement step, which leads to closed\-form solutions that are known only up to an intractable normalization constant\.

In*behavior\-constrained*RL\([Peters et al\., 2010](https://arxiv.org/html/2610.12848#bib.bib68);[Abdolmaleki et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib2);[Peng et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib67);[Nair et al\., 2020](https://arxiv.org/html/2610.12848#bib.bib62)\), which includes offline\-to\-online fine\-tuning, the improvement step is penalized for deviating from a reference policyprefp\_\{\\mathrm\{ref\}\}, typically a policy pretrained on the offline dataset\. Maximizing𝔼x∼π\[Q\(s,x\)\]−λDKL\(π\(⋅∣s\)∥pref\(⋅∣s\)\)\\mathbb\{E\}\_\{x\\sim\\pi\}\[Q\(s,x\)\]\-\\lambda\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid s\)\\,\\\|\\,p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid s\)\\right\)over all policies yields the*behavior\-tilted*optimum

π⋆​\(x∣s\)=pref​\(x∣s\)​exp⁡\(Q⁡\(s,x\)/λ\)𝒵⁡\(s\),𝒵⁡\(s\)=∫𝒜pref​\(x∣s\)​exp⁡\(Q⁡\(s,x\)/λ\)​𝑑x,\\pi^\{\\star\}\(x\\mid s\)\\;=\\;\\frac\{p\_\{\\mathrm\{ref\}\}\(x\\mid s\)\\exp\\\!\\big\(Q\(s,x\)/\\lambda\\big\)\}\{\\mathcal\{Z\}\(s\)\},\\qquad\\mathcal\{Z\}\(s\)=\\int\_\{\\mathcal\{A\}\}p\_\{\\mathrm\{ref\}\}\(x\\mid s\)\\exp\\\!\\big\(Q\(s,x\)/\\lambda\\big\)\\,\\mathrm\{d\}x,\(1\)whereλ\>0\\lambda\>0trades off the penalty for deviating fromprefp\_\{\\mathrm\{ref\}\}against maximizing the expected return\.

In*maximum entropy*RL, the reference is dropped and the rewards are instead augmented by the entropy of the current policyℋ\(π\(⋅∣s\)\)=−𝔼x∼π\[logπ\(x∣s\)\]\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\)\)=\-\\mathbb\{E\}\_\{x\\sim\\pi\}\[\\log\\pi\(x\\mid s\)\], which leads to the soft state\-action value functionQπ\(s,x\)=𝔼π\[∑k≥0γk\(r\(sk,xk\)\+λℋ\(π\(⋅∣sk\)\)\)∣s0=s,x0=x\]Q^\{\\pi\}\(s,x\)=\\mathbb\{E\}\_\{\\pi\}\\big\[\\sum\_\{k\\geq 0\}\\gamma^\{k\}\\big\(r\(s\_\{k\},x\_\{k\}\)\+\\lambda\\,\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{k\}\)\)\\big\)\\mid s\_\{0\}=s,\\,x\_\{0\}=x\\big\]\. The improvement step of soft policy iteration is the*Boltzmann policy*\([Ziebart et al\., 2008](https://arxiv.org/html/2610.12848#bib.bib104);[Haarnoja et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib38);[Levine, 2018](https://arxiv.org/html/2610.12848#bib.bib48);[Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39)\)

π⋆​\(x∣s\)=exp⁡\(Q⁡\(s,x\)/λ\)𝒵⁡\(s\),𝒵⁡\(s\)=∫𝒜exp⁡\(Q⁡\(s,x\)/λ\)​𝑑x,\\pi^\{\\star\}\(x\\mid s\)\\;=\\;\\frac\{\\exp\\\!\\big\(Q\(s,x\)/\\lambda\\big\)\}\{\\mathcal\{Z\}\(s\)\},\\qquad\\mathcal\{Z\}\(s\)=\\int\_\{\\mathcal\{A\}\}\\exp\\\!\\big\(Q\(s,x\)/\\lambda\\big\)\\,\\mathrm\{d\}x,\(2\)with a temperatureλ\>0\\lambda\>0, which now trades off maximizing the expected return against the policy’s stochasticity\.

In both cases, the normalization constant𝒵⁡\(s\)\\mathcal\{Z\}\(s\)is intractable, and generating samples from the potentially complex and multimodal targetπ⋆\\pi^\{\\star\}requires fitting a tractable policy parameterization to its density\. FFM optimizes a flow\-based policy to this end\.

##### Flow policies from flow matching\.

Assume for now that we have access to samplesX1∼π⋆\(⋅∣s\)X\_\{1\}\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)from the optimal policy\. By drawing independent Gaussian noiseε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\), one can construct a stochastic interpolant\([Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\)as

Xt=α⁡\(t\)​X1\+β⁡\(t\)​ε,t∈\[0,1\],X\_\{t\}\\;=\\;\\alpha\(t\)\\,X\_\{1\}\\;\+\\;\\beta\(t\)\\,\\varepsilon,\\qquad t\\in\[0,1\],\(3\)with smooth schedules satisfyingα⁡\(0\)=0\\alpha\(0\)=0,α⁡\(1\)=1\\alpha\(1\)=1,β⁡\(0\)=σ0\\beta\(0\)=\\sigma\_\{0\},β⁡\(1\)=0\\beta\(1\)=0\(e\.g\.α⁡\(t\)=t\\alpha\(t\)=t,β⁡\(t\)=σ0​\(1−t\)\\beta\(t\)=\\sigma\_\{0\}\(1\-t\)\)\. Hence,X0∼𝒩⁡\(⋅,0,σ02​I\)X\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\)is pure noise,X1X\_\{1\}is a clean action fromπ⋆\(⋅∣s\)\\pi^\{\\star\}\(\\cdot\\mid s\), andXtX\_\{t\}in between is a partially noised action with marginal densityρt\\rho\_\{t\}\. The interpolant defines thevelocity field

v⁡\(x,t∣s\):=𝔼X1∼π⋆,ε​\[X˙t\|Xt=x\],X˙t=α˙​\(t\)​X1\+β˙​\(t\)​ε,v\(x,t\\mid s\)\\;:=\\;\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}\\big\[\\dot\{X\}\_\{t\}\\,\\big\|\\,X\_\{t\}=x\\big\],\\qquad\\dot\{X\}\_\{t\}=\\dot\{\\alpha\}\(t\)\\,X\_\{1\}\+\\dot\{\\beta\}\(t\)\\,\\varepsilon,\(4\)which represents the mean direction in which interpolating actions move\. Here and in the following, the subscript of the expectation lists the random variables of the interpolant it is taken over:𝔼X1∼π⋆,ε\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}is the expectation over the pair\(X1,ε\)\(X\_\{1\},\\varepsilon\)of[Equation3](https://arxiv.org/html/2610.12848#S3.E3)with clean actionsX1∼π⋆\(⋅∣s\)X\_\{1\}\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\. Conditioning onXt=xX\_\{t\}=xhence averages over all pairs\(X1,ε\)\(X\_\{1\},\\varepsilon\)that are consistent with the noised actionxxat timett\. Integrating the ordinary differential equation \(ODE\)

d​Xt=v⁡\(Xt,t∣s\)​d​t,X0∼𝒩⁡\(⋅,0,σ02​I\),\\mathrm\{d\}X\_\{t\}\\;=\\;v\(X\_\{t\},t\\mid s\)\\,\\mathrm\{d\}t,\\quad X\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\),\(5\)transports the prior along the marginalsρt\\rho\_\{t\}, so that the terminal samplesX1X\_\{1\}are distributed according toπ⋆\(⋅∣s\)\\pi^\{\\star\}\(\\cdot\\mid s\)\([Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\)\. To learn the velocity, one can use*flow matching*\([Lipman et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib52);[Liu et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib55);[Albergo & Vanden\-Eijnden, 2023](https://arxiv.org/html/2610.12848#bib.bib7)\), which provides a simple and scalable regression objective, i\.e\.,

ℒFM​\(θ\)=𝔼t,X1∼π⋆,ε​\[‖vθ​\(Xt,t∣s\)−X˙t‖2\],\\mathcal\{L\}\_\{\\mathrm\{FM\}\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}\\big\[\\,\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\dot\{X\}\_\{t\}\\big\\\|^\{2\}\\,\\big\],\(6\)where the expectation is additionally taken overt∼𝒰⁡\(0,1\)t\\sim\\mathcal\{U\}\(0,1\)\. Optimizing[Equation6](https://arxiv.org/html/2610.12848#S3.E6), however, requires samples from the optimal policy, which are not available in online RL\. Instead, one typically has access to a criticQQ, and in the next section we leverage it to obtain a similar regression objective that can be optimized without samples fromπ⋆\\pi^\{\\star\}\.

## 4Fixed\-point target flow matching

Now assume we are given a critic functionQQwithπ⋆\(⋅∣s\)∝exp\(Q\(s,⋅\)/λ\)\\pi^\{\\star\}\(\\cdot\\mid s\)\\propto\\exp\(Q\(s,\\cdot\)/\\lambda\)instead of samplesX1X\_\{1\}\. Furthermore, assume that the critic is differentiable, allowing us to compute theQQ\-score

∇x​log​π⋆​\(x∣s\)=∇xQ​\(s,x\)/λ,\\nabla\_\{x\}\\log\\pi^\{\\star\}\(x\\mid s\)\\;=\\;\\nabla\_\{x\}Q\(s,x\)/\\lambda,\(7\)which does not require knowledge of the intractable normalization constant in[Equation2](https://arxiv.org/html/2610.12848#S3.E2)\. We arrive at the proposed method in two steps\. First, we introduce thetarget flow identity, which expresses the velocity as a conditional expectation, similar to flow matching, but in terms of theQQ\-score\. The corresponding objective,target flow matching, still requires samplesX1∼π⋆X\_\{1\}\\sim\\pi^\{\\star\}\. This is addressed in the second step, where we introducefixed\-point flow matching \(FFM\), which uses samples of the current policy instead of the optimal one, resulting in a fixed\-point iteration with theoretical guarantees\.

##### The target flow identity\.

Consider the interpolant in[Equation3](https://arxiv.org/html/2610.12848#S3.E3)and define

κ⁡\(t\):=α˙​\(t\)α⁡\(t\)​β2​\(t\)−β˙​\(t\)​β​\(t\)\>0on​\(0,1\),\\kappa\(t\)\\;:=\\;\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\beta^\{2\}\(t\)\-\\dot\{\\beta\}\(t\)\\beta\(t\)\\;\>\\;0\\quad\\text\{on \}\(0,1\),\(8\)which recurs throughout the paper\. The following proposition provides an alternative to flow matching \(see[Equation6](https://arxiv.org/html/2610.12848#S3.E6)\) for representing the velocity field\.

###### Proposition 1\(Target flow identity\)\. Letπ⋆\(⋅∣s\)∝exp\(Q\(s,⋅\)/λ\)\\pi^\{\\star\}\(\\cdot\\mid s\)\\propto\\exp\(Q\(s,\\cdot\)/\\lambda\)with𝒵⁡\(s\)<∞\\mathcal\{Z\}\(s\)<\\infty, letQ⁡\(s,⋅\)Q\(s,\\cdot\)be continuously differentiable with𝔼X1∼π⋆​\[‖∇xQ​\(s,X1\)‖\]<∞\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi^\{\\star\}\}\\big\[\\\|\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\\|\\big\]<\\infty, and letvvbe the velocity field[Equation4](https://arxiv.org/html/2610.12848#S3.E4)of the interpolant[Equation3](https://arxiv.org/html/2610.12848#S3.E3)withX1∼π⋆\(⋅∣s\)X\_\{1\}\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\. Then, for allt∈\(0,1\]t\\in\(0,1\],v⁡\(x,t∣s\)=α˙​\(t\)α⁡\(t\)​x\+κ⁡\(t\)λ​α​\(t\)​𝔼X1∼π⋆,ε​\[∇xQ​\(s,X1\)\|Xt=x\]\.v\(x,t\\mid s\)\\;=\\;\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,x\\;\+\\;\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}\\big\[\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\,\\big\|\\,X\_\{t\}=x\\big\]\.\(9\)

Intuitively, the velocity field that generates the Boltzmann policy is, up to a rescaling of the current noisy action, a conditional average ofQQ\-scores at clean actions\. The proof of Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)is given in[AppendixA](https://arxiv.org/html/2610.12848#A1)and uses the fact that the velocity can be expressed in terms of the score for Gaussian source distributions in combination with the target score identity\([De Bortoli et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib17)\)\. Note that the factor1/α⁡\(t\)1/\\alpha\(t\)in[Equation9](https://arxiv.org/html/2610.12848#S4.E9)diverges ast→0t\\to 0, whereXtX\_\{t\}is almost pure noise and theQQ\-score at the clean actionX1X\_\{1\}carries little information about it\. The conditional expectation itself remains finite, but a regression target built from[Equation9](https://arxiv.org/html/2610.12848#S4.E9)has large variance near the noise end\. In[SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2)we show that mixing the target with the flow\-matching target using a variance\-optimal control variate removes this singularity\.

Because theL2L^\{2\}\-optimal predictor is the conditional expectation,[Equation9](https://arxiv.org/html/2610.12848#S4.E9)can directly be turned into a regression objective as stated below\.

###### Corollary 2\(Target flow matching\)\. Drawt∼𝒰⁡\(0,1\)t\\sim\\mathcal\{U\}\(0,1\),X1∼π⋆\(⋅∣s\)X\_\{1\}\\sim\\pi^\{\\star\}\(\\cdot\\mid s\),ε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)and formXtX\_\{t\}by[Equation3](https://arxiv.org/html/2610.12848#S3.E3), and letwwbe a time\-dependent weighting withw⁡\(t\)\>0w\(t\)\>0on\(0,1\)\(0,1\)\. The*target flow matching*objectiveℒTFM​\(θ\)=𝔼t,X1∼π⋆,ε​\[w⁡\(t\)​‖vθ​\(Xt,t∣s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)λ​α​\(t\)​∇xQ​\(s,X1\)‖2\]\\mathcal\{L\}\_\{\\mathrm\{TFM\}\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}\\Big\[w\(t\)\\,\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\big\\\|^\{2\}\\Big\]\(10\)shares its population minimizer with flow matching,ℒFM\\mathcal\{L\}\_\{\\mathrm\{FM\}\}of[Equation6](https://arxiv.org/html/2610.12848#S3.E6)\.

In contrast to flow matching, whose targetX˙t\\dot\{X\}\_\{t\}encodes where the data moves, the target of target flow matching encodes where the critic points\. However, the main obstacle remains: the expectation in[Equation10](https://arxiv.org/html/2610.12848#S4.E10)still requires samplesX1∼π⋆X\_\{1\}\\sim\\pi^\{\\star\}\.

##### Fixed\-point flow matching\.

While we cannot sample fromπ⋆\\pi^\{\\star\}, we can sampleX1∼πθ\(⋅∣s\)X\_\{1\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\), whereπθ\\pi\_\{\\theta\}denotes the terminal density att=1t=1obtained by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)with the learned velocity fieldvθv\_\{\\theta\}in place ofvv\. In other words, we can sample from the current policy instead of the optimal one\. FFM leverages this fact by building the interpolant in[Equation10](https://arxiv.org/html/2610.12848#S4.E10)from the policy’s own samples\. To this end, consider the loss where the interpolant is built from clean actions of an arbitrary flow policyπθ¯\\pi\_\{\\bar\{\\theta\}\}with parametersθ¯\\bar\{\\theta\}, that is,

ℒFFM​\(θ,θ¯\)=𝔼t,X1∼πθ¯,ε​\[w⁡\(t\)​‖vθ​\(Xt,t∣s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)λ​α​\(t\)​∇xQ​\(s,X1\)‖2\],\\mathcal\{L\}\_\{\\mathrm\{FFM\}\}\(\\theta,\\bar\{\\theta\}\)=\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\},\\,\\varepsilon\}\\Big\[w\(t\)\\,\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\big\\\|^\{2\}\\Big\],\(11\)where the expectation is again over the interpolant[Equation3](https://arxiv.org/html/2610.12848#S3.E3), now withX1∼πθ¯\(⋅∣s\)X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid s\)instead ofX1∼π⋆\(⋅∣s\)X\_\{1\}\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\. Given some initial parametersθ0\\theta\_\{0\}, FFM computes the updates

θk\+1=arg​minθ⁡ℒFFM​\(θ,θk\),\\theta\_\{k\+1\}\\;=\\;\\argmin\_\{\\theta\}\\;\\mathcal\{L\}\_\{\\mathrm\{FFM\}\}\(\\theta,\\theta\_\{k\}\),\(12\)which has the optimal policy as a fixed point, as stated below\.

###### Theorem 3\(Fixed point\)\.

Consider the iteration[Equation12](https://arxiv.org/html/2610.12848#S4.E12)in its population form, that is, with the regression solved exactly over all velocity fields\. Under regularity conditions on the criticQQthat we state in[AppendixA](https://arxiv.org/html/2610.12848#A1), if the current policy is optimal,πθk\(⋅∣s\)=π⋆\(⋅∣s\)\\pi\_\{\\theta\_\{k\}\}\(\\cdot\\mid s\)=\\pi^\{\\star\}\(\\cdot\\mid s\), then the update returns the velocity field[Equation4](https://arxiv.org/html/2610.12848#S3.E4)ofπ⋆\\pi^\{\\star\}, so thatπθk\+1\(⋅∣s\)=π⋆\(⋅∣s\)\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\\mid s\)=\\pi^\{\\star\}\(\\cdot\\mid s\)and every subsequent update returns the same velocity field\. The optimal policy is therefore a fixed point of FFM\.

The proof is given in[AppendixA](https://arxiv.org/html/2610.12848#A1)\. In practice, FFM repeats the following steps\. First, clean actionsX1∼πθk\(⋅∣s\)X\_\{1\}\\sim\\pi\_\{\\theta\_\{k\}\}\(\\cdot\\mid s\)are generated by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)with the current velocity field \(forward passes only, no gradients\)\. Next, theQQ\-scores∇xQ​\(s,X1\)\\nabla\_\{x\}Q\(s,X\_\{1\}\)are computed, which can be stored in a replay buffer for sample reuse\. Then, fresh noiseε\\varepsilonand timesttare drawn to form the noised actionsXtX\_\{t\}, on which the regression loss[Equation11](https://arxiv.org/html/2610.12848#S4.E11)is minimized; see[Figure1](https://arxiv.org/html/2610.12848#S1.F1)for an illustration\.

## 5Applications in reinforcement learning

##### Offline\-to\-online RL\.

In offline\-to\-online RL, one commonly pre\-trains a flow policyvrefv\_\{\\mathrm\{ref\}\}by flow matching[Equation6](https://arxiv.org/html/2610.12848#S3.E6)and a critic by an offline method on a fixed dataset\. The policy is then fine\-tuned online, where it must stay close to the pretrained flow while it improves, since offline critics are reliable only near the data\. This motivates the behavior\-constrained objective of[Section3](https://arxiv.org/html/2610.12848#S3)\. Prior work for fine\-tuning flow policies typically converts the flow to a diffusion model with a specific \(memoryless\) noise schedule and leverages stochastic optimal control to compute adjoint states, which yield a stable regression objective for training\([Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib23);[Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49);[Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\)\. In contrast, FFM requires neither a conversion to a diffusion model nor the computation of adjoint states\.

Keeping the policy close to the pretrained policy is commonly realized by considering the behavior\-tilted target policyπ⋆∝pref​eQ/λ\\pi^\{\\star\}\\propto p\_\{\\mathrm\{ref\}\}\\,e^\{Q/\\lambda\}, whereprefp\_\{\\mathrm\{ref\}\}is the terminal distribution of the pretrained flow policy\([Uehara et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib86);[Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib23)\)\. Note that FFM only requires the score𝗌ref​\(x\)≔∇x​log​pref​\(x∣s\)\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(x\)\\coloneqq\\nabla\_\{x\}\\log p\_\{\\mathrm\{ref\}\}\(x\\mid s\)of the reference policy, which can be read off the pretrained velocity field via the velocity–score bridge \(see[SectionB\.1](https://arxiv.org/html/2610.12848#A2.SS1)\),

𝗌ref​\(x\)=\(vref​\(x,t∣s\)−α˙​\(t\)α⁡\(t\)​x\)/κ⁡\(t\)\(t→1\)\.\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(x\)\\;=\\;\\big\(v\_\{\\mathrm\{ref\}\}\(x,t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,x\\big\)\\big/\\kappa\(t\)\\qquad\(t\\to 1\)\.\(13\)Similar to[Shin et al\. \(2026a\)](https://arxiv.org/html/2610.12848#bib.bib80), FFM can then directly be used for behavior\-constrained fine\-tuning by considering

ℒBT−FFM​\(θ,θ¯\)=𝔼t,X1∼πθ¯,ε​\[‖vθ​\(Xt,t∣s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)α⁡\(t\)​g​\(X1\)‖2\],\\mathcal\{L\}\_\{\\mathrm\{BT\-FFM\}\}\(\\theta,\\bar\{\\theta\}\)=\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\},\\,\\varepsilon\}\\Big\[\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\alpha\(t\)\}\\,g\(X\_\{1\}\)\\big\\\|^\{2\}\\Big\],\(14\)withg⁡\(x\):=𝗌ref​\(x\)\+1λ​∇xQ​\(s,x\)g\(x\):=\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(x\)\+\\tfrac\{1\}\{\\lambda\}\\nabla\_\{x\}Q\(s,x\)the score of the behavior\-tilted target, in place ofℒFFM\\mathcal\{L\}\_\{\\mathrm\{FFM\}\}in the fixed\-point iteration[Equation12](https://arxiv.org/html/2610.12848#S4.E12)\. By the argument of Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)\([AppendixA](https://arxiv.org/html/2610.12848#A1)\), the behavior\-tilted policyπ⋆∝pref​eQ/λ\\pi^\{\\star\}\\propto p\_\{\\mathrm\{ref\}\}\\,e^\{Q/\\lambda\}is a fixed point of the resulting iteration\.

##### Maximum entropy reinforcement learning\.

The optimal policy in maximum entropy RL, as outlined in[Section3](https://arxiv.org/html/2610.12848#S3), is given byπ⋆​\(x∣s\)∝exp⁡\(Q⁡\(s,x\)/λ\)\\pi^\{\\star\}\(x\\mid s\)\\propto\{\\exp\\\!\\big\(Q\(s,x\)/\\lambda\\big\)\}\. Hence, we can directly use FFM to realize the policy improvement step\. However, computing the softQQ\-function requires the entropy of the current flow policy,ℋ\(πθ\(⋅∣s\)\)=−𝔼X1∼πθ\[logπθ\(X1∣s\)\]\\mathcal\{H\}\(\\pi\_\{\\theta\}\(\\cdot\\mid s\)\)=\-\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi\_\{\\theta\}\}\[\\log\\pi\_\{\\theta\}\(X\_\{1\}\\mid s\)\]\. To that end, we use the*instantaneous change of variables*\([Chen et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib14)\)to compute the log\-density of a generated action along its own sampling trajectory as

log⁡πθ​\(X1∣s\)=log⁡𝒩⁡\(X0,0,σ02​I\)−∫01∇x⋅vθ​\(Xt,t∣s\)​𝑑t,\\log\\pi\_\{\\theta\}\(X\_\{1\}\\mid s\)\\;=\\;\\log\\mathcal\{N\}\\\!\\left\(X\_\{0\};\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\)\\;\-\\;\\int\_\{0\}^\{1\}\\nabla\_\{x\}\\\!\\cdot v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\\,\\mathrm\{d\}t,\(15\)and estimate the entropy by averaging over generated actions\. However, the integral in[Equation15](https://arxiv.org/html/2610.12848#S5.E15)requires the divergence of the velocity field in each integration step, which can be computationally expensive\. Therefore, we follow common practice\([Grathwohl et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib36)\)and rely on Hutchinson’s trace estimator\([Hutchinson, 1989](https://arxiv.org/html/2610.12848#bib.bib43);[Skilling, 1989](https://arxiv.org/html/2610.12848#bib.bib82)\)\. More specifically, the estimator computes the divergence as the expectation∇x⋅v=𝔼z​\[z⊤​\(∂xv\)​z\]\\nabla\_\{x\}\\\!\\cdot v=\\mathbb\{E\}\_\{z\}\\big\[z^\{\\top\}\(\\partial\_\{x\}v\)\\,z\\big\]over random probeszzwith𝔼⁡\[z​z⊤\]=I\\mathbb\{E\}\[zz^\{\\top\}\]=I\. The cost is one vector–Jacobian product per integration step, and no auxiliary density network is needed\. For details, including an algorithmic description, we refer to[AppendixD](https://arxiv.org/html/2610.12848#A4)\.

##### Trust\-region constraints\.

Recently, on\-policy maximum entropy RL has gained popularity, in particular when using massively parallel simulators\. For on\-policy RL it is well known that trust\-region methods stabilize policy optimization and prevent premature convergence\([Peters et al\., 2010](https://arxiv.org/html/2610.12848#bib.bib68);[Schulman et al\., 2015](https://arxiv.org/html/2610.12848#bib.bib77);[Schulman et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib78);[Abdolmaleki et al\., 2015](https://arxiv.org/html/2610.12848#bib.bib1);[Otto et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib63);[Akrour et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib5);[Voelcker et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib90);[Le et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib47);[Thilges et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib85)\)\. Similar to[Thilges et al\. \(2026\)](https://arxiv.org/html/2610.12848#bib.bib85), we employ a trust\-region loss that keeps the velocity close to the previous one via

ℒTR​\(θ,θ¯\)=∫01𝔼X1∼πθ¯,ε​\[‖vθ​\(Xt,t∣s\)−vθ¯​\(Xt,t∣s\)‖2\]​𝑑t\.\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\bar\{\\theta\}\)\\;=\\;\\int\_\{0\}^\{1\}\\,\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\},\\,\\varepsilon\}\\Big\[\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-v\_\{\\bar\{\\theta\}\}\(X\_\{t\},t\\mid s\)\\big\\\|^\{2\}\\Big\]\\,\\mathrm\{d\}t\.\(16\)[Equation16](https://arxiv.org/html/2610.12848#S5.E16)is then used to constrain the fixed\-point update[Equation12](https://arxiv.org/html/2610.12848#S4.E12)to stay close to the previous flow policy with parametersθk\\theta\_\{k\}, i\.e\.,

θk\+1=arg​minθ⁡ℒFFM​\(θ,θk\)s\.t\.ℒTR​\(θ,θk\)≤εTR\.\\theta\_\{k\+1\}=\\argmin\_\{\\theta\}\\ \\mathcal\{L\}\_\{\\mathrm\{FFM\}\}\(\\theta,\\theta\_\{k\}\)\\quad\\mathrm\{s\.t\.\}\\quad\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\theta\_\{k\}\)\\leq\\varepsilon\_\{\\mathrm\{TR\}\}\.\(17\)The constrained problem in[Equation17](https://arxiv.org/html/2610.12848#S5.E17)can be approached by the method of Lagrange multipliers\([Boyd & Vandenberghe, 2004](https://arxiv.org/html/2610.12848#bib.bib10)\)\. Introducing the multiplierν≥0\\nu\\geq 0, the Lagrangian reads

ℒLag​\(θ,ν,θk\)=ℒFFM​\(θ,θk\)\+ν⁡\(ℒTR​\(θ,θk\)−εTR\),\\mathcal\{L\}\_\{\\mathrm\{Lag\}\}\(\\theta,\\nu;\\theta\_\{k\}\)\\;=\\;\\mathcal\{L\}\_\{\\mathrm\{FFM\}\}\(\\theta,\\theta\_\{k\}\)\\;\+\\;\\nu\\,\\big\(\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\theta\_\{k\}\)\-\\varepsilon\_\{\\mathrm\{TR\}\}\\big\),\(18\)and[Equation17](https://arxiv.org/html/2610.12848#S5.E17)is replaced by the saddle\-point problem

θk\+1=arg​minθ⁡maxν≥0​ℒLag​\(θ,ν,θk\)\.\\theta\_\{k\+1\}\\;=\\;\\argmin\_\{\\theta\}\\,\\max\_\{\\nu\\geq 0\}\\;\\mathcal\{L\}\_\{\\mathrm\{Lag\}\}\(\\theta,\\nu;\\theta\_\{k\}\)\.\(19\)We solve[Equation19](https://arxiv.org/html/2610.12848#S5.E19)by alternating stochastic gradient descent onθ\\thetawith gradient ascent onν\\nu, such that the multiplier grows while the constraint is violated and decays once it is satisfied\. We refer to[AppendicesD](https://arxiv.org/html/2610.12848#A4)and[E](https://arxiv.org/html/2610.12848#A5)for the instantiations with the previous policy and with the pretrained flow, respectively, including the dual update\.

##### Correlated exploration\.

A flow policy is a deterministic function of its prior draw, and hence, integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)fromX0=σ0​εX\_\{0\}=\\sigma\_\{0\}\\varepsilonyields the actionx=Ψθ​\(ε∣s\)x=\\Psi\_\{\\theta\}\(\\varepsilon\\mid s\), whose only source of randomness isε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)\. Here,Ψθ\(⋅∣s\)\\Psi\_\{\\theta\}\(\\cdot\\mid s\)is simply the sampler of the flow policy: the map that takes a prior drawε\\varepsilonand returns the clean action obtained by integrating the ODE with the learned velocity field\. Hence, exploration can also be shaped by correlating the prior draws across environment steps\. Similar to colored\-noise exploration for Gaussian policies\([Eberhard et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib29)\), we draw the priors within an episode from a stationary Ornstein–Uhlenbeck chain with correlation timeτou\\tau\_\{\\mathrm\{ou\}\}, i\.e\.,

εk\+1=e−1/τouεk\+1−e−2/τouξk,ξk∼𝒩\(⋅;0,I\),ε0∼𝒩\(⋅;0,I\),\\varepsilon\_\{k\+1\}\\;=\\;e^\{\-1/\\tau\_\{\\mathrm\{ou\}\}\}\\,\\varepsilon\_\{k\}\\;\+\\;\\sqrt\{1\-e^\{\-2/\\tau\_\{\\mathrm\{ou\}\}\}\}\\;\\xi\_\{k\},\\qquad\\xi\_\{k\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\),\\qquad\\varepsilon\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\),\(20\)and executexk=Ψθ​\(εk∣sk\)x\_\{k\}=\\Psi\_\{\\theta\}\(\\varepsilon\_\{k\}\\mid s\_\{k\}\)\. Since the chain in[Equation20](https://arxiv.org/html/2610.12848#S5.E20)is stationary, everyεk\\varepsilon\_\{k\}is marginally distributed according to𝒩⁡\(⋅,0,I\)\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\), and every executed action remains a sample ofπθ\(⋅∣sk\)\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{k\}\)\(see[SectionB\.3](https://arxiv.org/html/2610.12848#A2.SS3)\)\. Therefore, the FFM regression[Equation11](https://arxiv.org/html/2610.12848#S4.E11), the entropy estimate[Equation15](https://arxiv.org/html/2610.12848#S5.E15), and the soft Bellman backup[Equation42](https://arxiv.org/html/2610.12848#A4.E42)are unchanged, and so is the fixed pointπ⋆\\pi^\{\\star\}of Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3); only the states the agent visits change\. Instead of resampling the prior at every step, the agent commits to a direction for roughlyτou\\tau\_\{\\mathrm\{ou\}\}steps, which produces the sustained, directed behavior that sparse\-reward tasks require before any reward is observed\. We use it only for the hard\-exploration tasks of[SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)\. Other colored noise processes, such as the pink noise of[Eberhard et al\. \(2023\)](https://arxiv.org/html/2610.12848#bib.bib29), can be used as well, which we leave to future work\.

## 6Experiments

We evaluate FFM in two settings: offline\-to\-online fine\-tuning, and online on\-policy maximum entropy RL, where we train a flow policy from scratch\. For the offline\-to\-online setting, we focus on the ten OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.12848#bib.bib65)\)locomotion and manipulation domains totaling 50 tasks\. For the online RL setting, we consider the GPU\-accelerated MJX\([Zakka et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib98)\)Playground and humanoid tasks, which result in 24 tasks in total\. We restrict ourselves to presenting the main results in this section, and refer the reader to[AppendixE](https://arxiv.org/html/2610.12848#A5)and to[AppendixD](https://arxiv.org/html/2610.12848#A4)for additional results and ablations, such as varying the number of integration steps, the interpolation schedules, or different trust region constraints\.

##### Offline\-to\-online RL\.

We pretrain a behavior\-cloning flow policy and a critic offline \(to11M gradient steps\) and then fine\-tune online \(to1\.51\.5M\), against TRQAM\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\), QAM \(\+QAM\-E\)\([Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49)\), FQL \(\+IFQL\)\([Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\), DSRL\([Wagenmaker et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib91)\), and CGQL\-L\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\)under an identical protocol \([Figure2](https://arxiv.org/html/2610.12848#S6.F2)\)\. FFM’s offline phase is deliberately conservative, keeping the policy inside the pretrained flow’s support instead of chasing an offline critic that is unreliable off\-support, so its offline average \(0\.630\.63\) trails the strongest offline baseline \(TRQAM,0\.680\.68;[Table5](https://arxiv.org/html/2610.12848#A5.T5)\)\. However, once the online phase starts and the interaction makes the critic trustworthy, the budget is relaxed and FFM improves rapidly and attains the best aggregate success at1\.51\.5M steps \(0\.950\.95mean vs\.0\.930\.93for the next\-best method;[Table6](https://arxiv.org/html/2610.12848#A5.T6)\), with the widest margins on the hard, long\-horizon tasks \(e\.g\. HumanoidMaze\-Large,0\.990\.99vs\.0\.760\.76\) and without a single divergence across seeds\.

000\.50\.511AntMaze\-LargeAntMaze\-GiantHumanoidMaze\-MediumHumanoidMaze\-Large000\.50\.511Cube\-DoubleCube\-Triple00\.511\.5Cube\-Quadruple00\.511\.5Scene000\.50\.5111\.51\.5000\.50\.511Puzzle\-3x3000\.50\.5111\.51\.5Puzzle\-4x4FFM \(ours\)TRQAMQAMQAM\-EIFQLFQLDSRLCGQL\-Lofflineonline

Figure 2:Per\-domain aggregated offline\-to\-online success rate\(mean over the five tasks and all seeds\)\. Offline \(≤1\\leq\\\!1M\) and online \(\>1\>\\\!1M\) phases are shaded; FFM \(ours\) is drawn in bold with a±1\\pm 1standard\-deviation band\. Per\-task curves and full tables are in[SectionE\.3](https://arxiv.org/html/2610.12848#A5.SS3)\.
##### Online RL\.

We additionally train FFM in the online on\-policy RL setting where the policy is trained from scratch\. Concretely, we train FFM on the 20 MuJoCo Playground DMC\([Zakka et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib98)\)tasks and on the four MuJoCo Playground humanoid tasks that require precise high\-dimensional control\. We compare FFM’s performance against state\-of\-the\-art Gaussian policy representations \(REPPO\([Voelcker et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib90)\), PPO\([Schulman et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib78)\), SPO\([Xie et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib96)\)\), diffusion policy representations \(TRUDI\([Le et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib47)\), AMDP\([Thilges et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib85)\), DIME\([Celik et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib11)\), DPPO\([Ren et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib72)\)\) and flow policy representations \(FPO\([McAllister et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib60)\)\)\. Following[Agarwal et al\. \(2021\)](https://arxiv.org/html/2610.12848#bib.bib3), every learning curve represents the aggregated interquartile mean \(IQM\) with95%95\\%confidence intervals from a stratified bootstrap for every task in[Figure4](https://arxiv.org/html/2610.12848#A4.F4)for the Playground tasks and in[Figure5](https://arxiv.org/html/2610.12848#A4.F5)for the humanoid tasks, where every task is run on1010seeds\. While FFM performs comparably on the lower\-dimensional Playground tasks \([Figure3](https://arxiv.org/html/2610.12848#S6.F3)left two panels and[Figure4](https://arxiv.org/html/2610.12848#A4.F4)\), FFM converges faster to the highest performance alongside TRUDI on the more sophisticated, high\-dimensional humanoid tasks\. This indicates that FFM’s benefits are most visible in high\-dimensional tasks\. We further integrated correlated exploration into FFM and ran experiments on the Playground tasks\. The resulting learning curves are shown in[Figure3](https://arxiv.org/html/2610.12848#S6.F3)left two panels and[Figure4](https://arxiv.org/html/2610.12848#A4.F4)for the aggregated and per\-task results, respectively\. Please note that we have run these experiments only on a subset of the tasks \(see[Figure4](https://arxiv.org/html/2610.12848#A4.F4)\)\. The aggregated FFM \+ OU curve \(dashed in[Figure3](https://arxiv.org/html/2610.12848#S6.F3)\) uses the FFM \+ OU learning curve on each task where we ran correlated exploration and the FFM learning curve on all other tasks\. FFM \+ OU shows a faster convergence to a similar return value as the highest\-performing baseline DIME, indicating that if computational resources permit, FFM’s performance can be boosted even more by tuningτou\\tau\_\{\\mathrm\{ou\}\}\.

Figure 3:Maximum Entropy Online RL\.We have run FFM on all 20 MJX Playground tasks and on the four MJX humanoid tasks and present the aggregated results over all Playground tasks \(left two panels\) and the aggregated results over the humanoid tasks \(right two panels\)\. We have evaluated all methods on1010seeds for every single environment \(see[Figure4](https://arxiv.org/html/2610.12848#A4.F4)and[Figure5](https://arxiv.org/html/2610.12848#A4.F5)for results\) and report the IQM with95%95\\%stratified bootstrapped confidence intervals following[Agarwal et al\. \(2021\)](https://arxiv.org/html/2610.12848#bib.bib3)\. FFM performs on par with the baselines on the Playground tasks \(left two panels\) but converges to the highest return much faster than TRUDI on the high\-dimensional humanoid tasks\. This result indicates that FFM’s benefits are most pronounced on high\-dimensional tasks\. Furthermore, we can boost FFM’s performance by leveraging its deterministic mapping properties and incorporating correlated exploration \(FFM \+ OU\), which leads to a significantly faster convergence to a similar return value as the best\-performing baseline DIME \(left two panels\) \(see[Figure15](https://arxiv.org/html/2610.12848#A4.F15)\)\.

## 7Conclusion

We introduced fixed\-point flow matching, a data\-free procedure for training flow policies using a flow\-matching\-style regression objective\. Its basis is the target flow identity, which expresses the velocity field as a conditional average ofQQ\-scores at clean actions and thereby replaces the data\-derived targets of flow matching by critic gradients\. Evaluating the resulting objective on samples from the current policy instead of the optimal policy yields a fixed\-point iteration that has the optimal policy as a fixed point\. The construction is conceptually simple, easy to implement, and needs no gradients through the sampling chain, no conversion of the flow into a stochastic differential equation, and no adjoint simulation\. We consider two instantiations of the proposed framework: an online maximum entropy actor–critic algorithm, and offline\-to\-online fine\-tuning against a behavior\-tilted target\. Across5050offline\-to\-online tasks and2424online tasks, both are competitive with or better than the strongest available baselines\.

##### Limitations\.

FFM requires a*differentiable*target\. The regression target is built from theQQ\-score∇xQ​\(s,x\)\\nabla\_\{x\}Q\(s,x\), so the critic must be differentiable in the action and the policy update is only as accurate as the critic’s gradient\. This restricts FFM to continuous action spaces\. Extensions to non\-differentiable targets and discrete action spaces are left for future work\.

## Acknowledgements

The authors thank Yonghoon Dong for providing the results for several baselines used in[Figure2](https://arxiv.org/html/2610.12848#S6.F2)\. D\.B\. acknowledges support by funding from a Google PhD fellowship in Machine Learning and ML Foundations, the pilot program Core Informatics of the Helmholtz Association \(HGF\) and the state of Baden\-Württemberg through bwHPC, as well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden\-Württemberg and by the German Federal Ministry of Education and Research\. Furthermore, all authors are supported by the European Research Council \(ERC\) under the European Union’s Horizon Europe program through the project SMARTI³ \(Grant Agreement No\. 101171393\)\.

## References

- Abdolmaleki et al\. \(2015\)Abbas Abdolmaleki, Rudolf Lioutikov, Jan R Peters, Nuno Lau, Luis Pualo Reis, and Gerhard Neumann\.Model\-based relative entropy stochastic search\.*Advances in Neural Information Processing Systems*, 28, 2015\.
- Abdolmaleki et al\. \(2018\)Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller\.Maximum a posteriori policy optimisation\.In*International Conference on Learning Representations*, 2018\.URL[https://openreview\.net/forum?id=S1ANxQW0b](https://openreview.net/forum?id=S1ANxQW0b)\.
- Agarwal et al\. \(2021\)Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare\.Deep reinforcement learning at the edge of the statistical precipice\.*Advances in neural information processing systems*, 34:29304–29320, 2021\.
- Akhound\-Sadegh et al\. \(2024\)Tara Akhound\-Sadegh, Jarrid Rector\-Brooks, Joey Bose, Sarthak Mittal, Pablo Lemos, Cheng\-Hao Liu, Marcin Sendera, Siamak Ravanbakhsh, Gauthier Gidel, Yoshua Bengio, Nikolay Malkin, and Alexander Tong\.Iterated denoising energy matching for sampling from boltzmann densities\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp \(eds\.\),*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 760–786\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/akhound\-sadegh24a\.html](https://proceedings.mlr.press/v235/akhound-sadegh24a.html)\.
- Akrour et al\. \(2019\)Riad Akrour, Joni Pajarinen, Jan Peters, and Gerhard Neumann\.Projections for approximate policy iteration algorithms\.In*International Conference on Machine Learning*, pp\. 181–190\. PMLR, 2019\.
- Albergo et al\. \(2025\)Michael Albergo, Nicholas M Boffi, and Eric Vanden\-Eijnden\.Stochastic interpolants: A unifying framework for flows and diffusions\.*Journal of Machine Learning Research*, 26\(209\):1–80, 2025\.
- Albergo & Vanden\-Eijnden \(2023\)Michael Samuel Albergo and Eric Vanden\-Eijnden\.Building normalizing flows with stochastic interpolants\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=li7qeBbCR1t](https://openreview.net/forum?id=li7qeBbCR1t)\.
- Black et al\. \(2024\)Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine\.Training diffusion models with reinforcement learning\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=YCWjhGrJFD](https://openreview.net/forum?id=YCWjhGrJFD)\.
- Blessing et al\. \(2026\)Denis Blessing, Lorenz Richter, Julius Berner, Egor Malitskiy, and Gerhard Neumann\.Bridge matching sampler: Scalable sampling via generalized fixed\-point diffusion matching\.*arXiv preprint arXiv:2603\.00530*, 2026\.
- Boyd & Vandenberghe \(2004\)Stephen Boyd and Lieven Vandenberghe\.*Convex optimization*, volume 1\.Cambridge university press Cambridge, 2004\.
- Celik et al\. \(2025\)Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann\.DIME: Diffusion\-based maximum entropy reinforcement learning\.In*Forty\-second International Conference on Machine Learning*, 2025\.URL[https://openreview\.net/forum?id=Aw6dBR7Vxj](https://openreview.net/forum?id=Aw6dBR7Vxj)\.
- Chen et al\. \(2023\)Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu\.Offline reinforcement learning via high\-fidelity generative behavior modeling\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=42zs3qa2kpy](https://openreview.net/forum?id=42zs3qa2kpy)\.
- Chen et al\. \(2024\)Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu\.Score regularized policy optimization through diffusion behavior\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=xCRr9DrolJ](https://openreview.net/forum?id=xCRr9DrolJ)\.
- Chen et al\. \(2018\)Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud\.Neural ordinary differential equations\.*Advances in neural information processing systems*, 31, 2018\.
- Chen et al\. \(2025\)Tianyi Chen, Haitong Ma, Na Li, Kai Wang, and Bo Dai\.One\-step flow policy mirror descent\.*arXiv preprint arXiv:2507\.23675*, 2025\.
- Chi et al\. \(2023\)Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin CM Burchfiel, and Shuran Song\.Diffusion Policy: Visuomotor Policy Learning via Action Diffusion\.In*Proceedings of Robotics: Science and Systems*, Daegu, Republic of Korea, July 2023\.doi:10\.15607/RSS\.2023\.XIX\.026\.
- De Bortoli et al\. \(2024\)Valentin De Bortoli, Michael Hutchinson, Peter Wirnsberger, and Arnaud Doucet\.Target score matching\.*arXiv preprint arXiv:2402\.08667*, 2024\.
- Dern et al\. \(2025\)Niclas Dern, Lennart Redl, Sebastian Pfister, Marcel Kollovieh, David Lüdke, and Stephan Günnemann\.Energy\-weighted flow matching: Unlocking continuous normalizing flows for efficient and scalable Boltzmann sampling\.*arXiv preprint arXiv:2509\.03726*, 2025\.
- Ding et al\. \(2024\)Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi\.Diffusion\-based reinforcement learning via Q\-weighted variational policy optimization\.In*Advances in Neural Information Processing Systems*, 2024\.
- Ding et al\. \(2025\)Shutong Ding, Ke Hu, Shan Zhong, Haoyang Luo, Weinan Zhang, Jingya Wang, Jun Wang, and Ye Shi\.Genpo: Generative diffusion models meet on\-policy reinforcement learning\.*Advances in neural information processing systems*, 38:130443–130474, 2025\.
- Ding & Jin \(2024\)Zihan Ding and Chi Jin\.Consistency models as a rich and efficient policy class for reinforcement learning\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=v8jdwkUNXb](https://openreview.net/forum?id=v8jdwkUNXb)\.
- Domingo\-Enrich & Han \(2026\)Carles Domingo\-Enrich and Jiequn Han\.Adjoint matching through the lens of the stochastic maximum principle in optimal control\.*arXiv preprint arXiv:2604\.08580*, 2026\.
- Domingo\-Enrich et al\. \(2025\)Carles Domingo\-Enrich, Michal Drozdzal, Brian Karrer, and Ricky T\. Q\. Chen\.Adjoint matching: Fine\-tuning flow and diffusion generative models with memoryless stochastic optimal control\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=xQBRrtQM8u](https://openreview.net/forum?id=xQBRrtQM8u)\.
- Domingo\-Enrich et al\. \(2026\)Carles Domingo\-Enrich, Yuanqi Du, and Michael S\. Albergo\.A unified perspective on fine\-tuning and sampling with diffusion and flow models\.*arXiv preprint arXiv:2605\.00229*, 2026\.
- Dong et al\. \(2025\)Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang\.Maximum entropy reinforcement learning with diffusion policy\.In*International Conference on Machine Learning*, 2025\.
- Dong et al\. \(2026a\)Xiaoyi Dong, Xi Sheryl Zhang, and Jian Cheng\.Mean flow policy optimization\.In*Forty\-third International Conference on Machine Learning*, 2026a\.URL[https://openreview\.net/forum?id=88H3V90Wzo](https://openreview.net/forum?id=88H3V90Wzo)\.
- Dong et al\. \(2026b\)Yonghoon Dong, Kyungmin Lee, Changyeon Kim, Jaehyuk Kim, and Jinwoo Shin\.Trust region q adjoint matching\.*arXiv preprint arXiv:2605\.27079*, 2026b\.
- Doo et al\. \(2026\)JaeHyeok Doo, Byeongguk Jeon, Seonghyeon Ye, Kimin Lee, and Minjoon Seo\.Q\-Flow: Stable and expressive reinforcement learning with flow\-based policy\.*arXiv preprint arXiv:2605\.13435*, 2026\.
- Eberhard et al\. \(2023\)Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius\.Pink noise is all you need: Colored noise exploration in deep reinforcement learning\.In*International Conference on Learning Representations*, 2023\.
- Efron \(2011\)Bradley Efron\.Tweedie’s formula and selection bias\.*Journal of the American Statistical Association*, 106\(496\):1602–1614, 2011\.
- Fan et al\. \(2023\)Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee\.DPOK: Reinforcement learning for fine\-tuning text\-to\-image diffusion models\.In*Advances in Neural Information Processing Systems*, 2023\.
- Farebrother et al\. \(2024\)Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal\.Stop regressing: Training value functions via classification for scalable deep RL\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp \(eds\.\),*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 13049–13071\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/farebrother24a\.html](https://proceedings.mlr.press/v235/farebrother24a.html)\.
- Fujimoto et al\. \(2018\)Scott Fujimoto, Herke Hoof, and David Meger\.Addressing function approximation error in actor\-critic methods\.In*International conference on machine learning*, pp\. 1587–1596\. Pmlr, 2018\.
- Gao et al\. \(2026\)Ting Gao, Stavros Orfanoudakis, Nan Lin, Winnie Daamen, Serge Hoogendoorn, and Elvin Isufi\.Flow matching policy optimization with mirror descent and entropy constraints\.*arXiv preprint arXiv:2603\.17685*, 2026\.
- Ghanem & Ghogho \(2026\)Abdelghani Ghanem and Mounir Ghogho\.Entropy\-regularized adjoint matching for offline reinforcement learning\.*arXiv preprint arXiv:2605\.06156*, 2026\.
- Grathwohl et al\. \(2019\)Will Grathwohl, Ricky T\. Q\. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud\.FFJORD: Free\-form continuous dynamics for scalable reversible generative models\.In*International Conference on Learning Representations*, 2019\.
- Guo et al\. \(2026\)Zhengyi Guo, Jiayuan Sheng, David D\. Yao, and Wenpin Tang\.Improved techniques for fine\-tuning flow models via adjoint matching: A deterministic control pipeline\.*arXiv preprint arXiv:2605\.06583*, 2026\.
- Haarnoja et al\. \(2017\)Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine\.Reinforcement learning with deep energy\-based policies\.In*International conference on machine learning*, pp\. 1352–1361\. PMLR, 2017\.
- Haarnoja et al\. \(2018\)Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.In*International conference on machine learning*, pp\. 1861–1870\. Pmlr, 2018\.
- Hansen\-Estruch et al\. \(2023\)Philippe Hansen\-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine\.Idql: Implicit q\-learning as an actor\-critic method with diffusion policies\.*arXiv preprint arXiv:2304\.10573*, 2023\.
- Havens et al\. \(2025\)Aaron Havens, Benjamin Kurt Miller, Bing Yan, Carles Domingo\-Enrich, Anuroop Sriram, Brandon Wood, Daniel Levine, Bin Hu, Brandon Amos, Brian Karrer, Xiang Fu, Guan\-Horng Liu, and Ricky T\. Q\. Chen\.Adjoint sampling: Highly scalable diffusion samplers via adjoint matching\.In*International Conference on Machine Learning*, 2025\.
- Ho et al\. \(2020\)Jonathan Ho, Ajay Jain, and Pieter Abbeel\.Denoising diffusion probabilistic models\.*Advances in neural information processing systems*, 33:6840–6851, 2020\.
- Hutchinson \(1989\)Michael F Hutchinson\.A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines\.*Communications in Statistics\-Simulation and Computation*, 18\(3\):1059–1076, 1989\.
- Janner et al\. \(2022\)Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine\.Planning with diffusion for flexible behavior synthesis\.In*International Conference on Machine Learning*, 2022\.
- Kang et al\. \(2023\)Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan\.Efficient diffusion policies for offline reinforcement learning\.*Advances in Neural Information Processing Systems*, 36:67195–67212, 2023\.
- Kim et al\. \(2026\)Sungha Kim, Gawon Lee, Jusuk Lee, Jonghae Park, H\. Jin Kim, and Daesol Cho\.FLAG: Flow policy MaxEnt\-RL by latent augmented guidance\.*arXiv preprint arXiv:2605\.30749*, 2026\.
- Le et al\. \(2026\)Huy Le, Onur Celik, Denis Blessing, Tai Hoang, Claas A Voelcker, Axel Brunnbauer, Felix Richter, Michael Volpp, and Gerhard Neumann\.Trust\-region diffusion policies for massively parallel on\-policy RL\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=mGu2fs7kJt](https://openreview.net/forum?id=mGu2fs7kJt)\.
- Levine \(2018\)Sergey Levine\.Reinforcement learning and control as probabilistic inference: Tutorial and review\.*arXiv preprint arXiv:1805\.00909*, 2018\.
- Li & Levine \(2026\)Qiyang Li and Sergey Levine\.Q\-learning with adjoint matching\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=vd4eNAdtO6](https://openreview.net/forum?id=vd4eNAdtO6)\.
- Li et al\. \(2024\)Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki\.Learning multimodal behaviors from scratch with diffusion policy gradient\.*Advances in Neural Information Processing Systems*, 37:38456–38479, 2024\.
- Li et al\. \(2026\)Zeyang Li, Sunbochen Tang, and Navid Azizan\.Reverse flow matching: A unified framework for online reinforcement learning with diffusion and flow policies\.*arXiv preprint arXiv:2601\.08136*, 2026\.
- Lipman et al\. \(2023\)Yaron Lipman, Ricky T\. Q\. Chen, Heli Ben\-Hamu, Maximilian Nickel, and Matthew Le\.Flow matching for generative modeling\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=PqvMRDCJT9t](https://openreview.net/forum?id=PqvMRDCJT9t)\.
- Liu et al\. \(2025a\)Guan\-Horng Liu, Jaemoo Choi, Yongxin Chen, Benjamin Kurt Miller, and Ricky T\. Q\. Chen\.Adjoint Schrödinger bridge sampler\.In*Advances in Neural Information Processing Systems*, 2025a\.
- Liu et al\. \(2025b\)Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang\.Flow\-GRPO: Training flow matching models via online RL\.*arXiv preprint arXiv:2505\.05470*, 2025b\.
- Liu et al\. \(2023\)Xingchao Liu, Chengyue Gong, and qiang liu\.Flow straight and fast: Learning to generate and transfer data with rectified flow\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=XVjTT1nw5z](https://openreview.net/forum?id=XVjTT1nw5z)\.
- Lu et al\. \(2023\)Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu\.Contrastive energy prediction for exact energy\-guided diffusion sampling in offline reinforcement learning\.In*International Conference on Machine Learning*, pp\. 22825–22855\. PMLR, 2023\.
- Lv et al\. \(2025\)Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma\.Flow\-based policy for online reinforcement learning\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.URL[https://openreview\.net/forum?id=CANUXhPoyn](https://openreview.net/forum?id=CANUXhPoyn)\.
- Lv et al\. \(2026\)Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, and Xiao Ma\.FLAC: Maximum entropy RL via kinetic energy regularized bridge matching\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=v0qBWYewKR](https://openreview.net/forum?id=v0qBWYewKR)\.
- Ma et al\. \(2025\)Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai\.Efficient online reinforcement learning for diffusion policy\.In*International Conference on Machine Learning*, 2025\.
- McAllister et al\. \(2026\)David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa\.Flow matching policy gradients\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=eoEmoKoQpJ](https://openreview.net/forum?id=eoEmoKoQpJ)\.
- Mosso et al\. \(2026\)Mattia Mosso, Jaemoo Choi, and Heng Yang\.Hard\-constrained sampling on embedded Riemannian manifolds via adjoint Schrödinger bridges\.*arXiv preprint arXiv:2608\.25838*, 2026\.
- Nair et al\. \(2020\)Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine\.AWAC: Accelerating online reinforcement learning with offline datasets\.*arXiv preprint arXiv:2006\.09359*, 2020\.
- Otto et al\. \(2021\)Fabian Otto, Philipp Becker, Vien Anh Ngo, Hanna Carolin Maria Ziesche, and Gerhard Neumann\.Differentiable trust region layers for deep reinforcement learning\.In*International Conference on Learning Representations*, 2021\.URL[https://openreview\.net/forum?id=qYZD\-AO1Vn](https://openreview.net/forum?id=qYZD-AO1Vn)\.
- OuYang et al\. \(2026\)RuiKang OuYang, Bo Qiang, and José Miguel Hernández\-Lobato\.BNEM: A Boltzmann sampler based on bootstrapped noised energy matching\.*Transactions on Machine Learning Research*, 2026\.
- Park et al\. \(2025a\)Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine\.Ogbench: Benchmarking offline goal\-conditioned rl\.In*International Conference on Learning Representations \(ICLR\)*, 2025a\.
- Park et al\. \(2025b\)Seohong Park, Qiyang Li, and Sergey Levine\.Flow q\-learning\.In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste\-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu \(eds\.\),*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pp\. 48104–48127\. PMLR, 13–19 Jul 2025b\.URL[https://proceedings\.mlr\.press/v267/park25f\.html](https://proceedings.mlr.press/v267/park25f.html)\.
- Peng et al\. \(2019\)Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine\.Advantage\-weighted regression: Simple and scalable off\-policy reinforcement learning\.*arXiv preprint arXiv:1910\.00177*, 2019\.
- Peters et al\. \(2010\)Jan Peters, Katharina Mulling, and Yasemin Altun\.Relative entropy policy search\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 24, pp\. 1607–1612, 2010\.
- Potaptchik et al\. \(2025\)Peter Potaptchik, Cheuk\-Kit Lee, and Michael S\. Albergo\.Tilt matching for scalable sampling and fine\-tuning\.*arXiv preprint arXiv:2512\.21829*, 2025\.
- Psenka et al\. \(2024\)Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma\.Learning a diffusion model policy from rewards via q\-score matching\.In*Forty\-first International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=35ahHydjXo](https://openreview.net/forum?id=35ahHydjXo)\.
- Raffin et al\. \(2022\)Antonin Raffin, Jens Kober, and Freek Stulp\.Smooth exploration for robotic reinforcement learning\.In*Conference on Robot Learning*, pp\. 1634–1644\. PMLR, 2022\.
- Ren et al\. \(2025\)Allen Z\. Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz\.Diffusion policy policy optimization\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=mEpqHvbD2h](https://openreview.net/forum?id=mEpqHvbD2h)\.
- Reuss et al\. \(2023\)Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov\.Goal\-conditioned imitation learning using score\-based diffusion policies\.*arXiv preprint arXiv:2304\.02532*, 2023\.
- Robbins \(1956\)Herbert Robbins\.An empirical Bayes approach to statistics\.In*Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability*, volume 1, pp\. 157–163, 1956\.
- Ruan et al\. \(2026\)Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alán Aspuru\-Guzik, Igor Gilitschenski, and Nicholas Rhinehart\.QPILOTS: Efficient test\-time Q\-steering for flow policies\.*arXiv preprint arXiv:2606\.14801*, 2026\.
- Sanokowski & Patil \(2025\)Sebastian Sanokowski and Kaustubh Patil\.Diffusion\-augmented Markov decision processes for maximum entropy reinforcement learning\.*arXiv preprint arXiv:2512\.02019*, 2025\.
- Schulman et al\. \(2015\)John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz\.Trust region policy optimization\.In*International conference on machine learning*, pp\. 1889–1897\. Pmlr, 2015\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Shi et al\. \(2026\)Kexin Shi, Junyao Shi, Poorvi Hebbar, Zhuolun Zhao, Tarun Amarnath, Yifan Su, Shikhar Bahl, and Deepak Pathak\.FlowDPG: Deterministic policy gradient on flow matching policies for real\-world manipulation\.*arXiv preprint arXiv:2606\.22303*, 2026\.
- Shin et al\. \(2026a\)Jeongwoo Shin, Dongsoo Shin, Yuchen Zhu, Wei Guo, Yongxin Chen, Joonseok Lee, Jaewoong Choi, and Jaemoo Choi\.Efficient adjoint matching for fine\-tuning diffusion models\.*arXiv preprint arXiv:2605\.11480*, 2026a\.
- Shin et al\. \(2026b\)Yongjae Shin, Jongseong Chae, Jongeui Park, and Youngchul Sung\.Flow matching with injected noise for offline\-to\-online reinforcement learning\.In*International Conference on Learning Representations*, 2026b\.
- Skilling \(1989\)John Skilling\.The eigenvalues of mega\-dimensional matrices\.In*Maximum Entropy and Bayesian Methods: Cambridge, England, 1988*, pp\. 455–466\. Springer, 1989\.
- So et al\. \(2026\)Oswin So, Brian Karrer, Chuchu Fan, Ricky T\. Q\. Chen, and Guan\-Horng Liu\.Discrete adjoint matching\.In*International Conference on Learning Representations*, 2026\.
- Song et al\. \(2021\)Yang Song, Jascha Sohl\-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole\.Score\-based generative modeling through stochastic differential equations\.In*International Conference on Learning Representations*, 2021\.URL[https://openreview\.net/forum?id=PxTIG12RRHS](https://openreview.net/forum?id=PxTIG12RRHS)\.
- Thilges et al\. \(2026\)Serge Thilges, Onur Celik, Denis Blessing, Emiliyan Gospodinov, and Gerhard Neumann\.Scalable maximum entropy reinforcement learning for diffusion policies via adjoint matching\.*arXiv preprint arXiv:2606\.22630*, 2026\.
- Uehara et al\. \(2024\)Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M\. Tseng, Tommaso Biancalani, and Sergey Levine\.Fine\-tuning of continuous\-time diffusion models as entropy\-regularized control\.*arXiv preprint arXiv:2402\.15194*, 2024\.
- Uehara et al\. \(2025\)Masatoshi Uehara, Yulai Zhao, Chenyu Wang, Xiner Li, Aviv Regev, Sergey Levine, and Tommaso Biancalani\.Inference\-time alignment in diffusion models with reward\-guided generation: Tutorial and review\.*arXiv preprint arXiv:2501\.09685*, 2025\.
- Vargas et al\. \(2023\)Francisco Vargas, Will Sussman Grathwohl, and Arnaud Doucet\.Denoising diffusion samplers\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=8pvnfTAbu1f](https://openreview.net/forum?id=8pvnfTAbu1f)\.
- Vincent \(2011\)Pascal Vincent\.A connection between score matching and denoising autoencoders\.*Neural computation*, 23\(7\):1661–1674, 2011\.
- Voelcker et al\. \(2026\)Claas A Voelcker, Axel Brunnbauer, Marcel Hussing, Michal Nauman, Pieter Abbeel, Radu Grosu, Eric Eaton, Amir massoud Farahmand, and Igor Gilitschenski\.Relative entropy pathwise policy optimization\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=4vmm8mlHkS](https://openreview.net/forum?id=4vmm8mlHkS)\.
- Wagenmaker et al\. \(2025\)Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine\.Steering your diffusion policy with latent space reinforcement learning\.In*Conference on Robot Learning*, 2025\.
- Wang et al\. \(2024\)Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, and Shengbo Eben Li\.Diffusion actor\-critic with entropy regulator\.In*Advances in Neural Information Processing Systems*, 2024\.
- Wang et al\. \(2023\)Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou\.Diffusion policies as an expressive policy class for offline reinforcement learning\.In*The Eleventh International Conference on Learning Representations*, 2023\.URL[https://openreview\.net/forum?id=AHvFDPi\-FA](https://openreview.net/forum?id=AHvFDPi-FA)\.
- Wang et al\. \(2026\)Ziqian Wang, Rui Zhang, Yitian Liu, Xingjian Mao, Minqian Wang, and Yao Mu\.Q\-VGM: Q\-guided value\-gradient matching for offline\-to\-online RL of flow\-matching VLA policies\.*arXiv preprint arXiv:2606\.08015*, 2026\.
- Woo & Ahn \(2024\)Dongyeop Woo and Sungsoo Ahn\.Iterated energy\-based flow matching for sampling from Boltzmann densities\.*arXiv preprint arXiv:2408\.16249*, 2024\.
- Xie et al\. \(2025\)Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu\.Simple policy optimization\.In*Forty\-second International Conference on Machine Learning*, 2025\.URL[https://openreview\.net/forum?id=SG8Yx1FyeU](https://openreview.net/forum?id=SG8Yx1FyeU)\.
- Yang et al\. \(2023\)Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin\.Policy representation via diffusion probability model for reinforcement learning\.*arXiv preprint arXiv:2305\.13122*, 2023\.
- Zakka et al\. \(2025\)Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A\. Kahrs, Carlo Sferrazza, Yuval Tassa, and Pieter Abbeel\.Mujoco playground: An open\-source framework for gpu\-accelerated robot learning and sim\-to\-real transfer\., 2025\.URL[https://github\.com/google\-deepmind/mujoco\_playground](https://github.com/google-deepmind/mujoco_playground)\.
- Zhang & Chen \(2022\)Qinsheng Zhang and Yongxin Chen\.Path integral sampler: A stochastic control approach for sampling\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=\_uCb2ynRu7Y](https://openreview.net/forum?id=_uCb2ynRu7Y)\.
- Zhang et al\. \(2025a\)Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang\.ReinFlow: Fine\-tuning flow matching policy with online reinforcement learning\.In*Advances in Neural Information Processing Systems*, 2025a\.
- Zhang et al\. \(2025b\)Yixian Zhang, Shu’ang Yu, Tonghe Zhang, Mo Guang, Haojia Hui, Kaiwen Long, Yu Wang, Chao Yu, and Wenbo Ding\.SAC Flow: Sample\-efficient reinforcement learning of flow\-based policies via velocity\-reparameterized sequential modeling\.*arXiv preprint arXiv:2509\.25756*, 2025b\.
- Zhou et al\. \(2026\)Zhiyuan Zhou, Andy Peng, Charles Xu, Qiyang Li, Tobias Springenberg, Kevin Frans, and Sergey Levine\.Test\-time gradient guidance of flow policies in reinforcement learning\.*arXiv preprint arXiv:2606\.11087*, 2026\.
- Ziakas et al\. \(2026\)Christos Ziakas, Alessandra Russo, and Avishek Joey Bose\.Aligning flow map policies with optimal Q\-guidance\.*arXiv preprint arXiv:2605\.12416*, 2026\.
- Ziebart et al\. \(2008\)Brian D\. Ziebart, Andrew Maas, J\. Andrew Bagnell, and Anind K\. Dey\.Maximum entropy inverse reinforcement learning\.In*Proceedings of the 23rd National Conference on Artificial Intelligence \- Volume 3*, AAAI’08, pp\. 1433–1438\. AAAI Press, 2008\.ISBN 9781577353683\.

Appendix contents

## Appendix AProofs

This appendix proves the three results of[Section4](https://arxiv.org/html/2610.12848#S4): the target flow identity \(Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)\), the target flow matching objective \(Corollary[2](https://arxiv.org/html/2610.12848#Thmtheorem2)\), and the fixed\-point property of FFM \(Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)\)\. We use the notation of[SectionB\.1](https://arxiv.org/html/2610.12848#A2.SS1)and suppress the statess\.

##### Proof of the target flow identity \(Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)\)

The proof combines the velocity–score bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28)with the*target score identity*of[De Bortoli et al\. \(2024\)](https://arxiv.org/html/2610.12848#bib.bib17): for a continuously differentiable densityμ\\muwith𝔼μ​\[‖∇log⁡μ​\(X1\)‖\]<∞\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\nabla\\log\\mu\(X\_\{1\}\)\\\|\\big\]<\\inftyand allt∈\(0,1\]t\\in\(0,1\],

𝗌μ​\(x,t\)=1α⁡\(t\)​𝔼μ​\[∇log⁡μ​\(X1\)\|Xt=x\],\\mathsf\{s\}^\{\\mu\}\(x,t\)\\;=\\;\\frac\{1\}\{\\alpha\(t\)\}\\,\\mathbb\{E\}\_\{\\mu\}\\big\[\\nabla\\log\\mu\(X\_\{1\}\)\\,\\big\|\\,X\_\{t\}=x\\big\],\(21\)where𝗌μ\\mathsf\{s\}^\{\\mu\}denotes the score[Equation26](https://arxiv.org/html/2610.12848#A2.E26)of the interpolant ofμ\\mu\. Whereas the noise identity[Equation27](https://arxiv.org/html/2610.12848#A2.E27)averages the noise,[Equation21](https://arxiv.org/html/2610.12848#A1.E21)averages the score of the*target*at clean samples; it follows from an integration by parts that moves the gradient from the Gaussian kernel ofρt\\rho\_\{t\}ontoμ\\mu, and the factor1/α⁡\(t\)1/\\alpha\(t\)accounts for the scaling ofX1X\_\{1\}in[Equation25](https://arxiv.org/html/2610.12848#A2.E25)\.

###### Proof of Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)\.

Apply[Equation21](https://arxiv.org/html/2610.12848#A1.E21)toμ=π⋆∝eQ⁡\(s,⋅\)/λ\\mu=\\pi^\{\\star\}\\propto e^\{Q\(s,\\cdot\)/\\lambda\}\. By[Equation7](https://arxiv.org/html/2610.12848#S4.E7),∇x​log​π⋆=∇xQ​\(s,⋅\)/λ\\nabla\_\{x\}\\log\\pi^\{\\star\}=\\nabla\_\{x\}Q\(s,\\cdot\)/\\lambda, and the integrability assumption of Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)is exactly that of the identity, so the score of the noised Boltzmann policy is

𝗌π⋆​\(x,t\)=1λ​α​\(t\)​𝔼π⋆​\[∇xQ​\(s,X1\)\|Xt=x\]\.\\mathsf\{s\}^\{\\pi^\{\\star\}\}\(x,t\)\\;=\\;\\frac\{1\}\{\\lambda\\,\\alpha\(t\)\}\\,\\mathbb\{E\}\_\{\\pi^\{\\star\}\}\\big\[\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\,\\big\|\\,X\_\{t\}=x\\big\]\.\(22\)Inserting[Equation22](https://arxiv.org/html/2610.12848#A1.E22)into the bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28)gives[Equation9](https://arxiv.org/html/2610.12848#S4.E9)\. ∎

##### Proof of target flow matching \(Corollary[2](https://arxiv.org/html/2610.12848#Thmtheorem2)\)

###### Proof of Corollary[2](https://arxiv.org/html/2610.12848#Thmtheorem2)\.

Both objectives are weighted least\-squares regressions ofvθ​\(Xt,t\)v\_\{\\theta\}\(X\_\{t\},t\)onto a random target under the interpolant ofπ⋆\\pi^\{\\star\}andt∼𝒰⁡\(0,1\)t\\sim\\mathcal\{U\}\(0,1\): flow matching[Equation6](https://arxiv.org/html/2610.12848#S3.E6)ontoX˙t\\dot\{X\}\_\{t\}, and target flow matching[Equation10](https://arxiv.org/html/2610.12848#S4.E10)onto

Y:=α˙​\(t\)α⁡\(t\)​Xt\+κ⁡\(t\)λ​α​\(t\)​∇xQ​\(s,X1\)\.Y\\;:=\\;\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\+\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\nabla\_\{x\}Q\(s,X\_\{1\}\)\.\(23\)When the minimization ranges over all velocity fields, the minimizer of a least\-squares regression is the conditional expectation of the target given the regressor, here\(Xt,t\)\(X\_\{t\},t\), for almost every\(x,t\)\(x,t\)with respect to the interpolant marginals; since the weightingw⁡\(t\)\>0w\(t\)\>0depends onttalone, it does not change this pointwise minimizer\. For flow matching the conditional expectation is𝔼π⋆​\[X˙t∣Xt=x\]=v⁡\(x,t\)\\mathbb\{E\}\_\{\\pi^\{\\star\}\}\[\\dot\{X\}\_\{t\}\\mid X\_\{t\}=x\]=v\(x,t\)by the definition[Equation4](https://arxiv.org/html/2610.12848#S3.E4)of the velocity\. For target flow matching, the first term ofYYis a function ofXtX\_\{t\}and therefore

𝔼π⋆​\[Y∣Xt=x\]=α˙​\(t\)α⁡\(t\)​x\+κ⁡\(t\)λ​α​\(t\)​𝔼π⋆​\[∇xQ​\(s,X1\)\|Xt=x\]=v⁡\(x,t\)\\mathbb\{E\}\_\{\\pi^\{\\star\}\}\[Y\\mid X\_\{t\}=x\]\\;=\\;\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,x\+\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\mathbb\{E\}\_\{\\pi^\{\\star\}\}\\big\[\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\,\\big\|\\,X\_\{t\}=x\\big\]\\;=\\;v\(x,t\)\(24\)by the target flow identity[Equation9](https://arxiv.org/html/2610.12848#S4.E9)\. Both objectives hence share the population minimizervv, for every positive weightingww\. ∎

##### Proof of the fixed point \(Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)\)

We first state Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)in full\. The regularity conditions the main text defers concern only the critic\.

###### Theorem\(Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3), full statement\)\.

LetQ⁡\(s,⋅\)Q\(s,\\cdot\)be continuously differentiable, letπ⋆∝eQ/λ\\pi^\{\\star\}\\propto e^\{Q/\\lambda\}with𝒵⁡\(s\)=∫eQ/λ​𝑑x<∞\\mathcal\{Z\}\(s\)=\\int e^\{Q/\\lambda\}\\,\\mathrm\{d\}x<\\inftyhave finite second moments and𝔼π⋆​\[‖∇xQ​\(s,X1\)‖2\]<∞\\mathbb\{E\}\_\{\\pi^\{\\star\}\}\\big\[\\\|\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\\|^\{2\}\\big\]<\\infty, and letw⁡\(t\)\>0w\(t\)\>0on\(0,1\)\(0,1\)\. Consider the iteration[Equation12](https://arxiv.org/html/2610.12848#S4.E12)in its population form, in which the regression is solved exactly over all velocity fields at everyt∈\(0,1\)t\\in\(0,1\)\. Ifπθk\(⋅∣s\)=π⋆\(⋅∣s\)\\pi\_\{\\theta\_\{k\}\}\(\\cdot\\mid s\)=\\pi^\{\\star\}\(\\cdot\\mid s\), thenvθk\+1v\_\{\\theta\_\{k\+1\}\}is the velocity fieldvvof[Equation4](https://arxiv.org/html/2610.12848#S3.E4),πθk\+1\(⋅∣s\)=π⋆\(⋅∣s\)\\pi\_\{\\theta\_\{k\+1\}\}\(\\cdot\\mid s\)=\\pi^\{\\star\}\(\\cdot\\mid s\), andvθk\+m=vθk\+1v\_\{\\theta\_\{k\+m\}\}=v\_\{\\theta\_\{k\+1\}\}for allm≥1m\\geq 1\.

The two moment conditions make the regression target of[Equation11](https://arxiv.org/html/2610.12848#S4.E11)square integrable at everyt∈\(0,1\)t\\in\(0,1\), so that the regression at each time is well posed, and they imply the integrability assumption of Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)\.

###### Proof of Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)\.

The loss[Equation11](https://arxiv.org/html/2610.12848#S4.E11)depends on the current parametersθk\\theta\_\{k\}only through the distributionπθk\(⋅∣s\)\\pi\_\{\\theta\_\{k\}\}\(\\cdot\\mid s\)of the clean actionsX1X\_\{1\}\. Forπθk=π⋆\\pi\_\{\\theta\_\{k\}\}=\\pi^\{\\star\}it therefore coincides with the target flow matching objective[Equation10](https://arxiv.org/html/2610.12848#S4.E10), whose minimizer over all velocity fields is the velocity fieldvvof the interpolant ofπ⋆\\pi^\{\\star\}by Corollary[2](https://arxiv.org/html/2610.12848#Thmtheorem2); hencevθk\+1=vv\_\{\\theta\_\{k\+1\}\}=v\. Integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)withvvtransports the prior toπ⋆\\pi^\{\\star\}\([SectionB\.1](https://arxiv.org/html/2610.12848#A2.SS1)\), soπθk\+1=π⋆\\pi\_\{\\theta\_\{k\+1\}\}=\\pi^\{\\star\}\. The next update therefore again minimizes[Equation10](https://arxiv.org/html/2610.12848#S4.E10)and returnsvv, and by induction does every later one\. ∎

##### The behavior\-tilted target

The argument applies verbatim to the offline\-to\-online objective[Equation14](https://arxiv.org/html/2610.12848#S5.E14)\. There the target isπ⋆∝pref​eQ/λ\\pi^\{\\star\}\\propto p\_\{\\mathrm\{ref\}\}\\,e^\{Q/\\lambda\}, whose terminal score isg=𝗌ref\+∇xQ/λg=\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\+\\nabla\_\{x\}Q/\\lambda, and the regression target of[Equation14](https://arxiv.org/html/2610.12848#S5.E14)isα˙​\(t\)α⁡\(t\)​Xt\+κ⁡\(t\)α⁡\(t\)​g​\(X1\)\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}X\_\{t\}\+\\tfrac\{\\kappa\(t\)\}\{\\alpha\(t\)\}\\,g\(X\_\{1\}\)\. Applying the target score identity[Equation21](https://arxiv.org/html/2610.12848#A1.E21)toμ=π⋆\\mu=\\pi^\{\\star\}, with∇log⁡π⋆=g\\nabla\\log\\pi^\{\\star\}=gsince the normalizer ofπ⋆\\pi^\{\\star\}is constant inxx, and inserting it into the bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28)shows that under sampling fromπ⋆\\pi^\{\\star\}the conditional expectation of this target is the velocity field ofπ⋆\\pi^\{\\star\}\. The proof above then goes through with∇xQ/λ\\nabla\_\{x\}Q/\\lambdareplaced bygg, under the conditions of the theorem withQQreplaced byQ\+λ​log⁡prefQ\+\\lambda\\log p\_\{\\mathrm\{ref\}\}\.

## Appendix BFurther method details

This appendix collects details of the method that the main text defers: the background on one\-sided stochastic interpolants that the main text and the proofs of[AppendixA](https://arxiv.org/html/2610.12848#A1)rely on \([SectionB\.1](https://arxiv.org/html/2610.12848#A2.SS1)\), the family of regression targets that mixes the FFM target with the flow\-matching target and its variance\-optimal member \([SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2)\), and the correlated exploration scheme of[Section5](https://arxiv.org/html/2610.12848#S5)\([SectionB\.3](https://arxiv.org/html/2610.12848#A2.SS3)\)\.

### B\.1One\-sided stochastic interpolants

This subsection collects the background on one\-sided stochastic interpolants\([Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\)that the main text and the proofs of[AppendixA](https://arxiv.org/html/2610.12848#A1)rely on: the score of the noised marginals, the denoising identities that express it as a conditional expectation, the velocity–score bridge that eliminates the score from all main\-text formulas, and the deterministic and stochastic samplers that share the marginals of the flow\. All identities are standard and we state them without proof\. Throughout,μ\\muis a probability density onℝd\\mathbb\{R\}^\{d\}\(an action distribution; the conditioning statessis suppressed\),X1∼μX\_\{1\}\\sim\\muandε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)are independent, and

Xt=α⁡\(t\)​X1\+β⁡\(t\)​ε,α⁡\(0\)=0,α⁡\(1\)=1,β⁡\(0\)=σ0,β⁡\(1\)=0,X\_\{t\}=\\alpha\(t\)X\_\{1\}\+\\beta\(t\)\\varepsilon,\\qquad\\alpha\(0\)=0,\\;\\;\\alpha\(1\)=1,\\;\\;\\beta\(0\)=\\sigma\_\{0\},\\;\\;\\beta\(1\)=0,\(25\)withα,β\\alpha,\\betacontinuously differentiable,α⁡\(t\)\>0\\alpha\(t\)\>0on\(0,1\]\(0,1\]andβ⁡\(t\)\>0\\beta\(t\)\>0on\[0,1\)\[0,1\)\. Expectations𝔼μ​\[⋅\]\\mathbb\{E\}\_\{\\mu\}\[\\cdot\]are over the pair\(X1,ε\)\(X\_\{1\},\\varepsilon\), that is, over the interpolant ofμ\\mu; they abbreviate the main\-text notation𝔼X1∼μ,ε\\mathbb\{E\}\_\{X\_\{1\}\\sim\\mu,\\,\\varepsilon\}of[Equation4](https://arxiv.org/html/2610.12848#S3.E4)\. We writeρt\\rho\_\{t\}for the density ofXtX\_\{t\}, which is the Gaussian convolutionρt​\(x\)=∫μ⁡\(x1\)​φβ2​\(t\)​\(x−α⁡\(t\)​x1\)​d​x1\\rho\_\{t\}\(x\)=\\int\\mu\(x\_\{1\}\)\\,\\varphi\_\{\\beta^\{2\}\(t\)\}\(x\-\\alpha\(t\)x\_\{1\}\)\\,\\mathrm\{d\}x\_\{1\}withφβ2​\(t\)\\varphi\_\{\\beta^\{2\}\(t\)\}the density of𝒩⁡\(⋅,0,β2​\(t\)​I\)\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\beta^\{2\}\(t\)I\\right\), and we call

𝗌⁡\(x,t\):=∇x​log​ρt​\(x\)\\mathsf\{s\}\(x,t\)\\;:=\\;\\nabla\_\{x\}\\log\\rho\_\{t\}\(x\)\(26\)the*score*of the noised marginals\. It is the second field, next to the velocity[Equation4](https://arxiv.org/html/2610.12848#S3.E4), around which the constructions of diffusion and flow models revolve\.

##### Denoising identities

SinceXt\|X1∼𝒩⁡\(⋅,α⁡\(t\)​X1,β2​\(t\)​I\)X\_\{t\}\\mid X\_\{1\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,\\alpha\(t\)X\_\{1\},\\,\\beta^\{2\}\(t\)I\\right\), the score of the Gaussian\-smoothed marginal is a conditional average of the noise, and the posterior mean of the clean sample is an affine function of the score\. Written for the interpolant[Equation25](https://arxiv.org/html/2610.12848#A2.E25), Tweedie’s formula\([Robbins, 1956](https://arxiv.org/html/2610.12848#bib.bib74);[Efron, 2011](https://arxiv.org/html/2610.12848#bib.bib30)\)reads

𝗌⁡\(x,t\)=−1β⁡\(t\)​𝔼μ​\[ε∣Xt=x\],x^1​\(x,t\):=𝔼μ​\[X1∣Xt=x\]=x\+β2​\(t\)​𝗌​\(x,t\)α⁡\(t\),\\mathsf\{s\}\(x,t\)=\-\\frac\{1\}\{\\beta\(t\)\}\\,\\mathbb\{E\}\_\{\\mu\}\[\\varepsilon\\mid X\_\{t\}=x\],\\qquad\\hat\{x\}\_\{1\}\(x,t\):=\\mathbb\{E\}\_\{\\mu\}\[X\_\{1\}\\mid X\_\{t\}=x\]=\\frac\{x\+\\beta^\{2\}\(t\)\\,\\mathsf\{s\}\(x,t\)\}\{\\alpha\(t\)\},\(27\)where the second identity follows from the first by taking the conditional expectation ofx=α⁡\(t\)​X1\+β⁡\(t\)​εx=\\alpha\(t\)X\_\{1\}\+\\beta\(t\)\\varepsilongivenXt=xX\_\{t\}=x\. We refer to the first identity as the*noise identity*and tox^1\\hat\{x\}\_\{1\}as the*denoiser*; the noise identity is also what turns score matching into the denoising regression of[Vincent \(2011\)](https://arxiv.org/html/2610.12848#bib.bib89)\.

##### The velocity–score bridge and sampling

The velocity[Equation4](https://arxiv.org/html/2610.12848#S3.E4)is the conditional mean ofX˙t=α˙​\(t\)​X1\+β˙​\(t\)​ε\\dot\{X\}\_\{t\}=\\dot\{\\alpha\}\(t\)X\_\{1\}\+\\dot\{\\beta\}\(t\)\\varepsilon, so inserting the two identities of[Equation27](https://arxiv.org/html/2610.12848#A2.E27)expresses it through the score,

v⁡\(x,t\)=α˙​\(t\)​𝔼μ​\[X1∣Xt=x\]\+β˙​\(t\)​𝔼μ​\[ε∣Xt=x\]=α˙​\(t\)α⁡\(t\)​x\+κ⁡\(t\)​𝗌​\(x,t\),v\(x,t\)=\\dot\{\\alpha\}\(t\)\\,\\mathbb\{E\}\_\{\\mu\}\[X\_\{1\}\\mid X\_\{t\}=x\]\+\\dot\{\\beta\}\(t\)\\,\\mathbb\{E\}\_\{\\mu\}\[\\varepsilon\\mid X\_\{t\}=x\]=\\frac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,x\+\\kappa\(t\)\\,\\mathsf\{s\}\(x,t\),\(28\)with the coefficientκ⁡\(t\)=α˙​\(t\)α⁡\(t\)​β2​\(t\)−β˙​\(t\)​β​\(t\)\>0\\kappa\(t\)=\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\beta^\{2\}\(t\)\-\\dot\{\\beta\}\(t\)\\beta\(t\)\>0of[Equation8](https://arxiv.org/html/2610.12848#S4.E8), and conversely

𝗌⁡\(x,t\)=1κ⁡\(t\)​\(v⁡\(x,t\)−α˙​\(t\)α⁡\(t\)​x\)\.\\mathsf\{s\}\(x,t\)\\;=\\;\\frac\{1\}\{\\kappa\(t\)\}\\Big\(v\(x,t\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,x\\Big\)\.\(29\)For the one\-sided interpolant either field therefore determines the other pointwise\([Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\): the*velocity–score bridge*\.[Equation29](https://arxiv.org/html/2610.12848#A2.E29)is how the main text reads the reference’s terminal score off a pretrained velocity field in[Equation13](https://arxiv.org/html/2610.12848#S5.E13); sinceκ⁡\(t\)→0\\kappa\(t\)\\to 0ast→1t\\to 1, the score is evaluated at a time close to, but not at, the clean endpoint\.

The marginalsρt\\rho\_\{t\}solve the continuity equation∂tρt\+∇⋅\(ρt​v\)=0\\partial\_\{t\}\\rho\_\{t\}\+\\nabla\\\!\\cdot\\\!\(\\rho\_\{t\}\\,v\)=0\([Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\), so integrating the ODE[Equation5](https://arxiv.org/html/2610.12848#S3.E5)from the priorρ0=𝒩⁡\(⋅,0,σ02​I\)\\rho\_\{0\}=\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\)transports it toρ1=μ\\rho\_\{1\}=\\mu: this is the flow policy of the main text\. The same marginals are shared by the family of SDEs

d​Xt=\(v\+σ2​\(t\)2​𝗌\)​\(Xt,t\)​d​t\+σ⁡\(t\)​d​Wt,X0∼𝒩⁡\(⋅,0,σ02​I\),\\mathrm\{d\}X\_\{t\}\\;=\\;\\Big\(v\+\\tfrac\{\\sigma^\{2\}\(t\)\}\{2\}\\,\\mathsf\{s\}\\Big\)\(X\_\{t\},t\)\\,\\mathrm\{d\}t\+\\sigma\(t\)\\,\\mathrm\{d\}W\_\{t\},\\qquad X\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\),\(30\)for any diffusion coefficientσ⁡\(t\)≥0\\sigma\(t\)\\geq 0, because the additional driftσ2​\(t\)2​𝗌\\tfrac\{\\sigma^\{2\}\(t\)\}\{2\}\\mathsf\{s\}cancels the diffusion term in the Fokker–Planck equation and leaves the continuity equation of the flow\([Song et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib84);[Albergo et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib6)\)\. A trained velocity field can hence be sampled either deterministically by[Equation5](https://arxiv.org/html/2610.12848#S3.E5)or stochastically by[Equation30](https://arxiv.org/html/2610.12848#A2.E30), with the score supplied by the bridge[Equation29](https://arxiv.org/html/2610.12848#A2.E29), and the time reversal of[Equation30](https://arxiv.org/html/2610.12848#A2.E30), with driftv−σ2​\(t\)2​𝗌v\-\\tfrac\{\\sigma^\{2\}\(t\)\}\{2\}\\mathsf\{s\}and run fromt=1t=1to00, transportsμ\\muback to the prior\([Song et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib84)\)\. We use the ODE throughout\.

### B\.2Generalized regression targets and control variates

The main text regresses the velocity head onto the target of[Equation10](https://arxiv.org/html/2610.12848#S4.E10), in which the critic gradient is, besidesXtX\_\{t\}, the only random quantity\. This subsection places that target in a one\-parameter family that mixes it with the flow\-matching target, and states the member of the family with the smallest variance\. The family is most transparent in score space, where it is the generalized target score identity of the bridge matching sampler\([Blessing et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib9)\), and transfers to velocity space through the bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28); the velocity form is what our offline\-to\-online implementation uses \([SectionE\.3](https://arxiv.org/html/2610.12848#A5.SS3)\)\. We state the results without proofs, which are given in[Blessing et al\. \(2026\)](https://arxiv.org/html/2610.12848#bib.bib9)and follow the pattern of[AppendixA](https://arxiv.org/html/2610.12848#A1)\.

##### The generalized target score identity

The noise identity[Equation27](https://arxiv.org/html/2610.12848#A2.E27)and the target score identity[Equation21](https://arxiv.org/html/2610.12848#A1.E21)express the same score𝗌μ​\(x,t\)\\mathsf\{s\}^\{\\mu\}\(x,t\)as conditional expectations of two different random variables, the scaled noise−ε/β\(t\)\-\\varepsilon/\\beta\(t\)and the scaled target score∇log⁡μ​\(X1\)/α⁡\(t\)\\nabla\\log\\mu\(X\_\{1\}\)/\\alpha\(t\)\. Any convex combination of the two is therefore again conditionally unbiased: for a mixing schedulec⁡\(t\)∈\[0,1\]c\(t\)\\in\[0,1\],

𝗌μ\(x,t\)=𝔼μ\[ζ\(X1,ε,t\)\|Xt=x\],ζ\(X1,ε,t\):=c⁡\(t\)α⁡\(t\)∇logμ\(X1\)−1−c⁡\(t\)β⁡\(t\)ε,\\mathsf\{s\}^\{\\mu\}\(x,t\)\\;=\\;\\mathbb\{E\}\_\{\\mu\}\\big\[\\zeta\(X\_\{1\},\\varepsilon,t\)\\,\\big\|\\,X\_\{t\}=x\\big\],\\qquad\\zeta\(X\_\{1\},\\varepsilon,t\)\\;:=\\;\\frac\{c\(t\)\}\{\\alpha\(t\)\}\\,\\nabla\\log\\mu\(X\_\{1\}\)\\;\-\\;\\frac\{1\-c\(t\)\}\{\\beta\(t\)\}\\,\\varepsilon,\(31\)which is the*generalized target score identity*of[Blessing et al\. \(2026\)](https://arxiv.org/html/2610.12848#bib.bib9);c≡1c\\equiv 1recovers[Equation21](https://arxiv.org/html/2610.12848#A1.E21)andc≡0c\\equiv 0the noise identity\. The two random variables have complementary variances: the target\-score term carries the factor1/α⁡\(t\)1/\\alpha\(t\), which diverges at the noise endt→0t\\to 0, and the noise term the factor1/β⁡\(t\)1/\\beta\(t\), which diverges at the clean endt→1t\\to 1\. Since the conditional mean of the noise term is known, it acts as a*control variate*for the target\-score term, and the mixing schedule can be chosen to minimize the variance of the combined estimator\. For a schedule that depends onttalone, the relevant quantity is the variance averaged overXtX\_\{t\},𝔼μ​\[‖ζ−𝗌μ​\(Xt,t\)‖2\]=𝔼μ​\[‖ζ‖2\]−𝔼μ​\[‖𝗌μ​\(Xt,t\)‖2\]\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\zeta\-\\mathsf\{s\}^\{\\mu\}\(X\_\{t\},t\)\\\|^\{2\}\\big\]=\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\zeta\\\|^\{2\}\\big\]\-\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\mathsf\{s\}^\{\\mu\}\(X\_\{t\},t\)\\\|^\{2\}\\big\], whose second term does not depend oncc\. BecauseX1X\_\{1\}andε\\varepsilonare independent, the cross term of the first vanishes and

𝔼μ​\[‖ζ⁡\(X1,ε,t\)‖2\]=d⁡\(c2​\(t\)α2​\(t\)​ℐ¯\+\(1−c⁡\(t\)\)2β2​\(t\)\),ℐ¯:=1d​𝔼μ​\[‖∇log⁡μ​\(X1\)‖2\],\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\zeta\(X\_\{1\},\\varepsilon,t\)\\\|^\{2\}\\big\]\\;=\\;d\\,\\Big\(\\frac\{c^\{2\}\(t\)\}\{\\alpha^\{2\}\(t\)\}\\,\\bar\{\\mathcal\{I\}\}\+\\frac\{\(1\-c\(t\)\)^\{2\}\}\{\\beta^\{2\}\(t\)\}\\Big\),\\qquad\\bar\{\\mathcal\{I\}\}:=\\tfrac\{1\}\{d\}\\,\\mathbb\{E\}\_\{\\mu\}\\big\[\\\|\\nabla\\log\\mu\(X\_\{1\}\)\\\|^\{2\}\\big\],\(32\)whereℐ¯\\bar\{\\mathcal\{I\}\}is the per\-coordinate Fisher information ofμ\\mu\. Minimizing[Equation32](https://arxiv.org/html/2610.12848#A2.E32)overc⁡\(t\)c\(t\)yields\([Blessing et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib9)\)

c⋆​\(t\)=α2​\(t\)α2​\(t\)\+β2​\(t\)​ℐ¯,c^\{\\star\}\(t\)\\;=\\;\\frac\{\\alpha^\{2\}\(t\)\}\{\\alpha^\{2\}\(t\)\+\\beta^\{2\}\(t\)\\,\\bar\{\\mathcal\{I\}\}\},\(33\)for any targetμ\\muwith finite Fisher information; in practice,ℐ¯\\bar\{\\mathcal\{I\}\}is estimated on the current batch\. Its shape is that of an inverse signal\-to\-noise ratio: the target\-score term dominates neart=1t=1, where samples are clean, and the noise anchor neart=0t=0\.

##### The generalized target flow identity

The velocity image of the family[Equation31](https://arxiv.org/html/2610.12848#A2.E31)has a direct reading\. Two velocity targets are available for the interpolant[Equation25](https://arxiv.org/html/2610.12848#A2.E25), the flow\-matching targetX˙t\\dot\{X\}\_\{t\}of[Equation6](https://arxiv.org/html/2610.12848#S3.E6)and the target\-flow\-identity target of[Equation10](https://arxiv.org/html/2610.12848#S4.E10), and their convex combination is

Yc​\(X1,ε,t\):=\(1−c⁡\(t\)\)​X˙t\+c⁡\(t\)​\(α˙​\(t\)α⁡\(t\)​Xt\+κ⁡\(t\)λ​α​\(t\)​∇xQ​\(s,X1\)\),c⁡\(t\)∈\[0,1\]\.Y\_\{c\}\(X\_\{1\},\\varepsilon,t\)\\;:=\\;\\big\(1\-c\(t\)\\big\)\\,\\dot\{X\}\_\{t\}\\;\+\\;c\(t\)\\,\\Big\(\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\+\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\,\\nabla\_\{x\}Q\(s,X\_\{1\}\)\\Big\),\\qquad c\(t\)\\in\[0,1\]\.\(34\)Under sampling from the Boltzmann policy, both endpoints have the same conditional mean, the velocity field generatingπ⋆\\pi^\{\\star\}\([Equation4](https://arxiv.org/html/2610.12848#S3.E4)forc≡0c\\equiv 0and Proposition[1](https://arxiv.org/html/2610.12848#Thmtheorem1)forc≡1c\\equiv 1\), which gives the*generalized target flow identity*

v⁡\(x,t∣s\)=𝔼X1∼π⋆,ε​\[Yc​\(X1,ε,t\)\|Xt=x\]for every​c​\(t\)∈\[0,1\]\.v\(x,t\\mid s\)\\;=\\;\\mathbb\{E\}\_\{X\_\{1\}\\sim\\pi^\{\\star\},\\,\\varepsilon\}\\big\[\\,Y\_\{c\}\(X\_\{1\},\\varepsilon,t\)\\,\\big\|\\,X\_\{t\}=x\\big\]\\qquad\\text\{for every \}c\(t\)\\in\[0,1\]\.\(35\)SubstitutingX˙t=α˙​\(t\)​X1\+β˙​\(t\)​ε\\dot\{X\}\_\{t\}=\\dot\{\\alpha\}\(t\)X\_\{1\}\+\\dot\{\\beta\}\(t\)\\varepsilon,Xt=α⁡\(t\)​X1\+β⁡\(t\)​εX\_\{t\}=\\alpha\(t\)X\_\{1\}\+\\beta\(t\)\\varepsilon, and the schedule identityα˙​\(t\)α⁡\(t\)​β​\(t\)=β˙​\(t\)\+κ⁡\(t\)β⁡\(t\)\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\beta\(t\)=\\dot\{\\beta\}\(t\)\+\\tfrac\{\\kappa\(t\)\}\{\\beta\(t\)\}into[Equation34](https://arxiv.org/html/2610.12848#A2.E34)shows that the family is exactly the velocity image of the score\-space targetζ\\zetaof[Equation31](https://arxiv.org/html/2610.12848#A2.E31),

Yc=α˙​\(t\)α⁡\(t\)​Xt\+κ⁡\(t\)​ζ​\(X1,ε,t\),Y\_\{c\}\\;=\\;\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\,X\_\{t\}\\;\+\\;\\kappa\(t\)\\,\\zeta\(X\_\{1\},\\varepsilon,t\),\(36\)so[Equation35](https://arxiv.org/html/2610.12848#A2.E35)is the bridge applied to[Equation31](https://arxiv.org/html/2610.12848#A2.E31)\. By[Equation35](https://arxiv.org/html/2610.12848#A2.E35), the proof of Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)carries over to the mixed target: under sampling fromπ⋆\\pi^\{\\star\}the regression ontoYcY\_\{c\}returns the velocity field ofπ⋆\\pi^\{\\star\}, soπ⋆\\pi^\{\\star\}remains a fixed point for every mixing schedulec⁡\(t\)∈\[0,1\]c\(t\)\\in\[0,1\]\. The mapζ↦Yc\\zeta\\mapsto Y\_\{c\}is affine with a shift that depends only onXtX\_\{t\}and a scaleκ⁡\(t\)\\kappa\(t\)that does not depend oncc, soVar⁡\(Yc∣Xt=x\)=κ2​\(t\)​Var​\(ζ∣Xt=x\)\\mathrm\{Var\}\(Y\_\{c\}\\mid X\_\{t\}=x\)=\\kappa^\{2\}\(t\)\\,\\mathrm\{Var\}\(\\zeta\\mid X\_\{t\}=x\)and the two families share their variance\-minimizing schedule:

c⋆​\(t\)=α2​\(t\)α2​\(t\)\+β2​\(t\)​ℐ¯,ℐ¯=1d​𝔼q​\[‖∇xQ/λ‖2\],c^\{\\star\}\(t\)\\;=\\;\\frac\{\\alpha^\{2\}\(t\)\}\{\\alpha^\{2\}\(t\)\+\\beta^\{2\}\(t\)\\,\\bar\{\\mathcal\{I\}\}\},\\qquad\\bar\{\\mathcal\{I\}\}=\\tfrac\{1\}\{d\}\\,\\mathbb\{E\}\_\{q\}\\big\[\\\|\\nabla\_\{x\}Q/\\lambda\\\|^\{2\}\\big\],\(37\)is the variance\-optimal control variate also in velocity space\. It also removes the endpoint singularity noted in[Section4](https://arxiv.org/html/2610.12848#S4): in[Equation34](https://arxiv.org/html/2610.12848#A2.E34)the critic gradient enters with the coefficientc⁡\(t\)​κ​\(t\)/\(λ​α​\(t\)\)c\(t\)\\,\\kappa\(t\)/\(\\lambda\\,\\alpha\(t\)\), and sincec⋆​\(t\)=Θ⁡\(α2​\(t\)\)c^\{\\star\}\(t\)=\\Theta\(\\alpha^\{2\}\(t\)\)ast→0t\\to 0whileα⁡\(t\)​κ​\(t\)→α˙​\(0\)​σ02<∞\\alpha\(t\)\\,\\kappa\(t\)\\to\\dot\{\\alpha\}\(0\)\\,\\sigma\_\{0\}^\{2\}<\\infty, this coefficient stays bounded at the noise end, whereasc⋆​\(t\)→1c^\{\\star\}\(t\)\\to 1ast→1t\\to 1hands the target back to the pure FFM form where actions are clean\. No time truncationt≥ϵ0t\\geq\\epsilon\_\{0\}is needed; our offline\-to\-online implementation uses this schedule \([SectionE\.3](https://arxiv.org/html/2610.12848#A5.SS3)\)\.

### B\.3Correlated exploration

This subsection expands on the correlated exploration scheme of[Section5](https://arxiv.org/html/2610.12848#S5)\.

##### The flow policy as a deterministic map

Integrating the ODE[Equation5](https://arxiv.org/html/2610.12848#S3.E5)with the learned velocity field from the prior drawX0=σ0​εX\_\{0\}=\\sigma\_\{0\}\\varepsilonup tot=1t=1is a deterministic computation onceε\\varepsilonand the statessare fixed\. We writeΨθ​\(ε∣s\)\\Psi\_\{\\theta\}\(\\varepsilon\\mid s\)for the resulting clean action, so thatΨθ\(⋅∣s\)\\Psi\_\{\\theta\}\(\\cdot\\mid s\)is the sampler of the flow policy: it maps a prior draw to an action, andπθ\(⋅∣s\)\\pi\_\{\\theta\}\(\\cdot\\mid s\)is the law ofΨθ​\(ε∣s\)\\Psi\_\{\\theta\}\(\\varepsilon\\mid s\)forε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)\. In particular, an action is a sample ofπθ\(⋅∣s\)\\pi\_\{\\theta\}\(\\cdot\\mid s\)whenever the prior draw it is computed from is distributed according to𝒩⁡\(⋅,0,I\)\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\), regardless of how that draw was generated\.

##### The Ornstein–Uhlenbeck chain is stationary

The chain[Equation20](https://arxiv.org/html/2610.12848#S5.E20)is a first\-order autoregressive process with coefficienta:=e−1/τou∈\(0,1\)a:=e^\{\-1/\\tau\_\{\\mathrm\{ou\}\}\}\\in\(0,1\), that is,εk\+1=a​εk\+1−a2​ξk\\varepsilon\_\{k\+1\}=a\\,\\varepsilon\_\{k\}\+\\sqrt\{1\-a^\{2\}\}\\,\\xi\_\{k\}\. Ifεk∼𝒩⁡\(⋅,0,I\)\\varepsilon\_\{k\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)andξk∼𝒩⁡\(⋅,0,I\)\\xi\_\{k\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)are independent, thenεk\+1\\varepsilon\_\{k\+1\}is Gaussian with mean zero and covariance

𝔼⁡\[εk\+1​εk\+1⊤\]=a2​I\+\(1−a2\)​I=I,\\mathbb\{E\}\\big\[\\varepsilon\_\{k\+1\}\\varepsilon\_\{k\+1\}^\{\\top\}\\big\]\\;=\\;a^\{2\}\\,I\+\(1\-a^\{2\}\)\\,I\\;=\\;I,\(38\)soεk\+1∼𝒩⁡\(⋅,0,I\)\\varepsilon\_\{k\+1\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\)as well\. Sinceε0∼𝒩⁡\(⋅,0,I\)\\varepsilon\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\), induction overkkshows that everyεk\\varepsilon\_\{k\}is marginally standard normal, and by the previous paragraph every executed actionxk=Ψθ​\(εk∣sk\)x\_\{k\}=\\Psi\_\{\\theta\}\(\\varepsilon\_\{k\}\\mid s\_\{k\}\)is a sample ofπθ\(⋅∣sk\)\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{k\}\)\. What changes is the dependence between draws: unrolling the recursion gives

𝔼\[εk\+jεk⊤\]=ajI=e−j/τouI,j≥0,\\mathbb\{E\}\\big\[\\varepsilon\_\{k\+j\}\\varepsilon\_\{k\}^\{\\top\}\\big\]\\;=\\;a^\{j\}\\,I\\;=\\;e^\{\-j/\\tau\_\{\\mathrm\{ou\}\}\}\\,I,\\qquad j\\geq 0,\(39\)so the correlation between two prior draws decays exponentially with their distance in time, withτou\\tau\_\{\\mathrm\{ou\}\}steps as the time constant\. The two limits recover familiar schemes:τou→0\\tau\_\{\\mathrm\{ou\}\}\\to 0givesa→0a\\to 0and independent draws at every step, which is the usual way of sampling the policy, andτou→∞\\tau\_\{\\mathrm\{ou\}\}\\to\\inftygivesa→1a\\to 1and a single drawε0\\varepsilon\_\{0\}that is kept for the whole episode, similar to the state\-dependent exploration of[Raffin et al\. \(2022\)](https://arxiv.org/html/2610.12848#bib.bib71)\.

## Appendix CExtended related work

This appendix complements[Section2](https://arxiv.org/html/2610.12848#S2)\. The first paragraph places FFM in the family of matching objectives that train diffusion and flow models from an unnormalized target rather than from samples, which is where its fixed\-point structure comes from\. The second collects work on critic\-driven updates of diffusion and flow policies that[Section2](https://arxiv.org/html/2610.12848#S2)groups but has no room to discuss individually\.

##### Fixed\-point methods for diffusion and flow models\.

Training a diffusion or flow model from an unnormalized density instead of samples is the problem of neural samplers\. Early approaches minimize a path\-space divergence by simulating the generative process\([Zhang & Chen, 2022](https://arxiv.org/html/2610.12848#bib.bib99);[Vargas et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib88)\)\. Matching\-based samplers instead regress onto a target computed from the energy and its gradient: iDEM\([Akhound\-Sadegh et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib4)\), iEFM\([Woo & Ahn, 2024](https://arxiv.org/html/2610.12848#bib.bib95)\), BNEM\([OuYang et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib64)\), and EWFM\([Dern et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib18)\)estimate the noised score, velocity, or energy by importance\-weighted Monte Carlo, target score matching\([De Bortoli et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib17)\), on which FFM builds, writes the noised score as a conditional expectation of the clean\-data score but requires samples of the target, and tilt matching\([Potaptchik et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib69)\)matches the velocity of a reward\-tilted interpolant without reward gradients\. These methods are iterative: the target is evaluated with samples of the current model and the regression is repeated, which is the fixed\-point structure that Theorem[3](https://arxiv.org/html/2610.12848#Thmtheorem3)makes explicit for critic\-driven targets\. Adjoint matching\([Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib23)\)forms a second family of such regressions\. Proposed for reward fine\-tuning, it has since been used for sampling from unnormalized densities\([Havens et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib41);[Liu et al\., 2025a](https://arxiv.org/html/2610.12848#bib.bib53);[Mosso et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib61)\), for discrete generative models\([So et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib83)\), for faster fine\-tuning pipelines\([Guo et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib37);[Shin et al\., 2026a](https://arxiv.org/html/2610.12848#bib.bib80)\), and in RL for policy extraction\([Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49);[Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27);[Ghanem & Ghogho, 2026](https://arxiv.org/html/2610.12848#bib.bib35)\)and maximum entropy RL\([Thilges et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib85)\); its theory has been connected to the stochastic maximum principle\([Domingo\-Enrich & Han, 2026](https://arxiv.org/html/2610.12848#bib.bib22)\)and unified with sampling\([Domingo\-Enrich et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib24)\)\. The bridge matching sampler\([Blessing et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib9)\)shows that these matching objectives, including adjoint sampling and ASBS, are fixed\-point iterations rooted in Nelson’s relation and stabilizes them by damping\. FFM belongs to this family with two differences: the target flow identity provides the regression target from critic gradients at clean actions, so no adjoint ODE is needed, and the target itself moves as the critic is learned\.

##### Further critic\-driven policy updates\.

Sorted by how the critic enters the actor update, the methods of[Section2](https://arxiv.org/html/2610.12848#S2)fall into a few families, and we record here the entries the main text omits\. Methods based on backpropagation through time differentiate the critic through the entire sampling chain: besides those cited in[Section2](https://arxiv.org/html/2610.12848#S2),[Lv et al\. \(2025\)](https://arxiv.org/html/2610.12848#bib.bib57)constrain the update relative to a behavior policy\. GenPO\([Ding et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib20)\)instead applies a PPO\-style objective\([Schulman et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib78)\)to a diffusion policy whose likelihood is made tractable by a coupling construction\. A second family avoids the chain by regressing onto improved*actions*: DIPO\([Yang et al\., 2023](https://arxiv.org/html/2610.12848#bib.bib97)\)and[Li et al\. \(2024\)](https://arxiv.org/html/2610.12848#bib.bib50)improve the actions stored in the replay buffer by gradient ascent onQQand then clone them with the standard denoising loss\. This is the closest relative of FFM among the action\-target methods, and the difference is that its regression target is an action rather than a velocity, so the noise level of the interpolant is not accounted for\. A third family leaves the denoising loss in place and weights it by the exponentiated critic value\([Ding et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib19);[Ma et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib59);[Dong et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib25)\), of which one\-step flow policy mirror descent\([Chen et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib15)\)is the one\-step variant, and flow map policies trained by mirror descent\([Dong et al\., 2026a](https://arxiv.org/html/2610.12848#bib.bib26);[Gao et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib34)\)are the multimodal ones\. None of these families obtains its target from the velocity field of the Boltzmann policy itself, which is what the target flow identity provides\.

## Appendix DMaximum\-entropy reinforcement learning

This section provides information on FFM’s implementation in the online on\-policy maximum\-entropy RL setting as proposed in[Section5](https://arxiv.org/html/2610.12848#S5)\. We review the maximum\-entropy actor\-critic variant similar to that of[Haarnoja et al\. \(2018\)](https://arxiv.org/html/2610.12848#bib.bib39), where we optimize the state\-action value function and the velocity field’s parameters\. Moreover, we give an overview of how FFM resolves bounding final actions that are executed in the environment, provide an algorithm box, report ablations on design choices, and provide detailed per\-task results\.

Figure 4:Maximum Entropy Online RL: MJX Playground tasks\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds\. On most of the tasks, FFM performs similarly to or on par with baselines\. However, we observe that there are some tasks such asAcrobotSwingupSparse, HopperHop, HopperStand, where FFM \(and most of the baselines\) perform badly, indicating that a more sophisticated exploration strategy is required\. On a subset of the tasks, we have therefore run FFM with correlated exploration \(FFM \+ OU\), where we show the chosenτou\\tau\_\{\\mathrm\{ou\}\}hyperparameter on the respective plot\. Please note that correlated exploration is only possible due to FFM’s deterministic mapping \([Section5](https://arxiv.org/html/2610.12848#S5)\) Correlated exploration can help improve performance, such as inHopperHop, or even more significantly in theHopperStandtask\. The aggregated results when using the correlated exploration curves whenever applicable are given in[Figure3](https://arxiv.org/html/2610.12848#S6.F3)\(left two panels\)\.Figure 5:Maximum Entropy Online RL: MJX Humanoid tasks\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds\. FFM converges faster to a similar performance level as TRUDI and performs significantly better than TRUDI on theT1JoystickRoughTerraintask\. The high\-dimensional humanoid tasks showcase FFM’s benefits more clearly\.### D\.1Actor\-Critic based Maximum\-entropy reinforcement learning

In maximum\-entropy RL\([Ziebart et al\., 2008](https://arxiv.org/html/2610.12848#bib.bib104);[Levine, 2018](https://arxiv.org/html/2610.12848#bib.bib48);[Haarnoja et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib38);[Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39)\), the rewards are augmented with the current policy’s entropy, leading to the objective

J\(π\)=𝔼π\[∑k≥0γk\(r\(sk,xk\)\+λℋ\(π\(⋅∣sk\)\)\)\],J\(\\pi\)\\;=\\;\\mathbb\{E\}\_\{\\pi\}\\Big\[\\textstyle\\sum\_\{k\\geq 0\}\\gamma^\{k\}\\big\(r\(s\_\{k\},x\_\{k\}\)\+\\lambda\\,\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{k\}\)\)\\big\)\\Big\],\(40\)whereλ\>0\\lambda\>0trades off maximizing the expected task return against maximizing the policy’s stochasticity by maximizing the entropyℋ\(π\(⋅∣sk\)\)\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{k\}\)\)\. The soft state\-action value function is given as

Qπ\(s,x\)=𝔼π\[∑k≥0γk\(r\(sk,xk\)\+λℋ\(π\(⋅∣sk\)\)\)∣s0=s,x0=x\],Q^\{\\pi\}\(s,x\)=\\mathbb\{E\}\_\{\\pi\}\\big\[\\sum\_\{k\\geq 0\}\\gamma^\{k\}\\big\(r\(s\_\{k\},x\_\{k\}\)\+\\lambda\\,\\mathcal\{H\}\(\\pi\(\\cdot\\mid s\_\{k\}\)\)\\big\)\\mid s\_\{0\}=s,\\,x\_\{0\}=x\\big\],\(41\)which can also be expressed using the recursive Bellman equation as

Qπ​\(s,x\)=r⁡\(s,x\)\+γ​𝔼s′∼P,x′∼π​\[Qπ​\(s′,x′\)−λ​log⁡π⁡\(x′∣s′\)\]\.Q^\{\\pi\}\(s,x\)=r\(s,x\)\+\\gamma\\,\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P,x^\{\\prime\}\\sim\\pi\}\\big\[Q^\{\\pi\}\(s^\{\\prime\},x^\{\\prime\}\)\-\\lambda\\log\\pi\(x^\{\\prime\}\\mid s^\{\\prime\}\)\\big\]\.\(42\)Based on this equation, we can then update the parameters ofQπQ^\{\\pi\}using the Bellman residual,

JQ​\(ϕ\)=𝔼⁡\[12​\(Qϕπ​\(s,x\)−y\)2\]J\_\{Q\}\(\\phi\)=\\mathbb\{E\}\\left\[\\frac\{1\}\{2\}\\left\(Q\_\{\\phi\}^\{\\pi\}\(s,x\)\-y\\right\)^\{2\}\\right\]\(43\)whereϕ\\phiare the parameters of the critic network that we have omitted so far to avoid cluttered notation, andyyare the critic targets, that are calculated by

y=r⁡\(s,x\)\+γ​𝔼s′∼p,x′∼π​\[Qϕ¯π​\(s′,x′\)−λ​log⁡π⁡\(x′∣s′\)\]y=r\(s,x\)\+\\gamma\\mathbb\{E\}\_\{s^\{\\prime\}\\sim p,x^\{\\prime\}\\sim\\pi\}\\left\[Q\_\{\\bar\{\\phi\}\}^\{\\pi\}\(s^\{\\prime\},x^\{\\prime\}\)\-\\lambda\\log\\pi\(x^\{\\prime\}\\mid s^\{\\prime\}\)\\right\]\(44\)for the one\-step lookahead, mainly used in the off\-policy RL setting\([Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39);[Fujimoto et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib33)\)\. Here,ϕ¯\\bar\{\\phi\}denotes the target network parameters, a Polyak averaging of the actual parametersϕ\\phi\.

In the on\-policy setting, commonly no target network is used, and the target is calculated using N\-step returns based on a Monte Carlo estimate with samples from the current rollout and the current target as

y=∑k=tNγk−t​\(r⁡\(sk,xk\)−λ​log⁡π⁡\(xk∣sk\)\)\+γN\+1​Qϕ​\(sN,xN\),y=\\sum\_\{k=t\}^\{N\}\\gamma^\{k\-t\}\\left\(r\(s\_\{k\},x\_\{k\}\)\-\\lambda\\log\\pi\(x\_\{k\}\\mid s\_\{k\}\)\\right\)\+\\gamma^\{N\+1\}Q\_\{\\phi\}\(s\_\{N\},x\_\{N\}\),\(45\)as suggested, for example, by[Voelcker et al\. \(2026\)](https://arxiv.org/html/2610.12848#bib.bib90)and used in our online on\-policy methods\. Note that the gradients for the critic’s parametersϕ\\phiare stopped for calculating the targets in[Equation45](https://arxiv.org/html/2610.12848#A4.E45)\. For calculating the entropy and the likelihoods of the current policy, we follow[Equation15](https://arxiv.org/html/2610.12848#S5.E15)\. Once the parameters of the Q\-function are updated in the policy evaluation step, the parameters of the policyπ\\piare updated in the policy improvement step such that the expected soft Q\-function is maximized\. In the standard case, e\.g\., with Gaussian policies, the objective for the policy improvement step follows from

L\(π\)=DKL\(π\(x∣s\)∥exp⁡\(Q⁡\(s,x\)/λ\)𝒵⁡\(s\)\),L\(\\pi\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(x\\mid s\)\\,\\\|\\,\\frac\{\\exp\\big\(Q\(s,x\)/\\lambda\\big\)\}\{\\mathcal\{Z\}\(s\)\}\\right\),\(46\)where𝒵⁡\(s\)=∫exp⁡\(Q⁡\(s,x\)/λ\)​𝑑x\\mathcal\{Z\}\(s\)=\\int\\exp\\big\(Q\(s,x\)/\\lambda\\big\)\\,\\mathrm\{d\}xis the normalizing constant and independent of the actionxx\. It can be shown that alternating both steps converges to the optimal maximum\-entropy policy in the tabular setting\([Haarnoja et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib38)\)\. For FFM to optimize the flow policy, we rely on[Equation17](https://arxiv.org/html/2610.12848#S5.E17)\.

##### Enforcing action bounds

In RL benchmarks and potentially in real\-world use cases, the applied actions are bounded due to physical constraints such as bounded motor torques\. FFM squashes the generated actions to\[−1,1\]d\[\-1,1\]^\{d\}, to avoid querying the Q\-function at unreasonable action regions\. However, please note that we keep the flow unconstrained onℝd\\mathbb\{R\}^\{d\}, such that the theory does not change, but we map the flow’s clean outputX1X\_\{1\}to the environment action through a coordinate\-wise squashing functiongg

xenv=g⁡\(X1\),x\_\{\\mathrm\{env\}\}=g\(X\_\{1\}\),\(47\)where, in the literature\([Haarnoja et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib39)\), thetanh\\tanhfunction is used as

g⁡\(z\)=tanh⁡\(z\),g\(z\)=\\tanh\(z\),\(48\)with the log derivative

log⁡gtanh′​\(z\)=log⁡\(1−tanh2⁡z\)\.\\log g^\{\\prime\}\_\{\\tanh\}\(z\)=\\log\\big\(1\-\\tanh^\{2\}z\\big\)\.\(49\)Recently,[Thilges et al\. \(2026\)](https://arxiv.org/html/2610.12848#bib.bib85)proposed an alternative squashing function and reported that using the error function

g⁡\(z\)=erf⁡\(z/2\)g\(z\)=\\mathrm\{erf\}\(z/\\sqrt\{2\}\)\(50\)yields improved numerical behavior\. Its log derivative is given by

log⁡gerf′​\(z\)=12​log⁡2π−z22\.\\log g^\{\\prime\}\_\{\\mathrm\{erf\}\}\(z\)=\\tfrac\{1\}\{2\}\\log\\tfrac\{2\}\{\\pi\}\-\\tfrac\{z^\{2\}\}\{2\}\.\(51\)
However, please note that squashing the actions changes the likelihood calculations by the change\-of\-variables rule, such that the likelihood of the new distribution is

πenv\(⋅∣s\)=π\(⋅∣s\)\|det\(d​xenvd​x\)\|−1,\\pi\_\{\\mathrm\{env\}\}\(\\cdot\\mid s\)=\\pi\(\\cdot\\mid s\)\\Biggl\|\\mathrm\{det\}\\left\(\\frac\{dx\_\{\\mathrm\{env\}\}\}\{dx\}\\right\)\\Biggr\|^\{\-1\},\(52\)which leads to the log\-likelihoods needed for calculating the entropy values as

logπenv\(⋅∣s\)=logπ\(⋅∣s\)−∑ilogg′\(X1,i\),\\log\\pi\_\{\\mathrm\{env\}\}\(\\cdot\\mid s\)=\\log\\pi\(\\cdot\\mid s\)\-\\sum\_\{i\}\\log g^\{\\prime\}\(X\_\{1,i\}\),\(53\)where, depending on which squashing function is used, either[Equation49](https://arxiv.org/html/2610.12848#A4.E49)or[Equation51](https://arxiv.org/html/2610.12848#A4.E51)is used for the Jacobian in[Equation52](https://arxiv.org/html/2610.12848#A4.E52)\. This log\-likelihood is required to estimate the entropy \([Equation15](https://arxiv.org/html/2610.12848#S5.E15)\) over a batch of samples during the rollout to calculate the soft Q\-function in maximum\-entropy RL\.

Because the FFM target from[Equation10](https://arxiv.org/html/2610.12848#S4.E10)is w\.r\.t\. the clean actionX1X\_\{1\}rather than the squashed actionxenvx\_\{\\mathrm\{env\}\}, a consequence of squashing the clean actions is that, by the chain rule, the critic’s gradient w\.r\.t\.X1\{X\_\{1\}\}is

∇X1Q​\(s,g⁡\(X1\)\)=g′​\(X1\)⊙∇xenvQ​\(s,xenv\),\\nabla\_\{X\_\{1\}\}Q\(s,g\(X\_\{1\}\)\)=g^\{\\prime\}\(X\_\{1\}\)\\odot\\nabla\_\{x\_\{\\mathrm\{env\}\}\}Q\(s,x\_\{\\mathrm\{env\}\}\),\(54\)where⊙\\odotdenotes coordinate\-wise multiplication\. In our experiments, we have used the error function as the squashing function, although, as shown in our ablation studies, the tanh squashing only performs slightly worse \([SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)\)\.

### D\.2Pseudocode

Algorithm 1Online on\-policy maximum\-entropy RL with an FFM actor1:input:Initialized parameters for velocity field

vθv\_\{\\theta\}, critic

QϕQ\_\{\\phi\}, temperature

λ\\lambdawith entropy target

ℋ⋆\\mathcal\{H\}^\{\\star\}, trust\-region budget

εTR\\varepsilon\_\{\\mathrm\{TR\}\}with multiplier

ν\\nu, schedules

α,β\\alpha,\\beta\([Equation57](https://arxiv.org/html/2610.12848#A4.E57)\), time weighting

ww
2:foreach iterationdo

3:rollout \(on\-policy, no gradients\):

4:• act

xk∼πθ\(⋅∣sk\)x\_\{k\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{k\}\)by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)

5:• optionally, correlate the prior draws via the OU chain of[Equation20](https://arxiv.org/html/2610.12848#S5.E20)

6:• collect

\{\(sk,xk,rk,sk\+1\)\}k=tN\\\{\(s\_\{k\},x\_\{k\},r\_\{k\},s\_\{k\+1\}\)\\\}\_\{k=t\}^\{N\}
7:likelihoods:

log⁡πθ​\(xk∣sk\)\\log\\pi\_\{\\theta\}\(x\_\{k\}\\mid s\_\{k\}\)by[Equation15](https://arxiv.org/html/2610.12848#S5.E15), divergences by Hutchinson’s estimator

8:policy evaluation:calculate

NN\-step soft returns of[Equation45](https://arxiv.org/html/2610.12848#A4.E45)\(target gradients stopped\)

9:

yt=∑k=tNγk−t​\(r⁡\(sk,xk\)−λ​log⁡πθ​\(xk∣sk\)\)\+γN\+1​Qϕ​\(sN,xN\)\\displaystyle y\_\{t\}=\\sum\_\{k=t\}^\{N\}\\gamma^\{\\,k\-t\}\\bigl\(r\(s\_\{k\},x\_\{k\}\)\-\\lambda\\log\\pi\_\{\\theta\}\(x\_\{k\}\\mid s\_\{k\}\)\\bigr\)\+\\gamma^\{\\,N\+1\}Q\_\{\\phi\}\(s\_\{N\},x\_\{N\}\)
10:foreach critic epoch over minibatches of the rolloutdo

11:update critic parameters

ϕ\\phiusing

\(Qϕ​\(st,xt\)−yt\)2\\bigl\(Q\_\{\\phi\}\(s\_\{t\},x\_\{t\}\)\-y\_\{t\}\\bigr\)^\{2\}, or if distributional using HL\-Gauss\([Farebrother et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib32);[Voelcker et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib90)\)

12:Calculate Q\-scores:

∇xQϕ​\(s,X1\)\\nabla\_\{x\}Q\_\{\\phi\}\(s,X\_\{1\}\)at the rollout actions

X1∼πθ¯\(⋅∣s\)X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid s\)
13:foreach actor epoch over minibatches of the rolloutdo

14:

t∼𝒰⁡\(0,1\)t\\sim\\mathcal\{U\}\(0,1\),

ε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\),

Xt←α⁡\(t\)​X1\+β⁡\(t\)​εX\_\{t\}\\leftarrow\\alpha\(t\)X\_\{1\}\+\\beta\(t\)\\varepsilon
15:minimize over

θ\\thetausing[Equation18](https://arxiv.org/html/2610.12848#S5.E18):

16:

𝔼t,X1,ε​\[w⁡\(t\)​‖vθ​\(Xt,t∣s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)λ​α​\(t\)​∇xQϕ​\(s,X1\)‖2\]\+ν⁡\(ℒTR​\(θ,θ¯\)−εTR\)\\displaystyle\\mathbb\{E\}\_\{t,X\_\{1\},\\varepsilon\}\\Bigl\[w\(t\)\\Bigl\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\lambda\\,\\alpha\(t\)\}\\nabla\_\{x\}Q\_\{\\phi\}\(s,X\_\{1\}\)\\Bigr\\\|^\{2\}\\Bigr\]\+\\nu\\bigl\(\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\bar\{\\theta\}\)\-\\varepsilon\_\{\\mathrm\{TR\}\}\\bigr\)
17:with

ℒTR\\mathcal\{L\}\_\{\\mathrm\{TR\}\}of[Equation16](https://arxiv.org/html/2610.12848#S5.E16)evaluated at the same points

\(Xt,t\)\(X\_\{t\},t\)
18:duals:

19:• ascend

ν\\nu\([Equation19](https://arxiv.org/html/2610.12848#S5.E19)\)

20:• descend

λ\\lambdaon

λ⁡\(𝔼⁡\[ℋ^\]−ℋ⋆\)\\lambda\\bigl\(\\mathbb\{E\}\\bigl\[\\widehat\{\\mathcal\{H\}\}\\bigr\]\-\\mathcal\{H\}^\{\\star\}\\bigr\), so

λ\\lambdagrows while the entropy is below

ℋ⋆\\mathcal\{H\}^\{\\star\}
21:update the trust\-region reference:

θ¯←θ\\bar\{\\theta\}\\leftarrow\\theta
22:evaluation:integrate[Equation5](https://arxiv.org/html/2610.12848#S3.E5)from

X0=0X\_\{0\}=0\([SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)\)

### D\.3Experimental details

##### Environments and tasks\.

In our online on\-policy RL experiments from[Section6](https://arxiv.org/html/2610.12848#S6)we have evaluated FFM on the GPU\-accelerated MuJoCo Playground\([Zakka et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib98)\)tasks\. More concretely, we distinguish between the 20 DeepMind Control \(DMC\) tasks of the suite, whose episode returns are bounded between\[0,1000\]\[0,1000\], and the four MJX humanoid joystick tasks, which require precise and high\-dimensional whole\-body control and whose returns are not normalized\.

For every learning curve, we follow[Agarwal et al\. \(2021\)](https://arxiv.org/html/2610.12848#bib.bib3)and report the interquartile mean \(IQM\) of the return with stratified bootstrapped95%95\\%confidence intervals over 10 seeds per task\. Furthermore, in[Figure3](https://arxiv.org/html/2610.12848#S6.F3)the aggregated results over all tasks for the Playground and the Humanoid tasks are reported, where the per\-task results are shown in[SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)\.

##### Baselines\.

We compare against Gaussian\-policy methods, REPPO\([Voelcker et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib90)\), PPO\([Schulman et al\., 2017](https://arxiv.org/html/2610.12848#bib.bib78)\), and SPO\([Xie et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib96)\), diffusion\-policy methods, TRUDI\([Le et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib47)\), AMDP\([Thilges et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib85)\), DIME\([Celik et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib11)\), and DPPO\([Ren et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib72)\), and the flow\-policy method FPO\([McAllister et al\., 2026](https://arxiv.org/html/2610.12848#bib.bib60)\)\.

##### Hyperparameters\.

[Table1](https://arxiv.org/html/2610.12848#A4.T1)lists the hyperparameters of FFM and the baselines\. Notably, for FFM the flow starts from the priorX0∼𝒩⁡\(⋅,0,σ02​I\)X\_\{0\}\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,\\sigma\_\{0\}^\{2\}I\\right\)withσ0=1\\sigma\_\{0\}=1under the geometric schedule of[Equation57](https://arxiv.org/html/2610.12848#A4.E57)and is integrated with1616Euler steps\. Furthermore, FFM employs action squashing using the error function map from[SectionD\.1](https://arxiv.org/html/2610.12848#A4.SS1)and the entropy estimate[Equation15](https://arxiv.org/html/2610.12848#S5.E15)from[Section5](https://arxiv.org/html/2610.12848#S5)with four Rademacher probes per integration node\.

##### Implementation details\.

*Distributional critic\.*As done in most baselines and related works recently, we parameterize the critic distributionally following HL\-Gauss\([Farebrother et al\., 2024](https://arxiv.org/html/2610.12848#bib.bib32)\), trained by cross\-entropy on the same targets as[Equation45](https://arxiv.org/html/2610.12848#A4.E45)\.

*Actor regression\.*The implemented policy\-improvement step is the FFM regression[Equation11](https://arxiv.org/html/2610.12848#S4.E11)with the regression target replaced by the variance\-optimal mixture of[Equation34](https://arxiv.org/html/2610.12848#A2.E34)and the time weighting fixed tow⁡\(t\)=1/κ2​\(t\)w\(t\)=1/\\kappa^\{2\}\(t\),

ℒ⁡\(θ,θ¯\)=𝔼t,X1∼πθ¯,ε​\[w⁡\(t\)​‖vθ​\(Xt,t∣s\)−Yc⋆​\(X1,ε,t\)‖2\],w⁡\(t\)=1κ2​\(t\),\\mathcal\{L\}\(\\theta,\\bar\{\\theta\}\)\\;=\\;\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\},\\,\\varepsilon\}\\Big\[w\(t\)\\,\\big\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-Y\_\{c^\{\\star\}\}\(X\_\{1\},\\varepsilon,t\)\\big\\\|^\{2\}\\Big\],\\qquad w\(t\)\\;=\\;\\frac\{1\}\{\\kappa^\{2\}\(t\)\},\(55\)whereYc⋆Y\_\{c^\{\\star\}\}is the velocity target[Equation34](https://arxiv.org/html/2610.12848#A2.E34)at the variance\-optimal mixing schedule of[Equation37](https://arxiv.org/html/2610.12848#A2.E37)\. The variance\-optimal mixing schedule is given asc⋆​\(t\)=α2​\(t\)/\(α2​\(t\)\+β2​\(t\)​ℐ¯\)c^\{\\star\}\(t\)=\\alpha^\{2\}\(t\)/\\big\(\\alpha^\{2\}\(t\)\+\\beta^\{2\}\(t\)\\,\\bar\{\\mathcal\{I\}\}\\big\)\([Equation37](https://arxiv.org/html/2610.12848#A2.E37)\), but we fixed the Fisher scale toℐ¯=1\\bar\{\\mathcal\{I\}\}=1, such that we obtainc⋆​\(t\)=α2​\(t\)/\(α2​\(t\)\+β2​\(t\)\)c^\{\\star\}\(t\)=\\alpha^\{2\}\(t\)/\\big\(\\alpha^\{2\}\(t\)\+\\beta^\{2\}\(t\)\\big\)\. Since the generated actions are squashed \([SectionD\.1](https://arxiv.org/html/2610.12848#A4.SS1)\), the Q\-score∇xQ​\(s,X1\)/λ\\nabla\_\{x\}Q\(s,X\_\{1\}\)/\\lambdainYc⋆Y\_\{c^\{\\star\}\}becomes∇x\[Q⁡\(s,g⁡\(x\)\)/λ\+log⁡\|det∂⁡g⁡\(x\)\|\]\|x=X1\\nabla\_\{x\}\\big\[Q\(s,g\(x\)\)/\\lambda\+\\log\|\{\\det\\partial g\(x\)\}\|\\big\]\\big\|\_\{x=X\_\{1\}\}, which is the score of the squashed targets\. Forc⋆=1c^\{\\star\}=1,[Equation55](https://arxiv.org/html/2610.12848#A4.E55)is[Equation11](https://arxiv.org/html/2610.12848#S4.E11)\. Please note that multiplying the residual byw⁡\(t\)=1/κ⁡\(t\)\\sqrt\{w\(t\)\}=1/\\kappa\(t\)converts the velocity difference into a score difference through the bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28), so[Equation55](https://arxiv.org/html/2610.12848#A4.E55)is the score\-field regression of[SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2)\. The regression[Equation55](https://arxiv.org/html/2610.12848#A4.E55)in its score\-field form is

ℒ\(θ,θ¯\)=𝔼t,X1∼πθ¯,ε\[∥α⁡\(t\)​\(vθ​\(Xt,t∣s\)−α˙​\(t\)​X1\)\+a⁡\(t\)​εα⁡\(t\)​κ​\(t\)−c⋆​\(t\)α⁡\(t\)∇x\[1λQ\(s,g\(x\)\)\+log\|det∂g\(x\)\|\]\|x=X1∥2\],\\mathcal\{L\}\(\\theta,\\bar\{\\theta\}\)\\;=\\;\\mathbb\{E\}\_\{t,\\,X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\},\\,\\varepsilon\}\\bigg\[\\,\\Big\\\|\\frac\{\\alpha\(t\)\\,\\bigl\(v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\dot\{\\alpha\}\(t\)\\,X\_\{1\}\\bigr\)\+a\(t\)\\,\\varepsilon\}\{\\alpha\(t\)\\,\\kappa\(t\)\}\\\\ \-\\frac\{c^\{\\star\}\(t\)\}\{\\alpha\(t\)\}\\,\\nabla\_\{x\}\\Big\[\\tfrac\{1\}\{\\lambda\}Q\\big\(s,g\(x\)\\big\)\+\\log\|\{\\det\\partial g\(x\)\}\|\\Big\]\\Big\|\_\{x=X\_\{1\}\}\\Big\\\|^\{2\}\\,\\bigg\],\(56\)with the coefficienta⁡\(t\):=−\(c⋆​α˙​β\+\(1−c⋆\)​α​β˙\)a\(t\):=\-\\big\(c^\{\\star\}\\dot\{\\alpha\}\\beta\+\(1\-c^\{\\star\}\)\\,\\alpha\\dot\{\\beta\}\\big\)\. However, since1/κ⁡\(t\)1/\\kappa\(t\)diverges ast→1t\\to 1for the linear schedule and1/α⁡\(t\)1/\\alpha\(t\)diverges ast→0t\\to 0, the implementation softens the two denominators of[Equation56](https://arxiv.org/html/2610.12848#A4.E56),α​κ→α​κ\+δ​σ02\\alpha\\kappa\\to\\alpha\\kappa\+\\delta\\sigma\_\{0\}^\{2\}andα→α\+δ\\alpha\\to\\alpha\+\\delta, by the constantδ=0\.05\\delta=0\.05\.

*Deterministic evaluation\.*During evaluation, we found that starting the flow’s integration from the central pointX0=0X\_\{0\}=0instead of a random prior draw yields improved performance \([SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)\); training rollouts keep stochastic prior sampling\.

##### Interpolant schedule\.

We instantiate the schedulesα,β\\alpha,\\betawith a*geometric*schedule\. Given interpolation scalesσmax\>σmin\>0\\sigma\_\{\\max\}\>\\sigma\_\{\\min\}\>0with ratior=σmax/σminr=\\sigma\_\{\\max\}/\\sigma\_\{\\min\}, we writeτ⁡\(t\)=σmax2​\(1−r−2​t\)\\tau\(t\)=\\sigma\_\{\\max\}^\{2\}\\bigl\(1\-r^\{\-2t\}\\bigr\)andτT=σmax2−σmin2\\tau\_\{T\}=\\sigma\_\{\\max\}^\{2\}\-\\sigma\_\{\\min\}^\{2\}, and set

α⁡\(t\)=τ⁡\(t\)τT=1−r−2​t1−r−2,β⁡\(t\)=τT−τ⁡\(t\)=σmax​r−2​t−r−2\.\\alpha\(t\)\\;=\\;\\frac\{\\tau\(t\)\}\{\\tau\_\{T\}\}\\;=\\;\\frac\{1\-r^\{\-2t\}\}\{1\-r^\{\-2\}\},\\qquad\\beta\(t\)\\;=\\;\\sqrt\{\\tau\_\{T\}\-\\tau\(t\)\}\\;=\\;\\sigma\_\{\\max\}\\sqrt\{r^\{\-2t\}\-r^\{\-2\}\}\.\(57\)The boundary conditions of the one\-sided interpolant hold exactly sinceτ⁡\(0\)=0\\tau\(0\)=0andτ⁡\(1\)=σmax2​\(1−r−2\)=τT\\tau\(1\)=\\sigma\_\{\\max\}^\{2\}\(1\-r^\{\-2\}\)=\\tau\_\{T\}, which givesα⁡\(0\)=0\\alpha\(0\)=0,α⁡\(1\)=1\\alpha\(1\)=1,β⁡\(1\)=0\\beta\(1\)=0, and a prior scale ofσ0=β⁡\(0\)=τT\\sigma\_\{0\}=\\beta\(0\)=\\sqrt\{\\smash\[b\]\{\\tau\_\{T\}\}\}\. In our experiments the geometric schedule led to slightly better results \([Figures6](https://arxiv.org/html/2610.12848#A4.F6),[7](https://arxiv.org/html/2610.12848#A4.F7)and[8](https://arxiv.org/html/2610.12848#A4.F8)\) than the linear scheduleα⁡\(t\)=t\\alpha\(t\)=t,β⁡\(t\)=σ0​\(1−t\)\\beta\(t\)=\\sigma\_\{0\}\(1\-t\)\. All runs useσ0=1\\sigma\_\{0\}=1andr=2\.5r=2\.5, corresponding toσmax=σ0/1−r−2≈1\.09\\sigma\_\{\\max\}=\\sigma\_\{0\}/\\sqrt\{1\-r^\{\-2\}\}\\approx 1\.09andσmin=σmax/r≈0\.44\\sigma\_\{\\min\}=\\sigma\_\{\\max\}/r\\approx 0\.44, identical across all task suites\.

Table 1:Hyperparameters of FFM and of the online baselines\.†All methods use discount0\.990\.99on the MuJoCo Playground DMC tasks and0\.970\.97on the humanoid locomotion tasks \(G1/T1 Joystick\); all other FFM hyperparameters are identical across the two suites\.

### D\.4Further experiments and design ablations

[Figure4](https://arxiv.org/html/2610.12848#A4.F4)and[Figure5](https://arxiv.org/html/2610.12848#A4.F5)show the per\-task learning curves on the 20 DMC tasks and the four humanoid tasks, respectively\. An aggregation over all environments can be found in[Figure3](https://arxiv.org/html/2610.12848#S6.F3)\.

##### Interpolant schedule\.

As detailed in[SectionD\.3](https://arxiv.org/html/2610.12848#A4.SS3), we use a geometric interpolant schedule for the on\-policy RL setting\. We refer to[SectionD\.3](https://arxiv.org/html/2610.12848#A4.SS3)for the exact parameter setting\. A valid and intuitive interpolant is the linear schedule as described in[Section3](https://arxiv.org/html/2610.12848#S3)\. However, the schedule type’s effect on performance is unclear, which is why we have conducted thorough experiments and report the aggregated results over the 20 Playground tasks together with their final performances and the Humanoid tasks together with their final performances in[Figure6](https://arxiv.org/html/2610.12848#A4.F6)\. In[Figure7](https://arxiv.org/html/2610.12848#A4.F7), we report the per\-task results of the Playground suite, and the per\-task results on the humanoid environments are shown in[Figure8](https://arxiv.org/html/2610.12848#A4.F8)\. While there seems to be only a minor, negligible difference in the aggregated results for the Playground tasks \([Figure6](https://arxiv.org/html/2610.12848#A4.F6)\(left\)\), a clear difference can be seen in favor of the geometric schedule in the aggregated results for the humanoid tasks \([Figure6](https://arxiv.org/html/2610.12848#A4.F6)\(right\)\)\. However, our experiments show that both schedules lead to a reasonably well\-performing policy on average\.

Figure 6:Ablation\. Geometric versus linear interpolant schedule\.As discussed in[SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4), FFM is not restricted to a specific interpolation schedule type such as the linear schedule\. Motivated by diffusion\-based methods, we have compared the performance of a linear schedule to the geometric schedule as described in[SectionD\.4](https://arxiv.org/html/2610.12848#A4.SS4)and report the aggregated results over all 20 MJX Playground tasks \(left two panels\) and the four humanoid tasks \(right two panels\), where every single task was run on1010seeds, and we report the IQM and the stratified bootstrapped95%95\\%confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)\. While the results indicate that there is barely a performance difference in the Playground tasks, a slight improvement on the humanoid tasks can be seen, which is why we have chosen the geometric schedule as default for the online RL tasks\. See[Figure7](https://arxiv.org/html/2610.12848#A4.F7)and[Figure8](https://arxiv.org/html/2610.12848#A4.F8)for the per\-task results\.Figure 7:Ablation\. Geometric versus linear interpolant schedule \(MJX Playground\)\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on the 20 MJX Playground tasks\. The performance between the schedule types is the same for most tasks and differs only for some tasks such asFingerSpinorPendulumSwingup, which, however, lie within the confidence intervals\. Therefore, there is no statistically clear result on which interpolant should be used for the Playground tasks\.Figure 8:Ablation\. Geometric versus linear interpolant schedule \(MJX Humanoid\)\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on the four MJX humanoid tasks\. While for theG1JoystickFlatTerrainand for theG1JoystickRoughTerraintasks there is no difference in performance, it seems that the geometric schedule performs slightly better on theT1tasks, which is why we have chosen the geometric schedule as default for the online RL experiments\.
##### Number of integration steps\.

We further analyze the effect of the number of integration steps on the humanoid tasks’ performance\.[Figure9](https://arxiv.org/html/2610.12848#A4.F9)\(left\) shows the results aggregated over the four MJX humanoid tasks for different numbers of Euler integration stepsKK\. Please note that we only varyKKand keep everything else as detailed in[Table1](https://arxiv.org/html/2610.12848#A4.T1)\. The corresponding per\-task curves to the aggregated results are shown in[Figure10](https://arxiv.org/html/2610.12848#A4.F10)\. As expected, the performance increases with the number of integration steps, since the integration error of the flow decreases\. However, we observe that performance plateaus atK=16K=16, where increasing toK=32K=32does not improve performance and yields similar results\. Based on this observation and the baseline’s hyperparameters, we useK=16K=16throughout the paper for a fair comparison against the other diffusion\- and flow\-based methods \([Table1](https://arxiv.org/html/2610.12848#A4.T1)\)\.

Figure 9:Ablation\. Number of Integration Steps \(left two panels\) and Action Squashing mapping \(right two panels\)\.Number of integration steps \(left two panels\):We analyze the effect of varying integration steps and report the aggregated results over all four humanoid tasks \(left two panels\)\. Every single task was run on1010seeds, and we report the IQM and the95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)\. The results indicate that with an increasing number of integration steps, the performance increases until it plateaus fromK=16K=16onwards, which is why we have chosenK=16K=16as default in our experiments\. Please see[Figure10](https://arxiv.org/html/2610.12848#A4.F10)for the per\-task learning curves\.Action Squashing Mapping \(right two panels\):We analyze the effect of utilizing thetanh\\mathrm\{tanh\}and theerf\\mathrm\{erf\}as squashing functions for mapping the generated actions to a defined action bound\. We have run experiments on the four MJX humanoid tasks and report the aggregated IQM with95%95\\%stratified bootstrapped confidence intervals \(1010seeds per task\)\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)in the two right panels\. The results show that FFM is not sensitive to the choice of the squashing function\. Please see[Figure11](https://arxiv.org/html/2610.12848#A4.F11)for the per\-task learning curves\.Figure 10:Ablation\. Number of Integration Steps\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on the four MJX humanoid tasks for varying numbers of integration stepsKK\. With increasing number of integration steps, the performance improves\. However, the performance plateaus atK=16K=16steps\.
##### Action squashing\.

A key factor for stable training in actor\-critic\-based RL is that the generated actions need to be squashed to a valid region to avoid querying the critic at unreasonable action regions during the policy update\. As detailed in[AppendixD](https://arxiv.org/html/2610.12848#A4), there are two distinctive choices for the squashing function, namely, thetanh\\mathrm\{tanh\}and theerf\\mathrm\{erf\}functions\. To assess the effect on the performance of the design choices, we have run FFM with atanh\\mathrm\{tanh\}squashing function while keeping all other design choices the same and report the aggregated learning curves on the humanoid tasks in[Figure9](https://arxiv.org/html/2610.12848#A4.F9)\(right\), together with the final returns\. The per\-task results are shown in[Figure11](https://arxiv.org/html/2610.12848#A4.F11)\. From the results, it becomes clear that both variants perform on par, indicating that FFM is not sensitive to this choice and that either function is a valid default\.

Figure 11:Ablation\. Action squashing mapping\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on the four MJX humanoid tasks for thetanh\\mathrm\{tanh\}anderf\\mathrm\{erf\}squashing functions\. The results indicate that FFM is insensitive to the choice of the squashing function\.
##### Deterministic evaluation\.

During our experiments, we observed that starting the flow’s integration always at the central pointX0=0X\_\{0\}=0during evaluation only, yields improved performance against the standard case, whereX0∼𝒩⁡\(0,I\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\)is drawn from a standard normal distribution as done during training\. This observation is depicted in[Figure12](https://arxiv.org/html/2610.12848#A4.F12), where we plot the aggregated learning curves on the Playground tasks together with their final returns \(left\) and the aggregated learning curves on the humanoid tasks together with their final returns \(right\)\. The per\-task learning curves are shown in[Figure13](https://arxiv.org/html/2610.12848#A4.F13)and[Figure14](https://arxiv.org/html/2610.12848#A4.F14)respectively\. While the improvement is not very obvious for the Playground tasks \([Figure12](https://arxiv.org/html/2610.12848#A4.F12)\(left\)\), a clear and significant performance boost in the humanoid tasks can be observed \([Figure12](https://arxiv.org/html/2610.12848#A4.F12)\(right\)\)\. This result indicates that, for high\-dimensional tasks in particular, removing the stochasticity from the action generation process yields improved performance\. While this observation is very interesting and worth a more detailed analysis, in particular from a theoretical point of view, we leave this analysis for future work and apply the deterministic evaluation throughout our experiments\.

Figure 12:Ablation\. Deterministic evaluation\.We compare the effect of evaluating FFM usingX0=0X\_\{0\}=0\(deterministic\) andX0∼𝒩⁡\(0,I\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\)\(stochastic\) on the 20 MJX Playground \(left two panels\) and four humanoid tasks \(right two panels\), where every task was run on1010seeds, and we plot the aggregated IQM with stratified bootstrapped95%95\\%confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)\. While there is almost no performance gap between both variants for the Playground tasks, a significant performance boost follows from the deterministic evaluation on the humanoid tasks\. This indicates that deterministic evaluation matters most for high\-dimensional tasks\. Please see[Figure13](https://arxiv.org/html/2610.12848#A4.F13)and[Figure14](https://arxiv.org/html/2610.12848#A4.F14)for the per\-task performances of each task suite\.Figure 13:Ablation\. Deterministic evaluation \(MJX Playground\)\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on each of the 20 MJX Playground tasks forX0=0X\_\{0\}=0\(deterministic\) andX0∼𝒩⁡\(0,I\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\)\(stochastic\) during evaluation\. While most of the results indicate that there are not many differences, forFingerSpin, orWalkerRun, deterministic evaluation seems to improve performance\.Figure 14:Ablation\. Deterministic evaluation \(MJX Humanoids\)\.We plot the IQM per\-task learning curves with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)over1010seeds on each of the four MJX humanoid tasks forX0=0X\_\{0\}=0\(deterministic\) andX0∼𝒩⁡\(0,I\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\)\(stochastic\) during evaluation\. For all four tasks, deterministic evaluation shows significantly improved performance, indicating that stochasticity can harm performance, in particular for high\-dimensional action spaces\. We use deterministic evaluation throughout all of our experiments\.
##### Correlated exploration\.

A flow policy’s key benefit is that the action generation process is a deterministic mapping up to the sampling of the starting sample\. As discussed in[Section5](https://arxiv.org/html/2610.12848#S5), this allows us to apply a correlated exploration scheme, where the exploration is shaped by correlating the prior draws across the environment steps\. To analyze the additional hyperparameterτou\\tau\_\{\\mathrm\{ou\}\}that is introduced by[Equation20](https://arxiv.org/html/2610.12848#S5.E20), we have conducted experiments on single Playground tasks, where the initial instantiation of FFM, i\.e\., without correlated exploration, performs unsatisfactorily \(see[Figure4](https://arxiv.org/html/2610.12848#A4.F4)\)\.[Figure15](https://arxiv.org/html/2610.12848#A4.F15)shows the learning curves on a subset of tasks that we have picked from the task suite to analyze the effect on performance for varyingτou\\tau\_\{\\mathrm\{ou\}\}values\. We can observe that for each task, there exists at least oneτou\\tau\_\{\\mathrm\{ou\}\}value that outperforms the initial case without correlated exploration, and only for theFingerSpinandWalkerRuntasks, we observe that a too\-high value yields decreased performance\. Especially for theAcrobotSwingupSparse, HopperHopandHopperStandtasks, correlated exploration yields a significant improvement over the case without correlated exploration\. However, the results also indicate that there is no global value forτou\\tau\_\{\\mathrm\{ou\}\}that performs best across every task, which makes tuning this hyperparameter difficult and is only possible if enough compute resources are available\. Nevertheless, we believe that correlated exploration is a useful feature that FFM can leverage\.

Figure 15:Ablation\. Correlated Exploration\.On a subset of the MJX Playground tasks, we analyze the effect of correlated exploration by varyingτou∈\{1,2,4,8,16,32\}\\tau\_\{\\mathrm\{ou\}\}\\in\\\{1,2,4,8,16,32\\\}, where we have run every curve on1010seeds and report the IQM with95%95\\%stratified bootstrapped confidence intervals\([Agarwal et al\., 2021](https://arxiv.org/html/2610.12848#bib.bib3)\)\. Correlated exploration can significantly boost FFM’s performance for the right choice ofτou\\tau\_\{\\mathrm\{ou\}\}\. For example, theHopperStandtask is solved, while FFM without correlated exploration struggles on this task\.

## Appendix EOffline\-to\-online fine\-tuning

This appendix collects the offline\-to\-online instantiation of FFM \([Section5](https://arxiv.org/html/2610.12848#S5)\): the behavior\-constrained setting and the behavior\-tilted target \([SectionE\.1](https://arxiv.org/html/2610.12848#A5.SS1)\), the fine\-tuning procedure \([SectionE\.2](https://arxiv.org/html/2610.12848#A5.SS2)\), and the protocol, full results, and design ablations of the offline\-to\-online experiments of[Section6](https://arxiv.org/html/2610.12848#S6)\([SectionsE\.3](https://arxiv.org/html/2610.12848#A5.SS3),[E\.4](https://arxiv.org/html/2610.12848#A5.SS4)and[E\.5](https://arxiv.org/html/2610.12848#A5.SS5)\)\. Throughout,vrefv\_\{\\mathrm\{ref\}\}and𝗌ref\\mathsf\{s\}\_\{\\mathrm\{ref\}\}denote the velocity and score fields of the pretrained \(behavior\) policyprefp\_\{\\mathrm\{ref\}\}, related by the bridge[Equation28](https://arxiv.org/html/2610.12848#A2.E28), andκ⁡\(t\)=α˙​\(t\)α⁡\(t\)​β2​\(t\)−β˙​\(t\)​β​\(t\)\\kappa\(t\)=\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}\\beta^\{2\}\(t\)\-\\dot\{\\beta\}\(t\)\\beta\(t\)is the bridge coefficient[Equation8](https://arxiv.org/html/2610.12848#S4.E8), assumed positive on\(0,1\)\(0,1\)\.

### E\.1Further details

In offline\-to\-online RL a fixed dataset𝒟\\mathcal\{D\}of transitions is available before any interaction\. Two objects are pretrained on it: a*behavior flow*prefp\_\{\\mathrm\{ref\}\}, fit by flow matching[Equation6](https://arxiv.org/html/2610.12848#S3.E6)to the dataset actions \(its velocity fieldvrefv\_\{\\mathrm\{ref\}\}is the reference of[Section5](https://arxiv.org/html/2610.12848#S5)\), and a criticQQ, fit by an offline method \(here pessimistic TD,[Algorithm2](https://arxiv.org/html/2610.12848#alg2)\)\. Fine\-tuning then continues both while the dataset grows with online interaction\. The difficulty is that an offline critic is trustworthy only near the data: its extrapolation errors off the support ofprefp\_\{\\mathrm\{ref\}\}are large and systematically optimistic, so an unconstrained improvement step chases them\([Peng et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib67);[Nair et al\., 2020](https://arxiv.org/html/2610.12848#bib.bib62);[Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\)\. The standard remedy penalizes deviation from the behavior policy: at each state, maximize𝔼x∼π\[Q\(s,x\)\]−λDKL\(π\(⋅∣s\)∥pref\(⋅∣s\)\)\\mathbb\{E\}\_\{x\\sim\\pi\}\[Q\(s,x\)\]\-\\lambda\\,D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\\mid s\)\\,\\\|\\,p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid s\)\\right\), whose maximizer is the*behavior\-tilted*Boltzmann policyπ⋆​\(x∣s\)∝pref​\(x∣s\)​eQ⁡\(s,x\)/λ\\pi^\{\\star\}\(x\\mid s\)\\propto p\_\{\\mathrm\{ref\}\}\(x\\mid s\)\\,e^\{Q\(s,x\)/\\lambda\}of[Section3](https://arxiv.org/html/2610.12848#S3)\([Abdolmaleki et al\., 2018](https://arxiv.org/html/2610.12848#bib.bib2);[Peng et al\., 2019](https://arxiv.org/html/2610.12848#bib.bib67);[Nair et al\., 2020](https://arxiv.org/html/2610.12848#bib.bib62)\)\. Its terminal score is

∇xlogπ⋆\(x∣s\)=𝗌ref\(x\)\+∇xQ\(s,x\)/λ,𝗌ref:=∇xlogpref\(⋅∣s\),\\nabla\_\{x\}\\log\\pi^\{\\star\}\(x\\mid s\)\\;=\\;\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(x\)\+\\nabla\_\{x\}Q\(s,x\)/\\lambda,\\qquad\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\\;:=\\;\\nabla\_\{x\}\\log p\_\{\\mathrm\{ref\}\}\(\\cdot\\mid s\),\(58\)so relative to the pure Boltzmann target of[Section4](https://arxiv.org/html/2610.12848#S4)exactly one new ingredient enters: the reference’s terminal score\. For a flow\-parameterized reference it is available without a density and without the dataset, by reading the score off the pretrained velocity field through the bridge[Equation29](https://arxiv.org/html/2610.12848#A2.E29)neart=1t=1, which is[Equation13](https://arxiv.org/html/2610.12848#S5.E13)\. Two mechanisms then keep the fine\-tuned policy close toprefp\_\{\\mathrm\{ref\}\}, and[Algorithm2](https://arxiv.org/html/2610.12848#alg2)combines them: the*target*itself, which by[AppendixA](https://arxiv.org/html/2610.12848#A1)has the tilt as a fixed point, and the trust region of[Section5](https://arxiv.org/html/2610.12848#S5), here taken with respect to the pretrained flow, that is,ℒTR​\(θ,ref\)≤εTR\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\mathrm\{ref\}\)\\leq\\varepsilon\_\{\\mathrm\{TR\}\}withvrefv\_\{\\mathrm\{ref\}\}in place ofvθ¯v\_\{\\bar\{\\theta\}\}in[Equation16](https://arxiv.org/html/2610.12848#S5.E16), which bounds every step\.

### E\.2Pseudocode

[Algorithm2](https://arxiv.org/html/2610.12848#alg2)summarizes the fine\-tuning procedure\. It alternates a pessimistic TD update of the critic ensemble with the behavior\-tilted FFM step of[Section5](https://arxiv.org/html/2610.12848#S5): clean actions are generated by the current flow, labeled with the scoreggof the behavior\-tilted target, noised, and regressed onto by[Equation14](https://arxiv.org/html/2610.12848#S5.E14), while the trust region[Equation16](https://arxiv.org/html/2610.12848#S5.E16), taken with respect to the pretrained flowvrefv\_\{\\mathrm\{ref\}\}instead of the previous policy, is enforced through the Lagrangian[Equation18](https://arxiv.org/html/2610.12848#S5.E18)and the dual update of[Equation19](https://arxiv.org/html/2610.12848#S5.E19)\. We writeℒTR​\(θ,ref\)\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\mathrm\{ref\}\)for[Equation16](https://arxiv.org/html/2610.12848#S5.E16)withvrefv\_\{\\mathrm\{ref\}\}in place ofvθ¯v\_\{\\bar\{\\theta\}\},θ¯\\bar\{\\theta\}for a frozen copy of the current actor parameters, andϕ¯\\bar\{\\phi\}for the EMA target parameters of the critic\. The implementation differs from the pseudocode in the form of the actor regression, which uses the variance\-reduced target of[SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2); this and the remaining implementation details are given in[SectionE\.3](https://arxiv.org/html/2610.12848#A5.SS3)\.

Algorithm 2Offline\-to\-online fine\-tuning with an FFM actor1:input:Pretrained velocity field

vrefv\_\{\\mathrm\{ref\}\}with terminal law

prefp\_\{\\mathrm\{ref\}\}, critic ensemble

Qϕ1:NQ\_\{\\phi\_\{1:N\}\}with EMA targets

Qϕ¯1:NQ\_\{\\bar\{\\phi\}\_\{1:N\}\}, dataset

𝒟\\mathcal\{D\}, temperature

λ\\lambda, pessimism

ϱ\\varrho, trust\-region budget

εTR\\varepsilon\_\{\\mathrm\{TR\}\}with multiplier

ν\\nuand dual step size

ην\\eta\_\{\\nu\}, schedules

α,β\\alpha,\\beta
2:initialization:warm start

vθ←vrefv\_\{\\theta\}\\leftarrow v\_\{\\mathrm\{ref\}\}, set

ν←ν0\\nu\\leftarrow\\nu\_\{0\}
3:foreach step, first offline on

𝒟\\mathcal\{D\}, then onlinedo

4:interaction \(online phase only, no gradients\):

5:• act

x∼πθ\(⋅∣s\)x\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\)by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)

6:• add

\(s,x,r,s′\)\(s,x,r,s^\{\\prime\}\)to

𝒟\\mathcal\{D\}
7:policy evaluation:pessimistic TD targets on a minibatch of

𝒟\\mathcal\{D\}with

x′∼πθ\(⋅∣s′\)x^\{\\prime\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s^\{\\prime\}\)\(target gradients stopped\)

8:

y=r\+γ⁡\(meanj​Qϕ¯j​\(s′,x′\)−ϱ​stdj​Qϕ¯j​\(s′,x′\)\)\\displaystyle y=r\+\\gamma\\,\\bigl\(\\mathrm\{mean\}\_\{j\}\\,Q\_\{\\bar\{\\phi\}\_\{j\}\}\(s^\{\\prime\},x^\{\\prime\}\)\-\\varrho\\,\\mathrm\{std\}\_\{j\}\\,Q\_\{\\bar\{\\phi\}\_\{j\}\}\(s^\{\\prime\},x^\{\\prime\}\)\\bigr\)
9:update critic parameters

ϕ1:N\\phi\_\{1:N\}using

∑j\(Qϕj​\(s,x\)−y\)2\\sum\_\{j\}\\bigl\(Q\_\{\\phi\_\{j\}\}\(s,x\)\-y\\bigr\)^\{2\}, then the EMA targets

ϕ¯1:N\\bar\{\\phi\}\_\{1:N\}
10:calculate scores \(no gradients\):

11:• clean actions

X1∼πθ¯\(⋅∣s\)X\_\{1\}\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid s\)by integrating[Equation5](https://arxiv.org/html/2610.12848#S3.E5)

12:• reference score

𝗌ref​\(X1\)\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(X\_\{1\}\)read off

vrefv\_\{\\mathrm\{ref\}\}by[Equation13](https://arxiv.org/html/2610.12848#S5.E13)

13:• behavior\-tilted score

g⁡\(X1\)=𝗌ref​\(X1\)\+1λ​∇xQ​\(s,X1\)g\(X\_\{1\}\)=\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\(X\_\{1\}\)\+\\tfrac\{1\}\{\\lambda\}\\nabla\_\{x\}Q\(s,X\_\{1\}\)
14:

t∼𝒰⁡\(0,1\)t\\sim\\mathcal\{U\}\(0,1\),

ε∼𝒩⁡\(⋅,0,I\)\\varepsilon\\sim\\mathcal\{N\}\\\!\\left\(\\cdot;\\,0,\\,I\\right\),

Xt←α⁡\(t\)​X1\+β⁡\(t\)​εX\_\{t\}\\leftarrow\\alpha\(t\)X\_\{1\}\+\\beta\(t\)\\varepsilon
15:minimize over

θ\\thetausing[Equation18](https://arxiv.org/html/2610.12848#S5.E18)with

ℒBT−FFM\\mathcal\{L\}\_\{\\mathrm\{BT\-FFM\}\}of[Equation14](https://arxiv.org/html/2610.12848#S5.E14):

16:

𝔼t,X1,ε​\[‖vθ​\(Xt,t∣s\)−α˙​\(t\)α⁡\(t\)​Xt−κ⁡\(t\)α⁡\(t\)​g​\(X1\)‖2\]\+ν⁡\(ℒTR​\(θ,ref\)−εTR\)\\displaystyle\\mathbb\{E\}\_\{t,X\_\{1\},\\varepsilon\}\\Bigl\[\\Bigl\\\|v\_\{\\theta\}\(X\_\{t\},t\\mid s\)\-\\tfrac\{\\dot\{\\alpha\}\(t\)\}\{\\alpha\(t\)\}X\_\{t\}\-\\tfrac\{\\kappa\(t\)\}\{\\alpha\(t\)\}\\,g\(X\_\{1\}\)\\Bigr\\\|^\{2\}\\Bigr\]\+\\nu\\bigl\(\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\mathrm\{ref\}\)\-\\varepsilon\_\{\\mathrm\{TR\}\}\\bigr\)
17:with

ℒTR​\(θ,ref\)\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\mathrm\{ref\}\)of[Equation16](https://arxiv.org/html/2610.12848#S5.E16)evaluated at the same points

\(Xt,t\)\(X\_\{t\},t\)
18:dual:ascend

ν←max⁡\(0,ν\+ην​\(ℒTR​\(θ,ref\)−εTR\)\)\\nu\\leftarrow\\max\\bigl\(0,\\,\\nu\+\\eta\_\{\\nu\}\\,\(\\mathcal\{L\}\_\{\\mathrm\{TR\}\}\(\\theta,\\mathrm\{ref\}\)\-\\varepsilon\_\{\\mathrm\{TR\}\}\)\\bigr\)\([Equation19](https://arxiv.org/html/2610.12848#S5.E19)\)

19:update the frozen actor copy:

θ¯←θ\\bar\{\\theta\}\\leftarrow\\theta
20:returnfine\-tuned velocity field

vθv\_\{\\theta\}

### E\.3Experimental details

#### E\.3\.1Environments and tasks

We evaluate on the offline\-to\-online benchmark of OGBench\([Park et al\., 2025a](https://arxiv.org/html/2610.12848#bib.bib65)\), comprising ten domains that split into*locomotion*\(goal\-reaching navigation\) and*manipulation*\(goal\-conditioned object rearrangement\);[Figure16](https://arxiv.org/html/2610.12848#A5.F16)shows a representative frame of each\. Every domain defines five*singletask*goals \(Task 1–5\): the offline dataset is shared across a domain’s tasks \(goal\-freenavigate/playdata\) and each task fixes a goal used for the sparse reward and for evaluation\.[Table2](https://arxiv.org/html/2610.12848#A5.T2)lists the state and action dimensionality, episode length, dataset size, and network width for each domain\.

![Refer to caption](https://arxiv.org/html/2610.12848v1/env_grid.png)Figure 16:The ten evaluation environments \(five tasks each\)\. Top: locomotion; bottom: manipulation\.Table 2:Environment specifications\. Network width and layer norm follow the dataset\-size convention \(1M: width512512; 10M/100M: width10241024with actor layer norm\), applied identically to the BC\-pretraining and fine\-tuning networks\.
#### E\.3\.2Offline\-to\-online protocol

We follow the protocol of TRQAM\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\)\. For every \(environment, task, seed\) we first pretrain a behavior\-cloning flow policy by flow matching for300300K gradient steps, then run11M offline gradient steps followed by500500K online environment steps \(1\.51\.5M total\); the online phase continues from the offline actor, critic, optimizer, and dual variables\. Policies are evaluated every5050K steps over5050episodes and we report the binary task\-success rate\. We use the eight\-seed set\{10001,20002,…,80008\}\\\{10001,20002,\\dots,80008\\\}\.

#### E\.3\.3Shared architecture and hyperparameters

All methods use four\-hidden\-layer MLP actors and critics; the critic is an ensemble with pessimistic backup\. The discount and pessimism are\(γ,ϱ\)=\(0\.995,0\.5\)\(\\gamma,\\varrho\)=\(0\.995,0\.5\)except on the long\-horizon HumanoidMaze domains, where\(γ,ϱ\)=\(0\.999,0\.0\)\(\\gamma,\\varrho\)=\(0\.999,0\.0\)\.[Table3](https://arxiv.org/html/2610.12848#A5.T3)lists the shared optimization hyperparameters\.

Table 3:Shared hyperparameters \(all methods\)\.
#### E\.3\.4FFM hyperparameters

FFM has a single per\-domain choice, the regime of[Table4](https://arxiv.org/html/2610.12848#A5.T4): Regime A on the sparse\-reward locomotion domains and Regime B on the chunked manipulation domains, with no per\-task or per\-seed tuning\. All other settings are shared\. The terminal reference score is always used and read off the pretrained flow by[Equation13](https://arxiv.org/html/2610.12848#S5.E13)att=1−δreft=1\-\\delta\_\{\\mathrm\{ref\}\}withδref=0\.1\\delta\_\{\\mathrm\{ref\}\}=0\.1; the endpoint softening isδ=0\.05\\delta=0\.05; four interpolant samples\(t,ε\)\(t,\\varepsilon\)are drawn per clean action; and the trust\-region budget isεTR=1\\varepsilon\_\{\\mathrm\{TR\}\}=1offline and88online\.

Table 4:FFM two\-regime recipe: the only per\-domain choice\.##### Actor update\.

The implementation differs from[Algorithm2](https://arxiv.org/html/2610.12848#alg2)in the actor regression\. Instead of the pure FFM target of[Equation14](https://arxiv.org/html/2610.12848#S5.E14), it regresses onto the mixed targetYcY\_\{c\}of[Equation34](https://arxiv.org/html/2610.12848#A2.E34)under the variance\-optimal schedulec⋆​\(t\)c^\{\\star\}\(t\)of[Equation37](https://arxiv.org/html/2610.12848#A2.E37), with the behavior\-tilted scoreg=𝗌ref\+∇xQ/λg=\\mathsf\{s\}\_\{\\mathrm\{ref\}\}\+\\nabla\_\{x\}Q/\\lambdain place of∇xQ/λ\\nabla\_\{x\}Q/\\lambdaand the Fisher scaleℐ¯\\bar\{\\mathcal\{I\}\}either fixed or tracked by an EMA \([Table4](https://arxiv.org/html/2610.12848#A5.T4)\)\. The residual is divided byκ⁡\(t\)\\kappa\(t\), so that the regression is carried out in score space \([SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2)\)\. All offline\-to\-online experiments use the linear interpolantα⁡\(t\)=t\\alpha\(t\)=t,β⁡\(t\)=1−t\\beta\(t\)=1\-t, for whichα˙​\(t\)/α​\(t\)=1/t\\dot\{\\alpha\}\(t\)/\\alpha\(t\)=1/tandκ⁡\(t\)=\(1−t\)/t\\kappa\(t\)=\(1\-t\)/t, and the regression line reads

vθ​\(Xt,t\)−Ycκ⁡\(t\)=t⁡\(vθ​\(Xt,t\)−X1\)\+\(t−c\)​ε1−t−ct​g​\(X1\)\.\\frac\{v\_\{\\theta\}\(X\_\{t\},t\)\-Y\_\{c\}\}\{\\kappa\(t\)\}\\;=\\;\\frac\{t\\big\(v\_\{\\theta\}\(X\_\{t\},t\)\-X\_\{1\}\\big\)\+\(t\-c\)\\,\\varepsilon\}\{1\-t\}\\;\-\\;\\frac\{c\}\{t\}\\,g\(X\_\{1\}\)\.\(59\)The denominators1−t1\-tandttare softened byδ\\delta, which replaces a time truncation \([SectionB\.2](https://arxiv.org/html/2610.12848#A2.SS2)\), and the trust region[Equation16](https://arxiv.org/html/2610.12848#S5.E16)is evaluated at the same points\(Xt,t\)\(X\_\{t\},t\)with the weight1/κ⁡\(t\)1/\\kappa\(t\)and the same softening\. The clean actionsX1X\_\{1\}are clipped to the action box before the critic gradient is taken, the critic target uses the best ofnnsampled next actions, and the pretrained flowvrefv\_\{\\mathrm\{ref\}\}continues to be trained by flow matching[Equation6](https://arxiv.org/html/2610.12848#S3.E6)on the growing dataset, so that the reference tracks the data\.

##### The two regimes\.

The rows of[Table4](https://arxiv.org/html/2610.12848#A5.T4)are two coordinated settings rather than four independent knobs\. In the sparse\-reward mazes the critic is smooth and∇xQ\\nabla\_\{x\}Qis small, so we amplify it with1/λ=301/\\lambda=30\. This only works if the Fisher scale is held fixed atℐ¯=1\\bar\{\\mathcal\{I\}\}=1: an adaptive normalizer estimates‖∇xQ/λ‖2\\\|\\nabla\_\{x\}Q/\\lambda\\\|^\{2\}and divides the amplification back out\. In manipulation the chunked critic gradient is informative, so1/λ=11/\\lambda=1suffices, and the adaptive EMA normalizer is preferable because the gradient magnitudes of an action chunk vary by orders of magnitude across coordinates and over training\. The reference score anchors the target to the pretrained flow in both regimes\. Locomotion additionally keeps the trust region[Equation16](https://arxiv.org/html/2610.12848#S5.E16), since with sparse rewards the critic is unreliable off the visited data and a navigation policy can otherwise drift into unseen parts of the maze\. In manipulation the product tilt and chunking already keep the actor on the data, and the trust region only slows online improvement, so it is switched off\. Action chunking with horizonh=5h=5is part of the protocol inherited from TRQAM and is applied to all methods on the manipulation domains \([Table3](https://arxiv.org/html/2610.12848#A5.T3)\); it appears in[Table4](https://arxiv.org/html/2610.12848#A5.T4)only because it coincides with the regime split\.

#### E\.3\.5Baselines

We compare against TRQAM\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\), QAM and QAM\-E\([Li & Levine, 2026](https://arxiv.org/html/2610.12848#bib.bib49)\), FQL and IFQL\([Park et al\., 2025b](https://arxiv.org/html/2610.12848#bib.bib66)\), DSRL\([Wagenmaker et al\., 2025](https://arxiv.org/html/2610.12848#bib.bib91)\), and CGQL\-L\([Dong et al\., 2026b](https://arxiv.org/html/2610.12848#bib.bib27)\), each run in the identical protocol above\.

### E\.4Full results

[Tables5](https://arxiv.org/html/2610.12848#A5.T5)and[6](https://arxiv.org/html/2610.12848#A5.T6)report the offline\-phase \(at11M steps\) and offline\-to\-online \(at1\.51\.5M steps\) success rates, aggregated per domain over the five tasks and all seeds, with the best method per row in bold\. The per\-domain aggregated learning curves are shown in[Figure2](https://arxiv.org/html/2610.12848#S6.F2)\(main text\), and[Figure17](https://arxiv.org/html/2610.12848#A5.F17)gives the full per\-\(environment, task\) grid; in both, the offline \(≤1\\leq\\\!1M\) and online \(\>1\>\\\!1M\) phases are shaded\. The tables corroborate the discussion of[Section6](https://arxiv.org/html/2610.12848#S6): FFM’s offline average trails the strongest offline baseline yet its offline\-to\-online average is the best overall, with the largest per\-environment gains on the hard, long\-horizon tasks and no divergence across seeds\.

Table 5:Offline\-phase success rate at the end of offline training \(1M steps\), mean over 5 tasks and seeds per environment\. Best per row in bold\.Table 6:Offline\-to\-online success rate at the end of online fine\-tuning \(1\.5M total steps\), mean over 5 tasks and seeds per environment\. Best per row in bold\.000\.50\.511AM\-LTask 1Task 2Task 3Task 4Task 5000\.50\.511AM\-G000\.50\.511HM\-M000\.50\.511HM\-L000\.50\.511C\-2000\.50\.511C\-3000\.50\.511C\-4000\.50\.511Scene000\.50\.511P\-3x3000\.50\.5111\.51\.5000\.50\.511P\-4x4000\.50\.5111\.51\.5000\.50\.5111\.51\.5000\.50\.5111\.51\.5000\.50\.5111\.51\.5FFM \(ours\)TRQAMQAMQAM\-EIFQLFQLDSRLCGQL\-Lofflineonline

Figure 17:Offline\-to\-online success rate for every environment \(rows\) and task \(columns\), mean over seeds\. Offline and online phases are shaded; FFM \(ours\) is drawn in bold\.
### E\.5Design ablations

The recipe of[Table4](https://arxiv.org/html/2610.12848#A5.T4)and the integration budget were fixed once and transferred across all ten domains\. The ablations below justify these choices\. Each varies a single setting on one task per family,cube\-doubletask 3 \(manipulation\) andantmaze\-largetask 3 \(locomotion\), with all other settings at their regime values, and reports the success rate averaged over the last150150K steps of the respective phase as mean±\\pmstandard deviation over three seeds\. Unless stated otherwise, the runs are offline only \(11M steps, reusing the pretrained flow\)\.

##### Critic scale and Fisher normalizer\.

The two left panels of[Figure18](https://arxiv.org/html/2610.12848#A5.F18)sweep1/λ∈\{1,2,4,8,30\}1/\\lambda\\in\\\{1,2,4,8,30\\\}for the fixed \(ℐ¯=1\\bar\{\\mathcal\{I\}\}=1\) and the adaptive \(EMA\) normalizer\. On manipulation, success falls monotonically with1/λ1/\\lambda\(from0\.400\.40to0\.000\.00\) and the two normalizers coincide, since at1/λ=11/\\lambda=1there is no amplification to cancel\. On locomotion the curves cross: with the fixed normalizer success rises with1/λ1/\\lambda\(0\.83→0\.950\.83\\to 0\.95\), with the adaptive one it falls \(0\.72→0\.140\.72\\to 0\.14\) because the EMA divides out the gain\. The pairs\(1/λ=30,fixed\)\(1/\\lambda\{=\}30,\\text\{fixed\}\)and\(1/λ=1,adaptive\)\(1/\\lambda\{=\}1,\\text\{adaptive\}\)of[Table4](https://arxiv.org/html/2610.12848#A5.T4)are therefore the two operating points, and no mixed combination improves on either\. A third variant,*adaptive\-shape*, tracks the Fisher normalizer per coordinate but rescales it to unit mean, so that it keeps the scale invariance across action dimensions without cancelling the overall1/λ21/\\lambda^\{2\}level\. On locomotion it rises with1/λ1/\\lambdalike the fixed normalizer \(0\.82→0\.930\.82\\to 0\.93\), and on manipulation all three variants coincide\. It would remove the normalizer row of[Table4](https://arxiv.org/html/2610.12848#A5.T4), but since it matches rather than improves on the per\-domain choice, we keep the explicit split\.

##### Integration steps\.

The two right panels of[Figure18](https://arxiv.org/html/2610.12848#A5.F18)vary the number of Euler steps over\{2,5,10,25\}\\\{2,5,10,25\\\}\. On manipulation two steps are too coarse \(0\.310\.31\) and performance saturates from five steps on \(≈0\.40\\approx 0\.40\); locomotion is insensitive \(0\.980\.98at two steps,0\.940\.94at2525\)\. The ten steps inherited from the TRQAM/QAM protocol lie on the plateau for both families\.

124830000\.250\.250\.50\.50\.750\.75111/λ1/\\lambdaoffline successCritic scale \(manip\.\)1248301/λ1/\\lambdaCritic scale \(loco\.\)251025ODE stepsIntegration steps \(manip\.\)251025ODE stepsIntegration steps \(loco\.\)fixed Fisheradaptive Fisheradaptive\-shapeintegration\-step sweep

Figure 18:Critic scale and integration steps\(offline success, mean±\\pmstd over three seeds, one manipulation and one locomotion task\)\.*Left two panels:*fixed, adaptive, and adaptive\-shape Fisher normalizer against1/λ1/\\lambda\.*Right two panels:*number of Euler steps\.
##### Behavior anchor\.

Two mechanisms keep FFM near the data: the reference score, which makes the*product*tiltpref​eQ/λp\_\{\\mathrm\{ref\}\}\\,e^\{Q/\\lambda\}the target, and the trust region[Equation16](https://arxiv.org/html/2610.12848#S5.E16)\. Without the reference score the target is the*pure*tilteQ/λe^\{Q/\\lambda\}\.[Figure19](https://arxiv.org/html/2610.12848#A5.F19)crosses the two \(product vs\. pure tilt, trust region on vs\. off\) in both phases\. Offline, removing both anchors collapses the policy \(0\.000\.00on manipulation,0\.190\.19on locomotion\)\. Either one recovers it: oncube\-doublethe pure tilt with trust region \(0\.580\.58\) even beats the product tilt \(0\.410\.41\), and onantmaze\-largethe product tilt with trust region is best \(0\.960\.96\)\. Online, at the budgetεTR=8\\varepsilon\_\{\\mathrm\{TR\}\}=8of the recipe, the pure tilt collapses oncube\-double\(0\.050\.05\) while the product tilt reaches0\.990\.99; on locomotion all four combinations converge to≈1\\approx 1\.

[Figure20](https://arxiv.org/html/2610.12848#A5.F20)sweeps the budgetεTR\\varepsilon\_\{\\mathrm\{TR\}\}on four tasks\. The budget the pure tilt needs is task\-dependent in both phases and points in opposite directions\. Online,cube\-doubleneeds a tight budget \(1\.01\.0atεTR=1\\varepsilon\_\{\\mathrm\{TR\}\}=1,0\.050\.05at88\) andpuzzle\-3x3a loose one \(0\.230\.23at11,1\.01\.0at88\), so at every fixed budget the worst task scores≤0\.42\\leq 0\.42\. Offline,cube\-doubleandsceneare best atεTR=1\\varepsilon\_\{\\mathrm\{TR\}\}=1\(0\.580\.58and1\.001\.00, falling to0\.000\.00and0\.020\.02at1616\),puzzle\-3x3needs at least44\(0\.210\.21at11,0\.980\.98at44\),antmaze\-largeis insensitive, and tightening below11does not help \(0\.350\.35,0\.550\.55,0\.580\.58atεTR=0\.25\\varepsilon\_\{\\mathrm\{TR\}\}=0\.25,0\.50\.5,11oncube\-double\)\. The product tilt, by contrast, is robust online: it stays≥0\.99\\geq 0\.99on all four tasks for everyεTR≥4\\varepsilon\_\{\\mathrm\{TR\}\}\\geq 4, so a single budget serves all tasks, which is why the recipe uses it\. Offline it is flat on three tasks but rises from0\.190\.19atεTR=1\\varepsilon\_\{\\mathrm\{TR\}\}=1to0\.990\.99at88onpuzzle\-3x3; a loose offline budget is equivalent to no trust region \(0\.990\.99at1616vs\.1\.001\.00with the trust region off\), which is the setting of the manipulation regime\.

producttiltpuretilt000\.250\.250\.50\.50\.750\.7511successManipulation \(offline\)producttiltpuretiltManipulation \(online\)producttiltpuretiltLocomotion \(offline\)producttiltpuretiltLocomotion \(online\)trust region offtrust region on

Figure 19:Behavior anchor\.Product tiltpref​eQ/λp\_\{\\mathrm\{ref\}\}\\,e^\{Q/\\lambda\}vs\. pure tilteQ/λe^\{Q/\\lambda\}, with the trust region on or off, offline and online \(mean±\\pmstd over three seeds; the online runs useεTR=8\\varepsilon\_\{\\mathrm\{TR\}\}=8\)\.14816000\.250\.250\.50\.50\.750\.7511budgetεTR\\varepsilon\_\{\\mathrm\{TR\}\}successPure tilt \(offline\)14816budgetεTR\\varepsilon\_\{\\mathrm\{TR\}\}Product tilt \(offline\)14816budgetεTR\\varepsilon\_\{\\mathrm\{TR\}\}Pure tilt \(online\)14816budgetεTR\\varepsilon\_\{\\mathrm\{TR\}\}Product tilt \(online\)cube\-doublescenepuzzle\-3x3antmaze\-large

Figure 20:Trust\-region budget sweep\.Success againstεTR\\varepsilon\_\{\\mathrm\{TR\}\}on four tasks \(mean±\\pmstd over three seeds\), offline \(left two panels\) and online \(right two panels\), for the pure tilt \(first and third panel\) and the product tilt \(second and fourth panel\)\. With the offline trust region off, the product tilt scores0\.410\.41,0\.940\.94,1\.001\.00, and0\.810\.81oncube\-double,scene,puzzle\-3x3, andantmaze\-large\.