GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

# GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios
Source: [https://arxiv.org/html/2606.06967](https://arxiv.org/html/2606.06967)
Ke Hu1∗Shutong Ding1Panxin Tao1Jingya Wang1Ye Shi1†

1ShanghaiTech University

\{huke2024, dingsht, taopx2022\}@shanghaitech\.edu\.cn \{wangjingya, shiye\}@shanghaitech\.edu\.cn

###### Abstract

Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning \(RL\) in complex continuous\-control tasks\. Among them, flow\-based policies are especially appealing because they generate actions through deterministic transport maps\. However, applying such generative policies to likelihood\-based on\-policy learning remains limited by the difficulty of evaluating the probability of executed actions\. Existing flow RL methods either replace the true action\-density ratio with approximate surrogates, which can introduce biased updates, or recover exact likelihoods through dummy\-action augmentation, which enlarges the policy space and increases computation\. In this work, we propose GenPO\+\+, a reversible generative policy optimization framework that uses history states as auxiliary memory in a high\-order reversible ODE solver, yielding exact inversion without changing the original action dimension\. The resulting generative policy map has a log\-determinant determined only by fixed solver coefficients, enabling exact and Jacobian\-free likelihood\-ratio computation\. This design preserves the expressiveness of generative flow policies while avoiding both action ratio bias and dummy\-action overhead\. We evaluate GenPO\+\+ on large\-scale simulated control, fine\-tuning, and real\-world robotic manipulation tasks, where it achieves competitive or superior performance over state\-of\-the\-art on\-policy RL methods, while improving training stability and computational efficiency\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.06967v1/x1.png)Figure 1:Comparison of FPO, GenPO, and GenPO\+\+\. FPO relies on an ELBO surrogate ratio that can be biased, while GenPO obtains exact inversion through dummy\-action augmentation at the cost of an enlarged action space\. GenPO\+\+ replaces dummy actions with solver\-history states, achieving exact inversion and Jacobian\-free likelihood\-ratio computation while preserving the original action dimension\.Reinforcement learning \(RL\) has achieved strong performance in continuous control, often with simple Gaussian policies and stable likelihood\-based updates such as PPOSchulmanet al\.\([2017](https://arxiv.org/html/2606.06967#bib.bib61)\)\. However, Gaussian policies are limited in expressiveness: they typically represent unimodal action distributions and can struggle in tasks with multimodal action choices, contact\-rich dynamics, or discontinuous action manifolds\. Generative policies based on diffusion and flow models offer a promising alternative\. Diffusion PolicyChiet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib20)\), DiffuserJanneret al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib42)\), diffusion policies for offline RLWanget al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib27)\); Ajayet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib43)\), and flow\-matching imitation policiesRouxelet al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib18)\)have shown that generative models can represent rich conditional action distributions and capture diverse behavior modes\. Flow matching and rectified flow further provide efficient deterministic samplers, making them attractive for scalable control and policy fine\-tuningLipmanet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib1),[2022](https://arxiv.org/html/2606.06967#bib.bib98)\); Liuet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib97)\)\.

Despite this progress, turning a good generative sampler into an effective on\-policy policy remains difficult\. The key obstacle is not action generation itself, but evaluating the probability of an already executed action under a new policy\. This state\-action likelihood ratio is required by clipped policy updates, KL control, and entropy regularization\. For Gaussian policies, the ratio is closed form\. For diffusion and flow policies, the sampler is usually a discretized neural ODE or denoising process: exact inversion can be nontrivial, and exact density evaluation may require neural\-network Jacobian determinants or costly trace estimatorsSonget al\.\([2020b](https://arxiv.org/html/2606.06967#bib.bib7)\)\. This creates a gap between the expressive power of generative policies and the likelihood\-based machinery that makes on\-policy learning stable\.

Recent methods have made important progress but still leave critical limitations\. Flow Policy Optimization \(FPO\)McAllisteret al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib120)\)avoids explicit density computation by replacing the true action\-density ratio with a tractable ELBO\-based surrogate\. This makes flow policies practical to optimize, but the surrogate ratio can differ from the true probability ratio when the variational gaps of the current and old policies differ\. As a result, clipping and regularization are applied to an approximate objective\. GenPODinget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib119)\)takes a complementary route by using exact diffusion inversion and dummy\-action augmentation to construct an invertible generative policy\. This enables exact likelihood\-ratio computation, but it doubles the effective action dimension, introduces redundant exploration variables, changes the standard flow/diffusion sampling procedure, and adds computational overhead\. Meanwhile, exact inversion methods such as EDICTWallaceet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib90)\), BDIAZhanget al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib94)\), and BELMWanget al\.\([2024a](https://arxiv.org/html/2606.06967#bib.bib92)\)show how to make diffusion trajectories reversible for reconstruction and editing, but they do not directly provide an efficient policy update mechanism for continuous control\.

In this work, we proposeGenPO\+\+, a reversible flow policy optimization framework for likelihood\-based on\-policy learning with expressive generative policies\. Our key observation is that the auxiliary memory needed for exact inversion does not have to be an independent dummy action\. Instead, it can be obtained from solver\-history states that naturally arise in high\-order ODE integration\. Based on this observation, GenPO\+\+ constructs a reversible high\-order flow\-policy solver whose inverse is available in closed form\. More importantly, the log\-determinant of the policy map is independent of the neural velocity network and depends only on fixed solver coefficients\. This yields an exact, Jacobian\-free likelihood ratio without changing the original action dimension\.

GenPO\+\+ has several practical advantages\. First, it avoids dummy\-action augmentation and therefore preserves the original action representation, which is important for fine\-tuning pretrained supervised flow or diffusion policies\. Second, it avoids repeated neural\-network Jacobian determinant computation, reducing the learning\-time overhead that appears in exact\-likelihood generative policy methods\. Third, by using an exact likelihood ratio rather than an ELBO surrogate, it improves training stability in settings where approximate objectives can drift from the true executed action distribution\.

We evaluate GenPO\+\+ on large\-scale simulated control, policy online fine\-tuning, and real\-world robotic manipulation tasks\. Across these settings, GenPO\+\+ achieves competitive or superior performance compared with Gaussian PPO, diffusion policy fine\-tuning, FPO, and GenPO, while improving stability and reducing inference or learning overhead\. Our contributions are summarized as follows:

- •Detailed Analysis of On\-Policy Generative RL\.We identify the central obstacle of applying flow\-based generative policies to on\-policy RL: the probability ratio of executed actions must be evaluated accurately after data collection\. We formalize how approximate ELBO\-ratio objectives can deviate from the true action\-density ratio, and explain why dummy\-action exact\-inversion methods introduce redundant policy dimensions and additional computation\.
- •High\-Order Reversible Generative Policy\.We propose GenPO\+\+, a high\-order reversible generative policy that replaces independent dummy actions with history states\. The resulting transition admits closed\-form inversion, preserves the original action dimension, and has a fixed\-coefficient log\-determinant independent of the neural velocity field, enabling exact Jacobian\-free likelihood\-ratio computation\. We further connect the update to Adams–Bashforth integration and derive its local truncation error\.
- •Simulation and Real\-World Evaluation\.We conduct experiments in large\-scale simulated benchmarks, imitation\-to\-RL fine\-tuning tasks, and real\-world dexterous hand manipulation tasks, demonstrating improved stability, efficiency, and final performance over prior generative policy optimization methods\.

## 2Related Works

### 2\.1Diffusion and Flow\-Based Generative Models

Diffusion models formulate generation as an iterative transformation from noise to data, typically by learning a reverse denoising process associated with a predefined denoising dynamicsSohl\-Dicksteinet al\.\([2015](https://arxiv.org/html/2606.06967#bib.bib4)\); Hoet al\.\([2020](https://arxiv.org/html/2606.06967#bib.bib3)\); Songet al\.\([2020b](https://arxiv.org/html/2606.06967#bib.bib7)\)\. Deterministic variants and probability\-flow perspectives, such as DDIMSonget al\.\([2020a](https://arxiv.org/html/2606.06967#bib.bib2)\)and score\-based probability\-flow ODEs, further connect diffusion sampling with continuous\-time transportSonget al\.\([2020a](https://arxiv.org/html/2606.06967#bib.bib2),[b](https://arxiv.org/html/2606.06967#bib.bib7)\)\. Flow\-based generative models provide a closely related transport view, where samples are generated by moving particles from a simple base distribution to a target distribution through an invertible or continuous\-time mappingRezende and Mohamed \([2015](https://arxiv.org/html/2606.06967#bib.bib105)\); Dinhet al\.\([2016](https://arxiv.org/html/2606.06967#bib.bib106)\); Kingma and Dhariwal \([2018](https://arxiv.org/html/2606.06967#bib.bib124)\); Chenet al\.\([2018](https://arxiv.org/html/2606.06967#bib.bib125)\); Grathwohlet al\.\([2018](https://arxiv.org/html/2606.06967#bib.bib126)\)\. Flow matching has recently emerged as a scalable way to train continuous\-time generative flows without simulation\-based maximum likelihood training\. Instead of optimizing exact likelihoods, it learns a velocity field by regressing conditional transport directions between source and target samplesLipmanet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib98)\)\. Conditional flow matching and optimal\-transport flow matching generalize this idea to broader coupling strategies and often produce simpler transport pathsTonget al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib127)\)\. Rectified flow further emphasizes straight transport trajectories and efficient generationLiuet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib97)\), while stochastic interpolants provide a unified framework connecting diffusion, stochastic dynamics, and deterministic flowsAlbergoet al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib128)\)\. These developments make flow models attractive for policy learning because they can represent expressive and multimodal conditional action distributions while often requiring fewer sampling steps than standard stochastic diffusion samplers\.

The practical behavior of flow policies is strongly shaped by the numerical solver used to discretize the underlying continuous\-time dynamics\. Higher\-order solvers reuse past model evaluations to improve trajectory accuracy or reduce the number of function evaluations, as shown by DPM\-SolverLuet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib100)\), DEISZhang and Chen \([2023](https://arxiv.org/html/2606.06967#bib.bib101)\), UniPCZhaoet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib102)\), and EDM\-style solversKarraset al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib103)\)\. Beyond sampling efficiency, the discretized solver also defines a transport map from latent noise to generated samples, whose invertibility becomes important when likelihood evaluation, latent recovery, or policy\-ratio computation is required\. Classical normalizing flows, including NICEDinhet al\.\([2014](https://arxiv.org/html/2606.06967#bib.bib129)\), RealNVPDinhet al\.\([2016](https://arxiv.org/html/2606.06967#bib.bib106)\), GlowKingma and Dhariwal \([2018](https://arxiv.org/html/2606.06967#bib.bib124)\), continuous normalizing flowsChenet al\.\([2018](https://arxiv.org/html/2606.06967#bib.bib125)\); Grathwohlet al\.\([2018](https://arxiv.org/html/2606.06967#bib.bib126)\), and residual\-flow models such as i\-ResNet and Residual FlowsBehrmannet al\.\([2019](https://arxiv.org/html/2606.06967#bib.bib130)\); Chenet al\.\([2019](https://arxiv.org/html/2606.06967#bib.bib132)\), obtain tractable likelihoods by building invertibility into the model architecture\. Recent diffusion inversion methods, including EDICTWallaceet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib90)\), BDIAZhanget al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib94)\), and BELMWanget al\.\([2024a](https://arxiv.org/html/2606.06967#bib.bib92)\), instead study how to make the sampling trajectory itself reversible through coupled transformations or bidirectional solver design\. These works highlight the importance of invertible generative maps, but they are mainly designed for density modeling, sampling, or image inversion rather than on\-policy likelihood\-ratio optimization\. In contrast, flow policies require a solver that simultaneously supports efficient action generation, stable inversion, and tractable likelihood ratios over large rollout batches\.

### 2\.2Generative Policy for Reinforcement Learning

Reinforcement learning aims to learn a policy through interactions with an environmentSutton and Barto \([2018](https://arxiv.org/html/2606.06967#bib.bib88)\)\. Compared with standard Gaussian policies, generative policies provide a more expressive class of conditional action distributions and can better capture multimodal behaviors\. They have shown strong empirical performance in imitation learning and offline decision making, including diffusion policies for behavior cloningChiet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib20)\); Reusset al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib23)\), trajectory generation and planningJanneret al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib42)\); Ajayet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib43)\), offline behavior modelingWanget al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib27)\); Chenet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib29)\), and offline actor\-critic learningHansen\-Estruchet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib31)\); Kanget al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib32)\)\. However, deploying generative policies in online RL is more challenging, since the policy must be improved from reward feedback rather than supervised action labels\.

Existing online generative\-policy methods are predominantly developed under off\-policy RL\. One line of work treats the generative policy as an expressive actor and optimizes it with value\-based or actor\-critic objectives, including normalizing\-flow policies combined with SAC or TRPOMazoureet al\.\([2020](https://arxiv.org/html/2606.06967#bib.bib110)\); Tang and Agrawal \([2018](https://arxiv.org/html/2606.06967#bib.bib109)\), DACERWanget al\.\([2024b](https://arxiv.org/html/2606.06967#bib.bib96)\), consistency\-policy learningChenet al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib116)\), DIMECeliket al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib57)\), and Flow Q\-LearningParket al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib26)\)\. Another line leverages the internal structure of generative models for policy improvement\. DIPOYanget al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib28)\)uses Q\-gradient guidance, QVPODinget al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib113)\)weights the diffusion variational objective by value estimates, QSMPsenkaet al\.\([2023](https://arxiv.org/html/2606.06967#bib.bib33)\)and maximum\-entropy diffusion policiesDonget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib117)\)interpret the denoising network as a score estimator of the target policy distribution, and MEowChaoet al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib108)\)exploits the layer\-wise structure of normalizing flows for maximum\-entropy policy learning\.

Recently, several works have begun to study generative policies in on\-policy RL\. DPPORenet al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib118)\)formulates the denoising process of a pretrained diffusion policy as an inner MDP and fine\-tunes it with policy\-gradient updates\. FPOMcAllisteret al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib120)\)brings flow matching into the on\-policy policy\-gradient framework by constructing a surrogate ratio from the conditional flow\-matching objective, thereby avoiding exact likelihood evaluation\. GenPODinget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib119)\)introduces an invertible diffusion\-policy construction to enable tractable action log\-likelihoods, KL estimation, and entropy regularization for PPO\-style updates\. PolicyFlowYanget al\.\([2026](https://arxiv.org/html/2606.06967#bib.bib133)\)approximates importance ratios through velocity field variations instead of evaluating likelihoods along the full flow path\. ReinFlowZhanget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib131)\)fine\-tunes flow\-matching policies by injecting learnable noise into the deterministic flow trajectory, converting it into a discrete\-time Markov process with tractable likelihood computation\.

## 3Limitation of Existing On\-Policy Generative RL

A flow policy generates actions by transporting a simple base distribution to the action space through a conditional neural ODEChenet al\.\([2018](https://arxiv.org/html/2606.06967#bib.bib125)\)\. Given a statess, the continuous flow is defined asd​xtd​t=vθ​\(xt,t\|s\)\\frac\{dx\_\{t\}\}\{dt\}=v\_\{\\theta\}\(x\_\{t\},t\|s\)\. In practice, action inference is performed by a finite\-step numerical solverxk\+1=xk\+vθ​\(xk,t\|s\)​Δtx\_\{k\+1\}=x\_\{k\}\+v\_\{\\theta\}\(x\_\{k\},t\|s\)\\Delta\_\{t\}, whereΔt=tk\+1−tk\\Delta\_\{t\}=t\_\{k\+1\}\-t\_\{k\}\. Therefore, the implemented policy is not only a continuous\-time transport model, but also a discretized sampler that maps latent noise to executable actions\. This distinction is crucial for on\-policy reinforcement learning\. Methods such as PPO optimize policies through likelihood ratios\. The clipped surrogate objective is

ℒclip​\(θ\)=𝔼t​\[min⁡\(rt​\(θ\)​A^t,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\],\\mathcal\{L\}\_\{\\mathrm\{clip\}\}\(\\theta\)=\\mathbb\{E\}\_\{t\}\\left\[\\min\(r\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\},\\,\\mathrm\{clip\}\\big\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)\\hat\{A\}\_\{t\}\)\\right\],\(1\)For Gaussian policies, this ratio is available in closed form\. However, for flow policies, evaluating its exact density requires either inverting the solver trajectory or accounting for the Jacobian determinant of the transport mapSonget al\.\([2020b](https://arxiv.org/html/2606.06967#bib.bib7)\)\. This makes the likelihood\-ratio computation substantially more difficult than action generation itself\.

Recent on\-policy generative\-policy methods therefore differ mainly in how they handle this likelihood\-ratio bottleneck\. FPOMcAllisteret al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib120)\)avoids direct density evaluation by replacing the true action likelihood with a variational surrogate\. In its formulation, the action likelihood admits the lower bound:

ELBOθ\(s,a\)=logπθ\(a\|s\)−DKL\(q\(z\|s,a\)∥pθ\(z\|s,a\)\),\\mathrm\{ELBO\}\_\{\\theta\}\(s,a\)=\\log\\pi\_\{\\theta\}\(a\|s\)\-D\_\{\\mathrm\{KL\}\}\\\!\\left\(q\(z\|s,a\)\\,\\\|\\,p\_\{\\theta\}\(z\|s,a\)\\right\),\(2\)wherezzdenotes the latent variable or latent path associated with the flow policy\. FPO then constructs the PPO ratio using the ELBO values,

ρFPO​\(s,a\)=exp⁡\(ELBOθ​\(s,a\)\)exp⁡\(ELBOθold​\(s,a\)\)\.\\rho\_\{\\rm FPO\}\(s,a\)=\\frac\{\\exp\(\\mathrm\{ELBO\}\_\{\\theta\}\(s,a\)\)\}\{\\exp\(\\mathrm\{ELBO\}\_\{\\theta\_\{\\mathrm\{old\}\}\}\(s,a\)\)\}\.\(3\)This makes the update tractable, but the resulting ratio is generally not the exact action\-density ratio because the variational gaps underθ\\thetaandθold\\theta\_\{\\mathrm\{old\}\}need not cancel\. Detailed proof can be seen in the Appendix[B](https://arxiv.org/html/2606.06967#A2)\. When these variational gaps vary across policy updates, the ELBO ratio can deviate from the true executed\-action density ratio\. Consequently, the resulting clipped update may be biased away from the likelihood\-ratio direction, which can contribute to instability in high\-dimensional on\-policy training\. This phenomenon is also clearly observed in our empirical experiments\.

GenPODinget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib119)\)takes the opposite direction and constructs an exactly invertible generative policy by augmenting the action space with dummy variables\. Given two action components\(xt,yt\)\(x\_\{t\},y\_\{t\}\), one step of its alternating update can be written as

x~t\+Δt\\displaystyle\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\}=xt\+vθ​\(yt,t\|s\)​Δt,\\displaystyle=x\_\{t\}\+v\_\{\\theta\}\(y\_\{t\},t\|s\)\\Delta\_\{t\},y~t\+Δt\\displaystyle\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\}=yt\+vθ​\(x~t\+Δt,t\|s\)​Δt,\\displaystyle=y\_\{t\}\+v\_\{\\theta\}\(\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\},t\|s\)\\Delta\_\{t\},\(4\)xt\+Δt\\displaystyle x\_\{t\+\\Delta\_\{t\}\}=p​x~t\+Δt\+\(1−p\)​y~t\+Δt,\\displaystyle=p\\,\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\}\+\(1\-p\)\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\},yt\+Δt\\displaystyle y\_\{t\+\\Delta\_\{t\}\}=p​y~t\+Δt\+\(1−p\)​xt\+Δt,\\displaystyle=p\\,\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\}\+\(1\-p\)x\_\{t\+\\Delta\_\{t\}\},while the inversion is followed as :

y~t\+Δt\\displaystyle\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\}=\(yt\+Δt−\(1−p\)​xt\+Δt\)/p,\\displaystyle=\(y\_\{t\+\\Delta\_\{t\}\}\-\(1\-p\)x\_\{t\+\\Delta\_\{t\}\}\)/p,x~t\+Δt\\displaystyle\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\}=\(xt\+Δt−\(1−p\)​y~t\+Δt\)/p,\\displaystyle=\(x\_\{t\+\\Delta\_\{t\}\}\-\(1\-p\)\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\}\)/p,\(5\)yt\\displaystyle y\_\{t\}=y~t\+Δt−vθ​\(x~t\+Δt,t\)​Δt,\\displaystyle=\\tilde\{y\}\_\{t\+\\Delta\_\{t\}\}\-v\_\{\\theta\}\(\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\},t\)\\Delta\_\{t\},xt\\displaystyle x\_\{t\}=x~t\+Δt−vθ​\(yt,t\)​Δt,\\displaystyle=\\tilde\{x\}\_\{t\+\\Delta\_\{t\}\}\-v\_\{\\theta\}\(y\_\{t\},t\)\\Delta\_\{t\},Because both the alternating update and the mixing operation are invertible, GenPO can compute likelihoods exactly in the augmented action space with change of variable\. However, GenPO is not well suited for fine\-tuning tasks\. Furthermore, each inference step requires two NFEs together with the computation of the Jacobian over the entire reverse process, substantially increasing computational overhead and hardware requirements\. In addition, the executed action is defined as the average of two latent variables, which further reduces the interpretability of the policy\.

To address these issues, we proposeGenPO\+\+, which builds a reversible high\-order flow\-policy solver using history states as auxiliary memory\. The resulting map admits exact inversion and a Jacobian\-free log\-determinant determined by fixed solver coefficients, enabling exact and efficient likelihood\-ratio optimization without changing the original action dimension\. With these properties, GenPO\+\+ achieves strong performance in both large\-scale simulated benchmarks and real\-world robotic tasks\.

## 4GenPO\+\+

We propose GenPO\+\+, a reversible flow policy optimization framework for likelihood\-based on\-policy reinforcement learning\. The method proceeds as follows\. First, we construct a high\-order reversible flow policy by using history states as auxiliary variables, which enables closed\-form inversion without augmenting the original action dimension\. Second, we show that the resulting history augmented transition has a fixed\-coefficient log\-determinant independent of the neural velocity field, allowing exact and Jacobian\-free likelihood ratio computation\. Finally, we instantiate this reversible flow policy by computing the exact augmented likelihood ratio for clipped policy optimization and adaptive KL control\.

![Refer to caption](https://arxiv.org/html/2606.06967v1/x2.png)Figure 2:Pipeline of GenPO\+\+\. GenPO obtains reversibility through dummy\-action augmentation, which doubles the action space and introduces redundant exploration, while GenPO\+\+ replaces dummy actions with solver\-history states in a reversible high\-order flow solver, achieving exact inversion and Jacobian\-free likelihood\-ratio computation while preserving the original action dimension\.### 4\.1Reversible Policy via High\-order Method

We reservessfor the environment state and usexix\_\{i\}to denote the flow variable at solver stepii\. The key difficulty in making an explicit flow solver exactly reversible is that the variable to be recovered in the inverse step is also the input used to query the velocity field\. For example, reversing a standard Euler\-type update requires evaluatingvθ​\(xi,ti∣s\)v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\\mid s\), whilexix\_\{i\}itself is unknown during inversion\. This coupling makes exact inversion require either a nonlinear solve or an approximation\.

GenPO\+\+ avoids this issue by decoupling the recovered variable from the velocity\-query variable\. Instead of evolving only a single flow variablexix\_\{i\}, we keep a augmented pair\(xi,xi−1\)\(x\_\{i\},x\_\{i\-1\}\)\. The next flow variable is constructed as a linear combination of the current variablexix\_\{i\}, the history variablexi−1x\_\{i\-1\}, and the velocity evaluated at the current variablevθ​\(xi,ti∣s\)v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\\mid s\)\. Since the output pair is\(xi\+1,xi\)\(x\_\{i\+1\},x\_\{i\}\), the inverse step already has access toxix\_\{i\}, and can therefore evaluatevθ​\(xi,ti∣s\)v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\\mid s\)directly and recoverxi−1x\_\{i\-1\}in closed form\. This gives an exactly reversible flow\-policy solver without introducing independent dummy actions\. We define a transitionFi​\(⋅\)F\_\{i\}\(\\cdot\)of the companion form at stepiiwhenσ≠0\\sigma\\neq 0:

\(xi\+1,xi\)\\displaystyle\(x\_\{i\+1\},x\_\{i\}\)=Fi,θ​\(xi,xi−1\|s\)\\displaystyle=F\_\{i,\\theta\}\(x\_\{i\},x\_\{i\-1\}\|s\)\(6\)xi\+1\\displaystyle x\_\{i\+1\}=\(1−σ\)​xi\+σ​xi−1\+\(1\+σ\)​Δt​vθ​\(xi,ti\|s\)\.\\displaystyle=\(1\-\\sigma\)x\_\{i\}\+\\sigma x\_\{i\-1\}\+\(1\+\\sigma\)\\Delta\_\{t\}\\,v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\|s\)\.This transition is nonlinear inxix\_\{i\}through the neural velocity field, but affine in the history statexi−1x\_\{i\-1\}with constant coefficientσ​I\\sigma I\. This structure makes the inverse available in closed form:

\(xi,xi−1\)\\displaystyle\(x\_\{i\},x\_\{i\-1\}\)=Fi,θ−1​\(xi\+1,xi\|s\)\\displaystyle=F^\{\-1\}\_\{i,\\theta\}\(x\_\{i\+1\},x\_\{i\}\|s\)\(7\)xi−1\\displaystyle x\_\{i\-1\}=σ−1​\[xi\+1−\(1−σ\)​xi−\(1\+σ\)​Δt​vθ​\(xi,ti\|s\)\]\\displaystyle=\\sigma^\{\-1\}\[x\_\{i\+1\}\-\(1\-\\sigma\)x\_\{i\}\-\(1\+\\sigma\)\\Delta\_\{t\}\\,v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\|s\)\]
###### Proposition 4\.1\(High\-order consistency of the reversible history update\)\.

The transition in Eq\. \([6](https://arxiv.org/html/2606.06967#S4.E6)\) can be interpreted as a relaxed history approximation to a second\-order Adams–Bashforth \(AB2\) method\. Specifically, the AB2 update can be decomposed as

xi\+1AB2=xi\+Δt​vi−12​Δt​\(vi−1−vi\),x\_\{i\+1\}^\{\\mathrm\{AB2\}\}=x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\-\\frac\{1\}\{2\}\\Delta\_\{t\}\\,\(v\_\{i\-1\}\-v\_\{i\}\),\(8\)Along a smooth ODE trajectory, this correction admits the state\-history approximation

−12​Δt​\(vi−1−vi\)=xi−1−xi\+Δt​vi\+O​\(Δt3\)\.\-\\frac\{1\}\{2\}\\Delta\_\{t\}\\,\(v\_\{i\-1\}\-v\_\{i\}\)=x\_\{i\-1\}\-x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+O\(\\Delta\_\{t\}^\{3\}\)\.\(9\)Therefore, the GenPO\+\+ update

x^i\+1=xi\+Δt​vi\+σ​\(xi−1−xi\+Δt​vi\)\\hat\{x\}\_\{i\+1\}=x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+\\sigma\\bigl\(x\_\{i\-1\}\-x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\\bigr\)\(10\)is a relaxed high\-order history correction\. Whenσ=1\\sigma=1, the update matches the AB2 correction up to anO​\(Δt3\)O\(\\Delta\_\{t\}^\{3\}\)local approximation error; when0<σ<10<\\sigma<1, it relaxes the high\-order correction while preserving the reversible state\-history structure\. Details see Appendix[C](https://arxiv.org/html/2606.06967#A3)\.

Next we quantify the numerical accuracy of this approximation by characterizing its local truncation error \(LTE\)Wanget al\.\([2024a](https://arxiv.org/html/2606.06967#bib.bib92)\)and making explicit how the history coefficientσ\\sigmacontrols the deviation from the underlying ODE trajectory\.

###### Theorem 4\.2\(Local truncation error of the state\-history update\)\.

Assume thatvθ​\(x,t\|s\)v\_\{\\theta\}\(x,t\|s\)is sufficiently smooth along the exact ODE trajectory\. For the GenPO\+\+ update Eq\. \([6](https://arxiv.org/html/2606.06967#S4.E6)\), the standard local truncation error under trajectory\-consistent history satisfies the following relation, see Appendix[D](https://arxiv.org/html/2606.06967#A4):

xi\+1−xi\+1⋆=σ−12​Δt2​dd​t​v​\(xi⋆,ti\|s\)\+O​\(Δt3\)\.\{x\}\_\{i\+1\}\-x^\{\\star\}\_\{i\+1\}=\\frac\{\\sigma\-1\}\{2\}\\Delta\_\{t\}^\{2\}\\frac\{d\}\{dt\}v\(x^\{\\star\}\_\{i\},t\_\{i\}\|s\)\+O\(\\Delta\_\{t\}^\{3\}\)\.\(11\)

### 4\.2Efficient generative policy optimization

Given the reversible transition in Section[4\.1](https://arxiv.org/html/2606.06967#S4.SS1), likelihood evaluation reduces to a change\-of\-variables computation over the augmented solver trajectory\. The crucial property of GenPO\+\+ is that this computation does not require differentiating through the neural velocity field\. Although each transition is nonlinear in the current flow variable throughvθv\_\{\\theta\}, its companion\-form dependence on the history variable makes the Jacobian determinant a fixed coefficient\. For compactness, we write the paired solver variable aszi=\(xi,xi−1\)z\_\{i\}=\(x\_\{i\},x\_\{i\-1\}\)\.

##### Jacobian\-free likelihood\.

The Jacobian of the forward map \([6](https://arxiv.org/html/2606.06967#S4.E6)\) with respect to this paired variable is given by:

Ji=∂Fi,θ∂zi=\[\(1−σ\)​I\+\(1\+σ\)​∇xvθ​\(xi,ti\|s\)σI0\]\.J\_\{i\}=\\frac\{\\partial F\_\{i,\\theta\}\}\{\\partial z\_\{i\}\}=\\begin\{bmatrix\}\(1\-\\sigma\)I\+\(1\+\\sigma\)\\nabla\_\{x\}v\_\{\\theta\}\(x\_\{i\},t\_\{i\}\|s\)&\\sigma\\\\ I&0\\end\{bmatrix\}\.\(12\)The determinant of this companion\-form matrix is constant,detJi=\(−1\)d​σd,\\det J\_\{i\}=\(\-1\)^\{d\}\\sigma^\{d\},whereddis the dimension of the action space\. This implies that the likelihood ratio ultimately depends only on the noise variable, and not on the specific forward generation process\. As long as a given action can be inverted back to the noise space, the exact ratio can be computed directly\.

r~​\(θ\)=π~θ​\(x1,x1−Δt\|s\)π~old​\(x1,x1−Δt\|s\)=π~θ​\(F0,θ​…​F1,θ​\(z0,θ\)\|s\)π~old​\(F0,old​…​F1,old​\(z0,old\)\|s\)=p~​\(z0,θ\|s\)​∏detJip~​\(z0,old\|s\)​∏detJi=p~​\(z0,θ\|s\)p~​\(z0,old\|s\)\\tilde\{r\}\(\\theta\)=\\frac\{\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\{\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}=\\frac\{\\tilde\{\\pi\}\_\{\\theta\}\(F\_\{0,\\theta\}\\dots F\_\{1,\\theta\}\(z\_\{0,\\theta\}\)\|s\)\}\{\\tilde\{\\pi\}\_\{\\rm old\}\(F\_\{0,\\rm old\}\\dots F\_\{1,\\rm old\}\(z\_\{0,\\rm old\}\)\|s\)\}=\\frac\{\\tilde\{p\}\(z\_\{0,\\theta\}\|s\)\\prod\\det J\_\{i\}\}\{\\tilde\{p\}\(z\_\{0,\\rm old\}\|s\)\\prod\\det J\_\{i\}\}=\\frac\{\\tilde\{p\}\(z\_\{0,\\theta\}\|s\)\}\{\\tilde\{p\}\(z\_\{0,\\rm old\}\|s\)\}\(13\)wherep~∼𝒩​\(0,I\)\\tilde\{p\}\\sim\\mathcal\{N\}\(0,I\)\. Although GenPO\+\+ computes the likelihood ratio using the joint distribution of two consecutive states, its interaction with the environment still occurs throughx1x\_\{1\}\. We further show that this optimization objective is consistent with the standard reinforcement learning objective\. Details see appendix[E](https://arxiv.org/html/2606.06967#A5)\.

###### Proposition 4\.3\(Consistency of augmented\-action likelihood ratios\)\.

For any statess, letπ~θ​\(x1,x1−Δt\|s\)\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)be the GenPO\+\+ augmented policy andπθ​\(x1\|s\)=∫π~θ​\(x1,x1−Δt\|s\)​𝑑x1−Δt\\pi\_\{\\theta\}\(x\_\{1\}\|s\)=\\int\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\,dx\_\{1\-\\Delta\_\{t\}\}be its executed\-action marginal\. Then, for any advantage function depending only on the executed actionx1x\_\{1\},

𝔼\(x1,x1−Δt\)∼π~θold\(⋅,⋅\|s\)​\[r~θ​\(x1,x1−Δt\|s\)​Aπold​\(s,x1\)\]=𝔼x1∼πθold\(⋅\|s\)​\[rθ​\(x1\|s\)​Aπold​\(s,x1\)\]\.\\mathbb\{E\}\_\{\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)\\sim\\tilde\{\\pi\}\_\{\\theta\_\{\\rm old\}\}\(\\cdot,\\cdot\|s\)\}\\left\[\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\right\]=\\mathbb\{E\}\_\{x\_\{1\}\\sim\\pi\_\{\\theta\_\{\\rm old\}\}\(\\cdot\|s\)\}\\left\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\right\]\.\(14\)

### 4\.3Practical Implementation

During rollout, the policy samples an augmented base variablez0∼p~0z\_\{0\}\\sim\\tilde\{p\}\_\{0\}, generates the terminal augmented solver statez1=\(x1,x1−Δt\)z\_\{1\}=\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)through the reversible solver in Eq\. \(6\), and executes only the first componenta=x1a=x\_\{1\}in the environment\. During policy updates, to enable adaptive scheduling of the learning rate, we use the augmented likelihood ratio to monitor policy deviation during PPO updates\. The empirical augmented KL is estimated as

D^KLaug=−𝔼\(x1,x1−Δt\)∼π~o​l​d​\(x1,x1−Δt\|s\)​\[log⁡r~θ\]\.\\widehat\{D\}\_\{\\rm KL\}^\{\\rm aug\}=\-\\mathbb\{E\}\_\{\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)\\sim\\tilde\{\\pi\}\_\{old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\\left\[\\log\\tilde\{r\}\_\{\\theta\}\\right\]\.\(15\)Since marginalization contracts KL divergence, the augmented KL upper\-bounds the corresponding KL over executed actions\. We use it as a conservative proxy for adaptive learning\-rate control, and use a slightly larger target KL in high\-dimensional tasks to avoid overly conservative updates that may prematurely limit exploration and policy improvement\. The policy is updated with the standard clipped surrogate objective:

ℒclipGenPO\+⁣\+​\(θ\)=𝔼\(x1,x1−Δt\)∼π~old\(⋅\|s\)​\[min⁡\(r~​\(θ\)​Aπold​\(x1\),clip​\(r~​\(θ\),1−ϵ,1\+ϵ\)​Aπold​\(x1\)\)\]\.\\mathcal\{L\}^\{\\mathrm\{GenPO\+\+\}\}\_\{\\rm clip\}\(\\theta\)=\\mathbb\{E\}\_\{\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)\\sim\\tilde\{\\pi\}\_\{\\mathrm\{old\}\}\(\\cdot\|s\)\}\[\\min\(\\tilde\{r\}\(\\theta\)\\,A^\{\\pi\_\{\\mathrm\{old\}\}\}\(x\_\{1\}\),\\;\\mathrm\{clip\}\(\\tilde\{r\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\,A^\{\\pi\_\{\\mathrm\{old\}\}\}\(x\_\{1\}\)\)\]\.\(16\)Finally, since GenPO\+\+ interacts with the environment using the terminal ODE statex1x\_\{1\}, it avoids the dummy\-action space introduced by GenPO and therefore does not require an additional compression loss to suppress ineffective exploration\.

## 5Experiments

![Refer to caption](https://arxiv.org/html/2606.06967v1/x3.png)Figure 3:Learning curves across 8 IsaacLab benchmarks\. Results are averaged over 5 runs\. The x\-axis denotes iterations, and the y\-axis shows average episodic return with one standard deviation shaded\.In this section, we demonstrate the performance of GenPO\+\+ in three on\-policy RL scenarios: fine\-tuning, learning from scratch, and real\-world manipulation\. These settings are chosen to test the main claims: i\) whether exact reversible likelihood\-ratio optimization can improve pretrained generative policies, ii\) remain stable when learning high\-dimensional control policies from scratch, and iii\) reduce the computational overhead of prior exact\-likelihood generative policy methods\. We compare against Gaussian PPO and representative generative\-policy baselines, including DPPORenet al\.\([2024](https://arxiv.org/html/2606.06967#bib.bib118)\), FPOYiet al\.\([2026](https://arxiv.org/html/2606.06967#bib.bib121)\); McAllisteret al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib120)\), GenPODinget al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib119)\), and PolicyFlowYanget al\.\([2026](https://arxiv.org/html/2606.06967#bib.bib133)\)\. Unless otherwise stated, all methods use the same environment configuration, rollout budget, and critic architecture\. We report means over five random seeds with standard\-error shading\.

### 5\.1Learning from Scratch on IsaacLab

![Refer to caption](https://arxiv.org/html/2606.06967v1/x4.png)Figure 4:Ablation experiments ofσ\\sigma\.We next evaluate methods in IsaacLab across locomotion, manipulation, and whole\-body control tasks, includingAnt,Humanoid,Open\-Drawer,Anymal\-D\-Rough,Go2\-Rough,G1\-Rough,H1\-Rough, andDigit\-LocoManip\.

Figure[3](https://arxiv.org/html/2606.06967#S5.F3)shows that GenPO\+\+ achieves superior returns across the benchmark\. In contrast, FPO often exhibits large variance and collapse, suggesting that approximate likelihood\-ratio objectives can be unstable in high\-dimensional control\. Table[1](https://arxiv.org/html/2606.06967#S5.T1)compares rollout collection and policy learning time on IsaacLabHumanoid\. The collection time is similar across flow\-based methods, whereas update costs vary greatly\. FPO is expensive due to its heavier optimization schedule, and GenPO is slow because exact likelihood evaluation requires Jacobian\-related computation through the inverse process\. GenPO\+\+ substantially reduces total learning time compared with GenPO under the reported training configuration, while maintaining comparable rollout collection time among flow\-based methods\. This is more obvious in the Figure[11](https://arxiv.org/html/2606.06967#A6.F11)\. Figure[4](https://arxiv.org/html/2606.06967#S5.F4)studies the effect of the coefficientσ\\sigma\. GenPO\+\+ achieves stable performance across the tested range, suggesting that its empirical behavior is robust to moderate variations of this parameter\.

### 5\.2Manipulation Fine\-tuning on Robomimic

![Refer to caption](https://arxiv.org/html/2606.06967v1/x5.png)Figure 5:Online fine\-tuning results on three Robomimic benchmarks\. Top row reports zero\-noise sampling success rate, and bottom row reports random\-noise sampling success rate\. Curves are averaged over 5 runs with one standard error shaded\.We first study online fine\-tuning of pretrained flow\-matching policies on three Robomimic manipulation tasks:Can,Box, andThreading\. Each method is initialized from the same pretrained checkpoint and fine\-tuned with online rewards\. We evaluate both zero\-noise sampling, which measures the learned action mode, and random\-noise sampling, which tests the full stochastic policy distribution\.

Figure[5](https://arxiv.org/html/2606.06967#S5.F5)shows that GenPO\+\+ consistently improves pretrained flow policies across all three Robomimic tasks\. It preserves strong zero\-sampling performance onCan, reaches high success quickly onBox, and achieves the best final performance onThreadingunder both zero and random sampling\. This indicates that exact reversible likelihood\-ratio optimization helps GenPO\+\+

Table 1:Comparison of training efficiency\. Training time are reported in humanoid env\.CategoryMethodTrainingTime \(min\)Gaussian\-basedPPO13\.2513\.25Flow\-basedFPO72\.0672\.06GenPO132\.30132\.30\\rowcolornipsgreen\!12GenPO\+\+20\.7820\.78improve the learned action distribution during online fine\-tuning, leading to more reliable stochastic policy improvement\.

### 5\.3Real\-World Dexterous Hand Manipulation

Finally, we deploy GenPO\+\+ on a real\-world dexterous manipulation task using the RobotEra Xhand platform\. The task requires the hand to rotate and loosen a nut from a bolt through repeated in\-hand contact, which involves hardware actuation errors that are difficult to model accurately in simulation\. We follow the simulation training framework ofHsiehet al\.\([2025](https://arxiv.org/html/2606.06967#bib.bib123)\), and instantiate the algorithm fromQiet al\.\([2022](https://arxiv.org/html/2606.06967#bib.bib122)\)by replacing its PPO optimizer with GenPO\+\+\.

![Refer to caption](https://arxiv.org/html/2606.06967v1/x6.png)Figure 6:Episode rewards of GenPO\+\+ and PPO during dexterous hand manipulation training\.Figure[6](https://arxiv.org/html/2606.06967#S5.F6)shows that GenPO\+\+ achieves faster reward improvement and higher final performance than PPO during simulation training\. Figure[7](https://arxiv.org/html/2606.06967#S5.F7)further demonstrates successful real\-world deployment, where the learned policy loosens nuts from bolts with different geometries\. These results suggest that GenPO\+\+ improves both training efficiency and robustness under sim\-to\-real variations\. More details are provided in Appendix[F\.4](https://arxiv.org/html/2606.06967#A6.SS4)\.

![Refer to caption](https://arxiv.org/html/2606.06967v1/x7.png)Figure 7:Sequential video frames of the real\-world evaluation task\. GenPO\+\+ controls the dexterous hand to rotate and loosen nuts from bolts of different sizes, where different colors indicate different bolt geometries\.

## 6Conclusion and Limitations

In this work, we presentedGenPO\+\+, a reversible flow policy optimization framework that replaces dummy\-action augmentation with solver\-history states, yielding exact inversion and a Jacobian\-free log\-determinant while preserving the original action dimension\. Experiments across simulated control, imitation\-to\-RL fine\-tuning, and real\-world robotic manipulation show that this design improves stability and efficiency over approximate\-ratio methods and avoids the usage of dummy actions\. However, GenPO\+\+ introduces the solver\-history coefficientσ\\sigma: whenσ\\sigmais too large, the reversible update may deviate too much from the original flow process and distort the generated policy; whenσ\\sigmais too small, numerical stability and reversibility can degrade\. How to setσ\\sigmaadaptively, or design improved reversible solvers that reduce sensitivity to this parameter, is an important problem for future work\.

## References

- \[1\]A\. Ajay, Y\. Du, A\. Gupta, J\. Tenenbaum, T\. Jaakkola, and P\. Agrawal\(2022\)Is conditional generative modeling all you need for decision\-making?\.arXiv preprint arXiv:2211\.15657\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[2\]M\. Albergo, N\. M\. Boffi, and E\. Vanden\-Eijnden\(2025\)Stochastic interpolants: a unifying framework for flows and diffusions\.Journal of Machine Learning Research26\(209\),pp\. 1–80\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[3\]J\. Behrmann, W\. Grathwohl, R\. T\. Chen, D\. Duvenaud, and J\. Jacobsen\(2019\)Invertible residual networks\.InInternational conference on machine learning,pp\. 573–582\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[4\]O\. Celik, Z\. Li, D\. Blessing, G\. Li, D\. Palanicek, J\. Peters, G\. Chalvatzaki, and G\. Neumann\(2025\)DIME: diffusion\-based maximum entropy reinforcement learning\.arXiv preprint arXiv:2502\.02316\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[5\]C\. Chao, C\. Feng, W\. Sun, C\. Lee, S\. See, and C\. Lee\(2024\)Maximum entropy reinforcement learning via energy\-based normalizing flow\.arXiv preprint arXiv:2405\.13629\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[6\]H\. Chen, C\. Lu, C\. Ying, H\. Su, and J\. Zhu\(2022\)Offline reinforcement learning via high\-fidelity generative behavior modeling\.arXiv preprint arXiv:2209\.14548\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[7\]R\. T\. Chen, J\. Behrmann, D\. K\. Duvenaud, and J\. Jacobsen\(2019\)Residual flows for invertible generative modeling\.Advances in neural information processing systems32\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[8\]R\. T\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. K\. Duvenaud\(2018\)Neural ordinary differential equations\.Advances in neural information processing systems31\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1),[§3](https://arxiv.org/html/2606.06967#S3.p1.4)\.
- \[9\]Y\. Chen, H\. Li, and D\. Zhao\(2024\)Boosting continuous control with consistency policy\.InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems,pp\. 335–344\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[10\]C\. Chi, S\. Feng, Y\. Du, Z\. Xu, E\. Cousineau, B\. Burchfiel, and S\. Song\(2023\)Diffusion policy: visuomotor policy learning via action diffusion\.arXiv preprint arXiv:2303\.04137\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[11\]S\. Ding, K\. Hu, Z\. Zhang, K\. Ren, W\. Zhang, J\. Yu, J\. Wang, and Y\. Shi\(2024\)Diffusion\-based reinforcement learning via q\-weighted variational policy optimization\.arXiv preprint arXiv:2405\.16173\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[12\]S\. Ding, K\. Hu, S\. Zhong, H\. Luo, W\. Zhang, J\. Wang, J\. Wang, and Y\. Shi\(2025\)GenPO: generative diffusion models meet on\-policy reinforcement learning\.arXiv preprint arXiv:2505\.18763\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p3.1),[§3](https://arxiv.org/html/2606.06967#S3.p3.1),[§5](https://arxiv.org/html/2606.06967#S5.p1.1)\.
- \[13\]L\. Dinh, D\. Krueger, and Y\. Bengio\(2014\)Nice: non\-linear independent components estimation\.arXiv preprint arXiv:1410\.8516\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[14\]L\. Dinh, J\. Sohl\-Dickstein, and S\. Bengio\(2016\)Density estimation using real nvp\.arXiv preprint arXiv:1605\.08803\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[15\]X\. Dong, J\. Cheng, and X\. S\. Zhang\(2025\)Maximum entropy reinforcement learning with diffusion policy\.arXiv preprint arXiv:2502\.11612\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[16\]W\. Grathwohl, R\. T\. Chen, J\. Bettencourt, I\. Sutskever, and D\. Duvenaud\(2018\)Ffjord: free\-form continuous dynamics for scalable reversible generative models\.arXiv preprint arXiv:1810\.01367\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[17\]P\. Hansen\-Estruch, I\. Kostrikov, M\. Janner, J\. G\. Kuba, and S\. Levine\(2023\)Idql: implicit q\-learning as an actor\-critic method with diffusion policies\.arXiv preprint arXiv:2304\.10573\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[18\]J\. Ho, A\. Jain, and P\. Abbeel\(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[19\]E\. Hsieh, W\. Hsieh, Y\. Wang, T\. Lin, J\. Malik, K\. Sreenath, and H\. Qi\(2025\)Learning dexterous manipulation skills from imperfect simulations\.arXiv:2512\.02011\.Cited by:[§5\.3](https://arxiv.org/html/2606.06967#S5.SS3.p1.1)\.
- \[20\]M\. Janner, Y\. Du, J\. B\. Tenenbaum, and S\. Levine\(2022\)Planning with diffusion for flexible behavior synthesis\.arXiv preprint arXiv:2205\.09991\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[21\]B\. Kang, X\. Ma, C\. Du, T\. Pang, and S\. Yan\(2024\)Efficient diffusion policies for offline reinforcement learning\.Advances in Neural Information Processing Systems36\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[22\]T\. Karras, M\. Aittala, T\. Aila, and S\. Laine\(2022\)Elucidating the design space of diffusion\-based generative models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 26565–26577\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[23\]D\. P\. Kingma and P\. Dhariwal\(2018\)Glow: generative flow with invertible 1x1 convolutions\.Advances in neural information processing systems31\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[24\]Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le\(2022\)Flow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[25\]Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le\(2023\)Flow matching for generative modeling\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1)\.
- \[26\]X\. Liu, C\. Gong, and Q\. Liu\(2022\)Flow straight and fast: learning to generate and transfer data with rectified flow\.arXiv preprint arXiv:2209\.03003\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[27\]C\. Lu, Y\. Zhou, F\. Bao, J\. Chen, C\. Li, and J\. Zhu\(2022\)DPM\-Solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 5775–5787\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[28\]B\. Mazoure, T\. Doan, A\. Durand, J\. Pineau, and R\. D\. Hjelm\(2020\)Leveraging exploration in off\-policy algorithms via normalizing flows\.InConference on Robot Learning,pp\. 430–444\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[29\]D\. McAllister, S\. Ge, B\. Yi, C\. M\. Kim, E\. Weber, H\. Choi, H\. Feng, and A\. Kanazawa\(2025\)Flow matching policy gradients\.arXiv preprint arXiv:2507\.21053\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p3.1),[§3](https://arxiv.org/html/2606.06967#S3.p2.4),[§5](https://arxiv.org/html/2606.06967#S5.p1.1)\.
- \[30\]S\. Park, Q\. Li, and S\. Levine\(2025\)Flow q\-learning\.arXiv preprint arXiv:2502\.02538\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[31\]M\. Psenka, A\. Escontrela, P\. Abbeel, and Y\. Ma\(2023\)Learning a diffusion model policy from rewards via q\-score matching\.arXiv preprint arXiv:2312\.11752\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[32\]H\. Qi, A\. Kumar, R\. Calandra, Y\. Ma, and J\. Malik\(2022\)In\-Hand Object Rotation via Rapid Motor Adaptation\.InConference on Robot Learning \(CoRL\),Cited by:[§5\.3](https://arxiv.org/html/2606.06967#S5.SS3.p1.1)\.
- \[33\]A\. Z\. Ren, J\. Lidard, L\. L\. Ankile, A\. Simeonov, P\. Agrawal, A\. Majumdar, B\. Burchfiel, H\. Dai, and M\. Simchowitz\(2024\)Diffusion policy policy optimization\.arXiv preprint arXiv:2409\.00588\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p3.1),[§5](https://arxiv.org/html/2606.06967#S5.p1.1)\.
- \[34\]M\. Reuss, M\. Li, X\. Jia, and R\. Lioutikov\(2023\)Goal\-conditioned imitation learning using score\-based diffusion policies\.arXiv preprint arXiv:2304\.02532\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[35\]D\. Rezende and S\. Mohamed\(2015\)Variational inference with normalizing flows\.InInternational conference on machine learning,pp\. 1530–1538\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[36\]Q\. Rouxel, A\. Ferrari, S\. Ivaldi, and J\. Mouret\(2024\)Flow matching imitation learning for multi\-support manipulation\.In2024 IEEE\-RAS 23rd International Conference on Humanoid Robots \(Humanoids\),pp\. 528–535\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1)\.
- \[37\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1)\.
- \[38\]J\. Sohl\-Dickstein, E\. Weiss, N\. Maheswaranathan, and S\. Ganguli\(2015\)Deep unsupervised learning using nonequilibrium thermodynamics\.InInternational conference on machine learning,pp\. 2256–2265\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[39\]J\. Song, C\. Meng, and S\. Ermon\(2020\)Denoising diffusion implicit models\.arXiv preprint arXiv:2010\.02502\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[40\]Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole\(2020\)Score\-based generative modeling through stochastic differential equations\.arXiv preprint arXiv:2011\.13456\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1),[§3](https://arxiv.org/html/2606.06967#S3.p1.5)\.
- \[41\]R\. S\. Sutton and A\. G\. Barto\(2018\)Reinforcement learning: an introduction\.MIT press\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[42\]Y\. Tang and S\. Agrawal\(2018\)Boosting trust region policy optimization by normalizing flows policy\.arXiv preprint arXiv:1809\.10326\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[43\]A\. Tong, K\. Fatras, N\. Malkin, G\. Huguet, Y\. Zhang, J\. Rector\-Brooks, G\. Wolf, and Y\. Bengio\(2023\)Improving and generalizing flow\-based generative models with minibatch optimal transport\.arXiv preprint arXiv:2302\.00482\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p1.1)\.
- \[44\]B\. Wallace, A\. Gokul, and N\. Naik\(2023\)Edict: exact diffusion inversion via coupled transformations\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22532–22541\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[45\]F\. Wang, H\. Yin, Y\. Dong, H\. Zhu, H\. Zhao, H\. Qian, C\. Li,et al\.\(2024\)Belm: bidirectional explicit linear multi\-step sampler for exact inversion in diffusion models\.Advances in Neural Information Processing Systems37,pp\. 46118–46159\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2606.06967#S4.SS1.p3.1)\.
- \[46\]Y\. Wang, L\. Wang, Y\. Jiang, W\. Zou, T\. Liu, X\. Song, W\. Wang, L\. Xiao, J\. Wu, J\. Duan,et al\.\(2024\)Diffusion actor\-critic with entropy regulator\.Advances in Neural Information Processing Systems37,pp\. 54183–54204\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[47\]Z\. Wang, J\. J\. Hunt, and M\. Zhou\(2022\)Diffusion policies as an expressive policy class for offline reinforcement learning\.InThe Eleventh International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p1.1),[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p1.1)\.
- \[48\]L\. Yang, Z\. Huang, F\. Lei, Y\. Zhong, Y\. Yang, C\. Fang, S\. Wen, B\. Zhou, and Z\. Lin\(2023\)Policy representation via diffusion probability model for reinforcement learning\.arXiv preprint arXiv:2305\.13122\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p2.1)\.
- \[49\]S\. Yang, B\. Liu, and H\. Chen\(2026\)PolicyFlow: policy optimization with continuous normalizing flow in reinforcement learning\.arXiv preprint arXiv:2602\.01156\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p3.1),[§5](https://arxiv.org/html/2606.06967#S5.p1.1)\.
- \[50\]B\. Yi, H\. Choi, H\. G\. Singh, X\. Huang, T\. E\. Truong, C\. Sferrazza, Y\. Ma, R\. Duan, P\. Abbeel, G\. Shi,et al\.\(2026\)Flow policy gradients for robot control\.arXiv preprint arXiv:2602\.02481\.Cited by:[§5](https://arxiv.org/html/2606.06967#S5.p1.1)\.
- \[51\]G\. Zhang, J\. P\. Lewis, and W\. B\. Kleijn\(2024\)Exact diffusion inversion via bidirectional integration approximation\.InEuropean Conference on Computer Vision,pp\. 19–36\.Cited by:[§1](https://arxiv.org/html/2606.06967#S1.p3.1),[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[52\]Q\. Zhang and Y\. Chen\(2023\)Fast sampling of diffusion models with exponential integrator\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.
- \[53\]T\. Zhang, C\. Yu, S\. Su, and Y\. Wang\(2025\)ReinFlow: fine\-tuning flow matching policy with online reinforcement learning\.arXiv preprint arXiv:2505\.22094\.Cited by:[§2\.2](https://arxiv.org/html/2606.06967#S2.SS2.p3.1)\.
- \[54\]W\. Zhao, L\. Bai, Y\. Rao, J\. Zhou, and J\. Lu\(2023\)UniPC: a unified predictor\-corrector framework for fast sampling of diffusion models\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 49842–49869\.Cited by:[§2\.1](https://arxiv.org/html/2606.06967#S2.SS1.p2.1)\.

## Appendix AAlgorithm

Algorithm 1GenPO\+\+Input:flow policyvθ​\(x,s,t\)v\_\{\\theta\}\(x,s,t\)with base densityp~0​\(z\)\\tilde\{p\}\_\{0\}\(z\), value networkVω​\(s\)V\_\{\\omega\}\(s\), solver stepsMM, rollout horizonNN, PPO epochsKK, history coefficientσ\\sigma\.

1:

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta
2:for

ttin

1,2,⋯,T1,2,\\cdots,Tdo

3:foreach actordo

4:Initialize an empty rollout buffer

𝒟\\mathcal\{D\}
5:for

nnin

1,2,⋯,N1,2,\\cdots,Ndo

6:Sample base action

z0,n∼p~0​\(z\)z\_\{0,n\}\\sim\\tilde\{p\}\_\{0\}\(z\)and generate the augmented action

z1,nz\_\{1,n\}by \([6](https://arxiv.org/html/2606.06967#S4.E6)\):

7:Execute the environment action

an=x1,na\_\{n\}=x\_\{1,n\}
8:Observe reward

rnr\_\{n\}and next state

sn\+1s\_\{n\+1\}
9:Store

\(sn,an,z1,n,rn,sn\+1\)\(s\_\{n\},a\_\{n\},z\_\{1,n\},r\_\{n\},s\_\{n\+1\}\)in

𝒟\\mathcal\{D\}
10:endfor

11:Compute old augmented log\-likelihoods

log⁡π~θold​\(x1,x1−Δt∣s\)\\log\\tilde\{\\pi\}\_\{\\theta\_\{\\mathrm\{old\}\}\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\\mid s\)by exact inversion

12:Estimate advantages

A^1,⋯,A^N\\hat\{A\}\_\{1\},\\cdots,\\hat\{A\}\_\{N\}and returns

R^1,⋯,R^N\\hat\{R\}\_\{1\},\\cdots,\\hat\{R\}\_\{N\}using GAE

13:endfor

14:for

kkin

1,2,⋯,K1,2,\\cdots,Kdo

15:Sample a mini\-batch

ℬ\\mathcal\{B\}from the rollout buffer

𝒟\\mathcal\{D\}
16:For each

\(s,z1\)∈ℬ\(s,z\_\{1\}\)\\in\\mathcal\{B\}, recover the base solver state under the current policy:

z0,θ=F0,θ−1∘⋯∘F1,θ−1​\(z1\|s\),z\_\{0,\\theta\}=F^\{\-1\}\_\{0,\\theta\}\\circ\\cdots\\circ F^\{\-1\}\_\{1,\\theta\}\(z\_\{1\}\|s\),
17:Compute the exact augmented likelihood ratio:

r~θ=exp⁡\(log⁡π~θ​\(z0,θ\|s\)−log⁡π~old​\(z0,old\|s\)\)\\tilde\{r\}\_\{\\theta\}=\\exp\(\\log\\tilde\{\\pi\}\_\{\\theta\}\(z\_\{0,\\theta\}\|s\)\-\\log\\tilde\{\\pi\}\_\{\\mathrm\{old\}\}\(z\_\{0,\\rm old\}\|s\)\)
18:Update

πθ\\pi\_\{\\theta\}by maximizing the clipped GenPO\+\+ objective \([16](https://arxiv.org/html/2606.06967#S4.E16)\):

19:Update the value network

Vω​\(s\)V\_\{\\omega\}\(s\)with

ℒV​\(ω\)=𝔼ℬ​\[‖Vω​\(s\)−R^‖2\]\\mathcal\{L\}\_\{V\}\(\\omega\)=\\mathbb\{E\}\_\{\\mathcal\{B\}\}\[\\\|V\_\{\\omega\}\(s\)\-\\hat\{R\}\\\|^\{2\}\]
20:endfor

21:

θold←θ\\theta\_\{\\mathrm\{old\}\}\\leftarrow\\theta
22:endfor

## Appendix BELBO\-Ratio Bias in FPO

FPO replaces the exact likelihood ratio by an ELBO ratio\. For a fixed state\-action pair\(s,a\)\(s,a\), write the ELBO as

ℰθ​\(s,a\)=log⁡πθ​\(a\|s\)−Δθ​\(s,a\),\\mathcal\{E\}\_\{\\theta\}\(s,a\)=\\log\\pi\_\{\\theta\}\(a\|s\)\-\\Delta\_\{\\theta\}\(s,a\),\(17\)whereΔθ​\(s,a\)≥0\\Delta\_\{\\theta\}\(s,a\)\\geq 0denotes the variational gap between the true log\-likelihood and the ELBO\.

###### Proposition B\.1\(ELBO\-ratio decomposition\)\.

Let

ρFPO​\(θ;s,a\)=exp⁡\(ℰθ​\(s,a\)−ℰθold​\(s,a\)\)\\rho\_\{\\rm FPO\}\(\\theta;s,a\)=\\exp\\\!\(\\mathcal\{E\}\_\{\\theta\}\(s,a\)\-\\mathcal\{E\}\_\{\\theta\_\{\\rm old\}\}\(s,a\)\)\(18\)be the ELBO ratio used in place of the exact policy ratio\. Then

ρFPO​\(θ;s,a\)=rπ​\(θ;s,a\)​exp⁡\(Δθold​\(s,a\)−Δθ​\(s,a\)\)\.\\rho\_\{\\rm FPO\}\(\\theta;s,a\)=r\_\{\\pi\}\(\\theta;s,a\)\\exp\\\!\(\\Delta\_\{\\theta\_\{\\rm old\}\}\(s,a\)\-\\Delta\_\{\\theta\}\(s,a\)\)\.\(19\)Moreover, for the unclipped local objective

ℒFPO​\(θ;s,a\)=A​\(s,a\)​ρE​\(θ;s,a\),\\mathcal\{L\}\_\{\\rm FPO\}\(\\theta;s,a\)=A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\),\(20\)we have

∇θℒFPO=A​\(s,a\)​ρE​\(θ;s,a\)​\[∇θlog⁡πθ​\(a\|s\)−∇θΔθ​\(s,a\)\]\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\rm FPO\}=A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\)\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)\-\\nabla\_\{\\theta\}\\Delta\_\{\\theta\}\(s,a\)\]\.\(21\)

###### Proof\.

Substituting Eq\. \([17](https://arxiv.org/html/2606.06967#A2.E17)\) into Eq\. \([18](https://arxiv.org/html/2606.06967#A2.E18)\) gives

ρFPO​\(θ;s,a\)\\displaystyle\\rho\_\{\\rm FPO\}\(\\theta;s,a\)=exp⁡\(log⁡πθ​\(a\|s\)−Δθ​\(s,a\)−log⁡πθold​\(a\|s\)\+Δθold​\(s,a\)\)\\displaystyle=\\exp\\\!\(\\log\\pi\_\{\\theta\}\(a\|s\)\-\\Delta\_\{\\theta\}\(s,a\)\-\\log\\pi\_\{\\theta\_\{\\rm old\}\}\(a\|s\)\+\\Delta\_\{\\theta\_\{\\rm old\}\}\(s,a\)\)=πθ​\(a\|s\)πθold​\(a\|s\)​exp⁡\(Δθold​\(s,a\)−Δθ​\(s,a\)\)\.\\displaystyle=\\frac\{\\pi\_\{\\theta\}\(a\|s\)\}\{\\pi\_\{\\theta\_\{\\rm old\}\}\(a\|s\)\}\\exp\\\!\(\\Delta\_\{\\theta\_\{\\rm old\}\}\(s,a\)\-\\Delta\_\{\\theta\}\(s,a\)\)\.\(22\)Sinceθold\\theta\_\{\\rm old\},A​\(s,a\)A\(s,a\), andℰθold​\(s,a\)\\mathcal\{E\}\_\{\\theta\_\{\\rm old\}\}\(s,a\)are fixed during the update,

∇θℒFPO\\displaystyle\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\rm FPO\}=A​\(s,a\)​ρE​\(θ;s,a\)​∇θℰθ​\(s,a\)\\displaystyle=A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\)\\nabla\_\{\\theta\}\\mathcal\{E\}\_\{\\theta\}\(s,a\)=A​\(s,a\)​ρE​\(θ;s,a\)​\[∇θlog⁡πθ​\(a\|s\)−∇θΔθ​\(s,a\)\],\\displaystyle=A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\)\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)\-\\nabla\_\{\\theta\}\\Delta\_\{\\theta\}\(s,a\)\],\(23\)∎

Eq\. \([21](https://arxiv.org/html/2606.06967#A2.E21)\) shows that the ELBO\-ratio gradient is not only a likelihood\-ratio gradient\. Its deviation from the likelihood direction is

−A​\(s,a\)​ρE​\(θ;s,a\)​∇θΔθ​\(s,a\)\.\-A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\)\\nabla\_\{\\theta\}\\Delta\_\{\\theta\}\(s,a\)\.\(24\)ForA​\(s,a\)<0A\(s,a\)<0, this term has a positive coefficient in the direction of∇θΔθ​\(s,a\)\\nabla\_\{\\theta\}\\Delta\_\{\\theta\}\(s,a\)\. In the degenerate case∇θlog⁡πθ​\(a\|s\)=0\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\|s\)=0,

∇θℒE=−A​\(s,a\)​ρE​\(θ;s,a\)​∇θΔθ​\(s,a\),\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\rm E\}=\-A\(s,a\)\\rho\_\{\\rm E\}\(\\theta;s,a\)\\nabla\_\{\\theta\}\\Delta\_\{\\theta\}\(s,a\),\(25\)so the update changes only the ELBO gap while leaving the executed\-action likelihood locally unchanged\. Thus, when the variational gap varies across policy updates, the ELBO ratio can differ from the true policy ratio and can induce clipped PPO\-style updates that are not aligned with the true executed\-action likelihood\.

## Appendix CProof of Proposition[4\.1](https://arxiv.org/html/2606.06967#S4.Thmtheorem1)

Recall the two\-step Adams–Bashforth update and rewrite as:

xi\+1AB2=xi\+Δt​vi⏟Euler step−12​Δt​\(vi−1−vi\)⏟derivative\-history correction\.x\_\{i\+1\}^\{\\mathrm\{AB2\}\}=\\underbrace\{x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\}\_\{\\text\{Euler step\}\}\-\\underbrace\{\\frac\{1\}\{2\}\\Delta\_\{t\}\(v\_\{i\-1\}\-v\_\{i\}\)\}\_\{\\text\{derivative\-history correction\}\}\.\(26\)
Letx​\(t\)x\(t\)be the smooth trajectory of the ODE

d​x​\(t\)d​t=vθ​\(x​\(t\),s,t\),\\frac\{dx\(t\)\}\{dt\}=v\_\{\\theta\}\(x\(t\),s,t\),\(27\)
and letxi=x​\(ti\)x\_\{i\}=x\(t\_\{i\}\),xi−1=x​\(ti−1\)x\_\{i\-1\}=x\(t\_\{i\-1\}\)\. This indexing convention matches the reverse solver direction:xi−1x\_\{i\-1\}is the history state one step beforexix\_\{i\}, whilexi\+1x\_\{i\+1\}is the next state to be predicted\.

Define the total derivative of the velocity field along the trajectory as

v˙i=dd​t​vθ​\(x​\(t\),t\)\|t=ti\.\\dot\{v\}\_\{i\}=\\frac\{d\}\{dt\}v\_\{\\theta\}\(x\(t\),t\)\|\_\{t=t\_\{i\}\}\.\(28\)By Taylor expansion ofvθ​\(x​\(t\),s,t\)v\_\{\\theta\}\(x\(t\),s,t\)aroundtit\_\{i\}, we have

vi−1=vi−Δt​v˙i\+O​\(Δt2\)\.v\_\{i\-1\}=v\_\{i\}\-\\Delta\_\{t\}\\,\\dot\{v\}\_\{i\}\+O\(\\Delta\_\{t\}^\{2\}\)\.\(29\)Hence the derivative\-history correction is:

−12​Δt​\(vi−1−vi\)=12​Δt​\(Δt​v˙i\+O​\(Δt2\)\)=12​Δt2​v˙i\+O​\(Δt3\)\.\-\\frac\{1\}\{2\}\\Delta\_\{t\}\(v\_\{i\-1\}\-v\_\{i\}\)=\\frac\{1\}\{2\}\\Delta\_\{t\}\(\\Delta\_\{t\}\\,\\dot\{v\}\_\{i\}\+O\(\\Delta\_\{t\}^\{2\}\)\)=\\frac\{1\}\{2\}\\Delta\_\{t\}^\{2\}\\,\\dot\{v\}\_\{i\}\+O\(\\Delta\_\{t\}^\{3\}\)\.\(30\)
On the other hand, Taylor expansion of the state trajectory gives

xi−1\\displaystyle x\_\{i\-1\}=xi−Δt​x˙​\(ti\)\+12​Δt2​x¨​\(ti\)\+O​\(Δt3\)\\displaystyle=x\_\{i\}\-\\Delta\_\{t\}\\,\\dot\{x\}\(t\_\{i\}\)\+\\frac\{1\}\{2\}\\Delta\_\{t\}^\{2\}\\,\\ddot\{x\}\(t\_\{i\}\)\+O\(\\Delta\_\{t\}^\{3\}\)\(31\)=xi−Δt​vi\+12​Δt2​v˙i\+O​\(Δt3\)\.\\displaystyle=x\_\{i\}\-\\Delta\_\{t\}\\,v\_\{i\}\+\\frac\{1\}\{2\}\\Delta\_\{t\}^\{2\}\\,\\dot\{v\}\_\{i\}\+O\(\\Delta\_\{t\}^\{3\}\)\.Therefore,

−12​Δt​\(vi−1−vi\)=xi−1−xi\+Δt​vi\+O​\(Δt3\)\.\-\\frac\{1\}\{2\}\\Delta\_\{t\}\(v\_\{i\-1\}\-v\_\{i\}\)=x\_\{i\-1\}\-x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+O\(\\Delta\_\{t\}^\{3\}\)\.\(32\)
GenPO\+\+ replaces the history correction with a relaxed state\-history residual, which gives:

xi\+1GenPO\+⁣\+\\displaystyle x\_\{i\+1\}^\{\\mathrm\{GenPO\+\+\}\}=xi\+Δt​vi\+σ​\(xi−1−xi\+Δt​vi\)\\displaystyle=x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+\\sigma\(x\_\{i\-1\}\-x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\)\(33\)=xi\+Δt​vi\+σ​xi−1−σ​xi\+σ​Δt​vi\\displaystyle=x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+\\sigma x\_\{i\-1\}\-\\sigma x\_\{i\}\+\\sigma\\Delta\_\{t\}\\,v\_\{i\}=\(1−σ\)​xi\+σ​xi−1\+\(1\+σ\)​Δt​vi\.\\displaystyle=\(1\-\\sigma\)x\_\{i\}\+\\sigma x\_\{i\-1\}\+\(1\+\\sigma\)\\Delta\_\{t\}\\,v\_\{i\}\.which is Eq\. \([6](https://arxiv.org/html/2606.06967#S4.E6)\)\.

Whenσ=1\\sigma=1, the GenPO\+\+ update becomes

xi−1GenPO\+⁣\+=xi\+Δt​vi\+xi\+1−xi\+Δt​vi,x\_\{i\-1\}^\{\\mathrm\{GenPO\+\+\}\}=x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\}\+x\_\{i\+1\}\-x\_\{i\}\+\\Delta\_\{t\}\\,v\_\{i\},\(34\)which matches the AB2 update up to the local approximation errorO​\(Δt3\)O\(\\Delta\_\{t\}^\{3\}\)\. For generalσ\\sigma, the coefficientσ\\sigmarelaxes the strength of the state\-history correction while preserving the explicit dependence on\(xi,xi\+1\)\(x\_\{i\},x\_\{i\+1\}\), which is the key property enabling the bidirectionally explicit GenPO\+\+ transition\.

![Refer to caption](https://arxiv.org/html/2606.06967v1/x8.png)Figure 8:We train flow\-matching model on toy data and compare Euler sampling with GenPO\+\+ sampling under differentσ\\sigma\.
## Appendix DProof of Theorem[4\.2](https://arxiv.org/html/2606.06967#S4.Thmtheorem2)

Define the second\-order derivative of the velocity field along the trajectory as:

v¨i=d2d​t2​vθ​\(x​\(t\),t\|s\)\|t=ti\.\\ddot\{v\}\_\{i\}=\\frac\{d^\{2\}\}\{dt^\{2\}\}v\_\{\\theta\}\(x\(t\),t\|s\)\\bigg\|\_\{t=t\_\{i\}\}\.\(35\)Taylor expansion gives

xi−1\\displaystyle x\_\{i\-1\}=xi−Δ​t​vi\+12​Δ​t2​v˙i−16​Δ​t3​v¨i\+O​\(Δ​t4\),\\displaystyle=x\_\{i\}\-\\Delta t\\,v\_\{i\}\+\\frac\{1\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\-\\frac\{1\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\),\(36\)xi\+1⋆\\displaystyle x\_\{i\+1\}^\{\\star\}=xi\+Δ​t​vi\+12​Δ​t2​v˙i\+16​Δ​t3​v¨i\+O​\(Δ​t4\)\.\\displaystyle=x\_\{i\}\+\\Delta t\\,v\_\{i\}\+\\frac\{1\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\+\\frac\{1\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\.
Substituting the exact history pair\(xi,xi−1\)\(x\_\{i\},x\_\{i\-1\}\)into Eq\. \([6](https://arxiv.org/html/2606.06967#S4.E6)\), the GenPO\+\+ update gives

x^i\+1\\displaystyle\\hat\{x\}\_\{i\+1\}=\(1−σ\)​xi\+σ​xi−1\+\(1\+σ\)​Δ​t​vi\\displaystyle=\(1\-\\sigma\)x\_\{i\}\+\\sigma x\_\{i\-1\}\+\(1\+\\sigma\)\\Delta t\\,v\_\{i\}\(37\)=\(1−σ\)​xi\+σ​\(xi−Δ​t​vi\+12​Δ​t2​v˙i−16​Δ​t3​v¨i\+O​\(Δ​t4\)\)\+\(1\+σ\)​Δ​t​vi\\displaystyle=\(1\-\\sigma\)x\_\{i\}\+\\sigma\\left\(x\_\{i\}\-\\Delta t\\,v\_\{i\}\+\\frac\{1\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\-\\frac\{1\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\\right\)\+\(1\+\\sigma\)\\Delta t\\,v\_\{i\}=xi\+Δ​t​vi\+σ2​Δ​t2​v˙i−σ6​Δ​t3​v¨i\+O​\(Δ​t4\)\.\\displaystyle=x\_\{i\}\+\\Delta t\\,v\_\{i\}\+\\frac\{\\sigma\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\-\\frac\{\\sigma\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\.Therefore,

x^i\+1−xi\+1⋆\\displaystyle\\hat\{x\}\_\{i\+1\}\-x\_\{i\+1\}^\{\\star\}=\(xi\+Δ​t​vi\+σ2​Δ​t2​v˙i−σ6​Δ​t3​v¨i\+O​\(Δ​t4\)\)\\displaystyle=\\left\(x\_\{i\}\+\\Delta t\\,v\_\{i\}\+\\frac\{\\sigma\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\-\\frac\{\\sigma\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\\right\)\(38\)−\(xi\+Δ​t​vi\+12​Δ​t2​v˙i\+16​Δ​t3​v¨i\+O​\(Δ​t4\)\)\\displaystyle\\quad\-\\left\(x\_\{i\}\+\\Delta t\\,v\_\{i\}\+\\frac\{1\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\+\\frac\{1\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\\right\)=σ−12​Δ​t2​v˙i−σ\+16​Δ​t3​v¨i\+O​\(Δ​t4\)\.\\displaystyle=\\frac\{\\sigma\-1\}\{2\}\\Delta t^\{2\}\\dot\{v\}\_\{i\}\-\\frac\{\\sigma\+1\}\{6\}\\Delta t^\{3\}\\ddot\{v\}\_\{i\}\+O\(\\Delta t^\{4\}\)\.Thus,

x^i\+1−xi\+1⋆=σ−12​Δ​t2​dd​t​vθ​\(x​\(t\),t\|s\)\|t=ti\+O​\(Δ​t3\)\.\\hat\{x\}\_\{i\+1\}\-x\_\{i\+1\}^\{\\star\}=\\frac\{\\sigma\-1\}\{2\}\\Delta t^\{2\}\\frac\{d\}\{dt\}v\_\{\\theta\}\(x\(t\),t\|s\)\\bigg\|\_\{t=t\_\{i\}\}\+O\(\\Delta t^\{3\}\)\.\(39\)Thisσ\\sigma\-dependent local error explains the role of the history coefficient\. Figure[8](https://arxiv.org/html/2606.06967#A3.F8)visualizes this effect on toy\-data sampling with the same trained flow\-matching model\.

## Appendix EConservative Policy Improvement with Augmented\-Action Ratios

###### Theorem E\.1\(Consistency of GenPO\+\+ augmented\-action ratios\)\.

The executed\-action likelihood ratio satisfies

rθ​\(x1\|s\)=𝔼x1−Δt∼π~old\(⋅\|x1,s\)​\[r~θ​\(x1,x1−Δt\|s\)\]\.r\_\{\\theta\}\(x\_\{1\}\|s\)=\\mathbb\{E\}\_\{x\_\{1\-\\Delta\_\{t\}\}\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot\|x\_\{1\},s\)\}\[\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\]\.\(40\)
Moreover, for advantage functionAπold​\(s,x1\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\),

𝔼s∼dρπold,x1∼πold\(⋅\|s\)​\[rθ​\(x1\|s\)​Aπold​\(s,x1\)\]=𝔼s∼dρπold,\(z1\)∼π~old\(⋅,⋅\|s\)​\[r~θ​\(z1\|s\)​Aπold​\(s,x1\)\]\.\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},x\_\{1\}\\sim\\pi\_\{\\rm old\}\(\\cdot\|s\)\}\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]=\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},\(z\_\{1\}\)\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(z\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\.\(41\)

###### Proof\.

ziz\_\{i\}is defined as\(xi,xi−Δt\)\(x\_\{i\},x\_\{i\-\\Delta\_\{t\}\}\)\. We first prove Eq\.\([40](https://arxiv.org/html/2606.06967#A5.E40)\)\. Fix a statessand an executed actionx1x\_\{1\}\. By the definition of conditional density,

π~old​\(x1−Δt\|x1,s\)=π~old​\(x1,x1−Δt\|s\)πold​\(x1\|s\)\.\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\-\\Delta\_\{t\}\}\|x\_\{1\},s\)=\\frac\{\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\{\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\}\.\(42\)Therefore,

𝔼x1−Δt∼π~old\(⋅\|x1,s\)​\[r~θ​\(x1,x1−Δt\|s\)\]\\displaystyle\\mathbb\{E\}\_\{x\_\{1\-\\Delta\_\{t\}\}\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot\|x\_\{1\},s\)\}\[\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\]\(43\)=∫r~θ​\(x1,x1−Δt\|s\)​π~old​\(x1−Δt\|x1,s\)​𝑑x1−Δt\\displaystyle=\\int\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\-\\Delta\_\{t\}\}\|x\_\{1\},s\)\\,dx\_\{1\-\\Delta\_\{t\}\}=∫r~θ​\(x1,x1−Δt\|s\)​π~old​\(x1,x1−Δt\|s\)πold​\(x1\|s\)​𝑑x1−Δt\\displaystyle=\\int\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\frac\{\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\{\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\}\\,dx\_\{1\-\\Delta\_\{t\}\}=∫π~θ​\(x1,x1−Δt\|s\)π~old​\(x1,x1−Δt\|s\)​π~old​\(x1,x1−Δt\|s\)πold​\(x1\|s\)​𝑑x1−Δt\\displaystyle=\\int\\frac\{\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\{\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\\frac\{\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\}\{\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\}\\,dx\_\{1\-\\Delta\_\{t\}\}=1πold​\(x1\|s\)​∫π~θ​\(x1,x1−Δt\|s\)​𝑑x1−Δt\\displaystyle=\\frac\{1\}\{\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\}\\int\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\,dx\_\{1\-\\Delta\_\{t\}\}=πθ​\(x1\|s\)πold​\(x1\|s\)=rθ​\(x1\|s\)\.\\displaystyle=\\frac\{\\pi\_\{\\theta\}\(x\_\{1\}\|s\)\}\{\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\}=r\_\{\\theta\}\(x\_\{1\}\|s\)\.
We now prove Eq\.\([41](https://arxiv.org/html/2606.06967#A5.E41)\)\. Fix a statess\. Starting from the standard executed\-action surrogate, we have

𝔼x1∼πold\(⋅\|s\)​\[rθ​\(x1\|s\)​Aπold​\(s,x1\)\]\\displaystyle\\mathbb\{E\}\_\{x\_\{1\}\\sim\\pi\_\{\\rm old\}\(\\cdot\|s\)\}\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\(44\)=∫rθ​\(x1\|s\)​Aπold​\(s,x1\)​πold​\(x1\|s\)​𝑑x1\\displaystyle=\\int r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\\,dx\_\{1\}=∫∫r~θ​\(x1,x1−Δt\|s\)​π~old​\(x1−Δt\|x1,s\)​𝑑x1−Δt​Aπold​\(s,x1\)​πold​\(x1\|s\)​𝑑x1\.\\displaystyle=\\int\\int\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\-\\Delta\_\{t\}\}\|x\_\{1\},s\)\\,dx\_\{1\-\\Delta\_\{t\}\}A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\\,dx\_\{1\}\.
With the bayes’s rules,π~old​\(x1,x1−Δt\|s\)=π~old​\(x1−Δt\|x1,s\)​πold​\(x1\|s\)\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)=\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\-\\Delta\_\{t\}\}\|x\_\{1\},s\)\\pi\_\{\\rm old\}\(x\_\{1\}\|s\), we obtain

∫∫r~θ​\(x1,x1−Δt\|s\)​Aπold​\(s,x1\)​π~old​\(x1−Δt\|x1,s\)​πold​\(x1\|s\)​𝑑x1−Δt​𝑑x1\\displaystyle\\int\\int\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\-\\Delta\_\{t\}\}\|x\_\{1\},s\)\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\\,dx\_\{1\-\\Delta\_\{t\}\}\\,dx\_\{1\}\(45\)=∫∫r~θ​\(x1,x1−Δt\|s\)​Aπold​\(s,x1\)​π~old​\(x1,x1−Δt\|s\)​𝑑x1−Δt​𝑑x1\\displaystyle=\\int\\int\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\\,dx\_\{1\-\\Delta\_\{t\}\}\\,dx\_\{1\}=𝔼\(x1,x1−Δt\)∼π~old\(⋅,⋅\|s\)​\[r~θ​\(x1,x1−Δt\|s\)​Aπold​\(s,x1\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\.Therefore, for every fixed statess,

𝔼x1∼πold\(⋅\|s\)​\[rθ​\(x1\|s\)​Aπold​\(s,x1\)\]=𝔼\(x1,x1−Δt\)∼π~old\(⋅,⋅\|s\)​\[r~θ​\(x1,x1−Δt\|s\)​Aπold​\(s,x1\)\]\.\\mathbb\{E\}\_\{x\_\{1\}\\sim\\pi\_\{\\rm old\}\(\\cdot\|s\)\}\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]=\\mathbb\{E\}\_\{\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\)\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\.\(46\)Finally, integrating both sides overs∼dρπolds\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\}gives

𝔼s∼dρπold,x1∼πold\(⋅\|s\)​\[rθ​\(x1\|s\)​Aπold​\(s,x1\)\]=𝔼s∼dρπold,\(z1\)∼π~old\(⋅,⋅\|s\)​\[r~θ​\(z1\|s\)​Aπold​\(s,x1\)\]\.\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},x\_\{1\}\\sim\\pi\_\{\\rm old\}\(\\cdot\|s\)\}\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]=\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},\(z\_\{1\}\)\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(z\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\.\(47\)This proves Eq\.\([41](https://arxiv.org/html/2606.06967#A5.E41)\)\. ∎

###### Lemma E\.2\(Marginal TV is bounded by augmented TV\)\.

For every statess,

TV\(πθ\(⋅\|s\),πold\(⋅\|s\)\)≤TV\(π~θ\(⋅,⋅\|s\),π~old\(⋅,⋅\|s\)\)\.\{\\rm TV\}\\\!\(\\pi\_\{\\theta\}\(\\cdot\|s\),\\pi\_\{\\rm old\}\(\\cdot\|s\)\)\\leq\{\\rm TV\}\\\!\(\\tilde\{\\pi\}\_\{\\theta\}\(\\cdot,\\cdot\|s\),\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\)\.\(48\)

###### Proof\.

By definition of total variation distance,

TV\(πθ\(⋅\|s\),πold\(⋅\|s\)\)=12∫\|πθ\(x1\|s\)−πold\(x1\|s\)\|dx1\.\{\\rm TV\}\\\!\(\\pi\_\{\\theta\}\(\\cdot\|s\),\\pi\_\{\\rm old\}\(\\cdot\|s\)\)=\\frac\{1\}\{2\}\\int\|\\pi\_\{\\theta\}\(x\_\{1\}\|s\)\-\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)\|dx\_\{1\}\.\(49\)Using the marginal definitions,

πθ​\(x1\|s\)−πold​\(x1\|s\)=∫\[π~θ​\(x1,x1−Δt\|s\)−π~old​\(x1,x1−Δt\|s\)\]​𝑑x1−Δt\.\\pi\_\{\\theta\}\(x\_\{1\}\|s\)\-\\pi\_\{\\rm old\}\(x\_\{1\}\|s\)=\\int\[\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\-\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\]dx\_\{1\-\\Delta\_\{t\}\}\.\(50\)Therefore,

TV\(πθ\(⋅\|s\),πold\(⋅\|s\)\)=12∫\|∫\[π~θ\(x1,x1−Δt\|s\)−π~old\(x1,x1−Δt\|s\)\]dx1−Δt\|dx1\.\\displaystyle\{\\rm TV\}\\\!\(\\pi\_\{\\theta\}\(\\cdot\|s\),\\pi\_\{\\rm old\}\(\\cdot\|s\)\)=\\frac\{1\}\{2\}\\int\|\\int\[\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\-\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\]dx\_\{1\-\\Delta\_\{t\}\}\|dx\_\{1\}\.\(51\)By the triangle inequality,

\|∫\[π~θ\(x1,x1−Δt\|s\)−π~old\(x1,x1−Δt\|s\)\]dx1−Δt\|≤∫\|π~θ\(x1,x1−Δt\|s\)−π~old\(x1,x1−Δt\|s\)\|dx1−Δt\.\|\\int\[\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\-\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\]dx\_\{1\-\\Delta\_\{t\}\}\|\\leq\\int\|\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\-\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\|dx\_\{1\-\\Delta\_\{t\}\}\.\(52\)Thus,

TV\(πθ\(⋅\|s\),πold\(⋅\|s\)\)\\displaystyle\{\\rm TV\}\\\!\(\\pi\_\{\\theta\}\(\\cdot\|s\),\\pi\_\{\\rm old\}\(\\cdot\|s\)\)≤12∫∫\|π~θ\(x1,x1−Δt\|s\)−π~old\(x1,x1−Δt\|s\)\|dx1−Δtdx1\\displaystyle\\leq\\frac\{1\}\{2\}\\int\\int\|\\tilde\{\\pi\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\-\\tilde\{\\pi\}\_\{\\rm old\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)\|dx\_\{1\-\\Delta\_\{t\}\}dx\_\{1\}\(53\)=TV\(π~θ\(⋅,⋅\|s\),π~old\(⋅,⋅\|s\)\)\.\\displaystyle=\{\\rm TV\}\\\!\(\\tilde\{\\pi\}\_\{\\theta\}\(\\cdot,\\cdot\|s\),\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\)\.∎

According to the Eq\.\([41](https://arxiv.org/html/2606.06967#A5.E41)\) and Eq\.\([48](https://arxiv.org/html/2606.06967#A5.E48)\), there is:

𝒥​\(πθ\)−𝒥​\(πold\)\\displaystyle\\mathcal\{J\}\(\\pi\_\{\\theta\}\)\-\\mathcal\{J\}\(\\pi\_\{\\rm old\}\)≥𝔼s∼dρπold,x1∼πold\(⋅\|s\)\[rθ\(x1\|s\)Aπold\(s,x1\)\]−CTVTV\(π~θ\(⋅\|s\),π~old\(⋅\|s\)\)\\displaystyle\\geq\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},x\_\{1\}\\sim\\pi\_\{\\rm old\}\(\\cdot\|s\)\}\[r\_\{\\theta\}\(x\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\-C\_\{\\rm TV\}\{\\rm TV\}\\\!\(\\tilde\{\\pi\}\_\{\\theta\}\(\\cdot\|s\),\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot\|s\)\)\(54\)=𝔼s∼dρπold,z1∼π~old\(⋅,⋅\|s\)\[r~θ\(z1\|s\)Aπold\(s,x1\)\]−CTVTV\(π~θ\(⋅\|s\),π~old\(⋅\|s\)\)\\displaystyle=\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},z\_\{1\}\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(z\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\-C\_\{\\rm TV\}\{\\rm TV\}\\\!\(\\tilde\{\\pi\}\_\{\\theta\}\(\\cdot\|s\),\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot\|s\)\)≥𝔼s∼dρπold,z1∼π~old\(⋅,⋅\|s\)\[r~θ\(z1\|s\)Aπold\(s,x1\)\]−CTVTV\(π~θ\(⋅,⋅\|s\),π~old\(⋅,⋅\|s\)\),\\displaystyle\\geq\\mathbb\{E\}\_\{s\\sim d\_\{\\rho\}^\{\\pi\_\{\\rm old\}\},z\_\{1\}\\sim\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\}\[\\tilde\{r\}\_\{\\theta\}\(z\_\{1\}\|s\)A^\{\\pi\_\{\\rm old\}\}\(s,x\_\{1\}\)\]\-C\_\{\\rm TV\}\{\\rm TV\}\\\!\(\\tilde\{\\pi\}\_\{\\theta\}\(\\cdot,\\cdot\|s\),\\tilde\{\\pi\}\_\{\\rm old\}\(\\cdot,\\cdot\|s\)\),Therefore, applying PPO clipping tor~θ​\(x1,x1−Δt\|s\)\\tilde\{r\}\_\{\\theta\}\(x\_\{1\},x\_\{1\-\\Delta\_\{t\}\}\|s\)gives a conservative trust\-region surrogate for the original MDP, even though the likelihood ratio is computed in augmented GenPO\+\+ coordinates\.

## Appendix FExperiment Details

### F\.1Hardware

Experiments were conducted on a dual\-socket server featuring Intel Xeon Gold 6430 CPUs \(128 total threads, 2\.1–3\.4 GHz\) and 8 NVIDIA RTX 4090 D GPUs \(24GB GDDR6X each\)\. The system utilized CUDA 12\.8 with driver version 570\.124, organized across two NUMA nodes\. No Multi\-Instance GPU \(MIG\) or ECC features were enabled\.

### F\.2IsaacLab

##### Reinforcement Learning Framework in IsaacLab\.

All reinforcement learning algorithms are implemented within IsaacLab using the RSL\-RL training framework\. IsaacLab provides official wrappers that convert environment observations, actions, rewards, and termination signals into the interface required by different RL libraries, including RSL\-RL\. In our implementation, we build all policy optimization methods on top of the RSL\-RL on\-policy runner and keep the same environment interaction pipeline, rollout storage, mini\-batch construction, and PPO\-style optimization interface across methods\. This unified implementation ensures that the reported differences come from the policy parameterization and likelihood\-ratio computation rather than from differences in the simulator or training infrastructure\. The RSL\-RL library is available at[https://github\.com/leggedrobotics/rsl\_rl](https://github.com/leggedrobotics/rsl_rl)\. The FPO is available at[https://github\.com/amazon\-far/fpo\-control](https://github.com/amazon-far/fpo-control)\. The PolicyFlow is available at[https://github\.com/PolicyFlow2026/PolicyFlow](https://github.com/PolicyFlow2026/PolicyFlow)\. The GenPO is available at[https://github\.com/wadx2019/genpo/](https://github.com/wadx2019/genpo/)\.

![Refer to caption](https://arxiv.org/html/2606.06967v1/x9.png)Figure 9:Learning curves for different flow policy time steps on the Isaaclab\-Vecocity\-Rough\-G1\-v0 benchmark\. Results are averaged over 5 runs\. The x\-axis denotes training epochs, and the y\-axis shows average episodic return with one standard deviation shaded\.![Refer to caption](https://arxiv.org/html/2606.06967v1/x10.png)Figure 10:Eight Isaaclab benchmark visualizations, eight images from[https://isaac\-sim\.github\.io/IsaacLab/main/source/overview/environments\.html](https://isaac-sim.github.io/IsaacLab/main/source/overview/environments.html)\. From \(a\) to \(h\) are Isaac\-Ant\-v0, Isaac\-Humanoid\-v0, Isaac\-Open\-Drawer\-Franka\-v0, Isaac\-Velocity\-Rough\-Anymal\-D\-v0, Isaac\-Velocity\-Rough\-Unitree\-Go2\-v0, Isaac\-Velocity\-Rough\-G1\-v0, Isaac\-Velocity\-Rough\-H1\-v0, and Isaac\-Tracking\-LocoManip\-Digit\-v0\.![Refer to caption](https://arxiv.org/html/2606.06967v1/x11.png)Figure 11:Learning curves across 8 IsaacLab benchmarks\. Results are averaged over 5 runs\. The x\-axis denotes wall\-clock\-time, and the y\-axis shows average episodic return with one standard deviation shaded\.
##### Benchmark suite\.

We evaluate all methods on eight IsaacLab tasks covering three representative categories: classical locomotion, articulated manipulation, and whole\-body locomotion\-control\. The benchmark containsIsaac\-Ant\-v0,Isaac\-Humanoid\-v0,Isaac\-Open\-Drawer\-Franka\-v0,Isaac\-Velocity\-Rough\-Anymal\-D\-v0,Isaac\-Velocity\-Rough\-Unitree\-Go2\-v0,Isaac\-Velocity\-Rough\-Unitree\-G1\-v0,Isaac\-Velocity\-Rough\-H1\-v0, andIsaac\-Tracking\-LocoManip\-Digit\-v0\. These tasks differ substantially in action dimension, contact complexity, reward scale, and episode length\. We keep the official IsaacLab environment configuration, reward terms, termination conditions, and simulation settings unchanged for all methods\. Therefore, each method is evaluated under the same control problem and the same simulator dynamics\.

##### Baselines and fairness\.

We compare GenPO\+\+ with Gaussian PPO and three representative generative policy optimization methods: FPO, GenPO, and PolicyFlow\. For all methods, the critic architecture, rollout horizon, advantage estimation, value loss, PPO clipping range, and mini\-batch construction follow the same RSL\-RL pipeline unless otherwise specified\. For FPO, we follow the parameter design in the code, the number of epochs is 32, and an EMA mechanism is introduced111[https://github\.com/amazon\-far/fpo\-control](https://github.com/amazon-far/fpo-control)\.

##### Wall\-clock comparison\.

Figure[11](https://arxiv.org/html/2606.06967#A6.F11)reports the IsaacLab learning curves using wall\-clock time as the x\-axis\. This complements the iteration\-based curves in the main paper\. GenPO\+\+ remains competitive or superior under wall\-clock comparison, indicating that the performance gain is not caused by using more expensive updates\. In contrast, GenPO incurs a larger learning overhead due to exact likelihood evaluation through the augmented inverse process, while FPO can be slower because of its heavier optimization schedule\. GenPO\+\+ avoids neural\-Jacobian computation and therefore substantially reduces policy\-update time while retaining exact augmented likelihood\-ratio optimization\.

##### Number of flow steps\.

Figure[9](https://arxiv.org/html/2606.06967#A6.F9)studies the number of flow\-policy solver steps onIsaac\-Velocity\-Rough\-G1\-v0\. Although more steps increase the expressiveness of the transport map, they also lengthen the differentiable inversion path used for likelihood\-ratio computation\. During PPO updates, gradients are back\-propagated through the unrolled reversible solver, making the optimization similar to BPTT\. A larger number of steps therefore increases memory and computation cost, and may cause poorly conditioned gradients due to repeated step\-wise Jacobian products\. Empirically, moderate step numbers provide the best trade\-off, while using more steps does not yield monotonic improvement\.

Table 2:Task\-specific hyperparameters for GenPO\+\+\.

### F\.3Fine\-tuning: Robomimic

![Refer to caption](https://arxiv.org/html/2606.06967v1/x12.png)Figure 12:Evaluation episodes on the three manipulation fine\-tuning tasks\.We evaluate online fine\-tuning of pretrained flow\-matching manipulation policies on three visual manipulation tasks:Can,Box Cleanup, andThreading\. TheCantask is from RoboMimic, whileBox CleanupandThreadingfollow the DexMimicGen manipulation benchmark used in prior flow\-policy fine\-tuning experiments\. These tasks cover different manipulation regimes, including single\-object grasping, object rearrangement, and long\-horizon precision insertion\.

Our Robomimic fine\-tuning experiments are implemented based on the FPO codebase and use the same pretrained base flow policies released by FPO:[https://github\.com/amazon\-far/fpo\-control/tree/main/manipulation\_experiments](https://github.com/amazon-far/fpo-control/tree/main/manipulation_experiments)\. We evaluate three manipulation tasks:Can,Box Cleanup, andThreading\. These tasks cover different robot embodiments and manipulation regimes:Canuses a single robot arm with a parallel\-jaw gripper,Threadinguses two dexterous hands, andBox Cleanupuses two robot arms with parallel\-jaw grippers\.

Following FPO, the policy outputs an action chunk with horizon length 16\. During environment interaction, all methods execute the full 16\-step action chunk before querying the policy again\. Thus, the compared methods use the same temporal abstraction, environment rollout protocol, and pretrained initialization\.

### F\.4Simulation Modeling

The task uses the RobotEra Xhand with 12 actuated joints\. Following DexScrew, we use a simplified screw\-and\-nut model with a fixed base and the nut is attached through a single revolute joint around the screw axis\. Simulation is performed in Isaac Gym\. The physics step is 0\.005 s and the policy decimation is 10, giving a 20 Hz high\-level control rate\. Each episode lasts at most 800 steps\. The policy action is a normalized1212\-dimensional vector clipped to\[−1,1\]\[\-1,1\]\. At each control step, the action is converted to an incremental joint target,

qt\+1tar=clip​\(qttar\+α⋅at,qmin,qmax\),q^\{\\mathrm\{tar\}\}\_\{t\+1\}=\\mathrm\{clip\}\\\!\(q^\{\\mathrm\{tar\}\}\_\{t\}\+\\alpha\\cdot\\,a\_\{t\},\\ q\_\{\\min\},q\_\{\\max\}\),\(55\)whereα\\alphais the action scale, andqminq\_\{\\min\}andqmaxq\_\{\\max\}are the XHand joint limits\.

Table[3](https://arxiv.org/html/2606.06967#A6.T3)summarizes the domain randomization settings\.

Table 3:Domain Randomization Parameters\.#### F\.4\.1Oracle Policy

The oracle policy is trained in simulation with proprioceptive observations, object geometry, and privileged information that is unavailable on the real robot\. The proprioceptive input contains the recent joint\-position and joint\-target history, and the privileged state follows the nut\-bolt oracle specification in Table[4](https://arxiv.org/html/2606.06967#A6.T4)\.

Table 4:Privileged information used by the nut\-bolt oracle policy\.The oracle rewards and weights are listed in Table[5](https://arxiv.org/html/2606.06967#A6.T5)\. The oracle\-stage training schedule is reported separately in Table[6](https://arxiv.org/html/2606.06967#A6.T6)\.

Table 5:Nut\-bolt reward weights\.Table 6:Hyperparameters for training the oracle GenPO\+\+ policy\.
#### F\.4\.2Sensorimotor Student Policy

After the oracle policy is trained, we distill it into a deployable sensorimotor policy that does not require privileged simulation state\. The student uses only proprioception: a current observation together with a historyht∈ℝ30×24h\_\{t\}\\in\\mathbb\{R\}^\{30\\times 24\}containing3030frames of joint positions and joint targets\. Following DexScrew, the student is trained with DAgger: at each step, the sensorimotor policy acts in the environment, while the oracle provides target hand actions and ground\-truth privileged embeddings\. The training hyperparameters are given in Table[7](https://arxiv.org/html/2606.06967#A6.T7)\.

Table 7:Hyperparameters for training the sensorimotor student policy\.Letztz\_\{t\}denote the privileged embedding used by the oracle and letz^t=ϕ​\(ht\)\\hat\{z\}\_\{t\}=\\phi\(h\_\{t\}\)be the embedding predicted from proprioceptive history\. The student loss follows the sensorimotor policy training objective:

ℒstudent=‖atHand−a^tHand‖22\+‖zt−z^t‖22,\\mathcal\{L\}\_\{\\mathrm\{student\}\}=\\left\\\|a\_\{t\}^\{\\mathrm\{Hand\}\}\-\\hat\{a\}\_\{t\}^\{\\mathrm\{Hand\}\}\\right\\\|\_\{2\}^\{2\}\+\\left\\\|z\_\{t\}\-\\hat\{z\}\_\{t\}\\right\\\|\_\{2\}^\{2\},\(56\)whereatHanda\_\{t\}^\{\\mathrm\{Hand\}\}is the oracle target hand action anda^tHand\\hat\{a\}\_\{t\}^\{\\mathrm\{Hand\}\}is the action produced by the sensorimotor policy\. The embedding predictorϕ\\phiis optimized with Adam until convergence\.

#### F\.4\.3Deployment

For real\-world deployment, the trained student is exported to TorchScript together with the running mean and variance of the proprioceptive observation and proprioceptive\-history normalizers\. The XHand controller sends commands at200200Hz, while policy inference is performed every1010low\-level command cycles, matching the2020Hz simulation control rate\.

The policy output is clipped to\[−1,1\]\[\-1,1\], the inactive pinky channels are zeroed for nut\-bolt manipulation, and the incremental target rule with the same action scale is applied\. To match the200200Hz hardware interface and avoid discontinuous jumps in joint targets, the high\-level target is not sent as a step command\. Instead, we interpolate between the previously commanded target and the new policy target over the next1010hardware cycles

![Refer to caption](https://arxiv.org/html/2606.06967v1/x13.png)Figure 13:nut\-bolts with different geometries\.Figure[13](https://arxiv.org/html/2606.06967#A6.F13)shows nut\-bolts with different geometries, which we use in real world deployment\. No privileged state, contact signal, nut pose, or simulator parameter is provided, and the robustness across different nut\-bolts is provided by the student adaptation module, domain randomization, and mostly the expressiveness of GenPO\+\+\.