Path-Coupled Bellman Flows for Distributional Reinforcement Learning

# Path-Coupled Bellman Flows for Distributional Reinforcement Learning
Source: [https://arxiv.org/html/2605.08253](https://arxiv.org/html/2605.08253)
###### Abstract

Distributional reinforcement learning \(DRL\) models the full return distribution, but existing finite\-support or quantile\-based methods rely on projections, while recent flow\-based approaches can suffer from*boundary mismatch*at the flow source or from*high\-variance*bootstrapping when current and successor noises are independent\. We propose Path\-Coupled Bellman Flows \(PCBF\), a continuous\-time DRL method that learns return distributions with flow matching usingsource\-consistent Bellman\-coupled paths: the current path starts from the required base prior att=0t\{=\}0, reaches the Bellman target att=1t\{=\}1, and maintains a pathwise affine relation to the successor flow at intermediate times \(without requiring time\-ttmarginals to satisfy a distributional Bellman fixed point for alltt\)\. PCBF couples current and successor return flows through shared base noise and uses aλ\\lambda\-parameterized control\-variate target:λ=0\\lambda\{=\}0recovers an unbiased sample Bellman target, whileλ\>0\\lambda\{\>\}0trades controlled bias for variance reduction\. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance\.

distributional reinforcement learning, Bellman operator, flow matching, generative modeling, offline reinforcement learning, variance reduction

## 1Introduction

Distributional reinforcement learning \(DRL\)\(Bellemareet al\.,[2017](https://arxiv.org/html/2605.08253#bib.bib2)\)models the full distribution of returns rather than only their expectation, enabling richer representations of uncertainty and often leading to improved empirical performance\. Practical DRL algorithms, however, typically rely on finite\-dimensional approximations over fixed supports\(Bellemareet al\.,[2017](https://arxiv.org/html/2605.08253#bib.bib2)\)or quantile assignments\(Dabneyet al\.,[2018b](https://arxiv.org/html/2605.08253#bib.bib3)\)\. However, these discretization strategies introduce inherent limitations\. Because the Bellman update rarely aligns with fixed support points or quantile locations, these methods rely on heuristic projection steps that introduce bias and limit the expressivity of the learned distribution\.

To overcome the limitations of discrete projections, it is natural to reframe DRL as a problem of continuous probability transport\. Fundamentally, the distributional Bellman equation defines an affine transport relationship: the return distribution at the current state is a direct transformation of the successor state’s distribution\. This perspective makes flow matching\(Lipmanet al\.,[2022](https://arxiv.org/html/2605.08253#bib.bib13)\)a highly attractive framework\. By learning a continuous neural velocity field that transports samples from a simple base prior \(e\.g\., Gaussian noise\) to a complex target distribution, flow matching sidesteps heuristic projections and gracefully models the geometry of the Bellman update\.

Yet, directly enforcing an uncorrected pointwise Bellman map inside flow composition fails in two critical ways\. First, it violates the initial boundary condition: flow matching requires the generation process to start from a fixed simple prior, such as a standard Gaussian\. However, an uncorrected Bellman update shifts this starting distribution through the reward and discounted successor value, making the standard flow\-matching objective ill\-posed\. Second, when the noise driving the current and successor distributions is sampled independently, their intermediate trajectories are not pathwise aligned\. As a result, Bellman consistency can only be enforced at the endpoint, leading to high\-variance per\-sample targets that destabilize critic learning\.

We proposePath\-Coupled Bellman Flows \(PCBF\), which addresses both issues through a source\-consistent Bellman path correction and shared\-noise path coupling\. Instead of applying the Bellman update directly to intermediate flow states, PCBF repairs the flow path so that it starts from the required base prior while still ending at the Bellman target\. This separates the geometric requirement of flow matching from the stochasticity of Bellman bootstrapping\. On top of this corrected geometry, PCBF couples current and successor return flows using shared base noise, aligning their intermediate trajectories rather than enforcing Bellman consistency only at the endpoint\. This pathwise structure induces aλ\\lambda\-parameterized family of training targets that interpolates between direct sample\-based Bellman supervision and variance\-reduced supervision using successor\-flow velocity predictions\. Thus, PCBF cleanly separates Bellman path correction from variance control\.

Theoretically, we characterize the population\-optimal velocity field induced by the PCBF path, analyze the bias–variance behavior of theλ\\lambda\-target, and show that shared\-noise Bellman generator updates inheritγ\\gamma\-contraction and induce at​γt\\gammacontraction for PCBF interpolants\. Empirically, PCBF accurately recovers ground\-truth return distributions on toy MRPs and achieves competitive performance on OGBench\(Parket al\.,[2025a](https://arxiv.org/html/2605.08253#bib.bib10)\)and D4RL Adroit\(Fuet al\.,[2020](https://arxiv.org/html/2605.08253#bib.bib11)\)\.

#### Contributions\.

Our main contributions are:\(i\)a source\-consistent Bellman\-interpolated path that fixes thet=0t\{=\}0boundary mismatch arising from uncorrected pointwise Bellman paths;\(ii\)shared\-noise path coupling that aligns current and successor return flows;\(iii\)aλ\\lambda\-parameterized control\-variate target with a generalL2L\_\{2\}bias bound \(and a Gaussian closed form\);\(iv\)empirical evidence on toy return laws, OGBench, D4RL, FloQ comparisons where available, coupling/discretization diagnostics, and ablations\.

## 2Related Work

#### Generative Models in Offline RL\.

Generative models have become prominent in offline RL for trajectory planning, behavior modeling, and policy extraction, including diffusion\-based planners and policies\(Janneret al\.,[2022](https://arxiv.org/html/2605.08253#bib.bib35); Wanget al\.,[2022](https://arxiv.org/html/2605.08253#bib.bib23); Hansen\-Estruchet al\.,[2023](https://arxiv.org/html/2605.08253#bib.bib24)\)and flow\- or energy\-guided policy methods\(Chenet al\.,[2023](https://arxiv.org/html/2605.08253#bib.bib25); Luet al\.,[2023](https://arxiv.org/html/2605.08253#bib.bib36); Zhanget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib37)\)\. These works primarily improve the actor or planner\. PCBF instead focuses on the critic side: learning expressive and stable return distributions that can guide offline policy extraction\.

#### Distributional Reinforcement Learning\.

To capture the full complexity of cumulative returns, the DRL paradigm shifts the focus from estimating expected scalar values to modeling the entire return distribution\(Bellemareet al\.,[2017](https://arxiv.org/html/2605.08253#bib.bib2)\)\. Classical approaches initially relied on categorical projections\(Bellemareet al\.,[2017](https://arxiv.org/html/2605.08253#bib.bib2)\)and quantile regression\(Dabneyet al\.,[2018b](https://arxiv.org/html/2605.08253#bib.bib3)\)for discrete action spaces\. This framework was later successfully scaled to continuous control domains by D4PG\(Barth\-Maronet al\.,[2018](https://arxiv.org/html/2605.08253#bib.bib28)\), which integrated categorical distributional critics with deterministic policy gradients, and was further advanced by continuous variants like DSAC\(Duanet al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib26)\)\. Despite their success, these traditional methods are inherently constrained by discrete projections, fixed support boundaries, or moment\-matching approximations\(Nguyen\-Tanget al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib27)\), fundamentally limiting continuous critic expressivity\. To overcome these limits and match the continuous nature of modern generative actors, recent efforts have attempted to unify DRL with continuous flow matching\.

#### Generative Value Modeling and Flow Matching\.

Beyond policy and trajectory generation, generative models are increasingly being used for other RL components\. For instance, EVOR\(Espinosa\-Diceet al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib30)\)performs inference\-time policy extraction using a distributional reward model learned via standard flow matching\. Approaches such asflo​Q\\text\{flo\}Q\(Agrawallaet al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib31)\)use flow matching to parameterize value functions, mapping noise to scalar value estimates to enable iterative compute scaling\. TD\-Flow\(Farebrotheret al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib34)\)is also closely related, as it combines temporal\-difference bootstrapping with flow matching over probability paths\. However, when applied to rewards, TD\-Flow naturally corresponds to a return distribution for a transformed random\-horizon MDP, where the process terminates at each step with probability1−γ1\-\\gamma\. This differs from the conventional discounted return distribution usually studied in DRL: as discussed byBellemareet al\.\([2023](https://arxiv.org/html/2605.08253#bib.bib20)\), the two interpretations agree in expectation but generally differ as distributions\. In contrast, PCBF is designed to model the standard discounted return distribution while preserving its Bellman geometry within flow matching\. Concurrent methods such as DFC\(Chenet al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib33)\)and Bellman Diffusion\(Liet al\.,[2024](https://arxiv.org/html/2605.08253#bib.bib32)\)attempt to align Bellman target distributions but rely on independently sampled noise\. Such endpoint\-level matching lacks pathwise coupling, which can lead to misaligned vector fields and high\-variance training targets\. Value Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\)also uses flow matching to model return distributions, combining an analytical velocity target from interpolation endpoints with a self\-consistency target from the target network’s ODE trajectory\. However, its DCFM\-style self\-consistency objective can conflict with the source boundary condition required by flow matching, namely recovering the base prior att=0t=0\. We revisit this construction, with an expanded comparison to PCBF, in Appendix[B](https://arxiv.org/html/2605.08253#A2)\.

Building on the unified flow\-matching framework ofLipmanet al\.\([2024](https://arxiv.org/html/2605.08253#bib.bib29)\), PCBF addresses these issues through source\-consistent Bellman path repair and shared\-noise path coupling\. Together with its control\-variate training target, PCBF mitigates high\-variance Bellman flow\-matching updates and provides a stable flow\-based distributional critic for offline RL\.

## 3Preliminaries

#### Distributional RL\.

We consider a Markov decision process \(MDP\)\(Suttonet al\.,[1998](https://arxiv.org/html/2605.08253#bib.bib12)\)defined by the tuple\(𝒮,𝒜,p,ℛ,γ\)\(\\mathcal\{S\},\\mathcal\{A\},p,\\mathcal\{R\},\\gamma\), with state space𝒮\\mathcal\{S\}and action space𝒜\\mathcal\{A\}, transition kernelp​\(s′∣s,a\)p\(s^\{\\prime\}\\mid s,a\), and discountγ∈\(0,1\)\\gamma\\in\(0,1\)\. At each step we observe a reward sampleRRdrawn from a reward kernelℛ\(⋅∣s,a,s′\)\\mathcal\{R\}\(\\cdot\\mid s,a,s^\{\\prime\}\)\(deterministic rewards are the special caseR=r​\(s,a\)R=r\(s,a\)a\.s\.\)\. Letπ​\(a∣s\)\\pi\(a\\mid s\)denote a stochastic policy\. In distributional reinforcement learning \(DRL\), the return is modeled as a random variableZπ​\(s,a\)Z^\{\\pi\}\(s,a\)\. The evolution of return distributions is governed by the*distributional Bellman equation*

Zπ​\(s,a\)​=𝑑​R​\(s,a\)\+γ​Zπ​\(S′,A′\),Z^\{\\pi\}\(s,a\)\\;\\overset\{d\}\{=\}\\;R\(s,a\)\+\\gamma\\,Z^\{\\pi\}\(S^\{\\prime\},A^\{\\prime\}\),\(1\)where\(S′,A′\)\(S^\{\\prime\},A^\{\\prime\}\)are random variables sampled conditional on\(s,a\)\(s,a\), and=𝑑\\overset\{d\}\{=\}denotes equality in distribution\.

#### Flow matching\.

Flow matching\(Lipmanet al\.,[2022](https://arxiv.org/html/2605.08253#bib.bib13)\)is a class of continuous\-time generative models that learns a time\-dependent vector field to transport samples from a simple base distribution to a target data distribution\. In contrast to denoising diffusion models\(Hoet al\.,[2020](https://arxiv.org/html/2605.08253#bib.bib14); Songet al\.,[2020](https://arxiv.org/html/2605.08253#bib.bib15)\), which rely on stochastic differential equations \(SDEs\) and iterative noise injection, flow matching models are based on ordinary differential equations \(ODEs\), enabling simpler training and faster inference, while often achieving comparable or superior sample quality\.

Given a data distributionp​\(x\)∈ℝdp\(x\)\\in\\mathbb\{R\}^\{d\}, flow matching learns the parametersθ\\thetaof a time\-dependent velocity fieldvθv\_\{\\theta\}such that the induced flowψθ\\psi\_\{\\theta\}, defined as the solution to the ordinary differential equation

dd​t​ψθ​\(t,x\)=vθ​\(t,ψθ​\(t,x\)\)\\frac\{d\}\{dt\}\\psi\_\{\\theta\}\(t,x\)=v\_\{\\theta\}\(t,\\psi\_\{\\theta\}\(t,x\)\)\(2\)When initialized from a simple base distribution att=0t=0\(e\.g\., a standard Gaussian\), the resulting dynamics transport samples so that the distribution att=1t=1matches the target data distributionp​\(x\)p\(x\)\.

In this work, we adopt the simplest form of flow matching based on linear interpolation paths and uniform time sampling\(Lipmanet al\.,[2022](https://arxiv.org/html/2605.08253#bib.bib13)\)\. Specifically, given samplesX0∼𝒩​\(0,Id\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\)andX1∼p​\(x1\)X\_\{1\}\\sim p\(x\_\{1\}\), we define the linear interpolation

Xt=\(1−t\)​X0\+t​X1\.X\_\{t\}=\(1\-t\)X\_\{0\}\+tX\_\{1\}\.\(3\)The flow matching objective minimizes the squared error between the learned velocity field and the ground\-truth displacement direction

minθ⁡𝔼X0∼𝒩​\(0,Id\)X1∼p​\(x1\)t∼Unif​\(\[0,1\]\)​\[‖vθ​\(t,Xt\)−\(X1−X0\)‖22\]\.\\min\_\{\\theta\}\\;\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}X\_\{0\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\\\\ X\_\{1\}\\sim p\(x\_\{1\}\)\\\\ t\\sim\\mathrm\{Unif\}\(\[0,1\]\)\\end\{subarray\}\}\\left\[\\left\\\|v\_\{\\theta\}\(t,X\_\{t\}\)\-\(X\_\{1\}\-X\_\{0\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\.\(4\)Intuitively, this objective trains the velocity fieldvθv\_\{\\theta\}to predict the average direction that transports samples from the base distribution toward the data distribution along straight\-line paths\. At convergence, the resulting vector field defines an ODE whose solution generates samples fromp​\(x\)p\(x\)when integrated fromt=0t=0tot=1t=1\. At inference time, new samples are generated by solving the ODE in Equation \([2](https://arxiv.org/html/2605.08253#S3.E2)\)\. In this work, we use the explicit Euler method, which we find to be sufficient in practice\. We refer readers toLipmanet al\.\([2022](https://arxiv.org/html/2605.08253#bib.bib13)\)for further theoretical and practical details\.

## 4Path\-Coupled Bellman Flows

We now introduce*Path\-Coupled Bellman Flows*, a continuous\-time distributional reinforcement learning framework that integrates flow matching with the recursive geometry of the distributional Bellman equation\. Our goal is to learn return distributions with flow matching while usingsource\-consistent Bellman\-coupled paths: the current path starts from the required base prior att=0t\{=\}0, reaches the Bellman target att=1t\{=\}1, and maintains a pathwise affine relation to the successor flow at intermediate times—without requiring every time\-ttmarginal to be a distributional Bellman fixed point\.

### 4\.1Boundary mismatch of Pointwise Bellman path

Write the successor linear flow \(shared base noiseUU, terminal returnX′X^\{\\prime\}\) as

Zt′=\(1−t\)​U\+t​X′\.Z^\{\\prime\}\_\{t\}=\(1\-t\)U\+tX^\{\\prime\}\.\(5\)A natural attempt is to enforce the Bellman affine map*pointwise at every flow time*by defining the*uncorrected pointwise Bellman path*

ZtD≔R\+γ​Zt′\.Z^\{D\}\_\{t\}\\;\\coloneqq\\;R\+\\gamma\\,Z^\{\\prime\}\_\{t\}\.\(6\)This construction reaches the desired Bellman endpoint,Z1D=R\+γ​X′Z^\{D\}\_\{1\}=R\+\\gamma X^\{\\prime\}, but it violates the source boundary required by flow matching:Z0D=R\+γ​UZ^\{D\}\_\{0\}=R\+\\gamma U, which is generally not equal to the prescribed base noiseU∼𝒩​\(0,1\)U\\sim\\mathcal\{N\}\(0,1\)\. ThusZtDZ^\{D\}\_\{t\}is not a valid flow\-matching path for a model whose source distribution is fixed to the base prior\.

This boundary failure also clarifies how prior Bellman flow\-matching objectives connect to pointwise supervision\. In particular, the DCFM\-style term in Value Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\)can be read as imposing intermediate\-time consistency in velocity space: given an intermediate pointZtZ\_\{t\}, it evaluates the successor velocity at the Bellman\-inverse point\(Zt−R\)/γ\(Z\_\{t\}\-R\)/\\gammaand aligns the two velocity fields\. The Value\-Flows DCFM term can be viewed as a same\-time self\-consistency penalty evaluated at the Bellman\-inverse point\. Unlike the exact path derivative ofZt=R\+γ​Zt′Z\_\{t\}=R\+\\gamma Z^\{\\prime\}\_\{t\}, this objective does not include the explicitγ\\gammavelocity scaling; this distinction is one source of mismatch with the affine Bellman path geometry\. Therefore DCFM\-style training inherits the same source\-boundary conflict: the induced trajectory reaches the Bellman endpoint att=1t\{=\}1but starts fromR\+γ​UR\+\\gamma Urather thanUU\. Appendix[B](https://arxiv.org/html/2605.08253#A2)revisits this incompatibility at the distributional level and records formal statements \(including a degeneracy analysis\)\.

###### Lemma 4\.1\(DCFM as Bellman\-inverse same\-time self\-consistency\)\.

The geometry\-consistent DCFM objective evaluates the successor velocity at the Bellman\-inverse pointZt′=\(Zt−R\)/γZ^\{\\prime\}\_\{t\}=\(Z\_\{t\}\-R\)/\\gammaand penalizes same\-time velocity mismatch with the affine scaling:ℒDCFM\(v,vk\)=𝔼\[\(v\(t,Zt∣s,a\)−γvk\(t,Zt−Rγ\|s′,a′\)\)2\]\\mathcal\{L\}\_\{\\mathrm\{DCFM\}\}\(v,v\_\{k\}\)=\\mathbb\{E\}\\\!\\left\[\\left\(v\(t,Z\_\{t\}\\mid s,a\)\-\\gamma\\,v\_\{k\}\\\!\\left\(t,\\frac\{Z\_\{t\}\-R\}\{\\gamma\}\\,\\middle\|\\,s^\{\\prime\},a^\{\\prime\}\\right\)\\right\)^\{2\}\\right\], wherevkv\_\{k\}denotes the lagged target\-network velocity field\. This objective matches the path\-derivative scaling implied byZt=R\+γ​Zt′Z\_\{t\}=R\+\\gamma Z^\{\\prime\}\_\{t\}\.

Appendix[B](https://arxiv.org/html/2605.08253#A2)expands on Value Flows \(DCFM/BCFM objectives\), explains the distributional\-level incompatibility with the Gaussian source att=0t\{=\}0, and contrasts this with PCBF’s source\-consistent coupled\-path construction\.

#### Path\-coupled Bellman flows\.

PCBF uses*path coupling*: shared base noise ties the current and successor flows so that their velocities satisfy a Bellman\-shaped geometric relation while respecting the flow source att=0t\{=\}0and the Bellman endpoint att=1t\{=\}1\.

Building on this coupling, PCBF derives a family of training targets parameterized byλ\\lambda\. Different choices ofλ\\lambdainterpolate between purely sample\-based Bellman supervision and variance\-reduced targets that incorporate model\-predicted successor dynamics\.λ=0\\lambda=0is unbiased;λ\>0\\lambda\>0trades bias for variance reduction\. With shared\-noise coupling, the induced bias is small\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/pcbf_valueflows_noise_comparison.png)Figure 1:The Architecture of Path\-Coupled Bellman Flows \(PCBF\)\.

### 4\.2Bellman\-consistent shared\-noise paths

We begin by formalizing the pathwise structure implied by the distributional Bellman equation \([1](https://arxiv.org/html/2605.08253#S3.E1)\), which relates the return distributions at\(s,a\)\(s,a\)and\(s′,a′\)\(s^\{\\prime\},a^\{\\prime\}\)through an affine transformation\. Rather than enforcing Bellman structure only at isolated endpoint samples, PCBF encodes a pathwise affine coupling between current and successor flows\.

#### Flow\-map notation\.

We writeψθ1​\(x0∣s,a\)\\psi\_\{\\theta\}^\{\\,1\}\(x\_\{0\}\\mid s,a\)for the solution at timet=1t\{=\}1of the ODE \([2](https://arxiv.org/html/2605.08253#S3.E2)\) with initial conditionx0x\_\{0\}att=0t\{=\}0and conditioning\(s,a\)\(s,a\); equivalently this is the previous “ψθ​\(X0∣s,a,1\)\\psi\_\{\\theta\}\(X\_\{0\}\\mid s,a,1\)” notation\. For a base noise drawX0∼𝒩​\(0,Id\)X\_\{0\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\), the random variableψθ1​\(X0∣s,a\)\\psi\_\{\\theta\}^\{\\,1\}\(X\_\{0\}\\mid s,a\)follows the learned return lawZ^θ​\(s,a\)\\hat\{Z\}\_\{\\theta\}\(s,a\)\.

To couple the current and successor flows, we generate the successor return using the*same*base noise and a slowly updated target networkvθ−v\_\{\\theta^\{\-\}\}, with flow mapψθ−1\\psi\_\{\\theta^\{\-\}\}^\{\\,1\}:

X′=ψθ−1​\(X0∣s′,a′\)\.X^\{\\prime\}=\\psi\_\{\\theta^\{\-\}\}^\{\\,1\}\(X\_\{0\}\\mid s^\{\\prime\},a^\{\\prime\}\)\.\(7\)The theory permits any coupling betweenUUandX1′X\_\{1\}^\{\\prime\}with the correct terminal marginal, while the implemented PCBF algorithm uses the deterministic target\-flow couplingX1′=ψθ−1​\(U∣S′,A′\)X\_\{1\}^\{\\prime\}=\\psi\_\{\\theta^\{\-\}\}^\{\\,1\}\(U\\mid S^\{\\prime\},A^\{\\prime\}\)\.

We then define two time\-synchronized linear interpolation paths as follows:

Zts′\\displaystyle Z^\{s^\{\\prime\}\}\_\{t\}=\(1−t\)​X0\+t​X′,\\displaystyle=\(1\-t\)X\_\{0\}\+tX^\{\\prime\},\(successor path\),\\displaystyle\\text\{\(successor path\)\},\(8\)Zts\\displaystyle Z^\{s\}\_\{t\}=\(1−t\)​X0\+t​\(R\+γ​X′\),\\displaystyle=\(1\-t\)X\_\{0\}\+t\\bigl\(R\+\\gamma X^\{\\prime\}\\bigr\),\(current path\)\.\\displaystyle\\text\{\(current path\)\}\.\(9\)
Both paths originate from the same noise realization att=0t=0and terminate at Bellman\-related samples att=1t=1\. An equivalent expression of Equation \([9](https://arxiv.org/html/2605.08253#S4.E9)\) that explicitly reveals the Bellman geometry is:

Zts=t​R\+γ​Zts′\+\(1−t\)​\(1−γ\)​X0\.Z^\{s\}\_\{t\}=tR\+\\gamma Z^\{s^\{\\prime\}\}\_\{t\}\+\(1\-t\)\(1\-\\gamma\)X\_\{0\}\.\(10\)
The final term acts as a residual anchor that guarantees the exact alignment of both paths att=0t=0, ensuring shared\-noise initialization regardless ofγ\\gamma\. Att=1t=1, the construction satisfies the distributional Bellman boundary conditionZ1s=R\+γ​X′Z^\{s\}\_\{1\}=R\+\\gamma X^\{\\prime\}\.

Differentiating \([10](https://arxiv.org/html/2605.08253#S4.E10)\) yieldsZ˙ts=R−\(1−γ\)​X0\+γ​Z˙ts′\\dot\{Z\}^\{s\}\_\{t\}=R\-\(1\-\\gamma\)X\_\{0\}\+\\gamma\\dot\{Z\}^\{s^\{\\prime\}\}\_\{t\}\. Substitutingvθ−​\(t,Zts′∣s′,a′\)v\_\{\\theta^\{\-\}\}\(t,Z^\{s^\{\\prime\}\}\_\{t\}\\mid s^\{\\prime\},a^\{\\prime\}\)forZ˙ts′\\dot\{Z\}^\{s^\{\\prime\}\}\_\{t\}gives the geometric target atλ=γ\\lambda=\\gamma, revealing thatλ=0\\lambda=0uses pure samples whileλ=γ\\lambda=\\gammaeliminatesX′X^\{\\prime\}via velocity prediction\.

Differentiating the current\-state path in Equation \([9](https://arxiv.org/html/2605.08253#S4.E9)\) yields the ideal Bellman\-consistent velocity

Z˙ts=R\+γ​X′−X0≕Y\.\\dot\{Z\}^\{s\}\_\{t\}=R\+\\gamma X^\{\\prime\}\-X\_\{0\}\\;\\eqqcolon\\;Y\.\(11\)This quantity corresponds to the BCFM target\. While unbiased,YYdepends directly on the sampled successor returnX′=x′X^\{\\prime\}=x^\{\\prime\}and thus suffers from high variance\.

### 4\.3λ\\lambda\-parameterized control variates

PCBF reduces this variance by exploiting information available along the successor path\. Evaluating the target velocity field alongzts′z^\{s^\{\\prime\}\}\_\{t\}yields

ct=vθ−​\(t,Zts′∣s′,a′\)\.c\_\{t\}=v\_\{\\theta^\{\-\}\}\(t,Z^\{s^\{\\prime\}\}\_\{t\}\\mid s^\{\\prime\},a^\{\\prime\}\)\.\(12\)For linear interpolation, the true successor\-path velocity is constant and equal toX′−X0X^\{\\prime\}\-X\_\{0\}\. We therefore define a control variateCt=ct−\(X′−X0\)C\_\{t\}=c\_\{t\}\-\(X^\{\\prime\}\-X\_\{0\}\), which captures the discrepancy between the model\-predicted successor velocity and the sample\-based velocity\. For the population\-optimal successor fieldv¯⋆\\bar\{v\}^\{\\star\}, the intrinsic piece ofCtC\_\{t\}has mean zero conditional on the successor interpolantZts′Z^\{s^\{\\prime\}\}\_\{t\}\(see Section[5](https://arxiv.org/html/2605.08253#S5)\); for learnedv¯\\bar\{v\}the residual remains random but acts as a control variate\.

Using this control variate, we define the PCBF training target as follows:

utλ≔\(R\+γ​X′−X0\)\+λ​\[vθ−​\(t,Zts′∣s′,a′\)−\(X′−X0\)\]\.u\_\{t\}^\{\\lambda\}\\coloneqq\(R\+\\gamma X^\{\\prime\}\-X\_\{0\}\)\+\\lambda\\\!\\left\[v\_\{\\theta^\{\-\}\}\(t,Z^\{s^\{\\prime\}\}\_\{t\}\\mid s^\{\\prime\},a^\{\\prime\}\)\-\(X^\{\\prime\}\-X\_\{0\}\)\\right\]\.\(13\)
Settingλ=0\\lambda=0recovers the baseline BCFM estimator \(unbiased, high variance\)\. Nonzeroλ\\lambdaintroduces a variance\-reducing correction at the cost of potential bias\. Early in training,λ≈γ\\lambda\\approx\\gammais often effective, as it replaces the noisy successor sample with smoother velocity predictions\. As the target network improves, the correction becomes a more reliable control variate: its conditional mean under successor\-path information approaches that ofv¯⋆\\bar\{v\}^\{\\star\}while still providing variance reduction\.

### 4\.4Policy extraction for offline control

At deployment, we extract actions using a candidate\-action protocol based on mean terminal returns under the learned flow; details are given in Appendix[D](https://arxiv.org/html/2605.08253#A4.SS0.SSS0.Px6)\.

#### Training procedure\.

Putting the above components together, PCBF trains a current velocity networkvθv\_\{\\theta\}on a fixed offline dataset𝒟\\mathcal\{D\}, together with a slowly updated target velocity networkvθ−v\_\{\\theta^\{\-\}\}maintained by Polyak averaging\. At each iteration, we sample a minibatch of transitions from𝒟\\mathcal\{D\}, draw shared base noises and flow times, generate successor return samples using the target flow, and construct the path\-coupled Bellman interpolants\. Theλ\\lambda\-target is then formed as a control\-variate correction of the sample\-based Bellman velocity target\. Pseudocode is provided in Appendix[A](https://arxiv.org/html/2605.08253#A1)\.

## 5Theory: Bellman\-Interpolated Flows andλ\\lambda\-Targets

#### Setup\.

Fix a policyπ\\piand a state–action pair\(s,a\)\(s,a\)\. Let\(S′,R\)∼p\(⋅,⋅∣s,a\)\(S^\{\\prime\},R\)\\sim p\(\\cdot,\\cdot\\mid s,a\)andA′∼π\(⋅∣S′\)A^\{\\prime\}\\sim\\pi\(\\cdot\\mid S^\{\\prime\}\)\. LetX1′∼ηS′,A′X\_\{1\}^\{\\prime\}\\sim\\eta\_\{S^\{\\prime\},A^\{\\prime\}\}denote the terminal next\-return \(under the true return law\), and letU∼𝒩​\(0,1\)U\\sim\\mathcal\{N\}\(0,1\)denote base noise, independent of the MDP randomness\.

### 5\.1Coupled Bellman interpolants

###### Definition 5\.1\(Coupled Bellman interpolants\)\.

Fort∈\[0,1\)t\\in\[0,1\)define the successor and current interpolants

Xt′\\displaystyle X\_\{t\}^\{\\prime\}:=t​X1′\+\(1−t\)​U,\\displaystyle:=t\\,X\_\{1\}^\{\\prime\}\+\(1\-t\)\\,U,\(14\)Xt\\displaystyle X\_\{t\}:=t​\(R\+γ​X1′\)\+\(1−t\)​U\.\\displaystyle:=t\\,\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\+\(1\-t\)\\,U\.\(15\)

### 5\.2Bellman\-interpolated marginals and posterior operator

For the independent\-source Gaussian special case, define the implied noise \(inverse map\) for fixed\(x1′,r,t\)\(x\_\{1\}^\{\\prime\},r,t\):

u​\(x,x1′,r,t\):=x−t​\(r\+γ​x1′\)1−t\.u\(x,x\_\{1\}^\{\\prime\},r,t\):=\\frac\{x\-t\(r\+\\gamma x\_\{1\}^\{\\prime\}\)\}\{1\-t\}\.\(16\)
###### Definition 5\.2\(Bellman\-interpolated marginal law\)\.

Let

Ps,a,t:=ℒ​\(\(1−t\)​U\+t​\(R\+γ​X1′\)∣S=s,A=a\),t∈\[0,1\),P\_\{s,a,t\}:=\\mathcal\{L\}\\\!\\left\(\(1\-t\)U\+t\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\\mid S=s,A=a\\right\),\\qquad t\\in\[0,1\),under Definition[5\.1](https://arxiv.org/html/2605.08253#S5.Thmtheorem1), where\(U,X1′\)\(U,X\_\{1\}^\{\\prime\}\)may be coupled\. IfPs,a,tP\_\{s,a,t\}is absolutely continuous, write its density asPs,a​\(x,t\)P\_\{s,a\}\(x,t\)\. In the special caseU⟂\(R,X1′\)U\\perp\(R,X\_\{1\}^\{\\prime\}\)withU∼𝒩​\(0,1\)U\\sim\\mathcal\{N\}\(0,1\), this density has the explicit Gaussian\-kernel form

Ps,a\(x,t\)=𝔼\[11−tϕ\(u\(x,X1′,R,t\)\)\|S=s,A=a\]\.P\_\{s,a\}\(x,t\)=\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{1\-t\}\\,\\phi\\\!\\bigl\(u\(x,X\_\{1\}^\{\\prime\},R,t\)\\bigr\)\\;\\middle\|\\;S=s,A=a\\right\]\.\(17\)

###### Proposition 5\.3\(Endpoints of the Bellman\-interpolated marginals\)\.

Under Definition[5\.1](https://arxiv.org/html/2605.08253#S5.Thmtheorem1)and[5\.2](https://arxiv.org/html/2605.08253#S5.Thmtheorem2):

1. 1\.Att=0t=0:X0=UX\_\{0\}=U, soPs,a,0=ℒ​\(U∣S=s,A=a\)=𝒩​\(0,1\)P\_\{s,a,0\}=\\mathcal\{L\}\(U\\mid S=s,A=a\)=\\mathcal\{N\}\(0,1\)\.
2. 2\.Ast↑1t\\uparrow 1:Xt⇒R\+γ​X1′X\_\{t\}\\Rightarrow R\+\\gamma X\_\{1\}^\{\\prime\}in distribution, andPs,a,t⇒ℒ​\(R\+γ​X1′∣S=s,A=a\)P\_\{s,a,t\}\\Rightarrow\\mathcal\{L\}\(R\+\\gamma X\_\{1\}^\{\\prime\}\\mid S=s,A=a\)weakly\.

Defining the endpoint law by

Ps,a,1:=ℒ​\(R\+γ​X1′∣S=s,A=a\),P\_\{s,a,1\}:=\\mathcal\{L\}\(R\+\\gamma X\_\{1\}^\{\\prime\}\\mid S=s,A=a\),we havePs,a,1=\(Tπ​η\)s,aP\_\{s,a,1\}=\(T^\{\\pi\}\\eta\)\_\{s,a\}\.

###### Definition 5\.4\(Posterior operator\)\.

For any integrable random variableg=g​\(S′,A′,R,X1′,U,t\)g=g\(S^\{\\prime\},A^\{\\prime\},R,X\_\{1\}^\{\\prime\},U,t\)define

ℬs,a​\[g\]​\(x,t\):=𝔼​\[g∣Xt=x,S=s,A=a\]\.\\mathcal\{B\}\_\{s,a\}\[g\]\(x,t\):=\\mathbb\{E\}\\bigl\[g\\mid X\_\{t\}=x,\\;S=s,\\;A=a\\bigr\]\.\(18\)

In the independent\-source Gaussian case,ℬs,a\\mathcal\{B\}\_\{s,a\}admits the usual Bayes form obtained by weighting samples with the Gaussian kernel in Eq\. \([17](https://arxiv.org/html/2605.08253#S5.E17)\); details are in Appendix[C](https://arxiv.org/html/2605.08253#A3)\.

### 5\.3Posterior velocity identity

###### Theorem 5\.5\(Continuity equation and posterior velocity\)\.

For each\(s,a\)\(s,a\)andt∈\[0,1\)t\\in\[0,1\), the pathwise velocity is\(R\+γ​X1′\)−U\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U, and the population velocity is

vs,a⋆​\(x,t\)=𝔼​\[\(R\+γ​X1′\)−U∣Xt=x,S=s,A=a\]\.v^\{\\star\}\_\{s,a\}\(x,t\)=\\mathbb\{E\}\\\!\\left\[\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U\\mid X\_\{t\}=x,\\;S=s,\\;A=a\\right\]\.WhenPs,a,tP\_\{s,a,t\}has densityPs,a​\(x,t\)P\_\{s,a\}\(x,t\), it satisfies the continuity equation

∂tPs,a​\(x,t\)\+∂x\(Ps,a​\(x,t\)​vs,a⋆​\(x,t\)\)=0,\\partial\_\{t\}P\_\{s,a\}\(x,t\)\+\\partial\_\{x\}\\\!\\bigl\(P\_\{s,a\}\(x,t\)\\,v^\{\\star\}\_\{s,a\}\(x,t\)\\bigr\)=0,and equivalently

vs,a⋆​\(x,t\)=ℬs,a​\[\(R\+γ​X1′\)−U\]​\(x,t\)\.v^\{\\star\}\_\{s,a\}\(x,t\)=\\mathcal\{B\}\_\{s,a\}\\\!\\left\[\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U\\right\]\(x,t\)\.\(19\)

#### Proof

See Appendix[C](https://arxiv.org/html/2605.08253#A3)\.

#### Connection to flow matching\.

Theorem[5\.5](https://arxiv.org/html/2605.08253#S5.Thmtheorem5)identifies the population\-optimal velocity field transporting the Bellman\-interpolated marginals\. PCBF’s training objective is precisely a Monte Carlo regression estimator of this conditional expectation; the following analysis shows how theλ\\lambda\-target changes the estimator’s bias/variance\.

### 5\.4What theλ\\lambda\-target learns \(and the induced bias\)

Letv¯​\(⋅,t,S′,A′\)\\bar\{v\}\(\\cdot,t,S^\{\\prime\},A^\{\\prime\}\)be any \(possibly lagged\) successor velocity field and define the control variate residual

Cv¯:=v¯​\(Xt′,t,S′,A′\)−\(X1′−U\)\.C\_\{\\bar\{v\}\}:=\\bar\{v\}\(X\_\{t\}^\{\\prime\},t,S^\{\\prime\},A^\{\\prime\}\)\-\(X\_\{1\}^\{\\prime\}\-U\)\.Define the per\-sampleλ\\lambda\-target

vλ:=\(R\+γ​X1′\)−U\+λ​Cv¯\.v\_\{\\lambda\}:=\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U\+\\lambda\\,C\_\{\\bar\{v\}\}\.\(20\)
###### Proposition 5\.6\(L2L\_\{2\}regression target and bias decomposition\)\.

Fix\(s,a,t\)\(s,a,t\)\. The population minimizer of𝔼​\[\(f​\(Xt\)−vλ\)2∣S=s,A=a\]\\mathbb\{E\}\[\(f\(X\_\{t\}\)\-v\_\{\\lambda\}\)^\{2\}\\mid S=s,A=a\]satisfies

f⋆​\(x\)=𝔼​\[vλ∣Xt=x,S=s,A=a\]=ℬs,a​\[vλ\]​\(x,t\)\.f^\{\\star\}\(x\)=\\mathbb\{E\}\[v\_\{\\lambda\}\\mid X\_\{t\}=x,S=s,A=a\]=\\mathcal\{B\}\_\{s,a\}\[v\_\{\\lambda\}\]\(x,t\)\.Consequently the learned marginal field equals

vs,a\(λ\)​\(x,t\)=vs,a⋆​\(x,t\)\+λ​ℬs,a​\[Cv¯\]​\(x,t\)\.v^\{\(\\lambda\)\}\_\{s,a\}\(x,t\)=v^\{\\star\}\_\{s,a\}\(x,t\)\+\\lambda\\,\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)\.\(21\)

Forv¯=v¯⋆\\bar\{v\}=\\bar\{v\}^\{\\star\}, the intrinsic pieceEintE\_\{\\mathrm\{int\}\}has mean zero conditional on\(Xt′,S′,A′\)\(X\_\{t\}^\{\\prime\},S^\{\\prime\},A^\{\\prime\}\); however the PCBF regression conditions on the*current*interpolantXtX\_\{t\}, which generally introduces the bias term in \([21](https://arxiv.org/html/2605.08253#S5.E21)\)\. Theλ\\lambda\-scheme matches the population Bellman\-interpolant fieldvs,a⋆v^\{\\star\}\_\{s,a\}*iff*ℬs,a​\[Cv¯\]​\(x,t\)=0\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)=0holdsPs,a​\(⋅,t\)P\_\{s,a\}\(\\cdot,t\)\-a\.e\.

### 5\.5GeneralL2L\_\{2\}bias bound

*Why this matters: it bounds theλ\\lambda\-induced bias without assuming Gaussianity, unimodality, or deterministic rewards\. The earlier Gaussian closed form \(App\.[C\.8](https://arxiv.org/html/2605.08253#A3.SS8)\) is a special case\.*

Define the intrinsic residual \(under the population\-optimal successor field\)

Eint\\displaystyle E\_\{\\mathrm\{int\}\}:=v¯⋆​\(Xt′,t,S′,A′\)−\(X1′−U\),\\displaystyle=\\bar\{v\}^\{\\star\}\(X\_\{t\}^\{\\prime\},t,S^\{\\prime\},A^\{\\prime\}\)\-\(X\_\{1\}^\{\\prime\}\-U\),v¯⋆​\(z′,t,s′,a′\)\\displaystyle\\bar\{v\}^\{\\star\}\(z^\{\\prime\},t,s^\{\\prime\},a^\{\\prime\}\):=𝔼​\[X1′−U∣Xt′=z′,S=s′,A=a′\]\.\\displaystyle=\\mathbb\{E\}\[X\_\{1\}^\{\\prime\}\-U\\mid X\_\{t\}^\{\\prime\}=z^\{\\prime\},\\,S=s^\{\\prime\},\\,A=a^\{\\prime\}\]\.and the conditional intrinsic standard deviation

σx2:=𝔼​\[Eint2∣Xt=x,S=s,A=a\]\.\\sigma\_\{x\}^\{2\}:=\\mathbb\{E\}\\\!\\left\[E\_\{\\mathrm\{int\}\}^\{2\}\\mid X\_\{t\}=x,S=s,A=a\\right\]\.Letμx,t:=ℒ\(Xt′,S′,A′∣Xt=x,S=s,A=a\)\\mu\_\{x,t\}:=\\mathcal\{L\}\(X\_\{t\}^\{\\prime\},S^\{\\prime\},A^\{\\prime\}\\mid X\_\{t\}=x,S=s,A=a\)denote the conditional law of the successor\-side latent variables given the current interpolant\.

###### Proposition 5\.7\(L2L\_\{2\}bias bound for theλ\\lambda\-target\)\.

Under Eqs\. \([14](https://arxiv.org/html/2605.08253#S5.E14)\)–\([15](https://arxiv.org/html/2605.08253#S5.E15)\), writingCv¯=Eapprox\+EintC\_\{\\bar\{v\}\}=E\_\{\\mathrm\{approx\}\}\+E\_\{\\mathrm\{int\}\}withEapprox:=v¯−v¯⋆E\_\{\\mathrm\{approx\}\}:=\\bar\{v\}\-\\bar\{v\}^\{\\star\}, we have

\|ℬs,a​\[Cv¯\]​\(x,t\)\|≤‖v¯−v¯⋆‖L2​\(μx,t\)⏟\(i\) approximation\+σx⏟\(ii\) intrinsic,\\bigl\|\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)\\bigr\|\\;\\leq\\;\\underbrace\{\\bigl\\\|\\bar\{v\}\-\\bar\{v\}^\{\\star\}\\bigr\\\|\_\{L\_\{2\}\(\\mu\_\{x,t\}\)\}\}\_\{\\text\{\(i\) approximation\}\}\\;\+\\;\\underbrace\{\\sigma\_\{x\}\}\_\{\\text\{\(ii\) intrinsic\}\},\(22\)where \(i\) is the approximation error underμx,t\\mu\_\{x,t\}and \(ii\) is the intrinsic uncertainty, and consequently\|λ​ℬs,a​\[Cv¯\]​\(x,t\)\|≤λ⋅\(\(i\)\+\(ii\)\)\|\\lambda\\,\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)\|\\leq\\lambda\\cdot\(\(\\text\{i\}\)\+\(\\text\{ii\}\)\)\.

Term \(i\) is small when the target successor flow is accurate, which is the regime encouraged by target\-network averaging\. Term \(ii\) reflects uncertainty from conditioning on the current interpolant rather than the successor interpolant; shared\-noise coupling reduces this mismatch by tying both paths to the same latent source, consistent with the Gaussian scaling reported in App\.[C\.8](https://arxiv.org/html/2605.08253#A3.SS8)\.

App\.[C\.8](https://arxiv.org/html/2605.08253#A3.SS8)gives the closed\-formκ​\(t,γ,σ,ρ\)=t​\(1−t\)​σ2​\(ρ−γ\)\(t2​σ2\+\(1−t\)2\)​\(\(γ​t\)2​σ2\+\(1−t\)2\)\\kappa\(t,\\gamma,\\sigma,\\rho\)=\\tfrac\{t\(1\-t\)\\sigma^\{2\}\(\\rho\-\\gamma\)\}\{\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)\(\(\\gamma t\)^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)\}in the linear\-Gaussian special case, whereρ:=Corr​\(U,U′\)\\rho:=\\mathrm\{Corr\}\(U,U^\{\\prime\}\)is the correlation between current\-path and successor\-path base noises\. This shows that shared\-noise coupling \(ρ=1\\rho=1\) drives the bias to scale asO​\(\(1−γ\)​\(1−t\)\)O\(\(1\-\\gamma\)\(1\-t\)\)\. With independent noise \(ρ=0\\rho=0\), the\(1−γ\)\(1\-\\gamma\)cancellation of the shared\-noise case disappears; for fixed interiortt, the bias coefficient need not vanish asγ→1\\gamma\\to 1, although it still vanishes ast→1t\\to 1due to the common\(1−t\)\(1\-t\)factor\. The same analysis yields the variance\-minimizingλ⋆​\(t\)=γ​\(1−t\)\+ρ​t\\lambda^\{\\star\}\(t\)=\\gamma\(1\-t\)\+\\rho t\.

### 5\.6Shared\-noise Bellman contraction

Letp≥1p\\geq 1\. LetG=\{Gs,a\}G=\\\{G\_\{s,a\}\\\}andH=\{Hs,a\}H=\\\{H\_\{s,a\}\\\}be return generators driven by a shared latent seedξ\\xi, and define

Dp​\(G,H\):=sups,a\(𝔼ξ​\[\|Gs,a​\(ξ\)−Hs,a​\(ξ\)\|p\]\)1/p\.D\_\{p\}\(G,H\):=\\sup\_\{s,a\}\\left\(\\mathbb\{E\}\_\{\\xi\}\\left\[\|G\_\{s,a\}\(\\xi\)\-H\_\{s,a\}\(\\xi\)\|^\{p\}\\right\]\\right\)^\{1/p\}\.For a fixed policyπ\\pi, define the shared\-noise Bellman generator update by

\(𝒯pcG\)s,a=R\+γGS′,A′\(ξ′\),\(S′,A′\)∼p\(⋅∣s,a\)π\(⋅∣S′\)\.\(\\mathcal\{T\}\_\{\\rm pc\}G\)\_\{s,a\}=R\+\\gamma G\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\),\\quad\(S^\{\\prime\},A^\{\\prime\}\)\\sim p\(\\cdot\\mid s,a\)\\pi\(\\cdot\\mid S^\{\\prime\}\)\.
###### Proposition 5\.8\(Shared\-noise Bellman contraction\)\.

Under the common coupling where the same transition, action, reward, and successor latent seedξ′\\xi^\{\\prime\}are used when comparing𝒯pc​G\\mathcal\{T\}\_\{\\rm pc\}Gand𝒯pc​H\\mathcal\{T\}\_\{\\rm pc\}H,

Dp​\(𝒯pc​G,𝒯pc​H\)≤γ​Dp​\(G,H\)\.D\_\{p\}\(\\mathcal\{T\}\_\{\\rm pc\}G,\\mathcal\{T\}\_\{\\rm pc\}H\)\\leq\\gamma D\_\{p\}\(G,H\)\.Moreover, for PCBF interpolantsXtΦ=\(1−t\)​U\+t​\(R\+γ​ΦS′,A′​\(ξ′\)\)X\_\{t\}^\{\\Phi\}=\(1\-t\)U\+t\(R\+\\gamma\\Phi\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\)withΦ∈\{G,H\}\\Phi\\in\\\{G,H\\\}, we have

sups,a\(𝔼​\|XtG−XtH\|p\)1/p≤t​γ​Dp​\(G,H\)\.\\sup\_\{s,a\}\\left\(\\mathbb\{E\}\|X\_\{t\}^\{G\}\-X\_\{t\}^\{H\}\|^\{p\}\\right\)^\{1/p\}\\leq t\\gamma D\_\{p\}\(G,H\)\.

Proof in Appendix[C\.4](https://arxiv.org/html/2605.08253#A3.SS4)\. This is a contraction for the shared coupling used by PCBF\. It does not claim contraction of arbitrary independently coupled samples, nor does it by itself give a finite\-sample neural optimization guarantee\. PCBF can be viewed as a latent synchronous\-coupling method: current and successor return flows are driven by shared latent noise realization, so Bellman comparisons are performed pathwise on aligned stochastic trajectories rather than between independently sampled return realizations\. This synchronization underlies PCBF’s variance reduction, contraction behavior, and improved interpolation stability; the additionaltt\-factor further indicates that discrepancies vanish near the source distribution and grow gradually over flow time\.

## 6Experiments

In this section, we empirically evaluate PCBF across a diverse set of tasks, ranging from simulated toy environments to large\-scale, challenging reinforcement learning benchmarks\. We compare PCBF against prior baseline models and further present ablation studies and detailed analyses\.

### 6\.1Experimental Setup

#### Toy Environments\.

To rigorously validate the distributional accuracy of PCBF in a controlled setting, we first consider a suite of analytically tractable toy environments with known return structures\. Unlike complex control benchmarks, these environments admit closed\-form return laws, allowing us to unambiguously measure the discrepancy between the learned and ground\-truth distributions\. We utilize three specific environments: \(i\)Solitaire Dice; \(ii\)Bernoulli MRP; and \(iii\)Discrete Monte Carlo Chain\. Detailed definitions and additional results for these environments are provided in Appendix[I](https://arxiv.org/html/2605.08253#A9)\.

#### Benchmarks\.

We evaluate on 38 offline RL tasks: 20 state\-based OGBench manipulation tasks, 10 pixel\-based OGBench tasks, and 8 D4RL Adroit tasks\. OGBench tests long\-horizon manipulation, compositional reasoning, and sparse or semi\-sparse rewards; D4RL Adroit adds dexterous contact\-rich control\. Full dataset details are in Appendix[H\.1](https://arxiv.org/html/2605.08253#A8.SS1)\.

#### Methods and Evaluation\.

We compare PCBF against several representative offline RL baselines covering both distributional return modeling and flow\-based value learning\. For distributional RL baselines, we includeIQN\(Dabneyet al\.,[2018a](https://arxiv.org/html/2605.08253#bib.bib16)\)andCODAC\(Maet al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib17)\), which represent return distributions through quantile\-based critics\. We also compare againstValue Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\), a closely related flow\-based distributional RL method that directly models the full return distribution using flow matching\.

To further evaluate the role of flow\-based critic parameterization, we compare with FloQ\(Agrawallaet al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib31)\), which uses flow matching to parameterize scalar Q\-functions via iterative numerical integration rather than modeling the full return distribution\. In addition, we include strong scalar value\-based offline RL baselines, includingIQL\(Kostrikovet al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib18)\)andFQL\(Parket al\.,[2025c](https://arxiv.org/html/2605.08253#bib.bib7)\)\. This comparison allows us to assess whether PCBF’s gains arise from flow\-based critic parameterization alone or from modeling full return laws with source\-consistent Bellman\-coupled flows\.

Our experiments are conducted on a total of 38 tasks, including 20 state\-based OGBench single\-task environments, 10 visual OGBench tasks with pixel observations, and 8 D4RL Adroit manipulation tasks\. For state\-based and D4RL tasks, we report the mean and standard deviation over 8 random seeds, while for visual tasks we report results over 4 random seeds, following common practice\. Detailed descriptions of environments and datasets are provided in Appendix[H\.1](https://arxiv.org/html/2605.08253#A8.SS1), experimental implementations in Appendix[D](https://arxiv.org/html/2605.08253#A4), and hyperparameter settings in Appendix[H\.2](https://arxiv.org/html/2605.08253#A8.SS2)\.

### 6\.2Results and Discussion

#### OGBench and D4RL\.

Table[1](https://arxiv.org/html/2605.08253#S6.T1)summarizes aggregated performance across the 38\-task main set; per\-task scores are in Table[3](https://arxiv.org/html/2605.08253#A7.T3)\(Appendix[G](https://arxiv.org/html/2605.08253#A7)\)\. PCBF is not intended to uniformly dominate all offline RL baselines: its strongest gains appear where distributional fidelity and variance\-controlled bootstrapping matter, notablycube\-double\-playandpuzzle\-4x4\-play, where it achieves the best or near\-best aggregate performance\. Onscene\-playand D4RL Adroit, PCBF is broadly competitive with the strongest baselines\. We also observe clear failure cases: PCBF underperforms Value Flows oncube\-triple\-playandvisual\-cube\-double\-play, suggesting that policy extraction,λ\\lambdaselection, and/or visual encoders are not yet fully optimized for every long\-horizon or pixel\-based setting\. Onvisual\-antmaze\-teleport, PCBF matches or slightly exceeds the best reported baselines in aggregate\. When does path coupling help most? PCBF is strongest when critic\-side return\-law fidelity affects action ranking, especially for high\-variance, heavy\-tailed, or multimodal returns \(as also seen in our toy MRPs, where there is no actor or representation bottleneck and PCBF tracks ground\-truth CDFs more reliably than Value Flows under larger DCFM weight\)\. In offline control, however, better distribution modeling improves policy quality only if it changes candidate\-action ranking, so gains are largest on domains where variance\-controlled bootstrapping and tail fidelity affect selection \(e\.g\.,cube\-double\-play,puzzle\-4x4\-play\) and smaller when dominant errors come from action\-proposal coverage, visual representation, or very long\-horizon sparse credit assignment \(e\.g\.,cube\-triple\-play,visual\-cube\-double\-play\)\. This interpretation also explains whyλ\\lambdais task\-dependent: increasingλ\\lambdacan reduce Bellman\-target variance, but over\-reliance on successor\-velocity predictions can introduce bias when target\-flow quality or conditioning alignment is weak\.

Table 1:Offline RL Results\.Mean±\\pmstd over 8 seeds \(4 for pixel tasks\)\. FloQ aggregates for overlapping OGBench domains are taken fromAgrawallaet al\.\([2025](https://arxiv.org/html/2605.08253#bib.bib31)\); “–” indicates FloQ does not report that domain\. Bold values are within 95% of the best per\-domain\.
#### Pathwise Bellman residual and discretization\.

PCBF enforces the Bellman endpoint att=1t\{=\}1by construction, but training uses a finite\-step Euler solver\. LetZ^ts\\hat\{Z\}^\{s\}\_\{t\}andZ^ts′\\hat\{Z\}^\{s^\{\\prime\}\}\_\{t\}denote numerically integrated current and successor paths withNNfunction evaluations \(NFE\), and letγ~=γ​\(1−d\)\\tilde\{\\gamma\}=\\gamma\(1\-d\)on the transition\. We report the corrected residual

rcorr​\(t,N\):=𝔼​\[\|Z^ts−\(t​R\+γ~​Z^ts′\+\(1−t\)​\(1−γ~\)​U\)\|\],r\_\{\\mathrm\{corr\}\}\(t,N\):=\\mathbb\{E\}\\\!\\left\[\\left\|\\hat\{Z\}^\{s\}\_\{t\}\-\\bigl\(tR\+\\tilde\{\\gamma\}\\,\\hat\{Z\}^\{s^\{\\prime\}\}\_\{t\}\+\(1\-t\)\(1\-\\tilde\{\\gamma\}\)U\\bigr\)\\right\|\\right\],\(23\)which compares the integrated path to the*closed\-form*PCBF interpolation using the same integrated successor path \(nonterminal transitions,d=0d\{=\}0\)\. We sweept∈\[0,1\]t\\in\[0,1\]andN∈\{4,8,16,32\}N\\in\\\{4,8,16,32\\\}on Solitaire Dice and compare shared\-noise PCBF to an*independent\-noise*ablation where the successor path uses an independentU′U^\{\\prime\}while holding the solver budget fixed\. Figure[2](https://arxiv.org/html/2605.08253#S6.F2)shows that shared\-noise coupling yields smallerrcorrr\_\{\\mathrm\{corr\}\}across\(t,N\)\(t,N\), indicating that coupling reduces mismatch introduced by coarse discretization\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/nfe.png)Figure 2:Corrected Bellman residualrcorr​\(t,N\)r\_\{\\mathrm\{corr\}\}\(t,N\)on Solitaire Dice\. Shared\-noise PCBF \(blue\) maintains lower residuals than independent\-noise coupling \(orange\) across times and budgets\.
#### Training cost\.

PCBF has similar memory and wall\-clock cost to Value Flows and is slower than scalar critics; full cost details are in Appendix[E](https://arxiv.org/html/2605.08253#A5)\.

#### Toy Environments\.

On analytically tractable MRPs, PCBF closely matches ground\-truth return laws across discrete heavy\-tailed, continuous uniform, and long\-horizon multimodal distributions\. The strongest gains over Value Flows appear in Discrete MC and Solitaire, where preserving tail mass and avoiding variance collapse are critical\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/physics_combined.png)Figure 3:Learned PCBF Maps on Toy Environments\.Left Top \(Solitaire\); Right Top \(Bernoulli\); and Bottom \(Discrete MC\)\.Additionally, to rigorously assess distributional fidelity, we evaluate PCBF against Value Flows with varying dcfm coefficients on the toy environments\. Figure[4](https://arxiv.org/html/2605.08253#S6.F4)directly compares the learned return CDFs against exact Monte Carlo \(MC\) references, enabling precise evaluation beyond policy\-level metrics\. Across all environments, PCBF consistently matches the ground\-truth CDFs, while Value Flows \(VF\) exhibit progressive distributional degradation as dcfm increases—with higher dcfm values causing systematic underestimation of return variance, particularly in long\-horizon settings\. Extended comparisons across additional states are provided in Appendix[I\.3](https://arxiv.org/html/2605.08253#A9.SS3)in Figure[10](https://arxiv.org/html/2605.08253#A9.F10)\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/toy22.png)Figure 4:Distributional accuracy comparison on toy environments\.Learned return CDFs for PCBF and Value Flows \(with dcfm∈\{0,0\.5,1\}\\in\\\{0,0\.5,1\\\}\) compared against ground\-truth references\.Figure[5](https://arxiv.org/html/2605.08253#S6.F5)contrasts the stability of our method against Value Flows on the Solitaire and Discrete MC tasks\. Increasing the DCFM coefficient \(dcfm\) in Value Flows systematically degrades distributional accuracy, consistent with enforcing a full\-ttBellman\-shaped self\-consistency term that conflicts with the Gaussian source boundary\. In contrast, PCBF’sλ\\lambda\-target decouples variance reduction from the source/Bellman\-endpoint geometry and remains comparatively stable across hyperparameters\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/two_ablation.png)Figure 5:Hyperparameter Sensitivity Analysis \(PCBF vs\. Value Flows\) on Solitaire and Discrete MC Environments\.A more detailed discussion can be found in Figure[9](https://arxiv.org/html/2605.08253#A9.F9)of Appendix[I\.3](https://arxiv.org/html/2605.08253#A9.SS3)\.

## 7Limitations

PCBF has several limitations\.\(i\)Computational cost: training and evaluation require numerical integration of a velocity field; Table[2](https://arxiv.org/html/2605.08253#A5.T2)shows PCBF is in the same ballpark as other flow\-based critics but slower than scalar methods\.\(ii\)λ\\lambdais task\-dependent; we tune it once per domain, while fully adaptive online selection is left for future work\.\(iii\)We study scalar return distributions only; vector\-valued returns are not evaluated\.\(iv\)PCBF does not uniformly dominate all baselines \(see Section[6](https://arxiv.org/html/2605.08253#S6)\)\.\(v\)Large return magnitudes increase transport cost from a Gaussian prior; adaptive priors or normalization may help\.

## 8Conclusion

We presented Path\-Coupled Bellman Flows, a continuous\-time distributional RL framework that uses*source\-consistent Bellman\-coupled paths*and shared\-noise coupling so that flow matching is compatible with the Bellman endpoint while controlling target variance\. Aλ\\lambda\-parameterized control variate trades off bias and variance; we provided population\-velocity identification, anL2L\_\{2\}bias analysis, shared\-noise contraction, and an Euler\-sensitivity bound\. Empirically, PCBF recovers known return laws on toy MRPs, achieves strong results on several OGBench domains and competitive D4RL performance, and shows smaller corrected pathwise Bellman residuals under coarse discretization when coupling is enabled\.

## Impact Statement

This paper presents methodological advancements in Distributional Reinforcement Learning\. By improving the accuracy and stability of return distribution estimation, our work facilitates the development of risk\-sensitive agents capable of safer operation in uncertain environments, such as robotics or autonomous control\. While the advancement of autonomous decision\-making systems carries inherent general societal implications regarding automation and safety, this work focuses on fundamental algorithmic improvements and does not introduce ethical or privacy concerns\.

## References

- K\. D\. B\. J\. Adamet al\.\(2014\)A method for stochastic optimization\.arXiv preprint arXiv:1412\.69801412\(6\)\.Cited by:[Table 4](https://arxiv.org/html/2605.08253#A8.T4.2.4.2.2)\.
- B\. Agrawalla, M\. Nauman, K\. Agrawal, and A\. Kumar \(2025\)Floq: training critics via flow\-matching for scaling compute in value\-based rl\.arXiv preprint arXiv:2509\.06863\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p2.1),[Table 1](https://arxiv.org/html/2605.08253#S6.T1),[Table 1](https://arxiv.org/html/2605.08253#S6.T1.2.1.1)\.
- G\. Barth\-Maron, M\. W\. Hoffman, D\. Budden, W\. Dabney, D\. Horgan, D\. Tb, A\. Muldal, N\. Heess, and T\. Lillicrap \(2018\)Distributed distributional deterministic policy gradients\.arXiv preprint arXiv:1804\.08617\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px2.p1.1)\.
- M\. G\. Bellemare, W\. Dabney, and R\. Munos \(2017\)A distributional perspective on reinforcement learning\.InInternational conference on machine learning,pp\. 449–458\.Cited by:[§1](https://arxiv.org/html/2605.08253#S1.p1.1),[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px2.p1.1)\.
- M\. G\. Bellemare, W\. Dabney, and M\. Rowland \(2023\)Distributional reinforcement learning\.MIT Press\.Cited by:[§I\.1](https://arxiv.org/html/2605.08253#A9.SS1.SSS0.Px1.p1.2),[§I\.1](https://arxiv.org/html/2605.08253#A9.SS1.SSS0.Px2.p1.3),[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3)\.
- J\. Bradbury, R\. Frostig, P\. Hawkins, M\. J\. Johnson, C\. Leary, D\. Maclaurin, G\. Necula, A\. Paszke, J\. VanderPlas, S\. Wanderman\-Milne, and Q\. Zhang \(2018\)JAX: composable transformations of Python\+NumPy programsExternal Links:[Link](http://github.com/jax-ml/jax)Cited by:[Appendix H](https://arxiv.org/html/2605.08253#A8.p1.1)\.
- D\. Chen, Y\. Liu, Z\. Zhou, C\. Qu, and Y\. Qi \(2025\)Unleashing flow policies with distributional critics\.arXiv preprint arXiv:2509\.23087\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3)\.
- H\. Chen, C\. Lu, Z\. Wang, H\. Su, and J\. Zhu \(2023\)Score regularized policy optimization through diffusion behavior\.arXiv preprint arXiv:2310\.07297\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Cheng and J\. Weare \(2024\)The surprising efficiency of temporal difference learning for rare event prediction\.Advances in Neural Information Processing Systems37,pp\. 81257–81286\.Cited by:[§I\.1](https://arxiv.org/html/2605.08253#A9.SS1.SSS0.Px3.p1.4)\.
- W\. Dabney, G\. Ostrovski, D\. Silver, and R\. Munos \(2018a\)Implicit quantile networks for distributional reinforcement learning\.InInternational conference on machine learning,pp\. 1096–1105\.Cited by:[3rd item](https://arxiv.org/html/2605.08253#A8.I3.i3.p1.1),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p1.1)\.
- W\. Dabney, M\. Rowland, M\. Bellemare, and R\. Munos \(2018b\)Distributional reinforcement learning with quantile regression\.InProceedings of the AAAI conference on artificial intelligence,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2605.08253#S1.p1.1),[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Dong, C\. Zheng, C\. Finn, D\. Sadigh, and B\. Eysenbach \(2025\)Value flows\.External Links:2510\.07650,[Link](https://arxiv.org/abs/2510.07650)Cited by:[§B\.1](https://arxiv.org/html/2605.08253#A2.SS1.SSS0.Px1.p3.2),[Appendix B](https://arxiv.org/html/2605.08253#A2.p1.2),[5th item](https://arxiv.org/html/2605.08253#A8.I3.i5.p1.1),[§I\.3](https://arxiv.org/html/2605.08253#A9.SS3.p1.1),[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3),[§4\.1](https://arxiv.org/html/2605.08253#S4.SS1.p2.7),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p1.1)\.
- J\. Duan, Y\. Guan, S\. E\. Li, Y\. Ren, Q\. Sun, and B\. Cheng \(2021\)Distributional soft actor\-critic: off\-policy reinforcement learning for addressing value estimation errors\.IEEE transactions on neural networks and learning systems33\(11\),pp\. 6584–6598\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning,et al\.\(2018\)Impala: scalable distributed deep\-rl with importance weighted actor\-learner architectures\.InInternational conference on machine learning,pp\. 1407–1416\.Cited by:[Table 4](https://arxiv.org/html/2605.08253#A8.T4.2.13.11.2)\.
- N\. Espinosa\-Dice, K\. Brantley, and W\. Sun \(2025\)Expressive value learning for scalable offline reinforcement learning\.arXiv preprint arXiv:2510\.08218\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3)\.
- J\. Farebrother, M\. Pirotta, A\. Tirinzoni, R\. Munos, A\. Lazaric, and A\. Touati \(2025\)Temporal difference flows\.arXiv preprint arXiv:2503\.09817\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3)\.
- J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. Levine \(2020\)D4rl: datasets for deep data\-driven reinforcement learning\.arXiv preprint arXiv:2004\.07219\.Cited by:[§H\.1](https://arxiv.org/html/2605.08253#A8.SS1.SSS0.Px2),[§H\.1](https://arxiv.org/html/2605.08253#A8.SS1.p1.1),[§1](https://arxiv.org/html/2605.08253#S1.p5.3)\.
- P\. Hansen\-Estruch, I\. Kostrikov, M\. Janner, J\. G\. Kuba, and S\. Levine \(2023\)Idql: implicit q\-learning as an actor\-critic method with diffusion policies\.arXiv preprint arXiv:2304\.10573\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Hendrycks \(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[Table 4](https://arxiv.org/html/2605.08253#A8.T4.2.7.5.2)\.
- J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.Advances in neural information processing systems33,pp\. 6840–6851\.Cited by:[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px2.p1.1)\.
- M\. Janner, Y\. Du, J\. B\. Tenenbaum, and S\. Levine \(2022\)Planning with diffusion for flexible behavior synthesis\.arXiv preprint arXiv:2205\.09991\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2021\)Offline reinforcement learning with implicit q\-learning\.arXiv preprint arXiv:2110\.06169\.Cited by:[1st item](https://arxiv.org/html/2605.08253#A8.I3.i1.p1.1),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p2.1)\.
- Y\. Li, C\. Lai, C\. Schönlieb, Y\. Mitsufuji, and S\. Ermon \(2024\)Bellman diffusion: generative modeling as learning a linear operator in the distribution space\.arXiv preprint arXiv:2410\.01796\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p1.3)\.
- Y\. Lipman, R\. T\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le \(2022\)Flow matching for generative modeling\.arXiv preprint arXiv:2210\.02747\.Cited by:[§1](https://arxiv.org/html/2605.08253#S1.p2.1),[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px2.p3.2),[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px2.p3.6)\.
- Y\. Lipman, M\. Havasi, P\. Holderrieth, N\. Shaul, M\. Le, B\. Karrer, R\. T\. Chen, D\. Lopez\-Paz, H\. Ben\-Hamu, and I\. Gat \(2024\)Flow matching guide and code\.arXiv preprint arXiv:2412\.06264\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px3.p2.1)\.
- C\. Lu, H\. Chen, J\. Chen, H\. Su, C\. Li, and J\. Zhu \(2023\)Contrastive energy prediction for exact energy\-guided diffusion sampling in offline reinforcement learning\.InInternational Conference on Machine Learning,pp\. 22825–22855\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Ma, D\. Jayaraman, and O\. Bastani \(2021\)Conservative offline distributional reinforcement learning\.Advances in neural information processing systems34,pp\. 19235–19247\.Cited by:[4th item](https://arxiv.org/html/2605.08253#A8.I3.i4.p1.1),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p1.1)\.
- T\. Nguyen\-Tang, S\. Gupta, and S\. Venkatesh \(2021\)Distributional reinforcement learning via moment matching\.InProceedings of the AAAI conference on artificial intelligence,Vol\.35,pp\. 9144–9152\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Park, K\. Frans, B\. Eysenbach, and S\. Levine \(2025a\)OGBench: benchmarking offline goal\-conditioned rl\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§H\.1](https://arxiv.org/html/2605.08253#A8.SS1.SSS0.Px1),[§H\.1](https://arxiv.org/html/2605.08253#A8.SS1.SSS0.Px1.p2.1),[§H\.1](https://arxiv.org/html/2605.08253#A8.SS1.p1.1),[§1](https://arxiv.org/html/2605.08253#S1.p5.3)\.
- S\. Park, K\. Frans, B\. Eysenbach, and S\. Levine \(2025b\)OGBench: benchmarking offline goal\-conditioned rl\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 94937–94982\.Cited by:[Table 5](https://arxiv.org/html/2605.08253#A8.T5),[Table 5](https://arxiv.org/html/2605.08253#A8.T5.6.3.3)\.
- S\. Park, Q\. Li, and S\. Levine \(2025c\)Flow q\-learning\.arXiv preprint arXiv:2502\.02538\.Cited by:[Appendix D](https://arxiv.org/html/2605.08253#A4.SS0.SSS0.Px6.p1.4),[2nd item](https://arxiv.org/html/2605.08253#A8.I3.i2.p1.1),[Table 4](https://arxiv.org/html/2605.08253#A8.T4.2.13.11.2),[Appendix H](https://arxiv.org/html/2605.08253#A8.p1.1),[§6\.1](https://arxiv.org/html/2605.08253#S6.SS1.SSS0.Px3.p2.1)\.
- Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole \(2020\)Score\-based generative modeling through stochastic differential equations\.arXiv preprint arXiv:2011\.13456\.Cited by:[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px2.p1.1)\.
- R\. S\. Sutton, A\. G\. Barto,et al\.\(1998\)Reinforcement learning: an introduction\.Vol\.1,MIT press Cambridge\.Cited by:[§3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px1.p1.10)\.
- Z\. Wang, J\. J\. Hunt, and M\. Zhou \(2022\)Diffusion policies as an expressive policy class for offline reinforcement learning\.arXiv preprint arXiv:2208\.06193\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Zhang, W\. Zhang, and Q\. Gu \(2025\)Energy\-weighted flow matching for offline reinforcement learning\.arXiv preprint arXiv:2503\.04975\.Cited by:[§2](https://arxiv.org/html/2605.08253#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix APCBF training algorithm

Algorithm 1PCBF Training \(offline RL\)Input:offline dataset

𝒟\\mathcal\{D\}, discount

γ∈\(0,1\)\\gamma\\in\(0,1\), batch size

BB, control parameter

λ\\lambda, target update rate

τ\\tau
Define

sg​\(⋅\)\\mathrm\{sg\}\(\\cdot\)as stop\-gradient \(no backprop through the argument\)\.

Initialize current velocity network

vθv\_\{\\theta\}
Initialize target network

vθ−←vθv\_\{\\theta^\{\-\}\}\\leftarrow v\_\{\\theta\}
repeat

Sample minibatch

\{\(Si,Ai,Ri,Si′,di\)\}i=1B∼𝒟\\\{\(S\_\{i\},A\_\{i\},R\_\{i\},S^\{\\prime\}\_\{i\},d\_\{i\}\)\\\}\_\{i=1\}^\{B\}\\sim\\mathcal\{D\}
for

i=1i=1to

BBdo

Sample next action

Ai′∼π\(⋅\|Si′\)A^\{\\prime\}\_\{i\}\\sim\\pi\(\\cdot\|S^\{\\prime\}\_\{i\}\)
Sample

X0,i∼𝒩​\(0,1\)X\_\{0,i\}\\sim\\mathcal\{N\}\(0,1\)and

ti∼Unif​\[0,1\]t\_\{i\}\\sim\\mathrm\{Unif\}\[0,1\]
Xi′←ψθ−1​\(X0,i∣Si′,Ai′\)X^\{\\prime\}\_\{i\}\\leftarrow\\psi\_\{\\theta^\{\-\}\}^\{\\,1\}\(X\_\{0,i\}\\mid S^\{\\prime\}\_\{i\},A^\{\\prime\}\_\{i\}\)

γ~i←γ​\(1−di\)\\tilde\{\\gamma\}\_\{i\}\\leftarrow\\gamma\(1\-d\_\{i\}\)\{effective discount;

0if terminal\}

λi←\(1−di\)​λ\\lambda\_\{i\}\\leftarrow\(1\-d\_\{i\}\)\\,\\lambda\{no successor correction at terminal transitions\}

Zt,is′←\(1−ti\)​X0,i\+ti​Xi′Z^\{s^\{\\prime\}\}\_\{t,i\}\\leftarrow\(1\-t\_\{i\}\)X\_\{0,i\}\+t\_\{i\}X^\{\\prime\}\_\{i\}

Zt,is←ti​Ri\+γ~i​Zt,is′\+\(1−ti\)​\(1−γ~i\)​X0,iZ^\{s\}\_\{t,i\}\\leftarrow t\_\{i\}R\_\{i\}\+\\tilde\{\\gamma\}\_\{i\}Z^\{s^\{\\prime\}\}\_\{t,i\}\+\(1\-t\_\{i\}\)\(1\-\\tilde\{\\gamma\}\_\{i\}\)X\_\{0,i\}

ci←vθ−​\(ti,Zt,is′∣Si′,Ai′\)c\_\{i\}\\leftarrow v\_\{\\theta^\{\-\}\}\(t\_\{i\},Z^\{s^\{\\prime\}\}\_\{t,i\}\\mid S^\{\\prime\}\_\{i\},A^\{\\prime\}\_\{i\}\)

Ci←ci−\(Xi′−X0,i\)C\_\{i\}\\leftarrow c\_\{i\}\-\(X^\{\\prime\}\_\{i\}\-X\_\{0,i\}\)

Yi←Ri\+γ~i​Xi′−X0,iY\_\{i\}\\leftarrow R\_\{i\}\+\\tilde\{\\gamma\}\_\{i\}X^\{\\prime\}\_\{i\}\-X\_\{0,i\}

ui←Yi\+λi​Ciu\_\{i\}\\leftarrow Y\_\{i\}\+\\lambda\_\{i\}C\_\{i\}

endfor

ℒ←1B∑i=1B∥vθ\(ti,Zt,is∣Si,Ai\)−sg\(ui\)∥22\\mathcal\{L\}\\leftarrow\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\\|v\_\{\\theta\}\(t\_\{i\},Z^\{s\}\_\{t,i\}\\mid S\_\{i\},A\_\{i\}\)\-\\mathrm\{sg\}\(u\_\{i\}\)\\\|\_\{2\}^\{2\}

Take a gradient step on

θ\\thetato minimize

ℒ\\mathcal\{L\}
θ−←τ​θ\+\(1−τ\)​θ−\\theta^\{\-\}\\leftarrow\\tau\\theta\+\(1\-\\tau\)\\theta^\{\-\}

untilconvergence

## Appendix BValue Flows \(DCFM/BCFM\) and comparison with PCBF

This appendix complements Lemma[4\.1](https://arxiv.org/html/2605.08253#S4.Thmtheorem1): we summarize Value Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\), state its DCFM/BCFM training objectives, explain why enforcing a full\-ttdistributional Bellman identity clashes with the Gaussian source att=0t\{=\}0, and contrast this with PCBF’s coupled\-path construction\.

Value Flows learns a time\-dependent vector fieldv:ℝ×\[0,1\]×𝒮×𝒜→ℝv:\\mathbb\{R\}\\times\[0,1\]\\times\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}that generates a diffeomorphic flowψ\\psitransforming samples from a standard GaussianX0∼𝒩​\(0,1\)X\_\{0\}\\sim\\mathcal\{N\}\(0,1\)into return samples\. The flow satisfies the ODEdd​t​ψ​\(X0\|t,s,a\)=v​\(ψ​\(X0\|t,s,a\)\|t,s,a\)\\frac\{d\}\{dt\}\\psi\(X\_\{0\}\|t,s,a\)=v\(\\psi\(X\_\{0\}\|t,s,a\)\|t,s,a\)withψ​\(X0\|0,s,a\)=X0\\psi\(X\_\{0\}\|0,s,a\)=X\_\{0\}\.

To train the return vector field, Value Flows combines two losses\. The first is the Distributional Conditional Flow Matching \(DCFM\) loss, which matches the velocity field recursively through bootstrapping from a target network \(cf\. Lemma[4\.1](https://arxiv.org/html/2605.08253#S4.Thmtheorem1)\)\. The second is the Bootstrapped Conditional Flow Matching \(BCFM\) loss, which provides direct supervision using TD targets:

ℒBCFM\(v\)=𝔼𝒟\[∥v\(t,ZtTD\|s,a\)−\(Z1TD−X0\)∥2\]\\mathcal\{L\}\_\{\\text\{BCFM\}\}\(v\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\Big\[\\big\\\|v\(t,Z\_\{t\}^\{\\text\{TD\}\}\|s,a\)\-\(Z\_\{1\}^\{\\text\{TD\}\}\-X\_\{0\}\)\\big\\\|^\{2\}\\Big\]\(24\)
whereZ1TD=R​\(s,a\)\+γ​X′Z\_\{1\}^\{\\text\{TD\}\}=R\(s,a\)\+\\gamma X^\{\\prime\}is the bootstrapped return \(withX′X^\{\\prime\}sampled from the target return distribution ats′,a′s^\{\\prime\},a^\{\\prime\}\), andZtTD=\(1−t\)​X0\+t​Z1TDZ\_\{t\}^\{\\text\{TD\}\}=\(1\-t\)X\_\{0\}\+tZ\_\{1\}^\{\\text\{TD\}\}is the linear interpolation between the noise and the target return\. The final loss isℒValue Flow=ℒDCFM\+λ​ℒBCFM\\mathcal\{L\}\_\{\\text\{Value Flow\}\}=\\mathcal\{L\}\_\{\\text\{DCFM\}\}\+\\lambda\\mathcal\{L\}\_\{\\text\{BCFM\}\}\.

#### Remark on DCFM scaling\.

For clarity, the original Value Flows DCFM expression is often written without an explicitγ\\gammamultiplier on the successor\-side velocity term\. In this paper, we also consider the geometry\-consistent corrected DCFM\-style form implied by the affine Bellman pathZt=R\+γ​Zt′Z\_\{t\}=R\+\\gamma Z\_\{t\}^\{\\prime\}, whose time derivative givesZ˙t=γ​Z˙t′\\dot\{Z\}\_\{t\}=\\gamma\\dot\{Z\}\_\{t\}^\{\\prime\}\. Accordingly, the corrected same\-time mismatch uses

∥v\(t,Zt∣s,a\)−γvk\(t,Zt−Rγ\|s′,a′\)∥2\.\\left\\\|v\(t,Z\_\{t\}\\mid s,a\)\-\\gamma\\,v\_\{k\}\\\!\\left\(t,\\frac\{Z\_\{t\}\-R\}\{\\gamma\}\\,\\middle\|\\,s^\{\\prime\},a^\{\\prime\}\\right\)\\right\\\|^\{2\}\.This distinction isolates path\-geometry consistency and does not change our description of the original Value Flows objective\.

### B\.1Bellman inconsistency in Value Flows

#### Direct fixed\-point incompatibility att=0t=0\(distribution\-level statement\)\.

In this11\-state MRP, “Bellman at eachtt” implies that the time\-ttmarginal density must satisfy

pv\(⋅∣t\)=\!𝒯pv\(⋅∣t\):=𝔼R\[1γpv\(⋅−Rγ\|t\)\],p\_\{v\}\(\\cdot\\mid t\)\\stackrel\{\{\\scriptstyle\!\}\}\{\{=\}\}\\mathcal\{T\}p\_\{v\}\(\\cdot\\mid t\):=\\mathbb\{E\}\_\{R\}\\left\[\\frac\{1\}\{\\gamma\}p\_\{v\}\\left\(\\left\.\\frac\{\\cdot\-R\}\{\\gamma\}\\right\\rvert\\,t\\right\)\\right\],which is exactly the density form of the distributional Bellman equation\.

Because𝒯\\mathcal\{T\}has a unique fixed point distributionpZ⋆p\_\{Z\}^\{\\star\}\(the return law\), the condition above impliespv\(⋅∣t\)=pZ⋆p\_\{v\}\(\\cdot\\mid t\)=p\_\{Z\}^\{\\star\}for everytt\. In particular, it would requirepv\(⋅∣0\)=pZ⋆p\_\{v\}\(\\cdot\\mid 0\)=p\_\{Z\}^\{\\star\}\. However, the flow boundary condition enforcespv\(⋅∣0\)=𝒩\(0,1\)p\_\{v\}\(\\cdot\\mid 0\)=\\mathcal\{N\}\(0,1\)\.

Consequently, a literal full\-ttdistributional Bellman fixed\-point constraint would be incompatible with the Gaussian source boundary\. DCFM\-style same\-time velocity consistency can be viewed as a tractable proxy for this idea, and may therefore introduce source\-boundary tension when weighted strongly\. In contrast, BCFM avoids this conflict by only “anchoring” the terminal distribution\(t=1\)\(t=1\)via a TD target\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\), effectively ignoring the consistency of the intermediate flow\.

We formalize the resulting degeneracy \(constant\-in\-zzsolutions under vanishing DCFM loss\) as follows\. We do not claim that BCFM is ineffective; rather, our concern is that a strongly weighted DCFM\-style same\-time consistency term can bias the learned intermediate vector field toward source\-incompatible solutions, whereas PCBF separates endpoint Bellman consistency from intermediate path construction\. In contrast, PCBF achieves Bellman consistency by explicitly coupling the flow paths so that the terminal distribution att=1t=1satisfies the Bellman equation pointwise while respecting the prescribed Gaussian source att=0t\{=\}0, avoiding the full\-ttfixed\-point conflict discussed above\.

###### Proposition B\.1\(DCFM admits constant\-in\-zzsolutions\)\.

Consider a11\-state/11\-action MRP with discountγ∈\(0,1\)\\gamma\\in\(0,1\)and i\.i\.d\. rewardRR\. Let the flow start from shared base noiseZ0=X0∼𝒩​\(0,1\)Z\_\{0\}=X\_\{0\}\\sim\\mathcal\{N\}\(0,1\)and evolve byd​Zt/d​t=v​\(Zt,t\)dZ\_\{t\}/dt=v\(Z\_\{t\},t\)\. At a fixed point of DCFM \(ignoring target\-network lag\), we may takevk=vv\_\{k\}=v\. Assumev​\(⋅,t\)v\(\\cdot,t\)is continuous for a\.e\.ttand the support ofRRhas non\-zero measure\. Assume additionally that0lies in the support ofRRand that the almost\-sure identityv​\(R\+γ​z,t\)=v​\(z,t\)v\(R\+\\gamma z,t\)=v\(z,t\)extends continuously toR=0R=0on the support of\(R,z\)\(R,z\)pairs occurring under the DCFM loss\. IfℒDCFM​\(v\)=0\\mathcal\{L\}\_\{\\mathrm\{DCFM\}\}\(v\)=0then for a\.e\.tt,v​\(⋅,t\)v\(\\cdot,t\)is constant inzzon the support ofpv\(⋅∣t\)p\_\{v\}\(\\cdot\\mid t\)\.

###### Proof\.

In the11\-state/11\-action case and at DCFM fixed pointvk=vv\_\{k\}=v, ifℒDCFM​\(v\)=0\\mathcal\{L\}\_\{\\mathrm\{DCFM\}\}\(v\)=0, the integrand vanishes almost surely, hencev​\(R\+γ​z,t\)=v​\(z,t\)v\(R\+\\gamma z,t\)=v\(z,t\)for a\.s\.\(R,z\)\(R,z\)\. Under the assumption that0lies in the support ofRRand that this identity extends continuously toR=0R=0, takingR→0R\\to 0along the support givesv​\(γ​z,t\)=v​\(z,t\)v\(\\gamma z,t\)=v\(z,t\)for a\.s\.zz\. Iterating yields

v​\(z,t\)=v​\(γ​z,t\)=v​\(γ2​z,t\)=⋯=v​\(γn​z,t\),n≥1\.v\(z,t\)=v\(\\gamma z,t\)=v\(\\gamma^\{2\}z,t\)=\\cdots=v\(\\gamma^\{n\}z,t\),\\quad n\\geq 1\.Sinceγ∈\(0,1\)\\gamma\\in\(0,1\),γn​z→0\\gamma^\{n\}z\\to 0asn→∞n\\to\\infty; by continuity ofv​\(⋅,t\)v\(\\cdot,t\),

v​\(z,t\)=limn→∞v​\(γn​z,t\)=v​\(0,t\)v\(z,t\)=\\lim\_\{n\\to\\infty\}v\(\\gamma^\{n\}z,t\)=v\(0,t\)for a\.s\.zz\. Therefore, for a\.e\.tt,v​\(⋅,t\)v\(\\cdot,t\)is constant inzzon the support ofpv\(⋅∣t\)p\_\{v\}\(\\cdot\\mid t\)\. ∎

## Appendix CProof sketches

### C\.1Derivation of the density formula and proof of Proposition[5\.3](https://arxiv.org/html/2605.08253#S5.Thmtheorem3)

Definition[5\.2](https://arxiv.org/html/2605.08253#S5.Thmtheorem2)is stated at the level of the pushforward lawPs,a,t=ℒ​\(Xt∣s,a\)P\_\{s,a,t\}=\\mathcal\{L\}\(X\_\{t\}\\mid s,a\)and does not require independence betweenUUandX1′X\_\{1\}^\{\\prime\}\. The explicit kernel formula \([17](https://arxiv.org/html/2605.08253#S5.E17)\) is the independent\-source Gaussian special case obtained by change\-of\-variables usingu​\(x,x1′,r,t\)u\(x,x\_\{1\}^\{\\prime\},r,t\)from \([16](https://arxiv.org/html/2605.08253#S5.E16)\) and the Jacobian1/\(1−t\)1/\(1\-t\)\.

Proof of Proposition[5\.3](https://arxiv.org/html/2605.08253#S5.Thmtheorem3):

1. 1\.Att=0t=0, from \([14](https://arxiv.org/html/2605.08253#S5.E14)\)–\([15](https://arxiv.org/html/2605.08253#S5.E15)\) we haveX0=UX\_\{0\}=U, hencePs,a,0=𝒩​\(0,1\)P\_\{s,a,0\}=\\mathcal\{N\}\(0,1\)\.
2. 2\.Ast→1t\\to 1, we haveXt=t​\(R\+γ​X1′\)\+\(1−t\)​U→R\+γ​X1′X\_\{t\}=t\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\+\(1\-t\)U\\to R\+\\gamma X\_\{1\}^\{\\prime\}pointwise, henceXt⇒R\+γ​X1′X\_\{t\}\\Rightarrow R\+\\gamma X\_\{1\}^\{\\prime\}in distribution\. The weak convergence ofPs,a​\(⋅,t\)P\_\{s,a\}\(\\cdot,t\)follows, and pointwise convergence of densities holds under dominated convergence when the limit law is absolutely continuous\.

### C\.2Properties of the posterior operator

The posterior operatorℬs,a\\mathcal\{B\}\_\{s,a\}defined in \([18](https://arxiv.org/html/2605.08253#S5.E18)\) satisfies:

1. 1\.\(Linearity\)ℬs,a​\[α​g1\+β​g2\]​\(x,t\)=α​ℬs,a​\[g1\]​\(x,t\)\+β​ℬs,a​\[g2\]​\(x,t\)\\mathcal\{B\}\_\{s,a\}\[\\alpha g\_\{1\}\+\\beta g\_\{2\}\]\(x,t\)=\\alpha\\mathcal\{B\}\_\{s,a\}\[g\_\{1\}\]\(x,t\)\+\\beta\\mathcal\{B\}\_\{s,a\}\[g\_\{2\}\]\(x,t\)\.
2. 2\.\(Tower property\) Ifh=h​\(Xt,S,A\)h=h\(X\_\{t\},S,A\)is measurable w\.r\.t\.\(Xt,S,A\)\(X\_\{t\},S,A\), thenℬs,a​\[h⋅g\]​\(x,t\)=h​\(x,s,a\)​ℬs,a​\[g\]​\(x,t\)\\mathcal\{B\}\_\{s,a\}\[h\\cdot g\]\(x,t\)=h\(x,s,a\)\\,\\mathcal\{B\}\_\{s,a\}\[g\]\(x,t\)and𝔼​\[ℬs,a​\[g\]​\(Xt,t\)∣S=s,A=a\]=𝔼​\[g∣S=s,A=a\]\\mathbb\{E\}\[\\mathcal\{B\}\_\{s,a\}\[g\]\(X\_\{t\},t\)\\mid S=s,A=a\]=\\mathbb\{E\}\[g\\mid S=s,A=a\]\.
3. 3\.\(Bayes form, independent\-source special case\) In the independent\-source Gaussian case, Bayes’ rule yields: Ps,a​\(x,t\)\\displaystyle P\_\{s,a\}\(x,t\)=ℬ~s,a​\[1\]​\(x,t\),\\displaystyle=\\widetilde\{\\mathcal\{B\}\}\_\{s,a\}\[1\]\(x,t\),\(25\)ℬs,a​\[g\]​\(x,t\)\\displaystyle\\mathcal\{B\}\_\{s,a\}\[g\]\(x,t\)=ℬ~s,a​\[g\]​\(x,t\)ℬ~s,a​\[1\]​\(x,t\)=ℬ~s,a​\[g\]​\(x,t\)Ps,a​\(x,t\)\.\\displaystyle=\\frac\{\\widetilde\{\\mathcal\{B\}\}\_\{s,a\}\[g\]\(x,t\)\}\{\\widetilde\{\\mathcal\{B\}\}\_\{s,a\}\[1\]\(x,t\)\}=\\frac\{\\widetilde\{\\mathcal\{B\}\}\_\{s,a\}\[g\]\(x,t\)\}\{P\_\{s,a\}\(x,t\)\}\.\(26\)This is the continuous form of Bayes’ rule: the densityPs,aP\_\{s,a\}normalizes the unnormalized posterior to yield the conditional expectationℬs,a\\mathcal\{B\}\_\{s,a\}\.

These follow from standard properties of conditional expectation; Eqs\. \([25](https://arxiv.org/html/2605.08253#A3.E25)\)–\([26](https://arxiv.org/html/2605.08253#A3.E26)\) additionally use the independent\-source Gaussian representation \([17](https://arxiv.org/html/2605.08253#S5.E17)\)\.

### C\.3Proof of Theorem[5\.5](https://arxiv.org/html/2605.08253#S5.Thmtheorem5)

DifferentiateXt=t​\(R\+γ​X1′\)\+\(1−t\)​UX\_\{t\}=t\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\+\(1\-t\)Uto obtain the pathwise velocity\(R\+γ​X1′\)−U\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U\. Taking conditional expectation given\(Xt=x,S=s,A=a\)\(X\_\{t\}=x,S=s,A=a\)yields \([19](https://arxiv.org/html/2605.08253#S5.E19)\)\. WhenPs,a,tP\_\{s,a,t\}is absolutely continuous, the continuity equation follows from the weak form for smooth compactly supported test functionsφ\\varphiand under the required integrability assumptions:

dd​t​𝔼​\[φ​\(Xt\)∣s,a\]=𝔼​\[φ′​\(Xt\)​\(\(R\+γ​X1′\)−U\)∣s,a\]\\frac\{d\}\{dt\}\\mathbb\{E\}\[\\varphi\(X\_\{t\}\)\\mid s,a\]=\\mathbb\{E\}\[\\varphi^\{\\prime\}\(X\_\{t\}\)\\,\(\(R\+\\gamma X\_\{1\}^\{\\prime\}\)\-U\)\\mid s,a\]and disintegration with respect toXtX\_\{t\}\.

### C\.4Proof of Proposition[5\.8](https://arxiv.org/html/2605.08253#S5.Thmtheorem8)

For any\(s,a\)\(s,a\), using the common coupling cancels the reward:

\(𝒯pc​G\)s,a−\(𝒯pc​H\)s,a=γ​\(GS′,A′​\(ξ′\)−HS′,A′​\(ξ′\)\)\.\(\\mathcal\{T\}\_\{\\rm pc\}G\)\_\{s,a\}\-\(\\mathcal\{T\}\_\{\\rm pc\}H\)\_\{s,a\}=\\gamma\\bigl\(G\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\-H\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\\bigr\)\.Thus

\(𝔼​\[\|\(𝒯pc​G\)s,a−\(𝒯pc​H\)s,a\|p\]\)1/p=γ​\(𝔼​\[\|GS′,A′​\(ξ′\)−HS′,A′​\(ξ′\)\|p\]\)1/p≤γ​Dp​\(G,H\)\.\\left\(\\mathbb\{E\}\\left\[\|\(\\mathcal\{T\}\_\{\\rm pc\}G\)\_\{s,a\}\-\(\\mathcal\{T\}\_\{\\rm pc\}H\)\_\{s,a\}\|^\{p\}\\right\]\\right\)^\{1/p\}=\\gamma\\left\(\\mathbb\{E\}\\left\[\|G\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\-H\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\|^\{p\}\\right\]\\right\)^\{1/p\}\\leq\\gamma D\_\{p\}\(G,H\)\.Taking the supremum over\(s,a\)\(s,a\)gives the first claim\. For the interpolants,

XtG−XtH=t​γ​\(GS′,A′​\(ξ′\)−HS′,A′​\(ξ′\)\),X\_\{t\}^\{G\}\-X\_\{t\}^\{H\}=t\\gamma\\bigl\(G\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\-H\_\{S^\{\\prime\},A^\{\\prime\}\}\(\\xi^\{\\prime\}\)\\bigr\),so the same argument gives thet​γt\\gammabound\.

### C\.5Proof of Proposition[5\.6](https://arxiv.org/html/2605.08253#S5.Thmtheorem6)

Use the standardL2L\_\{2\}projection lemma: the minimizer of𝔼​\[\(f​\(X\)−Y\)2\]\\mathbb\{E\}\[\(f\(X\)\-Y\)^\{2\}\]isf​\(x\)=𝔼​\[Y∣X=x\]f\(x\)=\\mathbb\{E\}\[Y\\mid X=x\]\. Apply withX=XtX=X\_\{t\}andY=vλY=v\_\{\\lambda\}to getf⋆​\(x\)=ℬs,a​\[vλ\]​\(x,t\)f^\{\\star\}\(x\)=\\mathcal\{B\}\_\{s,a\}\[v\_\{\\lambda\}\]\(x,t\), then expandvλv\_\{\\lambda\}using \([20](https://arxiv.org/html/2605.08253#S5.E20)\) to get \([21](https://arxiv.org/html/2605.08253#S5.E21)\)\.

### C\.6Proof of Proposition[5\.7](https://arxiv.org/html/2605.08253#S5.Thmtheorem7)

Decompose \(using the theory notationXt′X\_\{t\}^\{\\prime\}for the successor interpolant\)

Cv¯=\(v¯​\(Xt′,t,S′,A′\)−v¯⋆​\(Xt′,t,S′,A′\)\)⏟=⁣:Eapprox\+\(v¯⋆​\(Xt′,t,S′,A′\)−\(X1′−U\)\)⏟=⁣:Eint\.C\_\{\\bar\{v\}\}=\\underbrace\{\\bigl\(\\bar\{v\}\(X\_\{t\}^\{\\prime\},t,S^\{\\prime\},A^\{\\prime\}\)\-\\bar\{v\}^\{\\star\}\(X\_\{t\}^\{\\prime\},t,S^\{\\prime\},A^\{\\prime\}\)\\bigr\)\}\_\{=:E\_\{\\mathrm\{approx\}\}\}\\;\+\\;\\underbrace\{\\bigl\(\\bar\{v\}^\{\\star\}\(X\_\{t\}^\{\\prime\},t,S^\{\\prime\},A^\{\\prime\}\)\-\(X\_\{1\}^\{\\prime\}\-U\)\\bigr\)\}\_\{=:E\_\{\\mathrm\{int\}\}\}\.By construction ofv¯⋆\\bar\{v\}^\{\\star\},𝔼​\[Eint∣Xt′,S′,A′\]=0\\mathbb\{E\}\[E\_\{\\mathrm\{int\}\}\\mid X\_\{t\}^\{\\prime\},S^\{\\prime\},A^\{\\prime\}\]=0\. Therefore

\|ℬs,a​\[Cv¯\]​\(x,t\)\|≤\|ℬs,a​\[Eapprox\]​\(x,t\)\|\+\|ℬs,a​\[Eint\]​\(x,t\)\|\.\\bigl\|\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)\\bigr\|\\leq\\bigl\|\\mathcal\{B\}\_\{s,a\}\[E\_\{\\mathrm\{approx\}\}\]\(x,t\)\\bigr\|\+\\bigl\|\\mathcal\{B\}\_\{s,a\}\[E\_\{\\mathrm\{int\}\}\]\(x,t\)\\bigr\|\.For the first term, conditional Jensen gives

\|ℬs,a\[Eapprox\]\(x,t\)\|=\|𝔼\[Eapprox∣Xt=x,S=s,A=a\]\|≤𝔼\[\|Eapprox\|\|Xt=x,S=s,A=a\]≤∥v¯−v¯⋆∥L2​\(μx,t\)\.\\bigl\|\\mathcal\{B\}\_\{s,a\}\[E\_\{\\mathrm\{approx\}\}\]\(x,t\)\\bigr\|=\\bigl\|\\mathbb\{E\}\[E\_\{\\mathrm\{approx\}\}\\mid X\_\{t\}=x,S=s,A=a\]\\bigr\|\\leq\\mathbb\{E\}\\\!\\left\[\\bigl\|E\_\{\\mathrm\{approx\}\}\\bigr\|\\;\\big\|\\;X\_\{t\}=x,S=s,A=a\\right\]\\leq\\\|\\bar\{v\}\-\\bar\{v\}^\{\\star\}\\\|\_\{L\_\{2\}\(\\mu\_\{x,t\}\)\}\.For the second term, Cauchy–Schwarz yields\|ℬs,a​\[Eint\]​\(x,t\)\|≤σx\|\\mathcal\{B\}\_\{s,a\}\[E\_\{\\mathrm\{int\}\}\]\(x,t\)\|\\leq\\sigma\_\{x\}\. This proves \([22](https://arxiv.org/html/2605.08253#S5.E22)\)\.

Multiplying \([22](https://arxiv.org/html/2605.08253#S5.E22)\) byλ\\lambdagives the stated bound on\|λ​ℬs,a​\[Cv¯\]​\(x,t\)\|\|\\lambda\\,\\mathcal\{B\}\_\{s,a\}\[C\_\{\\bar\{v\}\}\]\(x,t\)\|\.

### C\.7Euler integration error in theλ\\lambda\-target

The training target uses a numerically integrated successor endpointX^′\\hat\{X\}^\{\\prime\}instead of an idealX′X^\{\\prime\}\. The following proposition isolates how this perturbation propagates into theλ\\lambda\-target on*nonterminal*transitions; at terminalsγ~=0\\tilde\{\\gamma\}=0andλi=0\\lambda\_\{i\}=0in Algorithm[1](https://arxiv.org/html/2605.08253#alg1), so successor endpoint error does not enter\.

###### Proposition C\.1\(Endpoint integration error in theλ\\lambda\-target\)\.

LetX′X^\{\\prime\}be the exact successor endpoint under the target flow map andX^′=X′\+δ\\hat\{X\}^\{\\prime\}=X^\{\\prime\}\+\\deltathe endpoint returned by a numerical ODE solver\. DefineZt′=\(1−t\)​U\+t​X′Z^\{\\prime\}\_\{t\}=\(1\-t\)U\+tX^\{\\prime\}andZ^t′=\(1−t\)​U\+t​X^′\\hat\{Z\}^\{\\prime\}\_\{t\}=\(1\-t\)U\+t\\hat\{X\}^\{\\prime\}\. Assume the successor velocity fieldv¯​\(t,z,s′,a′\)\\bar\{v\}\(t,z,s^\{\\prime\},a^\{\\prime\}\)isLtL\_\{t\}\-Lipschitz inzzfor the relevant range\. For a nonterminal transition, define

uλ:=R\+γ​X′−U\+λ​\(v¯​\(t,Zt′,s′,a′\)−\(X′−U\)\),u^\{\\lambda\}:=R\+\\gamma X^\{\\prime\}\-U\+\\lambda\\bigl\(\\bar\{v\}\(t,Z^\{\\prime\}\_\{t\},s^\{\\prime\},a^\{\\prime\}\)\-\(X^\{\\prime\}\-U\)\\bigr\),and defineu^λ\\hat\{u\}^\{\\lambda\}analogously with\(X^′,Z^t′\)\(\\hat\{X\}^\{\\prime\},\\hat\{Z\}^\{\\prime\}\_\{t\}\)\. Then

‖u^λ−uλ‖≤\(\|γ−λ\|\+λ​Lt​t\)​‖δ‖\.\\\|\\hat\{u\}^\{\\lambda\}\-u^\{\\lambda\}\\\|\\leq\\bigl\(\|\\gamma\-\\lambda\|\+\\lambda L\_\{t\}t\\bigr\)\\\|\\delta\\\|\.In particular, if0≤λ≤γ0\\leq\\lambda\\leq\\gamma,

‖u^λ−uλ‖≤\(γ−λ\+λ​Lt​t\)​‖δ‖\.\\\|\\hat\{u\}^\{\\lambda\}\-u^\{\\lambda\}\\\|\\leq\\bigl\(\\gamma\-\\lambda\+\\lambda L\_\{t\}t\\bigr\)\\\|\\delta\\\|\.

###### Proof\.

Writeuλ=\(R−U\)\+\(γ−λ\)​X′\+λ​v¯​\(t,Zt′,s′,a′\)u^\{\\lambda\}=\(R\-U\)\+\(\\gamma\-\\lambda\)X^\{\\prime\}\+\\lambda\\bar\{v\}\(t,Z^\{\\prime\}\_\{t\},s^\{\\prime\},a^\{\\prime\}\)after expanding the bracket\. Thenu^λ−uλ=\(γ−λ\)​\(X^′−X′\)\+λ​\(v¯​\(t,Z^t′\)−v¯​\(t,Zt′\)\)\\hat\{u\}^\{\\lambda\}\-u^\{\\lambda\}=\(\\gamma\-\\lambda\)\(\\hat\{X\}^\{\\prime\}\-X^\{\\prime\}\)\+\\lambda\(\\bar\{v\}\(t,\\hat\{Z\}^\{\\prime\}\_\{t\}\)\-\\bar\{v\}\(t,Z^\{\\prime\}\_\{t\}\)\)\. Use‖X^′−X′‖=‖δ‖\\\|\\hat\{X\}^\{\\prime\}\-X^\{\\prime\}\\\|=\\\|\\delta\\\|and Lipschitz continuity in the second term:‖v¯​\(t,Z^t′\)−v¯​\(t,Zt′\)‖≤Lt​‖Z^t′−Zt′‖=Lt​t​‖δ‖\\\|\\bar\{v\}\(t,\\hat\{Z\}^\{\\prime\}\_\{t\}\)\-\\bar\{v\}\(t,Z^\{\\prime\}\_\{t\}\)\\\|\\leq L\_\{t\}\\\|\\hat\{Z\}^\{\\prime\}\_\{t\}\-Z^\{\\prime\}\_\{t\}\\\|=L\_\{t\}t\\\|\\delta\\\|\. Combine terms\. ∎

#### Interpretation\.

Atλ=0\\lambda=0the bound scales withγ​‖δ‖\\gamma\\\|\\delta\\\|, reflecting usual sensitivity to the bootstrapped endpoint\. When0≤λ≤γ0\\leq\\lambda\\leq\\gamma, choosingλ\\lambdacloser toγ\\gammareplaces part of the explicitγ​X′\\gamma X^\{\\prime\}dependence with the smoother fieldv¯\\bar\{v\}, reducing the coefficient in front of‖δ‖\\\|\\delta\\\|wheneverLt​t<1L\_\{t\}t<1\(and potentially increasing it in nonsmooth regimes\)\.

### C\.8Closed\-form Gaussian bias and variance\-minimizingλ⋆\\lambda^\{\\star\}

We instantiate Proposition[5\.7](https://arxiv.org/html/2605.08253#S5.Thmtheorem7)in a one\-step linear\-Gaussian MRP, which yields a closed\-form bias and the variance\-minimizingλ⋆​\(t\)\\lambda^\{\\star\}\(t\)\.

###### Proposition C\.2\(Gaussian closed form\)\.

Fixt∈\(0,1\)t\\in\(0,1\),γ∈\(0,1\)\\gamma\\in\(0,1\), and a one\-step MRP with deterministic rewardR=rR=rand Gaussian successor returnZ1′∼𝒩​\(μ,σ2\)Z\_\{1\}^\{\\prime\}\\sim\\mathcal\{N\}\(\\mu,\\sigma^\{2\}\),σ\>0\\sigma\>0\. Let\(U,U′\)\(U,U^\{\\prime\}\)be a bivariate standard Gaussian noise pair \(independent ofZ1′Z\_\{1\}^\{\\prime\}\) withCorr​\(U,U′\)=ρ∈\[−1,1\]\\mathrm\{Corr\}\(U,U^\{\\prime\}\)=\\rho\\in\[\-1,1\], and formZt′:=t​Z1′\+\(1−t\)​U′Z\_\{t\}^\{\\prime\}:=tZ\_\{1\}^\{\\prime\}\+\(1\-t\)U^\{\\prime\},Zt:=t​\(r\+γ​Z1′\)\+\(1−t\)​UZ\_\{t\}:=t\(r\+\\gamma Z\_\{1\}^\{\\prime\}\)\+\(1\-t\)U\. Letv¯⋆​\(⋅,t\)\\bar\{v\}^\{\\star\}\(\\cdot,t\)be the population flow\-matching velocity for the successor interpolant andC:=v¯⋆​\(Zt′,t\)−\(Z1′−U′\)C:=\\bar\{v\}^\{\\star\}\(Z\_\{t\}^\{\\prime\},t\)\-\(Z\_\{1\}^\{\\prime\}\-U^\{\\prime\}\)\. Then𝔼​\[C\]=0\\mathbb\{E\}\[C\]=0and

𝔼​\[C∣Zt=x\]=κ​\(t,γ,σ,ρ\)​\(x−t​\(r\+γ​μ\)\),κ​\(t,γ,σ,ρ\)=t​\(1−t\)​σ2​\(ρ−γ\)\(t2​σ2\+\(1−t\)2\)​\(\(γ​t\)2​σ2\+\(1−t\)2\)\.\\mathbb\{E\}\[C\\mid Z\_\{t\}=x\]=\\kappa\(t,\\gamma,\\sigma,\\rho\)\\,\\bigl\(x\-t\(r\+\\gamma\\mu\)\\bigr\),\\qquad\\kappa\(t,\\gamma,\\sigma,\\rho\)=\\frac\{t\(1\-t\)\\sigma^\{2\}\(\\rho\-\\gamma\)\}\{\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)\\bigl\(\(\\gamma t\)^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\\bigr\)\}\.Moreover, the variance of theλ\\lambda\-targetutλ:=\(r\+γ​Z1′\)−U\+λ​Cu^\{\\lambda\}\_\{t\}:=\(r\+\\gamma Z\_\{1\}^\{\\prime\}\)\-U\+\\lambda Cat fixedttequals

Var​\(utλ∣t\)=1\+γ2​σ2\+σ2Dt​\(λ2−2​λ​\(γ​\(1−t\)\+ρ​t\)\),Dt:=t2​σ2\+\(1−t\)2,\\mathrm\{Var\}\(u^\{\\lambda\}\_\{t\}\\mid t\)=1\+\\gamma^\{2\}\\sigma^\{2\}\+\\frac\{\\sigma^\{2\}\}\{D\_\{t\}\}\\bigl\(\\lambda^\{2\}\-2\\lambda\(\\gamma\(1\-t\)\+\\rho t\)\\bigr\),\\qquad D\_\{t\}:=t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\},so the variance\-minimizing choice isλ⋆​\(t\)=γ​\(1−t\)\+ρ​t\\lambda^\{\\star\}\(t\)=\\gamma\(1\-t\)\+\\rho t, with corresponding reductionVar​\(Y\)−minλ⁡Var​\(utλ∣t\)=σ2Dt​\(γ​\(1−t\)\+ρ​t\)2\\mathrm\{Var\}\(Y\)\-\\min\_\{\\lambda\}\\mathrm\{Var\}\(u^\{\\lambda\}\_\{t\}\\mid t\)=\\frac\{\\sigma^\{2\}\}\{D\_\{t\}\}\(\\gamma\(1\-t\)\+\\rho t\)^\{2\}, increasing inρ\\rhoover the nonnegative\-correlation regimeρ∈\[0,1\]\\rho\\in\[0,1\]fort\>0t\>0\.

#### Derivation\.

WriteZ1′=μ\+σ​WZ\_\{1\}^\{\\prime\}=\\mu\+\\sigma WwithW∼𝒩​\(0,1\)W\\sim\\mathcal\{N\}\(0,1\), and represent\(U,U′\)\(U,U^\{\\prime\}\)asU′=V′U^\{\\prime\}=V^\{\\prime\},U=ρ​V′\+1−ρ2​VU=\\rho V^\{\\prime\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,VwithV,V′∼𝒩​\(0,1\)V,V^\{\\prime\}\\sim\\mathcal\{N\}\(0,1\)independent,W⟂\(V,V′\)W\\perp\(V,V^\{\\prime\}\)\. ForZt′=t​\(μ\+σ​W\)\+\(1−t\)​V′Z\_\{t\}^\{\\prime\}=t\(\\mu\+\\sigma W\)\+\(1\-t\)V^\{\\prime\}, the Gaussian regression formula givesv¯⋆​\(z′,t\)=𝔼​\[Z1′−U′∣Zt′=z′\]=μ\+β​\(t,σ\)​\(z′−t​μ\)\\bar\{v\}^\{\\star\}\(z^\{\\prime\},t\)=\\mathbb\{E\}\[Z\_\{1\}^\{\\prime\}\-U^\{\\prime\}\\mid Z\_\{t\}^\{\\prime\}=z^\{\\prime\}\]=\\mu\+\\beta\(t,\\sigma\)\(z^\{\\prime\}\-t\\mu\), withβ​\(t,σ\)=\(t​σ2−\(1−t\)\)/\(t2​σ2\+\(1−t\)2\)\\beta\(t,\\sigma\)=\(t\\sigma^\{2\}\-\(1\-t\)\)/\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)\. Substitutingz′=Zt′z^\{\\prime\}=Z\_\{t\}^\{\\prime\}and subtractingZ1′−U′Z\_\{1\}^\{\\prime\}\-U^\{\\prime\}yields the linear formC=a​\(t,σ\)​W\+b​\(t,σ\)​V′C=a\(t,\\sigma\)W\+b\(t,\\sigma\)V^\{\\prime\}witha=−σ​\(1−t\)/\(t2​σ2\+\(1−t\)2\)a=\-\\sigma\(1\-t\)/\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\),b=t​σ2/\(t2​σ2\+\(1−t\)2\)b=t\\sigma^\{2\}/\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)\. MeanwhileZt−𝔼​\[Zt\]=γ​t​σ​W\+\(1−t\)​\(ρ​V′\+1−ρ2​V\)Z\_\{t\}\-\\mathbb\{E\}\[Z\_\{t\}\]=\\gamma t\\sigma W\+\(1\-t\)\(\\rho V^\{\\prime\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,V\), and joint Gaussianity yields𝔼​\[C∣Zt=x\]=Cov​\(C,Zt\)/Var​\(Zt\)​\(x−𝔼​\[Zt\]\)\\mathbb\{E\}\[C\\mid Z\_\{t\}=x\]=\\mathrm\{Cov\}\(C,Z\_\{t\}\)/\\mathrm\{Var\}\(Z\_\{t\}\)\\,\(x\-\\mathbb\{E\}\[Z\_\{t\}\]\)\. A direct covariance calculation givesCov​\(C,Zt\)=t​\(1−t\)​σ2​\(ρ−γ\)/\(t2​σ2\+\(1−t\)2\)\\mathrm\{Cov\}\(C,Z\_\{t\}\)=t\(1\-t\)\\sigma^\{2\}\(\\rho\-\\gamma\)/\(t^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}\)andVar​\(Zt\)=\(γ​t\)2​σ2\+\(1−t\)2\\mathrm\{Var\}\(Z\_\{t\}\)=\(\\gamma t\)^\{2\}\\sigma^\{2\}\+\(1\-t\)^\{2\}, which yieldsκ​\(t,γ,σ,ρ\)\\kappa\(t,\\gamma,\\sigma,\\rho\)\. The variance formula andλ⋆​\(t\)\\lambda^\{\\star\}\(t\)follow from completing the square inλ\\lambda\.

#### Consequence \(shared\-noise vs\. independent paths\)\.

Settingρ=1\\rho=1\(PCBF’s shared noise\) givesκ=O​\(\(1−γ\)​\(1−t\)\)→0\\kappa=O\(\(1\-\\gamma\)\(1\-t\)\)\\to 0as eitherγ→1\\gamma\\to 1ort→1t\\to 1, so the bias contracts at the endpoints and for sparse\-reward, near\-undiscounted regimes\. Settingρ=0\\rho=0\(independent noise\) leaves anO​\(γ\)O\(\\gamma\)multiplicative factor that does not vanish, illustrating quantitatively why path coupling is essential for the variance\-bias trade\-off\.

### C\.9PCBF endpoint Bellman consistency

For the PCBF path in Eq\. \([9](https://arxiv.org/html/2605.08253#S4.E9)\), the endpoint satisfies

Z1s=R\+γ​X′,Z\_\{1\}^\{s\}=R\+\\gamma X^\{\\prime\},pointwise\. Therefore, ifX′∼ηs′,a′X^\{\\prime\}\\sim\\eta\_\{s^\{\\prime\},a^\{\\prime\}\}, then

ℒ​\(Z1s∣s,a\)=ℒ​\(R\+γ​X′∣s,a\)=\(Tπ​η\)s,a,\\mathcal\{L\}\(Z\_\{1\}^\{s\}\\mid s,a\)=\\mathcal\{L\}\(R\+\\gamma X^\{\\prime\}\\mid s,a\)=\(T^\{\\pi\}\\eta\)\_\{s,a\},which is exactly the distributional Bellman update at the endpoint\.

## Appendix DImplementation Details

#### Flow matching\.

We implement PCBF using apath\-coupled flow\-matchingformulation with linear interpolation paths and uniform time sampling, as described in Sections[3](https://arxiv.org/html/2605.08253#S3.SS0.SSS0.Px1)and[4](https://arxiv.org/html/2605.08253#S4)\. The return distribution is modeled via a time\-dependent velocity field and trained using a mean squared error objective defined by the path\-coupled Bellman targets\. We numerically solve the resulting ODE using the explicit Euler method with10 integration stepsacross all experiments\. For simplicity and stability, we do not use additional time embeddings for the flow time variable\. Ablation results for different values of theλ\\lambdaparameter, which controls the strength of the path\-coupled correction, are reported in Appendix[F](https://arxiv.org/html/2605.08253#A6)\.

#### Value learning\.

PCBF learns return distributions rather than scalar value functions\. We maintain a target velocity network, updated using Polyak averaging, to generate successor return samples for the path\-coupled Bellman objective\. Theλ\\lambda\-parameterized target interpolates between a sample\-based Bellman\-consistent update and a variance\-reduced correction derived from successor flow predictions\. All value learning components are trained using squared regression losses on the induced velocity targets\.

#### Network architectures\.

For all state\-based tasks, we use multilayer perceptrons with four hidden layers of width 512 and GELU activations to parameterize the velocity fields\. Layer normalization is applied to stabilize training\. For pixel\-based environments, we employ a lightweight convolutional encoder adapted from the IMPALA architecture to extract visual features before feeding them into the flow model\.

#### Image processing\.

For pixel\-based environments, observations consist of raw RGB images of size64×64×364\\times 64\\times 3\. We apply a random\-shift augmentation with probability 0\.5 and use frame stacking with three consecutive frames\. These preprocessing steps are shared across all pixel\-based methods and are critical for stable learning in visually complex environments\.

#### Training and evaluation\.

We train PCBF for 1M gradient steps on state\-based OGBench tasks and 500K gradient steps on pixel\-based OGBench and D4RL tasks\. Evaluation is performed every 100K steps using 50 episodes\. For OGBench tasks, we report the average success rate over the final three evaluation checkpoints, following the official evaluation protocol\. For D4RL Adroit tasks, we report normalized returns at the final training checkpoint\.

#### Policy extraction\.

At deployment, we follow the standard candidate\-action protocol used in our FQL\-style codebase\(Parket al\.,[2025c](https://arxiv.org/html/2605.08253#bib.bib7)\): a behavior\-cloned proposal policyπβ\\pi\_\{\\beta\}samplesKKcandidate actions\{ak\}k=1K\\\{a\_\{k\}\\\}\_\{k=1\}^\{K\}fromπβ\(⋅∣s\)\\pi\_\{\\beta\}\(\\cdot\\mid s\)\. Each candidate is scored by the mean terminal return under the learned flow,

Q^θ​\(s,a\)=1M​∑m=1Mψθ1​\(Um∣s,a\),Um∼𝒩​\(0,I\),\\hat\{Q\}\_\{\\theta\}\(s,a\)=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\psi\_\{\\theta\}^\{\\,1\}\(U\_\{m\}\\mid s,a\),\\qquad U\_\{m\}\\sim\\mathcal\{N\}\(0,I\),\(27\)and we executearg⁡maxak⁡Q^θ​\(s,ak\)\\arg\\max\_\{a\_\{k\}\}\\hat\{Q\}\_\{\\theta\}\(s,a\_\{k\}\)\. Unless stated otherwise we useK=16K\{=\}16candidates, matching the “rejection sampling candidates” entry in Table[4](https://arxiv.org/html/2605.08253#A8.T4)\.

#### Hyperparameters\.

Theλ\\lambdaparameter, which controls the strength of the path\-coupled correction, is the primary method\-specific hyperparameter in PCBF\. We tuneλ\\lambdaat the domain level using the task marked with∗\*and apply the selected value across all tasks within the same domain\. A detailed summary of hyperparameter choices for all methods and domains is provided in Tables[4](https://arxiv.org/html/2605.08253#A8.T4)and Table[5](https://arxiv.org/html/2605.08253#A8.T5)\.

## Appendix EAdditional benchmark details

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/ogbench.png)Figure 6:OGBench TasksTable 2:Training cost oncube\-double\-play\-task1\(single A100,10610^\{6\}steps\)\. Wall\-clock is relative to IQL\.
## Appendix FAblation Study

Figure[7](https://arxiv.org/html/2605.08253#A6.F7)studies the effect ofλ\\lambdain PCBF across representative OGBench and D4RL tasks, with all other hyperparameters held fixed\. As discussed in Section[4](https://arxiv.org/html/2605.08253#S4),λ\\lambdainterpolates between unbiased sample\-based targets \(λ=0\\lambda=0\) and variance\-reduced corrections \(λ\>0\\lambda\>0\), and performance is highly sensitive to its choice\. In several state\-based OGBench tasks \(e\.g\., scene\-play\-task2, cube\-double\-task2, puzzle\-4×4\-task4\) moderateλ\\lambdavalues yield the best success rates, whereas for some visual tasks \(e\.g\., visual\-antmaze\-task1\)λ=0\\lambda=0performed best\. For D4RL Adroit tasks \(e\.g\., hammer\-cloned and hammer\-expert\) differentλ\\lambdavalues can improve normalized returns\. This task\-dependent variability motivates tuningλ\\lambdaat the domain level, as reported in our main experiments\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/lambda_ablation.png)Figure 7:Ablation study of theλ\\lambdaparameter in PCBF\.Red stars denote the best\-performingλ\\lambdaon representative OGBench and D4RL tasks\.
## Appendix GFull Experiment Results

Table 3:Full Offline RL Results\.We report performance on the 38 tasks on OGBench and D4RL\. The symbol\(∗\)\(\*\)indicates the task used for hyperparameter tuning within each domain\. Results are averaged over 8 random seeds \(4 seeds for pixel\-based tasks\)\. Bold numbers denote values that are within95%95\\%of the best performing method on each task\.AlgorithmsTaskIQNCODACFloQFQLIQLValue FlowsPCBF \(Ours\)cube\-double\-play\-singletask\-task1\-v070±1470\\pm 1480±1180\\pm 1150±2450\\pm 2461±961\\pm 927±527\\pm 5𝟗𝟕±𝟏\\mathbf\{97\\pm 1\}92±392\\pm 3cube\-double\-play\-singletask\-task2\-v0 \(∗\*\)24±924\\pm 963±463\\pm 472±1572\\pm 1536±636\\pm 61±11\\pm 1𝟕𝟔±𝟕\\mathbf\{76\\pm 7\}𝟕𝟒±𝟕\\mathbf\{74\\pm 7\}cube\-double\-play\-singletask\-task3\-v025±625\\pm 666±966\\pm 957±1457\\pm 1422±522\\pm 50±00\\pm 073±473\\pm 4𝟖𝟏±𝟖\\mathbf\{81\\pm 8\}cube\-double\-play\-singletask\-task4\-v010±110\\pm 113±213\\pm 28±48\\pm 45±25\\pm 20±00\\pm 0𝟑𝟎±𝟓\\mathbf\{30\\pm 5\}22±522\\pm 5cube\-double\-play\-singletask\-task5\-v081±881\\pm 882±482\\pm 450±1150\\pm 1119±1019\\pm 104±34\\pm 369±569\\pm 5𝟖𝟒±𝟑\\mathbf\{84\\pm 3\}scene\-play\-singletask\-task1\-v0𝟏𝟎𝟎±𝟎\\mathbf\{100\\pm 0\}99±099\\pm 0100±1100\\pm 1𝟏𝟎𝟎±𝟎\\mathbf\{100\\pm 0\}94±394\\pm 399±099\\pm 0𝟏𝟎𝟎±𝟎\\mathbf\{100\\pm 0\}scene\-play\-singletask\-task2\-v0 \(∗\*\)1±01\\pm 085±485\\pm 483±1083\\pm 1076±976\\pm 912±312\\pm 3𝟗𝟕±𝟏\\mathbf\{97\\pm 1\}57±1357\\pm 13scene\-play\-singletask\-task3\-v094±294\\pm 290±390\\pm 398±298\\pm 2𝟗𝟖±𝟏\\mathbf\{98\\pm 1\}32±732\\pm 794±294\\pm 2𝟗𝟖±𝟐\\mathbf\{98\\pm 2\}scene\-play\-singletask\-task4\-v03±13\\pm 10±00\\pm 09±79\\pm 75±15\\pm 10±10\\pm 17±177\\pm 17𝟏𝟐±𝟑\\mathbf\{12\\pm 3\}scene\-play\-singletask\-task5\-v00±00\\pm 00±00\\pm 00±00\\pm 00±00\\pm 00±00\\pm 00±00\\pm 0𝟐±𝟏\\mathbf\{2\\pm 1\}puzzle\-4x4\-play\-singletask\-task1\-v0𝟒𝟏±𝟐\\mathbf\{41\\pm 2\}37±3237\\pm 3247±747\\pm 734±834\\pm 812±212\\pm 236±336\\pm 338±638\\pm 6puzzle\-4x4\-play\-singletask\-task2\-v012±412\\pm 410±1010\\pm 1021±621\\pm 616±516\\pm 57±47\\pm 4𝟐𝟕±𝟓\\mathbf\{27\\pm 5\}23±523\\pm 5puzzle\-4x4\-play\-singletask\-task3\-v0𝟒𝟓±𝟕\\mathbf\{45\\pm 7\}33±2933\\pm 2936±536\\pm 518±518\\pm 59±39\\pm 330±430\\pm 440±440\\pm 4puzzle\-4x4\-play\-singletask\-task4\-v0 \(∗\*\)23±223\\pm 212±1012\\pm 1019±519\\pm 511±311\\pm 35±25\\pm 2𝟐𝟖±𝟓\\mathbf\{28\\pm 5\}𝟐𝟖±𝟒\\mathbf\{28\\pm 4\}puzzle\-4x4\-play\-singletask\-task5\-v016±616\\pm 610±810\\pm 816±716\\pm 77±37\\pm 34±14\\pm 113±213\\pm 2𝟐𝟑±𝟑\\mathbf\{23\\pm 3\}cube\-triple\-play\-singletask\-task1\-v0 \(∗\*\)29±229\\pm 29±59\\pm 5−\-20±620\\pm 64±44\\pm 4𝟓𝟗±𝟏𝟐\\mathbf\{59\\pm 12\}18±418\\pm 4cube\-triple\-play\-singletask\-task2\-v00±00\\pm 0𝟏±𝟎\\mathbf\{1\\pm 0\}−\-𝟏±𝟐\\mathbf\{1\\pm 2\}0±00\\pm 00±00\\pm 00±10\\pm 1cube\-triple\-play\-singletask\-task3\-v01±01\\pm 00±00\\pm 0−\-0±00\\pm 00±00\\pm 0𝟕±𝟑\\mathbf\{7\\pm 3\}1±11\\pm 1cube\-triple\-play\-singletask\-task4\-v0𝟎±𝟎\\mathbf\{0\\pm 0\}𝟎±𝟎\\mathbf\{0\\pm 0\}−\-𝟎±𝟎\\mathbf\{0\\pm 0\}𝟎±𝟎\\mathbf\{0\\pm 0\}𝟎±𝟎\\mathbf\{0\\pm 0\}𝟎±𝟎\\mathbf\{0\\pm 0\}cube\-triple\-play\-singletask\-task5\-v00±00\\pm 00±00\\pm 0−\-0±00\\pm 01±11\\pm 1𝟐±𝟏\\mathbf\{2\\pm 1\}1±11\\pm 1pen\-cloned\-v180±1180\\pm 1176±276\\pm 2−\-74±1174\\pm 11𝟖𝟑\\mathbf\{83\}73±573\\pm 578±578\\pm 5pen\-expert\-v1118±19118\\pm 19136±2136\\pm 2−\-𝟏𝟒𝟐±𝟔\\mathbf\{142\\pm 6\}128128117±3117\\pm 3131±5131\\pm 5door\-cloned\-v10±00\\pm 00±00\\pm 0−\-2±12\\pm 1𝟑\\mathbf\{3\}0±00\\pm 01±01\\pm 0door\-expert\-v1105±0105\\pm 0104±0104\\pm 0−\-104±1104\\pm 1𝟏𝟎𝟕\\mathbf\{107\}104±1104\\pm 1𝟏𝟎𝟔±𝟎\\mathbf\{106\\pm 0\}hammer\-cloned\-v10±00\\pm 06±06\\pm 0−\-𝟏𝟏±𝟗\\mathbf\{11\\pm 9\}221±01\\pm 02±12\\pm 1hammer\-expert\-v1121±7121\\pm 7126±1126\\pm 1−\-125±3125\\pm 3𝟏𝟐𝟗\\mathbf\{129\}125±5125\\pm 5𝟏𝟐𝟔±𝟐\\mathbf\{126\\pm 2\}relocate\-cloned\-v10±00\\pm 00±00\\pm 0−\-0±00\\pm 0𝟐\\mathbf\{2\}0±00\\pm 00±00\\pm 0relocate\-expert\-v1103±0103\\pm 0103±2103\\pm 2−\-107±1107\\pm 1106106102±2102\\pm 2𝟏𝟎𝟗±𝟏\\mathbf\{109\\pm 1\}visual\-antmaze\-teleport\-navigate\-singletask\-task1\-v0 \(∗\*\)2±12\\pm 1−\-−\-2±12\\pm 15±25\\pm 2𝟏𝟎±𝟒\\mathbf\{10\\pm 4\}8±28\\pm 2visual\-antmaze\-teleport\-navigate\-singletask\-task2\-v07±37\\pm 3−\-−\-6±16\\pm 110±210\\pm 217±517\\pm 5𝟏𝟗±𝟒\\mathbf\{19\\pm 4\}visual\-antmaze\-teleport\-navigate\-singletask\-task3\-v06±46\\pm 4−\-−\-9±49\\pm 47±77\\pm 716±316\\pm 3𝟏𝟖±𝟒\\mathbf\{18\\pm 4\}visual\-antmaze\-teleport\-navigate\-singletask\-task4\-v04±24\\pm 2−\-−\-9±19\\pm 14±64\\pm 616±516\\pm 5𝟏𝟖±𝟓\\mathbf\{18\\pm 5\}visual\-antmaze\-teleport\-navigate\-singletask\-task5\-v02±12\\pm 1−\-−\-1±11\\pm 12±12\\pm 1𝟖±𝟐\\mathbf\{8\\pm 2\}𝟖±𝟑\\mathbf\{8\\pm 3\}visual\-cube\-double\-play\-singletask\-task1\-v0 \(∗\*\)4±14\\pm 1−\-−\-23±423\\pm 434±2334\\pm 23𝟑𝟓±𝟐\\mathbf\{35\\pm 2\}10±310\\pm 3visual\-cube\-double\-play\-singletask\-task2\-v00±00\\pm 0−\-−\-0±00\\pm 03±13\\pm 1𝟒±𝟐\\mathbf\{4\\pm 2\}0±00\\pm 0visual\-cube\-double\-play\-singletask\-task3\-v00±00\\pm 0−\-−\-0±00\\pm 07±47\\pm 4𝟏𝟏±𝟐\\mathbf\{11\\pm 2\}0±00\\pm 0visual\-cube\-double\-play\-singletask\-task4\-v00±00\\pm 0−\-−\-𝟒±𝟐\\mathbf\{4\\pm 2\}2±12\\pm 12±12\\pm 10±00\\pm 0visual\-cube\-double\-play\-singletask\-task5\-v01±11\\pm 1−\-−\-4±14\\pm 111±211\\pm 2𝟏𝟑±𝟑\\mathbf\{13\\pm 3\}3±13\\pm 1

## Appendix HExperiment Details

We implement PCBF and all baselines using JAX\(Bradburyet al\.,[2018](https://arxiv.org/html/2605.08253#bib.bib19)\), with implementations adapted from the FQL codebase\(Parket al\.,[2025c](https://arxiv.org/html/2605.08253#bib.bib7)\)\.

### H\.1Environments and Datasets

We evaluate PCBF on offline reinforcement learning benchmarks from OGBench\(Parket al\.,[2025a](https://arxiv.org/html/2605.08253#bib.bib10)\)and D4RL\(Fuet al\.,[2020](https://arxiv.org/html/2605.08253#bib.bib11)\), which provide diverse tasks with long\-horizon dependencies, semi\-sparse rewards, and multimodal return distributions\.

#### OGBench\(Parket al\.,[2025a](https://arxiv.org/html/2605.08253#bib.bib10)\)\.

OGBench is originally designed for offline goal\-conditioned reinforcement learning\. Following prior work, we adopt its single\-task variants \(“\-singletask”\) to benchmark standard reward\-maximizing offline RL methods\. In each environment, five predefined evaluation goals are provided, yielding five corresponding single\-task variants \(from \-singletask\-task1 to \-singletask\-task5\)\. Given a fixed evaluation goal, transitions in the dataset are labeled with a semi\-sparse reward function that reflects task progress\. For state\-based and pixel\-based OGBench, we use the following environments:

- •State\-based datasets - –cube\-double\-play\-v0 - –cube\-triple\-play\-v0 - –scene\-play\-v0 - –puzzle\-4x4\-play\-v0
- •Pixel\-based datasets - –visual\-cube\-double\-play\-v0 - –visual\-antmaze\-teleport\-navigate\-v0

We choose these environments to cover a diverse range of challenges in long\-horizon robotic manipulation and compositional reasoning, following the manipulation suite provided by OGBench\(Parket al\.,[2025a](https://arxiv.org/html/2605.08253#bib.bib10)\)\. The cube, scene, and puzzle environments are robot\-arm manipulation tasks built on a 6\-DoF robotic arm, where agents must interact with multiple objects over extended horizons\. The cube environments involve multi\-stage pick\-and\-place manipulation of colored blocks and require arranging objects into target configurations through sequences of coordinated actions\. The scene\-play environment further increases complexity by involving multiple interactive objects—such as drawers, locks, and movable blocks—where solving a task often requires executing a specific sequence of dependent subtasks, with the longest tasks involving up to eight atomic operations\. The puzzle\-4×4 environment corresponds to robotic variants of the Lights Out puzzle and are designed to test combinatorial generalization, as agents must reason over button configurations and long sequences of interactions to reach the desired goal state\. All state\-based manipulation tasks follow the standard play\-style dataset setting in OGBench, where datasets are collected from scripted policies performing random interactions rather than goal\-directed demonstrations\. As a result, the datasets exhibit high suboptimality and require agents to effectively stitch partial behaviors into coherent long\-horizon solutions\. We evaluate these environments using their single\-task variants, where each environment provides five predefined evaluation goals and rewards are semi\-sparse, reflecting incremental progress toward task completion\. In addition to state\-based environments, we include pixel\-based OGBench tasks that require learning directly from raw RGB observations of size64×64×364\\times 64\\times 3\. These visual variants introduce partial observability and additional perceptual complexity, further challenging return modeling and policy learning\. Due to the significantly higher computational cost of pixel\-based training, we evaluate a representative subset of visual tasks in our experiments\. For all OGBench tasks, we use the standard dataset types \(play for manipulation and navigate for navigation\) and report binary task success rates \(in percentage\), following the original OGBench evaluation protocol\.

#### D4RL\(Fuet al\.,[2020](https://arxiv.org/html/2605.08253#bib.bib11)\)\.

We further evaluate PCBF on tasks from theD4RLbenchmark, which is widely used for studying offline reinforcement learning\. In our experiments, the Adroit manipulation tasks require dexterous control with a high\-dimensional 24\-DoF action space\. These tasks involve contact\-rich interactions and long\-horizon dependencies, providing a complementary evaluation setting to OGBench manipulation environments\. Following the standard D4RL evaluation protocol, performance on Adroit tasks is reported using normalized returns\. We use the following 8 adroit tasks:

- •D4RL Adroit tasks - –pen\-cloned\-v1 - –pen\-expert\-v1 - –door\-cloned\-v1 - –door\-expert\-v1 - –hammer\-cloned\-v1 - –hammer\-expert\-v1 - –relocate\-cloned\-v1 - –relocate\-expert\-v1

### H\.2Methods and Hyperparameters

We compare PCBF against a set of strong offline reinforcement learning baselines, covering both scalar value\-based and distributional methods\. All methods use the same network architectures and discount factors\.

- •IQL\(Kostrikovet al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib18)\): Implicit Q\-Learning is a scalar value\-based offline RL method that learns conservative Q\-functions via expectile regression and extracts actions within the support of the offline dataset\.
- •FQL\(Parket al\.,[2025c](https://arxiv.org/html/2605.08253#bib.bib7)\): Flow Q\-Learning is a scalar value\-based method that employs a flow\-based policy to maximize Q\-values learned through standard temporal\-difference updates, together with behavioral regularization\.
- •IQN\(Dabneyet al\.,[2018a](https://arxiv.org/html/2605.08253#bib.bib16)\): Implicit Quantile Networks approximate the return distribution by predicting quantile values at randomly sampled quantile fractions using quantile regression\.
- •CODAC\(Maet al\.,[2021](https://arxiv.org/html/2605.08253#bib.bib17)\): CODAC extends distributional RL with conservative regularization to mitigate overestimation, combining quantile\-based critics with offline constraints\.
- •Value Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\): Value Flows is a flow\-based distributional RL method that models the full return distribution using flow matching and learns value functions via continuous density transformations\. Value Flows employs a weighted sampling procedure for timettduring training, whereas PCBF uses uniform time sampling\.
- •PCBF \(ours\): Path\-Coupled Bellman Flows model return distributions with flow matching using source\-consistent Bellman\-coupled paths, shared\-noise coupling, and theλ\\lambda\-target of Section[4](https://arxiv.org/html/2605.08253#S4)\. Details are in Appendix[D](https://arxiv.org/html/2605.08253#A4); hyperparameters are in Tables[4](https://arxiv.org/html/2605.08253#A8.T4)–[5](https://arxiv.org/html/2605.08253#A8.T5)\.

Table 4:Common hyperparameters for PCBF and baselines\.Table 5:Hyperparameters for PCBF and baselines\.Hyperparameters are tuned per domain on the task marked with∗\*in the OGBench benchmarks\. Entries marked with−\-indicate not applicable or inherited settings\(Parket al\.,[2025b](https://arxiv.org/html/2605.08253#bib.bib9)\)\. The discount factorγ\\gammais shared across methods within each domain\.DiscountIQLFQLIQNCODACPCBFDomain or taskγ\\gammaα\\alphaα\\alphaκ\\kappaα\\alphaα\\alphaλ\\lambdaλ\\lambdacube\-double\-play0\.9950\.31000\.95300130\.4cube\-triple\-play0\.995101000\.95100030\.030\.995puzzle\-4x4\-play0\.9933000\.95100031000\.2scene\-play0\.99101000\.9510010\.30\.2pen\-cloned0\.99−\-100000\.810000310\.7pen\-expert0\.99−\-30000\.81000030\.010\.7door\-cloned0\.99−\-300000\.930000100\.30\.5door\-expert0\.99−\-300000\.910000100\.30\.99hammer\-cloned0\.99−\-100000\.81000030\.30\.8hammer\-expert0\.99−\-300000\.8100001010\.9relocate\-cloned0\.99−\-300000\.93000030\.010\.99relocate\-expert0\.99−\-300000\.91000030\.10\.99visual\-antmaze\-teleport\-navigate0\.991100−\-−\-0\.30\.030visual\-cube\-double\-play0\.9950\.3100−\-−\-10\.30\.9

## Appendix IAdditional Results and Analysis

### I\.1Toy Environment Details\.

To directly evaluate whether PCBF learns accurate return distributions, we study a set of analytically tractable toy environments with known return structure\. In contrast to large\-scale control benchmarks such as OGBench, which necessarily involve action selection and policy optimization, these toy environments admit return laws that can be derived exactly or characterized in closed form\. This allows us to access distributional accuracy unambiguously, rather than inferring it indirectly from control performance\.

The environments are chosen to expose different forms of stochasticity in the Bellman recursion while remaining simple enough to permit precise interpretation\. We consider:

- •\(i\)Solitaire Diceenvironment, which produces long\-tailed and high\-variance returns through stochastic termination while remaining free of action\-dependent dynamics\.
- •\(ii\)Single\-state Bernoulli MRPwith a closed\-form return distribution, which serves as a sanity check for uniformly distributed over a bounded interval, enabling direct comparison between learned return distributions and the analytic ground truth\.
- •\(iii\)Discrete Monte Carlo Markov chainwith finite\-horizon returns induced by nearest\-neighbor dynamics, which enables controlled analysis of Bellman target variability as a function of effective horizon length\.

Together, these environments provide a principled testbed for analyzing how theλ\\lambda\-parameterized control variate in PCBF influences training behavior and return\-distribution accuracy as horizon length and return variance increase, and for directly comparing the fidelity of learned return distributions across PCBF and baseline distributional RL methods against known ground\-truth laws\.

#### Solitaire Dice\.

We first consider a stochastic termination process adapted from\(Bellemareet al\.,[2023](https://arxiv.org/html/2605.08253#bib.bib20)\), which we refer to as the*Solitaire Dice*environment\. The environment consists of repeatedly rolling a single fair six\-sided die\. If a11is rolled, the episode terminates immediately; otherwise, the agent receives a reward of11and the game continues\. The environment contains no actions, and the transition dynamics are entirely driven by the die outcome\.

LetTTdenote the number of non\-terminating rolls before the first terminating outcome11\. The undiscounted return is given by

G=∑t=0∞Rt=1\+1\+⋯\+1⏟T​times,G\\;=\\;\\sum\_\{t=0\}^\{\\infty\}R\_\{t\}\\;=\\;\\underbrace\{1\+1\+\\cdots\+1\}\_\{T\\text\{ times\}\},which is an integer\-valued random variable taking values inℕ=\{0,1,2,…\}\\mathbb\{N\}=\\\{0,1,2,\\dots\\\}\. Since each roll independently terminates the episode with probability16\\tfrac\{1\}\{6\}, the return follows a geometric distribution,

ℙ​\(G=k\)=16​\(56\)k,k∈ℕ,\\mathbb\{P\}\(G=k\)\\;=\\;\\tfrac\{1\}\{6\}\\left\(\\tfrac\{5\}\{6\}\\right\)^\{k\},\\qquad k\\in\\mathbb\{N\},corresponding to observingkknon\-terminating outcomes before the first terminating roll\.

When a discount factorγ∈\(0,1\)\\gamma\\in\(0,1\)is introduced, the return becomes

Gγ=∑t=0∞γt​Rt,G\_\{\\gamma\}\\;=\\;\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{t\},Conditioned onT=kT=k, this sum evaluates to the partial geometric series

Gγ=∑t=0k−1γt=1−γk1−γ,G\_\{\\gamma\}\\;=\\;\\sum\_\{t=0\}^\{k\-1\}\\gamma^\{t\}\\;=\\;\\frac\{1\-\\gamma^\{k\}\}\{1\-\\gamma\},Importantly, discounting changes the*support*of the return distribution but not the associated probabilities: for allk≥0k\\geq 0,

ℙ​\(Gγ=1−γk1−γ\)=16​\(56\)k\.\\mathbb\{P\}\\\!\\left\(G\_\{\\gamma\}=\\frac\{1\-\\gamma^\{k\}\}\{1\-\\gamma\}\\right\)\\;=\\;\\tfrac\{1\}\{6\}\\left\(\\tfrac\{5\}\{6\}\\right\)^\{k\}\.
This environment induces a highly skewed, long\-tailed return distribution driven entirely by stochastic termination\. As such, it provides a controlled setting for studying Bellman target variance arising from random episode lengths and for evaluating the stability of distributional learning methods under heavy\-tailed returns\.

#### Bernoulli\.

We next consider a single\-state, single\-action Markov decision process with purely stochastic rewards, adapted from the classical example in\(Bellemareet al\.,[2023](https://arxiv.org/html/2605.08253#bib.bib20)\)\. The state space is𝒳=\{x\}\\mathcal\{X\}=\\\{x\\\}and the action space is𝒜=\{a\}\\mathcal\{A\}=\\\{a\\\}\. The initial distribution isξ0=δx\\xi\_\{0\}=\\delta\_\{x\}, and the transition kernel is deterministic,

P​\(x∣x,a\)=1,P\(x\\mid x,a\)=1,so the system remains in the same state at all times\. The reward process\{Rt\}t≥0\\\{R\_\{t\}\\\}\_\{t\\geq 0\}is i\.i\.d\. with

ℙ​\(Rt=1∣x,a\)=ℙ​\(Rt=0∣x,a\)=12,\\mathbb\{P\}\(R\_\{t\}=1\\mid x,a\)=\\mathbb\{P\}\(R\_\{t\}=0\\mid x,a\)=\\tfrac\{1\}\{2\},
We fix the discount factor toγ=12\\gamma=\\tfrac\{1\}\{2\}\. The discounted return is therefore

G=∑t=0∞γt​Rt=R0\+12​R1\+14​R2\+⋯,G\\;=\\;\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\_\{t\}\\;=\\;R\_\{0\}\+\\tfrac\{1\}\{2\}R\_\{1\}\+\\tfrac\{1\}\{4\}R\_\{2\}\+\\cdots,
A key observation is thatGGadmits a binary expansion

G=R0\.R1​R2​…,G\\;=\\;R\_\{0\}\.R\_\{1\}R\_\{2\}\\ldots\\quad,where each digit is an independent Bernoulli random variable\. As a consequence, the support ofGGis the interval\[0,2\]\[0,2\], with0corresponding to the infinite sequence of zeros and22corresponding to the infinite sequence of ones\. Moreover, for any dyadic interval\[a,b\]⊂\[0,2\]\[a,b\]\\subset\[0,2\]whose endpoints admit finite binary expansions, the probability mass satisfies

ℙ​\(G∈\[a,b\]\)=b−a2,\\mathbb\{P\}\(G\\in\[a,b\]\)=\\tfrac\{b\-a\}\{2\},
This property uniquely characterizes the uniform distribution on\[0,2\]\[0,2\], implying that the return distribution is exactly

G∼Unif​\[0,2\]\.G\\sim\\mathrm\{Unif\}\[0,2\]\.
This environment provides a rare example of an MDP with a closed\-form return distribution despite infinite\-horizon bootstrapping\. Because the dynamics are trivial and all stochasticity arises solely from the reward sequence, it serves as a clean sanity check for distributional Bellman consistency and for analyzing variance\-reduction effects in PCBF without confounding effects from state transitions, exploration, or function approximation\.

#### Discrete Monte Carlo Chain\.

We next consider a finite\-state Markov reward process adapted from the discrete nearest\-neighbor Markov chains studied inCheng and Weare \([2024](https://arxiv.org/html/2605.08253#bib.bib21)\)\. The environment consists of a one\-dimensional Markov chain on a discrete state space𝒳=\{0,1,…,n−1\}\\mathcal\{X\}=\\\{0,1,\\dots,n\-1\\\}with no actions\. States0andn−1n\-1are absorbing terminal states, and episodes are initialized from a non\-terminal state in\{1,…,n−2\}\\\{1,\\dots,n\-2\\\}\.

The transition dynamics follow a nearest\-neighbor structure with state\-dependent probabilities\. Letp:𝒳→ℝ\>0p:\\mathcal\{X\}\\to\\mathbb\{R\}\_\{\>0\}be defined by

p​\(i\)∝exp⁡\(n−14​π​cos⁡\(4​π​\(i−1\)n−1\)\),p\(i\)\\;\\propto\\;\\exp\\\!\\left\(\\frac\{n\-1\}\{4\\pi\}\\cos\\\!\\left\(\\frac\{4\\pi\(i\-1\)\}\{n\-1\}\\right\)\\right\),and define the transition kernel fori∈\{1,…,n−2\}i\\in\\\{1,\\dots,n\-2\\\}by

P​\(i,i±1\)∝p​\(i±1\)p​\(i\)\+p​\(i±1\),P​\(i,i\)=1−P​\(i,i−1\)−P​\(i,i\+1\),P\(i,i\\pm 1\)\\;\\propto\\;\\frac\{p\(i\\pm 1\)\}\{p\(i\)\+p\(i\\pm 1\)\},\\qquad P\(i,i\)=1\-P\(i,i\-1\)\-P\(i,i\+1\),withP​\(0,0\)=P​\(n−1,n−1\)=1P\(0,0\)=P\(n\-1,n\-1\)=1\(absorbing boundaries\)\. This construction induces a multi\-well potential landscape in which local transitions remain stable while global escape times grow withnn\.

The reward function is deterministic: the agent receives a reward of11at each non\-terminal transition and0upon entering a terminal state\. The resulting return is

G=∑t=0T−11\.G\\;=\\;\\sum\_\{t=0\}^\{T\-1\}1\.whereTTis the \(finite\) first hitting time of the terminal set\. By construction, the return distribution is fully determined by the transition dynamics and the episode horizon\.

This environment provides a controlled finite\-horizon testbed in which Bellman targets accumulate stochasticity through repeated transitions rather than unbounded reward support\. It enables direct examination of how Bellman target variance scales with effective horizon length, and facilitates precise comparison of learned return distributions across PCBF and baseline methods in a setting where the underlying return law is exactly defined\.

### I\.2Internal Analysis: Variance Reduction and Stability

#### Variance reduction during training\.

Figure[8](https://arxiv.org/html/2605.08253#A9.F8)reports the within\-run standard deviation of the Bellman velocity regression loss\. Asλ\\lambdaincreases, this variability consistently decreases\. This demonstrates that theλ\\lambda\-parameterized control variate effectively dampens the noise in gradient updates caused by stochastic bootstrapping, validating its role as a variance\-reduction mechanism\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/variance_reduction.png)Figure 8:Variance reduction viaλ\\lambda\-parameterized control variates\.Largerλ\\lambdayields smoother loss trajectories \(lower standard deviation\), demonstrating effective variance reduction in Bellman targets\.
#### Bias–variance trade\-off\.

While increasingλ\\lambdareduces optimization variance, Figure[9](https://arxiv.org/html/2605.08253#A9.F9)illustrates the resulting trade\-off in distributional accuracy\. In the Bernoulli environment \(γ=0\.5\\gamma=0\.5\), the Wasserstein distance remains low forλ≤0\.3\\lambda\\leq 0\.3but increases asλ→γ\\lambda\\to\\gamma\. In the more challenging Discrete Monte Carlo environment \(γ=0\.95\\gamma=0\.95\), we observe a ”sweet spot” at moderateλ\\lambda, where the benefit of variance reduction outweighs the bias, yielding the lowest approximation error\.

### I\.3Comparative Analysis: PCBF vs\. Value Flows

We compare PCBF against Value Flows\(Donget al\.,[2025](https://arxiv.org/html/2605.08253#bib.bib1)\)to highlight the impact of our proposed boundary conditions versus the Bellman consistency used in Value Flows \(controlled by thedcfmcoefficient\)\.

#### Sensitivity to Consistency Regularization\.

Figure[9](https://arxiv.org/html/2605.08253#A9.F9)provides a direct comparison of hyperparameter sensitivity between our method \(PCBF\) and Value Flows\. The results reveal a sharp contrast in stability:

- •Value Flows Instability \(Orange\):Increasing the Deep Conjugate Flow Matching coefficient \(dcfm\) systematically degrades performance across all tasks\. This sensitivity is most extreme in theDiscrete MCenvironment \(Right\), where Wasserstein distance starts high \(≈1\.5\\approx 1\.5\) and explodes to\>4\.0\>4\.0asdcfm→1\\texttt\{dcfm\}\\to 1\. This is consistent with enforcing a full\-ttDCFM self\-consistency term that conflicts with the Gaussian source boundary required for accurate transport\.
- •PCBF Robustness \(Blue\):In contrast, our Control Variate formulation \(λ\\lambda\) demonstrates remarkable stability\. InDiscrete MC, PCBF maintains a consistently low Wasserstein distance \(≈0\.5\\approx 0\.5\) across the entire range ofλ∈\[0,0\.95\]\\lambda\\in\[0,0\.95\], effectively ignoring the bias that plagues Value Flows\. Similarly, inSolitaireandBernoulli, PCBF outperforms Value Flows at nearly all coefficient magnitudes, showing that our method reduces variance without introducing significant bias into the terminal distribution\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/combined_final_curve.png)Figure 9:Hyperparameter Sensitivity Analysis \(PCBF vs\. Value Flows\)We compare the impact of increasing the regularization coefficient on distributional accuracy \(Wasserstein Distance\)\.Orange \(Dashed\):Increasing the Value Flows consistency coefficient \(dcfm\) causes rapid performance degradation, particularly in complex environments like Discrete MC\.Blue \(Solid\):Our PCBF Control Variate \(λ\\lambda\) remains robust, maintaining low Wasserstein distances and high stability across a wide range of hyperparameter values, effectively decoupling variance reduction from distributional bias\.
#### Detailed Distributional Comparison\.

Figure[10](https://arxiv.org/html/2605.08253#A9.F10)presents a granular comparison of Cumulative Distribution Functions \(CDFs\) and Wasserstein distances\. Across all six evaluation settings, PCBF consistently achieves the lowest or near\-lowest Wasserstein distance to the ground truth\.

- •Long Horizons:In the Discrete MC environment \(S=5S=5\), VF with dcfm=1=1degrades to a Wasserstein distance of6\.8566\.856due to bias accumulation, while PCBF maintains0\.5860\.586\.
- •Tail Accuracy:The CDF plots reveal that VF systematically underestimates return variance \(producing overly concentrated distributions\), whereas PCBF closely tracks the reference tails\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/lambda_flow_comparison_dcfm.png)Figure 10:Full Distributional accuracy comparison\.PCBF \(blue\) consistently tracks the ground\-truth CDF \(dashed black\) more accurately than Value Flows \(red/green\), particularly in high\-variance regimes\.

### I\.4Visual Analysis of Transport Dynamics\.

Furthermore, we visually analyze the learned transport maps to verify that PCBF captures the correct distributional geometry\. Figure[11](https://arxiv.org/html/2605.08253#A9.F11)presents the learned flows for the Discrete Monte Carlo environment across statess=1,…,20s=1,\\dots,20\. The bottom panels illustrate the trajectories that deterministically transport the base noise to the complex, state\-dependent target return distributions\. The resulting flow\-transported densities \(top panels, blue\) align tightly with ground\-truth Monte Carlo histograms \(black dashed\) across the entire state space\. The visualization confirms that the method robustly approximates the true return lawZπ​\(x,a\)Z^\{\\pi\}\(x,a\)across diverse structures of stochasticity, justifying the low Wasserstein errors observed quantitatively\.

![Refer to caption](https://arxiv.org/html/2605.08253v1/figures/discrete_mc_all_states.png)Figure 11:Distributional Flow Analysis on the Discrete MC Environment\.We visualize the learned PCBF return distributions across statess=1s=1tos=20s=20\. The estimated probability density of the flow\-transported samples \(blue filled\) is compared against Ground Truth Monte Carlo rollouts\(black dashed lines\)\. Characteristic flow trajectories transporting random noise samples\(t=0\)\(t=0\)to the target return distribution\(t=1\)\(t=1\)over flow time\. Trajectory colors distinguish individual particles sampled from the base distributionp​\(x0\)p\(x\_\{0\}\), illustrating how the model maps stochastic noise to specific return outcomes\.