QuantFPFlow: Quantum Amplitude Estimation for Fokker--Planck Policy Optimisation in Continuous Reinforcement Learning

# Quantum Amplitude Estimation for Fokker–Planck Policy Optimisation in Continuous Reinforcement Learning
Source: [https://arxiv.org/html/2605.16429](https://arxiv.org/html/2605.16429)
\(Received: \)

###### Abstract

We introduceQuantFPFlow, a reinforcement learning framework that integrates quantum amplitude estimation into the Fokker–Planck \(FP\) formulation of stochastic policy optimisation\. Classical continuous\-space RL agents must estimate the FP partition functionZ=∫e−V​\(𝐱\)/D​𝑑𝐱Z=\\int e^\{\-V\(\\mathbf\{x\}\)/D\}\\,d\\mathbf\{x\}at cost𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\); QuantFPFlow replaces this with a Grover\-amplified amplitude estimator achieving𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)—a provable quadratic speedup\. While the full quantum acceleration requires fault\-tolerant hardware, the quantum\-inspired classical simulation demonstrated here already exhibits the𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)algorithmic structure\.

The estimated stationary distributionρ∗\\rho^\{\*\}drives a theoretically grounded exploration bonusraug=renv\+α​log⁡\(1/ρ∗​\(s\)\)r\_\{\\mathrm\{aug\}\}=r\_\{\\mathrm\{env\}\}\+\\alpha\\log\(1/\\rho^\{\*\}\(s\)\)\. This bonus steers the agent toward globally optimal regions of multimodal reward landscapes while simultaneously constraining policy variance through FP diffusion matching\.

On a continuous\-control task specifically designed to expose local\-optima failure, QuantFPFlow achieves mean reward1,295\.7±423\.21\{,\}295\.7\\pm 423\.2versus1,284\.0±474\.01\{,\}284\.0\\pm 474\.0for Soft Actor\-Critic \(SAC\), while discovering the global optimum10\.4 % more frequently\(33\.9 % vs\. 30\.7 %\)\. Policy entropy remains nearH​\(π\)≈6\.5H\(\\pi\)\\approx 6\.5nats throughout training, whereas SAC collapses to1\.51\.5nats, confirming that FP diffusion matching actively prevents premature convergence\. Dimensionality experiments further show computational scaling of𝒪​\(d0\.35\)\\mathcal\{O\}\(d^\{0\.35\}\)for QuantFPFlow versus𝒪​\(d0\.76\)\\mathcal\{O\}\(d^\{0\.76\}\)for classical FP estimation\. Keywords:Fokker–Planck equation, Quantum amplitude estimation, Reinforcement learning, Stochastic optimal control, Multimodal optimisation, Exploration

## 1Introduction

Reinforcement learning in continuous state–action spaces remains challenging due to the curse of dimensionality and the tendency of gradient\-based policy optimisation to converge to local optima\. Soft Actor\-Critic \(SAC\)\[[4](https://arxiv.org/html/2605.16429#bib.bib4)\]and DDPG\[[8](https://arxiv.org/html/2605.16429#bib.bib8)\]address exploration through entropy regularisation and Ornstein–Uhlenbeck noise respectively, but both are ultimately greedy methods that collapse onto dominant reward modes when the landscape is multimodal\.

The Fokker–Planck \(FP\) equation offers a principled alternative\[[10](https://arxiv.org/html/2605.16429#bib.bib10)\]\. Rather than optimising a deterministic policy, it characterises the evolution of a*probability distribution*over states under stochastic dynamics\. When the drift is a gradient fieldf​\(𝐱\)=−∇V​\(𝐱\)f\(\\mathbf\{x\}\)=\-\\nabla V\(\\mathbf\{x\}\), the unique stationary solution is the Boltzmann distributionρ∗​\(𝐱\)∝exp⁡\(−V​\(𝐱\)/D\)\\rho^\{\*\}\(\\mathbf\{x\}\)\\propto\\exp\(\-V\(\\mathbf\{x\}\)/D\), which encodes the globally optimal stochastic policy\[[6](https://arxiv.org/html/2605.16429#bib.bib6),[3](https://arxiv.org/html/2605.16429#bib.bib3)\]\. The computational bottleneck is the partition functionZ=∫e−V​\(𝐱\)/D​𝑑𝐱Z=\\int e^\{\-V\(\\mathbf\{x\}\)/D\}\\,d\\mathbf\{x\}, which requires𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\)Monte Carlo samples forε\\varepsilon\-precision estimation\.

Quantum amplitude estimation \(QAE\)\[[1](https://arxiv.org/html/2605.16429#bib.bib1)\]provides a quadratic speedup: the partition function can be encoded as the squared norm of a quantum state and estimated in𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)queries via Grover\-like amplification\. This paper operationalises this connection, yieldingQuantFPFlow, an FP\-Actor\-Critic with a quantum\-enhanced exploration mechanism\.

#### Contributions\.

1. 1\.QuantFPFlow algorithm: an FP\-Actor\-Critic that uses temperature\-annealed QAE to compute a theoretically grounded exploration bonus, FP\-regularised policy gradients, and adaptive diffusion matching\.
2. 2\.Proven quadratic query speedup\(Theorem[1](https://arxiv.org/html/2605.16429#Thmtheorem1)\):𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)partition function estimation versus classical𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\), with an empirical demonstration spanning three orders of magnitude in precision \([Figure5](https://arxiv.org/html/2605.16429#S5.F5)\)\.
3. 3\.Superior global optimum discovery: 10\.4 % relative improvement over SAC on a multimodal continuous control benchmark, with sustained high policy entropy \([Figures2](https://arxiv.org/html/2605.16429#S5.F2)and[3](https://arxiv.org/html/2605.16429#S5.F3)\)\.
4. 4\.Favourable dimensionality scaling:𝒪​\(d0\.35\)\\mathcal\{O\}\(d^\{0\.35\}\)versus classical FP𝒪​\(d0\.76\)\\mathcal\{O\}\(d^\{0\.76\}\)\([Figure6](https://arxiv.org/html/2605.16429#S5.F6)\)\.

## 2Background

This section reviews the two theoretical pillars of QuantFPFlow: the Fokker–Planck equation as a framework for stochastic policy optimisation, and quantum amplitude estimation as a tool for quadratic speedup in partition function computation\. Together they motivate the design of our algorithm in[Section3](https://arxiv.org/html/2605.16429#S3)\.

### 2\.1Fokker–Planck Equation in Reinforcement Learning

Consider a continuous\-state agent with Itô dynamics

d​𝐱=f​\(𝐱\)​d​t\+2​D​d​W,d\\mathbf\{x\}=f\(\\mathbf\{x\}\)\\,dt\+\\sqrt\{2D\}\\,dW,\(1\)wheref​\(𝐱\)f\(\\mathbf\{x\}\)is the policy\-induced drift,D\>0D\>0is the diffusion coefficient, andWWis a standard Wiener process\. The probability densityρ​\(𝐱,t\)\\rho\(\\mathbf\{x\},t\)satisfies the Fokker–Planck equation:

∂ρ∂t=−∇⋅\(f​\(𝐱\)​ρ\)\+D​∇2ρ\.\\frac\{\\partial\\rho\}\{\\partial t\}=\-\\nabla\\cdot\\bigl\(f\(\\mathbf\{x\}\)\\,\\rho\\bigr\)\+D\\,\\nabla^\{2\}\\rho\.\(2\)For gradient driftf​\(𝐱\)=−∇V​\(𝐱\)f\(\\mathbf\{x\}\)=\-\\nabla V\(\\mathbf\{x\}\), the unique stationary solution is

ρ∗​\(𝐱\)=1Z​e−V​\(𝐱\)/D,Z=∫ℝde−V​\(𝐱\)/D​𝑑𝐱,\\rho^\{\*\}\(\\mathbf\{x\}\)=\\frac\{1\}\{Z\}\\,e^\{\-V\(\\mathbf\{x\}\)/D\},\\quad Z=\\int\_\{\\mathbb\{R\}^\{d\}\}e^\{\-V\(\\mathbf\{x\}\)/D\}\\,d\\mathbf\{x\},\(3\)providedZ<∞Z<\\infty\. This is the*optimal stochastic policy*for the control problem with running costV​\(𝐱\)V\(\\mathbf\{x\}\)and diffusionDD\[[6](https://arxiv.org/html/2605.16429#bib.bib6),[11](https://arxiv.org/html/2605.16429#bib.bib11)\]\.

### 2\.2Quantum Amplitude Estimation

###### Definition 1\(Quantum oracle forZZ\)\.

Let𝒳=\{x1,…,xN\}\\mathcal\{X\}=\\\{x\_\{1\},\\ldots,x\_\{N\}\\\}be a uniform discretisation of the state space withN=2nN=2^\{n\}points\. Define the oracle𝒜\\mathcal\{A\}as the unitary preparing

𝒜​\|0⟩=1N​∑i=1N\(1−pi​\|0⟩\+pi​\|1⟩\)​\|i⟩,pi=e−V​\(xi\)/DM,\\mathcal\{A\}\|0\\rangle=\\frac\{1\}\{\\sqrt\{N\}\}\\sum\_\{i=1\}^\{N\}\\left\(\\sqrt\{1\-p\_\{i\}\}\\,\|0\\rangle\+\\sqrt\{p\_\{i\}\}\\,\|1\\rangle\\right\)\|i\\rangle,\\quad p\_\{i\}=\\frac\{e^\{\-V\(x\_\{i\}\)/D\}\}\{M\},\(4\)whereM=maxi⁡e−V​\(xi\)/DM=\\max\_\{i\}e^\{\-V\(x\_\{i\}\)/D\}normalises probabilities to\[0,1\]\[0,1\]\.

With this oracle, the amplitudea=𝔼​\[pi\]=Z/\(N​M\)a=\\sqrt\{\\mathbb\{E\}\[p\_\{i\}\]\}=\\sqrt\{Z/\(NM\)\}encodes the partition function\. Quantum amplitude estimation\[[1](https://arxiv.org/html/2605.16429#bib.bib1)\]estimatesa2a^\{2\}to additive precisionε\\varepsilonusing𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)applications of𝒜\\mathcal\{A\}and its inverse, via the quantum phase estimation subroutine applied to the Grover operatorQ=𝒜​\(I−2​\|0⟩​⟨0\|\)​𝒜−1​\(I−2​\|χ1⟩​⟨χ1\|\)Q=\\mathcal\{A\}\(I\-2\|0\\rangle\\langle 0\|\)\\mathcal\{A\}^\{\-1\}\(I\-2\|\\chi\_\{1\}\\rangle\\langle\\chi\_\{1\}\|\), where\|χ1⟩\|\\chi\_\{1\}\\ranglemarks the “good” states\. We encode the FP stationary distribution as

\|ψ⟩=1Z​∑x∈𝒳e−V​\(x\)/2​D​\|x⟩,\|\\psi\\rangle=\\frac\{1\}\{\\sqrt\{Z\}\}\\sum\_\{x\\in\\mathcal\{X\}\}e^\{\-V\(x\)/2D\}\|x\\rangle,\(5\)so that\|⟨x\|ψ⟩\|2=ρ∗​\(x\)\|\\langle x\|\\psi\\rangle\|^\{2\}=\\rho^\{\*\}\(x\)\. Grover\-like amplitude amplification then estimates these squared amplitudes with𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)queries\.

### 2\.3Related Work

#### FP in RL\.

Kappen\[[6](https://arxiv.org/html/2605.16429#bib.bib6)\]showed stochastic optimal control reduces to a Boltzmann path integral\. Todorov\[[11](https://arxiv.org/html/2605.16429#bib.bib11)\]connected this to KL111Kullback\-Leibler\-regularised RL\. Our work operationalises this connection computationally via quantum estimation\.

#### Quantum RL\.

Dunjko et al\.\[[2](https://arxiv.org/html/2605.16429#bib.bib2)\]established quantum speedups for model\-based RL\. Jerbi et al\.\[[5](https://arxiv.org/html/2605.16429#bib.bib5)\]showed quadratic speedups for policy evaluation\. QuantFPFlow is the first to exploit QAE specifically for FP partition function computation in continuous\-space RL\.

#### Entropy\-regularised RL\.

SAC\[[4](https://arxiv.org/html/2605.16429#bib.bib4)\]maximises an entropy\-augmented reward\. QuantFPFlow differs: entropy emerges from the FP diffusion structure via a consistency constraint rather than being added as a reward term\.

## 3QuantFPFlow Algorithm

Building on the FP framework and QAE speedup established in[Section2](https://arxiv.org/html/2605.16429#S2), we next present the full QuantFPFlow algorithm\. The algorithm couples three components—a quantum amplitude estimator, an FP\-guided actor, and a TD222Temporal Differencecritic—into a unified training loop whose key innovation is using the FP stationary distribution both as a theoretically grounded exploration signal and as a structural constraint on policy variance\.

### 3\.1Overview

QuantFPFlow has three coupled components: \(i\) a QAE computingρ∗​\(s\)\\rho^\{\*\}\(s\); \(ii\) an FP\-Actor updating policy meanμ\\muand log\-stdlog⁡σ\\log\\sigmavia FP\-guided gradients; \(iii\) a linear CriticVϕ​\(s\)=ϕ⊤​sV\_\{\\phi\}\(s\)=\\phi^\{\\top\}strained via TD learning\.

### 3\.2Temperature\-Annealed Quantum Amplitude Estimator

Algorithm 1Temperature\-Annealed QAE1:potential

V​\(x\)V\(x\), range

\[a,b\]\[a,b\], inverse temperature

β\\beta,

nqubitsn\_\{\\mathrm\{qubits\}\}
2:estimated

ρ^∗​\(x\)\\hat\{\\rho\}^\{\*\}\(x\)
3:

𝐱←linspace​\(a,b,2nqubits\)\\mathbf\{x\}\\leftarrow\\mathrm\{linspace\}\(a,b,\\,2^\{n\_\{\\mathrm\{qubits\}\}\}\);

𝐕←\[V​\(xi\)\]\\mathbf\{V\}\\leftarrow\[V\(x\_\{i\}\)\]
4:

ρacc←𝟎\\rho\_\{\\mathrm\{acc\}\}\\leftarrow\\mathbf\{0\}
5:for

βi\\beta\_\{i\}in

linspace​\(0\.3​β,β,6\)\\mathrm\{linspace\}\(0\.3\\beta,\\,\\beta,\\,6\)do

6:

𝐚←exp⁡\(−βi​𝐕/2\)\\mathbf\{a\}\\leftarrow\\exp\(\-\\beta\_\{i\}\\mathbf\{V\}/2\);

𝐚←𝐚/‖𝐚‖\\mathbf\{a\}\\leftarrow\\mathbf\{a\}/\\left\\\|\\mathbf\{a\}\\right\\\|
7:for

k=1k=1to

55do

8:

𝐚←2​a¯​1−𝐚\\mathbf\{a\}\\leftarrow 2\\bar\{a\}\\,\\mathbf\{1\}\-\\mathbf\{a\}⊳\\trianglerightinversion about mean

9:

𝐚←max⁡\(𝐚,0\)\\mathbf\{a\}\\leftarrow\\max\(\\mathbf\{a\},0\);

𝐚←𝐚/‖𝐚‖\\mathbf\{a\}\\leftarrow\\mathbf\{a\}/\\left\\\|\\mathbf\{a\}\\right\\\|
10:endfor

11:

ρacc←ρacc\+𝐚2\\rho\_\{\\mathrm\{acc\}\}\\leftarrow\\rho\_\{\\mathrm\{acc\}\}\+\\mathbf\{a\}^\{2\}
12:endfor

13:return

ρacc/∑ρacc\\rho\_\{\\mathrm\{acc\}\}/\\sum\\rho\_\{\\mathrm\{acc\}\}

The outer temperature annealing loop \([Algorithm1](https://arxiv.org/html/2605.16429#alg1), line 3\) initialises with a flat distribution atβi=0\.3​β\\beta\_\{i\}=0\.3\\betaand progressively sharpens the estimate towardρ∗\\rho^\{\*\}atβ\\beta\. This prevents premature collapse to a single mode—analogous to simulated annealing—while the inner Grover loop \(lines 5–9\) amplifies the probability mass of high\-density regions\.

### 3\.3Complexity Theorem

We now state and prove the main theoretical result\.

###### Theorem 1\(Quadratic query speedup for FP partition function\)\.

LetV:ℝd→ℝV:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}be a potential withZ=∫e−V​\(𝐱\)/D​𝑑𝐱<∞Z=\\int e^\{\-V\(\\mathbf\{x\}\)/D\}\\,d\\mathbf\{x\}<\\infty, and let𝒳=\{x1,…,xN\}\\mathcal\{X\}=\\\{x\_\{1\},\\ldots,x\_\{N\}\\\},N=2nN=2^\{n\}, be a uniform discretisation of a bounded domainΩ⊂ℝd\\Omega\\subset\\mathbb\{R\}^\{d\}with spacinghh\. Denote the discrete partition functionZN=hd​∑i=1Ne−V​\(xi\)/DZ\_\{N\}=h^\{d\}\\sum\_\{i=1\}^\{N\}e^\{\-V\(x\_\{i\}\)/D\}\.

1. 1\.\(Classical lower bound\)Any classical algorithm that estimatesZNZ\_\{N\}to additive precisionε\\varepsilonwith success probability≥2/3\\geq 2/3from i\.i\.d\. samples ofe−V​\(X\)/De^\{\-V\(X\)/D\},X∼Uniform​\(𝒳\)X\\sim\\mathrm\{Uniform\}\(\\mathcal\{X\}\), requiresΩ​\(1/ε2\)\\Omega\(1/\\varepsilon^\{2\}\)samples\.
2. 2\.\(Quantum upper bound\)The QAE algorithm with oracle𝒜\\mathcal\{A\}from[Definition1](https://arxiv.org/html/2605.16429#Thmdefinition1)estimatesZNZ\_\{N\}to additive precisionε\\varepsilonwith success probability≥8/π2\>0\.81\\geq 8/\\pi^\{2\}\>0\.81using𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)applications of𝒜\\mathcal\{A\}and𝒜−1\\mathcal\{A\}^\{\-1\}\.

###### Proof\.

Part 1 \(Classical lower bound\)\.LetX1,…,Xk∼Uniform​\(𝒳\)X\_\{1\},\\ldots,X\_\{k\}\\sim\\mathrm\{Uniform\}\(\\mathcal\{X\}\)be i\.i\.d\. samples and defineYj=e−V​\(Xj\)/DY\_\{j\}=e^\{\-V\(X\_\{j\}\)/D\}\. Then𝔼​\[Yj\]=ZN/\(N​h−d\)\\mathbb\{E\}\[Y\_\{j\}\]=Z\_\{N\}/\(Nh^\{\-d\}\)and the estimatorZ^N=\(N​hd/k\)​∑j=1kYj\\hat\{Z\}\_\{N\}=\(Nh^\{d\}/k\)\\sum\_\{j=1\}^\{k\}Y\_\{j\}is unbiased\. By the central limit theorem, the standard error isSE=N​hd⋅Std​\(Y\)/k\\mathrm\{SE\}=Nh^\{d\}\\cdot\\mathrm\{Std\}\(Y\)/\\sqrt\{k\}\. To achieve\|Z^N−ZN\|≤ε\|\\hat\{Z\}\_\{N\}\-Z\_\{N\}\|\\leq\\varepsilonwith probability≥2/3\\geq 2/3, Chebyshev’s inequality gives

k≥9​\(N​hd\)2​Var​\(Y\)ε2=Ω​\(1ε2\)\.k\\;\\geq\\;\\frac\{9\\,\(Nh^\{d\}\)^\{2\}\\,\\mathrm\{Var\}\(Y\)\}\{\\varepsilon^\{2\}\}\\;=\\;\\Omega\\\!\\left\(\\frac\{1\}\{\\varepsilon^\{2\}\}\\right\)\.\(6\)The lower bound is tight \(achieved by the sample mean\) and holds for any classical unbiased estimator by the Cramér–Rao bound\.

Part 2 \(Quantum upper bound\)\.By[Definition1](https://arxiv.org/html/2605.16429#Thmdefinition1), the oracle𝒜\\mathcal\{A\}prepares a state with “good amplitude”

sin2⁡θ=1N​∑i=1Npi=ZNN​M​hd,\\sin^\{2\}\\theta=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p\_\{i\}=\\frac\{Z\_\{N\}\}\{NMh^\{d\}\},\(7\)soZN=N​M​hd​sin2⁡θZ\_\{N\}=NMh^\{d\}\\sin^\{2\}\\theta\. We apply the quantum phase estimation \(QPE\) circuit to the Grover operatorQ=−𝒜​\(I−2​\|0⟩​⟨0\|\)​𝒜−1​\(I−2​\|χ1⟩​⟨χ1\|\)Q=\-\\mathcal\{A\}\(I\-2\|0\\rangle\\langle 0\|\)\\mathcal\{A\}^\{\-1\}\(I\-2\|\\chi\_\{1\}\\rangle\\langle\\chi\_\{1\}\|\), which has eigenvaluese±2​i​θe^\{\\pm 2i\\theta\}\. QPE withmmancilla qubits estimatesθ/π\\theta/\\pito precision2−m2^\{\-m\}with probability≥8/π2\\geq 8/\\pi^\{2\}, using2m2^\{m\}applications ofQQ\[[1](https://arxiv.org/html/2605.16429#bib.bib1)\]\. Setting2−m=ε/\(2​π​N​M​hd\)2^\{\-m\}=\\varepsilon/\(2\\pi NMh^\{d\}\)so that the induced error onZN=N​M​hd​sin2⁡θZ\_\{N\}=NMh^\{d\}\\sin^\{2\}\\thetais at mostε\\varepsilon\(via the Lipschitz bound\|d​\(sin2⁡θ\)/d​θ\|≤2\|d\(\\sin^\{2\}\\theta\)/d\\theta\|\\leq 2\) gives2m=𝒪​\(1/ε\)2^\{m\}=\\mathcal\{O\}\(1/\\varepsilon\), hence𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)oracle calls\. Together with Part 1 this establishes the quadratic separation\. ∎

###### Corollary 1\(Speedup for FP stationary distribution\)\.

Estimatingρ∗​\(xi\)\\rho^\{\*\}\(x\_\{i\}\)at any grid point to additive precisionε\\varepsilonrequires𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)QAE queries versus𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\)classically\.

###### Proof\.

ρ∗​\(xi\)=e−V​\(xi\)/D/ZN\\rho^\{\*\}\(x\_\{i\}\)=e^\{\-V\(x\_\{i\}\)/D\}/Z\_\{N\}\. GivenZ^N\\hat\{Z\}\_\{N\}from Theorem[1](https://arxiv.org/html/2605.16429#Thmtheorem1)Part 2 with precisionε​ZN/2\\varepsilon Z\_\{N\}/2and the exact numeratore−V​\(xi\)/De^\{\-V\(x\_\{i\}\)/D\}\(computed classically in𝒪​\(1\)\\mathcal\{O\}\(1\)\), error propagation gives\|ρ∗^​\(xi\)−ρ∗​\(xi\)\|≤ε\|\\hat\{\\rho^\{\*\}\}\(x\_\{i\}\)\-\\rho^\{\*\}\(x\_\{i\}\)\|\\leq\\varepsilonwith𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)oracle calls total\. ∎

### 3\.4Quantum Exploration Bonus

Givenρ^∗​\(s\)\\hat\{\\rho\}^\{\*\}\(s\), we augment the environment reward:

raug​\(s\)=renv​\(s\)\+α​log⁡1ρ^∗​\(s\),r\_\{\\mathrm\{aug\}\}\(s\)=r\_\{\\mathrm\{env\}\}\(s\)\+\\alpha\\,\\log\\frac\{1\}\{\\hat\{\\rho\}^\{\*\}\(s\)\},\(8\)whereα\>0\\alpha\>0\. The bonus is large whenρ^∗​\(s\)\\hat\{\\rho\}^\{\*\}\(s\)is small, incentivising the agent to visit states that are rare under the FP stationary distribution but are globally important\. Unlike curiosity\-based methods\[[9](https://arxiv.org/html/2605.16429#bib.bib9)\], this bonus is*theory\-grounded*: it is derived from the exact stationary distribution of the FP equation \([3](https://arxiv.org/html/2605.16429#S2.E3)\), which is the provably optimal long\-run occupancy measure\.

### 3\.5FP\-Actor\-Critic Update

The actor parameters\(μ,log⁡σ\)\(\\mu,\\log\\sigma\)are updated via a score\-function estimator of the FP drift gradient:

∇μℒFP=−𝔼a∼πμ​\[δTD⋅fFP​\(s\)⋅a−μσ2\],\\nabla\_\{\\mu\}\\mathcal\{L\}\_\{\\mathrm\{FP\}\}=\-\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\mu\}\}\\\!\\left\[\\delta\_\{\\mathrm\{TD\}\}\\cdot f\_\{\\mathrm\{FP\}\}\(s\)\\cdot\\frac\{a\-\\mu\}\{\\sigma^\{2\}\}\\right\],\(9\)whereδTD=raug\+γ​Vϕ​\(s′\)−Vϕ​\(s\)\\delta\_\{\\mathrm\{TD\}\}=r\_\{\\mathrm\{aug\}\}\+\\gamma V\_\{\\phi\}\(s^\{\\prime\}\)\-V\_\{\\phi\}\(s\)andfFP​\(s\)=−∇V​\(s\)f\_\{\\mathrm\{FP\}\}\(s\)=\-\\nabla V\(s\)is the FP drift\. Momentum is applied:mt\+1=βm​mt\+\(1−βm\)​∇ℒm\_\{t\+1\}=\\beta\_\{m\}m\_\{t\}\+\(1\-\\beta\_\{m\}\)\\nabla\\mathcal\{L\},μt\+1=μt\+ηa​mt\+1\\mu\_\{t\+1\}=\\mu\_\{t\}\+\\eta\_\{a\}m\_\{t\+1\}\.

The log\-standard\-deviation is updated via the*FP consistency loss*:

log⁡σt\+1=log⁡σt\+ηa​σt2−Dσt,\\log\\sigma\_\{t\+1\}=\\log\\sigma\_\{t\}\+\\eta\_\{a\}\\,\\frac\{\\sigma\_\{t\}^\{2\}\-D\}\{\\sigma\_\{t\}\},\(10\)which drivesσ2→D\\sigma^\{2\}\\to D, maintaining the structural correspondence between the policy and the FP diffusion\.

Algorithm 2QuantFPFlow Training1:

α=0\.5\\alpha\{=\}0\.5,

β=1\.5\\beta\{=\}1\.5,

D=0\.3D\{=\}0\.3,

ηa=5×10−3\\eta\_\{a\}\{=\}5\\\!\\times\\\!10^\{\-3\},

ηc=10−2\\eta\_\{c\}\{=\}10^\{\-2\},

γ=0\.99\\gamma\{=\}0\.99
2:Initialise

μ\\mu,

log⁡σ\\log\\sigma,

ϕ\\phi, QAE cache

3:forepisode

=1=1to

TTdo

4:

s←env\.reset​\(\)s\\leftarrow\\textsc\{env\.reset\}\(\)
5:whilenot donedo

6:

a←𝒩​\(s⊤​μ/‖s‖,elog⁡σ\)a\\leftarrow\\mathcal\{N\}\(s^\{\\top\}\\mu/\\left\\\|s\\right\\\|,\\,e^\{\\log\\sigma\}\)
7:

s′,r,done←env\.step​\(a\)s^\{\\prime\},r,\\mathrm\{done\}\\leftarrow\\textsc\{env\.step\}\(a\)
8:

ρ^∗​\(s\)←QAE​\(VFP,β\)\\hat\{\\rho\}^\{\*\}\(s\)\\leftarrow\\textsc\{QAE\}\(V\_\{\\mathrm\{FP\}\},\\beta\)⊳\\trianglerightcached every 10 steps

9:

raug←r\+α​log⁡\(1/ρ^∗​\(s\)\)r\_\{\\mathrm\{aug\}\}\\leftarrow r\+\\alpha\\log\(1/\\hat\{\\rho\}^\{\*\}\(s\)\)
10:

δ←raug\+γ​ϕ⊤​s′−ϕ⊤​s\\delta\\leftarrow r\_\{\\mathrm\{aug\}\}\+\\gamma\\phi^\{\\top\}s^\{\\prime\}\-\\phi^\{\\top\}s
11:

ϕ←ϕ\+ηc​δ​s\\phi\\leftarrow\\phi\+\\eta\_\{c\}\\,\\delta\\,s⊳\\trianglerightcritic TD

12:Update

μ\\muvia \([9](https://arxiv.org/html/2605.16429#S3.E9)\) with momentum

13:Update

log⁡σ\\log\\sigmavia \([10](https://arxiv.org/html/2605.16429#S3.E10)\)⊳\\trianglerightFP consistency

14:

s←s′s\\leftarrow s^\{\\prime\}
15:endwhile

16:endfor

## 4Experimental Setup

We evaluate QuantFPFlow against SAC, DDPG, and a random baseline on a purpose\-built continuous control environment designed to expose the local\-optima failure of greedy methods\. All experiments are run in a single\-CPU Google Colab environment; no GPU is required because the quantum\-inspired estimation is analytically cheap\. The environment, baselines, and evaluation metrics are described below\.

### 4\.1MultimodalRewardEnv

To expose the local\-optima failure of greedy methods we design a continuous control task \([Figure1](https://arxiv.org/html/2605.16429#S5.F1)\) with reward landscape

r​\(𝐱\)=8\.0​e−\(x1\+1\.5\)2/0\.32\+15\.0​e−\(x1−1\.5\)2/0\.32−5\.0​e−x12/0\.18−0\.05​‖𝐱2:d‖2\.r\(\\mathbf\{x\}\)=8\.0\\,e^\{\-\(x\_\{1\}\+1\.5\)^\{2\}/0\.32\}\+15\.0\\,e^\{\-\(x\_\{1\}\-1\.5\)^\{2\}/0\.32\}\-5\.0\\,e^\{\-x\_\{1\}^\{2\}/0\.18\}\-0\.05\\left\\\|\\mathbf\{x\}\_\{2:d\}\\right\\\|^\{2\}\.\(11\)Alocal optimumatx1≈−1\.5x\_\{1\}\\approx\{\-\}1\.5\(height 8\) is separated by a barrier from aglobal optimumatx1≈\+1\.5x\_\{1\}\\approx\{\+\}1\.5\(height 15\)\. State dynamics follow𝐬t\+1=0\.92​𝐬t\+0\.15​𝐚t\+0\.08​ξ\\mathbf\{s\}\_\{t\+1\}=0\.92\\mathbf\{s\}\_\{t\}\+0\.15\\mathbf\{a\}\_\{t\}\+0\.08\\xiwithξ∼𝒩​\(0,I\)\\xi\\sim\\mathcal\{N\}\(0,I\)\. State dimensiond=4d=4, action dimensionm=2m=2, episode horizonH=200H=200\.

### 4\.2Baselines

- •SAC\[[4](https://arxiv.org/html/2605.16429#bib.bib4)\]: entropy\-regularised actor\-critic \(αSAC=0\.2\\alpha\_\{\\mathrm\{SAC\}\}\{=\}0\.2,η=5×10−3\\eta\{=\}5\{\\times\}10^\{\-3\}, Adam optimiser\)\.
- •DDPG\[[8](https://arxiv.org/html/2605.16429#bib.bib8)\]: deterministic policy with Ornstein–Uhlenbeck exploration noise \(σOU=0\.2\\sigma\_\{\\mathrm\{OU\}\}\{=\}0\.2\)\.
- •Random: uniform random actions in\[−1,1\]m\[\-1,1\]^\{m\}\.

All agents train for 400 episodes\. Full hyperparameters are listed in[AppendixB](https://arxiv.org/html/2605.16429#A2)\.

## 5Results

We present results across six complementary analyses: the reward landscape and its implications for exploration \([Figure1](https://arxiv.org/html/2605.16429#S5.F1)\), learning curves and final performance \([Figures2](https://arxiv.org/html/2605.16429#S5.F2)and[1](https://arxiv.org/html/2605.16429#S5.T1)\), the exploration mechanism \([Figure3](https://arxiv.org/html/2605.16429#S5.F3)\), FP distribution estimation \([Figure4](https://arxiv.org/html/2605.16429#S5.F4)\), query complexity and sample efficiency \([Figure5](https://arxiv.org/html/2605.16429#S5.F5)\), and dimensionality scaling with qubit ablation \([Figure6](https://arxiv.org/html/2605.16429#S5.F6)\)\. Taken together, these results confirm all four contributions listed in[Section1](https://arxiv.org/html/2605.16429#S1)\.

### 5\.1Reward Landscape and Motivation

[Figure1](https://arxiv.org/html/2605.16429#S5.F1)shows the reward landscape defined by \([11](https://arxiv.org/html/2605.16429#S4.E11)\)\. The barrier between the two modes creates a trap for greedy agents: once the local optimum is found, gradient\-based methods have no incentive to cross the barrier\. QuantFPFlow’s exploration bonus \([8](https://arxiv.org/html/2605.16429#S3.E8)\) is large in the barrier region \(lowρ∗\\rho^\{\*\}\), actively encouraging barrier crossing\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_reward_landscape.png)Figure 1:Reward landscape of the multimodal continuous control environment\. The local optimum \(left dashed line, height 8\) and global optimum \(right dashed line, height 15\) are separated by a barrier atx1=0x\_\{1\}\{=\}0\. Greedy agents are trapped in the local optimum; the QuantFPFlow exploration bonus incentivises crossing the barrier\.
### 5\.2Learning Curves and Final Performance

[Figure2](https://arxiv.org/html/2605.16429#S5.F2)shows learning curves and final performance\. QuantFPFlow begins with negative cumulative reward during its exploration phase \(episodes 0–50\), as the quantum bonus incentivises visiting low\-density regions including suboptimal areas\. It then converges rapidly to∼\\sim1,300 cumulative reward, matching and slightly exceeding SAC’s final performance\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_learning_curves.png)Figure 2:Learning curves on the multimodal continuous control environment\.Left: smoothed cumulative reward \(window 25\) with±0\.3​σ\\pm 0\.3\\sigmashaded regions\.Centre: global optimum discovery rate—fraction of steps within\|x1−1\.5\|<0\.5\|x\_\{1\}\-1\.5\|<0\.5\.Right: mean reward over final 80 episodes with standard\-deviation error bars\. QuantFPFlow achieves the highest mean reward and discovery rate; DDPG collapses after episode 300\.[Table1](https://arxiv.org/html/2605.16429#S5.T1)summarises quantitative performance\. QuantFPFlow achieves the highest mean reward \(1,295\.71\{,\}295\.7\) and significantly outperforms DDPG \(628\.4628\.4\) and Random \(−670\.4\-670\.4\)\. Crucially, it discovers the global optimum in33\.9 %of steps versus SAC’s 30\.7 %—a10\.4 % relative improvement\.

Table 1:Performance summary \(400 training episodes, averaged over final 80 episodes\)\. Global opt\. rate: fraction of steps with\|x1−1\.5\|<0\.5\|x\_\{1\}\-1\.5\|<0\.5\. Bold indicates best result per column\.
### 5\.3Exploration Mechanism

[Figure3](https://arxiv.org/html/2605.16429#S5.F3)analyses the exploration mechanism in three panels\.

#### Reward augmentation \(left\)\.

The quantum exploration bonus \(shaded region, right axis\) contributes a consistent positive signal throughout training, stabilising at magnitude 5–6\. This bonus is derived fromρ∗\\rho^\{\*\}and is therefore theoretically grounded rather than heuristic\.

#### Policy entropy \(centre\)\.

QuantFPFlow entropy rises from3\.53\.5to a stable plateau of6\.5 natsvia FP diffusion matching \([10](https://arxiv.org/html/2605.16429#S3.E10)\)\. SAC entropy collapses from7\.57\.5to1\.5 nats—a5×5\\timesreduction despite explicit entropy regularisation\. This collapse explains SAC’s lower global optimum discovery rate\.

#### State visitation \(right\)\.

Both QuantFPFlow and SAC visit the local optimum \(x1≈−1\.5x\_\{1\}\\approx\{\-\}1\.5\), but QuantFPFlow shows a clearly higher secondary density at the global optimum \(x1≈\+1\.5x\_\{1\}\\approx\{\+\}1\.5\)\. DDPG’s policy collapses entirely to the local optimum\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_exploration_analysis.png)Figure 3:Exploration analysis\.Left: reward decomposition; environment reward \(dashed\) and quantum exploration bonus \(shaded, right axis\)\.Centre: policy entropy; QuantFPFlow maintainsH​\(π\)≈6\.5H\(\\pi\)\\approx 6\.5nats while SAC collapses to1\.51\.5nats\.Right: state\-visitation density of the trained policy alongx1x\_\{1\}; QuantFPFlow maintains mass at both optima\.

### 5\.4Fokker–Planck Distribution Estimation

[Figure4](https://arxiv.org/html/2605.16429#S5.F4)illustrates the FP framework\. The multimodal potentialV​\(x\)=12​\(x2−2\)2\+0\.3​sin⁡\(3​x\)V\(x\)=\\tfrac\{1\}\{2\}\(x^\{2\}\-2\)^\{2\}\+0\.3\\sin\(3x\)\(left panel\) has two wells corresponding to the two reward modes\. The stationary distributionρ∗\\rho^\{\*\}\(centre panel\) shows the ground truth \(dashed\) alongside the QuantFPFlow estimate and the classical FP solver; the classical solver closely tracks the ground truth while QuantFPFlow captures the bimodal structure with reduced peak height due to the finite29=5122^\{9\}=512grid discretisation\. The right panel confirms FP convergence dynamics: density propagates from a uniform initial condition to the stationary bimodal distribution\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_fp_distributions.png)Figure 4:Fokker–Planck framework\.Left: multimodal potentialV​\(x\)V\(x\)\.Centre: stationary distributionρ∗​\(x\)\\rho^\{\*\}\(x\)—ground truth \(dashed\), QuantFPFlow estimate \(red\), classical FP solver \(blue\)\.Right: FP evolution heatmap showing convergence to the stationary distribution\.
### 5\.5Query Complexity and Sample Efficiency

[Figure5](https://arxiv.org/html/2605.16429#S5.F5)confirms Theorem[1](https://arxiv.org/html/2605.16429#Thmtheorem1)empirically\. Classical estimation scales as𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\)while QuantFPFlow achieves𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\), yielding a1,000×1\{,\}000\\timesadvantage at precisionε=10−3\\varepsilon=10^\{\-3\}\. Sample efficiency \(centre\) shows QuantFPFlow’s faster convergence in the first 100 episodes\. Policy entropy \(right\) confirms sustained exploration throughout8×1048\\times 10^\{4\}update steps\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_complexity_efficiency.png)Figure 5:Complexity and efficiency analysis\.Left: query complexity for partition function estimation; shaded region is the quantum advantage gap \(Theorem[1](https://arxiv.org/html/2605.16429#Thmtheorem1)\)\.Centre: sample efficiency over 400 episodes\.Right: policy entropy during training\.
### 5\.6Dimensionality Scaling and Qubit Ablation

[Figure6](https://arxiv.org/html/2605.16429#S5.F6)presents the scaling analysis\. QuantFPFlow computation grows as𝒪​\(d0\.35\)\\mathcal\{O\}\(d^\{0\.35\}\)versus classical FP at𝒪​\(d0\.76\)\\mathcal\{O\}\(d^\{0\.76\}\), a significant advantage at high state dimensions\. The qubit ablation \(right panel\) shows MSE333Mean Squared Errorstabilising at8\.4×10−28\.4\{\\times\}10^\{\-2\}for≥5\\geq 5qubits; the slight non\-monotonicity at 4 qubits reflects discretisation artefacts at24=162^\{4\}=16grid points\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_scaling_ablation.png)Figure 6:Scaling and ablation\.Left: computation time vs\. state dimensionality \(log–log\); QuantFPFlow grows as𝒪​\(d0\.35\)\\mathcal\{O\}\(d^\{0\.35\}\)vs\. classical𝒪​\(d0\.76\)\\mathcal\{O\}\(d^\{0\.76\}\)\.Right: stationary distribution MSE vs\. qubit count; performance stabilises at≥5\\geq 5qubits\.
### 5\.7Phase Portrait and 2D Stationary Distribution

[Figure7](https://arxiv.org/html/2605.16429#S5.F7)visualises the FP vector field and 2D stationary distribution\. The phase portrait \(left\) shows the gradient drift field with two attractors corresponding to the double\-well potential minima\. The 2D stationary distribution \(right\) confirms the bimodal structure is preserved in two dimensions, consistent with[Equation11](https://arxiv.org/html/2605.16429#S4.E11)\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_phase_portrait.png)Figure 7:FP vector field and 2D stationary distribution\.Left: phase portrait coloured by drift magnitude; flow converges to two attractors\.Right: 2D quantum\-FP stationary distributionρ∗​\(x1,x2\)\\rho^\{\*\}\(x\_\{1\},x\_\{2\}\)as heatmap with contour lines\.

## 6Discussion

The experimental results raise three questions worth addressing in depth:*why*QuantFPFlow maintains exploration while SAC does not;*how*the quantum advantage manifests in a classical simulation; and*what*the current limitations imply for future work\. We address each in turn\.

### 6\.1Why QuantFPFlow Maintains Exploration

The critical mechanism is the FP consistency update \([10](https://arxiv.org/html/2605.16429#S3.E10)\), which drivesσ2→D\\sigma^\{2\}\\to Drather than toward zero\. In SAC, entropy regularisation creates a gradient toward higher entropy but is dominated by the value gradient as training progresses\. In QuantFPFlow, FP consistency is a*structural constraint*: the policy variance is required to match the diffusion implied by the FP equation, preventing entropy collapse and maintaining exploration \([Figure3](https://arxiv.org/html/2605.16429#S5.F3), centre\)\.

### 6\.2Quantum Advantage in Practice

The speedup established in Theorem[1](https://arxiv.org/html/2605.16429#Thmtheorem1)is in query complexity for partition function estimation\. On near\-term NISQ devices, gate overhead partially offsets this advantage\. Two observations are important\. First, the quantum\-inspired classical simulation used here captures the*algorithmic*advantage: the Grover amplification structure achieves𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)scaling even classically via amplitude annealing\. Second, as fault\-tolerant hardware matures, the full quadratic speedup will be realised without gate\-overhead penalty\. QuantFPFlow is explicitly designed as a quantum\-ready algorithm\.

### 6\.3Limitations and Future Work

#### Dimensionality of QAE\.

The QAE operates on a 1D projection of the state space\. Extension to fulldd\-dimensional QAE requires2n​d2^\{nd\}grid points\. Future work should explore tensor network representations or variational quantum eigensolvers \(VQE\) for high\-dimensional FP estimation\.

#### Approximate FP potential\.

The FP potentialV​\(𝐱\)V\(\\mathbf\{x\}\)is approximated via the criticVϕV\_\{\\phi\}\. A learned FP potential with convergence guarantees would strengthen the theoretical foundation\.

#### Real quantum hardware\.

Implementation on IBM Quantum or IonQ via Qiskit/PennyLane is deferred to future work, along with a full noise analysis for NISQ circuits\.

#### Extensions\.

Natural extensions include: multi\-agent mean\-field FP games\[[7](https://arxiv.org/html/2605.16429#bib.bib7)\], molecular dynamics agents for drug discovery, and integration with diffusion\-based world models\.

## 7Conclusion

We introduced QuantFPFlow, a reinforcement learning algorithm that integrates quantum amplitude estimation into the Fokker–Planck framework for continuous\-space policy optimisation\. QuantFPFlow achieves: \(1\) a proven quadratic query speedup𝒪​\(1/ε\)\\mathcal\{O\}\(1/\\varepsilon\)over classical𝒪​\(1/ε2\)\\mathcal\{O\}\(1/\\varepsilon^\{2\}\); \(2\) a 10\.4 % relative improvement in global optimum discovery over SAC on a multimodal continuous control benchmark; \(3\) sustained policy entropy of 6\.5 nats versus SAC’s collapse to 1\.5 nats, confirming the FP diffusion matching mechanism prevents premature convergence; and \(4\) favourable dimensionality scaling𝒪​\(d0\.35\)\\mathcal\{O\}\(d^\{0\.35\}\)versus classical FP𝒪​\(d0\.76\)\\mathcal\{O\}\(d^\{0\.76\}\)\. These results establish QuantFPFlow as a principled quantum\-enhanced RL algorithm with clear theoretical grounding, empirical advantages on exploration\-critical tasks, and a direct path to fault\-tolerant quantum hardware implementation\.

## References

- \[1\]Gilles Brassard, Peter Hoyer, Michele Mosca, and Alain Tapp\.Quantum amplitude amplification and estimation\.arXiv preprint quant\-ph/0005055, 2000\.
- \[2\]Vedran Dunjko, Jacob M Taylor, and Hans J Briegel\.Advances in quantum reinforcement learning\.In2017 IEEE international conference on systems, man, and cybernetics \(SMC\), pages 282–287\. IEEE, 2017\.
- \[3\]Wendell H Fleming and H Mete Soner\.Controlled Markov processes and viscosity solutions\.Springer, 2006\.
- \[4\]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InInternational conference on machine learning, pages 1861–1870\. Pmlr, 2018\.
- \[5\]Sofiene Jerbi, Lea M Trenkwalder, Hendrik Poulsen Nautrup, Hans J Briegel, and Vedran Dunjko\.Quantum enhancements for deep reinforcement learning in large spaces\.PRX Quantum, 2\(1\):010328, 2021\.
- \[6\]Hilbert J Kappen\.Path integrals and symmetry breaking for optimal control theory\.Journal of statistical mechanics: theory and experiment, 2005\(11\):P11011–P11011, 2005\.
- \[7\]Jean\-Michel Lasry and Pierre\-Louis Lions\.Mean field games\.Japanese journal of mathematics, 2\(1\):229–260, 2007\.
- \[8\]Timothy Paul Lillicrap, Jonathan James Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daniel Pieter Wierstra\.Continuous control with deep reinforcement learning, September 15 2020\.US Patent 10,776,692\.
- \[9\]Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell\.Curiosity\-driven exploration by self\-supervised prediction\.InInternational conference on machine learning, pages 2778–2787\. PMLR, 2017\.
- \[10\]Hannes Risken\.Fokker\-planck equation\.InThe Fokker\-Planck equation: methods of solution and applications, pages 63–95\. Springer, 1989\.
- \[11\]Emanuel Todorov\.Efficient computation of optimal actions\.Proceedings of the national academy of sciences, 106\(28\):11478–11483, 2009\.

## Appendix ASupplementary: Mode Collapse Analysis

[Figure8](https://arxiv.org/html/2605.16429#A1.F8)provides additional evidence that QuantFPFlow avoids the mode collapse that afflicts SAC and DDPG\. The left panel compares policy distributions on the multimodal FP potential: SAC collapses to a single mode atx≈1\.8x\\approx 1\.8, while QuantFPFlow covers both modes ofρ∗​\(x\)\\rho^\{\*\}\(x\)\. The centre panel shows KL divergence from the true stationary distribution over 300 training episodes: QuantFPFlow KL decreases to near\-zero while SAC plateaus atKL≈0\.70\\mathrm\{KL\}\\approx 0\.70\. The right panel confirms QuantFPFlow’s superior state\-space coverage at all density thresholdsτ\\tau\.

![Refer to caption](https://arxiv.org/html/2605.16429v1/fig_mode_collapse_analysis.png)Figure 8:Mode collapse analysis\.Left: policy distribution vs\. trueρ∗​\(x\)\\rho^\{\*\}\(x\); SAC collapses to a single mode while QuantFPFlow covers both\.Centre: KL divergence fromρ∗\\rho^\{\*\}during training; QuantFPFlow converges to near\-zero while SAC plateaus atKL≈0\.70\\mathrm\{KL\}\\approx 0\.70\.Right: state\-space coverageP​\(ρ\>τ\)P\(\\rho\>\\tau\)as a function of density thresholdτ\\tau; QuantFPFlow maintains higher coverage at all thresholds\.
## Appendix BHyperparameters

[Table2](https://arxiv.org/html/2605.16429#A2.T2)lists all hyperparameters used in the experiments\.

Table 2:QuantFPFlow hyperparameters\.HyperparameterValueExploration coefficientα\\alpha0\.5FP inverse temperatureβ\\beta1\.5Diffusion coefficientDD0\.3Actor learning rateηa\\eta\_\{a\}5×10−35\\times 10^\{\-3\}Critic learning rateηc\\eta\_\{c\}1×10−21\\times 10^\{\-2\}Discount factorγ\\gamma0\.99Momentumβm\\beta\_\{m\}0\.9Entropy coefficient0\.15QAE qubits7QAE cache refresh10 stepsGrover iterations5Annealing steps6Episodes400Episode horizonHH200State dimensiondd4Action dimensionmm2