流形约束的初始噪声优化：实现高效的生成模型对齐

# Manifold-Constrained Initial Noise Optimization for Efficient Generative Model Alignment
Source: [https://arxiv.org/html/2610.00365](https://arxiv.org/html/2610.00365)
Jinho Chang & Jong Chul Ye††thanks:Corresponding author\.Affiliation:Kim Jaechul Graduate School of AIAffiliation:KAISTAffiliation:Seoul, South KoreaEmail:[\{jinhojsk515,jong\.ye\}@kaist\.ac\.kr](mailto:)

###### Abstract

Recent advances in distillation and flow\-map models have enabled deterministic one\- or few\-step generation for high\-quality data, facilitating a new branch of reward alignment approaches that directly optimize the initial noise from a Gaussian distribution\. However, most existing initial\-noise optimization methods rely on first\-order gradient information, which is either inapplicable or suffers from instability and inefficiency in black\-box reward scenarios\. Here, we introduce ZeNOVA \(Zeroth\-orderNoiseOptimization withValueAnnealing\), a stable and efficient initial noise alignment method in a gradient\-free manner\. Specifically, we address existing algorithms’ major challenge in black\-box scenarios through annealed soft\-value guidance, manifold\-constrained hyperspherical Langevin dynamics, and Metropolis\-Hastings jumping\. Extensive experiments on image and video generative models show that ZeNOVA outperforms all evaluated zeroth\-order baselines by optimizing the initial noise toward higher rewards substantially more stably while exploiting the geometry of the Gaussian prior, demonstrating its practical applicability to various black\-box reward alignment\.

## 1Introduction

Reward alignment of a pre\-trained generative model aims to align its output distribution to admit higher external reward while minimizing its deviation from the original prior\. A wide range of fine\-tuning based approaches are presented grounded on several theoretical support, including entropy\-regularized reinforcement learning\([Black et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib43);[Fan et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib14);[Liu et al\., 2025a](https://arxiv.org/html/2610.00365#bib.bib12);[He et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib13);[Zheng et al\., 2026a](https://arxiv.org/html/2610.00365#bib.bib21)\), GFlowNets\([Zhang et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib16);[Liu et al\., 2025c](https://arxiv.org/html/2610.00365#bib.bib15);[Liu et al\., 2025b](https://arxiv.org/html/2610.00365#bib.bib11)\), and stochastic optimal control\([Domingo\-Enrich et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib10)\)\. Inference\-time methods employing various techniques such as diffusion posterior sampling\([Chung et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib25);[Ye et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib18);[Yu et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib17);[He et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib20);[Song et al\., 2023a](https://arxiv.org/html/2610.00365#bib.bib19)\)and path integral control\([Huang et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib45);[Kim et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib42)\)have also been proposed to implement the same effect without additional training\.

However, these works have to assume stochastic sampling trajectories involving many intermediate steps for their methods to analytically admit the reward\-tilted distribution and thereby satisfy both the prior and high reward\. Consequently, naively extending the results of these works to deterministic generative models is either limited or does not guarantee its behavior as theoretically designed\. Moreover, recent advances in generative modeling have led to the development of few\-step or one\-step models including distilled models\([Yin et al\., 2024b](https://arxiv.org/html/2610.00365#bib.bib40);[Yin et al\., 2024a](https://arxiv.org/html/2610.00365#bib.bib34);[Sauer et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib33)\)and flow\-map models\([Song et al\., 2023b](https://arxiv.org/html/2610.00365#bib.bib28);[Geng et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib41)\), which reduces the effect of control in between intermediate sampling process\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_zenova_main.png)Figure 1:Overview of ZeNOVA\.Naive initial\-noise optimization suffers from limited exploration and unstable updates under inaccurate zeroth\-order guidance from raw reward signal\. ZeNOVA combines annealed soft\-value guidance, hyperspherical updates, and Metropolis\-Hastings jumping to enable stable local refinement and global exploration while respecting the Gaussian prior geometry\.These recent advances facilitate a separate approach of initial\-noise optimization as a practical route to inference\-time reward alignment; a pretrained deterministic generator can produce different outputs by changing its Gaussian input, without updating its parameters\([Eyring et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib36);[Min et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib32);[Yoon et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib31)\)\. Most existing approaches, however, obtain their optimization directions by differentiating through both the generator and the reward computation\. This requirement limits their use when the reward is non\-differentiable or available only through black\-box evaluations, and can impose substantial computational overhead even when gradients are accessible\. Estimating directions from reward queries offers a gradient\-free alternative, but useful alignment also requires these estimates to support stable and efficient optimization in the high\-dimensional noise space\. In particular, seeking higher rewards should account for the Gaussian noise input geometry on which the generator was trained, which still remains challenging with optimizations in gradient\-free manner\.

Here, we proposeZeNOVA, a stable and efficient gradient\-free initial\-noise alignment method toward black\-box objectives\. Specifically, as illustrated in Figure[1](https://arxiv.org/html/2610.00365#S1.F1), we identify the major challenges faced by existing initial noise optimization algorithms in black\-box scenarios, and address them through three contributions of annealed soft\-value guidance, manifold\-constrained hyperspherical Langevin dynamics, and Metropolis\-Hastings jumping\. Together, these components require only forward evaluations of the generator and reward model, making ZeNOVA applicable to black\-box alignment without backpropagation\. Experiments on image and video generative models show that ZeNOVA achieves higher rewards than all evaluated zeroth\-order baselines while maintaining more stable initial\-noise optimization, supporting its practical use across diverse reward alignment tasks\.

## 2Preliminaries

Reward\-tilted Distribution\.Whenprefp^\{\\mathrm\{ref\}\}andpθp^\{\\theta\}denote the distribution from the pretrained model and the fine\-tuned model with parametersθ\\theta, Entropy\-regularized RL formulates the alignment objective as balancing expected terminal reward against divergence from the reference process with a balancing parameterλ\>0\\lambda\>0:

ℒ\(θ\)=𝔼𝒙0∼p0θ\[r\(𝒙0\)\]−λDKL\(pθ∥pref\),\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{\{\\bm\{x\}\}\_\{0\}\\sim p\_\{0\}^\{\\theta\}\}\\left\[r\(\{\\bm\{x\}\}\_\{0\}\)\\right\]\-\\lambda D\_\{\\mathrm\{KL\}\}\\left\(p^\{\\theta\}\\,\\\|\\,p^\{\\mathrm\{ref\}\}\\right\),\(1\)In many algorithms for solving Eq\. \([1](https://arxiv.org/html/2610.00365#S2.E1)\) in diffusion and flow\-matching models\([Fan et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib14)\), the trajectory\-level KL admits a decomposition into a sum or integral of local policy divergences along the sampling trajectory\. Whent∈\[0,1\]t\\in\[0,1\]denotes the noise timestep and𝒙0\{\\bm\{x\}\}\_\{0\}denotes the clean sample, it is known that the optimal policy for Eq\. \([1](https://arxiv.org/html/2610.00365#S2.E1)\) induces the following marginal distribution\([Uehara et al\., 2024b](https://arxiv.org/html/2610.00365#bib.bib35)\)with*Soft value function*V⁡\(𝒙t\)V\(\{\\bm\{x\}\}\_\{t\}\), which represents the expected reward and KL penalty term over the optimal policy from𝒙t\{\\bm\{x\}\}\_\{t\}:

pt⋆​\(𝒙t\)∝ptref​\(𝒙t\)​e1λ​V​\(𝒙t\),V⁡\(𝒙t\):=λ​log⁡𝔼ptref​\[exp⁡\(r⁡\(𝒙0\)λ\)\|𝒙t\]\\displaystyle p\_\{t\}^\{\\star\}\(\{\\bm\{x\}\}\_\{t\}\)\\propto p\_\{t\}^\{\\mathrm\{ref\}\}\(\{\\bm\{x\}\}\_\{t\}\)e^\{\\frac\{1\}\{\\lambda\}V\(\{\\bm\{x\}\}\_\{t\}\)\},\\quad V\(\{\\bm\{x\}\}\_\{t\}\):=\\lambda\\log\\mathbb\{E\}\_\{p\_\{t\}^\{\\mathrm\{ref\}\}\}\\\!\\left\[\\exp\\\!\\left\(\\frac\{r\(\{\\bm\{x\}\}\_\{0\}\)\}\{\\lambda\}\\right\)\\middle\|\{\\bm\{x\}\}\_\{t\}\\right\]\(2\)Substituting the relationship between the output of a diffusion/flow\-matching model and the score function\([Song et al\., 2021](https://arxiv.org/html/2610.00365#bib.bib27)\), Eq\. \([2](https://arxiv.org/html/2610.00365#S2.E2)\) shows that the difference between the optimal and reference models shall be proportional to the gradient of this value function∇𝒙tV​\(𝒙t\)\\nabla\_\{\{\\bm\{x\}\}\_\{t\}\}V\(\{\\bm\{x\}\}\_\{t\}\)\.

Among several mathematical frameworks, Path\-Integral Control\(PIC\)\([Kappen, 2005](https://arxiv.org/html/2610.00365#bib.bib47);[Huang et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib45);[Uehara et al\., 2024a](https://arxiv.org/html/2610.00365#bib.bib46)\)allows us to express the optimal policy and soft value function through weighted samples of the reference dynamics:

∇𝒙tV​\(𝒙t\)\\displaystyle\\nabla\_\{\{\\bm\{x\}\}\_\{t\}\}V\(\{\\bm\{x\}\}\_\{t\}\)=limd​t→0λg⁡\(t\)​d​t​𝔼Pref​\[er⁡\(𝒙0\)/λ​d​𝒘t∣𝒙t\]𝔼Pref​\[er⁡\(𝒙0\)/λ∣𝒙t\]\\displaystyle=\\lim\_\{dt\\to 0\}\\frac\{\\lambda\}\{g\(t\)dt\}\\frac\{\\mathbb\{E\}\_\{P^\{\\mathrm\{ref\}\}\}\\\!\\left\[e^\{r\(\{\\bm\{x\}\}\_\{0\}\)/\\lambda\}d\{\\bm\{w\}\}\_\{t\}\\mid\{\\bm\{x\}\}\_\{t\}\\right\]\}\{\\mathbb\{E\}\_\{P^\{\\mathrm\{ref\}\}\}\\\!\\left\[e^\{r\(\{\\bm\{x\}\}\_\{0\}\)/\\lambda\}\\mid\{\\bm\{x\}\}\_\{t\}\\right\]\}\(3\)This representation weights sampled noise increments by terminal desirability, allowing the control to be estimated from samples of the reference model and terminal reward evaluations alone\.

Reward\-tilted Noise Distribution\.Consider reward alignment of the input initial noise fed to the generative models, without assuming access to reward gradients\. Let𝒛∼p0:=𝒩⁡\(0,𝐈d\)\{\\bm\{z\}\}\\sim p\_\{0\}:=\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\_\{d\}\)denote thedd\-dimensional input noise, andGθ​\(𝒛,𝒄\)G\_\{\\theta\}\(\{\\bm\{z\}\},\{\\bm\{c\}\}\)be a generator conditioned on𝒄\{\\bm\{c\}\}, leading to the reward in noise space and the reward\-titled noise distribution:

R⁡\(𝒛\):=r⁡\(Gθ​\(𝒛,𝒄\)\),p⋆​\(𝒛\)∝p0​\(𝒛\)​exp⁡\(R⁡\(𝒛\)/λ\)R\(\{\\bm\{z\}\}\):=r\(G\_\{\\theta\}\(\{\\bm\{z\}\},\{\\bm\{c\}\}\)\),\\quad p^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)\\exp\\left\(R\(\{\\bm\{z\}\}\)/\\lambda\\right\)\(4\)To perform optimization of the initial noise𝒛\{\\bm\{z\}\}, the simplest approach is to search for a maximizer ofR⁡\(𝒛\)R\(\{\\bm\{z\}\}\)over the entire supportℝd\\mathbb\{R\}^\{d\}\. But this is not only prone to mode collapse toward a few high\-reward regions, it also disregards the Gaussian prior distribution of𝒛\{\\bm\{z\}\}on which the latter generative modelGθG\_\{\\theta\}is trained, leading to reward hacking and degradation of the image quality\. While several empirical regularizers were suggested to mitigate this\([Eyring et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib36);[Shi et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib24)\), whenRRis differentiable with respect to its input𝒛\{\\bm\{z\}\}, sampling from \([4](https://arxiv.org/html/2610.00365#S2.E4)\) is straightforward because its score function is tractable\. Hence, the following Langevin dynamics hasp⋆p^\{\\star\}as its stationary distribution\.

d​𝒛=\(−𝒛\+1λ​∇𝒛R​\(𝒛\)\)​d​t\+2​d​𝒘d\{\\bm\{z\}\}=\\left\(\-\{\\bm\{z\}\}\+\\frac\{1\}\{\\lambda\}\\nabla\_\{\{\\bm\{z\}\}\}R\(\{\\bm\{z\}\}\)\\right\)dt\+\\sqrt\{2\}\\,d\{\\bm\{w\}\}\(5\)This observation is utilized in several gradient\-based initial\-noise alignment algorithms\([Min et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib32);[Yoon et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib31)\)\.

## 3Method

Our interest is a more challenging setting in whichRRcan be evaluated but∇𝒛R\\nabla\_\{\{\\bm\{z\}\}\}Ris unavailable\. Here, we first show why a direct zeroth\-order extension of gradient\-based Langevin optimization is inadequate, and then describe the proposed contributions of ZeNOVA to address those challenges\.

### 3\.1Challenges for Black\-box Objectives\.

A natural zeroth\-order baseline\.For a small perturbation scaleδ\>0\\delta\>0, a classic gradient estimation method with function evaluations according to Stein’s Lemma\([Stein, 1981](https://arxiv.org/html/2610.00365#bib.bib23)\)is

𝔼ϵ​\[1δ​R​\(𝒛\+δ​ϵ\)​ϵ\]=∇𝒛𝔼ϵ∼𝒩⁡\(0,𝐈\)​\[R⁡\(𝒛\+δ​ϵ\)\]≈∇𝒛R​\(𝒛\),\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\}\\left\[\\frac\{1\}\{\\delta\}R\(\{\\bm\{z\}\}\+\\delta\{\\bm\{\{\\epsilon\}\}\}\)\{\\bm\{\{\\epsilon\}\}\}\\right\]=\\nabla\_\{\{\\bm\{z\}\}\}\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\}\[R\(\{\\bm\{z\}\}\+\\delta\{\\bm\{\{\\epsilon\}\}\}\)\]\\approx\\nabla\_\{\{\\bm\{z\}\}\}R\(\{\\bm\{z\}\}\),\(6\)which can be estimated fromNNparticles as

𝒈^​\(𝒛\)=1N​∑n=1N\[1δ​\(R⁡\(𝒛\+δ​ϵ\(n\)\)−R¯\)​ϵ\(n\)\]\\hat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\)=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\left\[\\frac\{1\}\{\\delta\}\\left\(R\(\{\\bm\{z\}\}\+\\delta\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\)\-\\bar\{R\}\\right\)\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\\right\]\(7\)whereR¯\\bar\{R\}is an optional control variate to reduce variance, typically set as the mean ofRR\. This is the familiar Gaussian\-search estimator used in evolution strategies such as NES\([Wierstra et al\., 2014](https://arxiv.org/html/2610.00365#bib.bib44)\)\. Therefore, the most direct gradient\-free extension of \([5](https://arxiv.org/html/2610.00365#S2.E5)\) is to replace∇𝒛R​\(𝒛\)\\nabla\_\{\{\\bm\{z\}\}\}R\(\{\\bm\{z\}\}\)by𝒈^​\(𝒛\)\\hat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\)\. We refer to this construction as*naive zeroth\-order\(ZO\) Langevin*\.

However, while this naive alternative seems theoretically sound, it performs poorly when applied to practical initial\-noise optimization\. We analyze the reasons for this failure through the following three failure modes\.

Failure mode I: Nature of Black\-box rewards\.Many black\-box rewards can be discrete, quantized, thresholded, ranking\-based, or simply locally constant\. In this case, the gradient \(including its estimator\) becomes00and does not carry any signal for the optimization\. Naively increasingδ\\deltain Eq\. \([7](https://arxiv.org/html/2610.00365#S3.E7)\) can search a broader range to recover reward signal, but this makes the estimator more deviate from the ground truth gradient\.

Failure mode II: Destabilization from few\-particle estimates\.To make the zeroth\-order estimator accurate, the required number of particlesNNhas to grow proportionally to the number of dimensionsdd\. However, in most high\-dimensional noise spaces for recent generative models, a limited computational budget usually admitsN≪dN\\ll d, which introduces unavoidable inaccuracy in the estimator\.

Empirically, we observe that this inaccuracy goes beyond making the optimization process less efficient, and frequently drives𝒛\{\\bm\{z\}\}completely out of distribution, leading to sample collapse\. Figure[2](https://arxiv.org/html/2610.00365#S3.F2)shows an example of reward\-tilted Langevin dynamics with Pickscore\([Kirstain et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib38)\)failing when the actual reward gradient is replaced by𝒈^​\(𝒛\)\\hat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\)\. Note that when this happens, the norm of𝒛\{\\bm\{z\}\}drifts away from the expected value for a Gaussian distribution, rapidly degrading the model output\.

Failure mode III: Hazard of bad initial states\.Even if the aforementioned issues are resolved, a sequential and incremental motion of Langevin dynamics suffers from moving between separated modes slowly, as it requires traversing intermediate low\-likelihood regions\. This makes the finite\-budget trajectories sensitive to their initialization\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure1-1.png)Figure 2:Replacing real gradients in Langevin dynamics\(grey\) with zeroth\-order estimates\(red\)\.
![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure1-2.png)
Figure 3:2D toy experiment for sampling from reward\-tilted target with zeroth\-order gradient estimation\.

### 3\.2ZeNOVA: Noise Optimization for Black\-box Rewards with Value Annealing

Zeroth\-order value\-gradient estimator\.Note that when the reward gradient control term is removed from Eq\. \([5](https://arxiv.org/html/2610.00365#S2.E5)\), the remainder is an Ornstein\-Uhlenbeck \(OU\) process\([Øksendal, 2003](https://arxiv.org/html/2610.00365#bib.bib30)\)that preservesp0=𝒩⁡\(0,𝐈\)p\_\{0\}=\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\. We take this process as a reference and consider a soft value that aggregates future terminal rewards\. The OU process admits an exact Gaussian transition over any time intervalh\>0h\>0: settingσ2=1−e−2​h\\sigma^\{2\}=1\-e^\{\-2h\}, we can express its transition using a single kernel,

qσ​\(𝒛′∣𝒛\)=𝒩⁡\(𝒛′,1−σ2​𝒛,σ2​𝐈\),0<σ<1\.q\_\{\\sigma\}\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)=\{\\mathcal\{N\}\}\\\!\\left\(\{\\bm\{z\}\}^\{\\prime\};\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\},\\sigma^\{2\}\{\\mathbf\{I\}\}\\right\),\\qquad 0<\\sigma<1\.\(8\)The parameterσ\\sigmaspecifies the*horizon*of the OU process: largeσ\\sigmanearly forgets the current state and broadly explores the reference prior, whereas smallσ\\sigmaconcentrates around the current state\.

Following the previous work in the soft RL\([Uehara et al\., 2024b](https://arxiv.org/html/2610.00365#bib.bib35)\)and PIC\([Kappen, 2005](https://arxiv.org/html/2610.00365#bib.bib47)\), we define the soft value as the log\-expectation of the terminal rewardexp⁡\(R/λ\)\\exp\(R/\\lambda\)under this transition kernel:

Vσ\(𝒛\):=λlog𝔼𝒛′∼qσ\(⋅∣𝒛\)\[exp\(R⁡\(𝒛′\)λ\)\]V\_\{\\sigma\}\(\{\\bm\{z\}\}\):=\\lambda\\log\\mathbb\{E\}\_\{\{\\bm\{z\}\}^\{\\prime\}\\sim q\_\{\\sigma\}\(\\cdot\\mid\{\\bm\{z\}\}\)\}\\left\[\\exp\\left\(\\frac\{R\(\{\\bm\{z\}\}^\{\\prime\}\)\}\{\\lambda\}\\right\)\\right\]\(9\)Then, we propose to sample from an intermediate distribution obtained by exponentially tilting the prior with this soft value, instead of the raw rewardRR:

pσ⋆​\(𝒛\)∝p0​\(𝒛\)​exp⁡\(Vσ​\(𝒛\)λ\),p\_\{\\sigma\}^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)\\exp\\left\(\\frac\{V\_\{\\sigma\}\(\{\\bm\{z\}\}\)\}\{\\lambda\}\\right\),\(10\)This choice is motivated by two properties of the value function: First, even when the reward function is poorly behaved \(*e\.g\.*, discrete or piecewise constant\), the value function is defined as its expectation over a Gaussian kernel, and therefore continuous and well\-behaved\([Jensen et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib22)\)\. Moreover, this intermediate distribution induced by the soft\-value function has an exact relationship top⋆p^\{\\star\}:

###### Proposition 1\(Interpolation of target distribution\)\.

Let reward\-tilted distributionp⋆p^\{\\star\}be defined as Eq\. \([4](https://arxiv.org/html/2610.00365#S2.E4)\) and soft\-value\-tilted distributionpσ⋆p\_\{\\sigma\}^\{\\star\}as Eq\. \([10](https://arxiv.org/html/2610.00365#S3.E10)\)\. Thenpσ⋆p\_\{\\sigma\}^\{\\star\}is equal top⋆p^\{\\star\}passed through a Gaussian kernel of Eq\. \([8](https://arxiv.org/html/2610.00365#S3.E8)\):

pσ⋆​\(𝒛\)∝∫qσ​\(𝒛∣𝒛′\)​p⋆​\(𝒛′\)​d​𝒛′\.p\_\{\\sigma\}^\{\\star\}\(\{\\bm\{z\}\}\)\\propto\\int q\_\{\\sigma\}\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)\\,p^\{\\star\}\(\{\\bm\{z\}\}^\{\\prime\}\)\\,d\{\\bm\{z\}\}^\{\\prime\}\.\(11\)

This implies thatpσ⋆→p0p\_\{\\sigma\}^\{\\star\}\\to p\_\{0\}asσ→1\\sigma\\to 1, andpσ⋆→p⋆p\_\{\\sigma\}^\{\\star\}\\to p^\{\\star\}asσ→0\\sigma\\to 0, where the proof is in Appendix[A\.1](https://arxiv.org/html/2610.00365#A1.SS1)\. Motivated by this interpolation, we use a decreasing horizon schedule1\>σ1\>σ2\>⋯\>σK\>01\>\\sigma\_\{1\}\>\\sigma\_\{2\}\>\\cdots\>\\sigma\_\{K\}\>0with the aim of tracking the sequencepσ1⋆,pσ2⋆,…,pσK⋆p\_\{\\sigma\_\{1\}\}^\{\\star\},p\_\{\\sigma\_\{2\}\}^\{\\star\},\\ldots,p\_\{\\sigma\_\{K\}\}^\{\\star\}in a coarse\-to\-fine manner; larger horizons aggregate reward information over a broad neighborhood, while smaller horizons bring the target closer to the original reward tilt and support local refinement\.

Crucially,∇Vσ\\nabla V\_\{\\sigma\}can be estimated without differentiatingRR\. Applying the Gaussian score identity to \([9](https://arxiv.org/html/2610.00365#S3.E9)\), we obtain

∇𝒛Vσ​\(𝒛\)=λ​1−σ2σ​𝔼ϵ​\[exp⁡\(R⁡\(1−σ2​𝒛\+σ​ϵ\)/λ\)​ϵ\]𝔼ϵ​\[exp⁡\(R⁡\(1−σ2​𝒛\+σ​ϵ\)/λ\)\]\.\\nabla\_\{\{\\bm\{z\}\}\}V\_\{\\sigma\}\(\{\\bm\{z\}\}\)=\\frac\{\\lambda\\sqrt\{1\-\\sigma^\{2\}\}\}\{\\sigma\}\\frac\{\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\}\\left\[\\exp\(R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}\)/\\lambda\)\\,\{\\bm\{\{\\epsilon\}\}\}\\right\]\}\{\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\}\\left\[\\exp\(R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}\)/\\lambda\)\\right\]\}\.\(12\)See Appendix[A\.3](https://arxiv.org/html/2610.00365#A1.SS3)for the derivation\. WithNNreward evaluations, we use

𝒈^σ​\(𝒛\)\\displaystyle\\widehat\{\{\\bm\{g\}\}\}\_\{\\sigma\}\(\{\\bm\{z\}\}\):=λ​1−σ2σ​∑n=1Nexp⁡\(R⁡\(1−σ2​𝒛\+σ​ϵ\(n\)\)/λ\)∑j=1Nexp⁡\(R⁡\(1−σ2​𝒛\+σ​ϵ\(j\)\)/λ\)​ϵ\(n\)ϵ\(n\)∼𝒩⁡\(0,𝐈\)\.\\displaystyle:=\\frac\{\\lambda\\sqrt\{1\-\\sigma^\{2\}\}\}\{\\sigma\}\\sum\_\{n=1\}^\{N\}\\frac\{\\exp\(R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\)/\\lambda\)\}\{\\sum\_\{j=1\}^\{N\}\\exp\(R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}^\{\(j\)\}\)/\\lambda\)\}\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\\qquad\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\.\(13\)The normalized exponential weights are implemented with the usual log\-sum\-exp stabilization\. For a fixedσ\\sigma, we can replace the reward gradient in Eq\. \([5](https://arxiv.org/html/2610.00365#S2.E5)\) with Eq\. \([13](https://arxiv.org/html/2610.00365#S3.E13)\)\.

Hyperspherical Langevin dynamics\.Even for soft value gradient functions, their Monte Carlo estimate can still be noisy for smallNN\. Meanwhile, the standard Gaussian prior is rotationally symmetric, and its density depends only on the radius\. Moreover, in high dimensions, Gaussian samples exhibit a uniform likelihood for any direction, whereas their norm concentrates within a narrow range aroundd\\sqrt\{d\}\. Indeed, as described in Figure[2](https://arxiv.org/html/2610.00365#S3.F2), failure of the update with zeroth\-order estimation is often accompanied by a drift of the noise norm away from this Gaussian typical range\.

This observation suggests treating the two degrees of freedom separately and formulating the Langevin dynamics in hyperspherical coordinates, separating the prior\-sensitive radial motion from the angular motion\. Note that sampling𝒛\{\\bm\{z\}\}from the standard Gaussian can be decomposed as:

𝒛=ρ​𝒖whereρ∼χd,𝒖∈Unif​\(𝕊d−1\),\{\\bm\{z\}\}=\\rho\{\\bm\{u\}\}\\quad\\text\{where\}\\quad\\rho\\sim\\chi\_\{d\},\{\\bm\{u\}\}\\in\\text\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\),\(14\)
Therefore, for a given current state𝒛k\{\\bm\{z\}\}\_\{k\}, we decompose it into its normρk\\rho\_\{k\}and its unit direction vector𝒖k\{\\bm\{u\}\}\_\{k\}, and separately perform*angular*Langevin update and*radial*Langevin update\.

Angular Langevin on the sphere\.Let𝐏𝒖⟂:=𝐈−𝒖​𝒖⊤\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\}^\{\\perp\}:=\{\\mathbf\{I\}\}\-\{\\bm\{u\}\}\{\\bm\{u\}\}^\{\\top\}denote the projection onto the tangent space of𝒖\{\\bm\{u\}\}\. For𝒈σ​\(𝒛\):=∇𝒛Vσ​\(𝒛\)\{\\bm\{g\}\}\_\{\\sigma\}\(\{\\bm\{z\}\}\):=\\nabla\_\{\{\\bm\{z\}\}\}V\_\{\\sigma\}\(\{\\bm\{z\}\}\), the angular update uses a gradient and noise on a tangent space, followed by spherical exponential map\([Lee, 2018](https://arxiv.org/html/2610.00365#bib.bib26)\):

𝒗k\\displaystyle\{\\bm\{v\}\}\_\{k\}=𝐏𝒖k⟂​\(ηA​ρkλ​𝒈^σk​\(𝒛k\)\+2​ηA​𝝃k\),𝝃k∼𝒩⁡\(0,𝐈\)\\displaystyle=\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\_\{k\}\}^\{\\perp\}\\left\(\\frac\{\\eta\_\{A\}\\rho\_\{k\}\}\{\\lambda\}\\widehat\{\{\\bm\{g\}\}\}\_\{\\sigma\_\{k\}\}\(\{\\bm\{z\}\}\_\{k\}\)\+\\sqrt\{2\\eta\_\{A\}\}\\,\\bm\{\\xi\}\_\{k\}\\right\),\\qquad\\bm\{\\xi\}\_\{k\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\(15\)𝒖k\+1\\displaystyle\{\\bm\{u\}\}\_\{k\+1\}=Exp𝒖k⁡\(𝒗k\)\\displaystyle=\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\_\{k\}\}\(\{\\bm\{v\}\}\_\{k\}\)\(16\)whereExp⁡\(⋅\)\\operatorname\{Exp\}\(\\cdot\)denotes an exponential mapping as follows:

Exp𝒖⁡\(𝒗\)=cos⁡\(‖𝒗‖2\)​𝒖\+sin⁡\(‖𝒗‖2\)​𝒗‖𝒗‖2\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\}\(\{\\bm\{v\}\}\)=\\cos\(\\\|\{\\bm\{v\}\}\\\|\_\{2\}\)\{\\bm\{u\}\}\+\\sin\(\\\|\{\\bm\{v\}\}\\\|\_\{2\}\)\\frac\{\{\\bm\{v\}\}\}\{\\\|\{\\bm\{v\}\}\\\|\_\{2\}\}\(17\)
Radial Langevin\.Since the reference radial distribution isρ∼χd\\rho\\sim\\chi\_\{d\}, a radial update step is

ρk\+1=ρk\+ηR​\(d−1ρk−ρk\+1λ​𝒖k⊤​𝒈^σk​\(𝒛k\)\)\+2​ηR​ξk,ξk∼𝒩⁡\(0,1\)\.\\rho\_\{k\+1\}=\\rho\_\{k\}\+\\eta\_\{R\}\\left\(\\frac\{d\-1\}\{\\rho\_\{k\}\}\-\\rho\_\{k\}\+\\frac\{1\}\{\\lambda\}\{\\bm\{u\}\}\_\{k\}^\{\\top\}\\widehat\{\{\\bm\{g\}\}\}\_\{\\sigma\_\{k\}\}\(\{\\bm\{z\}\}\_\{k\}\)\\right\)\+\\sqrt\{2\\eta\_\{R\}\}\\,\\xi\_\{k\},\\qquad\\xi\_\{k\}\\sim\{\\mathcal\{N\}\}\(0,1\)\.\(18\)The radius remains stochastic and reward\-adaptive, while noisy high\-dimensional guidance is regularized by an analytic one\-dimensional prior score rather than by an explicit norm constraint\.

For any fixedσ\\sigma, this continuous spherical\-radial Langevin dynamics has \([10](https://arxiv.org/html/2610.00365#S3.E10)\) as its stationary distribution ofρ​𝒖\\rho\{\\bm\{u\}\}\. This spherical\-radial Langevin update allows us to design their dynamics separately\(*e\.g\.*, small step size for radial update\), which stabilizes noisy zeroth\-order guidance without freezing the radius or relying on an explicit shell projection\.

Metropolis\-Hastings Jumping\.Since incremental updates may be a somewhat inevitable limitation of Langevin dynamics to simulate an SDE, we briefly examine a fundamentally different alternative of utilizing the Metropolis–Hastings\(MH\) algorithm:

###### Proposition 2\(Metropolis\-Hastings algorithm for reward\-tilted distribtuion\)\.

The Metropolis\-Hastings chain with proposal𝐳′∼p0\{\\bm\{z\}\}^\{\\prime\}\\sim p\_\{0\}and acceptance probabilityα⁡\(𝐳,𝐳′\)=min⁡\{1,exp⁡\(\(R⁡\(𝐳′\)−R⁡\(𝐳\)\)/λ\)\}\\alpha\(\{\\bm\{z\}\},\{\\bm\{z\}\}^\{\\prime\}\)=\\min\\\{1,\\exp\(\(R\(\{\\bm\{z\}\}^\{\\prime\}\)\-R\(\{\\bm\{z\}\}\)\)/\\lambda\)\\\}converges to the reward\-tilted distributionp⋆​\(𝐳\)∝p0​\(𝐳\)​eR⁡\(𝐳\)/λp^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}\)/\\lambda\}\.

See Appendix[A\.2](https://arxiv.org/html/2610.00365#A1.SS2)for the derivation\. MH chain targets the same distribution with Langevin dynamics and provides global exploration, but it is inefficient as a stand\-alone sampler: Namely, for a finite number of proposals, the MH algorithm is strictly worse than Best\-of\-K\(BoK\) sampling\. Moreover, as the sample reward increases, every proposal is drawn afresh regardless of the current high\-reward state, and the MH algorithm rejects almost all proposals\. Nonetheless, this has a complementary behavior against Langevin dynamics, allowing occasional global moves to new reward modes with much smaller computation than gradient estimation\. We therefore interleave a small number of independent MH transitions with Langevin updates, dubbed as*MH jumping*\. MH jumping is effective in the early stage of the optimization, after which the annealed soft\-value and hyperspherical Langevin updates perform coarse\-to\-fine refinement\.

ZeNOVA: The Algorithm\.Combining the three components yieldsZeNOVA\(Zeroth\-orderNoiseOptimization withValueAnnealing\), where the complete procedure is summarized in Algorithm[1](https://arxiv.org/html/2610.00365#alg1)of Appendix[B\.1](https://arxiv.org/html/2610.00365#A2.SS1)\. The two principal hyperparameters governing inference\-time computation are the number of particlesNNand the number of sampling iterationKK\. IncreasingNNimproves the Monte Carlo estimate of the soft value gradient, while increasingKKallows the sampler to traverse the interpolation more gradually\. Importantly, ZeNOVA requires only forward evaluations of the generator and reward model\.

## 4Experiments

We evaluate ZeNOVA across a range of deterministic generators and reward functions to demonstrate its effectiveness, scalability, and broad applicability\.

Toy Experiment\.We design a reward\-aligned sampling problem of 2\-D data to identify the effect of existing sampling algorithms under zeroth\-order gradient estimation\. Starting fromp0=𝒩⁡\(0,𝐈\)p\_\{0\}=\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\), we use a synthetic reward with narrow high\-reward bands and aim to sample fromp⋆​\(𝒛\)∝p0​\(𝒛\)​eR⁡\(𝒛\)p^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}\)\}\. Figure[3](https://arxiv.org/html/2610.00365#S3.F3)compares \(a\) Gradient Ascent\(GA\) with fixed\-radius projection\([Kim et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib9)\), GA with radialχd\\chi\_\{d\}regularization loss\([Eyring et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib36)\), GA with exponential\-map updates\([Shi et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib24)\), \(d\) Cartesian Langevin updates, and \(e\) our hyperspherical Langevin updates against \(f\) ground\-truth samples obtained by reward\-weighted resampling of prior samples\. Unlike the other baselines \(a\)\-\(d\) whose samples concentrate in partial high\-reward regions due to their empirical regularizers, our hyperspherical update faithfully reproduces the ground\-truth distribution, without artifacts such as samples far along the reward bands in the naive cartesian update\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative.png)Figure 4:Qualitative image results from SDXL\-Turbo and DMD2\.Table 1:Initial noise reward alignment performance on text\-to\-image models\.Bold: best performance, “\-”: Not available\.ImageRewardPickScoreGenEvalCountingOCRMethodReward↑\\uparrowAes\.↑\\uparrowVal\.↑\\uparrowReward↑\\uparrowAes\.↑\\uparrowVal\.↑\\uparrowReward↑\\uparrowAes\.↑\\uparrowReward↓\\downarrowAes\.↑\\uparrowReward↑\\uparrowAes\.↑\\uparrowSDXL\-Turbobase model1\.05056\.028222\.880921\.53785\.40200\.53210\.53755\.385328\.3755\.69650\.13685\.4256Best\-of\-KK1\.69326\.114823\.282923\.19255\.57470\.89490\.72325\.37172\.95\.70960\.47845\.3303ReNO1\.73786\.039522\.958525\.10315\.57210\.7755\-\-\-\-\-\-ORIGEN1\.81985\.164021\.178924\.23215\.57480\.9286\-\-\-\-\-\-MoNO1\.67816\.055823\.022124\.62505\.55790\.7675\-\-\-\-\-\-ZO Langevin1\.39685\.812922\.845222\.27345\.50920\.74020\.68755\.41236\.0255\.70780\.38235\.3439ZO Exponential Map1\.64526\.050723\.220122\.62305\.54370\.78800\.66735\.39123\.3255\.67770\.42395\.4677NES1\.63546\.079523\.085022\.35465\.50500\.75830\.66135\.40065\.85\.68110\.41955\.3516ZeNOVA \(ours\)1\.80786\.127923\.358423\.45645\.59571\.03290\.73325\.42780\.555\.72220\.51145\.3864DMD2base model0\.57095\.490821\.433920\.44825\.36770\.34380\.45475\.138714\.5755\.14680\.09215\.3661Best\-of\-KK1\.64565\.697022\.167622\.05105\.55110\.89150\.72245\.13871\.855\.15570\.46465\.0526ReNO1\.37895\.423220\.987123\.40255\.41590\.5440\-\-\-\-\-\-ORIGEN1\.27084\.973119\.695921\.85865\.30210\.2489\-\-\-\-\-\-MoNO1\.28165\.362721\.066723\.15075\.41400\.3623\-\-\-\-\-\-ZO Langevin1\.48985\.487721\.846821\.80615\.40430\.78110\.65975\.13889\.7754\.69220\.39615\.3055ZO Exponential Map1\.41005\.593521\.904121\.96115\.55480\.67820\.63835\.14361\.355\.26100\.23515\.3417NES1\.29965\.576321\.799421\.61495\.50070\.61930\.62515\.15311\.9755\.07840\.20735\.3637ZeNOVA \(ours\)1\.76835\.721022\.402222\.63985\.61300\.91870\.72755\.17030\.0755\.19020\.52375\.1575

Text\-to\-Image Generation\.We evaluate initial noise alignment for pretrained text\-to\-image generation with SDXL\-turbo\([Sauer et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib33)\)and DMD2\([Yin et al\., 2024a](https://arxiv.org/html/2610.00365#bib.bib34)\)on five tasks: ImageReward\([Xu et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib29)\), PickScore\([Kirstain et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib38)\), GenEval\([Ghosh et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib37)\), Query object counting\([Qian et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib2)\), and OCR\([Cui et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib1)\)\. The generator parameters remain fixed, and alignment is performed by updating the initial noise\. ImageReward and PickScore provide differentiable preference rewards, while GenEval, Counting, and OCR evaluate compositional alignment, object counting, and text rendering through black\-box reward evaluations\. We additionally report aesthetic score\([Schuhmann, 2022](https://arxiv.org/html/2610.00365#bib.bib39)\)as a separate evaluation metric to assess how task\-specific alignment affects visual quality\. See Appendix[B\.3](https://arxiv.org/html/2610.00365#A2.SS3)for the detailed settings on the reward functions and utilized text prompt datasets\.

We include uncontrolled generation and Best\-of\-K, which selects the best sample amongKKindependently drawn initial noises\. The zeroth\-order baselines comprise ZO Langevin, exponential\-map updates with a ZO gradient estimator, and Natural Evolution Strategies \(NES\)\([Wierstra et al\., 2014](https://arxiv.org/html/2610.00365#bib.bib44)\)\. For differentiable rewards, we also include ReNO\([Eyring et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib36)\), ORIGEN\([Min et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib32)\), and MoNO\([Shi et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib24)\)as baselines that utilize backpropagation through the generator and reward model\.

Table[1](https://arxiv.org/html/2610.00365#S4.T1)shows that ZeNOVA improves over uncontrolled generation, Best\-of\-KK, and every reported zeroth\-order optimization baseline on all five task metrics for both generative models\. Specifically, in black\-box reward tasks, ZeNOVA achieves performance gains of\+0\.046\+0\.046on GenEval and\+0\.088\+0\.088on OCR from the best zeroth\-order baseline, and reduces MAE of counting by around40%40\\%\. These results show improvements over the alternatives on objectives that do not provide gradients\. Note that on ImageReward and PickScore where the reward gradient is accessible, gradient\-based algorithms of ReNO and ORIGEN often attain higher reward than ours\. Here, leveraging the fact that ImageReward and PickScore both measure the same aspect of human preference, we evaluated both runs with the reward not employed during optimization as a validation metric\. We observe that gradient\-based methods are prone to reward hacking, allowing our method to achieve the highest validation metric\.

In terms of image quality, ZeNOVA achieved the highest Aesthetic score across various scenarios, while some unstable zeroth\-order methods and first\-order methods prone to reward hacking exhibited significant drops\. While this is partly because some optimized rewards are already associated with image quality, but the results in other cases suggest that ZeNOVA properly preserves the prior distribution of the original generative model’s input\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_video.png)Figure 5:Qualitative video results from rCM with ZeNOVA, optimizing VideoAlign scores\.Text\-to\-Video Generation\.Due to the increased data dimensionality introduced by the additional temporal axis, a growing interest for models that generate video frames or chunks in a few steps was emerged\. We further applied ZeNOVA to a few\-step distilled text\-to\-video flow\-matching model rCM\([Zheng et al\., 2026b](https://arxiv.org/html/2610.00365#bib.bib8)\)\. Following[Xue et al\. \(2025\)](https://arxiv.org/html/2610.00365#bib.bib7), we employed three VideoAlign\([Liu et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib6)\)benchmark score for Visual Quality \(VQ\), Motion Quality \(MQ\), and Text Alignment \(TA\)\. Text prompts from VidProM\([Wang and Yang, 2024](https://arxiv.org/html/2610.00365#bib.bib5)\)dataset were used\. More details can be found in Appendix[B\.4](https://arxiv.org/html/2610.00365#A2.SS4)\.

Table 2:Quantitative performance on text\-to\-video models\.Bold: Best performance\.VideoAlignMethodVQMQTAAllbase model3\.48541\.36842\.17316\.1997ZeNOVA6\.44643\.31243\.908810\.8146

Table[2](https://arxiv.org/html/2610.00365#S4.T2)and Figure[5](https://arxiv.org/html/2610.00365#S4.F5)show that ZeNOVA improves over uncontrolled generation in all four optimization settings\. These gains demonstrate the scalability of the proposed initial\-noise alignment method for a task with much more dimensionality and complexity, supporting both individual reward objectives and their combined optimization without modifying the pretrained generator\.

## 5Discussion

Effect of Particle Count per Iteration\.Zeroth\-order methods require reward evaluations for a fixed number\(NN\) of particles per iteration to estimate each method’s intended update direction of the initial noise; asNNincreases, this estimate approaches the theoretical ground truth and the performance converges\. In practice, since the number of available reward evaluations is finite, the performance behavior with a small number of particles becomes a critical factor in the practicality of the algorithm\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure2.png)
Figure 6:Effect of particle count per iteration for zeroth\-order methods\.
![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure3.png)
Figure 7:Pareto curve between PickScore and Vendi diversity at different guidance scales\.

Figure[6](https://arxiv.org/html/2610.00365#S5.F6)compares ZeNOVA with Cartesian Langevin, exponential\-map updates, and NES with ImageReward on HPDv2 prompts, holding the number of iterations fixed atK=25K=25\. Reward increases withNNfor every evaluated method, while ZeNOVA achieves the highest reward at every reported particle count and exhibits a small degradation as the particle budget decreases; reducingNNfrom6464to88lowers ZeNOVA’s ImageReward by only0\.07430\.0743\. Notably, ZeNOVA with only88particles outperforms all three baselines with6464particles\. These results demonstrate that ZeNOVA preserves strong alignment under limited reward\-query budgets and are consistent with the stabilizing role of hyperspherical updates under noisy zeroth\-order guidance\.

Reward\-Diversity Trade\-off\.ZeNOVA and the baselines have a guidance scale hyperparameterλ\\lambdato control the trade\-off between prior preservation and reward\. Figure[7](https://arxiv.org/html/2610.00365#S5.F7)plots the Pareto curve between PickScore reward and Vendi diversity\([Friedman and Dieng, 2023](https://arxiv.org/html/2610.00365#bib.bib48)\)of 64 images per prompt, for different methods at three guidance scales\. For all methods, increasing1/λ1/\\lambdaimproves the average reward while diversity decreases, indicating that stronger guidance yields reward gains by making the output distribution focus on a narrow high\-reward region\. Meanwhile, our approach achieves a dominant Pareto front located at the most upright of the plot, indicating higher diversity in its output for a fixed level of reward alignment\.

Table 3:Ablation study of the components of ZeNOVA\.VariantsImageRewardPickScoreOCRZeNOVA1\.807823\.45640\.5114\- MH jumping1\.726223\.18670\.4412\- Value Annealing1\.698423\.29600\.3756\- Hyperspherical update1\.396822\.27340\.3823

Ablation Study\.We’ve conducted an ablation study on more detailed design choices of the proposed ZeNOVA in Table[3](https://arxiv.org/html/2610.00365#S5.T3)to analyze and emphasize their role\. It is shown that all three components contributed to the method’s performance and stability\. We note that introducing the annealed soft value function yielded marginal or no performance improvements on ImageReward and PickScore, but led to a substantial improvement on OCR reward; the reward landscapes of the former two rewards are continuous with many high\-reward modes, allowing sufficiently high rewards to be achieved using only local reward gradient information around the initial point\. In the more challenging black\-box reward of OCR, the value function allows the method to access a more global and meaningful reward gradient signal\.

## 6Conclusion

We introduced ZeNOVA, a stable and efficient initial\-noise optimization framework for generative models through initial\-noise optimization\. Guided by a reward\-tilted sampling objective, ZeNOVA combines annealed soft\-value guidance to extract useful signals from irregular rewards, where several crucial challenges are adequately resolved to enable stability and global exploration\. Experimental results on various scenarios demonstrate that ZeNOVA consistently achieves higher rewards than the zeroth\-order baselines and retains strong performance with less computational cost\.

### AI use statement

In this work, we used generative AI tools to \[assist with translation\]\. We have not used generative AI tools to \[help develop theoretical models or conceptual frameworks, formulate mathematical claims, provide critical ingredients for proving mathematical claims, assist in the writing of proofs, propose or refine hypotheses, design or provide feedback on research methodology or experiments, implement methods, clean and reformat dataset, interpret results\], and \[generate synthetic data sets, support qualitative and thematic data analysis\] are not applicable to this work\. Additionally, we used generative AI tools to \[create or edit software code, edit a research paper to improve readability\]\. We have reviewed all AI\-assisted work\. \[All authors reviewed and refined LLM\-generated drafts and code snippets before using them\]\. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

### Ethics statement

This paper presents work on image generative models, and we acknowledge that generative models can produce misleading, biased, or not\-safe\-for\-work visual content, which may amplify misinformation, prejudice, or privacy violations\.

### Reproducibility statement

For theoretical results, clear explanations of any assumptions and a complete proof of the claims are included in the Appendix\. Any datasets used in the experiments are open access; a complete description of the data preparation are provided in the Appendix\.

## References

- K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. LevineTraining diffusion models with reinforcement learning\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=YCWjhGrJFD)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Chunget al\.\(2023\)H\. Chung, J\. Kim, M\. T\. Mccann, M\. L\. Klasky, and J\. C\. YeDiffusion posterior sampling for general noisy inverse problems\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=OnD9zGAGT0k)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Cuiet al\.\(2025\)C\. Cui, T\. Sun, M\. Lin, T\. Gao, Y\. Zhang, J\. Liu, X\. Wang, Z\. Zhang, C\. Zhou, H\. Liu,et al\.Paddleocr 3\.0 technical report\.arXiv preprint arXiv:2507\.05595\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Domingo\-Enrichet al\.\(2025\)C\. Domingo\-Enrich, M\. Drozdzal, B\. Karrer, and R\. T\. Q\. ChenAdjoint matching: fine\-tuning flow and diffusion generative models with memoryless stochastic optimal control\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xQBRrtQM8u)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Eyringet al\.\(2024\)L\. Eyring, S\. Karthik, K\. Roth, A\. Dosovitskiy, and Z\. AkataReNO: enhancing one\-step text\-to\-image models through reward\-based noise optimization\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=MXY0qsGgeO)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p3.1),[§2](https://arxiv.org/html/2610.00365#S2.p3.2),[§4](https://arxiv.org/html/2610.00365#S4.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p4.1)\.
- Fanet al\.\(2023\)Y\. Fan, O\. Watkins, Y\. Du, H\. Liu, M\. Ryu, C\. Boutilier, P\. Abbeel, M\. Ghavamzadeh, K\. Lee, and K\. LeeReinforcement learning for fine\-tuning text\-to\-image diffusion models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=8OTPepXzeh)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1),[§2](https://arxiv.org/html/2610.00365#S2.p1.2)\.
- Fleming and Rishel \(2012\)W\. H\. Fleming and R\. W\. RishelDeterministic and stochastic optimal control\.Vol\.1,Springer Science & Business Media\.Cited by:[§A\.3](https://arxiv.org/html/2610.00365#A1.SS3.p3.1)\.
- Friedman and Dieng \(2023\)D\. Friedman and A\. B\. DiengThe vendi score: a diversity evaluation metric for machine learning\.Transactions on Machine Learning Research\.External Links:ISSN 2835\-8856Cited by:[§5](https://arxiv.org/html/2610.00365#S5.p3.1)\.
- Genget al\.\(2025\)Z\. Geng, M\. Deng, X\. Bai, J\. Z\. Kolter, and K\. HeMean flows for one\-step generative modeling\.arXiv preprint arXiv:2505\.13447\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p2.1)\.
- Ghoshet al\.\(2023\)D\. Ghosh, H\. Hajishirzi, and L\. SchmidtGenEval: an object\-focused framework for evaluating text\-to\-image alignment\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=Wbr51vK331)Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Heet al\.\(2026\)X\. He, S\. Fu, Y\. Zhao, W\. Li, J\. Yang, D\. Yin, F\. Rao, and B\. ZhangTempFlow\-GRPO: when timing matters for GRPO in flow models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=7mCo3R3Wyn)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Heet al\.\(2023\)Y\. He, N\. Murata, C\. Lai, Y\. Takida, T\. Uesaka, D\. Kim, W\. Liao, Y\. Mitsufuji, J\. Z\. Kolter, R\. Salakhutdinov,et al\.Manifold preserving guided diffusion\.arXiv preprint arXiv:2311\.16424\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Huanget al\.\(2024\)Y\. Huang, A\. Ghatare, Y\. Liu, Z\. Hu, Q\. Zhang, C\. S\. Sastry, S\. Gururani, S\. Oore, and Y\. YueSymbolic music generation with non\-differentiable rule guided diffusion\.arXiv preprint arXiv:2402\.14285\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1),[§2](https://arxiv.org/html/2610.00365#S2.p2.1)\.
- Jensenet al\.\(2026\)C\. Jensen, L\. Schaufelberger, R\. De Santi, K\. Jorner, and A\. KrauseValue matching: scalable and gradient\-free reward\-guided flow adaptation\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 144926–144958\.Cited by:[§3\.2](https://arxiv.org/html/2610.00365#S3.SS2.p2.3)\.
- Kappen \(2005\)H\. KappenPath integrals and symmetry breaking for optimal control theory\.J Stat Mech\- Theory E2005,pp\.\.External Links:[Document](https://dx.doi.org/10.1088/1742-5468/2005/11/P11011)Cited by:[§A\.3](https://arxiv.org/html/2610.00365#A1.SS3.p3.2),[§2](https://arxiv.org/html/2610.00365#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.00365#S3.SS2.p2.1)\.
- Kimet al\.\(2025\)J\. Kim, B\. S\. Kim, and J\. C\. YeFree2Guide: training\-free text\-to\-video alignment using image lvlm\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 17920–17929\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Kimet al\.\(2026\)J\. Kim, H\. Kim, J\. Wang, and J\. C\. YeGradient\-free noise optimization for reward alignment in generative models\.arXiv preprint arXiv:2605\.11347\.Cited by:[§4](https://arxiv.org/html/2610.00365#S4.p2.1)\.
- Kirstainet al\.\(2023\)Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. LevyPick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=G5RwHpBUv0)Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p2.1),[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p3.1),[§3\.1](https://arxiv.org/html/2610.00365#S3.SS1.p5.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Lee \(2018\)J\. M\. LeeIntroduction to riemannian manifolds\.Vol\.2,Springer\.Cited by:[§A\.4](https://arxiv.org/html/2610.00365#A1.SS4.p4.2),[§3\.2](https://arxiv.org/html/2610.00365#S3.SS2.p8.1)\.
- Liuet al\.\(2025a\)J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. OuyangFlow\-GRPO: training flow matching models via online RL\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=oCBKGw5HNf)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Liuet al\.\(2026\)J\. Liu, G\. Liu, J\. Liang, Z\. Yuan, X\. Liu, M\. Zheng, X\. Wu, Q\. Wang, M\. Xia, X\. Wang,et al\.Improving video generation with human feedback\.Advances in Neural Information Processing Systems38,pp\. 82155–82192\.Cited by:[§B\.4](https://arxiv.org/html/2610.00365#A2.SS4.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p7.1)\.
- Liuet al\.\(2025b\)Q\. Liu, Z\. Liu, D\. Zhang, and K\. JiaNabla\-R2D3: effective and efficient 3D diffusion alignment with 2D rewards\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=Dk2qprCnu8)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Liuet al\.\(2025c\)Z\. Liu, T\. Z\. Xiao, W\. Liu, Y\. Bengio, and D\. ZhangEfficient diversity\-preserving diffusion alignment via gradient\-informed GFlownets\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Aye5wL6TCn)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Minet al\.\(2026\)Y\. Min, D\. Choi, K\. Yeo, J\. Lee, and M\. SungORIGEN: zero\-shot 3d orientation grounding in text\-to\-image generation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=CKOyN2HShu)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p3.1),[§2](https://arxiv.org/html/2610.00365#S2.p3.3),[§4](https://arxiv.org/html/2610.00365#S4.p4.1)\.
- Qianet al\.\(2025\)Y\. Qian, Z\. Guo, B\. Deng, C\. T\. Lei, S\. Zhao, C\. P\. Lau, X\. Hong, and M\. P\. PoundT2icount: enhancing cross\-modal understanding for zero\-shot counting\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 25336–25345\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Saueret al\.\(2024\)A\. Sauer, D\. Lorenz, A\. Blattmann, and R\. RombachAdversarial diffusion distillation\.InEuropean Conference on Computer Vision,pp\. 87–103\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p1.1),[§1](https://arxiv.org/html/2610.00365#S1.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Schuhmann \(2022\)C\. SchuhmannImproved aesthetic predictor\.Note:[https://github\.com/christophschuhmann/improved\-aesthetic\-predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor)GitHub repositoryCited by:[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Shiet al\.\(2026\)Q\. Shi, C\. Jin, Z\. Liu, and Y\. GuManifold\-constrained noise optimization for diverse diffusion sampling\.arXiv preprint arXiv:2607\.23937\.Cited by:[§2](https://arxiv.org/html/2610.00365#S2.p3.2),[§4](https://arxiv.org/html/2610.00365#S4.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p4.1)\.
- Songet al\.\(2023a\)J\. Song, Q\. Zhang, H\. Yin, M\. Mardani, M\. Liu, J\. Kautz, Y\. Chen, and A\. VahdatLoss\-guided diffusion models for plug\-and\-play controllable generation\.InInternational Conference on Machine Learning,pp\. 32483–32498\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Songet al\.\(2023b\)Y\. Song, P\. Dhariwal, M\. Chen, and I\. SutskeverConsistency models\.arXiv preprint arXiv:2303\.01469\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p2.1)\.
- Songet al\.\(2021\)Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. PooleScore\-based generative modeling through stochastic differential equations\.In9th International Conference on Learning Representations, ICLR,Cited by:[§2](https://arxiv.org/html/2610.00365#S2.p1.3)\.
- Stein \(1981\)C\. M\. SteinEstimation of the mean of a multivariate normal distribution\.The Annals of Statistics9\(6\),pp\. 1135–1151\.Cited by:[§3\.1](https://arxiv.org/html/2610.00365#S3.SS1.p1.1)\.
- Ueharaet al\.\(2024a\)M\. Uehara, Y\. Zhao, T\. Biancalani, and S\. LevineUnderstanding reinforcement learning\-based fine\-tuning of diffusion models: a tutorial and review\.arXiv preprint arXiv:2407\.13734\.Cited by:[§2](https://arxiv.org/html/2610.00365#S2.p2.1)\.
- Ueharaet al\.\(2024b\)M\. Uehara, Y\. Zhao, K\. Black, E\. Hajiramezanali, G\. Scalia, N\. L\. Diamant, A\. M\. Tseng, T\. Biancalani, and S\. LevineFine\-tuning of continuous\-time diffusion models as entropy\-regularized control\.arXiv preprint arXiv:2402\.15194\.Cited by:[§2](https://arxiv.org/html/2610.00365#S2.p1.2),[§3\.2](https://arxiv.org/html/2610.00365#S3.SS2.p2.1)\.
- Wang and Yang \(2024\)W\. Wang and Y\. YangVidprom: a million\-scale real prompt\-gallery dataset for text\-to\-video diffusion models\.Advances in Neural Information Processing Systems37,pp\. 65618–65642\.Cited by:[§B\.4](https://arxiv.org/html/2610.00365#A2.SS4.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p7.1)\.
- Wierstraet al\.\(2014\)D\. Wierstra, T\. Schaul, T\. Glasmachers, Y\. Sun, J\. Peters, and J\. SchmidhuberNatural evolution strategies\.The Journal of Machine Learning Research15\(1\),pp\. 949–980\.Cited by:[§B\.2](https://arxiv.org/html/2610.00365#A2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2610.00365#S3.SS1.p1.3),[§4](https://arxiv.org/html/2610.00365#S4.p4.1)\.
- Wuet al\.\(2023\)X\. Wu, Y\. Hao, K\. Sun, Y\. Chen, F\. Zhu, R\. Zhao, and H\. LiHuman preference score v2: a solid benchmark for evaluating human preferences of text\-to\-image synthesis\.arXiv preprint arXiv:2306\.09341\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p3.1)\.
- Xuet al\.\(2024\)J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. DongImagereward: learning and evaluating human preferences for text\-to\-image generation\.Advances in Neural Information Processing Systems36\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Xueet al\.\(2025\)Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu, M\. Chen, Z\. Liu, W\. Liu, Q\. Guo, W\. Huang,et al\.Dancegrpo: unleashing grpo on visual generation\.arXiv preprint arXiv:2505\.07818\.Cited by:[§B\.4](https://arxiv.org/html/2610.00365#A2.SS4.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p7.1)\.
- Yeet al\.\(2024\)H\. Ye, H\. Lin, J\. Han, M\. Xu, S\. Liu, Y\. Liang, J\. Ma, J\. Y\. Zou, and S\. ErmonTfg: unified training\-free guidance for diffusion models\.Advances in Neural Information Processing Systems37,pp\. 22370–22417\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Yinet al\.\(2024a\)T\. Yin, M\. Gharbi, T\. Park, R\. Zhang, E\. Shechtman, F\. Durand, and W\. T\. FreemanImproved distribution matching distillation for fast image synthesis\.Advances in neural information processing systems37,pp\. 47455–47487\.Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p1.1),[§1](https://arxiv.org/html/2610.00365#S1.p2.1),[§4](https://arxiv.org/html/2610.00365#S4.p3.1)\.
- Yinet al\.\(2024b\)T\. Yin, M\. Gharbi, R\. Zhang, E\. Shechtman, F\. Durand, W\. T\. Freeman, and T\. ParkOne\-step diffusion with distribution matching distillation\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 6613–6623\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p2.1)\.
- Yoonet al\.\(2026\)T\. Yoon, Y\. Min, K\. Yeo, and M\. Sung$\\psi$\-sampler: initial particle sampling for SMC\-based inference\-time reward alignment in score models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=slVqJAI5sT)Cited by:[§B\.3](https://arxiv.org/html/2610.00365#A2.SS3.p3.1),[§1](https://arxiv.org/html/2610.00365#S1.p3.1),[§2](https://arxiv.org/html/2610.00365#S2.p3.3)\.
- Yuet al\.\(2023\)J\. Yu, Y\. Wang, C\. Zhao, B\. Ghanem, and J\. ZhangFreedom: training\-free energy\-guided conditional diffusion model\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 23174–23184\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Zhanget al\.\(2025\)D\. Zhang, Y\. Zhang, J\. Gu, R\. Zhang, J\. M\. Susskind, N\. Jaitly, and S\. ZhaiImproving GFlownets for text\-to\-image diffusion alignment\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=XDbY3qhM42)Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Zhenget al\.\(2026a\)K\. Zheng, H\. Chen, H\. Ye, H\. Wang, Q\. Zhang, K\. Jiang, H\. Su, S\. Ermon, J\. Zhu, and M\. LiuDiffusionnft: online diffusion reinforcement with forward process\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 134129–134150\.Cited by:[§1](https://arxiv.org/html/2610.00365#S1.p1.1)\.
- Zhenget al\.\(2026b\)K\. Zheng, Y\. Wang, Q\. Ma, H\. Chen, J\. Zhang, Y\. Balaji, J\. Chen, M\. Liu, J\. Zhu, and Q\. ZhangLarge scale diffusion distillation via score\-regularized continuous\-time consistency\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 2582–2603\.Cited by:[§B\.4](https://arxiv.org/html/2610.00365#A2.SS4.p1.1),[§4](https://arxiv.org/html/2610.00365#S4.p7.1)\.
- Øksendal \(2003\)B\. ØksendalStochastic differential equations\.InStochastic differential equations: an introduction with applications,pp\. 38–50\.Cited by:[§3\.2](https://arxiv.org/html/2610.00365#S3.SS2.p1.1)\.

## Appendix ATheoretical Results

### A\.1Proof of Proposition 1

See[1](https://arxiv.org/html/2610.00365#Thmproposition1)

###### Proof\.

Starting from the defition of the soft value function,

Vσ​\(𝒛\)\\displaystyle V\_\{\\sigma\}\(\{\\bm\{z\}\}\):=λlog𝔼𝒛′∼qσ\(⋅∣𝒛\)\[eR⁡\(𝒛′\)/λ\]\\displaystyle:=\\lambda\\log\\mathbb\{E\}\_\{\{\\bm\{z\}\}^\{\\prime\}\\sim q\_\{\\sigma\}\(\\cdot\\mid\{\\bm\{z\}\}\)\}\\left\[e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\right\]\(19\)⇔exp⁡\(Vσ​\(𝒛\)λ\)\\displaystyle\\Leftrightarrow\\exp\\left\(\\frac\{V\_\{\\sigma\}\(\{\\bm\{z\}\}\)\}\{\\lambda\}\\right\)=exp\(log𝔼𝒛′∼qσ\(⋅∣𝒛\)\[eR⁡\(𝒛′\)/λ\]\)\\displaystyle=\\exp\\\!\\left\(\\log\\mathbb\{E\}\_\{\{\\bm\{z\}\}^\{\\prime\}\\sim q\_\{\\sigma\}\(\\cdot\\mid\{\\bm\{z\}\}\)\}\\left\[e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\right\]\\right\)=𝔼𝒛′∼qσ\(⋅∣𝒛\)\[eR⁡\(𝒛′\)/λ\]\\displaystyle=\\mathbb\{E\}\_\{\{\\bm\{z\}\}^\{\\prime\}\\sim q\_\{\\sigma\}\(\\cdot\\mid\{\\bm\{z\}\}\)\}\\left\[e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\right\]=∫qσ​\(𝒛′∣𝒛\)​eR⁡\(𝒛′\)/λ​d​𝒛′\\displaystyle=\\int q\_\{\\sigma\}\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\,d\{\\bm\{z\}\}^\{\\prime\}Substituting this into \([10](https://arxiv.org/html/2610.00365#S3.E10)\) gives

pσ⋆​\(𝒛\)\\displaystyle p\_\{\\sigma\}^\{\\star\}\(\{\\bm\{z\}\}\)∝p0​\(𝒛\)​exp⁡\(Vσ​\(𝒛\)λ\)\\displaystyle\\propto p\_\{0\}\(\{\\bm\{z\}\}\)\\exp\\left\(\\frac\{V\_\{\\sigma\}\(\{\\bm\{z\}\}\)\}\{\\lambda\}\\right\)\(20\)=∫p0​\(𝒛\)​qσ​\(𝒛′∣𝒛\)​eR⁡\(𝒛′\)/λ​d​𝒛′\\displaystyle=\\int p\_\{0\}\(\{\\bm\{z\}\}\)q\_\{\\sigma\}\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\,d\{\\bm\{z\}\}^\{\\prime\}
Here, since OU kernel is reversible with respect top0p\_\{0\}, we havep0​\(𝒛\)​qσ​\(𝒛′∣𝒛\)=p0​\(𝒛′\)​qσ​\(𝒛∣𝒛′\)p\_\{0\}\(\{\\bm\{z\}\}\)q\_\{\\sigma\}\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)=p\_\{0\}\(\{\\bm\{z\}\}^\{\\prime\}\)q\_\{\\sigma\}\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)\. Thus,

pσ⋆​\(𝒛\)\\displaystyle p\_\{\\sigma\}^\{\\star\}\(\{\\bm\{z\}\}\)∝∫p0​\(𝒛′\)​qσ​\(𝒛∣𝒛′\)​eR⁡\(𝒛′\)/λ​d​𝒛′\\displaystyle\\propto\\int p\_\{0\}\(\{\\bm\{z\}\}^\{\\prime\}\)q\_\{\\sigma\}\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}\\,d\{\\bm\{z\}\}^\{\\prime\}\(21\)∝∫qσ​\(𝒛∣𝒛′\)​p⋆​\(𝒛′\)​d​𝒛′\.\\displaystyle\\propto\\int q\_\{\\sigma\}\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)p^\{\\star\}\(\{\\bm\{z\}\}^\{\\prime\}\)\\,d\{\\bm\{z\}\}^\{\\prime\}\.∎

### A\.2Proof of Proposition 2

See[2](https://arxiv.org/html/2610.00365#Thmproposition2)

###### Proof\.

To construct a Markov chain for sampling from an unnormalized densityp~​\(𝒛\)\\widetilde\{p\}\(\{\\bm\{z\}\}\), Metropolis\-Hastings algorithm draws a candidate𝒛′∼q\(⋅∣𝒛\)\{\\bm\{z\}\}^\{\\prime\}\\sim q\(\\cdot\\mid\{\\bm\{z\}\}\)and accepts it with probability

αMH​\(𝒛,𝒛′\)=min⁡\{1,p~​\(𝒛′\)​q​\(𝒛∣𝒛′\)p~​\(𝒛\)​q​\(𝒛′∣𝒛\)\},\\alpha\_\{\\mathrm\{MH\}\}\(\{\\bm\{z\}\},\{\\bm\{z\}\}^\{\\prime\}\)=\\min\\\!\\left\\\{1,\\frac\{\\widetilde\{p\}\(\{\\bm\{z\}\}^\{\\prime\}\)q\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)\}\{\\widetilde\{p\}\(\{\\bm\{z\}\}\)q\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)\}\\right\\\},\(22\)retaining the current state if the proposal is rejected\.

p⋆​\(𝒛\)∝p0​\(𝒛\)​exp⁡\(R⁡\(𝒛\)λ\),p^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)\\exp\\\!\\left\(\\frac\{R\(\{\\bm\{z\}\}\)\}\{\\lambda\}\\right\),\(23\)To apply Metropolis\-Hastings to an unnormalized reward\-tilted distributionp~​\(𝒛\)=p0​\(𝒛\)​eR⁡\(𝒛\)/λ\\widetilde\{p\}\(\{\\bm\{z\}\}\)=p\_\{0\}\(\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}\)/\\lambda\}, on can choose the independent Gaussian prior proposal and reward\-based acceptance probability as

q⁡\(𝒛′∣𝒛\)=p0​\(𝒛′\)=𝒩⁡\(𝒛′,0,𝐈\)\.q\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)=p\_\{0\}\(\{\\bm\{z\}\}^\{\\prime\}\)=\{\\mathcal\{N\}\}\(\{\\bm\{z\}\}^\{\\prime\};0,\{\\mathbf\{I\}\}\)\.\(24\)
By this selection, Metropolis\-Hastings acceptance ratio forα⁡\(𝒛,𝒛′\)\\alpha\(\{\\bm\{z\}\},\{\\bm\{z\}\}^\{\\prime\}\)becomes

p~​\(𝒛′\)​q​\(𝒛∣𝒛′\)p~​\(𝒛\)​q​\(𝒛′∣𝒛\)=p0​\(𝒛′\)​eR⁡\(𝒛′\)/λ​p0​\(𝒛\)p0​\(𝒛\)​eR⁡\(𝒛\)/λ​p0​\(𝒛′\)=exp⁡\(R⁡\(𝒛′\)−R⁡\(𝒛\)λ\),\\displaystyle\\frac\{\\widetilde\{p\}\(\{\\bm\{z\}\}^\{\\prime\}\)q\(\{\\bm\{z\}\}\\mid\{\\bm\{z\}\}^\{\\prime\}\)\}\{\\widetilde\{p\}\(\{\\bm\{z\}\}\)q\(\{\\bm\{z\}\}^\{\\prime\}\\mid\{\\bm\{z\}\}\)\}=\\frac\{p\_\{0\}\(\{\\bm\{z\}\}^\{\\prime\}\)e^\{R\(\{\\bm\{z\}\}^\{\\prime\}\)/\\lambda\}p\_\{0\}\(\{\\bm\{z\}\}\)\}\{p\_\{0\}\(\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}\)/\\lambda\}p\_\{0\}\(\{\\bm\{z\}\}^\{\\prime\}\)\}=\\exp\\\!\\left\(\\frac\{R\(\{\\bm\{z\}\}^\{\\prime\}\)\-R\(\{\\bm\{z\}\}\)\}\{\\lambda\}\\right\),\(25\)which reconstructs the claimed algorithm to generate samples fromp⋆​\(𝒛\)∝p0​\(𝒛\)​eR⁡\(𝒛\)/λp^\{\\star\}\(\{\\bm\{z\}\}\)\\propto p\_\{0\}\(\{\\bm\{z\}\}\)e^\{R\(\{\\bm\{z\}\}\)/\\lambda\}\. ∎

### A\.3Derivation of Zeroth\-order Value Gradient Estimator

Stochastic Optimal Control theory describes the optimal control term injected into a given reference process to maximize an entropy\-regularized objective and relates it to the soft value function\. Specifically, in our OU reference process over a timestept=0t=0tot=ht=h,

d​𝒛s=−𝒛s​d​s\+2​d​𝒘s,𝒛0=𝒛\.d\{\\bm\{z\}\}\_\{s\}=\-\{\\bm\{z\}\}\_\{s\}\\,ds\+\\sqrt\{2\}\\,d\{\\bm\{w\}\}\_\{s\},\\qquad\{\\bm\{z\}\}\_\{0\}=\{\\bm\{z\}\}\.\(26\)the optimal control𝒖⋆​\(𝒛,0\)\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)is defined to maximize the following objectives under the controlled process\.

max⁡𝔼u​\[R⁡\(𝒛hu\)−λ2​∫0h‖𝒖s‖2​𝑑s\|𝒛0u=𝒛\]\\displaystyle\\max\\mathbb\{E\}\_\{u\}\\left\[R\(\{\\bm\{z\}\}\_\{h\}^\{u\}\)\-\\frac\{\\lambda\}\{2\}\\int\_\{0\}^\{h\}\\\|\{\\bm\{u\}\}\_\{s\}\\\|^\{2\}ds\\middle\|\\,\{\\bm\{z\}\}\_\{0\}^\{u\}=\{\\bm\{z\}\}\\right\]\(27\)s\.t\.​d​𝒛su=−𝒛su​d​s\+2​\(𝒖s​d​s\+d​𝒘s\),𝒛0u=𝒛\\displaystyle\\text\{s\.t\. \}d\{\\bm\{z\}\}\_\{s\}^\{u\}=\-\{\\bm\{z\}\}\_\{s\}^\{u\}ds\+\\sqrt\{2\}\\bigl\(\{\\bm\{u\}\}\_\{s\}\\,ds\+d\{\\bm\{w\}\}\_\{s\}\\bigr\),\{\\bm\{z\}\}\_\{0\}^\{u\}=\{\\bm\{z\}\}\(28\)
Withσ2:=1−e−2​h\\sigma^\{2\}:=1\-e^\{\-2h\}, Hamilton\-Jacobi\-Bellman equation\([Fleming and Rishel, 2012](https://arxiv.org/html/2610.00365#bib.bib4)\)states that the optimal control att=0t=0𝒖⋆​\(𝒛,0\)\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)satisfies

𝒖⋆​\(𝒛,0\)=2λ​∇𝒛Vσ​\(𝒛\)\.\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)=\\frac\{\\sqrt\{2\}\}\{\\lambda\}\\nabla\_\{\{\\bm\{z\}\}\}V\_\{\\sigma\}\(\{\\bm\{z\}\}\)\.\(29\)Furthermore, the path\-integral control identity\([Kappen, 2005](https://arxiv.org/html/2610.00365#bib.bib47)\)gives

𝒖⋆​\(𝒛,0\)=limd​t→0𝔼ref​\[eR⁡\(𝒛h\)/λ​d​𝒘0∣𝒛0=𝒛\]d​t​𝔼ref​\[eR⁡\(𝒛h\)/λ∣𝒛0=𝒛\]\.\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)=\\lim\_\{dt\\to 0\}\\frac\{\\mathbb\{E\}\_\{\\text\{ref\}\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{0\}=\{\\bm\{z\}\}\]\}\{dt\\mathbb\{E\}\_\{\\text\{ref\}\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\\mid\{\\bm\{z\}\}\_\{0\}=\{\\bm\{z\}\}\]\}\.\(30\)Here,d​𝒘0d\{\\bm\{w\}\}\_\{0\}andd​tdtdenote the injected Brownian motion of a single trajectory att=0t=0and its corresponding timestep increment\. All expectations in this identity are under the uncontrolled reference process\. The terminal reward is evaluated at𝒛h\{\\bm\{z\}\}\_\{h\}\.

Now, we reformulate Eq\. \([30](https://arxiv.org/html/2610.00365#A1.E30)\) into terms that are more easily tractable\. First, we can get the exact OU solution for𝒛h\{\\bm\{z\}\}\_\{h\}as

𝒛h=1−σ2​𝒛\+σ​ϵ,ϵ∼𝒩⁡\(0,𝐈\)\.\{\\bm\{z\}\}\_\{h\}=\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\},\\qquad\{\\bm\{\{\\epsilon\}\}\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\.\(31\)
Our goal is to rewrite the numerator of Eq\. \([30](https://arxiv.org/html/2610.00365#A1.E30)\), which contains both the terminal reward and the initial Brownian incrementd​𝒘0d\{\\bm\{w\}\}\_\{0\}, using only the terminal Gaussian sample𝒛h\{\\bm\{z\}\}\_\{h\}\. Here, note that the reward weight depends only on𝒛h\{\\bm\{z\}\}\_\{h\}\. Therefore, applying the tower property to the numerator of Eq\. \([30](https://arxiv.org/html/2610.00365#A1.E30)\) yields

𝔼⁡\[eR⁡\(𝒛h\)/λ​d​𝒘0\]\\displaystyle\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}d\{\\bm\{w\}\}\_\{0\}\]=𝔼⁡\[𝔼⁡\[eR⁡\(𝒛h\)/λ​d​𝒘0∣𝒛h\]\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{h\}\]\\right\]\(32\)=𝔼⁡\[eR⁡\(𝒛h\)/λ​𝔼​\[d​𝒘0∣𝒛h\]\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\\mathbb\{E\}\[d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{h\}\]\\right\]\(33\)
For𝔼⁡\[d​𝒘0∣𝒛h\]\\mathbb\{E\}\[d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{h\}\], we use the conditional mean formula jointly Gaussian variables:

𝔼⁡\[X∣Y\]=𝔼⁡\[X\]\+Cov⁡\(X,Y\)​Var⁡\(Y\)−1​\(Y−𝔼⁡\[Y\]\)\.\\mathbb\{E\}\[X\\mid Y\]=\\mathbb\{E\}\[X\]\+\\operatorname\{Cov\}\(X,Y\)\\operatorname\{Var\}\(Y\)^\{\-1\}\(Y\-\\mathbb\{E\}\[Y\]\)\.\(34\)Applying this formula withX=d​𝒘0X=d\{\\bm\{w\}\}\_\{0\}andY=𝒛hY=\{\\bm\{z\}\}\_\{h\}, and using𝔼⁡\[d​𝒘0\]=0\\mathbb\{E\}\[d\{\\bm\{w\}\}\_\{0\}\]=0and𝔼⁡\[𝒛h\]=1−σ2​𝒛\\mathbb\{E\}\[\{\\bm\{z\}\}\_\{h\}\]=\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}, gives

𝔼⁡\[d​𝒘0∣𝒛h\]=0\+Cov⁡\(d​𝒘0,𝒛h\)​Var⁡\(𝒛h\)−1​\(𝒛h−1−σ2​𝒛\)\\mathbb\{E\}\[d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{h\}\]=0\+\\operatorname\{Cov\}\(d\{\\bm\{w\}\}\_\{0\},\{\\bm\{z\}\}\_\{h\}\)\\operatorname\{Var\}\(\{\\bm\{z\}\}\_\{h\}\)^\{\-1\}\(\{\\bm\{z\}\}\_\{h\}\-\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\)\(35\)
Here, we have a stochastic\-integral representation of the terminal state of the OU process as𝒛h=e−h​𝒛\+2​∫0he−\(h−s\)​d​𝒘s\{\\bm\{z\}\}\_\{h\}=e^\{\-h\}\{\\bm\{z\}\}\+\\sqrt\{2\}\\int\_\{0\}^\{h\}e^\{\-\(h\-s\)\}d\{\\bm\{w\}\}\_\{s\}\. Bothd​𝒘0d\{\\bm\{w\}\}\_\{0\}and𝒛h\{\\bm\{z\}\}\_\{h\}are thus linear functions of Gaussian noise and are jointly Gaussian\. The initial noise contributes to𝒛h\{\\bm\{z\}\}\_\{h\}through the interval\[0,d​t\]\[0,dt\], while later increments are independent of it\. Their cross\-covariance and terminal covariance are consequently

Cov⁡\(d​𝒘0,𝒛h\)\\displaystyle\\operatorname\{Cov\}\(d\{\\bm\{w\}\}\_\{0\},\{\\bm\{z\}\}\_\{h\}\)=2​∫0d​te−\(h−s\)​ds​𝐈=2​1−σ2​\(ed​t−1\)​𝐈,\\displaystyle=\\sqrt\{2\}\\int\_\{0\}^\{dt\}e^\{\-\(h\-s\)\}ds\{\\mathbf\{I\}\}=\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\(e^\{dt\}\-1\)\{\\mathbf\{I\}\},\(36\)Var⁡\(𝒛h\)\\displaystyle\\operatorname\{Var\}\(\{\\bm\{z\}\}\_\{h\}\)=σ2​𝐈\.\\displaystyle=\\sigma^\{2\}\{\\mathbf\{I\}\}\.
Now we can complete the formula:

𝔼⁡\[d​𝒘0∣𝒛h\]\\displaystyle\\mathbb\{E\}\[d\{\\bm\{w\}\}\_\{0\}\\mid\{\\bm\{z\}\}\_\{h\}\]=2​1−σ2​\(ed​t−1\)σ2​\(𝒛h−1−σ2​𝒛\)\\displaystyle=\\frac\{\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\(e^\{dt\}\-1\)\}\{\\sigma^\{2\}\}\(\{\\bm\{z\}\}\_\{h\}\-\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\)\(37\)𝔼⁡\[eR⁡\(𝒛h\)/λ​d​𝒘0\]\\displaystyle\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}d\{\\bm\{w\}\}\_\{0\}\]=2​1−σ2​\(ed​t−1\)σ2​𝔼​\[eR⁡\(𝒛h\)/λ​\(𝒛h−1−σ2​𝒛\)\]\.\\displaystyle=\\frac\{\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\(e^\{dt\}\-1\)\}\{\\sigma^\{2\}\}\\mathbb\{E\}\\\!\\left\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\(\{\\bm\{z\}\}\_\{h\}\-\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\)\\right\]\.\(38\)Substituting it into Eq\. \([30](https://arxiv.org/html/2610.00365#A1.E30)\) gives

𝒖⋆​\(𝒛,0\)\\displaystyle\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)=limd​t→02​1−σ2σ2​ed​t−1d​t​𝔼⁡\[eR⁡\(𝒛h\)/λ​\(𝒛h−1−σ2​𝒛\)\]𝔼⁡\[eR⁡\(𝒛h\)/λ\]\\displaystyle=\\lim\_\{dt\\to 0\}\\frac\{\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\}\{\\sigma^\{2\}\}\\frac\{e^\{dt\}\-1\}\{dt\}\\frac\{\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\(\{\\bm\{z\}\}\_\{h\}\-\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\)\]\}\{\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\]\}\(39\)=2​1−σ2σ2​𝔼⁡\[eR⁡\(𝒛h\)/λ​\(𝒛h−1−σ2​𝒛\)\]𝔼⁡\[eR⁡\(𝒛h\)/λ\]\\displaystyle=\\frac\{\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\}\{\\sigma^\{2\}\}\\frac\{\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\(\{\\bm\{z\}\}\_\{h\}\-\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\)\]\}\{\\mathbb\{E\}\[e^\{R\(\{\\bm\{z\}\}\_\{h\}\)/\\lambda\}\]\}Finally, we substitute𝒛h=1−σ2​𝒛\+σ​ϵ\{\\bm\{z\}\}\_\{h\}=\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}:

𝒖⋆​\(𝒛,0\)=2​1−σ2σ​𝔼ϵ​\[eR⁡\(1−σ2​𝒛\+σ​ϵ\)/λ​ϵ\]𝔼ϵ​\[eR⁡\(1−σ2​𝒛\+σ​ϵ\)/λ\]\.\{\\bm\{u\}\}^\{\\star\}\(\{\\bm\{z\}\},0\)=\\frac\{\\sqrt\{2\}\\sqrt\{1\-\\sigma^\{2\}\}\}\{\\sigma\}\\frac\{\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\}\[e^\{R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}\)/\\lambda\}\{\\bm\{\{\\epsilon\}\}\}\]\}\{\\mathbb\{E\}\_\{\{\\bm\{\{\\epsilon\}\}\}\}\[e^\{R\(\\sqrt\{1\-\\sigma^\{2\}\}\{\\bm\{z\}\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}\)/\\lambda\}\]\}\.\(40\)
The resulting expression depends only on a Gaussian perturbation and its terminal reward, so no full OU trajectory or initial Brownian increment needs to be sampled\. Multiplying byλ/2\\lambda/\\sqrt\{2\}recovers \([12](https://arxiv.org/html/2610.00365#S3.E12)\)\.

### A\.4Derivation of Hyperspherical Langevin Dynamics

We decompose a sample𝒛\{\\bm\{z\}\}from a standard Gaussian as its normρ\\rhoand its unit direction vector𝒖\{\\bm\{u\}\}\.

𝒛=ρ​𝒖,ρ:=‖𝒛‖2\>0,𝒖∈𝕊d−1\{\\bm\{z\}\}=\\rho\{\\bm\{u\}\},\\qquad\\rho:=\\\|\{\\bm\{z\}\}\\\|\_\{2\}\>0,\\qquad\{\\bm\{u\}\}\\in\\mathbb\{S\}^\{d\-1\}\(41\)When we convert the soft\-value\-tilted distribtuion withρ\\rhoand𝒖\{\\bm\{u\}\}as well, Eq\. \([10](https://arxiv.org/html/2610.00365#S3.E10)\) becomes

p~σ⋆​\(ρ,𝒖\)∝ρd−1​exp⁡\(−ρ22\+Vσ​\(ρ​𝒖\)λ\)\.\\widetilde\{p\}\_\{\\sigma\}^\{\\star\}\(\\rho,\{\\bm\{u\}\}\)\\propto\\rho^\{d\-1\}\\exp\\left\(\-\\frac\{\\rho^\{2\}\}\{2\}\+\\frac\{V\_\{\\sigma\}\(\\rho\{\\bm\{u\}\}\)\}\{\\lambda\}\\right\)\.\(42\)This expression exposes two qualitatively different directions: the angular component moves along the Gaussian shell, whereas the radial component controls how far the sample lies from the origin\. Rather than allowing a noisydd\-dimensional update to perturb both components implicitly, we construct Langevin dynamics for them explicitly\.

Angular Langevin on the sphere\.With a fixed radialρ\\rho, the score function of the angular conditional distribution induced by \([42](https://arxiv.org/html/2610.00365#A1.E42)\) is

∇𝕊​log​p~σ⋆​\(ρ,𝒖\)=𝐏𝒖⟂​\(ρλ​𝒈σ​\(ρ​𝒖\)\),\\nabla\_\{\\mathbb\{S\}\}\\log\\widetilde\{p\}\_\{\\sigma\}^\{\\star\}\(\\rho,\{\\bm\{u\}\}\)=\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\}^\{\\perp\}\\left\(\\frac\{\\rho\}\{\\lambda\}\{\\bm\{g\}\}\_\{\\sigma\}\(\\rho\{\\bm\{u\}\}\)\\right\),\(43\)
where𝒈σ​\(𝒛\):=∇𝒛Vσ​\(𝒛\)\{\\bm\{g\}\}\_\{\\sigma\}\(\{\\bm\{z\}\}\):=\\nabla\_\{\{\\bm\{z\}\}\}V\_\{\\sigma\}\(\{\\bm\{z\}\}\)and𝐏𝒖⟂:=𝐈−𝒖​𝒖⊤\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\}^\{\\perp\}:=\{\\mathbf\{I\}\}\-\{\\bm\{u\}\}\{\\bm\{u\}\}^\{\\top\}denotes the projection onto the tangent space of𝒖\{\\bm\{u\}\}\.

Since𝒖\{\\bm\{u\}\}must lie on a unit sphere, we use intrinsic spherical Langevin dynamics

d​𝒖s=∇𝕊​log​p~σ⋆​\(ρs,𝒖s\)​d​s\+2​d​𝒘s𝕊,d\{\\bm\{u\}\}\_\{s\}=\\nabla\_\{\\mathbb\{S\}\}\\log\\widetilde\{p\}\_\{\\sigma\}^\{\\star\}\(\\rho\_\{s\},\{\\bm\{u\}\}\_\{s\}\)\\,ds\+\\sqrt\{2\}\\,d\{\\bm\{w\}\}\_\{s\}^\{\\mathbb\{S\}\},\(44\)where𝒘s𝕊\{\\bm\{w\}\}\_\{s\}^\{\\mathbb\{S\}\}denotes Brownian motion also lying on the unit sphere\. A practical Euler step implementation of this with step sizeηA\\eta\_\{A\}can be a tangent Gaussian followed by the spherical exponential map\([Lee, 2018](https://arxiv.org/html/2610.00365#bib.bib26)\):

𝒗k\\displaystyle\{\\bm\{v\}\}\_\{k\}=ηA​ρkλ​𝐏𝒖k⟂​𝒈^σk​\(𝒛k\)\+2​ηA​𝐏𝒖k⟂​𝝃k,\\displaystyle=\\eta\_\{A\}\\frac\{\\rho\_\{k\}\}\{\\lambda\}\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\_\{k\}\}^\{\\perp\}\\widehat\{\{\\bm\{g\}\}\}\_\{\\sigma\_\{k\}\}\(\{\\bm\{z\}\}\_\{k\}\)\+\\sqrt\{2\\eta\_\{A\}\}\\,\{\\mathbf\{P\}\}\_\{\{\\bm\{u\}\}\_\{k\}\}^\{\\perp\}\\bm\{\\xi\}\_\{k\},\(45\)𝒖k\+1\\displaystyle\{\\bm\{u\}\}\_\{k\+1\}=Exp𝒖k⁡\(𝒗k\),𝝃k∼𝒩⁡\(0,𝐈\),\\displaystyle=\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\_\{k\}\}\(\{\\bm\{v\}\}\_\{k\}\),\\qquad\\bm\{\\xi\}\_\{k\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\),\(46\)whereExp𝒖​\(𝒗\)\\text\{Exp\}\_\{\{\\bm\{u\}\}\}\(\{\\bm\{v\}\}\)implements the rotation of𝒖\{\\bm\{u\}\}to the origin toward the direction of𝒗\{\\bm\{v\}\}as

Exp𝒖⁡\(𝒗\)=cos⁡\(‖𝒗‖2\)​𝒖\+sin⁡\(‖𝒗‖2\)​𝒗‖𝒗‖2\.\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\}\(\{\\bm\{v\}\}\)=\\cos\(\\\|\{\\bm\{v\}\}\\\|\_\{2\}\)\{\\bm\{u\}\}\+\\sin\(\\\|\{\\bm\{v\}\}\\\|\_\{2\}\)\\frac\{\{\\bm\{v\}\}\}\{\\\|\{\\bm\{v\}\}\\\|\_\{2\}\}\.\(47\)with the continuous extensionExp𝒖⁡\(𝟎\)=𝒖\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\}\(\{\\bm\{0\}\}\)=\{\\bm\{u\}\}\. By construction, both the estimated reward drift and the Langevin noise remain tangent to the sphere\.

Radial Langevin\.For soft\-value\-tilted distribution, with a fixed angular state𝒖\{\\bm\{u\}\}, the score function of the radial conditional induced by \([42](https://arxiv.org/html/2610.00365#A1.E42)\) is

dd​ρ​log⁡p~σ⋆​\(ρ∣𝒖\)=d−1ρ−ρ\+1λ​𝒖⊤​𝒈σ​\(ρ​𝒖\)\.\\frac\{d\}\{d\\rho\}\\log\\widetilde\{p\}\_\{\\sigma\}^\{\\star\}\(\\rho\\mid\{\\bm\{u\}\}\)=\\frac\{d\-1\}\{\\rho\}\-\\rho\+\\frac\{1\}\{\\lambda\}\{\\bm\{u\}\}^\{\\top\}\{\\bm\{g\}\}\_\{\\sigma\}\(\\rho\{\\bm\{u\}\}\)\.\(48\)Note that the first two terms are equal to the score function of the standard Gaussian’s radial distributionρ∼χd\\rho\\sim\\chi\_\{d\}\. We therefore directly apply one\-dimensional Langevin dynamics to the radius,

d​ρs=\(d−1ρs−ρs\+1λ​𝒖s⊤​𝒈σ​\(ρs​𝒖s\)\)​d​s\+2​d​ws\.d\\rho\_\{s\}=\\left\(\\frac\{d\-1\}\{\\rho\_\{s\}\}\-\\rho\_\{s\}\+\\frac\{1\}\{\\lambda\}\{\\bm\{u\}\}\_\{s\}^\{\\top\}\{\\bm\{g\}\}\_\{\\sigma\}\(\\rho\_\{s\}\{\\bm\{u\}\}\_\{s\}\)\\right\)ds\+\\sqrt\{2\}\\,dw\_\{s\}\.\(49\)A practical Euler step with step sizeηR\\eta\_\{R\}is

ρk\+1=ρk\+ηR​\(d−1ρk−ρk\+1λ​𝒖k⊤​𝒈σk​\(𝒛k\)\)\+2​ηR​ξk,ξk∼𝒩⁡\(0,1\)\.\\rho\_\{k\+1\}=\\rho\_\{k\}\+\\eta\_\{R\}\\left\(\\frac\{d\-1\}\{\\rho\_\{k\}\}\-\\rho\_\{k\}\+\\frac\{1\}\{\\lambda\}\{\\bm\{u\}\}\_\{k\}^\{\\top\}\{\\bm\{g\}\}\_\{\\sigma\_\{k\}\}\(\{\\bm\{z\}\}\_\{k\}\)\\right\)\+\\sqrt\{2\\eta\_\{R\}\}\\,\\xi\_\{k\},\\qquad\\xi\_\{k\}\\sim\{\\mathcal\{N\}\}\(0,1\)\.\(50\)The first two terms provide the exact Gaussian radial bias, while the scalar projection𝒖⊤​𝒈^σ\{\\bm\{u\}\}^\{\\top\}\\widehat\{\{\\bm\{g\}\}\}\_\{\\sigma\}captures the reward\-induced radial preference\. Thus the radius remains stochastic and reward\-adaptive, unlike several empirical regularizations like explicit norm constraints\.

## Appendix BImplementation Details

### B\.1Detailed Algorithm of ZeNOVA

Algorithm[1](https://arxiv.org/html/2610.00365#alg1)describes a detailed algorithm of the proposed ZeNOVA with Annealed Soft Value, Hyperspherical Langevin, and MH Jumping\. Note that the reciprocal of the temperature1/λ1/\\lambdacan be considered as an effective reward guidance scale, making the method more focused on reward maximization than prior preservation\.

Algorithm 1ZeNOVA with Annealed Soft Value, Hyperspherical Langevin, and MH Jumping1:Generator

GθG\_\{\\theta\}, reward

rr, temperature

λ\\lambda, horizons

1\>σ1\>⋯\>σK\>01\>\\sigma\_\{1\}\>\\cdots\>\\sigma\_\{K\}\>0, number of particles

NN, number of MH jumping

NMHN\_\{\\text\{MH\}\}, number of iteration

KK, step sizes

ηA,ηR\\eta\_\{A\},\\eta\_\{R\}
2:

𝒛0∼𝒩⁡\(0,𝐈\)\{\\bm\{z\}\}\_\{0\}\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\);

ρ0←‖𝒛0‖2\\rho\_\{0\}\\leftarrow\\\|\{\\bm\{z\}\}\_\{0\}\\\|\_\{2\};

𝒖0←𝒛0/ρ0\{\\bm\{u\}\}\_\{0\}\\leftarrow\{\\bm\{z\}\}\_\{0\}/\\rho\_\{0\}
3:for

k=0,…,K−1k=0,\\ldots,K\-1do

4:for

j=1,…,NMHj=1,\\ldots,N\_\{\\text\{MH\}\}do⊳\\trianglerightindependent MH jumping

5:

𝒛′∼p0\{\\bm\{z\}\}^\{\\prime\}\\sim p\_\{0\};

a∼Unif⁡\(0,1\)a\\sim\\operatorname\{Unif\}\(0,1\)
6:ifa<min⁡\{1,exp⁡\(\(R⁡\(z′\)−R⁡\(z\)\)/λ\)\}a<\\min\\\{1,\\exp\(\(R\(\{\\bm\{z\}\}^\{\\prime\}\)\-R\(\{\\bm\{z\}\}\)\)/\\lambda\)\\\}thenzk←z′\{\\bm\{z\}\}\_\{k\}\\leftarrow\{\\bm\{z\}\}^\{\\prime\}

7:

ρk←‖𝒛k‖2\\rho\_\{k\}\\leftarrow\\\|\{\\bm\{z\}\}\_\{k\}\\\|\_\{2\};

𝒖k←𝒛k/ρk\{\\bm\{u\}\}\_\{k\}\\leftarrow\{\\bm\{z\}\}\_\{k\}/\\rho\_\{k\}
8:for

n=1,…,Nn=1,\\ldots,Ndo⊳\\trianglerightparallel reward queries

9:

ϵk\(n\)∼𝒩⁡\(0,𝐈\)\{\\bm\{\{\\epsilon\}\}\}\_\{k\}^\{\(n\)\}\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\)
10:

Rk\(n\)←r⁡\(Gθ​\(1−σk2​𝒛k\+σk​ϵk\(n\)\)\)R\_\{k\}^\{\(n\)\}\\leftarrow r\\\!\\left\(G\_\{\\theta\}\(\\sqrt\{1\-\\sigma\_\{k\}^\{2\}\}\{\\bm\{z\}\}\_\{k\}\+\\sigma\_\{k\}\{\\bm\{\{\\epsilon\}\}\}\_\{k\}^\{\(n\)\}\)\\right\)
11:

wk\(n\)←softmax⁡\(Rk\(n\)/λ\)w\_\{k\}^\{\(n\)\}\\leftarrow\\operatorname\{softmax\}\(R\_\{k\}^\{\(n\)\}/\\lambda\)
12:

𝒈^k←λ​1−σk2σk​∑n=1Nwk\(n\)​ϵk\(n\)\\widehat\{\{\\bm\{g\}\}\}\_\{k\}\\leftarrow\\dfrac\{\\lambda\\sqrt\{1\-\\sigma\_\{k\}^\{2\}\}\}\{\\sigma\_\{k\}\}\\sum\_\{n=1\}^\{N\}w\_\{k\}^\{\(n\)\}\{\\bm\{\{\\epsilon\}\}\}\_\{k\}^\{\(n\)\}⊳\\trianglerightsoft value gradient

13:

gkrad←𝒖k⊤​𝒈^kg\_\{k\}^\{\\mathrm\{rad\}\}\\leftarrow\{\\bm\{u\}\}\_\{k\}^\{\\top\}\\widehat\{\{\\bm\{g\}\}\}\_\{k\}
14:

𝒈ktan←𝒈^k−gkrad​𝒖k\{\\bm\{g\}\}\_\{k\}^\{\\mathrm\{tan\}\}\\leftarrow\\widehat\{\{\\bm\{g\}\}\}\_\{k\}\-g\_\{k\}^\{\\mathrm\{rad\}\}\{\\bm\{u\}\}\_\{k\}
15:

𝝃k∼𝒩⁡\(0,𝐈\)\\bm\{\\xi\}\_\{k\}\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\);

𝝃ktan←𝝃k−\(𝒖k⊤​𝝃k\)​𝒖k\\bm\{\\xi\}\_\{k\}^\{\\mathrm\{tan\}\}\\leftarrow\\bm\{\\xi\}\_\{k\}\-\(\{\\bm\{u\}\}\_\{k\}^\{\\top\}\\bm\{\\xi\}\_\{k\}\)\{\\bm\{u\}\}\_\{k\}
16:

𝒖k\+1←Exp𝒖k⁡\(ηA​ρkλ​𝒈ktan\+2​ηA​𝝃ktan\)\{\\bm\{u\}\}\_\{k\+1\}\\leftarrow\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\_\{k\}\}\(\\eta\_\{A\}\\dfrac\{\\rho\_\{k\}\}\{\\lambda\}\{\\bm\{g\}\}\_\{k\}^\{\\mathrm\{tan\}\}\+\\sqrt\{2\\eta\_\{A\}\}\\bm\{\\xi\}\_\{k\}^\{\\mathrm\{tan\}\}\)⊳\\trianglerightspherical update

17:

ρk\+1←ρk\+ηR​\[d−1ρk−ρk\+1λ​gkrad\]\+2​ηR​ξk​where​ξk∼𝒩⁡\(0,1\)\\rho\_\{k\+1\}\\leftarrow\\rho\_\{k\}\+\\eta\_\{R\}\\left\[\\dfrac\{d\-1\}\{\\rho\_\{k\}\}\-\\rho\_\{k\}\+\\dfrac\{1\}\{\\lambda\}g\_\{k\}^\{\\mathrm\{rad\}\}\\right\]\+\\sqrt\{2\\eta\_\{R\}\}\\xi\_\{k\}\\textbf\{ where \}\\xi\_\{k\}\\sim\\mathcal\{N\}\(0,1\)⊳\\trianglerightradial update

18:

𝒛k\+1←ρk\+1​𝒖k\+1\{\\bm\{z\}\}\_\{k\+1\}\\leftarrow\\rho\_\{k\+1\}\{\\bm\{u\}\}\_\{k\+1\}
19:return

Gθ​\(𝒛K\)G\_\{\\theta\}\(\{\\bm\{z\}\}\_\{K\}\)

### B\.2Implementation of Zeroth\-Order Baselines

ZO Langevin\.We substitute the zeroth\-order gradient estimator𝒈^​\(𝒛k\)\\widehat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\_\{k\}\)in Eq\. \([7](https://arxiv.org/html/2610.00365#S3.E7)\) into Eq\. \([5](https://arxiv.org/html/2610.00365#S2.E5)\) and apply an Euler\-Maruyama step of sizeη\\eta:

𝒛k\+1=𝒛k\+η⁡\(−𝒛k\+1λ​𝒈^​\(𝒛k\)\)\+2​η​𝝃k,𝝃k∼𝒩⁡\(0,𝐈\)\.\{\\bm\{z\}\}\_\{k\+1\}=\{\\bm\{z\}\}\_\{k\}\+\\eta\\left\(\-\{\\bm\{z\}\}\_\{k\}\+\\frac\{1\}\{\\lambda\}\\widehat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\_\{k\}\)\\right\)\+\\sqrt\{2\\eta\}\\,\\bm\{\\xi\}\_\{k\},\\qquad\\bm\{\\xi\}\_\{k\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\.\(51\)
ZO Exponential Map\.Letρk=‖𝒛k‖2\\rho\_\{k\}=\\\|\{\\bm\{z\}\}\_\{k\}\\\|\_\{2\}and𝒖k=𝒛k/ρk\{\\bm\{u\}\}\_\{k\}=\{\\bm\{z\}\}\_\{k\}/\\rho\_\{k\}\. We project the same estimator onto the tangent space and update the direction through the spherical exponential map with step sizeη\\eta:

𝒈ktan=\(𝐈−𝒖k​𝒖k⊤\)​𝒈^​\(𝒛k\),𝒖k\+1=Exp𝒖k⁡\(η​ρkλ​𝒈ktan\),𝒛k\+1=ρk​𝒖k\+1\.\{\\bm\{g\}\}\_\{k\}^\{\\mathrm\{tan\}\}=\(\{\\mathbf\{I\}\}\-\{\\bm\{u\}\}\_\{k\}\{\\bm\{u\}\}\_\{k\}^\{\\top\}\)\\widehat\{\{\\bm\{g\}\}\}\(\{\\bm\{z\}\}\_\{k\}\),\\qquad\{\\bm\{u\}\}\_\{k\+1\}=\\operatorname\{Exp\}\_\{\{\\bm\{u\}\}\_\{k\}\}\\\!\\left\(\\frac\{\\eta\\rho\_\{k\}\}\{\\lambda\}\{\\bm\{g\}\}\_\{k\}^\{\\mathrm\{tan\}\}\\right\),\\qquad\{\\bm\{z\}\}\_\{k\+1\}=\\rho\_\{k\}\{\\bm\{u\}\}\_\{k\+1\}\.\(52\)
Natural Evolution Strategies\.NES\([Wierstra et al\., 2014](https://arxiv.org/html/2610.00365#bib.bib44)\)maximizes the expected rewardJ⁡\(𝜽\)=𝔼𝒛∼π𝜽​\[R⁡\(𝒛\)\]J\(\\bm\{\\theta\}\)=\\mathbb\{E\}\_\{\{\\bm\{z\}\}\\sim\\pi\_\{\\bm\{\\theta\}\}\}\[R\(\{\\bm\{z\}\}\)\]by updating the parameters of a search distribution along its natural gradient:

𝜽k\+1=𝜽k\+η​𝐅​\(𝜽k\)−1​∇^𝜽​J,∇^𝜽​J=1N​∑n=1N\(R⁡\(𝒛\(n\)\)−R¯\)​∇𝜽​log⁡π𝜽​\(𝒛\(n\)\),\\bm\{\\theta\}\_\{k\+1\}=\\bm\{\\theta\}\_\{k\}\+\\eta\\mathbf\{F\}\(\\bm\{\\theta\}\_\{k\}\)^\{\-1\}\\widehat\{\\nabla\}\_\{\\bm\{\\theta\}\}J,\\qquad\\widehat\{\\nabla\}\_\{\\bm\{\\theta\}\}J=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\(R\(\{\\bm\{z\}\}^\{\(n\)\}\)\-\\bar\{R\}\)\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\(\{\\bm\{z\}\}^\{\(n\)\}\),\(53\)where𝐅\\mathbf\{F\}is the Fisher information matrix\. Among the possible search distributions, we use an isotropic Gaussianπ𝒛k=𝒩⁡\(𝒛k,σ2​𝐈\)\\pi\_\{\{\\bm\{z\}\}\_\{k\}\}=\{\\mathcal\{N\}\}\(\{\\bm\{z\}\}\_\{k\},\\sigma^\{2\}\{\\mathbf\{I\}\}\)with fixedσ=grad\_sigma\\sigma=\\texttt\{grad\\\_sigma\}and optimize only its mean\. For𝒛k\(n\)=𝒛k\+σ​ϵ\(n\)\{\\bm\{z\}\}\_\{k\}^\{\(n\)\}=\{\\bm\{z\}\}\_\{k\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}, the resulting update is

𝒛k\+1=𝒛k\+η​σ2​𝒈^k,𝒈^k=1N​σ​∑n=1N\(R⁡\(𝒛k\+σ​ϵ\(n\)\)−R¯\)​ϵ\(n\),ϵ\(n\)∼𝒩⁡\(0,𝐈\)\.\{\\bm\{z\}\}\_\{k\+1\}=\{\\bm\{z\}\}\_\{k\}\+\\eta\\sigma^\{2\}\\widehat\{\{\\bm\{g\}\}\}\_\{k\},\\qquad\\widehat\{\{\\bm\{g\}\}\}\_\{k\}=\\frac\{1\}\{N\\sigma\}\\sum\_\{n=1\}^\{N\}\\left\(R\(\{\\bm\{z\}\}\_\{k\}\+\\sigma\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\)\-\\bar\{R\}\\right\)\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\},\\qquad\{\\bm\{\{\\epsilon\}\}\}^\{\(n\)\}\\sim\{\\mathcal\{N\}\}\(0,\{\\mathbf\{I\}\}\)\.\(54\)

### B\.3Text\-to\-Image Generation

Models\.We utilized SDXL\-turbo\([Sauer et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib33)\)and DMD2\([Yin et al\., 2024a](https://arxiv.org/html/2610.00365#bib.bib34)\)without Classifier\-Free Guidance scale for our text\-to\-image generative models\. All experiments for zeroth\-order methods are conducted using NVIDIA RTX 4090 GPUs with VRAM 24GB\. For first\-order methods which require backpropagation through reward model and generator, NVIDIA A100 with VRAM 40GB was used\.

Reward function and text prompts\.ImageReward\([Xu et al\., 2024](https://arxiv.org/html/2610.00365#bib.bib29)\)and PickScore\([Kirstain et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib38)\)return the human preference of a given image related to the text prompt\. GenEval\([Ghosh et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib37)\)evaluates the alignment between a given image and text prompt by aggregating detection result for text attributes\. For counting reward, we employ an external model T2ICount\([Qian et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib2)\)that estimates the count of given query objects in a given image\. Here, to implement a challenging black\-box reward scenario, we rounded the continuous T2ICount output to a nearest whole number before used by any initial noise optimization methods\. OCR reward is computed using PaddleOCR\([Cui et al\., 2025](https://arxiv.org/html/2610.00365#bib.bib1)\)according to a normalized Levenshtein distance between the recognized and target text\.

For counting reward task, we imported a quantity\-aware generation prompt set from[Yoon et al\. \(2026\)](https://arxiv.org/html/2610.00365#bib.bib31)which contains text prompts of various objects attached with a certain query number\. For other tasks, we used HPDv2\([Wu et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib3)\)prompts for ImageReward, Pick\-a\-Pic\([Kirstain et al\., 2023](https://arxiv.org/html/2610.00365#bib.bib38)\)prompts for PickScore and OCR, GenEval test prompts for GenEval, where we randomly selected 200 prompts\.

Hyperparameters\.For the results in Table[1](https://arxiv.org/html/2610.00365#S4.T1), we useN=16N=16particles per update for the particle\-based zeroth\-order optimization methods\. The number of iteration was set toK=25K=25for ImageReward and Counting, andK=50K=50for PickScore, GenEval, and OCR\. These settings correspond toN×K=400N\\times K=400and800800number of reward function evaluations, respectively\. First\-order methods require one function evaluation and its backpropagation per iteration, so we allocatedN×KN\\times Kiterations for each task\. Note that ZeNOVA utilizedNMHN\_\{\\text\{MH\}\}number of reward evaluations for MH jumping, so onlyN−NMHN\-N\_\{\\text\{MH\}\}number of particles were used for value gradient estimation unlike other zeroth\-order methods\. We usedNMH=4N\_\{\\text\{MH\}\}=4for all tasks\.

We set the guidance scale to1/λ=100\.01/\\lambda=100\.0for the Counting reward and1/λ=1000\.01/\\lambda=1000\.0for the rest of the reward function, unless specified\. Step size for ZeNOVA was set toηA=0\.1\\eta\_\{A\}=0\.1andηR=0\.03\\eta\_\{R\}=0\.03\. For horizon scheduling, we fixσ1=0\.9\\sigma\_\{1\}=0\.9andσK=0\.1\\sigma\_\{K\}=0\.1, then use geometric interpolation to get the intermediate horizons so that the consecutiveσ\\sigmahave the same ratio\.

### B\.4Text\-to\-Video Generation

Model\.We use rCM\([Zheng et al\., 2026b](https://arxiv.org/html/2610.00365#bib.bib8)\), a few\-step distilled text\-to\-video flow\-matching model as our reference video generator\. All experiments are conducted on a single NVIDIA RTX PRO 6000 Blackwell\.

Reward function and text prompts\.Following[Xue et al\. \(2025\)](https://arxiv.org/html/2610.00365#bib.bib7), we use VideoAlign\([Liu et al\., 2026](https://arxiv.org/html/2610.00365#bib.bib6)\)to assess generated videos along three dimensions: Visual Quality \(VQ\), Motion Quality \(MQ\), and Text Alignment \(TA\)\. We randomly select 30 text prompts from VidProM\([Wang and Yang, 2024](https://arxiv.org/html/2610.00365#bib.bib5)\)\. We conduct four separate optimization settings, using VQ, MQ, TA, or their sum \(*i\.e\.*, All\) as the reward objective\.

Hyperparameters\.We useN=32N=32particles per update andK=15K=15optimization iterations in all four reward settings, corresponding toN×K=480N\\times K=480particle reward evaluations\. The other hyperparameters are set as the same as in the text\-to\-image generation\.

## Appendix CAdditional Results

Figures[8](https://arxiv.org/html/2610.00365#A3.F8)\-[13](https://arxiv.org/html/2610.00365#A3.F13)show qualitative examples with various black\-box rewards from different text\-to\-image generative models\(SDXL\-Turbo, DMD2\) and the text\-to\-video generative model\(rCM\) with ZeNOVA\.

![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_ours_imagereward.png)Figure 8:Images generated by SDXL\-Turbo and DMD2 with ZeNOVA, using ImageReward as the reward signal\.![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_ours_pickscore.png)Figure 9:Images generated by SDXL\-Turbo and DMD2 with ZeNOVA, using PickScore as the reward signal\.![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_ours_geneval.png)Figure 10:Images generated by SDXL\-Turbo and DMD2 with ZeNOVA, using GenEval as the reward signal\.![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_ours_counting.png)Figure 11:Images generated by SDXL\-Turbo and DMD2 with ZeNOVA, using the counting reward as the reward signal\.![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_ours_ocr.png)Figure 12:Images generated by SDXL\-Turbo and DMD2 with ZeNOVA, using the OCR reward as the reward signal\.![Refer to caption](https://arxiv.org/html/2610.00365v1/figs/figure_qualitative_video_ex.png)Figure 13:Vidoes generated by rCM with ZeNOVA, using a sum of three VideoAlign rewards as the reward signal\.