D2PO: Optimizing Diffusion Samplers via Dynamic Preference
Summary
D2PO proposes a dynamic preference optimization framework that aligns diffusion sampling policies with perceptual quality using direct preference optimization, outperforming regression-based methods under low-NFE constraints.
arXiv:2607.06609v1 Announce Type: new
Abstract: We propose D2PO (Dynamic Direct Preference Optimization), a principled framework for optimizing diffusion sampling policies with respect to timestep schedules and classifier-free guidance (CFG) weights. Our work is motivated by a fundamental limitation of existing student-teacher regression frameworks; low-NFE student samplers are trained to mimic high-NFEteachers, often sacrificing high-frequency texture fidelity while preserving coarse global structures, thereby misaligning the sampler with perceptual quality. D2PO addresses this challenge by reformulating sampler optimization as a preference-based alignment problem, leveraging the Direct Preference Optimization (DPO) framework. To make DPO applicable to diffusion samplers, we model the sampling policy as an energy-based model (EBM), transforming preference comparisons into tractable energy differences. We further introduce a novel energy formulation derived directly from the pretrained score network, enabling preference evaluation in perturbed spaces that jointly capture structural consistency and fine-grained details. Moreover, we introduce dynamic preferences, where the preferred samples used for alignment progressively improve as the sampling policies are learned. This self-improving mechanism replaces rigid static teacher supervision with an iterative, preference-guided refinement process, providing progressively stronger alignment signals. Extensive experiments demonstrate that D2PO aligns diffusion samplers with perceptual quality more faithfully, unlocking the full potential of high-quality teachers and consistently outperforming conventional regression-based schedulers under low-NFE constraints.
View Cached Full Text
Cached at: 07/09/26, 07:41 AM
# D2PO: Optimizing Diffusion Samplers via Dynamic Preference
Source: [https://arxiv.org/html/2607.06609](https://arxiv.org/html/2607.06609)
11institutetext:1ECE &2IPAI, Seoul National University, Korea
3AIGS, Ulsan National Institute of Science and Technology, Korea###### Abstract
We propose D2PO \(Dynamic Direct Preference Optimization\), a principled framework for optimizing diffusion sampling policies with respect to timestep schedules and classifier\-free guidance \(CFG\) weights\. Our work is motivated by a fundamental limitation of existing student\-teacher regression frameworks; low\-NFE student samplers are trained to mimic high\-NFE teachers, often sacrificing high\-frequency texture fidelity while preserving coarse global structures, thereby misaligning the sampler with perceptual quality\. D2PO addresses this challenge by reformulating sampler optimization as a preference\-based alignment problem, leveraging the Direct Preference Optimization \(DPO\) framework\. To make DPO applicable to diffusion samplers, we model the sampling policy as an energy\-based model \(EBM\), transforming preference comparisons into tractable energy differences\. We further introduce a novel energy formulation derived directly from the pretrained score network, enabling preference evaluation in perturbed spaces that jointly capture structural consistency and fine\-grained details\. Moreover, we introduce dynamic preferences, where the preferred samples used for alignment progressively improve as the sampling policies are learned\. This self\-improving mechanism replaces rigid static teacher supervision with an iterative, preference\-guided refinement process, providing progressively stronger alignment signals\. Extensive experiments demonstrate that D2PO aligns diffusion samplers with perceptual quality more faithfully, unlocking the full potential of high\-quality teachers and consistently outperforming conventional regression\-based schedulers under low\-NFE constraints\.
## 1Introduction
Diffusion Probabilistic Models \(DPMs\)\[sohl2015deep,ho2020denoising,song2021scorebased\]have achieved unprecedented fidelity in high\-resolution image synthesis, text\-to\-image generation\[rombach2022high,dhariwal2021diffusion\], and video generation\[ho2022video,singer2022make,zhou2022magic,wang2023modelscope\]\. However, this performance comes at a substantial computational cost\. DPMs are inherently iterative, requiring many function evaluations \(NFE\) during sampling, which makes high\-quality generation expensive and limits practical deployment\.
A broad range of approaches has been explored to mitigate this bottleneck, including accelerated numerical solvers\[dockhorn2022genie,lu2022fastode,lu2023dpm,liu2022pseudo,zhao2023unipc,zhang2023lookahead,choi2025rx\-dpm\], few\-step knowledge distillation\[salimans2022progressive,song2023consistency,kim2024consistency,zheng2024trajectory,salimans2024moment,yin2024onestep,yin2024improved,zhou2024score,zhou2025adversarial\], architectural modifications\[ma2024learning,ye2024training\], and training\-time improvements\[kingma2021variational,vahdat2021score,xiao2021tackling,kang2024ogdm\]\. More recently, directly optimizing the sampling policy parameters—such as timestep schedules\[li2023autodiffusion,watson2021learning,tong2024learning,sabour2024align,xue2024accelerating,frankels4s\], classifier\-free guidance weights\[galashov2025learn\], and high\-order solver coefficients\[frankels4s,zhang2024iia,wang2026image\]—has emerged as a critical direction for acceleration\.
“Two birds that are sitting in a marsh area\.”
Δ=1\\Delta=1
Δ=2\\Delta=2
Δ=3\\Delta=3
Δ=4\\Delta=4
“There is a small bus with several people standing next to it\."
Δ=1\\Delta=1
Δ=2\\Delta=2
Δ=3\\Delta=3
Δ=4\\Delta=4
Figure 1:Qualitative evidence of the performance bottleneck in LD3\[tong2024learning\]\. All images are generated by the same model withNFE=4\\text\{NFE\}=4\. The columns show the impact of increasing the NFE gap \(Δ=T−S\\Delta=T\-S\) between the teacher \(TT\) and the student \(SS\)\. While a small gap \(Δ=1\\Delta=1\) yields high\-quality outputs, larger gaps \(up toΔ=4\\Delta=4\) lead to severe artifacts, demonstrating LD3’s inability to leverage high\-fidelity teachers\.In optimizing these sampler parameters, prior works typically adopt either a distribution or an instance perspective\. Specifically, one line of work optimizes distribution\-level metrics, such as FID\[li2023autodiffusion\]or KID\[watson2022learning\]over large sample sets, but such population\-level objectives yield weak, high\-variance gradients for the low\-dimensional sampler parameters\. Alternatively, instance\-wise distillation methods\[tong2024learning,frankels4s\]regress a low\-NFE student sampler onto the trajectories or outputs of a fixed high\-NFE teacher viaℓ2\\ell\_\{2\}or LPIPS\[zhang2018unreasonable\]losses\. Although effective when the student\-teacher gap is moderate, this regression paradigm exhibits a structural limitation when aggressive acceleration is desired\.
When the NFE gap between the student and teacher becomes substantial—a common scenario when striving for maximum acceleration—this regression\-based objective forces the student sampler to approximate a high\-fidelity trajectory within its restricted capacity, compromising generation quality\. This structural constraint often suppresses high\-frequency textures and fine\-grained details, preventing the student from fully benefiting from stronger teachers\. We empirically validate this phenomenon in[Fig\.˜1](https://arxiv.org/html/2607.06609#S1.F1); as the teacher NFE increases while the student NFE remains fixed, the perceptual quality degrades, even for a state\-of\-the\-art scheduler \(LD3\[tong2024learning\]\)\. This degradation directly reflects the structural rigidity of fixed\-teacher regression, where the student is forced to prioritize low\-level alignment over perceptual quality, failing to discover more effective sampling paths\.
To address this limitation, we reinterpret sampler optimization as a preference\-based alignment problem rather than a regression\-to\-teacher task\. We introduce D2PO \(Dynamic Direct Preference Optimization\), a framework inspired by DPO but adapted to diffusion sampling policies\. Applying DPO to diffusion samplers is non\-trivial because the marginalized log\-probability is intractable\. To obtain a tractable surrogate, we model the policy\-induced distribution as an Energy\-Based Model \(EBM\)\. We define the energy using a novel score\-based distance that measures discrepancies between samples leveraging the pretrained diffusion score model\. By comparing score predictions across multiple noise levels, this metric captures both structural and high\-frequency differences that conventional perceptual metrics fail to reflect\.
D2PO replaces the static teacher framework with a dynamic reference mechanism that evolves alongside the student policy\. Specifically, at each training step, the preference pair is constructed by executing the current policy under two different computational budgets: the losing sample is generated using a fast, sparse timestep schedule, while the winning sample is obtained via a denser, more refined schedule of the same policy\. Instead of imitating an external, immutable target, the student is encouraged to align with its own high\-quality, dense\-schedule approximation\. This dynamic preference loop eliminates the fixed error floor inherent in static distillation and implicitly drives the sampler to minimize discretization errors, thereby promoting highly accurate and self\-improving sampling trajectories\.
Our contributions are summarized as follows:
- ∙\\bulletWe propose D2PO, a preference\-based framework for optimizing diffusion samplers, establishing a tractable alignment objective by modeling the deterministic policy as an energy\-based surrogate\.
- ∙\\bulletWe formulate a novel score\-based energy metric derived from the pretrained score network, providing a multi\-scale learning signal that captures fine\-grained textural and structural details beyond conventional perceptual losses\.
- ∙\\bulletWe introduce a dynamic preference mechanism that replaces static teacher supervision with a refinement\-based target, enabling continual self\-improvement without being bounded by a fixed residual error\.
- ∙\\bulletWe comprehensively validate that D2PO learns superior sampling policies, outperforming state\-of\-the\-art distillation\-based baselines under various experimental settings\.
## 2Related Work
### 2\.1Optimizing diffusion sampling parameters
Since the trajectory of time steps profoundly impacts generation quality under a fixed computational budget, substantial research has focused on finding optimal sampling schedules\. Early heuristic approaches, such as EDM\[karras2022elucidating\], employ polynomial spacing to densify steps near the clean data manifold, while Watson*et al*\.\[watson2021learning\]introduce a dynamic programming framework to search for optimal discrete schedules that maximize log\-likelihood\. Analytic\-DPM\[bao2022analytic\]improves efficiency by deriving training\-free, optimal analytical forms for reverse variances directly from the pretrained score network, while obtaining the corresponding optimal trajectory via dynamic programming\[watson2021learning\]\. To automate and generalize schedule optimization, AutoDiffusion\[li2023autodiffusion\]employs an evolutionary search targeted at minimizing FID, while DDSS\[watson2022learning\]optimizes sampler parameters via direct sample\-quality feedback such as KID\[binkowski2018demystifying\]\.
Another line of work derives analytical error bounds or geometric properties of ODE/SDE trajectories to optimize time discretization\. Methods such as those by Chen*et al*\.\[chen2024adaptive,chen2024trajectory\], AYS\[sabour2024align\], and Xue*et al*\.\[xue2024accelerating\]dynamically adjust step sizes based on trajectory curvature or upper bounds of solver errors\. More recently, LD3\[tong2024learning\]adopts a relaxed matching objective to learn discretized trajectories through student\-teacher regression\.
Beyond timestep optimization, recent literature explores tuning other sampling parameters to further accelerate inference\. For instance, Galashov*et al*\.\[galashov2025learn\]learn time\-dependent CFG weights via a self\-consistency objective\. Similarly, S4S\[frankels4s\]optimizes solver coefficients at each step using teacher\-student matching\. Extending this direction, ConsistencySolver\[wang2026image\]employs a learnable high\-order solver to dynamically predict optimal integration coefficients\.
### 2\.2Aligning pretrained models with preferences
Driven by the limitations of predefined training objectives, aligning generative models directly with pairwise human or AI preferences has emerged as a dominant paradigm\. This approach originated in large language models via Reinforcement Learning from Human Feedback \(RLHF\)\[ouyang2022training\], which optimizes policies using a separate reward model\. To simplify this multi\-stage pipeline, Direct Preference Optimization \(DPO\)\[rafailov2024direct\]integrates the reward implicitly into the classification loss, enabling stable and direct policy updates\. Subsequent self\-play frameworks like SPIN\[chen2024self\]further remove the need for preference annotations, generating negatives from the model itself and contrasting them with SFT responses\.
Recently, this preference alignment paradigm has been actively adapted to text\-to\-image diffusion and flow\-matching models to enhance visual quality and aesthetic appeal\. Standard post\-training methods, including DPO\-style formulations\[wallace2024diffusion,liang2025aesthetic,yang2024using,yuan2024self\]and online reinforcement learning variants\[black2023training,fan2024reinforcement,liu2025flow\], predominantly focus on fine\-tuning the foundational weights of the denoiser or velocity networks\. While effective, optimizing high\-dimensional model parameters is computationally expensive and risks degrading the quality of outputs\. In contrast, D2PO keeps the generative backbone frozen and exclusively optimizes the low\-dimensional sampling policy, offering a highly lightweight, orthogonal, and complementary solution to existing weight\-tuning approaches\.
## 3Preliminaries
This section briefly reviews score\-based diffusion models and Direct Preference Optimization \(DPO\), which provide the theoretical foundation for our dynamic sampler optimization framework\.
### 3\.1Diffusion probabilistic models
We consider score\-based diffusion models\[ho2020denoising,song2021scorebased\], which transform a data distributionpreal\(𝒙0\)p\_\{\\mathrm\{real\}\}\(\\bm\{x\}\_\{0\}\)into a tractable prior through a gradual noising process\. Specifically, we adopt the Variance Preserving \(VP\) stochastic differential equation \(SDE\), which is given by
d𝒙t=−12β\(t\)𝒙tdt\+β\(t\)d𝐰t,\\displaystyle d\\bm\{x\}\_\{t\}=\-\\frac\{1\}\{2\}\\beta\(t\)\\bm\{x\}\_\{t\}\\,dt\+\\sqrt\{\\beta\(t\)\}\\,d\\mathbf\{w\}\_\{t\},\(1\)where𝐰t\\mathbf\{w\}\_\{t\}denotes a standard Wiener process andt∈\[0,T\]t\\in\[0,T\]\. This forward process admits a closed\-form marginal:
pt\(𝒙t∣𝒙0\)=𝒩\(𝒙t;αt𝒙0,σt2𝐈\),\\displaystyle p\_\{t\}\(\\bm\{x\}\_\{t\}\\mid\\bm\{x\}\_\{0\}\)=\\mathcal\{N\}\\\!\\left\(\\bm\{x\}\_\{t\};\\alpha\_\{t\}\\bm\{x\}\_\{0\},\\sigma\_\{t\}^\{2\}\\mathbf\{I\}\\right\),\(2\)where
αt=exp\(−12∫0tβ\(s\)𝑑s\)andσt2=1−αt2\.\\displaystyle\\alpha\_\{t\}=\\exp\\\!\\left\(\-\\frac\{1\}\{2\}\\\!\\int\_\{0\}^\{t\}\\beta\(s\)\\,ds\\right\)\\qquad\\text\{and\}\\qquad\\sigma\_\{t\}^\{2\}=1\-\\alpha\_\{t\}^\{2\}\.\(3\)
To construct the reverse process, a neural networksθ\(𝒙t,t\)s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)is trained to approximate the score function of the marginal distribution,*i\.e*\.,s\(𝒙t,t\)=∇𝒙tlogpt\(𝒙t\)s\(\\bm\{x\}\_\{t\},t\)=\\nabla\_\{\\bm\{x\}\_\{t\}\}\\log p\_\{t\}\(\\bm\{x\}\_\{t\}\)\. This is achieved via denoising score matching \(DSM\), which minimizes
ℒDSM=∫0Tλ\(t\)𝔼𝒙0∼preal,𝒙t∼pt\(⋅\|𝒙0\)\[∥sθ\(𝒙t,t\)−∇𝒙tlogpt\(𝒙t\|𝒙0\)∥22\]dt,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{DSM\}\}=\\int\_\{0\}^\{T\}\\lambda\(t\)\\,\\mathbb\{E\}\_\{\\bm\{x\}\_\{0\}\\sim p\_\{\\mathrm\{real\}\},\\,\\bm\{x\}\_\{t\}\\sim p\_\{t\}\(\\cdot\|\\bm\{x\}\_\{0\}\)\}\\left\[\\bigl\\\|s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)\-\\nabla\_\{\\bm\{x\}\_\{t\}\}\\log p\_\{t\}\(\\bm\{x\}\_\{t\}\|\\bm\{x\}\_\{0\}\)\\bigr\\\|\_\{2\}^\{2\}\\right\]dt,\(4\)where
∇𝒙tlogpt\(𝒙t\|𝒙0\)=−𝒙t−αt𝒙0σt2\.\\displaystyle\\nabla\_\{\\bm\{x\}\_\{t\}\}\\log p\_\{t\}\(\\bm\{x\}\_\{t\}\|\\bm\{x\}\_\{0\}\)=\-\\frac\{\\bm\{x\}\_\{t\}\-\\alpha\_\{t\}\\bm\{x\}\_\{0\}\}\{\\sigma\_\{t\}^\{2\}\}\.\(5\)
Oncesθs\_\{\\theta\}is trained, samples are generated by solving the corresponding reverse\-SDE fromt=Tt=Ttot=0t=0, which is given by
d𝒙t=\[−12β\(t\)𝒙t−β\(t\)sθ\(𝒙t,t\)\]dt\+β\(t\)d𝐰¯t,\\displaystyle d\\bm\{x\}\_\{t\}=\\Bigl\[\-\\frac\{1\}\{2\}\\beta\(t\)\\bm\{x\}\_\{t\}\-\\beta\(t\)\\,s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)\\Bigr\]dt\+\\sqrt\{\\beta\(t\)\}\\,d\\bar\{\\mathbf\{w\}\}\_\{t\},\(6\)where𝐰¯t\\bar\{\\mathbf\{w\}\}\_\{t\}is a reverse\-time Wiener process\.
This score\-based formulation is equivalent to the noise\-prediction parameterizationϵθ\(𝒙t,t\)\\epsilon\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)in DDPM\[ho2020denoising\]\. The two representations are related by
sθ\(𝒙t,t\)=−ϵθ\(𝒙t,t\)σt\.\\displaystyle s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)=\-\\frac\{\\epsilon\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)\}\{\\sigma\_\{t\}\}\.\(7\)
### 3\.2Direct preference optimization
We build upon Direct Preference Optimization \(DPO\)\[rafailov2024direct\], a framework for aligning generative policies with preference data\. DPO provides a closed\-form solution to the KL\-regularized reward maximization problem commonly used in RLHF\[ouyang2022training\], which is defined as
maxπ𝔼𝐱∼π\(⋅\|c\)\[r\(𝐱,c\)\]−βDKL\(π\(⋅\|c\)∥πref\(⋅\|c\)\),\\displaystyle\\max\_\{\\pi\}\\;\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\pi\(\\cdot\|c\)\}\\bigl\[r\(\\mathbf\{x\},c\)\\bigr\]\-\\beta D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\pi\(\\cdot\|c\)\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\|c\)\\right\),\(8\)whereπref\\pi\_\{\\mathrm\{ref\}\}is a reference policy andβ\(\>0\)\\beta\(\>0\)controls the strength of regularization\.
Rather than explicitly learning a reward modelr\(𝐱,c\)r\(\\mathbf\{x\},c\)under the contextccand performing reinforcement learning, DPO leverages preference pairs\(𝐱w,𝐱l\)\(\\mathbf\{x\}\_\{w\},\\mathbf\{x\}\_\{l\}\), where𝐱w\\mathbf\{x\}\_\{w\}is preferred over𝐱l\\mathbf\{x\}\_\{l\}\. Specifically, by analyzing the optimal solution of Eq\. \([8](https://arxiv.org/html/2607.06609#S3.E8)\), the reward difference between two samples can be expressed via the log\-likelihood ratio of the optimal policy relative to the reference policy\. This leads to a logistic classification objective on preference pairs as
ℒDPO\(ϕ\)=−𝔼𝒟\[logσ\(β\(logπϕ\(𝐱w\|c\)πref\(𝐱w\|c\)−logπϕ\(𝐱l\|c\)πref\(𝐱l\|c\)\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{DPO\}\}\(\\phi\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\log\\sigma\\\!\\left\(\\beta\\Bigl\(\\log\\frac\{\\pi\_\{\\phi\}\(\\mathbf\{x\}\_\{w\}\|c\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\mathbf\{x\}\_\{w\}\|c\)\}\-\\log\\frac\{\\pi\_\{\\phi\}\(\\mathbf\{x\}\_\{l\}\|c\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(\\mathbf\{x\}\_\{l\}\|c\)\}\\Bigr\)\\right\)\\right\],\(9\)whereσ\(⋅\)\\sigma\(\\cdot\)denotes the sigmoid function\. This formulation eliminates the need for an explicit reward model, directly updating the policyπϕ\\pi\_\{\\phi\}to maximize the relative log\-likelihood of preferred samples over unpreferred ones while remaining anchored to the reference policy\.
## 4D2PO: Dynamic Direct Preference Optimization
### 4\.1Problem formulation
We aim to optimize the sampling policy of a pretrained diffusion modelsθs\_\{\\theta\}by learning a set of sampler parametersϕ=\{𝒮,𝝎\}\\phi=\\\{\\mathcal\{S\},\\bm\{\\omega\}\\\}, where𝒮\\mathcal\{S\}and𝝎\\bm\{\\omega\}denote the timestep schedule and the per\-step classifier\-free guidance \(CFG\) weights, respectively\. Given a promptccand an initial noise𝒙T∼𝒩\(𝟎,𝐈\)\\bm\{x\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), our sampler,*i\.e*\., ODE solver, deterministically generates an image as
𝒙ϕ=Φϕ\(𝒙T,c;θ\),\\bm\{x\}\_\{\\phi\}=\\Phi\_\{\\phi\}\(\\bm\{x\}\_\{T\},c;\\theta\),\(10\)whereΦϕ\\Phi\_\{\\phi\}denotes a fixed numerical solver parameterized byϕ\\phi\. AlthoughΦϕ\\Phi\_\{\\phi\}is deterministic, it induces a conditional distribution over generated images through the randomness of the initial noise𝒙T\\bm\{x\}\_\{T\}:
qϕ\(𝒙\|c\)=∫δ\(𝒙−Φϕ\(𝒙T,c;θ\)\)p\(𝒙T\)𝑑𝒙T,q\_\{\\phi\}\(\\bm\{x\}\|c\)=\\int\\delta\(\\bm\{x\}\-\\Phi\_\{\\phi\}\(\\bm\{x\}\_\{T\},c;\\theta\)\)\\,p\(\\bm\{x\}\_\{T\}\)\\,d\\bm\{x\}\_\{T\},\(11\)whereδ\(⋅\)\\delta\(\\cdot\)denotes the Dirac delta function\.
Our goal is to alignqϕ\(𝒙\|c\)q\_\{\\phi\}\(\\bm\{x\}\|c\)with perceptually preferred outputs using pairwise preference tuples\(c,𝒙T,𝒙w,𝒙l\)\(c,\\bm\{x\}\_\{T\},\\bm\{x\}\_\{w\},\\bm\{x\}\_\{l\}\), where𝒙w\\bm\{x\}\_\{w\}is preferred over𝒙l\\bm\{x\}\_\{l\}\. A central challenge is thatqϕ\(𝒙\|𝒙T,c\)=δ\(𝒙−𝒙ϕ\)q\_\{\\phi\}\(\\bm\{x\}\|\\bm\{x\}\_\{T\},c\)=\\delta\(\\bm\{x\}\-\\bm\{x\}\_\{\\phi\}\)is degenerate, rendering its likelihood non\-differentiable and direct preference optimization ill\-posed\. We address this by introducing a tractable surrogate formulation\.
### 4\.2Energy\-based surrogate for deterministic policies
To apply Direct Preference Optimization \(DPO\) to our sampler, we replaceqϕ\(𝒙\|𝒙T,c\)q\_\{\\phi\}\(\\bm\{x\}\|\\bm\{x\}\_\{T\},c\)with a smooth surrogate policyπϕ\(𝒙\|c,𝒙T\)\\pi\_\{\\phi\}\(\\bm\{x\}\|c,\\bm\{x\}\_\{T\}\), defined as an Energy\-Based Model \(EBM\):
πϕ\(𝒙\|c,𝒙T\)=1Z\(ϕ,c,𝒙T\)exp\(−αE\(𝒙;ϕ,c,𝒙T\)\),\\pi\_\{\\phi\}\(\\bm\{x\}\|c,\\bm\{x\}\_\{T\}\)=\\frac\{1\}\{Z\(\\phi,c,\\bm\{x\}\_\{T\}\)\}\\exp\\\!\\left\(\-\\alpha E\(\\bm\{x\};\\phi,c,\\bm\{x\}\_\{T\}\)\\right\),\(12\)whereZZis the partition function andα\(\>0\)\\alpha~\(\>0\)a temperature parameter\. We define the energy as a distance to the sampler output, which is given by
E\(𝒙;ϕ,c,𝒙T\)≡d\(𝒙,𝒙ϕ\),E\(\\bm\{x\};\\phi,c,\\bm\{x\}\_\{T\}\)\\equiv d\(\\bm\{x\},\\bm\{x\}\_\{\\phi\}\),\(13\)whered\(⋅,⋅\)d\(\\cdot,\\cdot\)is a predefined distance metric\.
This surrogate assigns high probability to images close to the sampler’s output𝒙ϕ\\bm\{x\}\_\{\\phi\}and smoothly decays as the proximity decreases\. Such functional relaxations are commonly employed to bypass the non\-differentiability of objective functions for optimization\. For example, score\-based models apply Gaussian perturbations—mathematically forming EBMs withℓ2\\ell^\{2\}energy—to resolve undefined gradients\. Soft Actor\-Critic\[haarnoja2018soft\]and the Gumbel–Softmax\[jang2017categorical\]use probabilistic relaxations on discrete policies and operations to enable backpropagation\. Our surrogate plays a similar role for generative optimization: it replaces the non\-differentiable Dirac delta function with a smooth landscape whose mode coincides with𝒙ϕ\\bm\{x\}\_\{\\phi\}, thereby binding updates to the true generative process while keeping the objective differentiable\.
Importantly, when computing the log\-probability ratio between two candidates sharing the same context\(c,𝐱T\)\(c,\\mathbf\{x\}\_\{T\}\), the identical partition functionZ\(ϕ,c,𝐱T\)Z\(\\phi,c,\\mathbf\{x\}\_\{T\}\)cancels out, allowing the ratio to simplify exactly as follows:
logπϕ\(𝒙w\|c,𝒙T\)πϕ\(𝒙l\|c,𝒙T\)=−α\(E\(𝒙w;ϕ,c,𝒙T\)−E\(𝒙l;ϕ,c,𝒙T\)\)\.\\displaystyle\\log\\frac\{\\pi\_\{\\phi\}\(\\bm\{x\}\_\{w\}\|c,\\bm\{x\}\_\{T\}\)\}\{\\pi\_\{\\phi\}\(\\bm\{x\}\_\{l\}\|c,\\bm\{x\}\_\{T\}\)\}=\-\\alpha\\left\(E\(\\bm\{x\}\_\{w\};\\phi,c,\\bm\{x\}\_\{T\}\)\-E\(\\bm\{x\}\_\{l\};\\phi,c,\\bm\{x\}\_\{T\}\)\\right\)\.\(14\)By applying this to both the student samplerϕ\\phiand the reference samplerϕref\\phi\_\{\\text\{ref\}\}and substituting this expression into the DPO objective \([Eq\.˜9](https://arxiv.org/html/2607.06609#S3.E9)\), we derive the final D2PO objective:
ℒD2PO\(ϕ\)=−𝔼𝒟\[logσ\(βΔϕ\(𝒙w,𝒙l\)\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{D2PO\}\}\(\\phi\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\left\[\\log\\sigma\\left\(\\beta\\Delta\_\{\\phi\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{l\}\)\\right\)\\right\],\(15\)where
Δϕ\(𝒙w,𝒙l\)=\(d\(𝒙w,𝒙ϕref\)−d\(𝒙w,𝒙ϕ\)\)−\(d\(𝒙l,𝒙ϕref\)−d\(𝒙l,𝒙ϕ\)\),\\displaystyle\\Delta\_\{\\phi\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{l\}\)=\\big\(d\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\_\{\\mathrm\{ref\}\}\}\)\-d\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\}\)\\big\)\-\\big\(d\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\_\{\\mathrm\{ref\}\}\}\)\-d\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\}\)\\big\),andϕref\\phi\_\{\\mathrm\{ref\}\}denotes the reference policy\. The temperature parameterα\\alphais implicitly absorbed into the scaling factorβ\\betafor simplicity\.
### 4\.3Score\-based distance
The effectiveness of D2PO depends on the choice of the distance functiond\(⋅,⋅\)d\(\\cdot,\\cdot\)used in the energy definition of the surrogate policyπϕ\\pi\_\{\\phi\}\. A naïve choice would adopt a predefined metric such asℓ2\\ell\_\{2\}or LPIPS, or train a separate network to approximate the energy\. However, such choices fail to exploit the rich representations already encoded within the pretrained diffusion model\.
#### 4\.3\.1Score\-induced energy\.
Our key insight is to derive the energy directly from the pretrained score networksθ\(𝒙t,t\)s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)\. Recall thatsθs\_\{\\theta\}approximates the data score∇𝒙tlogpt\(𝒙t\)\\nabla\_\{\\bm\{x\}\_\{t\}\}\\log p\_\{t\}\(\\bm\{x\}\_\{t\}\)at noise leveltt\. Since the score characterizes the geometry of the data distribution, it naturally quantifies sample likelihood\.
A sample that lies on the true data manifold should be locally consistent with this score field\. Conversely, a sample out of the data distribution fails to align with the score trajectory\. We therefore define the ideal score\-induced energy of a sample as the degree of its misalignment with the learned score geometry as follows:
E\(𝒙;ϕ\)≡∫0Tw\(t\)‖sθ\(𝒙t,t\)−sθ\(𝒙ϕ,t,t\)‖22𝑑t,\\displaystyle E\(\\bm\{x\};\\phi\)\\equiv\\int\_\{0\}^\{T\}w\(t\)\\,\\left\\\|s\_\{\\theta\}\(\\bm\{x\}\_\{t\},t\)\-s\_\{\\theta\}\(\\bm\{x\}\_\{\\phi,t\},t\)\\right\\\|\_\{2\}^\{2\}\\,dt,\(16\)wherew\(t\)w\(t\)is a weighting function over noise levels, and𝒙t\\bm\{x\}\_\{t\}and𝒙ϕ,t\\bm\{x\}\_\{\\phi,t\}denote the perturbed versions of𝒙\\bm\{x\}and𝒙ϕ\\bm\{x\}\_\{\\phi\}at noise leveltt\.
We evaluate the score discrepancy over the perturbed data distributions rather than the clean data distribution \(t=0t=0\) for both theoretical and practical reasons\. In standard score\-based generative modeling\[song2021scorebased,yin2024onestep,wang2023prolificdreamer\], the clean data score is unavailable and pretrained diffusion models do not directly learn this clean score\. Instead, they are trained to approximate the scores of perturbed distributionsptp\_\{t\}across a continuous spectrum of noise levels\. These noisy score fields encode the multi\-scale geometry of the data manifold, capturing coarse semantic structures at largettand fine\-grained details at smalltt\. Leveraging these perturbed distributions is therefore tractable and consistent with the objective of the pretrained diffusion model\.
#### 4\.3\.2Noise\-prediction distance\.
To transform the ideal energy formulation in Eq\. \([16](https://arxiv.org/html/2607.06609#S4.E16)\) into a practical optimization objective, we reframe the score discrepancy via the noise\-prediction error as follows:
dθ\(𝐱,𝐱ϕ;t\)=‖ϵθ\(𝐱t,t\)−ϵθ\(𝐱ϕ,t,t\)‖22d\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{x\}\_\{\\phi\};t\)=\\left\\\|\\epsilon\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)\-\\epsilon\_\{\\theta\}\(\\mathbf\{x\}\_\{\\phi,t\},t\)\\right\\\|\_\{2\}^\{2\}\(17\)which leverages the implicit relation,sθ=−ϵθ/σts\_\{\\theta\}=\-\\epsilon\_\{\\theta\}/\\sigma\_\{t\}\. Directly substituting this noise\-prediction distance for the score\-based distance in Eq\. \([16](https://arxiv.org/html/2607.06609#S4.E16)\) under a uniform weighting, however, introduces a scale mismatch across different noise levels\. Specifically, because the score discrepancy equals the noise\-prediction distance up to a scaling factor ofσt−2\\sigma\_\{t\}^\{\-2\}, the integrand tends to diverge numerically ast→0t\\to 0, causing the low\-noise terms to dominate the overall energy\. To resolve this imbalance and stabilize the optimization, we follow the established practice in DDPM\[ho2020denoising\]by adopting the weighting functionw\(t\)=σt2w\(t\)=\\sigma\_\{t\}^\{2\}, which cancels theσt−2\\sigma\_\{t\}^\{\-2\}factor; the score\-induced energy in Eq\. \([16](https://arxiv.org/html/2607.06609#S4.E16)\) reduces to the total noise\-prediction distance over the continuous trajectory, which is given by
E\(𝐱;ϕ\)=∫0Tdθ\(𝐱,𝐱ϕ;t\)𝑑t\.E\(\\mathbf\{x\};\\phi\)=\\int\_\{0\}^\{T\}d\_\{\\theta\}\(\\mathbf\{x\},\\mathbf\{x\}\_\{\\phi\};t\)\\,dt\.\(18\)As evaluating this continuous integral is computationally expensive, we approximate it in practice via Monte Carlo sampling\.
#### 4\.3\.3Practical D2PO objective\.
Substituting the weighted score\-based distance for the surrogate policyπϕ\\pi\_\{\\phi\}and applying the DPO objective between the student policyπϕ\\pi\_\{\\phi\}and the reference policyπϕref\\pi\_\{\\phi\_\{\\mathrm\{ref\}\}\}, we obtain the following objective:
ℒD2PO\(ϕ\)=−𝔼𝒟\[logσ\(𝔼t∼𝒰\(0,T\)\[βΔϕ\(𝒙w,𝒙l;t\)\]\)\],\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{D2PO\}\}\(\\phi\)=\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\Bigg\[\\log\\sigma\\Big\(\\mathbb\{E\}\_\{t\\sim\\mathcal\{U\}\(0,T\)\}\\Big\[\\beta\\Delta\_\{\\phi\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{l\};t\)\\Big\]\\Big\)\\Bigg\],\(19\)where
Δϕ\(𝒙w,𝒙l;t\)=\(dθ\(𝒙w,𝒙ϕref;t\)−dθ\(𝒙w,𝒙ϕ;t\)\)−\(dθ\(𝒙l,𝒙ϕref;t\)−dθ\(𝒙l,𝒙ϕ;t\)\)\.\\displaystyle\\Delta\_\{\\phi\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{l\};t\)\\hskip\-1\.42262pt=\\hskip\-1\.42262pt\\big\(d\_\{\\theta\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\_\{\\mathrm\{ref\}\}\};t\)\\hskip\-1\.42262pt\-\\hskip\-1\.42262ptd\_\{\\theta\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\};t\)\\big\)\\hskip\-1\.42262pt\-\\hskip\-1\.42262pt\\big\(d\_\{\\theta\}\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\_\{\\mathrm\{ref\}\}\};t\)\\hskip\-1\.42262pt\-\\hskip\-1\.42262ptd\_\{\\theta\}\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\};t\)\\big\)\.A key benefit of the proposed noise\-prediction distance lies in its evaluation over multi\-level noise\. The score field captures coarse semantic structure at large noise levelsttand fine\-grained detail at smalltt, and the weightingw\(t\)=σt2w\(t\)=\\sigma\_\{t\}^\{2\}aggregates these scales into a single well\-conditioned signal overtt\. This yields a significantly richer preference signal compared to perceptual metrics such as LPIPS\.
### 4\.4Dynamic preference
To understand D2PO and its dynamic optimization mechanism, it is essential to identify three key components: the*student sampler*, the*reference sampler*, and the*winning sampler*, which are parameterized byϕ\\phi,ϕref\\phi\_\{\\text\{ref\}\}, andϕ′\\phi^\{\\prime\}, respectively\.
The student sampler, governed by the target parametersϕ=\{𝒮,𝝎\}\\phi=\\\{\\mathcal\{S\},\\bm\{\\omega\}\\\}, represents the policy we aim to optimize\. To establish a preference comparison for DPO training, we utilize its output𝒙ϕ\\bm\{x\}\_\{\\phi\}as a baseline rather than relying on an external target\. Specifically, the losing sample𝒙l\\bm\{x\}\_\{l\}is synthesized by applying a degradation operator𝒢\\mathcal\{G\}\(e\.g\., a low\-pass filter\) to the student output, expressed as𝒙l=𝒢\(sg\[𝒙ϕ\]\)\\bm\{x\}\_\{l\}=\\mathcal\{G\}\(\\mathrm\{sg\}\[\\bm\{x\}\_\{\\phi\}\]\), wheresg\[⋅\]\\mathrm\{sg\}\[\\cdot\]denotes the stop\-gradient operator\.
Unlike standard DPO which employs a static reference model, D2PO dynamically updates the reference parametersϕref=\{𝒮ref,𝝎ref\}\\phi\_\{\\text\{ref\}\}=\\\{\\mathcal\{S\}\_\{\\text\{ref\}\},\\bm\{\\omega\}\_\{\\text\{ref\}\}\\\}\. Inspired by SPIN\[chen2024self\], the reference timestep schedule𝒮ref\\mathcal\{S\}\_\{\\text\{ref\}\}is synchronized by copying the student schedule𝒮\\mathcal\{S\}at the end of each epoch\. Meanwhile, the reference CFG weights𝝎ref\\bm\{\\omega\}\_\{\\text\{ref\}\}are adjusted at each training step via an Exponential Moving Average \(EMA\) with a momentum parameterλ\\lambda, i\.e\.,𝝎ref←λ𝝎ref\+\(1−λ\)𝝎\.\\bm\{\\omega\}\_\{\\text\{ref\}\}\\leftarrow\\lambda\\bm\{\\omega\}\_\{\\text\{ref\}\}\+\(1\-\\lambda\)\\bm\{\\omega\}\.
Designing a dynamic winning sampler is a core contribution of D2PO\. Instead of introducing a pre\-computed, fixed teacher, we formulate a dynamic teacher sampler whose implied distribution,πϕ′\\pi\_\{\\phi^\{\\prime\}\}, is generated relative to the current student parametersϕ\\phiat each training step\. For instance, if the student schedule𝒮\\mathcal\{S\}dictates a coarse trajectory withNNtimesteps, we construct the dynamic teacher’s schedule𝒮′\\mathcal\{S\}^\{\\prime\}by refining𝒮\\mathcal\{S\}with additional intermediate timesteps, yielding a denser2N2N\-step trajectory \(e\.g\., via linear interpolation\)\. The dynamic teacher sampler, parameterized byϕ′\\phi^\{\\prime\}, is then induced by the same numerical solver operating under this finer\-grained schedule𝒮′\.\\mathcal\{S\}^\{\\prime\}\.
This dynamic framework provides a more robust learning signal than a static teacher policy \(πfix\\pi^\{\\text\{fix\}\}\)\. By design, the dynamic teacher represents a higher\-fidelity trajectory derived from the student’s current parameters\. Consequently, the D2PO loss penalizes the discrepancy between the student’s coarse numerical path and this refined counterpart\. This formulation encourages the student sampler to yield a trajectory that remains consistent under step\-size refinement, which is achieved when the discrete path closely approximates the true continuous\-time trajectory\. Ultimately, rather than tracking an arbitrary external target, the student effectively learns to minimize its own discretization error\.
### 4\.5Theoretical analysis
We provide a theoretical justification for the efficacy of the dynamic teacher mechanism in D2PO, thereby reducing discretization error\.
#### 4\.5\.1Setup
Letπ\\pidenote the true continuous\-time policy representing the target distribution\. We defineπϕ\\pi\_\{\\phi\}as the student policy induced by the parametersϕ\\phiunder a discrete numerical schedule withNNtimesteps\. The dynamic teacher corresponds to a refined policyπϕ′\\pi\_\{\\phi^\{\\prime\}\}evaluated on a finer discretization schedule \(e\.g\.,2N2Ntimesteps\)\. To quantify discrepancies between policies, we employ a metricρ\(⋅,⋅\)\\rho\(\\cdot,\\cdot\)that satisfies the triangle inequality\. The true error of the student policy relative to the continuous\-time target is defined as
ϵϕtrue=ρ\(πϕ,π\)\.\\displaystyle\\epsilon^\{\\text\{true\}\}\_\{\\phi\}=\\rho\(\\pi\_\{\\phi\},\\pi\)\.\(20\)
#### 4\.5\.2Dynamic teacher
The dynamic DPO objective minimizes the discrepancy between the student and its refined counterpart:
ℒdyn=ρ\(πϕ,πϕ′\)\.\\displaystyle\\mathcal\{L\}\_\{\\text\{dyn\}\}=\\rho\(\\pi\_\{\\phi\},\\pi\_\{\\phi^\{\\prime\}\}\)\.\(21\)Assuming the underlying numerical solver exhibits a convergence order ofk\>0k\>0\[suli2003introduction\], the dynamic teacherπϕ′\\pi\_\{\\phi^\{\\prime\}\}constructed via a2×2\\timesrefinement yields a reduced true error relative to the continuous\-time target, which is given by
ϵϕ′true=ρ\(πϕ′,π\)≈12kϵϕtrue\.\\displaystyle\\epsilon^\{\\text\{true\}\}\_\{\\phi^\{\\prime\}\}=\\rho\(\\pi\_\{\\phi^\{\\prime\}\},\\pi\)\\approx\\frac\{1\}\{2^\{k\}\}\\,\\epsilon^\{\\text\{true\}\}\_\{\\phi\}\.\(22\)By applying the triangle inequality,ρ\(πϕ,π\)≤ρ\(πϕ,πϕ′\)\+ρ\(πϕ′,π\)\\rho\(\\pi\_\{\\phi\},\\pi\)\\leq\\rho\(\\pi\_\{\\phi\},\\pi\_\{\\phi^\{\\prime\}\}\)\+\\rho\(\\pi\_\{\\phi^\{\\prime\}\},\\pi\), we establish a lower bound on the dynamic loss:
ℒdyn≥\|ϵϕtrue−ϵϕ′true\|≈\(1−12k\)ϵϕtrue\.\\displaystyle\\mathcal\{L\}\_\{\\text\{dyn\}\}\\geq\\left\|\\epsilon^\{\\text\{true\}\}\_\{\\phi\}\-\\epsilon^\{\\text\{true\}\}\_\{\\phi^\{\\prime\}\}\\right\|\\approx\\left\(1\-\\frac\{1\}\{2^\{k\}\}\\right\)\\epsilon^\{\\text\{true\}\}\_\{\\phi\}\.\(23\)Consequently,ℒdyn\\mathcal\{L\}\_\{\\text\{dyn\}\}serves as a non\-trivial surrogate that upper\-bounds \(and scales proportionally with\) the student’s true error\. Minimizingℒdyn\\mathcal\{L\}\_\{\\text\{dyn\}\}enforces trajectory consistency across different discretization granularities, thereby driving a systematic reduction in discretization error\. Sinceϵϕ′true=O\(2−k\)\\epsilon^\{\\text\{true\}\}\_\{\\phi^\{\\prime\}\}=O\(2^\{\-k\}\), the residual error of the teacher vanishes progressively under refinement, aligningℒdyn\\mathcal\{L\}\_\{\\text\{dyn\}\}more closely with the true optimization objective as training proceeds\.
#### 4\.5\.3Fixed teacher
The conventional fixed\-teacher objective minimizes
ℒfix=ρ\(πϕ,πfix\),\\displaystyle\\mathcal\{L\}\_\{\\text\{fix\}\}=\\rho\(\\pi\_\{\\phi\},\\pi^\{\\text\{fix\}\}\),\(24\)whereπfix\\pi^\{\\text\{fix\}\}denotes a static teacher policy that is independent of the student parametersϕ\\phi\. Let the intrinsic error of this fixed teacher be
ϵfix=ρ\(πfix,π\),\\displaystyle\\epsilon^\{\\text\{fix\}\}=\\rho\(\\pi^\{\\text\{fix\}\},\\pi\),\(25\)which remains constant with respect toϕ\\phi\. Applying the triangle inequality yields the following bounds on the empirical loss:
\|ϵϕtrue−ϵfix\|≤ℒfix≤ϵϕtrue\+ϵfix\.\\displaystyle\\left\|\\epsilon^\{\\text\{true\}\}\_\{\\phi\}\-\\epsilon^\{\\text\{fix\}\}\\right\|\\leq\\mathcal\{L\}\_\{\\text\{fix\}\}\\leq\\epsilon^\{\\text\{true\}\}\_\{\\phi\}\+\\epsilon^\{\\text\{fix\}\}\.\(26\)Even ifℒfix\\mathcal\{L\}\_\{\\text\{fix\}\}is optimized to its global minimum \(ℒfix=0\\mathcal\{L\}\_\{\\text\{fix\}\}=0\), the resulting student policy is bounded byϵϕtrue=ϵfix\\epsilon^\{\\text\{true\}\}\_\{\\phi\}=\\epsilon^\{\\text\{fix\}\}\. Therefore, minimizingℒfix\\mathcal\{L\}\_\{\\text\{fix\}\}cannot reduce the true error below this asymptotic error floor\. D2PO bypasses this performance bottleneck because the dynamic teacher evolves alongside the student, preventing the optimization from stagnation at a fixed residual error floor\.
## 5Experiment
### 5\.1Experimental setup
We comprehensively evaluate D2PO across multiple generation tasks, architectures, and datasets\. For text\-to\-image synthesis, we use the pre\-trained Stable Diffusion v1\.5 model\[rombach2022high\]using prompts from the COCO\[lin2014microsoft\]dataset\. To demonstrate the generalizability of our approach, we apply this exact same sampler optimization to ImageNet \(256×\\times256\) generation in the latent space, as well as to the sampling process of the flow\-matching\-based InstaFlow model\. Crucially, we keep all pre\-trained model parameters strictly frozen; our method exclusively optimizes the sampling policy, consisting of the continuously parameterized timestep schedule𝒮\\mathcal\{S\}and the per\-step CFG weights𝝎\\bm\{\\omega\}for several advanced ODE solvers, including iPNDM\[zhang2023fast\], UniPC\[zhao2023unipc\]and DPM\-Solver\+\+\[lu2023dpm\]\. We compare D2PO against state\-of\-the\-art discretization methods, including DMN\[xue2024accelerating\], GITS\[chen2024trajectory\], and LD3\[tong2024learning\]\. For evaluation, we measure distributional fidelity using FID\[heusel2017gans\], and further assess text\-to\-image perceptual quality using HPSv2\[wu2023human\]and Aesthetic scores\[Schuhmann2022LaionAesthetics\]111Baselines for text\-to\-image tasks are re\-evaluated on newly generated samples since HPSv2 and Aesthetic scores are omitted in the original papers\. For ImageNet, we copy the FID values reported in the LD3\[tong2024learning\]paper\.\.
Table 1:Quantitative comparison of sampler optimization methods on text\-to\-image synthesis using Stable Diffusion v1\.5 across different ODE solvers\.iPNDMUniPCDPM\-Solver\+\+StepsMethodHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrowHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrowHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrow4DMN\[xue2024accelerating\]0\.20305\.093621\.390\.20305\.108422\.030\.19795\.097824\.33GITS\[chen2024trajectory\]0\.21285\.141318\.120\.21095\.159220\.140\.20965\.151919\.86LD3\[tong2024learning\]0\.21915\.175617\.600\.21805\.175518\.340\.21915\.173617\.46D2PO0\.22375\.202415\.690\.21855\.176116\.970\.22165\.185416\.845DMN\[xue2024accelerating\]0\.21465\.151417\.350\.21755\.173216\.990\.21085\.154919\.17GITS\[chen2024trajectory\]0\.22745\.192614\.070\.22685\.192815\.460\.22605\.205815\.29LD3\[tong2024learning\]0\.23465\.246313\.590\.23555\.259113\.890\.23445\.237813\.27D2PO0\.23855\.270113\.380\.23695\.277614\.470\.23745\.274014\.166DMN\[xue2024accelerating\]0\.22835\.210713\.660\.23225\.237113\.740\.22675\.218814\.51GITS\[chen2024trajectory\]0\.23825\.240712\.330\.24045\.246112\.380\.23975\.261012\.41LD3\[tong2024learning\]0\.23755\.256513\.100\.24025\.268912\.900\.23865\.244512\.62D2PO0\.24825\.330913\.540\.24415\.319214\.380\.24585\.323714\.007DMN\[xue2024accelerating\]0\.24285\.277811\.890\.24745\.301112\.120\.23995\.273413\.02GITS\[chen2024trajectory\]0\.23945\.236812\.160\.23795\.206512\.910\.23555\.217413\.16LD3\[tong2024learning\]0\.24345\.276312\.410\.24215\.266012\.770\.24555\.279512\.09D2PO0\.25135\.325712\.710\.24995\.347213\.610\.25025\.345813\.52
Table 2:Quantitative comparison on ImageNet\-256 \(latent space\) using the 3rd\-order \(3M\) iPNDM solver\. Baseline results are reported from the LD3\[tong2024learning\]paper\.StepsMethod4567Uniform13\.867\.806\.035\.35GITS\[chen2024trajectory\]56\.0043\.5619\.3310\.33DMN\[xue2024accelerating\]10\.157\.337\.257\.40LD3\[tong2024learning\]9\.196\.035\.094\.68D2PO7\.285\.484\.804\.70
Table 3:Quantitative comparison on the InstaFlow\[liu2024instaflow\]model using the prompts from the COCO dataset\. Higher HPS and Aesthetic scores \(↑\\uparrow\) are better, while lower FID scores \(↓\\downarrow\) are better\.StepsMethodHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrow2Uniform0\.18655\.006044\.27LD3\[tong2024learning\]0\.17084\.627063\.55D2PO0\.18725\.092440\.684Uniform0\.21895\.122216\.85LD3\[tong2024learning\]0\.20865\.071223\.21D2PO0\.21975\.141515\.506Uniform0\.23175\.184713\.68LD3\[tong2024learning\]0\.22995\.174015\.60D2PO0\.23425\.185512\.70
Table 4:Ablation study of D2PO on text\-to\-image synthesis using Stable Diffusion v1\.5 and the iPNDM solver using prompts from the COCO dataset\. We report performance at \# steps=4 and 5 after removing or altering key components: dynamic preference, score\-based energy, and the reference timestep update strategy\.StepsMethodAesthetic↑\\uparrowFID↓\\downarrow4D2PO \(Full\)5\.202415\.69w/o dynamic preference5\.181016\.91w/o score\-based energy5\.179617\.88w/ EMA reference timestep5\.193715\.755D2PO \(Full\)5\.270113\.38w/o dynamic preference5\.261513\.70w/o score\-based energy5\.263014\.59w/ EMA reference timestep5\.263913\.42
### 5\.2Main results
Our quantitative results on text\-to\-image synthesis using Stable Diffusion v1\.5 are presented in[Tab\.˜1](https://arxiv.org/html/2607.06609#S5.T1), where we apply D2PO to three representative ODE solvers—iPNDM, UniPC, and DPM\-Solver\+\+\. For evaluation, we generate 30k samples using text prompts from COCO dataset\[lin2014microsoft\], following the standard\[tong2024learning,frankels4s\]\. Overall, the results demonstrate that D2PO achieves superior performance compared to existing baselines, asserting its effectiveness and robustness across multiple advanced solvers\.
At a low number of steps \(4 and 5\), D2PO functions as a superior error corrector\. In this regime, D2PO achieves state\-of\-the\-art perceptual quality—showing the highest Aesthetic scores across all solvers and the highest HPS scores for iPNDM\. Furthermore, when using 4 steps, D2PO surpasses all baselines in FID across all three solvers\. This directly validates our hypothesis\. Regression\-based methods sacrifice high\-frequency details, in part because their loss metrics,*e\.g*\., LPIPS, fail to capture these errors\. Our novel score\-based distance metric captures both structural and textural discrepancies by measuring score differences at various noise levels\. This high sensitivity to fine\-grained texture loss allows D2PO to correct the foundational flaws of the baseline, simultaneously improving both quality and fidelity\.
At a higher number of steps \(6 and 7\), as the baseline’s severe discretization error is reduced, the expected quality\-diversity trade\-off\[brock2018large,kingma2018glow,ho2022classifier\]emerges, and D2PO’s behavior shifts to its primary goal of preference alignment\. While D2PO maintains its significant lead in perceptual scores \(HPS/Aesthetic\), its FID score becomes comparable or slightly higher than the baselines\. This shift is the expected signature of successful alignment\. D2PO’s objective function, which aligns with its dynamic, higher\-fidelity2N2Nteacher, optimizes the sampler towards a distribution that maximizes perceptual quality\. This distribution is distinct from the average of the real data distribution, which FID measures\. This result demonstrates that D2PO is not failing; rather, it is successfully harnessing the full potential of the model by optimizing for its intended perceptual objective, which is to reduce its own discretization error\.
Steps = 4: “Two dogs curled up asleep on a couch\."
Steps = 5: “A man riding a snow board on top of a snow covered slope\."
Steps = 6: “A male tennis player in white shorts is playing tennis\."
Steps = 7: “A little boy about to hit a baseball during a game\."
DMN\[xue2024accelerating\]
GITS\[chen2024trajectory\]
LD3\[tong2024learning\]
D2PO
Figure 2:Qualitative comparison of various discretization methods on Stable Diffusion v1\.5 using iPNDM across different number of time steps \(4 to 7\)\. D2PO consistently produces sharper details and fewer artifacts compared to baselines\.
### 5\.3Generalization across domains and architectures
To evaluate the robustness of our learned policy, we tested D2PO beyond standard text\-to\-image tasks\. As shown in[Tab\.˜3](https://arxiv.org/html/2607.06609#S5.T3), D2PO matches or outperforms baseline methods on ImageNet\-256 \(latent space\) using the 3rd\-order iPNDM solver\. Furthermore,[Tab\.˜3](https://arxiv.org/html/2607.06609#S5.T3)illustrates D2PO’s successful application to InstaFlow, a flow\-matching model on the COCO dataset\.
### 5\.4Qualitative results
[Fig\.˜2](https://arxiv.org/html/2607.06609#S5.F2)visualizes the qualitative results on Stable Diffusion v1\.5 \(iPNDM\) for all steps reported in[Tab\.˜1](https://arxiv.org/html/2607.06609#S5.T1)\. Across all cases, D2PO consistently produces images with higher perceptual quality, sharper details, and fewer artifacts compared to baselines\. This visual evidence directly supports our quantitative findings\. For instance, baseline methods like LD3 and DMN frequently exhibit significant blurriness, water\-color artifacts, or loss of fine\-grained detail\. In more complex scenes \(Steps=7\), D2PO generates a coherent and detailed image, while other methods suffer from severe structural distortion or artifacts\. More qualitative results on different settings are provided in the supplementary material\.
### 5\.5Ablation study
We ablate D2PO’s key components in[Tab\.˜4](https://arxiv.org/html/2607.06609#S5.T4)\. First, the w/o dynamic preference variant replaces our dynamic2N2N\-step teacher with a fixedN\+1N\+1\-step teacher \(as in LD3\[tong2024learning\]\)\. This degrades performance, validating that our dynamic mechanism provides a stronger, more consistent learning signal than regression to a static teacher\. Second, the w/o score\-based energy variant, which instead uses LPIPS, causes the most significant drop in fidelity, confirming our metric’s necessity for capturing fine\-grained discretization errors\. Finally, the w/ EMA reference timestep variant replaces our epoch\-wise copy strategy for the reference time step schedule with an EMA update\. Its sub\-optimal performance highlights that while EMA suits continuous parameters \(like CFG weights\), direct epoch\-wise copying provides a more stable anchor for optimizing discrete time steps\.
## 6Conclusion
We identified a critical performance bottleneck in dominant student\-teacher regression frameworks for optimizing diffusion samplers\. We demonstrated that as the teacher\-student NFE gap increases, standard regression losses force the low\-NFE student to sacrifice high\-frequency texture fidelity, leading to degraded perceptual quality\. To address this, we proposed D2PO, a novel framework that reframes sampler optimization as a preference\-based alignment task\. We introduced a novel score\-based energy function that leverages the score model itself to capture the fine\-grained textural and structural errors that standard metrics miss, and a dynamic preference mechanism that creates a self\-improving loop, where the student policy is aligned with a dynamically refined, higher\-fidelity version of itself\. This dynamic teacher provides a stronger, theoretically\-grounded learning signal that forces the student to minimize its own discretization error, rather than converging to a suboptimal fixed teacher\. Our extensive experiments across multiple solvers demonstrated that D2PO successfully aligns diffusion samplers with true perceptual quality, effectively solving the existing bottleneck of static teacher regression\.
#### Acknowledgements\.
This work was partly supported by the Samsung Electronics Co\., Ltd\. \(IO250418\-12669\-01\)\. It was also partly supported by the NRF grant \[RS\-2022\-NR070855\] and the IITP grants \[RS\-2025\-25442338; RS\-2026\-25526850; No\.RS\-2021\-II211343; No\.RS\-2020\-II201336\] funded by the Korea government \(MSIT\)\.
## References
D2PO: Optimizing Diffusion Samplers via Dynamic Preference Supplementary Document
## AExperimental Setup
### A\.1Implementation details
We implement our method using PyTorch\[paszke2019pytorch\]and adopt the pretrained, frozen Stable Diffusion v1\.5\[rombach2022high\]as the base denoiserϵθ\\epsilon\_\{\\theta\}, following prior work\[frankels4s,tong2024learning\]\. All experiments are performed on a single NVIDIA A6000 GPU\.
Our learnable policy isϕ=\{𝒮,𝝎\}\\phi=\\\{\\mathcal\{S\},\\bm\{\\omega\}\\\}\. Following LD3\[tong2024learning\], the timestep schedule𝒮\\mathcal\{S\}is parameterized by two coupled sequences:𝒮1\\mathcal\{S\}\_\{1\}defines the ODE solver discretization grid, while𝒮2\\mathcal\{S\}\_\{2\}provides the time\-conditioning inputs to the diffusion model\. We optimize each parameter group with a separate optimizer:𝒮1\\mathcal\{S\}\_\{1\}uses RMSprop \(momentum0\.90\.9\), while𝒮2\\mathcal\{S\}\_\{2\}and the CFG weights𝝎\\bm\{\\omega\}use SGD\. For the D2PO objective, we set the temperatureβ∈\{10,50,100\}\\beta\\in\\\{10,50,100\\\}\. To keep the effective step size stable, the base learning rates are scaled as1×10−3β\\frac\{1\\times 10^\{\-3\}\}\{\\beta\}for𝒮1\\mathcal\{S\}\_\{1\}and2×10−4βN\\frac\{2\\times 10^\{\-4\}\}\{\\beta N\}for𝒮2\\mathcal\{S\}\_\{2\}and𝝎\\bm\{\\omega\}, whereNNis the number of sampling steps\. We employ gradient accumulation over four iterations, yielding an effective batch size of 4\.
To construct the winning policyϕ′\\phi^\{\\prime\}, we linearly interpolate the current student timestep schedule𝒮\\mathcal\{S\}to generate a denser schedule with2N2Nfunction evaluations\. The linear interpolation is performed in continuous timet∈\[0,T\]t\\in\[0,T\]\.
### A\.2Evaluation protocol
Following standard practice in text\-to\-image evaluation\[frankels4s,tong2024learning\], we conduct experiments on the COCO 2014 validation set\[lin2014microsoft\]under the zero\-shot generation setting\. For quantitative evaluation, we use a random subset of 30,000 captions sampled from the validation set\. Training and evaluation prompts remain fully disjoint: D2PO is trained using 400 prompts from the training split, whereas evaluation is performed on unseen captions from the validation split\. On the generated samples, we compute standard metrics assessing distributional fidelity and perceptual quality: HPSv2\[wu2023human\], the Aesthetic score\[Schuhmann2022LaionAesthetics\], and FID\[heusel2017gans\]\.
### A\.3Gradient rematerialization
Optimizing sampling parameters requires backpropagating gradients through the entire reverse\-diffusion trajectory, which involves repeated evaluations of the heavy diffusion backboneϵθ\\epsilon\_\{\\theta\}\. A naive implementation would store all intermediate activations, resulting in memory consumption that scales linearly with the number of function evaluations, which is infeasible for large diffusion models\.
We therefore apply gradient rematerialization\[chen2016training\], following the protocol established in\[tong2024learning\]\. Instead of storing intermediate activations during the forward pass, rematerialization recomputes necessary activations on demand during backpropagation\. This trades additional compute for a substantial reduction in memory, enabling efficient training on a single GPU\.
## BAdditional Quantitative Results
Table A:Comparison of D2PO with standard LD3 and an enhanced variant \(LD3†\) on the COCO dataset\[lin2014microsoft\]\. LD3†is trained with a significantly larger computational budget \(16×16\\timestraining time\), more training data \(16×16\\times\), and a stronger teacher \(Δ=4\\Delta=4\) to match the resource allocation of D2PO\. Higher HPS and Aesthetic scores \(↑\\uparrow\) are better, while lower FID scores \(↓\\downarrow\) are better\.StepsMethodHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrow4LD3\[tong2024learning\]0\.21915\.175617\.60LD3†0\.21465\.167821\.34D2PO0\.22375\.202415\.695LD3\[tong2024learning\]0\.23465\.246313\.59LD3†0\.23615\.264414\.89D2PO0\.23855\.270113\.386LD3\[tong2024learning\]0\.23755\.256513\.10LD3†0\.24635\.319613\.46D2PO0\.24825\.330913\.547LD3\[tong2024learning\]0\.24345\.276312\.41LD3†0\.25025\.326413\.15D2PO0\.25135\.325712\.71
Table B:Quantitative comparison of text\-image alignment using CLIP score on the COCO dataset\[lin2014microsoft\]with the iPNDM solver\. Higher CLIP scores \(↑\\uparrow\) indicate better alignment\.MethodSteps = 4Steps = 5Steps = 6Steps = 7LD3\[tong2024learning\]25\.8726\.2126\.2226\.37D2PO26\.0126\.3026\.5226\.57
### B\.1Decoupling methodological gains from computational budget
We analyze whether D2PO’s gains stem from its design rather than its larger training budget\. LD3\[tong2024learning\]training is typically lightweight, utilizing a small subset of 25 prompts for 5 epochs\. D2PO trains on a significantly larger scale using 400 prompts\. To verify that D2PO’s superiority stems from its methodological design rather than resource scaling, we compare against an enhanced baseline, LD3†, designed to match D2PO’s resource budget\. LD3†’s budget has16×16\\timestraining duration,16×16\\timesmore data, and a stronger teacher with an increased step gap \(Δ=T−S=4\\Delta=T\-S=4\)\. As reported in[Tab\.˜A](https://arxiv.org/html/2607.06609#S2.T1), granting LD3 the same budget as D2PO fails to close the gap on the perceptual metrics D2PO is designed to optimize: D2PO retains the highest HPS at every step count and matches or exceeds LD3†on Aesthetic, confirming that its perceptual advantage is methodological rather than a by\-product of resource scaling\.
### B\.2Evaluation of text\-image semantic alignment
Beyond visual aesthetics and distributional fidelity, faithfully reflecting the conditioning prompt is a central requirement for text\-to\-image synthesis\. We therefore measure CLIP score on COCO to assess semantic alignment\.
As shown in[Tab\.˜B](https://arxiv.org/html/2607.06609#S2.T2), D2PO achieves higher CLIP scores than LD3 at every step count\. We attribute this to a difference in objective\. Distillation methods such as LD3 enforce pointwiseℓ2\\ell^\{2\}or LPIPS matching to a fixed teacher; in the extreme few\-step regime, where the solver already incurs large truncation error, this rigid structural constraint leaves little freedom to preserve prompt\-relevant content, diluting semantic alignment\. D2PO instead optimizes an ordinal preference toward its own refined, higher\-fidelity output, which does not tie the student to a single pointwise target and thus retains more flexibility to keep the generation consistent with the prompt\. As a result, D2PO improves faithfulness to the input text alongside its perceptual gains\.
Table C:Generalization to the modern Stable Diffusion 3\.5\-Medium\[esser2024scalingrectifiedflowtransformers\]backbone \(MM\-DiT, flow matching, Euler solver\) on the COCO dataset\. D2PO improves all metrics, demonstrating that the gains are not specific to SD v1\.5\.StepsMethodHPS↑\\uparrowAesthetic↑\\uparrowFID↓\\downarrow4Uniform0\.12894\.618592\.09LD3\[tong2024learning\]0\.13364\.647686\.34D2PO0\.14684\.781273\.896Uniform0\.16754\.892250\.94LD3\[tong2024learning\]0\.16984\.918448\.95D2PO0\.19025\.019033\.64
Table D:Blind user study: vote shares on 30 COCO prompts \(iPNDM, 7 steps, same initial noise, hidden method names\)\.MethodAlign\.Qual\.OverallAvg\.GITS\[chen2024trajectory\]15\.8%9\.4%11\.2%12\.1%LD3\[tong2024learning\]29\.4%26\.1%27\.3%27\.6%D2PO54\.8%64\.5%61\.5%60\.3%
Table E:FID comparison on the pixel\-space AFHQv2 dataset with the iPNDM solver\.Method46810Uniform23\.209\.554\.493\.19GITS\[chen2024trajectory\]12\.896\.104\.033\.26LD3\[tong2024learning\]9\.963\.632\.632\.27D2PO9\.943\.602\.612\.20
Table F:FID comparison on ImageNet\-256 \(latent space\) against search\-based optimization \(CMA\-ES and the GITS search baseline\) with the iPNDM solver\.Method4567GITS\[chen2024trajectory\]56\.0043\.5619\.3310\.33CMA\-ES\[hansen2016cma\]20\.0117\.837\.206\.99D2PO7\.285\.484\.804\.70
### B\.3Generalization to modern backbones
To confirm that the gains are not specific to the SD v1\.5 backbone, we apply D2PO to the modern Stable Diffusion 3\.5\-Medium\[esser2024scalingrectifiedflowtransformers\], a multimodal diffusion transformer \(MM\-DiT\) trained with flow matching and sampled with the Euler solver\. As reported in[Tab\.˜C](https://arxiv.org/html/2607.06609#S2.T3), D2PO outperforms both the Time\-Uniform baseline and LD3 on all metrics, including a large FID improvement, demonstrating that the benefits of our preference\-based optimization transfer to state\-of\-the\-art backbones\.
### B\.4Human evaluation
To verify that our gains on proxy metrics reflect genuine perceptual improvements, we conduct a blind, randomized user study on3030COCO prompts \(iPNDM,77steps\), in which all methods share the same initial noise and the method names are hidden\. Participants select the best result among GITS, LD3, and D2PO under three criteria—prompt alignment, visual quality, and overall preference \(9090votes per participant\)\. As shown in[Tab\.˜D](https://arxiv.org/html/2607.06609#S2.T4), D2PO receives60\.3%60\.3\\%of the votes on average, far ahead of LD3 \(27\.6%27\.6\\%\) and GITS \(12\.1%12\.1\\%\), confirming that the improvements in HPSv2 and Aesthetic scores correspond to human\-perceived quality\.
### B\.5Generalization to pixel\-space generation
We assess D2PO on the pixel\-space AFHQv2 dataset with the iPNDM solver \([Tab\.˜E](https://arxiv.org/html/2607.06609#S2.T5)\), where it matches or slightly outperforms LD3 across44–1010steps\. The marginal gap is partly because AFHQv2 uses unconditional generation, so the per\-step CFG weights𝝎\\bm\{\\omega\}carry no optimization signal and only the timestep schedule𝒮\\mathcal\{S\}is effectively optimized\.
### B\.6Comparison with search\-based optimization
On ImageNet\-256 \(latent space\), we compare against search\-based optimization \([Tab\.˜F](https://arxiv.org/html/2607.06609#S2.T6)\)\. Although the sampler parameters are low\-dimensional, black\-box search such as CMA\-ES\[hansen2016cma\]requires many sample evaluations per candidate\. Under the same parameterization and score\-based distance, D2PO substantially outperforms both CMA\-ES and the GITS\[chen2024trajectory\]search baseline, as it backpropagates a preference signal through the solver rather than relying on scalar function evaluations alone\.
DMN
GITS
LD3
D2PO
“An old photo of an umbrella and chairs at the beach\."
“black teddy bear placed in front of mirror\."\(a\)Steps = 4
“A cat laying on a cushion on top of a table\."
“A television that is sitting on a stand\."\(b\)Steps = 5
Figure A:Side\-by\-side comparison of selected images generated with Stable Diffusion and iPNDM solver in the low NFE regime \(Steps∈\{4,5\}\\in\\\{4,5\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.DMN
GITS
LD3
D2PO
“A large green truck on a city street\."
“A sports motorcycle is parked on a gravel road by a river\."\(a\)Steps = 6
“A large clock tower with a clock on it’s face\."
“A gold and white clock on street next to a building\."\(b\)Steps = 7
Figure B:Side\-by\-side comparison of selected images generated with Stable Diffusion and iPNDM solver in the high NFE regime \(Steps∈\{6,7\}\\in\\\{6,7\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.DMN
GITS
LD3
D2PO
“A boy that is standing next to an animal\."
“The blue train is passing through a wooded area\."\(a\)Steps = 4
“A woman sits on a bed with pillows\."
“A motorcycle parked in a field with a sky background\."\(b\)Steps = 5
Figure C:Side\-by\-side comparison of selected images generated with Stable Diffusion and UniPC solver in the low NFE regime \(Steps∈\{4,5\}\\in\\\{4,5\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.DMN
GITS
LD3
D2PO
“Giraffes in their wood and grass zoo enclosure\."
“A young man and woman sitting at a table\."\(a\)Steps = 6
“Hard plastic chairs in a dimly lit room\."
“Two dogs play in a dammed up river\."\(b\)Steps = 7
Figure D:Side\-by\-side comparison of selected images generated with Stable Diffusion and UniPC solver in the high NFE regime \(Steps∈\{6,7\}\\in\\\{6,7\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.DMN
GITS
LD3
D2PO
“A multicolored train passing another set of tracks\."
“A kitchen that is very clean in a house\."\(a\)Steps = 4
“A person with a skateboard on a street\."
“A small wooden table covered with delicious vegetables\."\(b\)Steps = 5
Figure E:Side\-by\-side comparison of selected images generated with Stable Diffusion and DPM\-Solver\+\+ in the low NFE regime \(Steps∈\{4,5\}\\in\\\{4,5\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.DMN
GITS
LD3
D2PO
“A stuffed animal has been placed inside of blankets\."
“A woman wearing a jacket and jeans skiing down a hill\."\(a\)Steps = 6
“A lady staring lovingly into her pizza\."
“A man riding a motorcycle with a helmet on\."\(b\)Steps = 7
Figure F:Side\-by\-side comparison of selected images generated with Stable Diffusion and DPM\-Solver\+\+ in the high NFE regime \(Steps∈\{6,7\}\\in\\\{6,7\\\}\)\. Methods \(from left to right\): DMN, GITS, LD3, and D2PO\.
## CAdditional Qualitative Results
In this section, we provide an extensive visual comparison to corroborate the quantitative findings presented in the main paper\. We evaluate Stable Diffusion v1\.5 coupled with three advanced ODE solvers \(iPNDM, UniPC, DPM\-Solver\+\+\) with number of steps ranging from 4 to 7\. In the low NFE regime \(4\-5 steps\), D2PO effectively mitigates the structural collapse and artifacts frequently observed in baselines, as shown in[Figs\.˜A](https://arxiv.org/html/2607.06609#S2.F1),[C](https://arxiv.org/html/2607.06609#S2.F3)and[E](https://arxiv.org/html/2607.06609#S2.F5)\. As the computational budget increases to 6\-7 steps, the advantage shifts towards fine\-grained details \([Figs\.˜B](https://arxiv.org/html/2607.06609#S2.F2),[D](https://arxiv.org/html/2607.06609#S2.F4)and[F](https://arxiv.org/html/2607.06609#S2.F6)\)\. These results confirm that our preference\-based optimization is robust across different solver architectures\.
## DAlgorithmic Details
In this section, Algorithm[1](https://arxiv.org/html/2607.06609#alg1)outlines the complete training procedure of D2PO \(Dynamic Direct Preference Optimization\), and Algorithm[2](https://arxiv.org/html/2607.06609#alg2)details the computation of our novel score\-based distance metric\.
### D\.1Training procedure \(Algorithm[1](https://arxiv.org/html/2607.06609#alg1)\)
The core of D2PO lies in its dynamic preference generation mechanism\. Unlike standard distillation methods that rely on a fixed teacher, D2PO iteratively refines the student policy to create a dynamic winner sample𝒙w\\bm\{x\}\_\{w\}\. As shown in Algorithm[1](https://arxiv.org/html/2607.06609#alg1), the winning policyϕ′\\phi^\{\\prime\}is derived on\-the\-fly by refining the current student parameters \(e\.g\., via timestep interpolation\)\. The model is then optimized with the DPO loss, which favors the winning sample𝒙w\\bm\{x\}\_\{w\}over the degraded losing sample𝒙l\\bm\{x\}\_\{l\}, both measured relative to the reference policyϕref\\phi\_\{\\text\{ref\}\}\.
### D\.2Score\-based distance \(Algorithm[2](https://arxiv.org/html/2607.06609#alg2)\)
A critical component of our objective is the energy function used to define the preference\. Instead of relying on pixel\-space metrics \(e\.g\., MSE\) or external networks \(e\.g\., LPIPS\), we leverage the pre\-trained diffusion model itself as a critic\. Algorithm[2](https://arxiv.org/html/2607.06609#alg2)describes this procedure\. Given two samples, we perturb both with the same noiseϵ\\bm\{\\epsilon\}at a randomly sampled timesteptt, and pass them through the pretrained networkϵθ\\epsilon\_\{\\theta\}\. The distancedθd\_\{\\theta\}is the squaredℓ2\\ell\_\{2\}difference between the two noise predictions, which \(up to the fixed factorσt2\\sigma\_\{t\}^\{2\}absorbed by our weighting\) equals the score discrepancy at leveltt\. This metric effectively captures the discrepancy between the sample’s trajectory and the vector field of the pre\-trained diffusion prior, providing a fine\-grained signal for structural and textural alignment\.
Algorithm 1D2PO \(Dynamic Direct Preference Optimization\)1:Input:Learnable params
ϕ=\{𝒮,𝝎\}\\phi=\\\{\\mathcal\{S\},\\bm\{\\omega\}\\\},
β\\beta, learning rate
η\\eta, EMA momentum
λ\\lambda, Sampler
Φϕ\(⋅\)\\Phi\_\{\\phi\}\(\\cdot\), Degradation operator
𝒢\(⋅\)\\mathcal\{G\}\(\\cdot\)
2:Initialize learnable sampler parameters
ϕ\\phi
3:Initialize reference policy
ϕref←ϕ\\phi\_\{\\text\{ref\}\}\\leftarrow\\phi
4:whilenot convergeddo
5:Sample batch of contexts
\(c,𝒙T\)\(c,\\bm\{x\}\_\{T\}\)
6:\# 1\. Generate student and reference outputs
7:
𝒙ϕ←Φϕ\(c,𝒙T\)\\bm\{x\}\_\{\\phi\}\\leftarrow\\Phi\_\{\\phi\}\(c,\\bm\{x\}\_\{T\}\)
8:
𝒙ϕref←Φϕref\(c,𝒙T\)\\bm\{x\}\_\{\\phi\_\{\\text\{ref\}\}\}\\leftarrow\\Phi\_\{\\phi\_\{\\text\{ref\}\}\}\(c,\\bm\{x\}\_\{T\}\)
9:\# 2\. Generate dynamic preference \(Sec 4\.4\)
10:
ϕ′←Refine\(ϕ\)\\phi^\{\\prime\}\\leftarrow\\text\{Refine\}\(\\phi\)
11:
𝒙w←Φϕ′\(c,𝒙T\)\\bm\{x\}\_\{w\}\\leftarrow\\Phi\_\{\\phi^\{\\prime\}\}\(c,\\bm\{x\}\_\{T\}\)
12:
𝒙l←𝒢\(sg\[𝒙ϕ\]\)\\bm\{x\}\_\{l\}\\leftarrow\\mathcal\{G\}\(\\mathrm\{sg\}\[\\bm\{x\}\_\{\\phi\}\]\)
13:\# 3\. Compute final loss
14:
Lw←β\(dθ\(𝒙w,𝒙ϕref\)−dθ\(𝒙w,𝒙ϕ\)\)L\_\{w\}\\leftarrow\\beta\\left\(d\_\{\\theta\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\_\{\\text\{ref\}\}\}\)\-d\_\{\\theta\}\(\\bm\{x\}\_\{w\},\\bm\{x\}\_\{\\phi\}\)\\right\)
15:
Ll←β\(dθ\(𝒙l,𝒙ϕref\)−dθ\(𝒙l,𝒙ϕ\)\)L\_\{l\}\\leftarrow\\beta\\left\(d\_\{\\theta\}\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\_\{\\text\{ref\}\}\}\)\-d\_\{\\theta\}\(\\bm\{x\}\_\{l\},\\bm\{x\}\_\{\\phi\}\)\\right\)
16:
ℒD2PO←−logσ\(Lw−Ll\)\\mathcal\{L\}\_\{\\text\{D2PO\}\}\\leftarrow\-\\log\\sigma\(L\_\{w\}\-L\_\{l\}\)
17:\# 4\. Update Parameters
18:
ϕ←ϕ−η∇ϕℒD2PO\\phi\\leftarrow\\phi\-\\eta\\nabla\_\{\\phi\}\\mathcal\{L\}\_\{\\text\{D2PO\}\}
19:
ωref←λωref\+\(1−λ\)ω\\omega\_\{\\text\{ref\}\}\\leftarrow\\lambda\\,\\omega\_\{\\text\{ref\}\}\+\(1\-\\lambda\)\\,\\omega
20:ifend of epochthen
21:
𝒮ref←𝒮\\mathcal\{S\}\_\{\\text\{ref\}\}\\leftarrow\\mathcal\{S\}
22:endif
23:endwhile
24:Return
ϕ\\phi
Algorithm 2Score\-based Distance1:Input:Two samples
\(𝒙a,𝒙b\)\(\\bm\{x\}\_\{a\},\\bm\{x\}\_\{b\}\), noise prediction network
ϵθ\(⋅,⋅\)\\epsilon\_\{\\theta\}\(\\cdot,\\cdot\), noise schedule
\(α\(t\),σ\(t\)\)\(\\alpha\(t\),\\sigma\(t\)\), uniform range
\[tmin,tmax\]\[t\_\{\\min\},t\_\{\\max\}\], conditioning
cc
2:Sample a single time step
t∼𝒰\(tmin,tmax\)t\\sim\\mathcal\{U\}\(t\_\{\\min\},t\_\{\\max\}\)
3:Sample random noise
ϵ∼𝒩\(0,𝐈\)\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(0,\\mathbf\{I\}\)
4:\# 1\. Compute forward\-marginal parameters
5:
αt←α\(t\)\\alpha\_\{t\}\\leftarrow\\alpha\(t\)
6:
σt←σ\(t\)\\sigma\_\{t\}\\leftarrow\\sigma\(t\)
7:\# 2\. Construct noisy samples
8:
𝒙a,t=αt𝒙a\+σtϵ\\bm\{x\}\_\{a,t\}=\\alpha\_\{t\}\\bm\{x\}\_\{a\}\+\\sigma\_\{t\}\\bm\{\\epsilon\}
9:
𝒙b,t=αt𝒙b\+σtϵ\\bm\{x\}\_\{b,t\}=\\alpha\_\{t\}\\bm\{x\}\_\{b\}\+\\sigma\_\{t\}\\bm\{\\epsilon\}
10:\# 3\. Predict noise
11:
ϵ^a=ϵθ\(𝒙a,t,t;c\)\\hat\{\\bm\{\\epsilon\}\}\_\{a\}=\\epsilon\_\{\\theta\}\(\\bm\{x\}\_\{a,t\},t;c\)
12:
ϵ^b=ϵθ\(𝒙b,t,t;c\)\\hat\{\\bm\{\\epsilon\}\}\_\{b\}=\\epsilon\_\{\\theta\}\(\\bm\{x\}\_\{b,t\},t;c\)
13:\# 4\. Compute the score\-based energy
14:
dθ=‖ϵ^a−ϵ^b‖22d\_\{\\theta\}=\\\|\\hat\{\\bm\{\\epsilon\}\}\_\{a\}\-\\hat\{\\bm\{\\epsilon\}\}\_\{b\}\\\|\_\{2\}^\{2\}
15:Return
dθd\_\{\\theta\}Similar Articles
Diffusion Policy Optimization without Drifting Apart
DiPOD stabilizes diffusion policy optimization by interleaving self-distillation with policy-gradient updates to maintain a tight ELBO, preventing the double-drift phenomenon and achieving higher rewards in both language and continuous control tasks.
$\xi$-DPO: Direct Preference Optimization via Ratio Reward Margin
This paper introduces xi-DPO, a novel preference optimization method that reformulates the objective to minimize distance to optimal ratio reward margins, addressing hyperparameter tuning challenges in SimPO. Experimental results show that xi-DPO outperforms existing methods on open benchmarks.
GroupDPO: Memory efficient Group-wise Direct Preference Optimization
GroupDPO introduces a memory-efficient algorithm for group-wise direct preference optimization that leverages multiple candidate responses per prompt while reducing peak memory usage through decoupled backpropagation. The method demonstrates consistent improvements over standard DPO across offline and online alignment settings.
TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue
This paper proposes TD-DPO, a token-level difference-aware preference optimization method to mitigate sycophancy in LLMs for clinical autism intervention dialogue, achieving a better trade-off between sycophancy reduction and intervention ability retention.
Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs
This paper introduces PNAPO, an offline preference optimization framework for rectified flow models that augments preference data with noise samples and uses dynamic regularization to improve training efficiency and sample efficiency.