Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Hugging Face Daily Papers Papers
rectified-flow preference-optimization noise-aware offline-alignment diffusion-models text-to-image sample-efficiency

Summary

This paper introduces PNAPO, an offline preference optimization framework for rectified flow models that augments preference data with noise samples and uses dynamic regularization to improve training efficiency and sample efficiency.

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.
Original Article
View Cached Full Text

Cached at: 05/14/26, 08:17 AM

Paper page - Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Source: https://huggingface.co/papers/2605.09433

Abstract

Rectified flow models require prior noise information for effective preference optimization, which PNAPO addresses by augmenting preference data with noise samples and employing dynamic regularization for improved training efficiency.

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient forrectified flow(RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, priorDPO-style alignment fordiffusion modelscommonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-AwarePreference Optimization(PNAPO), an off-policy alignment framework specialized forrectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states vianoise-image interpolation, which constrains thetrajectory estimationspace and yields a tighter surrogate objective forpreference optimization. In addition, we introduce a dynamic regularization strategy that adapts theDPOregularization based on (i) thereward gapbetween winner and loser and (ii)training progress, improving stability andsample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.09433

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.09433 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.09433 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.09433 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

D2PO: Optimizing Diffusion Samplers via Dynamic Preference

arXiv cs.LG

D2PO proposes a dynamic preference optimization framework that aligns diffusion sampling policies with perceptual quality using direct preference optimization, outperforming regression-based methods under low-NFE constraints.

PAPA: Online Personalized Active Preference Alignment

arXiv cs.LG

This paper introduces Personalized Active Preference Alignment (PAPA), a method for fine-tuning diffusion models using real-time user feedback without a parameterized reward model, enhancing efficiency in personalized tasks like recommendations and image generation.

Gradient-Aligned Pair Selection for Personalized Preference Optimization

arXiv cs.AI

This paper formalizes personalized preference learning as a geometry-aligned optimization problem and proposes GAP-DPO, an iterative algorithm that performs utility-aware, gradient-aligned preference pair selection, showing improved stylistic fidelity and preference alignment over standard DPO variants.

Personalized Federated Direct Preference Optimization via LoRA Adapters

arXiv cs.LG

The paper proposes pFedDPO, a personalized federated direct preference optimization method where a shared base model is synchronized via FedAvg while private LoRA adapters capture client-specific preferences, reducing gradient diversity heterogeneity compared to vanilla FedDPO.