Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Hugging Face Daily Papers Papers
rectified-flow preference-optimization noise-aware offline-alignment diffusion-models text-to-image sample-efficiency

Summary

This paper introduces PNAPO, an offline preference optimization framework for rectified flow models that augments preference data with noise samples and uses dynamic regularization to improve training efficiency and sample efficiency.

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient for rectified flow (RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, prior DPO-style alignment for diffusion models commonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states via noise-image interpolation, which constrains the trajectory estimation space and yields a tighter surrogate objective for preference optimization. In addition, we introduce a dynamic regularization strategy that adapts the DPO regularization based on (i) the reward gap between winner and loser and (ii) training progress, improving stability and sample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.
Original Article
View Cached Full Text

Cached at: 05/14/26, 08:17 AM

Paper page - Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs

Source: https://huggingface.co/papers/2605.09433

Abstract

Rectified flow models require prior noise information for effective preference optimization, which PNAPO addresses by augmenting preference data with noise samples and employing dynamic regularization for improved training efficiency.

Existing preference datasets for text-to-image models typically store only the final winner/loser images. This representation is insufficient forrectified flow(RF) models, whose generation is naturally indexed by a specific prior noise sample and follows a nearly straight denoising trajectory. In contrast, priorDPO-style alignment fordiffusion modelscommonly estimates trajectories using an independent forward noising process, which can be mismatched to the true reverse dynamics and introduces unnecessary variance. We propose Prior Noise-AwarePreference Optimization(PNAPO), an off-policy alignment framework specialized forrectified flow. PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image, turning the standard (prompt, winner, loser) triplet into a sextuple. Leveraging the straight-line property of RF, we estimate intermediate states vianoise-image interpolation, which constrains thetrajectory estimationspace and yields a tighter surrogate objective forpreference optimization. In addition, we introduce a dynamic regularization strategy that adapts theDPOregularization based on (i) thereward gapbetween winner and loser and (ii)training progress, improving stability andsample efficiency. Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.09433

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.09433 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.09433 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.09433 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Implicit Preference Alignment for Human Image Animation

Hugging Face Daily Papers

This paper introduces Implicit Preference Alignment (IPA), a data-efficient post-training framework that improves hand motion generation in human image animation without requiring paired preference data. It utilizes implicit reward maximization and hand-aware local optimization to enhance generation quality while reducing data curation costs.

DOG-DPO:Dynamic Optimization in Geometry for Safety Alignment

arXiv cs.LG

DOG-DPO is a training-free data selection framework that treats preference pairs as structured geometric signals, decomposing multi-dataset preference geometry into anchor and residual subspaces to select diverse subsets for safety alignment. It achieves strong utility-robustness trade-offs using only 11% of preference pairs across six safety benchmarks.

Active Learners as Efficient PRP Rerankers

Hugging Face Daily Papers

This paper reframes pairwise ranking prompting as active learning from noisy comparisons, introducing a noise-robust framework with a randomized-direction oracle to improve ranking quality under call constraints and address position bias.