The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

Hugging Face Daily Papers Papers
flow-matching reinforcement-learning discriminator-guided-rl alignment generative-models computer-vision

Summary

This paper proposes Discriminator-Guided Reinforcement Learning (DRL) to correct alignment issues in score- and flow-matching models by using a pretrained representation space discriminator as an optimal reward signal, significantly improving visual fidelity and semantic quality without human preferences.

Score- and flow-matching models often rely on preference-based reinforcement learning for two purposes: aligning with subjective preferences and, surprisingly, recovering properties such as visual realism and coherent object structure that matching-based training is intended to learn from the data itself. We argue that this reflects a structural mismatch. Matching losses measure ell_2 regression error on the velocity or score field under training-time marginals, a proxy poorly aligned with the visual and semantic properties that determine sample quality at inference. Given a reward aligned with these properties, RL sidesteps the mismatch by evaluating the model on its own samples and following the reward landscape directly. The challenge is to obtain such a reward without relying on human preferences, which are expensive and conflate data realism with annotator inclinations. We propose Discriminator-Guided RL (DRL). DRL trains a discriminator to separate data from base-model samples in a pretrained representation space and uses its logit as the reward in KL-regularized RL. The pretrained space restricts the discriminator to perceptually meaningful directions, and the logit estimates the log-likelihood ratio between data and model, which is the optimal reward for targeting the data distribution. Across SiT, JiT, REPA, and RAE, DRL reduces guidance-free FID (e.g., 9.38 to 2.62 on SiT) and semantic-space FD (e.g., 88.2 to 19.3 on DINOv3 for SiT), with consistent gains across all backbones, and improves human-preference rewards without training on them. It also yields a better Pareto frontier between preference reward and image fidelity under subsequent preference-based post-training, increasing alignment while reducing low-level artifacts such as oversaturation and excessive brightness.
Original Article
View Cached Full Text

Cached at: 06/18/26, 03:57 PM

Paper page - The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

Source: https://huggingface.co/papers/2606.19162

Abstract

Discriminator-Guided Reinforcement Learning (DRL) addresses alignment issues in score- and flow-matching models by using a pretrained representation space discriminator as an optimal reward signal, improving both visual fidelity and semantic quality without human preferences.

Score- andflow-matching modelsoften rely onpreference-based reinforcement learningfor two purposes: aligning with subjective preferences and, surprisingly, recovering properties such asvisual realismandcoherent object structurethat matching-based training is intended to learn from the data itself. We argue that this reflects a structural mismatch.Matching lossesmeasure ell_2 regression error on the velocity orscore fieldunder training-time marginals, a proxy poorly aligned with the visual and semantic properties that determine sample quality at inference. Given a reward aligned with these properties, RL sidesteps the mismatch by evaluating the model on its own samples and following the reward landscape directly. The challenge is to obtain such a reward without relying on human preferences, which are expensive and conflate data realism with annotator inclinations. We proposeDiscriminator-Guided RL(DRL). DRL trains a discriminator to separate data from base-model samples in apretrained representation spaceand uses its logit as the reward in KL-regularized RL. The pretrained space restricts the discriminator to perceptually meaningful directions, and the logit estimates thelog-likelihood ratiobetween data and model, which is the optimal reward for targeting the data distribution. Across SiT, JiT, REPA, and RAE, DRL reduces guidance-freeFID(e.g., 9.38 to 2.62 on SiT) andsemantic-space FD(e.g., 88.2 to 19.3 onDINOv3for SiT), with consistent gains across all backbones, and improves human-preference rewards without training on them. It also yields a betterPareto frontierbetween preference reward and imagefidelity under subsequent preference-based post-training, increasing alignment while reducing low-level artifacts such as oversaturation and excessive brightness.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.19162 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.19162 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.19162 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Reinforcement Learning via Fixed-point Flow Matching

arXiv cs.LG

The paper proposes Fixed-point Flow Matching (FFM), a data-free method for training and fine-tuning flow-based policies in online RL by formulating the actor update as a fixed-point iteration where the critic's action gradients serve as flow-matching regression targets, showing effective offline-to-online fine-tuning and from-scratch training.

Exploring the Design Space of Reward Backpropagation for Flow Matching

Hugging Face Daily Papers

FlowBP introduces a unified surrogate-trajectory framework for aligning flow matching models with human preferences via reward backpropagation, reducing memory usage and gradient chaining while maintaining performance across multiple text-to-image models.