Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Hugging Face Daily Papers Papers
multimodal text-to-image reward-model reinforcement-learning zero-shot generation mllm

Summary

This paper introduces SpectraReward, a training-free reward function that leverages pretrained multimodal large language models (MLLMs) as zero-shot reward models for reinforcement learning in text-to-image generation, demonstrating consistent improvements over prior methods.

In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM's pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy's own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/
Original Article
View Cached Full Text

Cached at: 07/15/26, 04:21 AM

Paper page - Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation

Source: https://huggingface.co/papers/2607.11886

Abstract

Inthispaper,weproposeSpectraReward,atraining-freerewardfunctionthatturnspretrainedMLLMsintooff-the-shelfrewardmodelsforimage-generationreinforcementlearning.InsteadofaskingtheMLLMtojudgeageneratedimageoranswerdecomposedverificationquestions,SpectraRewardmeasureshowwelltheoriginalpromptcanberecoveredfromthegeneratedimagethroughasingleimage-conditioned,teacher-forcedforwardpass.Weusetheaverageimage-conditionedpromptlog-likelihoodasthereward,directlyreusingtheMLLM’spretrainedimage-textalignmentabilitywithoutpreferencelabels,reward-modelfine-tuning.WefurtherintroduceSelf-SpectraReward,aspecialcaseforunifiedmultimodalmodelswherethepolicy’sownunderstandingbranchservesastherewardmodelforitsgenerationbranch,formingaclosed-loopself-improvingframeworkwithoutexternalrewardmodelsorexternalknowledge.ExtensiveexperimentsvalidateSpectraRewardthroughabroadimage-generationRLstudycoveringtwodiffusionmodels,threeRLalgorithms,ninerewardMLLMbackbonesfromfourMLLMfamiliesspanning4Bto235Bparameters,andfiveout-of-distributiontext-to-imagebenchmarks.ResultsshowthatbothSpectraRewardandSelf-SpectraRewardsignificantlyandconsistentlyimprovegenerationperformanceandoutperformpriorMLLM-derivedrewardtrainingmethods.FurtheranalysisrevealsthatlargerrewardMLLMsarenotalwaysbetter,whileSelf-SpectraRewardcanmatchorsurpassmuchlargerexternalrewardmodels,suggestingthatreward-policyalignmentisakeyfactorforeffectiveimage-generationRL.ProjectPage:https://huangrh99.github.io/SpectraReward/

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2607\.11886

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11886 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.11886 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11886 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles