无监督多模态奖励建模

# Unsupervised Multimodal Reward Modeling
Source: [https://arxiv.org/html/2610.12707](https://arxiv.org/html/2610.12707)
Yueying Li††thanks:Work done outside of NVIDIA, while conducting research at MIT and Cornell\.Jingxuan FanAffiliation:Scale AIKianté BrantleyAffiliation:Harvard UniversitySham M\. KakadeAffiliation:Harvard UniversityHanlin ZhangAffiliation:October 8, 2026Affiliation:Harvard University

###### Abstract

Vision\-language reward models are trained on annotated preference pairs, which are expensive to collect and can disagree with each other, and are ranked by benchmark accuracy\. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3\-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer\. The layer is the decisive choice, worth 5\.8 points over the last layer, and one recipe transferred without re\-tuning beats or at least matches GPT\-4o on three benchmarks at 32B, 6\.6 points ahead on VL\-RewardBench, from a few thousand parameters at about a thousandth of fine\-tuning compute\. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF\-V pairs\. Because the backbone is frozen the objective is convex and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6\.4 points in hallucination judging\. Retraining the head on the policy’s own verifier\-labelled candidates cuts caption hallucination by 5\.1 CHAIR points and lifts image\-generation pass rate by 5\.4, while scoring lower on every reward benchmark\. Together, an intermediate\-layer probe, free same\-image perturbations and a verifier\-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine\-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank\.

Figure 1:Reward modeling with no human preference label\.\(a\)A linear probe on a frozen Qwen3\-VL backbone, trained on caption perturbations, beats or at least matches GPT\-4o at 32B\.\(b\)Two probes trained on disjoint halves of the same data tie on the benchmark and differ by 6\.4 points on hallucination judging: the sampler, not the score, sets the reward\.\(c\)Retraining the head on the policy’s own verifier\-labelled candidates cuts caption hallucination 5\.1 CHAIR points below the base generators \([Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)\)\.## 1Introduction

Reinforcement learning from human feedback\([Christiano et al\., 2017](https://arxiv.org/html/2610.12707#bib.bib5);[Ouyang et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib6)\)has made the reward model the channel through which preferences enter post\-training\. For vision\-language models the same design is now standard: a scalar scorer over an image, a prompt and a response, trained on annotated preference pairs and ranked on preference benchmarks\([Yasunaga et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib1);[Li et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib2);[Hu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib31)\)\. The design has two costs: multimodal preference pairs are expensive to annotate and the annotations disagree with each other\([Yu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib11)\)\. Once the model is built, benchmark accuracy is how it is compared, selected and trusted\. VL\-RewardBench accuracy correlates with Best\-of\-NNaccuracy on MMMU\-Pro for generative judges \(Pearson’sr\>0\.9r\>0\.9;[Li et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib2)\); what sets that number for a scalar reward model, and when it stops predicting deployment, is the question this paper takes up\.

Text reward models show that a reward is a comparison before it is a number: Bradley–Terry training identifies a reward only up to a shift that may differ across prompts\([Bradley and Terry, 1952](https://arxiv.org/html/2610.12707#bib.bib4);[Rafailov et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib41)\), so a score of \.8 has no stable meaning across prompts, candidate pools or checkpoints\. The scalar hides the relative preference that was always the operative signal\([Sun et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib57);[Zhang, 2026](https://arxiv.org/html/2610.12707#bib.bib30)\), so what a reward model learns is set by which comparisons it is trained on\. And those comparisons need no annotator: web continuations as chosen against in\-batch continuations as rejected train a text reward model with no label at all\([Fan et al\., 2026](https://arxiv.org/html/2610.12707#bib.bib29)\)\. We take both findings to the vision\-language setting, make the comparison the object of study, and remove the human labels entirely\. The reward model is a linear head on the hidden states of an intermediate layer of a frozen backbone, trained by in\-batch Bradley–Terry\([Bradley and Terry, 1952](https://arxiv.org/html/2610.12707#bib.bib4)\)against same\-image perturbations of web captions or same\-seed re\-renders of a perturbed prompt \([Fig\.2](https://arxiv.org/html/2610.12707#S1.F2),[Section2](https://arxiv.org/html/2610.12707#S2)\)\. The claim is about the training data: the probe layer is selected on one labeled benchmark, which we therefore treat as development and never as transfer evidence, and the loop of[Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)uses verifier labels, not annotators\. Two things then decide what the head learns: the representation it reads, which sets the signal a linear head can extract, and the comparison distribution it is trained on, which sets the direction of the reward\. Every choice in the paper is one of the two, and the verified policy errors of[Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)are the second variable’s final instance\. A head of a few thousand parameters built this way beats GPT\-4o by 6\.6 points on VL\-RewardBench at 32B, for about a thousandth of the compute of full backbone fine\-tuning, and is level with GPT\-4o on Multimodal RewardBench and MMRB2\. At an equal pair count, the mined pairs teach a frozen head more than curated RLAIF\-V pairs do\.

Figure 2:\(a\) The reward from a linear probe\.A pair \(image, prompt, real caption, same\-image perturbation\) passes once through the frozen backboneϕ\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}\\phi\}; the layer\-N−k\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}N\{\-\}k\}states are mean\-pooled over response tokens intoh\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}h\}and scored by a bias\-free headw\{\\color\[rgb\]\{0\.6523,0\.4141,0\}w\}, trained with in\-batch matrix Bradley–Terry; the samplerqneg\{\\color\[rgb\]\{0\.0039,0\.418,0\.3047\}q\_\{\\mathrm\{neg\}\}\}is the only place supervision enters\.\(b\) One turn of the bootstrap loop\([Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)\): cheap verifiers label the policy’s own Best\-of\-NNcandidates and the same head is retrained on the minted pairs\.Bluefrozen,orangetrained,greensupervision\.Because the backbone is frozen, the objective is transparent enough to say what that number is\. It is ranking noise\-contrastive estimation over fixed features\([Gutmann and Hyvärinen, 2010](https://arxiv.org/html/2610.12707#bib.bib19);[Ma and Collins, 2018](https://arxiv.org/html/2610.12707#bib.bib20)\), so with a fixed quadratic penalty it is convex with a unique minimizer, characterized as an average of the negative sampler’s difference vectors weighted by how badly the head still misorders each pair \([Section2\.2](https://arxiv.org/html/2610.12707#S2.SS2)\)\. AdamW does not optimize that fixed objective, so where the trained heads land is measured rather than derived\. In our runs no training\-side change \(pool size, steps, curriculum, loss form or optimizer setting\) moves the benchmark by more than \.014, so the representation and the sampler are the two choices that matter\. The representation is the backbone’s, and our findings about it hold across backbones\. On four Qwen3\-VL sizes and on a second family, GLM\-4\.1V\-9B, the best probe layer sits well below the top, twelve layers down for 8B, 32B and GLM\-4\.1V\-9B\. The same recipe with no re\-tuning lifts each of them above its own untrained judge \([Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2)\)\. The sampler is the other choice, and it is where the benchmark stops being informative\.

Which pairs train the head decides which reward it learns\. Two probes trained on disjoint subsets of the same perturbations, split by whether CLIP can see the edit, reach similar benchmark accuracy and differ by 6\.4 points in hallucination judging \([Fig\.1](https://arxiv.org/html/2610.12707#S0.F1)b\)\. Deployed for Best\-of\-NNreranking, benchmark rank orders utility in neither direction, and where the probe does beat majority vote, by 7 to 11 points, a training\-free log\-likelihood judge matches it\. The analysis leaves one sampler to try: the policy’s own errors\. We mint those labels with cheap verifiers on Best\-of\-NNcandidates, for captions and for generated images; the resulting head cuts caption hallucination by 5\.1 CHAIR points below the base Qwen3\-VL\-2B and 8B generators, while its VL\-RewardBench accuracy falls below that of the probe it replaced \([Fig\.1](https://arxiv.org/html/2610.12707#S0.F1)c\)\. One DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib41)\)turn on the policy’s own pairs also cuts its hallucination by 3 to 6 CHAIR\([Rohrbach et al\., 2018](https://arxiv.org/html/2610.12707#bib.bib28)\)points at no QA cost \([Table8](https://arxiv.org/html/2610.12707#A7.T8)\)\.

Our contributions combine probing, negative sampling and self\-improvement, each studied on the policy side before \([Section4](https://arxiv.org/html/2610.12707#S4)\), into one reward\-side recipe that uses no human preference label\. \(i\)*Probed, not trained in:*a frozen vision\-language backbone with a linear head \(2,048 parameters at 2B\) at a depth set by the backbone, at or above GPT\-4o on three benchmarks at 32B \([Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2)\)\. \(ii\)*Supervision as a sampler:*same\-image perturbations written as a mixture of operators that apply to captions and to rendered images alike, with an argument for which reward direction they select and an ablation of the operators in both modalities \([Section2\.2](https://arxiv.org/html/2610.12707#S2.SS2),[Table3](https://arxiv.org/html/2610.12707#A4.T3)\)\. \(iii\)*What the score tracks:*benchmark accuracy plateaus under every training\-side change we tried, the learned direction is set by the sampler, and benchmark rank predicts Best\-of\-NNrank in neither direction \([Sections3\.3](https://arxiv.org/html/2610.12707#S3.SS3)and[3\.3](https://arxiv.org/html/2610.12707#S3.SS3.SSS0.Px2)\)\. \(iv\)*A bootstrap loop*that replaces annotators with cheap verifiers on the policy’s own candidates, the one supervision that moved deployment, shown for captions and for image generation \([Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)\)\.

## 2Methodology

### 2\.1Algorithm

We score each response with a bias\-free linear head on mean\-pooled response\-token states from layerN−k\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}N\{\-\}k\}of a frozen Qwen3\-VL backbone\([Qwen Team, 2025b](https://arxiv.org/html/2610.12707#bib.bib3)\)\. The triple \(layer, pooling, head\) is the design space the rest of the paper sweeps \([Fig\.2](https://arxiv.org/html/2610.12707#S1.F2)a; colours as in its caption\)\. For an imagexx, a promptqqand a responseyy, letht\(N−k\)\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}h^\{\(N\-k\)\}\_\{t\}\}be the hidden state of layerN−k\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}N\{\-\}k\}at response tokentt; the reward is

sw​\(x,q,y\)=w⊤​1\|y\|​∑t∈yht\(N−k\),\{\\color\[rgb\]\{0\.6523,0\.4141,0\}s\_\{w\}\}\(x,q,y\)\\;=\\;\{\\color\[rgb\]\{0\.6523,0\.4141,0\}w\}^\{\\top\}\\,\\frac\{1\}\{\|y\|\}\\sum\_\{t\\in y\}\{\\color\[rgb\]\{0\.0078,0\.3125,0\.4883\}h^\{\(N\-k\)\}\_\{t\}\},\(1\)so the 2B model trains2,0482\{,\}048parameters and the backbone none\. A micro\-batch ofBBpreference pairs\{\(xi,qi,yi\+,yi−\)\}i=1B\\\{\(x\_\{i\},q\_\{i\},y\_\{i\}^\{\+\},y\_\{i\}^\{\-\}\)\\\}\_\{i=1\}^\{B\}gives the in\-batch matrix Bradley–Terry loss, every chosen response against every rejected one,

ℒ=1B2∑i=1B∑j=1B−logσ\(β\(si\+−sj−\)\),\\mathcal\{L\}=\\frac\{1\}\{B^\{2\}\}\\sum\_\{i=1\}^\{B\}\\sum\_\{j=1\}^\{B\}\-\\log\\sigma\\\!\\big\(\\beta\\,\(\{\\color\[rgb\]\{0\.6523,0\.4141,0\}s^\{\+\}\_\{i\}\}\-\{\\color\[rgb\]\{0\.6523,0\.4141,0\}s^\{\-\}\_\{j\}\}\)\\big\),\(2\)withβ=0\.1\\beta\{=\}0\.1the inverse temperature andB=8B\{=\}8, so each positive meets eight negatives per step rather than one\. Note that supervision enters only through the negative sampler, which is what makes the sampler the object[Section2\.2](https://arxiv.org/html/2610.12707#S2.SS2)reasons about\. For a positive\(x,q,y\+\)\(x,q,y^\{\+\}\)the rejected response is drawn from a mixture of perturbation operators applied to the positive itself,

y−∼qneg\(⋅∣x,q,y\+\)=∑oπo𝒯o\(y\+;x,q\),∑oπo=1,\{\\color\[rgb\]\{0\.0039,0\.418,0\.3047\}y^\{\-\}\}\\;\\sim\\;\{\\color\[rgb\]\{0\.0039,0\.418,0\.3047\}q\_\{\\mathrm\{neg\}\}\}\(\\cdot\\mid x,q,y^\{\+\}\)\\;=\\;\\sum\_\{o\}\\pi\_\{o\}\\,\{\\color\[rgb\]\{0\.0039,0\.418,0\.3047\}\\mathcal\{T\}\_\{o\}\}\(y^\{\+\};x,q\),\\qquad\\textstyle\\sum\_\{o\}\\pi\_\{o\}=1,\(3\)and the modality fixes what the operators act on\. Unlike the pixel\-level perturbations of ImageNet\-C\([Hendrycks and Dietterich, 2019](https://arxiv.org/html/2610.12707#bib.bib55)\), which perturb the input, these operators edit the*response*and change a fact while leaving the input untouched\. For text responses𝒯near\\mathcal\{T\}\_\{\\mathrm\{near\}\}substituteskkeditable spans ofy\+y^\{\+\}\(colour, number, size or spatial words\),𝒯splice\\mathcal\{T\}\_\{\\mathrm\{splice\}\}replaces its tail with another caption’s,𝒯shuf\\mathcal\{T\}\_\{\\mathrm\{shuf\}\}permutes its clauses, and𝒯×\\mathcal\{T\}\_\{\\times\}returns the caption of another image; the recipe usesπ×=0\.15\\pi\_\{\\times\}\{=\}0\.15andk=1k\{=\}1\([Figs\.6](https://arxiv.org/html/2610.12707#A4.F6)and[D](https://arxiv.org/html/2610.12707#A4)\)\. For example, a caption reading*two brown horses beside a red fence*becomes*two brown horses beside a blue fence*: one colour word moves, the token count is unchanged, and the pair differs only in a fact about the image\. For image responses the operator acts on the prompt and a fixed\-seed generatorGGrenders both sides,y\+=G⁡\(q,s\)y^\{\+\}=G\(q;s\)andy−=G⁡\(𝒯near​\(q\),s\)y^\{\-\}=G\(\\mathcal\{T\}\_\{\\mathrm\{near\}\}\(q\);s\)\. The usual label\-free alternative is to have a model write the rejected response, as in constitutional and AI\-feedback pipelines\([Bai et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib51);[Lee et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib52);[Yu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib11)\)\. We do not, because such a negative differs from the human positive in author, length and style as well as in fact, and by[Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)the reward learns every one of those differences; length alone is a known shortcut for reward models\([Singhal et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib10)\)\. In our own test a head trained on human captions against the policy’s captions for the same images scores \.36 on Multimodal RewardBench, below chance, and picks the more hallucinated caption in every Best\-of\-NNcell \([AppendixF](https://arxiv.org/html/2610.12707#A6)\)\. Editing one fact inside the same response holds author, length and style fixed by construction\. These negatives cost nothing to label: the caption operators are string edits with no model in the loop, and the 22k\-pair bank builds in 10 seconds on one CPU core\. Rendering the image arm’s 1,212 pairs takes 5 GPU\-hours \([AppendixK](https://arxiv.org/html/2610.12707#A11)\)\.

### 2\.2What the Objective Identifies

We read[Eq\.2](https://arxiv.org/html/2610.12707#S2.E2)as ranking noise\-contrastive estimation\([Gutmann and Hyvärinen, 2010](https://arxiv.org/html/2610.12707#bib.bib19);[Ma and Collins, 2018](https://arxiv.org/html/2610.12707#bib.bib20)\), with the reward a linear energy over frozen features\([LeCun et al\., 2006](https://arxiv.org/html/2610.12707#bib.bib21)\)andβ\\betathe InfoNCE temperature\([van den Oord et al\., 2018](https://arxiv.org/html/2610.12707#bib.bib14)\)\. Writedi​j=hi\+−hj−d\_\{ij\}=h^\{\+\}\_\{i\}\-h^\{\-\}\_\{j\}, so that[Eq\.2](https://arxiv.org/html/2610.12707#S2.E2)is1B2​∑i​jℓ⁡\(β​w⊤​di​j\)\\frac\{1\}\{B^\{2\}\}\\sum\_\{ij\}\\ell\(\\beta\\,w^\{\\top\}d\_\{ij\}\)withℓ⁡\(z\)=log⁡\(1\+e−z\)\\ell\(z\)=\\log\(1\+e^\{\-z\}\)\. We keep three objects apart: an idealized objective over unrestricted scores \([Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)\); the linear head under a fixed penaltyλ2​w⊤​D​w\\frac\{\\lambda\}\{2\}w^\{\\top\}Dw\([Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1)\), a strongly convex problem with one minimizer that the sampler andβ\\betadefine and gradient descent reaches from any start; and the heads AdamW trains, whose stationary points depend on its preconditioner\([Xie and Li, 2024](https://arxiv.org/html/2610.12707#bib.bib74)\), which we measure \([Section3\.1](https://arxiv.org/html/2610.12707#S3.SS1.SSS0.Px3)\)\.[AppendixI](https://arxiv.org/html/2610.12707#A9)reads the two statements and[AppendixJ](https://arxiv.org/html/2610.12707#A10)proves them\.

###### Proposition 1\(the optimum is a density ratio against the sampler\)\.

LetPPandQQbe the joint laws of a positive and of a negative triple\(x,q,y\)\(x,q,y\), the latter with the sampler of[Eq\.3](https://arxiv.org/html/2610.12707#S2.E3)averaged over the positive it edits,q¯​\(y∣x,q\)\\bar\{q\}\(y\\mid x,q\), and assumePPandQQare mutually absolutely continuous\. Over unrestricted scores, the cross\-entry terms of[Eq\.2](https://arxiv.org/html/2610.12707#S2.E2)are minimized exactly by

s⋆​\(x,q,y\)=1β​log⁡d​Pd​Q​\(x,q,y\)\+C=1β​log⁡p⁡\(y∣x,q\)q¯​\(y∣x,q\)\+C,s^\{\\star\}\(x,q,y\)\\;=\\;\\frac\{1\}\{\\beta\}\\log\\frac\{\\mathrm\{d\}P\}\{\\mathrm\{d\}Q\}\(x,q,y\)\\;\+\\;C\\;=\\;\\frac\{1\}\{\\beta\}\\log\\frac\{p\(y\\mid x,q\)\}\{\\bar\{q\}\(y\\mid x,q\)\}\\;\+\\;C,\(4\)withCCa constant\. TheBBsame\-entry terms, wherey−y^\{\-\}is an edit of the veryy\+y^\{\+\}it is scored against, are coupled and are not minimized by[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)\.

Those coupled terms carry weight1/B=12\.5%1/B=12\.5\\%atB=8B\{=\}8and move even the unrestricted optimum off[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)\([AppendixJ](https://arxiv.org/html/2610.12707#A10)\)\. Our sampler also violates the absolute\-continuity assumption, since a caption of another image has probability zero for this one, so[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)describes what the cross\-entry terms reward and the linear head is analyzed through[Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1)\. Length drops out of the ratio when the negatives match its law conditionally on content, and a matched marginal is not enough \([AppendixI](https://arxiv.org/html/2610.12707#A9)\); our bank holds length fixed on two thirds of its pairs and shortens the negative by about 5% on the rest, so nothing makes the target length\-free, and[Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)finds the length artifact this allows\.

#### Why perturbations provide useful supervision\.

Independent negatives from the caption marginal, our𝒯×\\mathcal\{T\}\_\{\\times\}, makes⋆s^\{\\star\}the image–caption pointwise mutual information up to a constant \([SectionI\.2](https://arxiv.org/html/2610.12707#A9.SS2)\): a score of topic association, which can rank an accurate and an inaccurate response on the same topic equally\. A same\-context edit changes one fact and nothing else\. In the stylized model of[SectionI\.3](https://arxiv.org/html/2610.12707#A9.SS3)the quality factor cancels from the PMI score, while the edited pairs recover the correct ordering with marginlog⁡η1−η\\log\\frac\{\\eta\}\{1\-\\eta\}even when a fraction1−η1\-\\etaof the originals is wrong\. Perturbations thus supply one quality comparison, an original above its degraded copy, without requesting a judgment\.

###### Corollary 1\(the sampler’s pair law sets the direction\)\.

With the penaltyλ2​w⊤​D​w\\frac\{\\lambda\}\{2\}w^\{\\top\}Dw, the population minimizer of[Eq\.2](https://arxiv.org/html/2610.12707#S2.E2)satisfies

D​w⋆=βλ​𝔼​\[σ⁡\(−β​w⋆⁣⊤​d\)​d\],d=h\+−h−,D\\,w^\{\\star\}\\;=\\;\\frac\{\\beta\}\{\\lambda\}\\;\\mathbb\{E\}\\big\[\\,\\sigma\(\-\\beta\\,w^\{\\star\\top\}d\)\\;d\\,\\big\],\\qquad d=h^\{\+\}\-h^\{\-\},\(5\)the expectation over the pair law of the matrix loss:D​w⋆Dw^\{\\star\}is the mean difference vector of the sampler’s pairs, each weighted by the probability the head still misorders it, so two samplers with the same perturbations but different pair laws give different heads in general\.

The law of the differences, not their span, sets the direction; this is how the two CLIP\-split heads of[Section3\.3](https://arxiv.org/html/2610.12707#S3.SS3)can reach similar benchmark accuracy while differing by 6\.4 points in hallucination judging\. Nothing in the objective sets the head’s length coefficient to zero, and[Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)finds it with opposite signs at 2B and 8B under one sampler\. Finally,[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)orders a policy’s own samples by a ratio against the perturbation process, so it ranks them by how unlike a perturbation each one looks and has no term for how wrong it is\. The design hypothesis this leaves is that a sampler drawn from the policy’s own errors ranks them better:[Section3\.3](https://arxiv.org/html/2610.12707#S3.SS3.SSS0.Px2)measures how far the perturbation sampler falls short and[Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)tests the hypothesis\.

### 2\.3Experimental Setup

We build pairs from 70k PixMo web captions\([Deitke et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib12)\)and train heads on Qwen3\-VL Instruct backbones at 2B, 4B, 8B and 32B, with the same negative bank and a frozen vision tower at every size\. The headline recipe is the 2B backbone at probe layerN−4N\{\-\}4, head only, 200 steps of 256 pairs with AdamW at learning rate10−310^\{\-3\}and weight decay0\.010\.01; every later run changes one element of it, and the curated comparison set is RLAIF\-V\([Yu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib11)\)\. Every head trains on the same schedule of 256 pairs per step, so two runs are compared at equal pairs seen along the training timeline \([Fig\.3](https://arxiv.org/html/2610.12707#S3.F3), right\)\.

We evaluate on Multimodal RewardBench\([Yasunaga et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib1)\), VL\-RewardBench\([Li et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib2)\)and MMRB2\([Hu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib31)\), with GPT\-4o\([OpenAI, 2024](https://arxiv.org/html/2610.12707#bib.bib40)\)as the anchor on all three and no re\-tuning between them\. Per\-benchmark pair counts, the MMRB2 scoring construction and the confidence bands are in[AppendixC](https://arxiv.org/html/2610.12707#A3)\.

For Best\-of\-NNwe sampleN=16N\{=\}16candidates from a Qwen3\-VL\-2B or 8B policy over 2,500 prompts of COCO captioning\([Lin et al\., 2014](https://arxiv.org/html/2610.12707#bib.bib36)\), MathVista\([Lu et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib37)\), MMMU\([Yue et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib38)\)and Claude\-written hard questions on COCO images\. We measure CHAIRs\([Rohrbach et al\., 2018](https://arxiv.org/html/2610.12707#bib.bib28)\)or exact match against six training\-free controls, from random to the oracle, with 95% confidence intervals from a paired bootstrap over prompts\.

## 3Experiments

### 3\.1Two Choices Set the Accuracy: The Probe Layer and the Sampler

Two choices set the accuracy, and neither is an optimizer setting: the probe layer alone adds 5\.8 trainable points over the last layer, and mined negatives beat curated RLAIF\-V pairs by 3\.5 at equal pair count\.[Table2](https://arxiv.org/html/2610.12707#A4.T2)puts those two and every other ablation of the paper on one backbone under one recipe\. Every number is a real end\-to\-end run except the layer sweep that nominated the probe depth: linear heads trained briefly on precomputed hidden states of every layer, a relative instrument that appears only in[Fig\.3](https://arxiv.org/html/2610.12707#S3.F3)and[AppendixD](https://arxiv.org/html/2610.12707#A4)\.

Figure 3:Left:layer\-sweep trainable accuracy against layers below the top for 2B / 4B / 8B / 32B \(bands: 75% binomial intervals\); the optimum is never the last layer, and sweep numbers rank depths within a model only\.Right:Multimodal RewardBench pair accuracy over the first 250 steps by negative type \(2B full fine\-tune, 674\-pair eval subset\): same\-image perturbations transfer, cross\-image negatives fall below the untrained base, curated pairs plateau lower\.#### Probe depth\.

The best probe layer is never the top of the network: it isN−4N\{\-\}4for 2B,N−6N\{\-\}6for 4B andN−12N\{\-\}12for both 8B and 32B\. That depth is chosen on Multimodal RewardBench trainable accuracy, so we treat that benchmark as development throughout and VL\-RewardBench and MMRB2, which no choice in the recipe has seen, as the transfer evidence\. The gap over the last layer is 3 to 7 points in the layer sweep \([Fig\.3](https://arxiv.org/html/2610.12707#S3.F3)\) and 5\.8 trainable points on the real 4B run\. The optimum deepens from 2B to 8B and then holds at twelve layers below the top for 8B, 32B and GLM\-4\.1V\-9B \(36, 64 and 40 layers\), while the 36\-layer 4B peaks atN−6N\{\-\}6, so the depth is set by the backbone and not by its layer count; each peak is a plateau several layers wide \([AppendixD](https://arxiv.org/html/2610.12707#A4)\)\.

#### Negative construction\.

We find that same\-image perturbations transfer in both modalities and that cross\-image negatives do not \([Table3](https://arxiv.org/html/2610.12707#A4.T3)\)\. For text,𝒯×\\mathcal\{T\}\_\{\\times\}alone reaches 99% training accuracy and falls below the untrained base on the benchmark, because it amplifies a length and style shortcut the same\-image operators mostly remove: that head scores \.224 on the length\-sensitive safety/bias column against \.451\. For images the replica is exact, with the deranged negative transferring nothing to MMRB2 text\-to\-image \(52\.7 against a label prior of 53\.2\) while the same\-seed planted edit accounts for all of it \(56\.0\)\. Perturbation density and the operator mix are in[AppendixD](https://arxiv.org/html/2610.12707#A4)\.

#### Every training\-side change leaves the benchmark within \.014 of the recipe\.

Changing the loss form or the negative composition, training on four times the pairs or for twice the steps, or ordering the negatives from easy to hard or from hard to easy moves a real 2B run by at most \.014\. In the sweep setting, changing the negative pool or the temperature schedule moves it by less than \.005\. The sharpest row is a 540\-configuration grid over head learning rate,β\\betaand weight decay: all 540 reach \.995 or better in distribution, and selecting among them out of distribution changes the benchmark by \.0003\.

#### Mined negatives against curated pairs at a matched pair budget\.

At 15k pairs the mined\-negative head is 3 to 4 trainable points ahead of curated RLAIF\-V pairs on Multimodal RewardBench and 2 to 5 on VL\-RewardBench, including the hallucination split RLAIF\-V was built for \([Table5](https://arxiv.org/html/2610.12707#A4.T5)\)\. Unfreezing the backbone on the same 15k curated pairs recovers 11 points overall to \.613: a frozen head matches full fine\-tuning on mined\-negative data where real pairs need the unfrozen model \([AppendixD](https://arxiv.org/html/2610.12707#A4)\)\.

### 3\.2One Untuned Recipe Reaches GPT\-4o Across Backbones and Benchmarks

Table 1:One recipe on three reward benchmarks\.Each probe is one frozen linear head at its probe depth, trained once on caption perturbations and evaluated with no re\-tuning\. Published judges are quoted from each benchmark’s paper; our Multimodal RewardBench rows \(†\\dagger\) are over the 4,711 released pairs, which omit the withheld toxicity split \(GPT\-4o recomputed on them: \.728 overall, Gemini 1\.5 Pro \.734\), and safety/bias is a length artifact for every mined\-negative head \([Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)\)\. VL\-RewardBench macro is the mean of its three splits\. The 2B and 8B rows use the 60k\-pair bank, the 4B and 32B rows the 15k\-pair rebuild \([AppendixD](https://arxiv.org/html/2610.12707#A4)\)\. Bold: column maximum among our Qwen3\-VL rows\. The GLM\-4\.1V\-9B rows are a second backbone family under the identical recipe \(15k pairs, 200 steps,N−12N\{\-\}12of 40 layers, no re\-tuning\)\.VL\-RewardBenchOverallMacroGeneralHalluc\.Reason\.Gemini\-1\.5\-Pro \(strongest published\)\.672––––GPT\-4o \(published\)\.658––––Ours: 32B probe,N−12N\{\-\}12\.724\.649\.475\.810\.663Ours: 8B probe,N−12N\{\-\}12\.682\.607\.425\.765\.631Ours: 4B probe,N−6N\{\-\}6\.680\.600\.420\.774\.606Ours: 2B probe,N−4N\{\-\}4\.669\.592\.425\.762\.587Ours: GLM\-4\.1V\-9B probe,N−12N\{\-\}12\.602\.531\.348\.678\.568base 8B \(log\-likelihood judge\)\.542\.525\.459\.554\.561base 4B \(log\-likelihood judge\)\.525\.518\.464\.522\.568base 2B \(log\-likelihood judge\)\.569\.550\.525\.598\.527base GLM\-4\.1V\-9B \(log\-likelihood judge\)\.570\.529\.425\.616\.546

The 32B probe is at or above GPT\-4o on all three benchmarks and separably above it on one \([Table1](https://arxiv.org/html/2610.12707#S3.T1)\)\. On VL\-RewardBench it is the strongest entry on the table, 6\.6 points above GPT\-4o at 3\.6 standard errors of the difference; on Multimodal RewardBench and on MMRB2 reasoning the benchmark cannot resolve the margin, and separating either at two standard errors would take 13 and 3\.5 times the released pairs\. The stronger reference than GPT\-4o is the backbone itself as a generative judge, which on MMRB2, the one benchmark we ran it on, leads the probe by 4\.7 points \(64\.6 against 59\.9 on the four\-task average,[AppendixC](https://arxiv.org/html/2610.12707#A3)\)\. In exchange, the probe decodes nothing: it needs one forward pass per response, where the judge generates a judgment in both presentation orders\. Larger backbones improve the intermediate\-layer probe on all three benchmarks, while the base log\-likelihood judge stays near chance and the last\-layer head trails at every size \([Fig\.7](https://arxiv.org/html/2610.12707#A5.F7)\)\. The recipe is not specific to Qwen3\-VL: on GLM\-4\.1V\-9B, a second family with 40 layers, the same recipe atN−12N\{\-\}12beats its last layer and its untrained judge on both benchmarks \([Table1](https://arxiv.org/html/2610.12707#S3.T1)\)\. A head trained only on caption perturbations also transfers a usable signal to response\-image tokens once the backbone is large enough \(MMRB2 editing, level with GPT\-4o\), which starts[Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4.SSS0.Px2)\. Our training data shares no prompt with the benchmarks and 0\.1% of images with Multimodal RewardBench \([AppendixB](https://arxiv.org/html/2610.12707#A2)\)\.

#### What the columns mean\.

The safety/bias column of Multimodal RewardBench is a length artifact, the one reward models are known to carry\([Singhal et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib10)\)\. A head trained on same\-image perturbations, two thirds of which match length exactly, scores \.998 on it at 2B and \.19 at 8B, where its correlation with response length flips sign\. The gap is not seed noise: eight layer\-sweep seeds give\.967±\.007\.967\\pm\.007at 2B and\.569±\.013\.569\\pm\.013at 8B, the same ordering \([AppendixC](https://arxiv.org/html/2610.12707#A3)\)\. We therefore compare heads on the five*trainable*categories, where the 8B probe’s \.659 overall against \.708 at 2B becomes \.658 against \.602\.

### 3\.3Two Probes, One Score, Two Rewards

#### Same benchmark score, different reward\.

We train two probes on the two extreme quartiles of the same mined\-negative data, ranked by the drop in CLIP\([Radford et al\., 2021](https://arxiv.org/html/2610.12707#bib.bib35)\)image–text cosine from the real to the perturbed caption, 15k pairs each at 2B\. They reach similar accuracy on Multimodal RewardBench, \.597 against \.603 trainable, and then separate three ways \([Fig\.1](https://arxiv.org/html/2610.12707#S0.F1)b\): VL\-RewardBench hallucination 72\.0 against 78\.4, reasoning 63\.4 against 56\.8, and as Best\-of\-NNscorers the CLIP\-visible head beats probe\-8B on CHAIRs by 3\.8 points while losing 2\.9 on QA\. The same pattern, a small benchmark gap against a large hallucination gap, holds across sizes, consistent with[Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1): the two samplers differ from the positives along different coordinates, and the benchmark does not separate them\.

#### Whether a scorer helps Best\-of\-NNdepends on the pool, not on its benchmark score\.

Across eight captioning and QA cells and a hard\-QA ladder, a probe’s benchmark accuracy does not predict how much it improves Best\-of\-NNselection, in either direction \([Tables6](https://arxiv.org/html/2610.12707#A6.T6)and[F](https://arxiv.org/html/2610.12707#A6)\)\. On captioning every mined\-negative probe stays at greedy level where the oracle reaches \.06 to \.09 CHAIRs, and on MathVista and MMMU, where candidate agreement tracks correctness, majority vote is 1\.4 to 3\.8 points ahead of the probe, whose one positive role there is as a vote weighter\. Where the 8B generator is confidently wrong instead \(majority wrong on 32% of prompts\), every external scorer beats majority vote: probe\-2B by 10\.8 points, probe\-8B by 7\.2 and the training\-free log\-likelihood judge by 9\.2, all with 95% CIs excluding zero\. The trained head does not significantly beat the training\-free judge: probe\-2B against log\-likelihood is\+1\.6\+1\.6to\+2\.1\+2\.1points with CIs including zero, and probe\-2B beats probe\-8B by 3\.6 points, the reverse of their benchmark order\. What decides the gain is the pool: whether the generator’s majority can be trusted, and so whether any external scorer should replace the vote\. The benchmark measures neither\.

### 3\.4Learning from the Policy’s Own Errors: Better Deployment, Lower Benchmarks

Figure 4:One turn of the loop\.\(a\)CHAIRs@16 per captioning cell \(hatched: verified probe trained without COCO\-2\)\.\(b\)Judging \(VL\-RewardBench\) against ranking \(Δ\\DeltaCHAIRs vs greedy\)\.[Figure2](https://arxiv.org/html/2610.12707#S1.F2)b changes one element of the recipe of[Section2\.1](https://arxiv.org/html/2610.12707#S2.SS1), the negative sampler: the policy’s own Best\-of\-NNcandidates, labelled by a cheap verifier, are paired correct against wrong for the same prompt\. We train the head to separate verified successes from failures among the policy’s own outputs, the comparison that Best\-of\-NNselection makes at deployment\.

It lowers captioning CHAIRs where the perturbation probes did not: a head trained on 3,462 such pairs cuts CHAIRs by 5\.1 points against greedy and by 4\.1 points on a task it never trained on \([Fig\.1](https://arxiv.org/html/2610.12707#S0.F1)c,[Fig\.4](https://arxiv.org/html/2610.12707#S3.F4)a,[Table7](https://arxiv.org/html/2610.12707#A6.T7)\)\. On QA it matches probe\-8B as a selector, so the utility claim rests on captioning, the open\-ended task where no vote is available\. CHAIR is both harvest label and metric, so the gain is in absent COCO\-object mentions rather than faithfulness at large \([AppendixA](https://arxiv.org/html/2610.12707#A1)\)\.

The verified head is the weakest judge of the three heads \([Fig\.4](https://arxiv.org/html/2610.12707#S3.F4)b\) on VL\-RewardBench, \.615 against \.669 and \.682 for the perturbation probes, yet the one that ranks the policy’s captions below greedy’s hallucination rate \(the other benchmarks in[Table7](https://arxiv.org/html/2610.12707#A6.T7)\)\. Both follow from the sampler: 70% of the verified pairs are QA correctness pairs, so the categories they do not cover fall while math rises\. The two rewards differ because the two comparisons do\. Benchmark pairs contrast other models’ responses on annotator\-chosen errors, close to what perturbations produce; verified pairs contrast this policy’s own successes and failures\. In our runs the head trained on the second kind improves candidate selection and loses VL\-RewardBench accuracy\. Serving both the benchmark and the deployment is open: train on the union of the two samplers, give the head more than one direction, or draw benchmark pairs from the policy under deployment\.

#### The actor side of the turn\.

One DPO turn on pairs the policy generated itself lowers its COCO hallucination rate from \.446 to \.390 CHAIRs with probe\-picked pairs, \.414 with verified and \.412 with random pairs, at no cost on MathVista or MMMU \([Table8](https://arxiv.org/html/2610.12707#A7.T8)\)\. The label adds little over random pairs, so the selector’s effect on the actor is still to be shown \([AppendixG](https://arxiv.org/html/2610.12707#A7)\)\.

![Refer to caption](https://arxiv.org/html/2610.12707v1/image_bon_hard.png)Figure 5:The loop transposes to image generation: verifier\-minted pairs gain 5\.4 points over random selection at 8B, while the 2B head trained on caption perturbations loses 6\.5\.\(a\)Best\-of\-NNon 160 held\-out hard compositional prompts: verifier pass rate of the image each selector picks fromNNQwen\-Image samples \(bands: 75% binomial intervals; paired differences against random atN=12N\{=\}12in the text\)\.\(b\)One prompt and each selector’s pick out of twelve, judged by the GenEval\-style OWLv2 verifier\([Ghosh et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib32);[Minderer et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib33)\)on object presence, count, colour and relative position\. The selectors see only the image\.
#### The same turn on image generation\.

The loop transposes to image generation point for point \([Fig\.5](https://arxiv.org/html/2610.12707#S3.F5)\): Qwen\-Image\([Qwen Team, 2025a](https://arxiv.org/html/2610.12707#bib.bib39)\)produces pools of 12 on GenEval\-style prompts\([Ghosh et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib32)\)and a label\-free OWLv2 verifier\([Minderer et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib33)\)plays CHAIR’s role\. On 160 held\-out hard prompts \(random \.765, oracle \.956\), the text\-trained 2B head is 6\.5 points below random \(95% CI−11\.6\-11\.6to−1\.6\-1\.6\), and the mined\-negative image heads are\+2\.5\+2\.5, on par with CLIP \(\+3\.0\+3\.0\)\. A head trained on 3,000 verifier\-minted pairs is\+5\.4\+5\.4at 8B \(95% CI\+0\.5\+0\.5to\+10\.3\+10\.3\), 2\.4 points above CLIP and 2\.9 above the mined\-negative image heads on the same prompts\. The two verified heads cross over as MMRB2 judges, so here too the better ranker is the worse judge\.

## 4Related Work

Multimodal reward models and the benchmarks that rank them\.A multimodal reward model is built by annotating preference pairs and fitting a scorer on them, from LLaVA\-RLHF and RLHF\-V\([Sun et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib45);[Yu et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib46)\)to MM\-RLHF’s 120k human comparisons\([Zhang et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib47)\)and the scoring heads of IXC\-2\.5\-Reward and UnifiedReward\([Zang et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib48);[Wang et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib49)\)\. Removing the annotator usually means a model in their place\([Bai et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib51);[Lee et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib52);[Li et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib70);[Yu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib11)\); we use a sampler, as hallucination\-injection DPO does for the policy\([Zhou et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib64);[Zhao et al\., 2023b](https://arxiv.org/html/2610.12707#bib.bib65)\)\. Multimodal RewardBench\([Yasunaga et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib1)\), VL\-RewardBench\([Li et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib2)\)and MMRB2\([Hu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib31)\)rank those models and generative judges\([Zheng et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib44);[Xiong et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib69)\)by pairwise accuracy against human labels\([Lambert et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib7);[Malik et al\., 2026](https://arxiv.org/html/2610.12707#bib.bib61);[Frick et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib59), cf\.\)\. We take them as given and ask what the number measures\. In text, policies optimized toward reward models of similar accuracy can perform quite differently\([Wen et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib60)\), while an outcome reward model’s score tracks downstream accuracy better than validation loss does\([Qi et al\., 2026](https://arxiv.org/html/2610.12707#bib.bib58)\); we ask when a reward model’s own benchmark score stops tracking deployment\.

Probing and probe depth\.The scorer we set against those judges is a probe, and linear probes are classically an analysis tool for encoders\([Alain and Bengio, 2017](https://arxiv.org/html/2610.12707#bib.bib8);[Tenney et al\., 2019](https://arxiv.org/html/2610.12707#bib.bib9);[Burns et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib53)\); intermediate layers often give better features than the last\([Skean et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib66)\)\. LLaVA\-Reward\([Zhou et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib34)\)is the closest prior use of a probe as a reward, mixing depths through a cross\-attention module trained on annotated preferences; ours is a 2,048\-parameter head with no label\.

Density\-ratio estimation and negative sampling\.A probe on a frozen backbone approximates a density ratio, which noise\-contrastive estimation fits against a noise distribution\([Gutmann and Hyvärinen, 2010](https://arxiv.org/html/2610.12707#bib.bib19);[van den Oord et al\., 2018](https://arxiv.org/html/2610.12707#bib.bib14)\)that the negative\-sampler literature designs to shape the representation\([He et al\., 2020](https://arxiv.org/html/2610.12707#bib.bib15);[Robinson et al\., 2021](https://arxiv.org/html/2610.12707#bib.bib16);[Kalantidis et al\., 2020](https://arxiv.org/html/2610.12707#bib.bib17);[Wang and Isola, 2020](https://arxiv.org/html/2610.12707#bib.bib18)\), and in vision\-language models as word\-order and attribute swaps of the true caption\([Yuksekgonul et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib62);[Hsieh et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib63)\)\. With frozen features those choices act only through the objective\. Closest to our objective is[Fan et al\. \(2026\)](https://arxiv.org/html/2610.12707#bib.bib29), a label\-free in\-batch Bradley–Terry reward on text with web continuations as the negative process; their invariance theorem gives a nuisance’s coefficient in the unrestricted optimum, which[Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)restates for our matrix loss\.

Best\-of\-NNand learning from self\-generated data\.A reward built from that ratio is then used, like a verifier\([Cobbe et al\., 2021](https://arxiv.org/html/2610.12707#bib.bib50);[Uesato et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib68);[Lightman et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib67)\), under optimization pressure, and proxy rewards degrade under it\([Gao et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib27);[Coste et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib73);[Stiennon et al\., 2020](https://arxiv.org/html/2610.12707#bib.bib25)\)\. We locate an earlier failure, before any pressure is applied, where the proxy’s benchmark score does not predict its Best\-of\-NNvalue\([Beirami et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib72)\)against self\-consistency\([Wang et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib26)\)\. Expert iteration and STaR\([Anthony et al\., 2017](https://arxiv.org/html/2610.12707#bib.bib24);[Zelikman et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib23);[Singh et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib71)\)improve a policy on its own verified outputs, and GKD\([Agarwal et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib22)\)diagnoses the train–inference mismatch on the policy side; ours is the reward\-side analogue\.

## 5Concluding Remarks

Frozen vision\-language representations support strong reward modeling with only a linear head and caption perturbations, but probe depth and training comparisons determine which signals the head extracts\. Similar benchmark scores can conceal different specializations, and higher preference accuracy need not yield better Best\-of\-NNselection\. Retraining on verified policy outputs improves caption selection while lowering benchmark accuracy, so the supervision that helps deployment is drawn from the policy’s own errors\. Anyone choosing a reward model should test how well it picks the best of their own policy’s samples, and a paper that reports one should state which pairs it was trained on\.

## Acknowledgement

S\.K\. acknowledges the support from the National Science Foundation Grant under award IIS 2229881; H\.Z\. and S\.K\. acknowledge the Chan Zuckerberg Initiative Foundation for establishing the Kempner Institute for the Study of Natural and Artificial Intelligence\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InICLR,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Alain and Bengio \(2017\)G\. Alain and Y\. BengioUnderstanding intermediate layers using linear classifier probes\.InInternational Conference on Learning Representations \(ICLR\) Workshop,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p2.1)\.
- Anthonyet al\.\(2017\)T\. Anthony, Z\. Tian, and D\. BarberThinking fast and slow with deep learning and tree search\.InNeurIPS,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Baiet al\.\(2022\)Y\. Bai, S\. Kadavath, S\. Kundu, A\. Askell, J\. Kernion, A\. Jones, A\. Chen, A\. Goldie, A\. Mirhoseini, C\. McKinnon,et al\.Constitutional ai: harmlessness from ai feedback\.arXiv preprint arXiv:2212\.08073\.Cited by:[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.4),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Beiramiet al\.\(2025\)A\. Beirami, A\. Agarwal, J\. Berant, A\. D’Amour, J\. Eisenstein, C\. Nagpal, and A\. T\. SureshTheoretical guarantees on the best\-of\-nnalignment policy\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p2.1)\.
- Burnset al\.\(2023\)C\. Burns, H\. Ye, D\. Klein, and J\. SteinhardtDiscovering latent knowledge in language models without supervision\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p2.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Costeet al\.\(2024\)T\. Coste, U\. Anwar, R\. Kirk, and D\. KruegerReward model ensembles help mitigate overoptimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Deitkeet al\.\(2024\)M\. Deitke, C\. Clark, S\. Lee,et al\.Molmo and pixmo: open weights and open data for state\-of\-the\-art multimodal models\.arXiv preprint arXiv:2409\.17146\.Cited by:[Appendix B](https://arxiv.org/html/2610.12707#A2.p1.1),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p1.1)\.
- Fanet al\.\(2026\)J\. Fan, Y\. Li, Z\. Qi, D\. Zhang, K\. Brantley, S\. M\. Kakade, and H\. ZhangScaling reward modeling without human supervision\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix G](https://arxiv.org/html/2610.12707#A7.p2.1),[Appendix I](https://arxiv.org/html/2610.12707#A9.p2.1),[§1](https://arxiv.org/html/2610.12707#S1.p2.1),[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Fricket al\.\(2025\)E\. Frick, T\. Li, C\. Chen, W\. Chiang, A\. N\. Angelopoulos, J\. Jiao, B\. Zhu, J\. E\. Gonzalez, and I\. StoicaHow to evaluate reward models for RLHF\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InICML,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Ghoshet al\.\(2023\)D\. Ghosh, H\. Hajishirzi, and L\. SchmidtGenEval: an object\-focused framework for evaluating text\-to\-image alignment\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks,Cited by:[Figure 5](https://arxiv.org/html/2610.12707#S3.F5),[§3\.4](https://arxiv.org/html/2610.12707#S3.SS4.SSS0.Px2.p1.1)\.
- Gutmann and Hyvärinen \(2010\)M\. Gutmann and A\. HyvärinenNoise\-contrastive estimation: a new estimation principle for unnormalized statistical models\.InAISTATS,Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.12707#S2.SS2.p1.1),[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Heet al\.\(2020\)K\. He, H\. Fan, Y\. Wu, S\. Xie, and R\. GirshickMomentum contrast for unsupervised visual representation learning\.InCVPR,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Hendrycks and Dietterich \(2019\)D\. Hendrycks and T\. DietterichBenchmarking neural network robustness to common corruptions and perturbations\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Figure 6](https://arxiv.org/html/2610.12707#A4.F6),[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.4)\.
- Hsiehet al\.\(2023\)C\. Hsieh, J\. Zhang, Z\. Ma, A\. Kembhavi, and R\. KrishnaSugarCrepe: fixing hackable benchmarks for vision\-language compositionality\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Appendix G](https://arxiv.org/html/2610.12707#A7.p3.1)\.
- Huet al\.\(2025\)Y\. Hu, R\. Askari\-Hemmat, M\. Hall, E\. Dinan, L\. Zettlemoyer, and M\. GhazvininejadMultimodal rewardbench 2: evaluating omni reward models for interleaved text and image\.arXiv preprint arXiv:2512\.16899\.Cited by:[Appendix B](https://arxiv.org/html/2610.12707#A2.p1.1),[§1](https://arxiv.org/html/2610.12707#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p2.1),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Kalantidiset al\.\(2020\)Y\. Kalantidis, M\. B\. Sariyildiz, N\. Pion, P\. Weinzaepfel, and D\. LarlusHard negative mixing for contrastive learning\.InNeurIPS,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Lambertet al\.\(2025\)N\. Lambert, V\. Pyatkin, J\. Morrison,et al\.RewardBench: evaluating reward models for language modeling\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 1755–1797\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- LeCunet al\.\(2006\)Y\. LeCun, S\. Chopra, R\. Hadsell, M\. Ranzato, and F\. J\. HuangA tutorial on energy\-based learning\.InPredicting Structured Data,Cited by:[§2\.2](https://arxiv.org/html/2610.12707#S2.SS2.p1.1)\.
- Leeet al\.\(2024\)H\. Lee, S\. Phatale, H\. Mansoor, T\. Mesnard, J\. Ferret, K\. Lu, C\. Bishop, E\. Hall, V\. Carbune, A\. Rastogi, and S\. PrakashRLAIF vs\. rlhf: scaling reinforcement learning from human feedback with ai feedback\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.4),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Liet al\.\(2025\)L\. Li, Y\. Wei, Z\. Xie,et al\.VL\-rewardbench: a challenging benchmark for vision\-language generative reward models\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Appendix B](https://arxiv.org/html/2610.12707#A2.p1.1),[§1](https://arxiv.org/html/2610.12707#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p2.1),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Liet al\.\(2024\)L\. Li, Z\. Xie, M\. Li, S\. Chen, P\. Wang, L\. Chen, Y\. Yang, B\. Wang, L\. Kong, and Q\. LiuVLFeedback: a large\-scale AI feedback dataset for large vision\-language models alignment\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Liet al\.\(2023\)Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. WenEvaluating object hallucination in large vision\-language models\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Appendix A](https://arxiv.org/html/2610.12707#A1.p3.1)\.
- Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft coco: common objects in context\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p3.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[Appendix J](https://arxiv.org/html/2610.12707#A10.SS0.SSS0.Px1.p4.1)\.
- Luet al\.\(2024\)P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. GaoMathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p3.1)\.
- Ma and Collins \(2018\)Z\. Ma and M\. CollinsNoise contrastive estimation and negative sampling for conditional models: consistency and statistical efficiency\.InEMNLP,Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.12707#S2.SS2.p1.1)\.
- Maliket al\.\(2026\)S\. Malik, V\. Pyatkin, S\. Land, J\. Morrison, N\. A\. Smith, H\. Hajishirzi, and N\. LambertRewardBench 2: advancing reward model evaluation\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Mindereret al\.\(2023\)M\. Minderer, A\. Gritsenko, and N\. HoulsbyScaling open\-vocabulary object detection\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Figure 5](https://arxiv.org/html/2610.12707#S3.F5),[§3\.4](https://arxiv.org/html/2610.12707#S3.SS4.SSS0.Px2.p1.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p2.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p1.1)\.
- Qiet al\.\(2026\)Z\. Qi, F\. Nie, A\. Alahi, J\. Zou, H\. Lakkaraju, Y\. Du, E\. Xing, S\. Kakade, and H\. ZhangEvoLM: in search of lost training dynamics for language model reasoning\.Advances in Neural Information Processing Systems38,pp\. 82906–82937\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Qwen Team \(2025a\)Qwen TeamQwen\-image technical report\.arXiv preprint arXiv:2508\.02324\.Cited by:[Appendix G](https://arxiv.org/html/2610.12707#A7.p3.1),[§3\.4](https://arxiv.org/html/2610.12707#S3.SS4.SSS0.Px2.p1.1)\.
- Qwen Team \(2025b\)Qwen TeamQwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Note:Qwen3\-VL\-2B/8B\-Instruct vision\-language modelsCited by:[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§3\.3](https://arxiv.org/html/2610.12707#S3.SS3.SSS0.Px1.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix G](https://arxiv.org/html/2610.12707#A7.p1.1),[§1](https://arxiv.org/html/2610.12707#S1.p2.1),[§1](https://arxiv.org/html/2610.12707#S1.p4.1)\.
- Robinsonet al\.\(2021\)J\. Robinson, C\. Chuang, S\. Sra, and S\. JegelkaContrastive learning with hard negative samples\.InICLR,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Rohrbachet al\.\(2018\)A\. Rohrbach, L\. A\. Hendricks, K\. Burns, T\. Darrell, and K\. SaenkoObject hallucination in image captioning\.InEMNLP,Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p4.1),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p3.1)\.
- Singhet al\.\(2024\)A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu,et al\.Beyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research \(TMLR\)\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Singhalet al\.\(2024\)P\. Singhal, T\. Goyal, J\. Xu, and G\. DurrettA long way to go: investigating length correlations in rlhf\.arXiv preprint arXiv:2310\.03716\.Cited by:[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.4),[§3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1.p1.1)\.
- Skeanet al\.\(2025\)O\. Skean, M\. R\. Arefin, D\. Zhao, N\. Patel, J\. Naghiyev, Y\. LeCun, and R\. Shwartz\-ZivLayer by layer: uncovering hidden representations in language models\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p2.1)\.
- Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. ChristianoLearning to summarize from human feedback\.InNeurIPS,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Sunet al\.\(2025\)H\. Sun, Y\. Shen, and J\. TonRethinking reward modeling in preference\-based large language model alignment\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2411\.04991Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p2.1)\.
- Sunet al\.\(2024\)Z\. Sun, S\. Shen, S\. Cao, H\. Liu, C\. Li, Y\. Shen, C\. Gan, L\. Gui, Y\. Wang, Y\. Yang, K\. Keutzer, and T\. DarrellAligning large multimodal models with factually augmented RLHF\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 13088–13110\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Tenneyet al\.\(2019\)I\. Tenney, D\. Das, and E\. PavlickBERT rediscovers the classical nlp pipeline\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p2.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- van den Oordet al\.\(2018\)A\. van den Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§2\.2](https://arxiv.org/html/2610.12707#S2.SS2.p1.1),[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Wallaceet al\.\(2024\)B\. Wallace, M\. Dang, R\. Rafailov, L\. Zhou, A\. Lou, S\. Purushwalkam, S\. Ermon, C\. Xiong, S\. Joty, and N\. NaikDiffusion model alignment using direct preference optimization\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Appendix G](https://arxiv.org/html/2610.12707#A7.p3.1)\.
- Wang and Isola \(2020\)T\. Wang and P\. IsolaUnderstanding contrastive representation learning through alignment and uniformity on the hypersphere\.InICML,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.InICLR,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Wanget al\.\(2025\)Y\. Wang, Y\. Zang, H\. Li, C\. Jin, and J\. WangUnified reward model for multimodal understanding and generation\.arXiv preprint arXiv:2503\.05236\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Wenet al\.\(2025\)X\. Wen, J\. Lou, Y\. Lu, H\. Lin, X\. Yu, X\. Lu, B\. He, X\. Han, D\. Zhang, and L\. SunRethinking reward model evaluation: are we barking up the wrong tree?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Xie and Li \(2024\)S\. Xie and Z\. LiImplicit bias of AdamW:ℓ∞\\ell\_\{\\infty\}\-norm constrained optimization\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix J](https://arxiv.org/html/2610.12707#A10.SS0.SSS0.Px1.p4.2),[§2\.2](https://arxiv.org/html/2610.12707#S2.SS2.p1.1)\.
- Xionget al\.\(2025\)T\. Xiong, X\. Wang, D\. Guo, Q\. Ye, H\. Fan, Q\. Gu, H\. Huang, and C\. LiLLaVA\-Critic: learning to evaluate multimodal models\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Yasunagaet al\.\(2025\)M\. Yasunaga, L\. Zettlemoyer, and M\. GhazvininejadMultimodal rewardbench: holistic evaluation of reward models for vision language models\.arXiv preprint arXiv:2502\.14191\.Cited by:[Appendix B](https://arxiv.org/html/2610.12707#A2.p1.1),[§1](https://arxiv.org/html/2610.12707#S1.p1.1),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p2.1),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Yuet al\.\(2024\)T\. Yu, Y\. Yao, H\. Zhang, T\. He, Y\. Han, G\. Cui, J\. Hu, Z\. Liu, H\. Zheng, M\. Sun, and T\. ChuaRLHF\-v: towards trustworthy mllms via behavior alignment from fine\-grained correctional human feedback\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Yuet al\.\(2025\)T\. Yu, H\. Zhang, Y\. Yao,et al\.RLAIF\-V: open\-source AI feedback leads to super GPT\-4V trustworthiness\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[Appendix B](https://arxiv.org/html/2610.12707#A2.p1.1),[§1](https://arxiv.org/html/2610.12707#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12707#S2.SS1.p1.4),[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p1.1),[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Yueet al\.\(2024\)X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§2\.3](https://arxiv.org/html/2610.12707#S2.SS3.p3.1)\.
- Yuksekgonulet al\.\(2023\)M\. Yuksekgonul, F\. Bianchi, P\. Kalluri, D\. Jurafsky, and J\. ZouWhen and why vision\-language models behave like bags\-of\-words, and what to do about it?\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p3.1)\.
- Zanget al\.\(2025\)Y\. Zang, X\. Dong, P\. Zhang, Y\. Cao, Z\. Liu, S\. Ding, S\. Wu, Y\. Ma, H\. Duan, W\. Zhang, K\. Chen, D\. Lin, and J\. WangInternLM\-xcomposer2\.5\-reward: a simple yet effective multi\-modal reward model\.InFindings of the Association for Computational Linguistics \(ACL\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. GoodmanSTaR: bootstrapping reasoning with reasoning\.InNeurIPS,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p4.1)\.
- Zhang \(2026\)D\. ZhangWhat Is RLCD? The Secret Behind Jev\.Note:Blog post,[https://di\-zhang\-llm\.github\.io/blog/what\-is\-rlcd\-the\-secret\-behind\-jev](https://di-zhang-llm.github.io/blog/what-is-rlcd-the-secret-behind-jev)Posted September 21, 2026; accessed September 25, 2026Cited by:[§1](https://arxiv.org/html/2610.12707#S1.p2.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, T\. Yu, H\. Tian, C\. Fu, P\. Li, J\. Zeng, W\. Xie, Y\. Shi, H\. Zhang, J\. Wu,et al\.MM\-RLHF: the next step forward in multimodal LLM alignment\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Zhaoet al\.\(2023a\)Y\. Zhao, A\. Gu, R\. Varma,et al\.PyTorch fsdp: experiences on scaling fully sharded data parallel\.Proceedings of the VLDB Endowment\.Cited by:[Appendix D](https://arxiv.org/html/2610.12707#A4.p2.1)\.
- Zhaoet al\.\(2023b\)Z\. Zhao, B\. Wang, L\. Ouyang, X\. Dong, J\. Wang, and C\. HeBeyond hallucinations: enhancing LVLMs through hallucination\-aware direct preference optimization\.arXiv preprint arXiv:2311\.16839\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.
- Zhouet al\.\(2025\)S\. Zhou, R\. Zhang, H\. Zhu, B\. Kveton, Y\. Zhou, J\. Gu, J\. Chen, and C\. ChenMultimodal llms as customized reward models for text\-to\-image generation\.InIEEE/CVF International Conference on Computer Vision \(ICCV\),Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p2.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, C\. Cui, R\. Rafailov, C\. Finn, and H\. YaoAligning modalities in vision large language models via preference fine\-tuning\.arXiv preprint arXiv:2402\.11411\.Cited by:[§4](https://arxiv.org/html/2610.12707#S4.p1.1)\.

###### Appendix Contents

1. [1Introduction](https://arxiv.org/html/2610.12707#S1)
2. [2Methodology](https://arxiv.org/html/2610.12707#S2)1. [2\.1Algorithm](https://arxiv.org/html/2610.12707#S2.SS1) 2. [2\.2What the Objective Identifies](https://arxiv.org/html/2610.12707#S2.SS2) 3. [2\.3Experimental Setup](https://arxiv.org/html/2610.12707#S2.SS3)
3. [3Experiments](https://arxiv.org/html/2610.12707#S3)1. [3\.1Two Choices Set the Accuracy: The Probe Layer and the Sampler](https://arxiv.org/html/2610.12707#S3.SS1) 2. [3\.2One Untuned Recipe Reaches GPT\-4o Across Backbones and Benchmarks](https://arxiv.org/html/2610.12707#S3.SS2) 3. [3\.3Two Probes, One Score, Two Rewards](https://arxiv.org/html/2610.12707#S3.SS3) 4. [3\.4Learning from the Policy’s Own Errors: Better Deployment, Lower Benchmarks](https://arxiv.org/html/2610.12707#S3.SS4)
4. [4Related Work](https://arxiv.org/html/2610.12707#S4)
5. [5Concluding Remarks](https://arxiv.org/html/2610.12707#S5)
6. [References](https://arxiv.org/html/2610.12707#bib)
7. [ALimitations and Broader Impacts](https://arxiv.org/html/2610.12707#A1)
8. [BReproducibility](https://arxiv.org/html/2610.12707#A2)
9. [CEvaluation Protocol](https://arxiv.org/html/2610.12707#A3)
10. [DNegative Mining and Probe Depth](https://arxiv.org/html/2610.12707#A4)
11. [EExtended Results for Reward\-Model Training](https://arxiv.org/html/2610.12707#A5)
12. [FExtended Results for Best\-of\-NN](https://arxiv.org/html/2610.12707#A6)
13. [GActor Training](https://arxiv.org/html/2610.12707#A7)
14. [HCLIP Studies](https://arxiv.org/html/2610.12707#A8)
15. [IReading the Two Statements](https://arxiv.org/html/2610.12707#A9)1. [I\.1What Constructed Comparisons Supervise](https://arxiv.org/html/2610.12707#A9.SS1) 2. [I\.2Independent Negatives: Density Ratios and PMI](https://arxiv.org/html/2610.12707#A9.SS2) 3. [I\.3When Perturbations Expose an Additional Quality Signal](https://arxiv.org/html/2610.12707#A9.SS3)
16. [JProofs](https://arxiv.org/html/2610.12707#A10)
17. [KCompute Resources](https://arxiv.org/html/2610.12707#A11)

## Appendix ALimitations and Broader Impacts

Our approach lowers the additional training and preference\-annotation requirements for multimodal reward modeling by learning a small scoring head over a frozen backbone\. This may make task\-specific reward models more accessible to researchers and practitioners with limited annotation budgets, while retaining the inference cost of the underlying backbone\. Constructing training comparisons explicitly also makes the intended supervision easier to inspect and revise\.

The broader implication concerns how reward models are evaluated and selected\. Our experiments show that higher preference\-benchmark accuracy need not yield better selection of a policy’s outputs\. Reporting the training comparison distribution alongside deployment\-specific evaluations can therefore support more informed model selection\. Conversely, using benchmark performance alone to justify deployment risks overlooking systematic errors and shortcuts, including response\-length biases\.

These benefits have a limited demonstrated scope\. Our experiments cover two backbone families, and verifier\-based adaptation is evaluated for one round\. Captioning improvements are measured with CHAIR, which also supplies the training labels; they establish fewer absent COCO\-object mentions rather than general factuality or preserved informativeness, and a CHAIR\-independent metric such as POPE\([Li et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib54)\)would be needed to settle the latter\. Best\-of\-NNcells are single\-seed, the hard\-QA set was written and verified by one LLM family, and our Multimodal RewardBench overall omits the withheld toxicity split \([Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)\)\. The method does not establish safety or fairness capabilities, and broader deployment requires independent evaluation of output quality, coverage, and task\-specific failure modes\.

## Appendix BReproducibility

The recipe is fully specified by[Eqs\.1](https://arxiv.org/html/2610.12707#S2.E1),[2](https://arxiv.org/html/2610.12707#S2.E2)and[3](https://arxiv.org/html/2610.12707#S2.E3)and the hyper\-parameters of[Section2\.3](https://arxiv.org/html/2610.12707#S2.SS3)and[AppendixD](https://arxiv.org/html/2610.12707#A4): a frozen Qwen3\-VL checkpoint, a probe layer, mean pooling over response tokens, and one bias\-free linear head trained with AdamW for 200 steps\. Every input is public\. Positives are PixMo\-Cap captions\([Deitke et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib12)\); negatives are string edits of those captions with no model in the loop, so the 22k\-pair bank rebuilds in 10 seconds on one CPU core from the operator definitions in[AppendixD](https://arxiv.org/html/2610.12707#A4)\. The curated comparison set is public RLAIF\-V\([Yu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib11)\), and all three evaluation benchmarks\([Yasunaga et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib1);[Li et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib2);[Hu et al\., 2025](https://arxiv.org/html/2610.12707#bib.bib31)\)are used as released, with the pair counts and the scoring construction stated per benchmark in[Section2\.3](https://arxiv.org/html/2610.12707#S2.SS3)\. Contamination is checked by a 256\-bit difference hash of every training image against every evaluation image, counting a pair as a near\-duplicate at≤10\\leq 10differing bits, plus exact prompt matching\. Against Multimodal RewardBench the perturbation set shares 5 near\-duplicate images of 4,711 \(0\.1%\) and no prompt; against VL\-RewardBench, the 1,000 MMRB2 prompt images and the 1,000 COCO Best\-of\-NNimages it shares none\. The image arm renders 1,212 PixMo\-derived prompts with Qwen\-Image, so its training images are generated and its prompts are not those of MMRB2\. The same audit exposes the curated baseline: RLAIF\-V pairs overlap VL\-RewardBench on 326 of 1,247 rows and Multimodal RewardBench on 18, which is why every RLAIF\-V\-trained row also reports the decontaminated subset\. Source for pair construction, training, every selector in the Best\-of\-NNtables and the plotting scripts behind every figure will be released with the paper\.

## Appendix CEvaluation Protocol

Multimodal RewardBench releases 4,711 of its 5,211 pairs, the toxicity split being withheld, in eight categories; our overall number is therefore comparable to published ones only through the recomputation of[Table1](https://arxiv.org/html/2610.12707#S3.T1)\. VL\-RewardBench has 1,247 pairs \(general 181, hallucination 749, reasoning 317\), of which 326 overlap the RLAIF\-V training set, so we also score anything trained on RLAIF\-V on the 921 clean rows\. On the 326 overlapping rows the RLAIF\-V fine\-tunes score \.80 against \.60 to \.64 for models that never saw the data; on the clean rows their lead disappears\. MMRB2 has 1,000 judge\-labelled pairs per task \(text\-to\-image, editing, interleaved generation, reasoning; 996 reasoning pairs are scoreable\)\. We score it with the prompt in the user turn and the response, images included, in the assistant turn, the construction the heads were trained on\. A scalar reward is order\-invariant, so its accuracy equals MMRB2’s dual\-order accuracy; exact ties count 0\.5\. We also report reasoning on the 350\-pair text\-only\-output subset, the convention that benchmark’s paper uses for scalar reward models\. On the four\-task average under this construction the 32B probe scores 59\.9 and the same backbone used as a generative judge 64\.6; GPT\-4o as a judge scores 59\.7\. Scoring prompt and response in one user turn instead costs the text heads 2 to 3 points on reasoning and 5 to 9 on its text\-only subset\. The GPT\-4o margins in[Table1](https://arxiv.org/html/2610.12707#S3.T1)are quoted as unpaired standard errors because the benchmark papers do not release per\-item predictions, so a paired interval is not available\. Shaded bands in every figure are 75% binomial intervals of the plotted accuracy over its evaluation set; seed\-to\-seed spread of the heads is narrower than the line width\. Eight seeds of the layer\-sweep heads put the safety/bias column of[Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)at\.967±\.007\.967\\pm\.007at 2B and\.569±\.013\.569\\pm\.013at 8B, where the correlation between head score and response length is−0\.57\-0\.57at 2B and\+0\.44\+0\.44at 8B\. The Best\-of\-NNintervals of[Section3\.3](https://arxiv.org/html/2610.12707#S3.SS3.SSS0.Px2)are over prompts at one selector seed\.

The Best\-of\-NNprompts are 500 each of COCO captioning \(two prompt sets, COCO and COCO\-2\), MathVista and MMMU, plus 500 Claude\-written hard questions on COCO images, of which 241 are counting or spatial; candidates are sampled at temperature 1\.0 and top\-pp0\.95, plus the greedy decode\. The mid\-training runs of[Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)continue from the mined\-negative probe on 3,462 verified pairs harvested from those pools \(MathVista 1,352, MMMU 1,072, COCO 511, COCO\-2 527\), with leave\-one\-task\-out runs that drop every COCO\-2 or every MMMU pair, and a non\-convex check that unfreezes the backbone on 15k RLAIF\-V pairs from either checkpoint\. Heads train with AdamW, learning rate10−310^\{\-3\}, weight decay0\.010\.01,β=0\.1\\beta\{=\}0\.1, micro\-batch 8 and effective batch 256 or 1,024 as stated; the backbones have 28, 36, 36 and 64 layers\.

## Appendix DNegative Mining and Probe Depth

Two pair banks appear in the benchmark tables: the 2B and 8B rows of[Table1](https://arxiv.org/html/2610.12707#S3.T1)use the original 60k\-pair bank, the 4B and 32B rows and every ablation the rebuilt 15k\-pair bank, which costs the 2B probe 2 to 3 points \(\.708 to \.682 on Multimodal RewardBench\), so the 32B row leads from the smaller bank\. Perturbation density does not move either benchmark: one, three or all editable spans stay within \.008 on both benchmarks \([Table4](https://arxiv.org/html/2610.12707#A4.T4)\) while the training task gets much easier\. The three token\-edit heads beat the mixed\-operator recipe on VL\-RewardBench by 2\.8 to 3\.4 points at equal Multimodal RewardBench accuracy \([Table3](https://arxiv.org/html/2610.12707#A4.T3)\)\. That comparison also drops the cross\-image and the length\-changing operators, so it does not isolate the mix\.

Table 2:Every ablation and analysis of the paper on one backbone \(Qwen3\-VL\-4B\)\.Frozen linear head, mean pooling, probe on layerN−6N\{\-\}6unless stated, 200 steps of 256 pairs, single seed, no re\-tuning between benchmarks\. MMRB1 = Multimodal RewardBench \(4,711 pairs;*trainable*= the five categories without safety/bias and VQA\); VL\-RB = VL\-RewardBench \(1,247 pairs, hallucination split 749\)\. TheΔ\\Delta\-CLIP heads are 5k\-pair quartiles of the same perturbations split by whether CLIP can see the edit; the verified heads train on 2,400 verifier\-minted pairs from a fresh head or from the mined\-negative head; the base judge scores by response log\-likelihood with no training\.![Refer to caption](https://arxiv.org/html/2610.12707v1/corruption_examples.png)Figure 6:Negative mining by perturbation, laid out after ImageNet\-C\([Hendrycks and Dietterich, 2019](https://arxiv.org/html/2610.12707#bib.bib55)\)\.\(a\)A PixMo image and three operators applied to its human caption: attribute swaps every colour, number and spatial word; shuffle reorders and drops sentences; splice continues the true caption with another image’s\. Red marks the edit, orange the foreign text\.\(b\)The image\-response arm perturbs the prompt instead and renders both prompts with Qwen\-Image under the same seed, so the two prompts differ only in the edited span\.Table 3:Negative\-mining ablation in both modalities\.Top: text responses, 2B backbone\. The frozen\-head rows use 15k pairs atN−4N\{\-\}4on the official benchmarks; the two full\-fine\-tune rows are the June series that first separated the negative types \(overall Multimodal RewardBench; their safety/bias column reads \.224 and \.451, the length shortcut\)\. Bottom: image responses, 2B image\-trained head atN−12N\{\-\}12on 1,091 same\-seed prompt pairs;*held\-out*is accuracy on the head’s own negative type, MMRB2 text\-to\-image is judge agreement \(label prior 53\.2\)\. The*by kind*row splits held\-out accuracy of the combined head by the planted edit\.
Table 4:Perturbation density,k=1k\{=\}1/ 3 / all editable spans per rejected caption \(all==13\.6 edits on average\), 21,521 identical images and captions per setting, token substitutions only\. Frozen 2BN−4N\{\-\}4head, 200 steps of 1,024, single seed\. The training task gets much easier \(val \.82→\\to\.996\) and neither benchmark moves\.Table 5:Mined negatives against curated RLAIF\-V pairs at a matched pair budget,2B backbone, 15k pairs per setting\. Frozen heads:N−4N\{\-\}4, 200 steps of 1,024 pairs\. Full fine\-tune: lr10−510^\{\-5\}, 400 steps of 256, vision tower frozen\. The*tokens*column is the response\-token count of the 15k pairs \(both sides; Qwen3\-VL tokenizer\): perturbation captions are longer than RLAIF\-V responses \(204 against 86 tokens per side\), so the RLAIF\-V heads see fewer tokens at the same pair count\. The*val*column is in\-distribution pair accuracy on each setting’s own held\-out pairs\. 326 of the 1,247 VL\-RewardBench rows overlap the RLAIF\-V training set, so RLAIF\-V models are also read on the 921 clean rows; mined\-negative pairs overlap none\. Single seed; the comparison is matched in pairs and not in tokens\.[Figure3](https://arxiv.org/html/2610.12707#S3.F3)gives the depth sweep per model,[Table3](https://arxiv.org/html/2610.12707#A4.T3)the operator ablation in both modalities,[Table4](https://arxiv.org/html/2610.12707#A4.T4)the density variants and[Table5](https://arxiv.org/html/2610.12707#A4.T5)the matched\-budget comparison against RLAIF\-V\. On that matched budget the mined\-negative head reads \.574 to \.584 trainable on Multimodal RewardBench against \.544 for RLAIF\-V and \.679 to \.709 on VL\-RewardBench against \.660, with hallucination \.769 to \.781 against \.737; its in\-distribution validation accuracy is \.82 to \.92 against \.55 for the curated pairs\. The comparison is matched in pairs and not in tokens, since perturbation captions run 2\.4 times longer, which leaves the RLAIF\-V head 2\.6M response tokens against 6\.1M\. Unfreezing the backbone on the same 15k real pairs gains 11 points overall and 7 trainable, to \.613, which puts it above the mined\-negative probe on the trainable categories and level with it on decontaminated VL\-RewardBench \(\.727 against \.747\)\. Perturbation operators act on one caption at a time: a near\-miss substitution of a colour, number or spatial word; a splice of another caption’s tail; a clause shuffle; and, for 15% of pairs, the caption of another image\. In the 15k\-pair bank the operators and their share are near\-miss substitution 50%, attribute substitution 15%, splice 17%, shuffle 10% and cross\-image 8%\. Measured as the ratio of rejected to chosen length in words, the two substitutions leave length unchanged \(1\.00\), splice shortens to 0\.82, shuffle to 0\.69 \(it drops about 30% of sentences\) and cross\-image captions are 1\.10 times as long, so across the bank a negative is about 5% shorter than its positive\. On GLM\-4\.1V\-9B the same recipe at the last layer scores \.628 on Multimodal RewardBench and \.598 on VL\-RewardBench against \.692 and \.602 atN−12N\{\-\}12\([Table1](https://arxiv.org/html/2610.12707#S3.T1)\); the untrained judge scores \.565 and \.570\. Density variants perturb one, three or all editable spans \(13\.6 on average\) of the same 21,521 captions\. Heads train with AdamW, learning rate10−310^\{\-3\}, weight decay0\.010\.01,β=0\.1\\beta\{=\}0\.1, micro\-batch 8, effective batch 256 or 1,024 as stated, 200 steps, on precomputed vision\-tower outputs; the 32B confirm run used FSDP\([Zhao et al\., 2023a](https://arxiv.org/html/2610.12707#bib.bib13)\)over eight GPUs with 1,024 effective\. The layer sweep trains a linear head on the precomputed hidden states of each layer for 8k pairs and five seeds; it ranks depths within a model and nothing else, and it misleads in two documented ways\. Last\-token pooling ranks first in the sweep at 4B, 8B and 32B and loses 7\.5 points to mean pooling in the one real run\. Sweep heads trained under a different regime than the real pipeline disagree with it by several points, in either direction\.

## Appendix EExtended Results for Reward\-Model Training

Figure 7:Three reward benchmarks against backbone size, by component \(shaded: confidence bands\)\. The linear probe atN−kN\{\-\}kimproves with size on all three and beats the last\-layer probe at every size\. The full fine\-tune matches it on MMRB1 only from 8B up and trails it on VL\-RewardBench at 32B \(early\-stopped at step 150 of 400\)\. The RLAIF\-V probe trails on MMRB1 everywhere and leads on VL\-RewardBench only at 32B, where the two share source images\.[Figure7](https://arxiv.org/html/2610.12707#A5.F7)separates the components behind[Table1](https://arxiv.org/html/2610.12707#S3.T1)at each backbone size\.

## Appendix FExtended Results for Best\-of\-NN

Figure 8:Accuracy of the selected candidate againstNNon 500 hard questions and the 241\-question counting/spatial subset, for the 2B and 8B generators, with every training\-free control on every panel\. Where the generator is confidently wrong \(8B\) every external scorer beats majority vote; on the 2B pool probe, log\-likelihood and vote are within noise\. probe\-8B omitted \(it tracks probe\-2B\); its numbers are in[AppendixF](https://arxiv.org/html/2610.12707#A6)\.Table 6:Best\-of\-16 on hard verifiable QA\(500 questions on COCO images, and the 241\-question counting/spatial subset\), accuracy of the selected candidate for each selector\.*greedy*isN=1N\{=\}1;*oracle*is pass@16\. Paired bootstrap over prompts; single seed per selector\. Where the 8B generator is confidently wrong \(mean top\-share \.85\) every external scorer beats majority vote\. On the 2B pool, the probes and log\-likelihood are within noise of each other and of the vote\.Table 7:One turn of the loop, 2B\.Top: CHAIRs atN=16N\{=\}16per captioning cell \(lower is better\);*verified*is the head trained on 3,462 verifier\-minted pairs,*held\-out*the same recipe with every COCO\-2 pair excluded, scored on COCO\-2\. Pooled paired bootstrap against greedy:−5\.1\-5\.1CHAIR points, 95% CI−7\.1\-7\.1to−3\.1\-3\.1; held\-out−4\.1\-4\.1, 95% CI−6\.9\-6\.9to−1\.1\-1\.1\. Bottom: the same heads as judges on the two unseen reward benchmarks, against the mined\-negative probe they replace \(the 15k\-pair rebuild that scores every Best\-of\-NNcell\)\. The best ranker is the worst recent judge\.
[Figure8](https://arxiv.org/html/2610.12707#A6.F8)and[Table6](https://arxiv.org/html/2610.12707#A6.T6)give the hard\-QA ladder with every control on every panel and[Table7](https://arxiv.org/html/2610.12707#A6.T7)the loop turn per captioning cell\. Initializing the verified head from the mined\-negative probe matters only when the pair budget is too small to reach the optimum\. At 2B a fresh head and a continued one score within 0\.3 points of each other everywhere \([Table7](https://arxiv.org/html/2610.12707#A6.T7)\); at 4B, where 2,400 pairs do not reach the optimum, the continued head is 7 points ahead \([Table2](https://arxiv.org/html/2610.12707#A4.T2)\)\. A planted\-decoy test \(3\.4 LLM\-written single\-error decoys and 4 template perturbations per captioning pool\) shows that the mined\-negative feature fires inside a real pool: every scorer ranks a source above its own one\-edit decoy at least 93% of the time\. The margin, however, is 0\.3 to 1\.1 within\-pool standard deviations, and CHAIRs of the pick does not move\. The probe therefore detects that an edited sentence is worse than its source but cannot tell which of 16 real captions is the most truthful\. Truth\-preserving edits separate the signals: the probes do not prefer the source over a true edit \(probe\-2B \.512, probe\-8B \.291\) while log\-likelihood prefers it 91% of the time\.

#### Model\-written negatives\.

Two 2B heads were trained with the policy’s own captions in the pair instead of a string edit, same recipe otherwise\. Self\-corruption \(chosen: the policy’s caption; rejected: its perturbation\) ties greedy and random in all four captioning cells and scores \.689 on Multimodal RewardBench, the same as the off\-policy head\. Human\-versus\-policy \(chosen: the PixMo caption; rejected: the policy’s caption for the same image\) scores \.360 on Multimodal RewardBench, below chance, and picks a more hallucinated caption than random in all four cells \(CHAIRs@16 \.422 / \.432 / \.414 / \.422 against random \.370 / \.384 / \.406 / \.402\)\. This head learned to tell a human author from a model; on a benchmark where both responses are model\-written, that direction anti\-correlates with the label\.

## Appendix GActor Training

We train the 2B policy on its own selected candidates against a size\-matched random\-selection control\. DPO\([Rafailov et al\., 2023](https://arxiv.org/html/2610.12707#bib.bib41)\)usesβ=0\.1\\beta\{=\}0\.1, learning rate10−610^\{\-6\}, two epochs, and three corpora from the same 500 prompts per task: verifier\-minted pairs, probe\-8B argmax\-versus\-argmin pairs, and random pairs as the control\. Policies are re\-evaluated greedily on all four tasks\.

The actor half of the turn shows no selector effect, though it has a safe recipe\. DPO costs nothing on QA: no change is outside±1\.8\\pm 1\.8points, and the per\-object hallucination rate on COCO falls for every corpus \(CHAIRi \.083 to \.073, \.079 and \.082 for the verified, RM\-picked and random pairs\), with the label adding little over random pairs \([Table8](https://arxiv.org/html/2610.12707#A7.T8)\)\. The size\-matched random control plays the role of the untrained\-initialization control of[Fan et al\. \(2026\)](https://arxiv.org/html/2610.12707#bib.bib29)\. The RM\-picked corpus is the only one that shortens captions \(7%\), yet its per\-object rate also falls, so the change is not purely length\. One DPO turn on self\-generated pairs is a mild, label\-agnostic hallucination reducer, and the actor side of the loop still lacks a demonstrated selector effect\.

Table 8:Actor, DPO \(2B policy\)\.Full\-parameter DPO \(β=0\.1\\beta\{=\}0\.1, lr10−610^\{\-6\}, 2 epochs, vision tower frozen\) on three corpora from the same 500 prompts×\\times4 tasks:*verified*\(3,462 pairs\),*rm*\(probe\-8B argmax vs argmin, 2,000\),*random*\(two random candidates, 1,999; the control\)\. Greedy regeneration at 512 tokens;Δ\\Deltaagainst the regenerated base, paired bootstrap over prompts \(2,000 resamples\), every QA CI including zero\. CHAIRi is hallucinated object mentions over all object mentions, pooled over the 500 COCO captions, which shorter captions do not lower by themselves\. Best value per column in bold\.The DPO base policy was regenerated at 512 new tokens under the identical protocol\. Between 19 and 27% of DPO pairs were skipped by the 1,024\-token cap, mostly long QA chains of thought, so the QA share of each corpus is below nominal; the update was small \(loss \.69 to \.64 to \.68\)\. The image\-generation counterpart \(LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.12707#bib.bib43)\)on Qwen\-Image\([Qwen Team, 2025a](https://arxiv.org/html/2610.12707#bib.bib39)\)for 160 steps on 640 on\-policy images, across seven Diffusion\-DPO\([Wallace et al\., 2024](https://arxiv.org/html/2610.12707#bib.bib42)\)and related fine\-tuning variants\) has the same shape\. No variant differs from the base generator on the hard prompts \(allΔ\\Deltawithin\[−\.04,\+\.07\]\[\-\.04,\+\.07\]\), and every selection variant minus its random control is inside±2\\pm 2points\.

## Appendix HCLIP Studies

Figure 9:CLIP view of the perturbation pairs\(22,000 same\-image pairs, CLIP ViT\-B/32\)\. \(a\) Image–caption cosine of the real and the perturbed caption; the perturbation lowers the median from \.303 to \.294 and adds a low\-score tail\. \(b\) The dropΔ\\DeltaCLIP per pair, which the dissociation split of[Section3\.3](https://arxiv.org/html/2610.12707#S3.SS3)ranks on, on a log count axis; the dashed line is the top\-quartile cut \(Δ≥\.008\\Delta\\geq\.008\) that defines the CLIP\-visible arm\. CLIP accepts 77 tokens and the captions average 150 words, so for 43% of pairs the edit falls beyond its window andΔ=0\\Delta=0exactly; the CLIP\-blind quartile is drawn from those pairs and is omitted from the histogram\.[Figure9](https://arxiv.org/html/2610.12707#A8.F9)shows the score distribution the quartile cuts are taken from\. Filtering the positives by image–caption CLIP score gives \.685 / \.698 / \.678 on Multimodal RewardBench for the top, random and bottom quartile and \.670 / \.686 / \.682 on VL\-RewardBench\. By contrast, theΔ\\DeltaCLIP split ranks each pair by the drop in CLIP cosine from the real to the perturbed caption\. The CLIP\-blind quartile has near\-zero drop, part of it because CLIP truncates at 77 tokens and the edit falls outside the window\.

## Appendix IReading the Two Statements

The convexity fact of[Section2\.2](https://arxiv.org/html/2610.12707#S2.SS2)predicts that, when gradient descent runs on a fixed\-penalty objective, initialization, step size and step count cannot move the minimizer\. It predicts nothing about AdamW, whose stationary points depend on its preconditioner, and nothing about the negative pool, the hardness schedule or the temperature, each of which changes the population objective rather than the path to its minimizer\. The temperature is the least obvious of these: writingu=β​wu=\\beta wturns the weight decay intoλ2​β2​∥u∥2\\frac\{\\lambda\}\{2\\beta^\{2\}\}\\lVert u\\rVert^\{2\}, soβ\\betaacts as an effective regularization strength that rescalesw⋆w^\{\\star\}and tilts its direction\. That those interventions nonetheless leave the benchmark where it was \([Section3\.1](https://arxiv.org/html/2610.12707#S3.SS1.SSS0.Px3)\) is an empirical finding, and[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)explains it\. Each intervention re\-weights the same difference vectors, and on the benchmark’s pairs the re\-weighted heads order alike\. The CLIP split of[Section3\.3](https://arxiv.org/html/2610.12707#S3.SS3)is the same equation evaluated at a re\-weighting large enough to separate on hallucination\. The value at the minimizer is the best linear separation of positives from negatives in the layerN−kN\{\-\}kfeature space, which is consistent with depth moving the plateau while optimizer settings do not; convexity fixes the endpoint of the fixed\-penalty problem and not the distance to it, which is where initialization re\-enters \([Section3\.4](https://arxiv.org/html/2610.12707#S3.SS4)\)\.

[Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)is stated for the decoupled objective because that is the one with a closed\-form optimum, and for mutually absolutely continuousPPandQQbecause that is what keeps the log ratio finite\. Our sampler does not satisfy the hypothesis: a caption of another image has probability zero as a caption of this one, so on those pairs the unrestricted optimum runs to±∞\\pm\\inftyand separates the two laws with unbounded margin\. The in\-distribution accuracy of the recipe is the finite\-sample face of that, with all 540 configurations of[Section3\.1](https://arxiv.org/html/2610.12707#S3.SS1.SSS0.Px3)at \.995 or better, and it is why we analyze the linear head through the fixed\-penalty problem of[Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1)rather than through[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)\. The diagonal terms alone would leave a context\-dependent offset free, making scores comparable within a prompt and not across prompts; the matrix form removes that freedom, and our same\-image\-only ablation \([Section3\.1](https://arxiv.org/html/2610.12707#S3.SS1.SSS0.Px3)\) confirms that the offset does not change within\-prompt accuracy\. The matrix form also decides what the target does with length\. Write a response as contentccand lengthll\. If the negatives match the positives’ length law conditionally on content,p​\(l∣c\)=q¯​\(l∣c\)p\(l\\mid c\)=\\bar\{q\}\(l\\mid c\), thend​P/d​Q\\mathrm\{d\}P/\\mathrm\{d\}Qdepends onccalone and length drops out of[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)\(cf\.[Fan et al\., 2026](https://arxiv.org/html/2610.12707#bib.bib29)\)\. A matched marginal is not enough: with two contents and two lengths, the joint tables

P=\(\.4\.1\.1\.4\),Q=\(\.1\.4\.4\.1\)P=\\begin\{pmatrix\}\.4&\.1\\\\ \.1&\.4\\end\{pmatrix\},\\qquad Q=\\begin\{pmatrix\}\.1&\.4\\\\ \.4&\.1\\end\{pmatrix\}share the length marginal\(\.5,\.5\)\(\.5,\.5\)and still have ratios44and1/41/4that depend on length\. Our substitution edits hold length fixed pairwise and splice and shuffle shorten the negative, so the bank meets neither condition exactly and nothing makes the target of[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)length\-free\. A linear head cannot represent[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)exactly and is characterized by the error\-weighted mean difference of[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)instead, whose coefficient on length is not the target’s; the sign flip of[Section3\.2](https://arxiv.org/html/2610.12707#S3.SS2.SSS0.Px1)is that head\-side coefficient changing with the backbone under one sampler\.

### I\.1What Constructed Comparisons Supervise

Fix a contextxx\(an image and its prompt\) and a finite response set𝒴\\mathcal\{Y\}, and letJx​\(y,y′\)J\_\{x\}\(y,y^\{\\prime\}\)be the probability that a training comparison designatesyyas chosen andy′y^\{\\prime\}as rejected, so that the conditional population loss is

ℒx​\(s\)=∑y,y′∈𝒴Jx​\(y,y′\)​ℓ​\(s⁡\(x,y\)−s⁡\(x,y′\)\),ℓ⁡\(v\)=log⁡\(1\+e−v\)\.\\mathcal\{L\}\_\{x\}\(s\)\\;=\\;\\sum\_\{y,y^\{\\prime\}\\in\\mathcal\{Y\}\}J\_\{x\}\(y,y^\{\\prime\}\)\\,\\ell\\big\(s\(x,y\)\-s\(x,y^\{\\prime\}\)\\big\),\\qquad\\ell\(v\)=\\log\(1\+e^\{\-v\}\)\.\(6\)Ifμ⁡\(y∣x\)\\mu\(y\\mid x\)is the distribution of real captions andKx​\(y′∣y\)K\_\{x\}\(y^\{\\prime\}\\mid y\)constructs the negative, thenJx​\(y,y′\)=μ⁡\(y∣x\)​Kx​\(y′∣y\)J\_\{x\}\(y,y^\{\\prime\}\)=\\mu\(y\\mid x\)K\_\{x\}\(y^\{\\prime\}\\mid y\): independent negatives useKx​\(y′∣y\)=ν⁡\(y′\)K\_\{x\}\(y^\{\\prime\}\\mid y\)=\\nu\(y^\{\\prime\}\), our cross\-image operator𝒯×\\mathcal\{T\}\_\{\\times\}, and a deterministic edit usesKx\(y′∣y\)=𝟏\{y′=𝒞\(y\)\}K\_\{x\}\(y^\{\\prime\}\\mid y\)=\\mathbf\{1\}\\\{y^\{\\prime\}=\\mathcal\{C\}\(y\)\\\}\. Judged candidates give aJxJ\_\{x\}in which winner and loser are dependent after labeling, so their marginals alone do not determine the objective; the joint comparison law does\. These are population statements about unrestricted scalar scores and carry no finite\-sample or downstream guarantee\.

### I\.2Independent Negatives: Density Ratios and PMI

WithJx​\(y,y′\)=μ⁡\(y∣x\)​ν​\(y′\)J\_\{x\}\(y,y^\{\\prime\}\)=\\mu\(y\\mid x\)\\,\\nu\(y^\{\\prime\}\)and a common supportSxS\_\{x\}, the argument of[Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)with the context held fixed gives the unrestricted minimizers of[Eq\.6](https://arxiv.org/html/2610.12707#A9.E6)onSxS\_\{x\},

s∗​\(x,y\)=log⁡μ⁡\(y∣x\)ν⁡\(y\)\+ϕ⁡\(x\),y∈Sx,s^\{\\ast\}\(x,y\)\\;=\\;\\log\\frac\{\\mu\(y\\mid x\)\}\{\\nu\(y\)\}\+\\phi\(x\),\\qquad y\\in S\_\{x\},\(7\)withϕ⁡\(x\)\\phi\(x\)arbitrary\. Ifν=μY\\nu=\\mu\_\{Y\}, the caption marginal of the corpus lawμ⁡\(x,y\)\\mu\(x,y\), then

s∗​\(x,y\)=log⁡μ⁡\(x,y\)μX​\(x\)​μY​\(y\)\+ϕ⁡\(x\)=PMIμ​\(x,y\)\+ϕ⁡\(x\),s^\{\\ast\}\(x,y\)\\;=\\;\\log\\frac\{\\mu\(x,y\)\}\{\\mu\_\{X\}\(x\)\\,\\mu\_\{Y\}\(y\)\}\+\\phi\(x\)\\;=\\;\\mathrm\{PMI\}\_\{\\mu\}\(x,y\)\+\\phi\(x\),\(8\)the idealized case of using other images’ captions as negatives\. PMI measures image–caption association, not correctness or preference agreement; shared support is an assumption a finite corpus does not guarantee; and the result does not apply to edited copies or to judge\-ordered candidates\.

### I\.3When Perturbations Expose an Additional Quality Signal

The following stylized example separates topic association from response quality\. Represent a response asy=\(t,c\)y=\(t,c\), wherettdenotes its topic andc∈\{0,1\}c\\in\\\{0,1\\\}indicates an incorrect or correct version\. Suppose the corpus has distribution

μ⁡\(\(t,c\)∣x\)=px​\(t\)​bη​\(c\),bη​\(1\)=η,bη​\(0\)=1−η,12<η<1\.\\mu\\big\(\(t,c\)\\mid x\\big\)=p\_\{x\}\(t\)\\,b\_\{\\eta\}\(c\),\\qquad b\_\{\\eta\}\(1\)=\\eta,\\quad b\_\{\\eta\}\(0\)=1\-\\eta,\\quad\\tfrac\{1\}\{2\}<\\eta<1\.\(9\)Thus topics depend on the context, but the fraction of correct versions is the same across contexts and topics\. Letp⁡\(t\)p\(t\)be the corpus topic marginal and assumepx​\(t\)\>0p\_\{x\}\(t\)\>0on a common finite topic set\. Independent corpus negatives have distributionμY​\(t,c\)=p⁡\(t\)​bη​\(c\)\\mu\_\{Y\}\(t,c\)=p\(t\)b\_\{\\eta\}\(c\), so[Eq\.7](https://arxiv.org/html/2610.12707#A9.E7)gives

sind∗​\(x,t,c\)=log⁡px​\(t\)​bη​\(c\)p⁡\(t\)​bη​\(c\)\+ϕ⁡\(x\)=log⁡px​\(t\)p⁡\(t\)\+ϕ⁡\(x\)\.s^\{\\ast\}\_\{\\mathrm\{ind\}\}\(x,t,c\)\\;=\\;\\log\\frac\{p\_\{x\}\(t\)\\,b\_\{\\eta\}\(c\)\}\{p\(t\)\\,b\_\{\\eta\}\(c\)\}\+\\phi\(x\)\\;=\\;\\log\\frac\{p\_\{x\}\(t\)\}\{p\(t\)\}\+\\phi\(x\)\.\(10\)The quality factor cancels: the optimum distinguishes topics by their association with the context but assigns equal scores to correct and incorrect versions within a topic\.

Now consider an idealized edit𝒞⁡\(t,c\)=\(t,1−c\)\\mathcal\{C\}\(t,c\)=\(t,1\-c\)that switches the version while preserving the topic, with the original always labeled chosen\. This label agrees with correctness with probabilityη\\etaand is reversed with probability1−η1\-\\eta; the corpus need not be perfectly correct\. For a fixed topic, writeΔt=s⁡\(x,t,1\)−s⁡\(x,t,0\)\\Delta\_\{t\}=s\(x,t,1\)\-s\(x,t,0\)\. Its conditional loss and minimizer are

ℒedit,t​\(Δt\)=η​ℓ​\(Δt\)\+\(1−η\)​ℓ​\(−Δt\),Δt∗=log⁡η1−η\>0\.\\mathcal\{L\}\_\{\\mathrm\{edit\},t\}\(\\Delta\_\{t\}\)=\\eta\\,\\ell\(\\Delta\_\{t\}\)\+\(1\-\\eta\)\\,\\ell\(\-\\Delta\_\{t\}\),\\qquad\\Delta\_\{t\}^\{\\ast\}=\\log\\frac\{\\eta\}\{1\-\\eta\}\>0\.\(11\)Indeed, the derivative isσ⁡\(Δt\)−η\\sigma\(\\Delta\_\{t\}\)\-\\eta, which vanishes uniquely at the stated margin\. Perturbation training therefore ranks the correct version above the incorrect one, a distinction absent from the independent\-negative optimum in this example\.

A Bradley–Terry judge with rewardr⁡\(x,t,c\)=γ​cr\(x,t,c\)=\\gamma c,γ\>0\\gamma\>0, gives the same within\-topic ordering with marginγ\\gamma; the edited pairs recover the ordering with a margin set byη\\etainstead\. Under the stated factorization and edit assumptions, the example shows one additional quality distinction, not a general superiority of perturbations\. Changing quality\-bearing content inside an otherwise matched response can supply a preference that corpus association alone does not teach; the experiments test whether actual edit rules do so and whether that preference transfers\.

## Appendix JProofs

#### Convexity of the regularized objective\.

Write the population objective as

L⁡\(w\)=𝔼​ℓ​\(β​w⊤​d\),d=h\+−h−,ℓ⁡\(z\)=log⁡\(1\+e−z\)\.L\(w\)\\;=\\;\\mathbb\{E\}\\,\\ell\\big\(\\beta\\,w^\{\\top\}d\\big\),\\qquad d=h^\{\+\}\-h^\{\-\},\\qquad\\ell\(z\)=\\log\\big\(1\+e^\{\-z\}\\big\)\.The functionℓ\\ellis twice differentiable with

ℓ′′​\(z\)=σ⁡\(z\)​σ​\(−z\)\>0,\\ell^\{\\prime\\prime\}\(z\)\\;=\\;\\sigma\(z\)\\,\\sigma\(\-z\)\\;\>\\;0,so it is strictly convex, and a strictly convex function of an affine map is convex\.

The Hessian ofLLis

∇2L​\(w\)=β2​𝔼​\[ℓ′′​\(β​w⊤​d\)​d​d⊤\]\.\\nabla^\{2\}L\(w\)\\;=\\;\\beta^\{2\}\\,\\mathbb\{E\}\\Big\[\\ell^\{\\prime\\prime\}\\big\(\\beta w^\{\\top\}d\\big\)\\,d\\,d^\{\\top\}\\Big\]\.It is positive definite onspan⁡\{d\}\\operatorname\{span\}\\\{d\\\}, because the weightsℓ′′\\ell^\{\\prime\\prime\}are strictly positive, and zero on the orthogonal complement, becausew⊤​dw^\{\\top\}ddoes not depend on that component\. HenceLLis strictly convex on the span and constant on its complement\.

Adding the penalty turns the objective into

Lλ​\(w\)=L⁡\(w\)\+λ2​w⊤​D​w,λ\>0,D≻0,L\_\{\\lambda\}\(w\)\\;=\\;L\(w\)\\;\+\\;\\frac\{\\lambda\}\{2\}\\,w^\{\\top\}Dw,\\qquad\\lambda\>0,\\;D\\succ 0,which isλ​λmin​\(D\)\\lambda\\,\\lambda\_\{\\min\}\(D\)\-strongly convex on all ofℝm\\mathbb\{R\}^\{m\}, withmmthe feature dimension\. It therefore has exactly one minimizer\. If𝔼​∥d∥2<∞\\mathbb\{E\}\\lVert d\\rVert^\{2\}<\\infty, it is alsoLsL\_\{s\}\-smooth withLs=β24​𝔼​∥d∥2\+λ​λmax​\(D\)L\_\{s\}=\\tfrac\{\\beta^\{2\}\}\{4\}\\mathbb\{E\}\\lVert d\\rVert^\{2\}\+\\lambda\\lambda\_\{\\max\}\(D\), sinceℓ′′≤14\\ell^\{\\prime\\prime\}\\leq\\tfrac\{1\}\{4\}, and gradient descent with a constant step below2/Ls2/L\_\{s\}converges to the minimizer from every initialization\. Isotropic weight decay isD=ID=I\.

AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.12707#bib.bib56)\)is not gradient descent onLλL\_\{\\lambda\}for a fixedDD\. With a frozen diagonal preconditionerMM, its updatew←w−α\(M∇L\(w\)\+λw\)w\\leftarrow w\-\\alpha\(M\\nabla L\(w\)\+\\lambda w\)stops where

∇L​\(w\)\+λ​M−1​w=0,\\nabla L\(w\)\+\\lambda M^\{\-1\}w=0,which moves withMM: forL⁡\(w\)=log⁡\(1\+e−w\)L\(w\)=\\log\(1\+e^\{\-w\}\)andλ=1\\lambda=1the fixed point solvesw=M​σ​\(−w\)w=M\\sigma\(\-w\), giving0\.400\.40atM=1M=1and1\.041\.04atM=4M=4\. The actualMMfollows the gradient history, and[Xie and Li \(2024\)](https://arxiv.org/html/2610.12707#bib.bib74)characterize AdamW’s limit points as KKT points of anℓ∞\\ell\_\{\\infty\}\-constrained problem instead\. The statements in this appendix therefore concern the fixed\-penalty problem; the AdamW heads of the experiments are measured, not derived \([Section3\.1](https://arxiv.org/html/2610.12707#S3.SS1.SSS0.Px3)\)\.

For gradient descent, initialization, step size, sample order and the number of steps change only the iterate sequence, so at fixedpp,qnegq\_\{\\mathrm\{neg\}\},β\\beta,λ\\lambdaandDDnone of them changes the minimizer\.

The negative pool, its hardness and its composition changeqnegq\_\{\\mathrm\{neg\}\}and therefore the law ofdd, so they may move the minimizer;[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)says how, through the error\-weighted mean ofdd\. The temperature is not neutral either\. Writingu=β​wu=\\beta w,

Lλ​\(w\)=𝔼​ℓ​\(u⊤​d\)\+λ2​β2​‖u‖2,L\_\{\\lambda\}\(w\)\\;=\\;\\mathbb\{E\}\\,\\ell\\big\(u^\{\\top\}d\\big\)\\;\+\\;\\frac\{\\lambda\}\{2\\beta^\{2\}\}\\,\\\|u\\\|^\{2\},soβ\\betaenters as an effective regularization strengthλ/β2\\lambda/\\beta^\{2\}and a change in it rescalesw⋆w^\{\\star\}and, since ridge shrinkage is isotropic while𝔼⁡\[d​d⊤\]\\mathbb\{E\}\[dd^\{\\top\}\]is not, tilts its direction as well\. Only the accuracy of a*ranking*is invariant to the scale ofw⋆w^\{\\star\}, not to its direction\. ∎

#### Proof of[Proposition1](https://arxiv.org/html/2610.12707#Thmproposition1)\.

LetP=μ⊗pP=\\mu\\otimes pandQ=μ⊗q¯Q=\\mu\\otimes\\bar\{q\}be the joint laws of a positive and of a negative triplea=\(x,q,y\)a=\(x,q,y\), withq¯\\bar\{q\}the marginal law of a negative as in the statement\. The off\-diagonal population loss over unrestricted scoresssis

J⁡\(s\)=𝔼a∼P,b∼Q​ℓ​\(β⁡\(s⁡\(a\)−s⁡\(b\)\)\),J\(s\)\\;=\\;\\mathbb\{E\}\_\{a\\sim P,\\;b\\sim Q\}\\;\\ell\\big\(\\beta\\,\(s\(a\)\-s\(b\)\)\\big\),whereaaandbbare independent because they come from different micro\-batch entries\. We assume throughout thatPPandQQare mutually absolutely continuous, so thatd​P/d​Q\\mathrm\{d\}P/\\mathrm\{d\}Qis finite and positive almost everywhere\. The hypothesis is what makes the minimizer a real\-valued function: on a pair where one of the two densities vanishes, the stationarity condition below putsδ⋆\\delta^\{\\star\}at±∞\\pm\\infty, the infimum ofJJover real\-valued scores is not attained, and every sequence approaching it separates the two laws with unbounded margin\. Our own sampler is of that kind, which is why the recipe’s weight decay and its linear class, and not[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4), are what pin down the head that gets trained \([Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1)\)\.

Group the contributions of each unordered pair\{a,b\}\\\{a,b\\\}of triples\. The pair appears once withaaas the positive, at densityP⁡\(a\)​Q​\(b\)P\(a\)Q\(b\), and once withbbas the positive, at densityP⁡\(b\)​Q​\(a\)P\(b\)Q\(a\)\. For a fixed differenceδ=s⁡\(a\)−s⁡\(b\)\\delta=s\(a\)\-s\(b\)its contribution is

P⁡\(a\)​Q​\(b\)​ℓ​\(β​δ\)\+P⁡\(b\)​Q​\(a\)​ℓ​\(−β​δ\)\.P\(a\)Q\(b\)\\,\\ell\(\\beta\\delta\)\\;\+\\;P\(b\)Q\(a\)\\,\\ell\(\-\\beta\\delta\)\.This is strictly convex inδ\\delta, and its derivative vanishes where

σ⁡\(β​δ\)=P⁡\(a\)​Q​\(b\)P⁡\(a\)​Q​\(b\)\+P⁡\(b\)​Q​\(a\),that is, atβ​δ⋆=log⁡P⁡\(a\)Q⁡\(a\)−log⁡P⁡\(b\)Q⁡\(b\)\.\\sigma\(\\beta\\delta\)\\;=\\;\\frac\{P\(a\)Q\(b\)\}\{P\(a\)Q\(b\)\+P\(b\)Q\(a\)\},\\qquad\\text\{that is, at\}\\qquad\\beta\\,\\delta^\{\\star\}\\;=\\;\\log\\frac\{P\(a\)\}\{Q\(a\)\}\\;\-\\;\\log\\frac\{P\(b\)\}\{Q\(b\)\}\.
The functions⋆=β−1​log⁡\(d​P/d​Q\)\+Cs^\{\\star\}=\\beta^\{\-1\}\\log\(\\mathrm\{d\}P/\\mathrm\{d\}Q\)\+Cattainsδ⋆\\delta^\{\\star\}for every pair simultaneously, so it minimizes every summand and thereforeJJ\. Any other minimizer attainsδ⋆\\delta^\{\\star\}on almost every pair as well, because each summand is strictly convex inδ\\delta, so it differs froms⋆s^\{\\star\}by a constant on the support ofPPandQQ\. Since the two laws share the context lawμ\\mu,

d​Pd​Q​\(x,q,y\)=μ⁡\(x,q\)​p​\(y∣x,q\)μ⁡\(x,q\)​q¯​\(y∣x,q\)=p⁡\(y∣x,q\)q¯​\(y∣x,q\),\\frac\{\\mathrm\{d\}P\}\{\\mathrm\{d\}Q\}\(x,q,y\)\\;=\\;\\frac\{\\mu\(x,q\)\\,p\(y\\mid x,q\)\}\{\\mu\(x,q\)\\,\\bar\{q\}\(y\\mid x,q\)\}\\;=\\;\\frac\{p\(y\\mid x,q\)\}\{\\bar\{q\}\(y\\mid x,q\)\},which is[Eq\.4](https://arxiv.org/html/2610.12707#S2.E4)\. The pairs run over triples and not over responses within one context, so no context\-dependent offset is left free; only the constantCCis\. The factorβ−1\\beta^\{\-1\}is where the temperature enters the target, and a linear head absorbs it into the scale ofww\.

TheBBdiagonal terms are outside this argument\. Therebbis an edit of the sameaait is scored against, so the pair law isP⁡\(a\)​qneg​\(b∣a\)P\(a\)\\,q\_\{\\mathrm\{neg\}\}\(b\\mid a\)rather than the productP⁡\(a\)​Q​\(b\)P\(a\)Q\(b\), the two orderings of\{a,b\}\\\{a,b\\\}no longer enter at weights that depend only on the marginals, andδ⋆\\delta^\{\\star\}above is not the pointwise minimizer\. We therefore state the proposition for the decoupled objective, whose terms carry weight1−1/B=87\.5%1\-1/B=87\.5\\%atB=8B\{=\}8\.

The coupling moves the optimum\. Take two responses withp=\(\.8,\.2\)p=\(\.8,\.2\), an edit that always swaps them, soq¯=\(\.2,\.8\)\\bar\{q\}=\(\.2,\.8\), andβ=1\\beta=1\. Up to a constant from equal\-response pairs, the full objective atB=8B\{=\}8is

78​\(\.64​ℓ​\(Δ\)\+\.04​ℓ​\(−Δ\)\)\+18​\(\.8​ℓ​\(Δ\)\+\.2​ℓ​\(−Δ\)\)=\.66​ℓ​\(Δ\)\+\.06​ℓ​\(−Δ\),\\tfrac\{7\}\{8\}\\big\(\.64\\,\\ell\(\\Delta\)\+\.04\\,\\ell\(\-\\Delta\)\\big\)\+\\tfrac\{1\}\{8\}\\big\(\.8\\,\\ell\(\\Delta\)\+\.2\\,\\ell\(\-\\Delta\)\\big\)\\;=\\;\.66\\,\\ell\(\\Delta\)\+\.06\\,\\ell\(\-\\Delta\),minimized atΔ=log⁡11\\Delta=\\log 11, while the cross\-entry terms alone givelog⁡16\\log 16\. ∎

#### Proof of[Corollary1](https://arxiv.org/html/2610.12707#Thmcorollary1)\.

The population objective with the penalty is

Lλ​\(w\)=𝔼​ℓ​\(β​w⊤​d\)\+λ2​w⊤​D​w,L\_\{\\lambda\}\(w\)\\;=\\;\\mathbb\{E\}\\,\\ell\\big\(\\beta\\,w^\{\\top\}d\\big\)\\;\+\\;\\frac\{\\lambda\}\{2\}\\,w^\{\\top\}Dw,with the expectation over the pair law of the matrix loss: each micro\-batch contributesBBsame\-entry pairs andB2−BB^\{2\}\-Bcross\-entry pairs, at total weights1/B1/Band1−1/B1\-1/B\. Sinceℓ′​\(z\)=−σ⁡\(−z\)\\ell^\{\\prime\}\(z\)=\-\\sigma\(\-z\),

∇Lλ​\(w\)=−β​𝔼​\[σ⁡\(−β​w⊤​d\)​d\]\+λ​D​w,\\nabla L\_\{\\lambda\}\(w\)\\;=\\;\-\\beta\\,\\mathbb\{E\}\\big\[\\sigma\(\-\\beta\\,w^\{\\top\}d\)\\,d\\big\]\\;\+\\;\\lambda\\,Dw,andLλL\_\{\\lambda\}is strongly convex \(previous paragraph\), so its unique minimizer is the unique zero of this gradient, which is[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)\. The inner product ofD​w⋆Dw^\{\\star\}with a unit vectorvvgives its coefficient alongvv\. In particularD​w⋆Dw^\{\\star\}is a positive combination of difference vectors and lies in their span, which is all ofℝm\\mathbb\{R\}^\{m\}whenever the difference vectors span it, and then constrains nothing; what[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)fixes is the combination\.

Now takeD=ID=Iand suppose the features encode length along a unit directionvvwithv⊤​d=0v^\{\\top\}d=0almost surely on the same\-entry pairs\. This is an assumption on the backbone, which pairwise length preservation \(exact for the substitution operators, approximate for the bank\) makes plausible and does not imply\. Those pairs then contribute nothing to the head’s coefficientv⊤​w⋆v^\{\\top\}w^\{\\star\}, and the cross\-entry pairs, which compare captions of different images, contribute

βλ​\(1−1B\)​𝔼×​\[σ⁡\(−β​w⋆⁣⊤​d\)​v⊤​d\],\\frac\{\\beta\}\{\\lambda\}\\Big\(1\-\\frac\{1\}\{B\}\\Big\)\\,\\mathbb\{E\}\_\{\\times\}\\big\[\\sigma\(\-\\beta\\,w^\{\\star\\top\}d\)\\,v^\{\\top\}d\\big\],an error\-weighted mean of gaps between unrelated captions, which the objective does not set to zero\. Pairwise preservation on the diagonal therefore leaves the coefficient alongvvto the cross\-entry pairs rather than fixing it at zero, the same conclusion as for the unrestricted optimum above, where a comparison that holds a nuisance fixed leaves its coefficient undetermined\. ForD≠ID\\neq Ithe same computation givesv⊤​D​w⋆v^\{\\top\}Dw^\{\\star\}, and the head’s coefficientv⊤​w⋆v^\{\\top\}w^\{\\star\}can be nonzero even when every difference is orthogonal tovv: withD=diag⁡\(1,2\)D=\\mathrm\{diag\}\(1,2\)andd=\(1,−1\)d=\(1,\-1\),w⋆∝\(1,−12\)w^\{\\star\}\\propto\(1,\-\\tfrac\{1\}\{2\}\)andv=\(1,1\)/2v=\(1,1\)/\\sqrt\{2\}givesv⊤​d=0v^\{\\top\}d=0andv⊤​w⋆\>0v^\{\\top\}w^\{\\star\}\>0\.

For two samplersq¯1,q¯2\\bar\{q\}\_\{1\},\\bar\{q\}\_\{2\}with difference lawsρ1≠ρ2\\rho\_\{1\}\\neq\\rho\_\{2\},[Eq\.5](https://arxiv.org/html/2610.12707#S2.E5)gives minimizersw1,w2w\_\{1\},w\_\{2\}that coincide only if the two error\-weighted means do, a condition on the laws and not on their spans\. A benchmark pair set\{ek\}\\\{e\_\{k\}\\\}measures the sign pattern

sign⁡\(w⊤​ek\),k=1,…,K,\\operatorname\{sign\}\\big\(w^\{\\top\}e\_\{k\}\\big\),\\qquad k=1,\\dots,K,which coincides forw1w\_\{1\}andw2w\_\{2\}whenever the two heads order the benchmark’s pairs alike, however far apart they are on other pairs\. ∎

## Appendix KCompute Resources

Training a head is the cheapest step in the study\. Once the frozen backbone’s hidden states are stored, a 540\-configuration grid at 2B fits in about nine GPU\-minutes, and a single head is 200 steps of ad→1d\\to 1linear map\. The 32B confirm run used FSDP over eight GPUs at effective batch 1,024\. The compute goes to everything that is not the reward model: one forward pass of the backbone per pair to store its hidden states, Best\-of\-NNsampling atN=16N\{=\}16over 2,500 prompts for two policy sizes, and the image arm’s renders\. The image arm needs two same\-seed Qwen\-Image renders per prompt at about 7 GPU\-seconds each, about 5 GPU\-hours for its 1,212 training pairs\. The caption negative bank itself is free: 10 seconds on one CPU core for 22k pairs, with no model in the loop\. The figure the paper quotes against fine\-tuning, about a thousandth of the compute, counts head training against the full fine\-tune of the same backbone on the same pairs and excludes the backbone forward passes both settings share\.