DEEPO: 针对多模态大语言模型幻觉的双熵增强策略优化

# DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs
Source: [https://arxiv.org/html/2609.28570](https://arxiv.org/html/2609.28570)
Miao PanAffiliation:Zhejiang UniversityWangjie GanAffiliation:Zhejiang UniversityJingxiao YangAffiliation:Zhejiang UniversityFan WangAffiliation:Zhejiang UniversityWeiming LiuAffiliation:Zhejiang UniversityCheng TanAffiliation:Shanghai Artificial Intelligence LaboratoryXuhong ZhangAffiliation:Zhejiang UniversityJintao ChenAffiliation:Zhejiang University

###### Abstract

Reinforcement learning \(RL\) is widely used to sharpen reasoning in multimodal large language models \(MLLMs\), yet its effect on hallucination is uneven\. We trace this to two weak points in the*correction chain*from reward to parameter update\. At the rollout level, hard queries—those with high semantic entropy—frequently produce unanimously wrong sample groups, collapsing the group\-relative advantage to zero exactly where hallucination risk is highest\. At the optimization level, confident\-but\-wrong tokens are gradient\-invisible: a categorical policy’s expected score\-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates\. We propose Dual\-Entropy Enhanced Policy Optimization \(DEEPO\), a dual\-stage enhancement combining signal variance regularization with gradient preconditioning: semantic\-entropy\-triggered expert prefixes inject grounded continuations on high\-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage\-sign\-aware Rényi preconditioning counteracts logit\-level saturation so correction reaches confident errors in the operational confidence regime\. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU—the most complex long\-horizon task in our evaluation suite \(\+4\.0\+4\.0, 95% CI\[1\.1,6\.9\]\[1\.1,6\.9\]\)—and additive elsewhere\. DEEPO reduces hallucination while preserving accuracy and training stability\.

## 1Introduction

Multimodal large language models \(MLLMs\) have greatly advanced vision–language understanding and multi\-step reasoning\([Yin et al\., 2024](https://arxiv.org/html/2609.28570#bib.bib48);[Wu et al\., 2024](https://arxiv.org/html/2609.28570#bib.bib49);[Li et al\., 2024b](https://arxiv.org/html/2609.28570#bib.bib50)\)\. However, hallucination—confident but incorrect or semantically inconsistent outputs—remains a key obstacle\([Fu et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib47)\), motivating work on its causes and mitigation\([Mahmoud et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib46)\)\.

While prior work attributes hallucination to dataset noise\([Bang et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib42)\), architectural bottlenecks\([Cossio, 2025](https://arxiv.org/html/2609.28570#bib.bib43)\), or decoding errors\([Wu et al\., 2025a](https://arxiv.org/html/2609.28570#bib.bib44)\), these explanations target static properties of the model or input\([Dang et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib41)\)\. Hallucination is also shaped during training: SFT overfits surface patterns by imitation, whereas RL can encourage guessing under uncertainty through reward\-driven rollouts\([Lin et al\., 2024](https://arxiv.org/html/2609.28570#bib.bib45);[Mahmoud et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib46);[Shenfeld et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib40)\)—an effect that is heterogeneous rather than universal \(in our measurements GRPO improves over its base model on POPE and HallusionBench yet slightly degrades on VideoHallucer; Table[1](https://arxiv.org/html/2609.28570#S4.T1)\): the issue is not that RL uniformly worsens hallucination, but that its corrective machinery is unreliable precisely on confident errors\. We therefore focus on the RL stage, where the correction chain from scalar reward to parameter update has two weak points that a training method can, and we argue should, address jointly\.

\(I\) Hard queries starve the correction signal\.In Fig[1](https://arxiv.org/html/2609.28570#S1.F1), hallucinated answers concentrate in regions of high semantic entropy—exactly the queries on which sampling is most likely to fail unanimously: with binary rewards an all\-wrong group hasμ=0\\mu=0,σ→0\\sigma\\to 0, so every response receivesAi≈0A\_\{i\}\\approx 0and the iteration extracts no gradient\([Kalai et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib51);[Khan and Fu, 2024](https://arxiv.org/html/2609.28570#bib.bib52)\)\. An*expert hint*—a truncated, answer\-free chain\-of\-thought \(CoT\) prefix—conditions one continuation on grounded visual reasoning, injecting direct supervision on hard queries and, under a shared group baseline, a high\-reward anchor that restores nonzero advantage variance, so failed responses receive genuine negative advantages instead of the degenerate zero; prefixes come from dataset traces\([Feng et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib39)\)or the policy’s own verified rollouts \(Sec\.[4\.1](https://arxiv.org/html/2609.28570#S4.SS1)\), never a test\-time oracle\.

![Refer to caption](https://arxiv.org/html/2609.28570v1/pic/Intro/Intro_Example_single_path.png)Figure 1:Model guessing and expert\-hint\-guided correction\. Under uncertain visual cues the model “guesses,” yielding high semantic entropy and frequently unanimous failures; the expert hint grounds one continuation on the visual evidence\.\(II\) Confident errors are gradient\-invisible\.Even a perfect advantage signal cannot correct confident hallucinations, because the delivery mechanism vanishes: for a softmax policy,𝔼​∥∇z​log​π∥2=1−e−H2\\mathbb\{E\}\\lVert\\nabla\_\{z\}\\log\\pi\\rVert^\{2\}=1\-e^\{\-H\_\{2\}\}\(Prop\.[1](https://arxiv.org/html/2609.28570#Thmproposition1)\), and a gradient\-masking pilot confirms that high\-entropy tokens carry the bulk of the update signal \(Fig\.[3](https://arxiv.org/html/2609.28570#A1.F3), App\.[A](https://arxiv.org/html/2609.28570#A1)\)\. Confident predictions—and confident hallucinations in particular, which are precisely the errors that survive decoding—thus receive vanishing corrective updates, while uncertain regions dominate the gradient budget\.

Why the two stages are complementary\.Rescued advantage variance is of little use if correction cannot reach confident errors, and curvature\-compensated updates cannot manufacture signal from a collapsed group\. DEEPO therefore combines the two: semantic entropy detects the queries whose groups are likely to collapse and triggers grounded continuations, while sign\-aware Rényi entropy parameterizes how strongly each token is corrected or reinforced—signals at different levels, not interchangeable \(App\.[D](https://arxiv.org/html/2609.28570#A4)\)\. The interaction is positive on all six benchmarks and statistically significant exactly where both mechanisms are most stressed—long\-horizon VideoMMMU \(\+4\.0\+4\.0, 95% CI\[1\.1,6\.9\]\[1\.1,6\.9\]; Table[2](https://arxiv.org/html/2609.28570#S4.T2)\)—and consistent with additivity elsewhere\.

Our contributions are summarized as follows:

- •Diagnosis\.Two weak points in the RL correction chain: on hard queries, unanimously wrong groups collapse the group\-relative advantage to zero, and confident errors are gradient\-invisible \(Prop\.[1](https://arxiv.org/html/2609.28570#Thmproposition1)\)\.
- •Dual\-stage enhancement\.DEEPO pairs entropy\-triggered expert prefixes at the rollout stage with advantage\-sign\-aware Rényi gradient preconditioning at the optimization stage; the scaling weight is a conservative damped proximal step with guaranteed ascent of a sample\-wise surrogate \(Thm\.[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)\), and sign\-symmetric rules provably cannot realize its allocation \(Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2)\)\.
- •Evidence\.Complementary single\-branch gains with a positive interaction—significant on VideoMMMU, additive elsewhere—supported by design, fairness, and mechanism\-diagnosis controls, a self\-generated\-hint variant, multi\-seed evaluation, and efficiency accounting\.

## 2Related Work

Hallucination mitigation in MLLMs\.Hallucination remains a persistent challenge for MLLMs[Bai et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib1), addressed through attention calibration[Xu et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib2);[Zhu et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib17), decoding\-time corrections[Leng and others \(2024\)](https://arxiv.org/html/2609.28570#bib.bib11);[Sun and others \(2024\)](https://arxiv.org/html/2609.28570#bib.bib12), grounding supervision[Zhao and others \(2025\)](https://arxiv.org/html/2609.28570#bib.bib13);[Chen and others \(2023\)](https://arxiv.org/html/2609.28570#bib.bib14), and multi\-object or cross\-image analyses[Zhou and others \(2023\)](https://arxiv.org/html/2609.28570#bib.bib9);[Chen and others \(2024\)](https://arxiv.org/html/2609.28570#bib.bib10);[Park and others \(2025\)](https://arxiv.org/html/2609.28570#bib.bib15);[Li and others \(2025\)](https://arxiv.org/html/2609.28570#bib.bib16)\. These are post\-hoc or decoding\-level fixes; we instead detect hallucination\-prone uncertainty during rollout and ground the generation\.

Training dynamics, rewards, and uncertainty\.RL can exacerbate hallucinations when rewards are weakly tied to factual correctness[He et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib3);[Yang et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib4)or encourage shortcut exploitation[Miao et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib54);[Jia et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib55)\. Token\-level credit assignment has been approached via perception\-aware rewards[Deng and others \(2025\)](https://arxiv.org/html/2609.28570#bib.bib61);[Huang et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib29)and verifiable\-data synthesis[Wu et al\. \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib28); these modify rewards or data, whereas DEEPO leaves the reward unchanged and modifies rollout conditioning and gradient delivery\. Uncertainty signals have mainly served detection or calibration[Farquhar et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib6);[Manakul et al\. \(2023\)](https://arxiv.org/html/2609.28570#bib.bib56);[Fadeeva et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib53);[Zhang et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib5);[Qiu and Miikkulainen \(2024\)](https://arxiv.org/html/2609.28570#bib.bib18);[Niu et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib38);[Jiang et al\. \(2023\)](https://arxiv.org/html/2609.28570#bib.bib37);[Yang et al\. \(2026a\)](https://arxiv.org/html/2609.28570#bib.bib71);[Zhuang et al\. \(2026a\)](https://arxiv.org/html/2609.28570#bib.bib69), not for controlling how uncertainty shapes rollouts and updates during RL fine\-tuning\.

Positioning against three adjacent lines\.*\(i\) Entropy\-modulated policy gradients*\(EMPG[Wang et al\. \(2025a\)](https://arxiv.org/html/2609.28570#bib.bib7)and relatives\) reshape token updates via uncertainty but are sign\-symmetric \(Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2)\) with no rollout\-time mechanism\.*\(ii\) Hallucination\-targeted preference optimization*\(RLHF\-V[Yu et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib64), RLAIF\-V[Yu et al\. \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib63), HA\-DPO[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib59), hallucination\-targeted DPO[Fu et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib47), SymMPO[Anonymous \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib67), POVID[Zhou et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib62)\) acts on offline preference pairs with no generation\-time intervention; DEEPO is dynamics\-centric and on\-policy\.*\(iii\) Expert\-guided RL*[Zhuang et al\. \(2026b\)](https://arxiv.org/html/2609.28570#bib.bib72);[Zhang et al\. \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib8);[Anonymous \(2025a\)](https://arxiv.org/html/2609.28570#bib.bib66);[Yang et al\. \(2026b\)](https://arxiv.org/html/2609.28570#bib.bib70)mixes complete off\-policy expert traces into on\-policy RL at a static ratio—we adopt the confidence weightϕt\\phi\_\{t\}of[Zhang et al\. \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib8)—whereas DEEPO uses expert text only as truncated, answer\-free prefixes, triggered per query by an adaptive semantic\-entropy threshold, with hint dropout and the prefix masked out of the RL loss\. To our knowledge, DEEPO is the first to pair answer\-level semantic entropy as a rollout\-time trigger for expert conditioning with a token\-level sign\-aware Rényi scaling law\.

## 3Preliminaries

#### GRPO\.

We build on Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.28570#bib.bib30)\), a critic\-free RL algorithm: for each queryqqthe policy samplesGGresponses with verifiable rewardsrir\_\{i\}\(task accuracy plus format compliance, following[Feng et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib39)\); the group\-relative advantageAi=\(ri−mean\)/stdA\_\{i\}=\(r\_\{i\}\-\\mathrm\{mean\}\)/\\mathrm\{std\}is broadcast to every token, and the policy is updated with a PPO\-style clipped surrogate plus a KL penalty to a frozen reference\. Two properties matter: \(i\) advantages are*sequence\-level*—the signal cannot distinguish which tokens caused failure; \(ii\) with binary rewards a unanimous group carries no signal—if all responses are wrong \(ri=0r\_\{i\}=0\) thenμ=0\\mu=0,σ→0\\sigma\\to 0, andAi≈0A\_\{i\}\\approx 0\(symmetrically for an all\-correct group\), so the iteration extracts no gradient precisely from the hardest queries\.

#### Semantic entropy\.

Following[Farquhar et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib6), semantic entropy measures guessing at the level of meaning: drawNNgenerations, partition them intoKKmeaning\-equivalent clusters\{Ck\}\\\{C\_\{k\}\\\}, and computeHs\(q\)=−∑kP\(Ck∣q\)logP\(Ck∣q\)H\_\{s\}\(q\)=\-\\sum\_\{k\}P\(C\_\{k\}\\mid q\)\\log P\(C\_\{k\}\\mid q\)over the cluster distribution; a highHs​\(q\)H\_\{s\}\(q\)indicates semantic\-level guessing and elevated hallucination risk\. We estimateHsH\_\{s\}from theGGrollouts already sampled for the current iteration \(N=GN\{=\}Gby default; App\.[A\.3](https://arxiv.org/html/2609.28570#A1.SS3)co\-varies both\), so estimation adds no extra sampling\.

#### Rényi entropy\.

At orderα=2\\alpha=2, the collision entropyH2=−log∑yπ\(y∣q,y<t\)2H\_\{2\}=\-\\log\\sum\_\{y\}\\pi\(y\\mid q,y\_\{<t\}\)^\{2\}characterizes token\-level sharpness and governs the expected score\-gradient norm \(Prop\.[1](https://arxiv.org/html/2609.28570#Thmproposition1)\); it is the quantity we use for gradient scaling\.

## 4Method

DEEPO addresses the two weak points identified in Sec\.[1](https://arxiv.org/html/2609.28570#S1)as a dual\-stage enhancement \(Fig\.[2](https://arxiv.org/html/2609.28570#S4.F2)\): Sec\.[4\.1](https://arxiv.org/html/2609.28570#S4.SS1)gives the rollout stage, where semantic\-entropy\-triggered expert prefixes provide grounded supervision and reshape in\-group advantages on hard queries \(signal repair\); Sec\.[4\.2](https://arxiv.org/html/2609.28570#S4.SS2)gives the optimization stage, where advantage\-sign\-aware Rényi gradient preconditioning delivers correction to confident tokens \(delivery repair\); Sec\.[4\.3](https://arxiv.org/html/2609.28570#S4.SS3)states the complementarity hypothesis and its falsifiable predictions\. Appendix[D](https://arxiv.org/html/2609.28570#A4)relates the two entropy quantities through predictive uncertainty\.

![Refer to caption](https://arxiv.org/html/2609.28570v1/pic/Structure_Refined_v6.png)Figure 2:Overview of Dual\-Entropy Enhanced Policy Optimization\. High semantic entropy activatesϕt\\phi\_\{t\}\-weighted prefix regularization and prefix\-guided rollouts, while the Rényi entropy factor scales policy gradients; the objective unifies both\.### 4\.1Expert Hint as Signal Repair

The first weak point in the correction chain is a starved learning signal on hard queries: when semantic entropy is high the sampled group fails frequently, and in the extreme unanimously \(ri=0r\_\{i\}=0for allii, henceμ=0\\mu=0,σ→0\\sigma\\to 0, andAi=\(ri−μ\)/\(σ\+ϵ\)≈0A\_\{i\}=\(r\_\{i\}\-\\mu\)/\(\\sigma\+\\epsilon\)\\approx 0for every token\), so no gradient is extracted exactly where hallucination risk is highest\. Entropy\-triggered expert prefixes counteract this through two channels, separated explicitly because the evidence weighs them differently \(Sec\.[5\.4](https://arxiv.org/html/2609.28570#S5.SS4)\):*direct grounding and supervision*\(the prefix steers the policy onto a grounded reasoning path, reinforced by RL and distilled throughℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}, Eq\.[2](https://arxiv.org/html/2609.28570#S4.E2)\) and*in\-group advantage reshaping*\(under a shared group baseline, the high\-reward grounded continuation raises the group mean, so unhinted failures receive real negative advantages instead of the degenerateAi≈0A\_\{i\}\\approx 0of a collapsed group\)\.

#### Hint source\.

Hints are truncated prefixes of model\-generated solution trajectories from the Video\-R1 dataset\([Feng et al\., 2025](https://arxiv.org/html/2609.28570#bib.bib39)\); they condition generation during training only \(never at inference\) and additionally receive the lightweightϕt\\phi\_\{t\}\-weighted cross\-entropy of Eq\.[2](https://arxiv.org/html/2609.28570#S4.E2), so DEEPO is a hybrid of RL on self\-generated tokens and auxiliary prefix supervision, ablated separately \(Table[7](https://arxiv.org/html/2609.28570#A1.T7), App\.[A\.4](https://arxiv.org/html/2609.28570#A1.SS4); Table[2](https://arxiv.org/html/2609.28570#S4.T2)\)\. The method is not tied to this dataset: Sec\.[5\.4](https://arxiv.org/html/2609.28570#S5.SS4)evaluates a self\-generated variant \(Deepo\-self\) using the prefix of the best verified\-correct rollout from the same group\.

#### Trigger and prefix construction\.

We estimateHsH\_\{s\}from theNNrollouts already generated for the current iteration \(defaultN=GN\{=\}G\), clustering answer\-level commitments by pairwise NLI entailment \(DeBERTa\-MNLI; App\.[G](https://arxiv.org/html/2609.28570#A7)\)\. A query is marked as challenging whenHs​\(q\)H\_\{s\}\(q\)exceeds an adaptive thresholdτ^\\hat\{\\tau\}, initialized atτs\\tau\_\{s\}\(default0\.80\.8\) and updated per iteration by exponential moving average,τ^←\(1−β\)​τ^\+β​H¯s\\hat\{\\tau\}\\leftarrow\(1\-\\beta\)\\hat\{\\tau\}\+\\beta\\overline\{H\}\_\{s\}withH¯s\\overline\{H\}\_\{s\}the batch mean; the trigger gates the intervention to queries whose groups fail frequently, unlike a rate\-matched random trigger \(Table[3](https://arxiv.org/html/2609.28570#S5.T3)\)\. Each expert trace is split at the final\-answer marker into a thought segmentythought∗y^\{\*\}\_\{\\mathrm\{thought\}\}\(observations and chain\-of\-thought\) and an answer segmentyans∗y^\{\*\}\_\{\\mathrm\{ans\}\}\(conclusion and options\); the prefix is drawn*exclusively from the thought segment*, with uncertainty\-proportional length hard\-capped away from the answer boundary:

Lh=min⁡\(⌊αmax⋅Hs​\(q\)Hsmax⋅Lthought⌋,Lthought−1\),L\_\{h\}=\\min\\Big\(\\big\\lfloor\\alpha\_\{\\max\}\\cdot\\tfrac\{H\_\{s\}\(q\)\}\{H\_\{s\}^\{\\max\}\}\\cdot L\_\{\\mathrm\{thought\}\}\\big\\rfloor,\\;L\_\{\\mathrm\{thought\}\}\-1\\Big\),\(1\)whereHsmax=log⁡NH\_\{s\}^\{\\max\}=\\log N,Lthought=\|ythought∗\|L\_\{\\mathrm\{thought\}\}=\|y^\{\*\}\_\{\\mathrm\{thought\}\}\|, andαmax∈\(0,1\)\\alpha\_\{\\max\}\\in\(0,1\)\(default0\.50\.5\)\. Sinceαmax<1\\alpha\_\{\\max\}<1andHs​\(q\)≤HsmaxH\_\{s\}\(q\)\\leq H\_\{s\}^\{\\max\},Lh≤Lthought−1L\_\{h\}\\leq L\_\{\\mathrm\{thought\}\}\-1for every trace:the prefix provably excludes the final answer and conclusion tokens, and an NLI entailment filter additionally discards any prefix that semantically entails the gold answer \(App\.[G](https://arxiv.org/html/2609.28570#A7)\)\.The policy then generates the continuationyqrem∼πθ\(⋅∣q⊕yqpre\)y\_\{q\}^\{\\mathrm\{rem\}\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid q\\oplus y\_\{q\}^\{\\mathrm\{pre\}\}\); the full trajectory is scored by the verifiable reward \(one extra generation pass for triggered queries; cost in Table[14](https://arxiv.org/html/2609.28570#A7.T14), App\.[G](https://arxiv.org/html/2609.28570#A7)\)\.

#### Two channels of benefit\.

For a triggered query, the hint\-conditioned continuation and the first\-pass unhinted rollouts*share a single group\-relative baseline*, and the prefix additionally receivesℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}\.*\(i\) Direct grounding and supervision*: the continuation is generated by the policy itself and reinforced when successful, whileℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}distills the prefix tokens; this channel operates whenever a grounded trace is available—including the policy’s own correct rollouts \(Deepo\-self\)—and does not depend on group composition\.*\(ii\) In\-group advantage reshaping*: under the shared baseline the high\-reward continuation raisesμ\\muand restores advantage variance, so on a query whose unaided group would have failed unanimously, the failed unhinted rollouts—frozen at zero advantage—now receive substantial negative advantages; this channel punishes failures with real updates and by construction cannot separate two equally rewarded responses, and it requires the shared baseline \(Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(c\)\)\. Hint dropout regularly reverts triggered queries to purely unhinted groups\.

#### Empirical diagnosis of advantage collapse\.

On the training distribution, the fraction of all\-wrong groups rises monotonically from12%12\\%\(lowestHsH\_\{s\}decile\) to63%63\\%\(highest\), and on triggered queries whose first\-pass group was unanimously wrong \(48%48\\%of triggered queries\), hint activation converts60%60\\%to mixed groups, moving the mean advantage of failed unhinted rollouts from≈0\\approx 0to−0\.212\-0\.212and the within\-group advantage variance from00to0\.60\.6\(Table[4](https://arxiv.org/html/2609.28570#S5.T4), Sec\.[5\.4](https://arxiv.org/html/2609.28570#S5.SS4)\)\. One coverage boundary is worth stating plainly: a policy that is confidently and*consistently*wrong produces*low*semantic entropy and is not triggered; and if such a group is also unanimously wrong, its advantage vanishes, so the optimization stage—which can only reallocate a nonzero advantage, sinceg⋆​\(H2,t,At\)​At=0g^\{\\star\}\(H\_\{2,t\},A\_\{t\}\)\\,A\_\{t\}=0whenAt=0A\_\{t\}=0—has no signal to amplify either\. Neither branch manufactures reward information the rollouts do not contain; this boundary is shared by every outcome\-reward RL method, and the delivery branch acts on confident errors inside groups that carry nonzero advantage variance\.

#### Optimization treatment of the prefix\.

The prefix is masked out of the RL loss \(policy gradients update only self\-generated tokens\), and the guidance is distilled by an auxiliary prefix loss with confidence weightϕt=πθ​\(yq,tpre∣q,yq,<tpre\)​\(1−πθ​\(yq,tpre∣q,yq,<tpre\)\)\\phi\_\{t\}=\\pi\_\{\\theta\}\(y^\{\\mathrm\{pre\}\}\_\{q,t\}\\mid q,y^\{\\mathrm\{pre\}\}\_\{q,<t\}\)\\,\(1\-\\pi\_\{\\theta\}\(y^\{\\mathrm\{pre\}\}\_\{q,t\}\\mid q,y^\{\\mathrm\{pre\}\}\_\{q,<t\}\)\)\([Zhang et al\., 2025b](https://arxiv.org/html/2609.28570#bib.bib8)\):

ℒaux=𝔼\(q,ysol\)∼𝒟\[𝟙\{Lh\>0\}Lh∑t=1LhϕtℓCE\(yq,tpre;πθ\(⋅∣q,yq,<tpre\)\)\],\\mathcal\{L\}\_\{\\mathrm\{aux\}\}=\\mathbb\{E\}\_\{\(q,\\,y^\{\\mathrm\{sol\}\}\)\\sim\\mathcal\{D\}\}\\Big\[\\tfrac\{\\mathbb\{1\}\\\{L\_\{h\}\>0\\\}\}\{L\_\{h\}\}\\textstyle\\sum\_\{t=1\}^\{L\_\{h\}\}\\phi\_\{t\}\\,\\ell\_\{\\mathrm\{CE\}\}\\big\(y^\{\\mathrm\{pre\}\}\_\{q,t\};\\,\\pi\_\{\\theta\}\(\\cdot\\mid q,y^\{\\mathrm\{pre\}\}\_\{q,<t\}\)\\big\)\\Big\],\(2\)whereϕt\\phi\_\{t\}concentrates learning on tokens of intermediate confidence;ℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}is a weak regularizer and leaves the RL objective unchanged\.

### 4\.2Rényi Gradient Scaling as Delivery Repair

A diagnostic signal has limited effect if it cannot reach the parameters responsible for confident hallucinations\. We now make the delivery bottleneck precise and derive the scaling rule, starting from the second\-order Rényi entropy of the token\-level policy,H2\(π\(⋅∣q,y<t\)\)=−log∑y∈𝒱π\(y∣q,y<t\)2H\_\{2\}\(\\pi\(\\cdot\\mid q,y\_\{<t\}\)\)=\-\\log\\sum\_\{y\\in\\mathcal\{V\}\}\\pi\(y\\mid q,y\_\{<t\}\)^\{2\}, which for softmax policies pins the expected magnitude of the score\-function gradient:

###### Proposition 1\(Score\-gradient norm via collision entropy; standard identity\)\.

For a policyπθ\\pi\_\{\\theta\}parameterized by logitszθ​\(q,y<t\)z\_\{\\theta\}\(q,y\_\{<t\}\),

𝔼yt∼πθ\(⋅\|q,y<t\)\[∥∇zθ​\(q,y<t\)logπθ\(yt∣q,y<t\)∥2\]=1−exp\(−H2\(πθ\)\)\.\\mathbb\{E\}\_\{y\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\|q,y\_\{<t\}\)\}\\big\[\\big\\lVert\\nabla\_\{z\_\{\\theta\}\(q,y\_\{<t\}\)\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid q,y\_\{<t\}\)\\big\\rVert^\{2\}\\big\]=1\-\\exp\\big\(\-H\_\{2\}\(\\pi\_\{\\theta\}\)\\big\)\.\(3\)

This is the trace of the categorical Fisher information in natural parameters \(proof in App\.[B](https://arxiv.org/html/2609.28570#A2)\): hallucinated outputs are often*confidently wrong*, and by Eq\.[3](https://arxiv.org/html/2609.28570#S4.E3)such low\-entropy predictions induce near\-zero score gradients, so RL provides the weakest correction exactly where it is most needed\.

#### From the identity to a weighting rule\.

EMPG\([Wang et al\., 2025a](https://arxiv.org/html/2609.28570#bib.bib7)\)reweights token updates with an entropy\-only factor, which is symmetric in update direction\. We instead derive a sign\-aware weight from a*sample\-wise*proximal analysis in logit space, conditioned on the current batch of*already generated*trajectories: everyAtA\_\{t\}is a fixed empirical scalar, and no expectation overyt∼πθy\_\{t\}\\sim\\pi\_\{\\theta\}is taken withAtA\_\{t\}inside—the only distribution\-level quantity is the Fisher matrixFt=diag⁡\(π\)−π​π⊤⪰0F\_\{t\}=\\operatorname\{diag\}\(\\pi\)\-\\pi\\pi^\{\\top\}\\succeq 0, a deterministic function of the policy at the current state\. Defining

τt:=Tr⁡\(Ft\)=1−exp⁡\(−H2,t\),\\tau\_\{t\}:=\\Tr\(F\_\{t\}\)=1\-\\exp\(\-H\_\{2,t\}\),\(4\)the following theorem \(proved in App\.[C](https://arxiv.org/html/2609.28570#A3)\) justifies a curvature\-compensated weight under softmax saturation; the subsequent remark states what entropy\-only rules cannot do\.

###### Theorem 4\.1\(Curvature\-compensated weight as a conservative damped proximal step\)\.

Fix a generated batch \(hence all advantages\{At\}\\\{A\_\{t\}\\\}are fixed empirical scalars,At≠0A\_\{t\}\\neq 0\) and a decoding stepttwith statest=\(q,y<t\)s\_\{t\}=\(q,y\_\{<t\}\), sampled tokenyty\_\{t\}, and score gradientgt=∇zt​log​πθ​\(yt∣st\)g\_\{t\}=\\nabla\_\{z\_\{t\}\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)\. Restrict one\-step logit updates to the score direction,Δ​zt=η​wt​At​gt\\Delta z\_\{t\}=\\eta\\,w\_\{t\}A\_\{t\}g\_\{t\}withwt≥0w\_\{t\}\\geq 0, and consider the sample\-wise second\-order proximal surrogate with Levenberg–Marquardt dampingε0\>0\\varepsilon\_\{0\}\>0,Qt​\(wt\)=At​gt⊤​Δ​zt−βkl2​Δ​zt⊤​\(Ft\+ε0​I\)​Δ​ztQ\_\{t\}\(w\_\{t\}\)=A\_\{t\}\\,g\_\{t\}^\{\\top\}\\Delta z\_\{t\}\-\\tfrac\{\\beta\_\{\\mathrm\{kl\}\}\}\{2\}\\,\\Delta z\_\{t\}^\{\\top\}\(F\_\{t\}\+\\varepsilon\_\{0\}I\)\\,\\Delta z\_\{t\}\. Then:

- \(i\)*Sample\-wise curvature bound\.*gt⊤​Ft​gt≤τt​∥gt∥2g\_\{t\}^\{\\top\}F\_\{t\}g\_\{t\}\\leq\\tau\_\{t\}\\lVert g\_\{t\}\\rVert^\{2\}, so the trace\-scalarized quadraticQ~t​\(w\)=η​At2​∥gt∥2​w−βkl​η22​At2​∥gt∥2​\(τt\+ε0\)​w2\\widetilde\{Q\}\_\{t\}\(w\)=\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\\,w\-\\tfrac\{\\beta\_\{\\mathrm\{kl\}\}\\eta^\{2\}\}\{2\}A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\,w^\{2\}is a tangent minorant ofQtQ\_\{t\}onℝ\+\\mathbb\{R\}\_\{\+\}\.
- \(ii\)*Damped optimum\.*The minorant maximizerwtε0=\(βkl​η​\(τt\+ε0\)\)−1≤\(βkl​η​ε0\)−1=:wmaxw\_\{t\}^\{\\varepsilon\_\{0\}\}=\\big\(\\beta\_\{\\mathrm\{kl\}\}\\,\\eta\\,\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\big\)^\{\-1\}\\leq\(\\beta\_\{\\mathrm\{kl\}\}\\,\\eta\\,\\varepsilon\_\{0\}\)^\{\-1\}=:w\_\{\\max\}is capped and guarantees strict sample\-wise ascentQt​\(wtε0\)≥At2​∥gt∥2/\(2​βkl​\(τt\+ε0\)\)\>0=Qt​\(0\)Q\_\{t\}\(w\_\{t\}^\{\\varepsilon\_\{0\}\}\)\\geq A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}/\\big\(2\\beta\_\{\\mathrm\{kl\}\}\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\big\)\>0=Q\_\{t\}\(0\)\.
- \(iii\)*Softmax\-saturation regime\.*Withδt=1−πθ​\(yt∣st\)\\delta\_\{t\}=1\-\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\),∥gt∥2≤2​δt2\\lVert g\_\{t\}\\rVert^\{2\}\\leq 2\\delta\_\{t\}^\{2\}andτt≤2​δt\\tau\_\{t\}\\leq 2\\delta\_\{t\}: asδt→0\\delta\_\{t\}\\to 0the unweighted step stalls \(Qt′​\(0\)=O⁡\(δt2\)Q\_\{t\}^\{\\prime\}\(0\)=O\(\\delta\_\{t\}^\{2\}\)\), the compensated weight grows asΩ⁡\(δt−1\)\\Omega\(\\delta\_\{t\}^\{\-1\}\)up to its cap, and the certified ascent decays only asΘ⁡\(δt\)\\Theta\(\\delta\_\{t\}\)forδt≫ε0\\delta\_\{t\}\\gg\\varepsilon\_\{0\}; in the deep tailδt≪ε0\\delta\_\{t\}\\ll\\varepsilon\_\{0\}the bound reverts toO⁡\(δt2/ε0\)O\(\\delta\_\{t\}^\{2\}/\\varepsilon\_\{0\}\)—a rate improvement over the operational confidence band, not an asymptotic removal of saturation\.

Three readings are essential: the statement is*sample\-wise and local*\(AtA\_\{t\}never appears inside an expectation overyt∼πθy\_\{t\}\\sim\\pi\_\{\\theta\}; one step at one token of a fixed trajectory, not the global return\); the damping is*load\-bearing*\(the\+ε0\+\\varepsilon\_\{0\}term, exactly the stabilizer in Eq\.[5](https://arxiv.org/html/2609.28570#S4.E5), caps the weight atwmaxw\_\{\\max\}\); and the theorem is*sign\-blind*\(QtQ\_\{t\}seesAtA\_\{t\}only throughAt2A\_\{t\}^\{2\}\), justifying the*magnitude*of the corrective weight, not its sign asymmetry\.

Guided by Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)and Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2), we apply a sign\-aware Rényi factor to the advantage:

g⋆​\(H2,t,At\)=ψ⁡\(H2,t,At\)/ψ¯,ψ⁡\(H2,t,At\)=\{\(τt\+ε0\)−1,At<0\(correction\),τt,At≥0\(reinforcement\),g^\{\\star\}\(H\_\{2,t\},A\_\{t\}\)=\\psi\(H\_\{2,t\},A\_\{t\}\)\\Big/\\bar\{\\psi\},\\qquad\\psi\(H\_\{2,t\},A\_\{t\}\)=\\begin\{cases\}\\bigl\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\\bigr\)^\{\-1\},&A\_\{t\}<0\\quad\\text\{\(correction\)\},\\\\\[2\.0pt\] \\tau\_\{t\},&A\_\{t\}\\geq 0\\quad\\text\{\(reinforcement\)\},\\end\{cases\}\(5\)whereψ¯\\bar\{\\psi\}is the batch mean over all tokens,τt=1−e−H2,t\\tau\_\{t\}=1\-e^\{\-H\_\{2,t\}\}, andε0=1/wmax\\varepsilon\_\{0\}=1/w\_\{\\max\}with weight capwmax=20w\_\{\\max\}=20\(App\.[G](https://arxiv.org/html/2609.28570#A7)\)\. The negative branch is the damped proximal weight of Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)\(ii\) withβkl,η\\beta\_\{\\mathrm\{kl\}\},\\etaabsorbed by batch normalization\. Its effect is best read through*effective step magnitude*, not token selection: a confident token has∥gt∥=O⁡\(δt\)\\lVert g\_\{t\}\\rVert=O\(\\delta\_\{t\}\), so its unweighted logit step collapses as the prediction saturates, and weighting by\(τt\+ε0\)−1≈\(2​δt\+ε0\)−1\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)^\{\-1\}\\approx\(2\\delta\_\{t\}\+\\varepsilon\_\{0\}\)^\{\-1\}recovers the effective step∥Δ​zt∥∝wt​∥gt∥\\lVert\\Delta z\_\{t\}\\rVert\\propto w\_\{t\}\\lVert g\_\{t\}\\rVertto aδt\\delta\_\{t\}\-independent scale whereverδt≫ε0\\delta\_\{t\}\\gg\\varepsilon\_\{0\}\(Thm\.[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)\(iii\)\)\. The rule thus un\-stalls optimization on confident errors within the operational confidence regime; it does not remove saturation asymptotically \(at fixedε0\>0\\varepsilon\_\{0\}\>0the weight saturates atwmaxw\_\{\\max\}and the step still vanishes asδt→0\\delta\_\{t\}\\to 0\)\. This amplification is*not*error\-selective by construction—low\-entropy structural tokens \(articles, connectives\) in failed trajectories are amplified as well—but three properties bound that collateral: the cap atwmaxw\_\{\\max\}keeps amplified steps proportional to an already\-small∥gt∥\\lVert g\_\{t\}\\rVert; the KL penalty to the frozen reference is deliberately left unweighted \(Sec\.[4\.3](https://arxiv.org/html/2609.28570#S4.SS3)\), anchoring high\-frequency tokens; andℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}adds an orthogonal cross\-entropy pull on fluent text\. The residual selectivity is empirical: on span\-annotated failed rollouts the updates achieve a1\.751\.75–1\.80×1\.80\\timesper\-token*weight*enrichment on hallucinated spans \(95% CIs exclude11; App\.[A\.5](https://arxiv.org/html/2609.28570#A1.SS5)\), i\.e\. a1\.151\.15–1\.25×1\.25\\timesenrichment of the effective logit\-step norm—a deliberately bounded effect: restoring a stalled gradient by roughly a fifth suffices, over thousands of updates, to bias the policy away from suboptimal deterministic attractors\. The positive branch is a separate, deliberately simple heuristic: reinforce in proportion to the available gradient signal𝔼​∥g∥2=τt\\mathbb\{E\}\\lVert g\\rVert^\{2\}=\\tau\_\{t\}, dampening reinforcement of confident tokens while leaving high\-uncertainty tokens near the unweighted baseline; confident correct tokens receive small positive weights, slowing further sharpening and acting against entropy collapse\. Batch\-mean normalization keeps the weights centered at one, so the rule reallocates update magnitude across tokens rather than rescaling the batch mean;g⋆\>0g^\{\\star\}\>0preserves the advantage sign, so the PPO clip*trigger set*is unchanged\.

#### Interaction with downstream optimization \(Adam and PPO\)\.

Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)characterizes a single damped proximal step; the deployed pipeline composes it with Adam and PPO clipping, which the theorem does not model\. Two observations keep the design meaningful:g⋆g^\{\\star\}acts in the*spatial*dimension \(per token, within a batch\) whereas Adam’s normalization acts*temporally*\(running statistics across steps\), rescaling each token’s update by roughly a common factor so the relative, mean\-one enrichment survives and steers subsequent updates; andg⋆\>0g^\{\\star\}\>0leaves the clip trigger set unchanged, so clipping intervenes on trajectories, not on the weighting\. We therefore read the theorem as a design\-level preconditioning argument, not a guarantee about the composed optimizer \(details in App\.[G](https://arxiv.org/html/2609.28570#A7)\)\.

### 4\.3The Unified Objective

#### Unified objective\.

The final training loss \(minimized\) combines the GRPO surrogate on self\-generated continuations with the weighted prefix regularizer:

ℒ=𝔼𝒟\[λℒaux−1\|yqrem\|∑t=1\|yqrem\|min\(rtA~t,clip\(rt,1−εclip,1\+εclip\)A~t\)\]\+βkl𝔼DKL\(πθ∥πref\),\\mathcal\{L\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\Big\[\\lambda\\,\\mathcal\{L\}\_\{\\mathrm\{aux\}\}\-\\tfrac\{1\}\{\|y\_\{q\}^\{\\mathrm\{rem\}\}\|\}\\textstyle\\sum\_\{t=1\}^\{\|y\_\{q\}^\{\\mathrm\{rem\}\}\|\}\\min\\big\(r\_\{t\}\\tilde\{A\}\_\{t\},\\mathrm\{clip\}\(r\_\{t\},1\{\-\}\\varepsilon\_\{\\mathrm\{clip\}\},1\{\+\}\\varepsilon\_\{\\mathrm\{clip\}\}\)\\tilde\{A\}\_\{t\}\\big\)\\Big\]\+\\beta\_\{\\mathrm\{kl\}\}\\,\\mathbb\{E\}\\,D\_\{\\mathrm\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\mathrm\{ref\}\}\),\(6\)wherertr\_\{t\}is the policy ratio,A~t=g⋆​\(H2,t,At\)⋅At\\tilde\{A\}\_\{t\}=g^\{\\star\}\(H\_\{2,t\},A\_\{t\}\)\\cdot A\_\{t\}the scaled advantage, andβkl=0\.04\\beta\_\{\\mathrm\{kl\}\}=0\.04; untriggered queries haveLh=0L\_\{h\}=0\. The KL term is deliberately*not*scaled byg⋆g^\{\\star\}: correction tokens are allowed to move further from the reference policy \(App\.[C](https://arxiv.org/html/2609.28570#A3)\)\.

#### Complementarity hypothesis\.

If the two stages repair distinct weak points, then \(P1\) grounded hints should help while shuffled hints do not \(Table[8](https://arxiv.org/html/2609.28570#A1.T8), App\.[A\.4](https://arxiv.org/html/2609.28570#A1.SS4)\); \(P2\) the sign\-aware rule should beat symmetric and entropy\-only weighting \(Table[3](https://arxiv.org/html/2609.28570#S5.T3)\); and \(P3\) the interaction should be positive and largest on long\-horizon tasks \(Table[2](https://arxiv.org/html/2609.28570#S4.T2)\)\. All three are borne out in Sec\.[5](https://arxiv.org/html/2609.28570#S5)\.

Table 1:Results on six benchmarks \(↑\\uparrow: higher is better\)\. Open\-source rows are mean±\\pmstd over four runs \(training seeds for fine\-tuned models; bootstrap over examples for frozen baselines\); closed\-source rows are single\-run\. Bold: best among training strategies on the Qwen2\.5\-VL\-7B backbone\.Hallucination Evaluations

General Reasoning

Table 2:Ablation of the two branches and their interaction \(↑\\uparrow: higher is better\)\. All variants use Qwen2\.5\-VL\-7B; all rows are four\-seed means\. “w/o Expert Hint” removes the whole signal\-repair branch; “w/o Gradient Scaling” removes only the delivery branch, keeping the entropy\-triggered prefixes, prefix regularizer, and dropout\. The last row reports the interactionI=Sfull−Shint−Sscaling\+SGRPOI=S\_\{\\mathrm\{full\}\}\-S\_\{\\mathrm\{hint\}\}\-S\_\{\\mathrm\{scaling\}\}\+S\_\{\\mathrm\{GRPO\}\}\. The 95% CIs are\[−0\.4,1\.2\]\[\-0\.4,1\.2\]/\[−1\.3,1\.7\]\[\-1\.3,1\.7\]/\[−0\.3,3\.1\]\[\-0\.3,3\.1\]/\[−0\.2,1\.0\]\[\-0\.2,1\.0\]/\[−0\.8,1\.2\]\[\-0\.8,1\.2\]/\[1\.1,6\.9\]\[1\.1,6\.9\]\(last bold\); only VideoMMMU excludes zero\.Hallucination Evaluations

General Reasoning

## 5Experiments

### 5\.1Experiment Setup

Baselines, datasets, and protocol\.We compare with closed\-source models \(GPT\-4o[OpenAI \(2024\)](https://arxiv.org/html/2609.28570#bib.bib19), Gemini\-1\.5\-Pro[Team et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib20)\), open\-source MLLMs \(LLaVA\-OneVision[Liu et al\. \(2024b\)](https://arxiv.org/html/2609.28570#bib.bib21), InternVL2\.5[Chen et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib22), LLaMA3\.2[Dubey et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib23)\), recent multimodal reasoning models \(R1\-OneVision[Wang et al\. \(2025b\)](https://arxiv.org/html/2609.28570#bib.bib24), R1\-VL[Zhang et al\. \(2025a\)](https://arxiv.org/html/2609.28570#bib.bib25), Vision\-R1[Zhan et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib26), Video\-R1[Feng et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib39)\), and, on the Qwen2\.5\-VL\-7B backbone[Bai et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib27), the training strategies SFT, GRPO[Shao et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib30), DAPO[Yu et al\. \(2025a\)](https://arxiv.org/html/2609.28570#bib.bib68), PAPO[Deng and others \(2025\)](https://arxiv.org/html/2609.28570#bib.bib61), and DEEPO\. We train on the Video\-R1 multimodal RL dataset[Feng et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib39)and evaluate on POPE[Li et al\. \(2023\)](https://arxiv.org/html/2609.28570#bib.bib31), VideoHallucer[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib32), HallusionBench[Guan et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib33)\(hallucination\) and MMBench[Liu et al\. \(2023\)](https://arxiv.org/html/2609.28570#bib.bib34), MMSTAR[Liu et al\. \(2024a\)](https://arxiv.org/html/2609.28570#bib.bib35), VideoMMMU[Li et al\. \(2024a\)](https://arxiv.org/html/2609.28570#bib.bib36)\(general reasoning\)\. Fine\-tuned models are means over four training seeds with greedy decoding; frozen baselines are evaluated once with bootstrap standard deviations \(Table[1](https://arxiv.org/html/2609.28570#S4.T1)\); the full implementation specification is in App\.[G](https://arxiv.org/html/2609.28570#A7)\.

### 5\.2Performance

Table[1](https://arxiv.org/html/2609.28570#S4.T1)reports results on three hallucination\-oriented and three general reasoning benchmarks\. DEEPO consistently achieves highly competitive performance across all benchmarks, setting new state\-of\-the\-art results among the evaluated Qwen\-based training strategies and outperforming GRPO, DAPO, and PAPO under the same backbone \(85\.8/57\.6/69\.7 on hallucination evaluations; 86\.4/60\.8/53\.3 on general reasoning\)—reducing hallucination does not trade off reasoning accuracy; the margin over GRPO’s four\-seed mean exceeds one seed standard deviation on five of six benchmarks \(largest on VideoMMMU,\+8\.6\+8\.6\), and the per\-seed protocol is summarized in Table[1](https://arxiv.org/html/2609.28570#S4.T1)\.

### 5\.3Ablation and Interaction

Table[2](https://arxiv.org/html/2609.28570#S4.T2)ablates the two branches and reports their interaction\.*The hint branch drives the reasoning\-heavy gains*\(w/o Gradient Scalingimproves five of six benchmarks, most sharply on VideoMMMU,\+4\.6\+4\.6\);*the scaling branch adds complementary gains*\(w/o Expert Hintimproves five of six and is flat on VideoMMMU; added on top of hints it lifts VideoMMMU from49\.349\.3to53\.353\.3\)\. The interaction is positive in the four\-seed means on all six benchmarks but statistically significant only on long\-horizon VideoMMMU; elsewhere the combination is consistent with additive gains\.

Table 3:Delivery\-branch design controls and trigger ablation \(four\-seed means, Qwen2\.5\-VL\-7B\); row 5 uses a rate\-matched random trigger\.Table 4:Mechanism diagnosis on the training distribution \(four\-seed pools\): \(a\) all\-wrong group fraction \(%\) and mean group reward byHsH\_\{s\}decile \(GRPO baseline\); \(b\) hint activation on triggered, originally all\-wrong queries; \(c\) baseline grouping ablation; \(d\) equal\-compute and equal\-supervision baselines\. All rows in \(c\)–\(d\) run the identical rollout pipeline of Algorithm[1](https://arxiv.org/html/2609.28570#alg1)\(G=8G\{=\}8unhinted rollouts plus one hint\-conditioned continuation per triggered query\); the “mixed” row is the default model itself, not a rerun, and in the “separated” variant the unhinted group is normalized within itself while the singleton hinted continuation is scored against a batch\-level EMA baseline over hint continuations—so no advantage is zeroed by construction and only the counterfactual penalty channel is removed\.HsH\_\{s\}decile12345678910All\-wrong \(%\)12162127333945515763Mean reward0\.760\.690\.610\.530\.460\.400\.340\.290\.240\.19

### 5\.4Controls and Mechanism Evidence

#### Design controls\.

Replacing the sign\-aware weight with any symmetric or sign\-only alternative degrades all three benchmarks, as does replacing the entropy trigger with a rate\-matched random one \(Table[3](https://arxiv.org/html/2609.28570#S5.T3)\); likewise, entropy\-triggered hints improve over the base method but trail full DEEPO \(Table[7](https://arxiv.org/html/2609.28570#A1.T7), App\.[A\.4](https://arxiv.org/html/2609.28570#A1.SS4)\)\. Conditioning the weight on the advantage sign—not entropy alone—is what delivers the correction, and the trigger gates the intervention to where it is needed\. The margin over EMPG’s entropy\-only factor is modest on POPE \(\+0\.5\+0\.5\) and largest on long\-horizon VideoMMMU \(\+1\.8\+1\.8\); the Rényi\-2 identity behind both rules is standard \(Prop\.[1](https://arxiv.org/html/2609.28570#Thmproposition1)\), so the added value lies in the sign\-aware allocation and the rollout\-stage trigger, not in the identity itself\.

#### Mechanism diagnosis\.

The rollout\-stage diagnosis of Sec\.[4\.1](https://arxiv.org/html/2609.28570#S4.SS1)is quantified in Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(a,b\): all\-wrong groups rise from12%12\\%to63%63\\%acrossHsH\_\{s\}deciles, and hints convert60%60\\%of triggered all\-wrong groups into mixed ones\. Removing the shared baseline—with the identical rollout pipeline, and the singleton hinted continuation scored against a batch\-level EMA baseline so that no advantage is zeroed by construction—degrades all three tracked benchmarks \(Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(c\), whose mixed row is the default model itself\), isolating the counterfactual penalty on unhinted failures\.

#### Fairness controls\.

The gains are not explained by extra compute or expert data \(Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(d\)\): matching wall\-clock with more GRPO steps, or matching traces with offline SFT before GRPO, both trail DEEPO, with the residual gap concentrated on VideoMMMU \(\+2\.1\+2\.1\)—offline SFT force\-fits full trajectories and shifts the policy off its on\-policy manifold, whereas DEEPO distills the same traces smoothly inside a single RL stage, avoiding the exploration collapse typical of multi\-stage SFT\-then\-RL pipelines\.*Paired analysis\.*Across the four seeds, DEEPO leads SFT\+GRPO on 4/4 seeds on VideoMMMU \(paired bootstrap 95% CI\[\+1\.2,\+3\.0\]\[\+1\.2,\+3\.0\]\) and on 3/4 seeds on POPE and HallusionBench \(CIs\[−0\.2,\+1\.4\]\[\-0\.2,\+1\.4\],\[−0\.1,\+0\.9\]\[\-0\.1,\+0\.9\]\)\. Further controls \(App\.[A\.4](https://arxiv.org/html/2609.28570#A1.SS4), App\.[A\.5](https://arxiv.org/html/2609.28570#A1.SS5)\): shuffled hints show no distinct advantage over GRPO;Deepo\-self retains≈71%\\approx 71\\%of the VideoMMMU gain without expert data—consistent with the direct\-grounding channel dominating, though source, quality, and coverage also differ between the variants, so the residual gap bounds rather than isolates the collapsed\-group channel; amplified updates are enriched1\.751\.75–1\.80×1\.80\\timeson span\-annotated hallucinated content at the weight level \(1\.151\.15–1\.25×1\.25\\timesat the effective\-step level\); results transfer to InternVL2\.5\-8B \(85\.085\.0vs\.83\.483\.4GRPO on POPE\)\.

## 6Conclusion

We studied hallucination in RL of MLLMs as two weak points of the correction chain—on hard queries, unanimously wrong groups collapse the group\-relative advantage to zero, and confident errors are gradient\-invisible—and proposed DEEPO, a dual\-stage enhancement pairing entropy\-triggered expert prefixes at rollout time with advantage\-sign\-aware Rényi gradient preconditioning at optimization time\. Both branches improve over GRPO individually; their interaction is significant on long\-horizon VideoMMMU and additive elsewhere; and the controls of Sec\.[5](https://arxiv.org/html/2609.28570#S5)isolate grounding content as the active ingredient, with stable training at\+18%\+18\\%per\-step cost over GRPO \(App\.[G](https://arxiv.org/html/2609.28570#A7)\)\. The guarantee currently concerns a local sample\-wise surrogate rather than the global expected return\.

## References

- Amari and Nagaoka \(2000\)S\. Amari and H\. NagaokaMethods of information geometry\.American Mathematical Society / Oxford University Press\.Cited by:[Appendix B](https://arxiv.org/html/2609.28570#A2.p1.4)\.
- Anonymous \(2025a\)AnonymousSRFT: a single\-stage method with supervised and reinforcement fine\-tuning for reasoning\.arXiv preprint arXiv:2506\.19767\.External Links:[Link](https://arxiv.org/abs/2506.19767)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Anonymous \(2025b\)AnonymousSymMPO: symmetric multimodal preference optimization for hallucination mitigation in lvlms\.arXiv preprint\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Baiet al\.\(2025\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. LinQwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.External Links:[Link](https://arxiv.org/abs/2502.13923)Cited by:[Table 13](https://arxiv.org/html/2609.28570#A7.T13.2.2.2.1),[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Baiet al\.\(2024\)Z\. Bai, P\. Wang, T\. Xiao, T\. He, Z\. Han, Z\. Zhang, and M\. Z\. ShouHallucination of multimodal large language models: a survey\.arXiv preprint arXiv:2404\.18930\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Banget al\.\(2025\)Y\. Bang, Z\. Ji, A\. Schelten, A\. Hartshorn, T\. Fowler, C\. Zhang, N\. Cancedda, and P\. FungHallulens: llm hallucination benchmark\.arXiv preprint arXiv:2504\.17550\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Chenet al\.\(2024\)T\. Chen, Z\. Zhang, J\. Wang,et al\.Expanding performance boundaries of open\-source multimodal large language models with internvl 2\.5\.arXiv preprint arXiv:2412\.05271\.External Links:[Link](https://arxiv.org/abs/2412.05271)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Chenet al\.\(2024\)X\. Chenet al\.Multi\-object hallucination in vision\-language models\.arXiv preprint arXiv:2407\.06192\.External Links:[Link](https://arxiv.org/abs/2407.06192)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Chenet al\.\(2023\)Z\. Chenet al\.Mitigating hallucination in visual language models with visual supervision\.arXiv preprint arXiv:2311\.16479\.External Links:[Link](https://arxiv.org/abs/2311.16479)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Cossio \(2025\)M\. CossioA comprehensive taxonomy of hallucinations in large language models\.arXiv preprint arXiv:2508\.01781\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Danget al\.\(2025\)H\. A\. Dang, V\. Tran, and L\. NguyenSurvey and analysis of hallucinations in large language models: attribution to prompting strategies or model behavior\.Frontiers in Artificial Intelligence8,pp\. 1622292\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Denget al\.\(2025\)Y\. Denget al\.Perception\-aware policy optimization for multimodal reasoning\.arXiv preprint arXiv:2507\.06448\.External Links:[Link](https://arxiv.org/abs/2507.06448)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1),[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey,et al\.Llama 3\.2: revolutionizing edge ai and vision with open models\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Fadeevaet al\.\(2024\)E\. Fadeeva, A\. Rubashevskii, A\. Shelmanov, S\. Petrakov, H\. Li, H\. Mubarak, E\. Tsymbalov, G\. Kuzmin, A\. Panchenko, T\. Baldwin, P\. Nakov, and M\. PanovFact‑checking the output of large language models via token‑level uncertainty quantification\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics: Findings of the Association for Computational Linguistics,pp\. 9367–9385\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.558)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Farquharet al\.\(2024\)S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. GalDetecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1),[§3](https://arxiv.org/html/2609.28570#S3.SS0.SSS0.Px2.p1.1)\.
- Fedotovet al\.\(2003\)A\. A\. Fedotov, P\. Harremoës, and F\. TopsøeRefinements of pinsker’s inequality\.IEEE Transactions on Information Theory49\(6\),pp\. 1491–1498\.Cited by:[Appendix E](https://arxiv.org/html/2609.28570#A5.p1.1)\.
- Fenget al\.\(2025\)K\. Feng, K\. Gong, B\. Li, Z\. Guo, Y\. Wang, T\. Peng, J\. Wu, X\. Zhang, B\. Wang, and X\. YueVideo\-r1: reinforcing video reasoning in mllms\.InProceedings of the Thirty\-Ninth Conference on Neural Information Processing Systems \(NeurIPS 2025\),External Links:[Link](https://arxiv.org/abs/2503.21776)Cited by:[Appendix G](https://arxiv.org/html/2609.28570#A7.SSx4.p1.1),[Table 13](https://arxiv.org/html/2609.28570#A7.T13.2.3.2.1),[§1](https://arxiv.org/html/2609.28570#S1.p3.1),[§3](https://arxiv.org/html/2609.28570#S3.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.28570#S4.SS1.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Fuet al\.\(2025\)Y\. Fu, R\. Xie, X\. Sun, Z\. Kang, and X\. LiMitigating hallucination in multimodal large language model via hallucination\-targeted direct preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 16563–16577\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p1.1),[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Guanet al\.\(2024\)W\. Guan, Y\. Wang, Z\. Wu, Y\. Zhang, W\. Wang,et al\.HallusionBench: a multi\-modal benchmark for hallucination evaluation in large vision\-language models\.arXiv preprint arXiv:2402\.15300\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Harremoës and Topsøe \(2001\)P\. Harremoës and F\. TopsøeInequalities between entropy and index of coincidence\.IEEE Transactions on Information Theory47\(7\),pp\. 2944–2960\.Cited by:[Appendix E](https://arxiv.org/html/2609.28570#A5.SS0.SSS0.Px2),[Appendix E](https://arxiv.org/html/2609.28570#A5.p1.1),[Proposition 2](https://arxiv.org/html/2609.28570#Thmproposition2.p1.1.1)\.
- Heet al\.\(2024\)L\. He, Z\. Chen, Z\. Shi, T\. Yu, J\. Shao, and L\. ShengA topic\-level self\-correctional approach to mitigate hallucinations in mllms\.arXiv preprint arXiv:2411\.17265\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Huanget al\.\(2025\)S\. Huang, X\. Qu, Y\. Li, Y\. Luo, Z\. He, D\. Liu, and Y\. ChengSpotlight on token perception for multimodal reinforcement learning\.arXiv preprint arXiv:2510\.09285\.External Links:[Link](https://arxiv.org/abs/2510.09285)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Jiaet al\.\(2025\)R\. Jia, Y\. Yang, Y\. Gai, K\. Luo, S\. Huang, J\. Lin, X\. Jiang, and J\. GuanjunWriting\-zero: bridge the gap between non\-verifiable tasks and verifiable rewards\.arXiv preprint arXiv:2506\.00103\.Note:PreprintCited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Jianget al\.\(2023\)M\. Jiang, Y\. Ruan, S\. Huang, S\. Liao, S\. Pitis, R\. Grosse, and J\. BaCalibrating language models via augmented prompt ensembles \(cape\)\.InProceedings of the ICML Workshop on Deployment Challenges for Generative AI,External Links:[Link](https://openreview.net/pdf?id=L0dc4wqbNs)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Jianget al\.\(2024\)Y\. Jiang, L\. Tang, H\. Xu, X\. Wang, W\. Li, P\. Gao, and Y\. QiaoVideoHallucer: benchmarking and mitigating hallucinations in video large language models\.arXiv preprint arXiv:2406\.06673\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Kalaiet al\.\(2025\)A\. T\. Kalai, O\. Nachum, S\. S\. Vempala, and E\. ZhangWhy language models hallucinate\.arXiv preprint arXiv:2509\.04664\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p3.1)\.
- Khan and Fu \(2024\)Z\. Khan and Y\. FuConsistency and uncertainty: identifying unreliable responses from black\-box vision\-language models for selective visual question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p3.1)\.
- Lenget al\.\(2024\)S\. Lenget al\.Mitigating object hallucinations in large vision\-language models through visual contrastive decoding\.arXiv preprint arXiv:2311\.16922\.External Links:[Link](https://arxiv.org/abs/2311.16922)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Liet al\.\(2023\)J\. Li, D\. Li, H\. Li, and S\. C\. HoiEvaluating object hallucination in large vision\-language models\.arXiv preprint arXiv:2305\.10355\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Liet al\.\(2024a\)X\. Li, Q\. Zhu, X\. Wang, L\. Tang, P\. Gao, and Y\. QiaoVideoMMMU: a comprehensive evaluation benchmark for video\-based massive multimodal understanding\.arXiv preprint arXiv:2406\.11636\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Liet al\.\(2025\)Y\. Liet al\.Evaluating and mitigating object hallucination in large vision\-language models\.Proceedings of ACL 2025\.External Links:[Link](https://aclanthology.org/2025.acl-main.299/)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Liet al\.\(2024b\)Z\. Li, D\. Liu, C\. Zhang, H\. Wang, T\. Xue, and W\. CaiEnhancing advanced visual reasoning ability of large language models\.arXiv preprint arXiv:2409\.13980\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p1.1)\.
- Linet al\.\(2024\)S\. Lin, L\. Gao, B\. Oguz, W\. Xiong, J\. Lin, W\. Yih, and X\. ChenFlame: factuality\-aware alignment for large language models\.Advances in Neural Information Processing Systems37,pp\. 115588–115614\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Liuet al\.\(2024a\)H\. Liu, Y\. Gong, Z\. Wu, P\. Zhang,et al\.MMStar: benchmarking multimodal large language models for massive multimodal understanding\.arXiv preprint arXiv:2406\.19678\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, D\. Li, and P\. ZhangMMBench: is your multi\-modal model an all\-around player?\.arXiv preprint arXiv:2307\.06281\.Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Liuet al\.\(2024b\)H\. Liu, C\. Li, Y\. Li, and Y\. J\. LeeLLaVA\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.External Links:[Link](https://arxiv.org/abs/2408.03326)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Mahmoudet al\.\(2025\)O\. Mahmoud, A\. Khalil, B\. L\. Semage, T\. G\. Karimpanal, and S\. RanaThe unintended trade\-off of ai alignment: balancing hallucination mitigation and safety in llms\.arXiv preprint arXiv:2510\.07775\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p1.1),[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Manakulet al\.\(2023\)P\. Manakul, A\. Liusie, and M\. J\. F\. GalesSelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Miaoet al\.\(2025\)Y\. Miao, S\. Zhang, L\. Ding, Y\. Zhang, L\. Zhang, and D\. TaoThe energy loss phenomenon in rlhf: a new perspective on mitigating reward hacking\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Niuet al\.\(2024\)R\. Niu, D\. Wu, R\. Yu, and Y\. MaFunctional‑level uncertainty quantification for calibrated fine‑tuning on llms \(uq4ct\)\.arXiv preprint arXiv:2410\.06431\.External Links:[Link](https://arxiv.org/abs/2410.06431)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- OpenAI \(2024\)OpenAIGPT\-4o system card\.arXiv preprint arXiv:2410\.21276\.External Links:[Link](https://arxiv.org/abs/2410.21276)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Parket al\.\(2025\)J\. Parket al\.Mitigating hallucination in multimodal reasoning via selective contrastive decoding\.arXiv preprint arXiv:2502\.01056\.External Links:[Link](https://arxiv.org/abs/2502.01056)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Qiu and Miikkulainen \(2024\)X\. Qiu and R\. MiikkulainenSemantic density: uncertainty calibration in llms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2405\.13845,[Link](https://arxiv.org/abs/2405.13845)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. GuoGroup relative policy optimization\.arXiv preprint\.Cited by:[Appendix G](https://arxiv.org/html/2609.28570#A7.SSx2.p1.1),[§3](https://arxiv.org/html/2609.28570#S3.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Shenfeldet al\.\(2025\)I\. Shenfeld, J\. Pari, and P\. AgrawalRL’s razor: why online reinforcement learning forgets less\.arXiv preprint arXiv:2509\.04259\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Simpson \(1949\)E\. H\. SimpsonMeasurement of diversity\.Nature163,pp\. 688\.Cited by:[Appendix B](https://arxiv.org/html/2609.28570#A2.p1.4)\.
- Sunet al\.\(2024\)W\. Sunet al\.Alleviating hallucination in large vision\-language models via multi\-view multi\-path reasoning\.arXiv preprint arXiv:2408\.17150\.External Links:[Link](https://arxiv.org/abs/2408.17150)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Teamet al\.\(2024\)G\. Team, G\. DeepMind, and GoogleGemini 1\.5: unlocking multimodal understanding across millions of tokens\.arXiv preprint arXiv:2403\.05530\.External Links:[Link](https://arxiv.org/abs/2403.05530)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Wanget al\.\(2025a\)J\. Wang, J\. Liu, Y\. Fu, Y\. Li, X\. Wang, Y\. Lin, Y\. Yue, L\. Zhang, Y\. Wang, and K\. WangHarnessing uncertainty: entropy\-modulated policy gradients for long\-horizon llm agents\.arXiv preprint arXiv:2509\.09265\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1),[§4\.2](https://arxiv.org/html/2609.28570#S4.SS2.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Y\. Wang L\. Zhanget al\.R1\-onevision: advancing generalized multimodal reasoning with cross\-modal alignment\.arXiv preprint arXiv:2503\.10615\.External Links:[Link](https://arxiv.org/abs/2503.10615)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Wuet al\.\(2025a\)J\. Wu, Y\. Shen, S\. Liu, Y\. Tang, S\. Song, X\. Wang, and L\. CaiImprove decoding factuality by token\-wise cross layer entropy of large language models\.arXiv preprint arXiv:2502\.03199\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p2.1)\.
- Wuet al\.\(2024\)J\. Wu, M\. Zhong, S\. Xing, Z\. Lai, Z\. Liu, Z\. Chen, W\. Wang, X\. Zhu, L\. Lu, T\. Lu,et al\.Visionllm v2: an end\-to\-end generalist multimodal large language model for hundreds of vision\-language tasks\.Advances in Neural Information Processing Systems37,pp\. 69925–69975\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p1.1)\.
- Wuet al\.\(2025b\)Z\. Wu, J\. Ni, X\. Liu, Z\. Liu, H\. Yan, and M\. Q\. ShiehSynthRL: scaling visual reasoning with verifiable data synthesis\.arXiv preprint arXiv:2506\.02096\.External Links:[Link](https://arxiv.org/abs/2506.02096)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Xuet al\.\(2025\)X\. Xu, H\. Chen, M\. Lyu, S\. Zhao, Y\. Xiong, Z\. Lin, J\. Han, and G\. DingMitigating hallucinations in multi\-modal large language models via image token attention\-guided decoding\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 1571–1590\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Yanget al\.\(2026a\)J\. Yang, W\. Gan, Y\. Zhuang, W\. Zhang, J\. Chen, and X\. ZhangReconciling process supervision with outcome\-based credit in agentic policy optimization\.External Links:2608\.31077,[Link](https://arxiv.org/abs/2608.31077)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Yanget al\.\(2026b\)J\. Yang, D\. He, M\. Pan, K\. Yao, G\. Su, W\. Zhang, Y\. Hu, T\. Li, Y\. Li, and X\. ZhangSARE: sample\-wise adaptive reasoning for training\-free fine\-grained visual recognition\.External Links:2603\.17729,[Link](https://arxiv.org/abs/2603.17729)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Yanget al\.\(2025\)Z\. Yang, X\. Luo, D\. Han, Y\. Xu, and D\. LiMitigating hallucinations in large vision\-language models via dpo: on\-policy data hold the key\.arXiv preprint arXiv:2501\.09695\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Yinet al\.\(2024\)S\. Yin, C\. Fu, S\. Zhao, K\. Li, X\. Sun, T\. Xu, and E\. ChenA survey on multimodal large language models\.National Science Review11\(12\),pp\. nwae403\.Cited by:[§1](https://arxiv.org/html/2609.28570#S1.p1.1)\.
- Yuet al\.\(2025a\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Zhang, T\. Fan, G\. Liu, L\. Liu,et al\.DAPO: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.External Links:[Link](https://arxiv.org/abs/2503.14476)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Yuet al\.\(2024\)T\. Yu, Y\. Yao, H\. Zhang, T\. He, Y\. Han, G\. Cui,et al\.RLHF\-v: towards trustworthy mllms via behavior alignment from fine\-grained correctional human feedback\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:[Link](https://arxiv.org/abs/2312.00849)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Yuet al\.\(2025b\)T\. Yu, H\. Zhang, Y\. Yao,et al\.RLAIF\-v: aligning mllms through open\-source ai feedback for super gpt\-4v trustworthiness\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:[Link](https://arxiv.org/abs/2405.17220)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Zhanet al\.\(2025\)J\. Zhan M\. Liet al\.Vision\-r1: incentivizing reasoning capability in multimodal large language models via vision\-guided reinforcement learning\.arXiv preprint arXiv:2503\.06749\.External Links:[Link](https://arxiv.org/abs/2503.06749)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Zhanget al\.\(2025a\)J\. Zhang H\. Wanget al\.R1\-vl: learning to reason with multimodal large language models via step\-wise group relative policy optimization\.arXiv preprint arXiv:2503\.12937\.External Links:[Link](https://arxiv.org/abs/2503.12937)Cited by:[§5\.1](https://arxiv.org/html/2609.28570#S5.SS1.p1.1)\.
- Zhanget al\.\(2024\)M\. Zhang, M\. Huang, R\. Shi, L\. Guo, C\. Peng, P\. Yan, Y\. Zhou, and X\. QiuCalibrating the confidence of large language models by eliciting fidelity\.arXiv preprint arXiv:2404\.02655\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Zhanget al\.\(2025b\)W\. Zhang, Y\. Xie, Y\. Sun, Y\. Chen, G\. Wang, Y\. Li, B\. Ding, and J\. ZhouOn\-policy rl meets off\-policy experts: harmonizing supervised fine\-tuning and reinforcement learning via dynamic weighting\.arXiv preprint arXiv:2508\.11408\.Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1),[§4\.1](https://arxiv.org/html/2609.28570#S4.SS1.SSS0.Px5.p1.1)\.
- Zhaoet al\.\(2025\)L\. Zhaoet al\.Mitigating object hallucination in large vision\-language models via image\-grounded guidance\.arXiv preprint arXiv:2402\.08680\.External Links:[Link](https://arxiv.org/abs/2402.08680)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Zhaoet al\.\(2024\)Z\. Zhao, B\. Wang, L\. Ouyang, X\. Dong, J\. Wang, and C\. HeBeyond hallucinations: enhancing lvlms through hallucination\-aware direct preference optimization\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2311.16839)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Zhouet al\.\(2023\)Y\. Zhouet al\.Analyzing and mitigating object hallucination in large vision\-language models\.arXiv preprint arXiv:2310\.00754\.External Links:[Link](https://arxiv.org/abs/2310.00754)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Zhouet al\.\(2024\)Y\. Zhou, C\. Cui, J\. Yoon, L\. Zhang, Z\. Deng, C\. Finn, M\. Bansal, and H\. YaoPOVID: preference optimization for hallucination mitigation in video llms\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2407.13881)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.
- Zhuet al\.\(2025\)Y\. Zhu, L\. Tao, M\. Dong, and C\. XuMitigating object hallucinations in large vision\-language models via attention calibration\.arXiv preprint arXiv:2502\.01969\.External Links:[Link](https://arxiv.org/abs/2502.01969)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p1.1)\.
- Zhuanget al\.\(2026a\)Y\. Zhuang, J\. Yang, M\. Pan, C\. Tan, Y\. Cai, S\. Tan, C\. Zhi, X\. Zhang, J\. Yin, and J\. ChenMitigating manifold departure: uncertainty\-aware subspace rectification for trustworthy mllm decoding\.External Links:2606\.09859,[Link](https://arxiv.org/abs/2606.09859)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p2.1)\.
- Zhuanget al\.\(2026b\)Y\. Zhuang, B\. Yu, J\. Yang, R\. Sun, Z\. Li, C\. Tan, X\. Zhang, J\. Yin, and J\. ChenDual\-axis policy optimization for llm agents: bayesian feedback attribution and trajectory mass normalization\.External Links:2609\.19830,[Link](https://arxiv.org/abs/2609.19830)Cited by:[§2](https://arxiv.org/html/2609.28570#S2.p3.1)\.

## Appendix AAdditional Experimental Results

### A\.1Gradient\-Mass Pilot

![Refer to caption](https://arxiv.org/html/2609.28570v1/pic/Intro/combined_three_metrics.png)Figure 3:Effect of entropy\-masked gradients\. Masking low\-entropy gradients preserves stable learning, while masking high\-entropy gradients destabilizes training: high\-entropy tokens carry the bulk of the update signal \(Prop\.[1](https://arxiv.org/html/2609.28570#Thmproposition1)\); this pilot shows where gradient mass lives, not a hallucination outcome\.
### A\.2Model Guess Analysis

To further examine uncertainty\-driven guessing, we transform multiple\-choice questions into cloze\-style fill\-in\-the\-blank queries, removing predefined options to compel models to answer from visual\-semantic grounding rather than option cues\. This setting exposes semantic instability in free\-form generation\.

![Refer to caption](https://arxiv.org/html/2609.28570v1/pic/Guess_Exp/combined_semantic_entropy.png)Figure 4:Semantic entropy on HallusionBench and MMStar \(cloze\-style conversion; questions sorted by the base model’sHsH\_\{s\}for readability\)\. Compared with the untrained base model, DAPO yields higher semantic entropy, while GRPO and PAPO reduce entropy but remain more variable\. DEEPO maintains consistently lower and less dispersed semantic entropy, indicating reduced uncertainty\-driven guessing\.As shown in Fig\.[4](https://arxiv.org/html/2609.28570#A1.F4), DEEPO consistently produces the lowest semantic entropy across both HallusionBench and MMStar, with the smallest dispersion across questions \(we operationalize “more stable” as the inter\-question standard deviation ofHsH\_\{s\}\)\. In contrast, the base model, DAPO, GRPO, SFT, and PAPO exhibit higher or more variable entropy, indicating less stable answer\-level commitments under ambiguous multimodal evidence\. We stress that lowerHsH\_\{s\}is evidence of reduced guessing only when accuracy is preserved—a confidently wrong model would also have low entropy—so this analysis should be read together with the accuracy numbers in Table[1](https://arxiv.org/html/2609.28570#S4.T1)\. These results support our motivation that semantic\-entropy\-triggered expert guidance reduces inconsistent semantic predictions and suppresses uncertainty\-driven guessing during RL fine\-tuning\.

### A\.3Hyperparameter Sensitivity

We briefly check sensitivity to the initial semantic\-entropy thresholdτs\\tau\_\{s\}\(Table[5](https://arxiv.org/html/2609.28570#A1.T5)\) and the rollout budgetNN\(Table[6](https://arxiv.org/html/2609.28570#A1.T6)\)\. Varyingτs\\tau\_\{s\}over\{0\.2,0\.4,0\.5,0\.8,0\.9\}\\\{0\.2,0\.4,0\.5,0\.8,0\.9\\\}moves results by at most0\.70\.7points on POPE and0\.50\.5on VideoMMMU \(four\-seed means\), as the EMA\-updated threshold self\-calibrates to the batch\. IncreasingNN\(co\-varied with the GRPO group sizeGG, so the joint effect of estimation budget and group size\) improves results up toN=8N\{=\}8–1212and degrades noticeably only in the small\-NNregime \(−4\.0\-4\.0on VideoMMMU atN=4N\{=\}4\); the defaultN=8N\{=\}8is a cost\-driven choice\.

Table 5:Sensitivity to the initial semantic entropy thresholdτs\\tau\_\{s\}\(four\-seed means\)\.Table 6:Sensitivity to the rollout budgetNN\(co\-varied with the GRPO group sizeGG; four\-seed means\)\.
### A\.4Causal Controls and Hallucination\-Targeted Baselines

This subsection accompanies Sec\.[5\.4](https://arxiv.org/html/2609.28570#S5.SS4)and collects the causal controls\. Control runs \(Table[8](https://arxiv.org/html/2609.28570#A1.T8), Table[9](https://arxiv.org/html/2609.28570#A1.T9), and Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(c\) in the main text\) follow the same four\-seed protocol as the main results and are reported as means\.

#### Grouping ablation\.

Table[4](https://arxiv.org/html/2609.28570#S5.T4)\(c\) \(main text\) compares the default mixed grouping against separated per\-condition baselines\. Both variants run the identical rollout pipeline of Algorithm[1](https://arxiv.org/html/2609.28570#alg1):G=8G\{=\}8first\-pass unhinted rollouts plus one hint\-conditioned continuation per triggered query, with matched compute and the same prefix regularizer\. They differ only in advantage computation: mixed pooling normalizes all nine samples jointly, whereas the separated variant normalizes the unhinted group within itself and scores the singleton hinted continuation against a batch\-level EMA baseline over hint continuations of other triggered queries \(so its advantage is well\-defined and not identically zero, avoiding the size\-1 group artifact\)\. The “mixed” row is the default DEEPO model itself, not a rerun, which is why its numbers coincide with Table[1](https://arxiv.org/html/2609.28570#S4.T1); the “separated” row is a separately trained run under this protocol\. The drop therefore isolates the loss of the counterfactual penalty on unhinted failures\.

Table 7:Controlled study on expert\-hint supervision \(four\-seed means\)\. Entropy\-triggered hints improve over the corresponding base method, yet remain below DEEPO—hint supervision alone does not explain the gains\. Hint rows use rollout conditioning only \(no prefix regularizer, no dropout\), unlike Table[2](https://arxiv.org/html/2609.28570#S4.T2)’s “w/o Gradient Scaling” row\.
#### Hint\-shuffle control\.

Table[8](https://arxiv.org/html/2609.28570#A1.T8)repeats DEEPO training with the same entropy trigger but with prefixes drawn from*mismatched*queries \(shuffled within the batch\)\. If the gain stemmed from generic regularization or longer context, shuffled hints would help equally\. The results confirm the prediction: shuffled hints show no distinct advantage over GRPO on any of the three benchmarks \(−0\.5\-0\.5/−0\.9\-0\.9/−0\.2\-0\.2; seed\-to\-seed variations overlap substantially\), while correct hints improve by\+2\.1\+2\.1/\+1\.1\+1\.1/\+8\.6\+8\.6—grounding content, not the conditioning machinery, drives the gains\.

Table 8:Hint\-shuffle control \(four\-seed means\): entropy\-triggered prefixes from mismatched queries\. Shuffled hints should fail to help despite identical context length and trigger statistics\.
#### Self\-generated hints\.

Table[9](https://arxiv.org/html/2609.28570#A1.T9)evaluatesDeepo\-self, in which the hint for a triggered query is the prefix of the highest\-reward verified\-correct rollout from the same GRPO group \(queries whose group contains no correct rollout receive no hint\)\. This variant requires no dataset traces at all\.Deepo\-self recovers most of the gain over GRPO \(\+0\.9\+0\.9/\+0\.0\+0\.0/\+6\.1\+6\.1\), trailing dataset\-derived hints only modestly \(−1\.2\-1\.2/−1\.1\-1\.1/−2\.5\-2\.5\)\. Because a self\-hint exists only when the group already contains a correct rollout,Deepo\-self cannot exercise the advantage\-reshaping channel on all\-wrong groups; its retained gain \(VideoMMMU:6\.16\.1of8\.68\.6,≈71%\\approx 71\\%\) indicates that the direct\-grounding and supervision channel carries the bulk of the gain; since the two variants also differ in hint source, quality, and coverage, the residual gap is an upper bound on—not a clean isolation of—the collapsed\-group coverage channel\.

Table 9:Self\-generated hints \(Deepo\-self, four\-seed means\): prefixes come from the highest\-reward verified\-correct rollout within the same GRPO group; no dataset traces are used\.

### A\.5Further Controls and Ablations

The following analyses isolate error localization \(weight vs\. effective step\) and backbone transfer; the weighting\-rule and trigger ablations are consolidated in Table[3](https://arxiv.org/html/2609.28570#S5.T3)of the main text\.

#### Error\-localization analysis\.

Table[10](https://arxiv.org/html/2609.28570#A1.T10)measures the relative per\-token update intensity that the sign\-aware rule assigns to hallucinated vs\. non\-error content on failed rollouts\.

*Annotation protocol\.*We randomly sampled 300 failed \(reward\-0\) rollouts per benchmark \(600 in total\) from the evaluation generations of the GRPO baseline\. Two trained annotators independently marked hallucinated spans at the token level—visual entities, attributes, counts, and spatial or temporal relations not supported by the input image/video—following a written guideline that explicitly requires articles, connectives, and other function words to be labeled as non\-error even when they are low\-entropy\. Disagreements were adjudicated by a third annotator; inter\-annotator agreement before adjudication was Cohen’sκ=0\.78\\kappa=0\.78\(HallusionBench\) and0\.810\.81\(VideoMMMU\)\. As an independent cross\-check, GPT\-4V was shown the same rollouts and visual inputs and asked to mark unsupported visual claims; it agreed with the adjudicated labels on92%92\\%of tokens on a50%50\\%subsample\. For each token we then recorded its scaling weightg⋆g^\{\\star\}under the DEEPO rule and its effective logit\-step norm∥Δ​zt∥∝wt​∥gt∥\\lVert\\Delta z\_\{t\}\\rVert\\propto w\_\{t\}\\lVert g\_\{t\}\\rVert; the table reports both quantities on error vs\. non\-error tokens within failed rollouts, with 95% bootstrap confidence intervals over rollouts \(10,000 resamples\)\.

*Interpretation\.*Hallucinated spans are sparse in long\-form generation: error tokens constitute6\.4%6\.4\\%\(HallusionBench\) and7\.1%7\.1\\%\(VideoMMMU\) of all tokens in the annotated failed rollouts; weighting by the enrichment factors, roughly10\.7%10\.7\\%/12\.1%12\.1\\%of the weighted update mass lands on error tokens and the remaining∼88%\\sim\\\!88\\%on non\-error tokens—as it inevitably does for any sequence\-level advantage formulation\. The table therefore supports a*relative budget shift*—nearly doubled per\-token update intensity on the error locus—not token\-orthogonal deletion of hallucinated content\. Weight is also not gradient contribution: the effective\-step enrichment \(1\.15×1\.15\\times/1\.25×1\.25\\times\) is attenuated relative to the weight enrichment because error tokens carry smaller raw score norms∥gt∥\\lVert g\_\{t\}\\rVert—exactly the stall the compensation is designed to counteract \(Thm\.[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)\(iii\)\)\.

Table 10:Error\-localization analysis on failed rollouts: per\-token update intensity on hallucinated vs\. non\-error content\. Annotation: token\-level hallucinated\-span labeling by two human annotators with adjudication \(600 rollouts; Cohen’sκ=0\.78\\kappa=0\.78/0\.810\.81\), cross\-checked by GPT\-4V \(92%92\\%token agreement on a50%50\\%subsample\)\.∥Δ​z∥\\lVert\\Delta z\\rVert: mean effective logit\-step norm\. CI: 95% bootstrap over rollouts\.
#### Second backbone\.

Table[11](https://arxiv.org/html/2609.28570#A1.T11)reports generalization to InternVL2\.5\-8B\.

Table 11:Generalization to a second backbone \(InternVL2\.5\-8B; four\-seed means\)\.

## Appendix BProof of the Entropy–Gradient Relationship

We prove Proposition[1](https://arxiv.org/html/2609.28570#Thmproposition1)\. Let the policy over theNN\-token vocabulary beπj:=πθ​\(yj∣q,y<t\)\\pi\_\{j\}:=\\pi\_\{\\theta\}\(y\_\{j\}\\mid q,y\_\{<t\}\)with logitsz=\(z1,…,zN\)z=\(z\_\{1\},\\dots,z\_\{N\}\), and letgk:=∇z​log​πk=ek−πg\_\{k\}:=\\nabla\_\{z\}\\log\\pi\_\{k\}=e\_\{k\}\-\\pibe the score\-function gradient for sampled tokenkk, whereeke\_\{k\}is thekk\-th standard basis vector andπ=\(π1,…,πN\)⊤\\pi=\(\\pi\_\{1\},\\dots,\\pi\_\{N\}\)^\{\\top\}\. For eachkk,

‖gk‖2=\(1−πk\)2\+∑j≠kπj2=1−2​πk\+∑j=1Nπj2\.\\\|g\_\{k\}\\\|^\{2\}=\(1\-\\pi\_\{k\}\)^\{2\}\+\\sum\_\{j\\neq k\}\\pi\_\{j\}^\{2\}=1\-2\\pi\_\{k\}\+\\sum\_\{j=1\}^\{N\}\\pi\_\{j\}^\{2\}\.\(7\)Taking the expectation overk∼πk\\sim\\pi,

𝔼k∼π​\[‖gk‖2\]=∑k=1Nπk​\(1−2​πk\+∑j=1Nπj2\)=1−2​∑kπk2\+∑jπj2=1−∑j=1Nπj2\.\\mathbb\{E\}\_\{k\\sim\\pi\}\\big\[\\\|g\_\{k\}\\\|^\{2\}\\big\]=\\sum\_\{k=1\}^\{N\}\\pi\_\{k\}\\Big\(1\-2\\pi\_\{k\}\+\\sum\_\{j=1\}^\{N\}\\pi\_\{j\}^\{2\}\\Big\)=1\-2\\sum\_\{k\}\\pi\_\{k\}^\{2\}\+\\sum\_\{j\}\\pi\_\{j\}^\{2\}=1\-\\sum\_\{j=1\}^\{N\}\\pi\_\{j\}^\{2\}\.\(8\)SinceH2\(π\)=−log∑jπj2H\_\{2\}\(\\pi\)=\-\\log\\sum\_\{j\}\\pi\_\{j\}^\{2\}, we have∑jπj2=exp⁡\(−H2​\(π\)\)\\sum\_\{j\}\\pi\_\{j\}^\{2\}=\\exp\(\-H\_\{2\}\(\\pi\)\), and therefore

𝔼k∼π​\[‖∇z​log​πk‖2\]=1−exp⁡\(−H2​\(π\)\),\\mathbb\{E\}\_\{k\\sim\\pi\}\\big\[\\big\\\|\\nabla\_\{z\}\\log\\pi\_\{k\}\\big\\\|^\{2\}\\big\]=1\-\\exp\\big\(\-H\_\{2\}\(\\pi\)\\big\),\(9\)which is Eq\.[3](https://arxiv.org/html/2609.28570#S4.E3)\. This is the trace of the categorical Fisher information matrixF=diag⁡\(π\)−π​π⊤F=\\operatorname\{diag\}\(\\pi\)\-\\pi\\pi^\{\\top\}in natural parameters, a standard identity \(cf\.[Amari and Nagaoka \(2000\)](https://arxiv.org/html/2609.28570#bib.bib57)\); equivalently, one minus Simpson’s index of coincidence[Simpson \(1949\)](https://arxiv.org/html/2609.28570#bib.bib65)\.

## Appendix CProof of Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)and Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2)

We first prove the sample\-wise damped\-proximal statement, then the sign\-symmetry impossibility\.

#### Setup and standing convention\.

Fix a training batch of*already generated*trajectories\. Every advantageAtA\_\{t\}below is a fixed empirical scalar computed from the batch rewards \(group\-relative normalization included\), and the sampled tokenyty\_\{t\}at each position is fixed as well\. Accordingly,*no expectation overyt∼πθy\_\{t\}\\sim\\pi\_\{\\theta\}is taken anywhere in this section withAtA\_\{t\}factored out*: the only distribution\-level quantities are the Fisher matrixFtF\_\{t\}and its traceτt\\tau\_\{t\}, which are deterministic functions of the current policy at the statest=\(q,y<t\)s\_\{t\}=\(q,y\_\{<t\}\)and do not depend on which token was sampled\. Writeπj:=πθ​\(yj∣st\)\\pi\_\{j\}:=\\pi\_\{\\theta\}\(y\_\{j\}\\mid s\_\{t\}\),j=1,…,Nj=1,\\dots,N, with logitsz=\(z1,…,zN\)z=\(z\_\{1\},\\dots,z\_\{N\}\); for the \(fixed\) sampled tokenyt=yky\_\{t\}=y\_\{k\}the score\-function gradient with respect to the logits isgt=gk=ek−πg\_\{t\}=g\_\{k\}=e\_\{k\}\-\\pi, whereeke\_\{k\}is thekk\-th standard basis vector\. The Fisher information matrix with respect to the logits is

Ft=𝔼y∼πθ\(⋅∣st\)\[gygy⊤\]=diag\(π\)−ππ⊤,τt:=Tr\(Ft\)=1−∑jπj2=1−exp\(−H2,t\),F\_\{t\}=\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\}\\big\[\\,g\_\{y\}g\_\{y\}^\{\\top\}\\,\\big\]=\\operatorname\{diag\}\(\\pi\)\-\\pi\\pi^\{\\top\},\\qquad\\tau\_\{t\}:=\\Tr\(F\_\{t\}\)=1\-\\textstyle\\sum\_\{j\}\\pi\_\{j\}^\{2\}=1\-\\exp\(\-H\_\{2,t\}\),\(10\)where the expectation definingFtF\_\{t\}involves the policy distribution only—it is the second moment of the score—and never the advantageAtA\_\{t\}\.

#### Part \(i\): sample\-wise curvature bound and tangent minorant\.

Ft⪰0F\_\{t\}\\succeq 0since for anyu∈ℝNu\\in\\mathbb\{R\}^\{N\},u⊤​Ft​u=∑jπj​uj2−\(∑jπj​uj\)2=VarJ∼π⁡\[uJ\]≥0u^\{\\top\}F\_\{t\}u=\\sum\_\{j\}\\pi\_\{j\}u\_\{j\}^\{2\}\-\\big\(\\sum\_\{j\}\\pi\_\{j\}u\_\{j\}\\big\)^\{2\}=\\operatorname\{Var\}\_\{J\\sim\\pi\}\[u\_\{J\}\]\\geq 0\. Consequentlyλmax​\(Ft\)≤Tr⁡\(Ft\)=τt\\lambda\_\{\\max\}\(F\_\{t\}\)\\leq\\Tr\(F\_\{t\}\)=\\tau\_\{t\}, and for the fixed sampled token,

gt⊤​Ft​gt≤λmax​\(Ft\)​∥gt∥2≤τt​∥gt∥2\.g\_\{t\}^\{\\top\}F\_\{t\}g\_\{t\}\\;\\leq\\;\\lambda\_\{\\max\}\(F\_\{t\}\)\\,\\lVert g\_\{t\}\\rVert^\{2\}\\;\\leq\\;\\tau\_\{t\}\\,\\lVert g\_\{t\}\\rVert^\{2\}\.\(11\)\(For interpretation only, the distributional moments𝔼y∼π​∥gy∥2=τt\\mathbb\{E\}\_\{y\\sim\\pi\}\\lVert g\_\{y\}\\rVert^\{2\}=\\tau\_\{t\}\(App\.[B](https://arxiv.org/html/2609.28570#A2)\) and𝔼y∼π​\[gy⊤​Ft​gy\]=Tr⁡\(Ft2\)=S2−2​S3\+S22≤τt2\\mathbb\{E\}\_\{y\\sim\\pi\}\[g\_\{y\}^\{\\top\}F\_\{t\}g\_\{y\}\]=\\Tr\(F\_\{t\}^\{2\}\)=S\_\{2\}\-2S\_\{3\}\+S\_\{2\}^\{2\}\\leq\\tau\_\{t\}^\{2\}, withSr:=∑jπjrS\_\{r\}:=\\sum\_\{j\}\\pi\_\{j\}^\{r\}, identifyτt\\tau\_\{t\}as the expected gradient mass; the proof below needs only the sample\-wise bound above, which holds for*every*sampled token rather than in expectation over it\.\) Restrict one\-step logit updates to the score direction,Δ​zt=η​wt​At​gt\\Delta z\_\{t\}=\\eta\\,w\_\{t\}A\_\{t\}g\_\{t\}withwt≥0w\_\{t\}\\geq 0\. The sample\-wise second\-order proximal surrogate with Levenberg–Marquardt dampingε0\>0\\varepsilon\_\{0\}\>0evaluates to

Qt​\(wt\)=At​gt⊤​Δ​zt−βkl2​Δ​zt⊤​\(Ft\+ε0​I\)​Δ​zt=η​At2​∥gt∥2​wt−βkl​η2​At22​\(gt⊤​Ft​gt\+ε0​∥gt∥2\)​wt2\.Q\_\{t\}\(w\_\{t\}\)\\;=\\;A\_\{t\}\\,g\_\{t\}^\{\\top\}\\Delta z\_\{t\}\-\\tfrac\{\\beta\_\{\\mathrm\{kl\}\}\}\{2\}\\,\\Delta z\_\{t\}^\{\\top\}\(F\_\{t\}\+\\varepsilon\_\{0\}I\)\\,\\Delta z\_\{t\}\\;=\\;\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\\,w\_\{t\}\-\\frac\{\\beta\_\{\\mathrm\{kl\}\}\\eta^\{2\}A\_\{t\}^\{2\}\}\{2\}\\,\\big\(g\_\{t\}^\{\\top\}F\_\{t\}g\_\{t\}\+\\varepsilon\_\{0\}\\lVert g\_\{t\}\\rVert^\{2\}\\big\)\\,w\_\{t\}^\{2\}\.\(12\)SinceAtA\_\{t\},gtg\_\{t\}are fixed sample quantities,QtQ\_\{t\}is an exact quadratic inwtw\_\{t\}; the only approximation we introduce is the scalarization of the curvature term\. Define the trace\-scalarized quadratic

Q~t​\(w\)=η​At2​∥gt∥2​w−βkl​η2​At22​∥gt∥2​\(τt\+ε0\)​w2\.\\widetilde\{Q\}\_\{t\}\(w\)=\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\\,w\-\\frac\{\\beta\_\{\\mathrm\{kl\}\}\\eta^\{2\}A\_\{t\}^\{2\}\}\{2\}\\,\\lVert g\_\{t\}\\rVert^\{2\}\\,\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\,w^\{2\}\.\(13\)Sincegt⊤​Ft​gt\+ε0​∥gt∥2≤\(τt\+ε0\)​∥gt∥2g\_\{t\}^\{\\top\}F\_\{t\}g\_\{t\}\+\\varepsilon\_\{0\}\\lVert g\_\{t\}\\rVert^\{2\}\\leq\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\lVert g\_\{t\}\\rVert^\{2\}by the bound above,Q~t​\(w\)≤Qt​\(w\)\\widetilde\{Q\}\_\{t\}\(w\)\\leq Q\_\{t\}\(w\)for allw≥0w\\geq 0, withQ~t​\(0\)=Qt​\(0\)=0\\widetilde\{Q\}\_\{t\}\(0\)=Q\_\{t\}\(0\)=0andQ~t′​\(0\)=Qt′​\(0\)=η​At2​∥gt∥2\\widetilde\{Q\}\_\{t\}^\{\\prime\}\(0\)=Q\_\{t\}^\{\\prime\}\(0\)=\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}:Q~t\\widetilde\{Q\}\_\{t\}is a tangent minorant ofQtQ\_\{t\}onℝ\+\\mathbb\{R\}\_\{\+\}\.

#### Part \(ii\): damped optimum, ascent, and cap\.

Q~t\\widetilde\{Q\}\_\{t\}is concave inwwwith unique maximizer

wtε0=η​At2​∥gt∥2βkl​η2​At2​∥gt∥2​\(τt\+ε0\)=1βkl​η​\(τt\+ε0\)≤1βkl​η​ε0=:wmax,w\_\{t\}^\{\\varepsilon\_\{0\}\}=\\frac\{\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\}\{\\beta\_\{\\mathrm\{kl\}\}\\,\\eta^\{2\}A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\}=\\frac\{1\}\{\\beta\_\{\\mathrm\{kl\}\}\\,\\eta\\,\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\}\\;\\leq\\;\\frac\{1\}\{\\beta\_\{\\mathrm\{kl\}\}\\,\\eta\\,\\varepsilon\_\{0\}\}=:w\_\{\\max\},\(14\)where the cap follows fromτt≥0\\tau\_\{t\}\\geq 0; in particular the weight remains bounded as the curvature vanishes, unlike the undamped inverse\-curvature rule1/τt1/\\tau\_\{t\}\. The achieved minorant value certifies strict sample\-wise ascent:

Qt​\(wtε0\)≥Q~t​\(wtε0\)=At2​∥gt∥22​βkl​\(τt\+ε0\)\>0=Qt​\(0\)\.Q\_\{t\}\(w\_\{t\}^\{\\varepsilon\_\{0\}\}\)\\;\\geq\\;\\widetilde\{Q\}\_\{t\}\(w\_\{t\}^\{\\varepsilon\_\{0\}\}\)=\\frac\{A\_\{t\}^\{2\}\\,\\lVert g\_\{t\}\\rVert^\{2\}\}\{2\\,\\beta\_\{\\mathrm\{kl\}\}\\,\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\}\\;\>\\;0=Q\_\{t\}\(0\)\.\(15\)The induced logit step itself stays controlled:∥Δ​zt∥=η​wtε0​\|At\|​∥gt∥=\|At\|​∥gt∥/\(βkl​\(τt\+ε0\)\)\\lVert\\Delta z\_\{t\}\\rVert=\\eta\\,w\_\{t\}^\{\\varepsilon\_\{0\}\}\\,\|A\_\{t\}\|\\,\\lVert g\_\{t\}\\rVert=\|A\_\{t\}\|\\,\\lVert g\_\{t\}\\rVert/\\big\(\\beta\_\{\\mathrm\{kl\}\}\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)\\big\), which vanishes smoothly asgt→0g\_\{t\}\\to 0while its amplification*relative to the unweighted rule*\(wt=1w\_\{t\}=1\) scales as\(τt\+ε0\)−1\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)^\{\-1\}\. This is a conservative damped proximal step in logit space: the damping constantε0\\varepsilon\_\{0\}caps the weight, and the guarantee degrades gracefully asτt→0\\tau\_\{t\}\\to 0\.

#### Part \(iii\): softmax\-saturation regime\.

Letπy:=πθ​\(yt∣st\)\\pi\_\{y\}:=\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)denote the probability of the sampled token andδt:=1−πy\\delta\_\{t\}:=1\-\\pi\_\{y\}the tail mass\. Using∑j≠yπj2≤\(∑j≠yπj\)2=δt2\\sum\_\{j\\neq y\}\\pi\_\{j\}^\{2\}\\leq\\big\(\\sum\_\{j\\neq y\}\\pi\_\{j\}\\big\)^\{2\}=\\delta\_\{t\}^\{2\},

∥gt∥2=\(1−πy\)2\+∑j≠yπj2∈\[δt2,2​δt2\],τt=1−πy2−∑j≠yπj2∈\[2​δt​\(1−δt\),2​δt−δt2\]\.\\lVert g\_\{t\}\\rVert^\{2\}=\(1\-\\pi\_\{y\}\)^\{2\}\+\\sum\_\{j\\neq y\}\\pi\_\{j\}^\{2\}\\;\\in\\;\\big\[\\,\\delta\_\{t\}^\{2\},\\;2\\,\\delta\_\{t\}^\{2\}\\,\\big\],\\qquad\\tau\_\{t\}=1\-\\pi\_\{y\}^\{2\}\-\\sum\_\{j\\neq y\}\\pi\_\{j\}^\{2\}\\;\\in\\;\\big\[\\,2\\delta\_\{t\}\(1\-\\delta\_\{t\}\),\\;2\\delta\_\{t\}\-\\delta\_\{t\}^\{2\}\\,\\big\]\.\(16\)Hence, as the prediction saturates \(πy→1\\pi\_\{y\}\\to 1, i\.e\.δt→0\\delta\_\{t\}\\to 0\): \(a\) the unweighted local slope vanishes*quadratically*,Qt′​\(0\)=η​At2​∥gt∥2≤2​η​At2​δt2Q\_\{t\}^\{\\prime\}\(0\)=\\eta A\_\{t\}^\{2\}\\lVert g\_\{t\}\\rVert^\{2\}\\leq 2\\eta A\_\{t\}^\{2\}\\,\\delta\_\{t\}^\{2\}—this is the softmax\-saturation stall; \(b\) the compensated weight grows aswtε0≥\(βkl​η​\(2​δt−δt2\+ε0\)\)−1=Ω⁡\(δt−1\)w\_\{t\}^\{\\varepsilon\_\{0\}\}\\geq\\big\(\\beta\_\{\\mathrm\{kl\}\}\\eta\\,\(2\\delta\_\{t\}\-\\delta\_\{t\}^\{2\}\+\\varepsilon\_\{0\}\)\\big\)^\{\-1\}=\\Omega\(\\delta\_\{t\}^\{\-1\}\)until it attains its capwmaxw\_\{\\max\}; \(c\) the certified ascent decays only*linearly*in the tail mass,

Qt​\(wtε0\)≥At2​δt22​βkl​\(2​δt−δt2\+ε0\)=Θ⁡\(δt\)for​δt≫ε0\.Q\_\{t\}\(w\_\{t\}^\{\\varepsilon\_\{0\}\}\)\\;\\geq\\;\\frac\{A\_\{t\}^\{2\}\\,\\delta\_\{t\}^\{2\}\}\{2\\beta\_\{\\mathrm\{kl\}\}\\,\(2\\delta\_\{t\}\-\\delta\_\{t\}^\{2\}\+\\varepsilon\_\{0\}\)\}\\;=\\;\\Theta\(\\delta\_\{t\}\)\\quad\\text\{for \}\\delta\_\{t\}\\gg\\varepsilon\_\{0\}\.\(17\)This is the formal content of saturation compensation: inverse\-curvature weighting converts the local correction rate on a confident token from quadratic to first\-order in the tail mass where the curvature dominates the damping \(δt≫ε0\\delta\_\{t\}\\gg\\varepsilon\_\{0\}\)\. In the deep tailδt≪ε0\\delta\_\{t\}\\ll\\varepsilon\_\{0\}the damping floor dominates: the weight saturates atwmax=\(βkl​η​ε0\)−1w\_\{\\max\}=\(\\beta\_\{\\mathrm\{kl\}\}\\eta\\varepsilon\_\{0\}\)^\{\-1\}and the certified bound reverts toO⁡\(δt2/ε0\)O\(\\delta\_\{t\}^\{2\}/\\varepsilon\_\{0\}\)—quadratic again, but with the1/ε01/\\varepsilon\_\{0\}constant—while the induced step∥Δ​zt∥≤\|At\|​∥gt∥/\(βkl​ε0\)\\lVert\\Delta z\_\{t\}\\rVert\\leq\|A\_\{t\}\|\\,\\lVert g\_\{t\}\\rVert/\(\\beta\_\{\\mathrm\{kl\}\}\\varepsilon\_\{0\}\)remains bounded and vanishes smoothly\. The theorem is thus a rate statement over the operational confidence band plus a boundedness statement in the tail; it does not eliminate saturation asymptotically\. This proves Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)\.

#### Sign\-symmetry impossibility \(Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2)\)\.

Letwt=ν⁡\(H2,t\)w\_\{t\}=\\nu\(H\_\{2,t\}\)be any advantage\-sign\-independent rule under a fixed batch budget∑twt=C\\sum\_\{t\}w\_\{t\}=C\. Suppose there exist two samplesa,ba,bwithH2,a=H2,bH\_\{2,a\}=H\_\{2,b\}andAa<0<AbA\_\{a\}<0<A\_\{b\}\. Then necessarilywa=ν⁡\(H2,a\)=ν⁡\(H2,b\)=wbw\_\{a\}=\\nu\(H\_\{2,a\}\)=\\nu\(H\_\{2,b\}\)=w\_\{b\}, so no such rule can satisfy the correction\-priority conditionwa\>wbw\_\{a\}\>w\_\{b\}\. More explicitly, define the corrective allocation objective

𝒞\(w\):=∑t:At<0\|At\|wtτt−1,\\mathcal\{C\}\(w\):=\\sum\_\{t:\\,A\_\{t\}<0\}\|A\_\{t\}\|\\,w\_\{t\}\\,\\tau\_\{t\}^\{\-1\},\(18\)measuring how much weighted update mass is assigned to negative\-advantage samples after curvature compensation\. Starting from any symmetric rule, construct the budget\-preserving perturbationwa′=wa\+δw\_\{a\}^\{\\prime\}=w\_\{a\}\+\\delta,wb′=wb−δw\_\{b\}^\{\\prime\}=w\_\{b\}\-\\deltawith0<δ<wb0<\\delta<w\_\{b\}, leaving other weights unchanged\. The total budget is preserved, and since only sampleaacontributes to𝒞\\mathcal\{C\}among this pair,

𝒞⁡\(w′\)−𝒞⁡\(w\)=\|Aa\|\(wa′−wa\)​τa−1=δ\|Aa\|τa−1\>0\.\\mathcal\{C\}\(w^\{\\prime\}\)\-\\mathcal\{C\}\(w\)=\|A\_\{a\}\|\\,\(w\_\{a\}^\{\\prime\}\-w\_\{a\}\)\\,\\tau\_\{a\}^\{\-1\}=\\delta\\,\|A\_\{a\}\|\\,\\tau\_\{a\}^\{\-1\}\>0\.\(19\)So there always exists a budget\-preserving asymmetric reallocation that strictly improves the corrective objective, while any symmetric rulewt=ν⁡\(H2,t\)w\_\{t\}=\\nu\(H\_\{2,t\}\)remains invariant over equal\-entropy pairs and cannot realize this preferential reallocation\.

#### Scope and caveats\.

We record four qualifications on how Theorem[4\.1](https://arxiv.org/html/2609.28570#S4.Thmtheorem1)should be read\. \(a\) The guarantee is*sample\-wise and local*: it concerns one damped proximal step in the logits of a single categorical conditional of one already\-generated trajectory, with the batch advantages held fixed\. It is not a monotonic\-improvement statement for the expected return underyt∼πθy\_\{t\}\\sim\\pi\_\{\\theta\}, it makes no claim across re\-sampled batches \(over which the empiricalAtA\_\{t\}are re\-drawn\), and it does not model shared parameters across token positions, PPO clipping, or Adam preconditioning—it is not a convergence or targeted\-correction guarantee for the full network\. \(b\)AtA\_\{t\}entersQtQ\_\{t\}only throughAt2A\_\{t\}^\{2\}: the theorem is sign\-blind, so it justifies the*magnitude*1/\(τt\+ε0\)1/\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)of the corrective branch of Eq\.[5](https://arxiv.org/html/2609.28570#S4.E5)but not its sign asymmetry—that asymmetry is a design choice motivated by Remark[4\.2](https://arxiv.org/html/2609.28570#S4.Thmtheorem2), and whether it outperforms symmetric rules is an empirical question, answered in Table[3](https://arxiv.org/html/2609.28570#S5.T3)\. \(c\) The analysis treats a single token position with a local logit update; loss reweighting equals per\-token logit stepping only to first order at the current iterate, and in practiceβkl,η\\beta\_\{\\mathrm\{kl\}\},\\etaare absorbed by batch normalization, so only the*shape*1/\(τt\+ε0\)1/\(\\tau\_\{t\}\+\\varepsilon\_\{0\}\)survives—which is why we label Eq\.[5](https://arxiv.org/html/2609.28570#S4.E5)a design rule rather than a corollary\. Finally, the KL penalty in Eq\.[6](https://arxiv.org/html/2609.28570#S4.E6)is deliberately left unweighted: tokens receiving large corrective weights are permitted to move further from the reference policy, which we consider a feature of the correction branch rather than an oversight\.

## Appendix DRelation Analysis of Different Entropies

DEEPO uses two entropy signals at different levels, and it is natural to ask whether they measure the same thing\. They do not: semantic entropyHsH\_\{s\}captures answer\-level dispersion across sampled rollouts \(Sec\.[3](https://arxiv.org/html/2609.28570#S3)\), while the Rényi\-2 entropyH2H\_\{2\}characterizes token\-level policy sharpness and, by Proposition[1](https://arxiv.org/html/2609.28570#Thmproposition1), the expected score\-gradient norm\. The two are linked through the token\-level Shannon entropyH1\(π\(⋅∣q,y<t\)\)=−∑y∈𝒱π\(y∣q,y<t\)logπ\(y∣q,y<t\)H\_\{1\}\(\\pi\(\\cdot\\mid q,y\_\{<t\}\)\)=\-\\sum\_\{y\\in\\mathcal\{V\}\}\\pi\(y\\mid q,y\_\{<t\}\)\\log\\pi\(y\\mid q,y\_\{<t\}\), theα→1\\alpha\\to 1limit of Rényi entropy, whose functional form mirrors that ofHsH\_\{s\}\(a Shannon entropy over meanings versus over tokens\)\. The chain

F⁡\(H2,n\)≥H1​\(p\)≥H2​\(p\)≥log⁡n−log⁡\(1\+2​n​\(log⁡n−H1​\(p\)\)\),F\(H\_\{2\};n\)\\;\\geq\\;H\_\{1\}\(p\)\\;\\geq\\;H\_\{2\}\(p\)\\;\\geq\\;\\log n\-\\log\\\!\\Big\(1\+2n\\big\(\\log n\-H\_\{1\}\(p\)\\big\)\\Big\),\(20\)summarized in the following proposition, makes the link quantitative\.

###### Proposition 2\(Entropy chain; known results\)\.

Letp=\(p1,…,pn\)p=\(p\_\{1\},\\dots,p\_\{n\}\)denote the softmax distribution over annn\-token vocabulary\. Then the chain above holds, whereF⁡\(H2,n\)=−1\+\(n−1\)​sn​log⁡\(1\+\(n−1\)​sn\)−\(n−1\)​\(1−s\)n​log⁡\(1−sn\)F\(H\_\{2\};n\)=\-\\tfrac\{1\+\(n\-1\)s\}\{n\}\\log\\\!\\big\(\\tfrac\{1\+\(n\-1\)s\}\{n\}\\big\)\-\\tfrac\{\(n\-1\)\(1\-s\)\}\{n\}\\log\\\!\\big\(\\tfrac\{1\-s\}\{n\}\\big\)ands⁡\(H2\):=n​e−H2−1n−1∈\[0,1\]s\(H\_\{2\}\):=\\sqrt\{\\tfrac\{ne^\{\-H\_\{2\}\}\-1\}\{n\-1\}\}\\in\[0,1\]\. The upper bound is the sharp bound of[Harremoës and Topsøe \(2001\)](https://arxiv.org/html/2609.28570#bib.bib60), attained exactly by the two\-level distributionp⋆=\(1\+\(n−1\)​sn,1−sn,…,1−sn\)p^\{\\star\}=\\big\(\\tfrac\{1\+\(n\-1\)s\}\{n\},\\tfrac\{1\-s\}\{n\},\\dots,\\tfrac\{1\-s\}\{n\}\\big\);H1≥H2H\_\{1\}\\geq H\_\{2\}is Rényi monotonicity \(equality iffppis uniform on its support\); the lower bound follows from Pinsker’s inequality and is tight only at the uniform distribution \(it is strict, and in fact vacuous, at degenerate distributions—see Remark[E\.1](https://arxiv.org/html/2609.28570#A5.Thmtheorem1)\)\. Self\-contained derivations are given in App\.[E](https://arxiv.org/html/2609.28570#A5)\.

The practical content for our design: the two branches do not require their entropy signals to be interchangeable or even strongly correlated—semantic entropy gates the rollout intervention, while token\-levelH2H\_\{2\}parameterizes optimization—and the chain above guarantees only that both are faithful measures of uncertainty at their respective levels\.*A note on scope:*we deliberately do not use an empiricalHsH\_\{s\}–H1H\_\{1\}correlation analysis to justify combining the two signals; such a correlation would neither be necessary for the design nor sufficient to establish complementarity, and theHsH\_\{s\}–H2H\_\{2\}chain is presented here solely to locate both quantities on a common uncertainty footing\.

## Appendix ERelation BetweenH1H\_\{1\}andH2H\_\{2\}

This appendix collects the relations between Shannon entropyH1H\_\{1\}and Rényi\-2 entropyH2H\_\{2\}used in App\.[D](https://arxiv.org/html/2609.28570#A4)\. None of the results is new:H1≥H2H\_\{1\}\\geq H\_\{2\}is Rényi monotonicity; the upper envelopeF⁡\(H2,n\)F\(H\_\{2\};n\)is the sharp entropy–index\-of\-coincidence bound of[Harremoës and Topsøe \(2001\)](https://arxiv.org/html/2609.28570#bib.bib60); the lower bound follows from Pinsker’s inequality \(cf\.[Fedotov et al\. \(2003\)](https://arxiv.org/html/2609.28570#bib.bib58)\)\. We re\-derive them in our notation for completeness\.

Letp=\(p1,…,pn\)p=\(p\_\{1\},\\dots,p\_\{n\}\)be a discrete distribution onn=\|𝒱\|n=\|\\mathcal\{V\}\|support points,H1\(p\)=−∑i=1npilogpiH\_\{1\}\(p\)=\-\\sum\_\{i=1\}^\{n\}p\_\{i\}\\log p\_\{i\},H2​\(p\)=−log⁡\(∑i=1npi2\)H\_\{2\}\(p\)=\-\\log\\big\(\\sum\_\{i=1\}^\{n\}p\_\{i\}^\{2\}\\big\),S2​\(p\)=∑ipi2=e−H2S\_\{2\}\(p\)=\\sum\_\{i\}p\_\{i\}^\{2\}=e^\{\-H\_\{2\}\}\.

#### \(ii\)H1≥H2H\_\{1\}\\geq H\_\{2\}\.

LetXXtake valuepip\_\{i\}with probabilitypip\_\{i\}\. Sincelog\\logis concave, Jensen’s inequality gives𝔼⁡\[log⁡X\]≤log⁡𝔼⁡\[X\]\\mathbb\{E\}\[\\log X\]\\leq\\log\\mathbb\{E\}\[X\], i\.e\.∑ipilogpi≤log∑ipi2\\sum\_\{i\}p\_\{i\}\\log p\_\{i\}\\leq\\log\\sum\_\{i\}p\_\{i\}^\{2\}; negating yieldsH1​\(p\)≥H2​\(p\)H\_\{1\}\(p\)\\geq H\_\{2\}\(p\), with equality iffppis uniform on its support\.

#### \(i​iii\) Upper envelopeH1≤F⁡\(H2,n\)H\_\{1\}\\leq F\(H\_\{2\};n\)[Harremoës and Topsøe \(2001\)](https://arxiv.org/html/2609.28570#bib.bib60)\.

Under the constraints∑ipi=1\\sum\_\{i\}p\_\{i\}=1,∑ipi2=C\\sum\_\{i\}p\_\{i\}^\{2\}=C, the symmetric Schur\-concaveH1H\_\{1\}is maximized \(by KKT / symmetry arguments\) at a two\-level distribution

p=\(a,b,…,b⏟n−1​entries\),b=1−an−1\.p=\(a,\\underbrace\{b,\\dots,b\}\_\{n\-1\\ \\text\{entries\}\}\),\\qquad b=\\frac\{1\-a\}\{n\-1\}\.\(21\)The quadratic constraint gives\(n−1\)​C=n​a2−2​a\+1\(n\-1\)C=na^\{2\}\-2a\+1; solving witha≥1/na\\geq 1/nand introducing

s:=n​C−1n−1=n​e−H2−1n−1∈\[0,1\],s:=\\sqrt\{\\frac\{nC\-1\}\{n\-1\}\}=\\sqrt\{\\frac\{ne^\{\-H\_\{2\}\}\-1\}\{n\-1\}\}\\in\[0,1\],\(22\)yieldsa=1\+\(n−1\)​sna=\\frac\{1\+\(n\-1\)s\}\{n\},1−a=\(n−1\)​\(1−s\)n1\-a=\\frac\{\(n\-1\)\(1\-s\)\}\{n\}, and substituting intoH1​\(a,b\)=−a​log⁡a−\(1−a\)​log⁡1−an−1H\_\{1\}\(a,b\)=\-a\\log a\-\(1\-a\)\\log\\frac\{1\-a\}\{n\-1\}gives the explicit envelope

F⁡\(H2,n\)=−1\+\(n−1\)​sn​log⁡\(1\+\(n−1\)​sn\)−\(n−1\)​\(1−s\)n​log⁡\(1−sn\)\.F\(H\_\{2\};n\)=\-\\frac\{1\+\(n\-1\)s\}\{n\}\\log\\\!\\Big\(\\frac\{1\+\(n\-1\)s\}\{n\}\\Big\)\-\\frac\{\(n\-1\)\(1\-s\)\}\{n\}\\log\\\!\\Big\(\\frac\{1\-s\}\{n\}\\Big\)\.\(23\)

#### \(i​i​iiii\) Lower bound via Pinsker\.

Letu=\(1/n,…,1/n\)u=\(1/n,\\dots,1/n\)\. Sinceχ2\(p∥u\)=nS2\(p\)−1=n∥p−u∥22\\chi^\{2\}\(p\\\|u\)=nS\_\{2\}\(p\)\-1=n\\\|p\-u\\\|\_\{2\}^\{2\}and

DKL\(p∥u\)≥12∥p−u∥12≥12∥p−u∥22=12\(S2\(p\)−1n\),D\_\{\\mathrm\{KL\}\}\(p\\\|u\)\\geq\\tfrac\{1\}\{2\}\\\|p\-u\\\|\_\{1\}^\{2\}\\geq\\tfrac\{1\}\{2\}\\\|p\-u\\\|\_\{2\}^\{2\}=\\tfrac\{1\}\{2\}\\Big\(S\_\{2\}\(p\)\-\\tfrac\{1\}\{n\}\\Big\),\(24\)we obtainS2\(p\)≤\(1\+2nDKL\(p∥u\)\)/nS\_\{2\}\(p\)\\leq\\big\(1\+2nD\_\{\\mathrm\{KL\}\}\(p\\\|u\)\\big\)/n\. Taking−log\-\\logand usingDKL\(p∥u\)=logn−H1\(p\)D\_\{\\mathrm\{KL\}\}\(p\\\|u\)=\\log n\-H\_\{1\}\(p\):

H2​\(p\)≥log⁡n−log⁡\(1\+2​n​\(log⁡n−H1​\(p\)\)\)\.H\_\{2\}\(p\)\\geq\\log n\-\\log\\\!\\Big\(1\+2n\\big\(\\log n\-H\_\{1\}\(p\)\\big\)\\Big\)\.\(25\)

## Appendix FProtocol for theHsH\_\{s\}–H1H\_\{1\}Alignment Analysis

This appendix specifies the protocol for the empirical alignment analysis referenced in App\.[D](https://arxiv.org/html/2609.28570#A4)\.

#### Quantities\.

For each evaluation questionqqon HallusionBench and MMStar \(multiple\-choice converted to cloze\-style, as in App\.[A\.2](https://arxiv.org/html/2609.28570#A1.SS2)\), we drawMMstochastic samples \(temperature1\.21\.2\) from the evaluated policy and compute: \(i\) the semantic entropyHs​\(q\)H\_\{s\}\(q\)over answer\-level clusters \(Sec\.[3](https://arxiv.org/html/2609.28570#S3), with the same protocol and budgetMM\); \(ii\) the token\-level Shannon entropyH1H\_\{1\}, averaged over answer\-span tokens and over theMMsamples,

H¯1\(q\)=1M∑i=1M1\|yians\|∑t∈yiansH1\(π\(⋅∣q,yi,<t\)\),\\bar\{H\}\_\{1\}\(q\)=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\frac\{1\}\{\|y\_\{i\}^\{\\mathrm\{ans\}\}\|\}\\sum\_\{t\\in y\_\{i\}^\{\\mathrm\{ans\}\}\}H\_\{1\}\\big\(\\pi\(\\cdot\\mid q,y\_\{i,<t\}\)\\big\),\(26\)whereyiansy\_\{i\}^\{\\mathrm\{ans\}\}denotes the extracted answer span of sampleii\. Both quantities are computed at inference; expert hints are used only during*training*, so “with\-hint” columns below compare*models trained with vs\. without*the hint branch, not hint\-conditioned decoding\.

#### Statistics\.

We report the Pearson coefficientρ\\rhoover questions betweenH¯1​\(q\)\\bar\{H\}\_\{1\}\(q\)andHs​\(q\)H\_\{s\}\(q\), and the Spearman rank coefficientρsp\\rho\_\{\\mathrm\{sp\}\}as a robustness check\. Significance uses a two\-tailedtt\-test at levelα=0\.01\\alpha=0\.01with statistict=ρ​M−2/1−ρ2t=\\rho\\sqrt\{M\-2\}/\\sqrt\{1\-\\rho^\{2\}\}, whereMMis the number of questions\.

Table[12](https://arxiv.org/html/2609.28570#A6.T12)reports the resulting correlations \(ρ\\rho: Pearson;ρsp\\rho\_\{\\mathrm\{sp\}\}: Spearman; allp<0\.01p<0\.01, two\-tailedtt\-test\)\. The alignment is already present in the GRPO baseline and strengthens substantially with the hint branch on both benchmarks, supporting the claim that the two entropy signals are linked through predictive uncertainty rather than interchangeable\.

Table 12:Correlation betweenH¯1\\bar\{H\}\_\{1\}andHsH\_\{s\}on multimodal benchmarks \(Pearsonρ\\rho/ Spearmanρsp\\rho\_\{\\mathrm\{sp\}\}\), for models trained without and with the hint branch\. All coefficients significant atp<0\.01p<0\.01\.

## Appendix GTraining Details

### G\.1Implementation Specification

For reproducibility we consolidate the design choices and hyperparameters that the main text references\.*Semantic\-entropy estimation:*answers are extracted from the rollout suffix after the final\-answer marker, lowercased and stripped of articles and punctuation; numeric answers are canonicalized\. Answers are clustered into semantic\-equivalence classes by pairwise DeBERTa\-MNLI entailment \(microsoft/deberta\-v2\-xlarge\-mnli\) under a lax rule—two answers are equivalent when they are mutually non\-contradictory and at least one direction entails the other; cluster probabilities are count\-based,P⁡\(Ck∣q\)=\|Ck\|/NP\(C\_\{k\}\\mid q\)=\|C\_\{k\}\|/N, andHsH\_\{s\}is the natural\-log Shannon entropy over these frequencies\.*Prefix truncation:*the same final\-answer marker delimits the thought/answer splity∗=\[ythought∗,yans∗\]y^\{\*\}=\[\\,y^\{\*\}\_\{\\mathrm\{thought\}\},y^\{\*\}\_\{\\mathrm\{ans\}\}\\,\]of each expert trace; prefixes are cut from the thought segment only, atLhL\_\{h\}tokens given by Eq\.[1](https://arxiv.org/html/2609.28570#S4.E1)with truncation capαmax=0\.5\\alpha\_\{\\max\}=0\.5, so the prefix never reaches the answer segment \(we verified on the full training set that no prefix contains the gold answer string\)\.*Semantic\-leakage audit:*token\-level exclusion does not by itself rule out semantic leakage, since an early reasoning prefix could still determine the conclusion\. We therefore audit every candidate prefix with the same DeBERTa\-MNLI model used for clustering: the gold answer is verbalized as a declarative statement, and a prefix is used only if the model does not predict entailment from the prefix to that statement\. On the 20k training traces,3\.1%3\.1\\%of candidate prefixes fail this test and are discarded \(the query then receives no prefix for that iteration\); a manual inspection of 500 filtered prefixes found none that determines the final label\.*Trigger:*the threshold is initialized atτs=0\.8\\tau\_\{s\}=0\.8and updated by exponential moving average with decayβ\\beta;Hsmax=log⁡NH\_\{s\}^\{\\max\}=\\log N; hint dropout with ratepdropp\_\{\\mathrm\{drop\}\}is applied, and the trigger is evaluated before dropout\.*Scaling:*τt\\tau\_\{t\}is computed fromπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}’s fp32 log\-probabilities with stop\-gradient;wmax=20w\_\{\\max\}=20\(ε0=0\.05\\varepsilon\_\{0\}=0\.05\);g⋆g^\{\\star\}is mean\-normalized over the batch and applied after group advantage normalization and before clipping; we log the effective sample size of the weights and the clipped\-token fraction as diagnostics\.*Optimization:*clipεclip=0\.2\\varepsilon\_\{\\mathrm\{clip\}\}=0\.2, KL coefficientβkl=0\.04\\beta\_\{\\mathrm\{kl\}\}=0\.04, prefix\-loss weightλ\\lambda, actor learning rate5×10−75\\times 10^\{\-7\};G=8G=8, sampling temperature1\.21\.2\.

![Refer to caption](https://arxiv.org/html/2609.28570v1/pic/Stability_Exp/combined_stability_metrics.png)Figure 5:Effect of Gradient Scaling on RL stability\. Adaptive weighting stabilizes RL loss, accelerates reward convergence, and reduces policy clipping, yielding smoother and more robust training\.We fine\-tune the Qwen2\.5\-VL\-7B model under the GRPO framework to enhance multimodal reasoning and reduce hallucination\. The training covers both image–text and video–text modalities, unified through a shared tokenizer to maintain consistent encoding across modalities\.

### Data Preprocessing\.

We employ a unified visual–text tokenizer that handles both static and temporal visual inputs\. For image–text pairs, the images are dynamically resized based on token limits \(4–16,384 tokens\) while preserving aspect ratio\. For video–text pairs, the preprocessing dynamically adjusts frame counts between 4 and 64 based on original FPS and duration, ensuring each clip maintains a consistent spatial resolution and frame factor of 2\.

### Training Framework\.

We adopt the GRPO reinforcement learning setup[Shao et al\. \(2024\)](https://arxiv.org/html/2609.28570#bib.bib30), which is critic\-free: advantages are estimated relative to the group of sampled responses \(Sec\.[3](https://arxiv.org/html/2609.28570#S3)\), and no value network is trained\. For each GRPO iteration,G=8G=8candidate responses are sampled per multimodal input \(image– or video–question pair\) using a temperature of 1\.2\. The policy is updated with Adam at a learning rate of5×10−75\\times 10^\{\-7\}\. A KL penalty of0\.040\.04constrains the updated policy within the trust region defined by the reference model, stabilizing optimization\.

### Hyperparameters\.

Table[13](https://arxiv.org/html/2609.28570#A7.T13)consolidates the configuration referenced throughout the paper\.

Table 13:Full hyperparameter configuration\.
### Dataset and Sampling\.

Our models are trained on the large\-scale multimodal dataset introduced in Video\-R1[Feng et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib39), selecting 20k samples as the final training subset\. Videos are preprocessed through the provided sampling pipeline, which adaptively determines frame counts based on original FPS and duration while maintaining aspect ratio and pixel limits\. Each sample is converted into an instruction\-style multimodal prompt for consistency with the training objective\. The reward is verifiable: task accuracy on the final answer plus a format\-compliance bonus, following[Feng et al\. \(2025\)](https://arxiv.org/html/2609.28570#bib.bib39)\.

### Hardware and Schedule\.

All experiments are conducted on 8×\\timesNVIDIA A100 \(80GB\) GPUs\. Training runs for 2,000 GRPO steps \(approximately 3 days of pure training time at≈130\{\\approx\}130s/step for the full model\), with evaluation conducted every 100 steps using a fixed set of held\-out prompts; periodic evaluation and checkpointing add a small additional overhead not included in the per\-step timing of Table[14](https://arxiv.org/html/2609.28570#A7.T14)\.

### Per\-Iteration Procedure

Algorithm[1](https://arxiv.org/html/2609.28570#alg1)summarizes one training iteration of DEEPO\. The procedure first estimates query\-level uncertainty from grouped rollouts and uses an EMA threshold to identify ambiguous samples that benefit from expert guidance\. For these samples, DEEPO conditionally injects a truncated expert prefix, while the token\-level scaling branch independently reweights policy gradients according to local predictive uncertainty\. Both branches are integrated into the same GRPO update, with hinted and unhinted trajectories sharing the group\-relative baseline\.

Algorithm 1One DEEPO training iteration0:policy

πθ\\pi\_\{\\theta\}, reference

πref\\pi\_\{\\mathrm\{ref\}\}, a batch of queries with expert traces, group size

GG, threshold state

τ^\\hat\{\\tau\}\(initialized at

τs\\tau\_\{s\}\), EMA decay

β\\beta, dropout rate

pdropp\_\{\\mathrm\{drop\}\}, cap

wmaxw\_\{\\max\},

λ\\lambda,

βkl\\beta\_\{\\mathrm\{kl\}\}
1:Sample

GGrollouts per query with

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(temp\.

1\.21\.2\)

2:Extract answers; cluster by DeBERTa\-MNLI entailment \(lax rule\); compute

Hs​\(q\)H\_\{s\}\(q\)per query

3:Update threshold:

τ^←\(1−β\)​τ^\+β​H¯s\\hat\{\\tau\}\\leftarrow\(1\-\\beta\)\\hat\{\\tau\}\+\\beta\\,\\overline\{H\}\_\{s\}
4:for allqueries

qqwith

Hs​\(q\)\>τ^H\_\{s\}\(q\)\>\\hat\{\\tau\}do

5:With prob\.

1−pdrop1\-p\_\{\\mathrm\{drop\}\}: truncate the*thought segment*of the expert trace to

LhL\_\{h\}tokens \(Eq\.[1](https://arxiv.org/html/2609.28570#S4.E1); answer segment

yans∗y^\{\*\}\_\{\\mathrm\{ans\}\}excluded\); sample continuation

yqrem∼πθold\(⋅∣q⊕yqpre\)y\_\{q\}^\{\\mathrm\{rem\}\}\\sim\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(\\cdot\\mid q\\oplus y\_\{q\}^\{\\mathrm\{pre\}\}\)
6:endfor

7:Score all trajectories with the verifiable reward

8:Compute group\-relative advantages over the full group: hinted continuations and first\-pass unhinted rollouts share one baseline \(in\-group variance anchoring, Sec\.[4\.1](https://arxiv.org/html/2609.28570#S4.SS1)\)

9:Compute

τt=1−e−H2,t\\tau\_\{t\}=1\-e^\{\-H\_\{2,t\}\}per token from

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}in fp32 \(stop\-grad\); form

g⋆g^\{\\star\}\(Eq\.[5](https://arxiv.org/html/2609.28570#S4.E5)\)

10:Update

πθ\\pi\_\{\\theta\}by minimizing Eq\.[6](https://arxiv.org/html/2609.28570#S4.E6)\(prefix masked in RL term;

A~t=g⋆​At\\tilde\{A\}\_\{t\}=g^\{\\star\}A\_\{t\}pre\-clip\)

### Efficiency Decomposition

Table[14](https://arxiv.org/html/2609.28570#A7.T14)decomposes the additional training cost introduced by the two DEEPO branches\. Gradient scaling adds little overhead relative to GRPO, whereas the expert\-hint branch accounts for most of the extra wall\-clock time due to answer clustering and conditional continuation generation\.

Table 14:Efficiency–performance trade\-off\. “SE Time Share” covers answer extraction and clustering forHsH\_\{s\}only; the conditional continuation pass and the prefix forward pass are separate delta components\.