Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning

# Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning
Source: [https://arxiv.org/html/2609.18057](https://arxiv.org/html/2609.18057)
Xinxin Song††thanks:Equal Contribution\.Siyuan Li11footnotemark:1Affiliation:Department of Automation, Tsinghua UniversityTingxiong XiaoAffiliation:Department of Automation, Tsinghua UniversityJinli Suo††thanks:Corresponding author\.Affiliation:Department of Automation, Tsinghua UniversityAffiliation:Institute for Brain and Cognitive Science, Tsinghua University

###### Abstract

Reinforcement learning with verifiable rewards \(RLVR\) has significantly improved the reasoning capabilities of large vision\-language models \(LVLMs\)\. However, standard on\-policy RLVR algorithms face a critical optimization bottleneck in preserving and reinforcing visually grounded reasoning behaviors: valuable visually\-grounded reasoning trajectories are discarded after a single update, while uniform token advantage allocation prevents the model from reinforcing critical perception or reasoning steps\. To bridge this gap, we propose PIVOT, a dual\-level learning framework that anchors policy optimization around informative visual reasoning signals\. Specifically, PIVOT introduces a self\-calibrated experience replay mechanism, which selectively collects and replays visually\-grounded historical experiences as stable reference anchors for policy optimization\. Building upon this, we further design a vision\-guided advantage allocation mechanism to allocate additional vision\-aware advantages to tokens based on their local visual support and impact on downstream reasoning\. Extensive experiments across diverse benchmarks demonstrate that PIVOT achieves highly competitive performance in enhancing the multimodal reasoning capabilities of LVLMs\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) has become a prevailing post\-training paradigm for enhancing the reasoning capabilities of large language models\([Zhang et al\., 2025c](https://arxiv.org/html/2609.18057#bib.bib42)\)\. Given its success in the text domain, recent studies have extended RLVR to large vision\-language models \(LVLMs\) and achieved promising results in visual reasoning tasks\([Liu et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib16)\)\. However, directly applying standard RLVR algorithms to LVLMs exposes a critical optimization bottleneck: sparse yet informative learning signals are neither sufficiently preserved nor effectively exploited, leading to inefficient optimization for visually grounded reasoning\. Specifically, this inefficiency manifests at two interconnected levels\.

At thetrajectory level, standard on\-policy RL algorithms typically discard the generated trajectories111We use the termstrajectory,experience, andresponseinterchangeably throughout this paper\.after each update phase\. For LVLMs, obtaining high\-quality visually grounded trajectories is non\-trivial due to the expanded multimodal search space\. These trajectories explicitly demonstrate how to ground visual evidence into logical steps, containing richer information than a sparse binary reward\. Discarding them indiscriminately results in a waste of optimization signals\. At thetoken level, the issue of uniform credit assignment further dilutes the learning signals\. Outcome\-based RLVR algorithms typically assign the same scalar advantage to all tokens within a trajectory\. However, the visual reasoning chain is highly heterogeneous: certain tokens act as pivotal points for visual perception or logical branching, whereas others are mere linguistic fillers\. Assigning uniform optimization strength to all tokens prevents the model from anchoring on the critical perception or reasoning steps, causing informative token\-level signals to be diluted across the entire response\.

To address the above limitations, we propose PIVOT, a dual\-level learning framework that Preserves and reinforces sparse but Informative learning signals for Visually\-grOunded reasoning at both the trajectory and Token levels\. Specifically, at the trajectory level, we introduce a self\-calibrated experience replay mechanism, which builds a dynamic experience pool to collect and selectively replay informative prompts alongside their high\-quality, visually\-grounded historical trajectories\. These trajectories act as retrospective anchors rather than direct training objectives, gently constraining the policy without stifling exploration\. Beyond trajectory\-level anchoring, we further introduce a vision\-guided advantage allocation strategy\. This mechanism obtains fine\-grained visual learning signals by calculating the counterfactual visual support of sampled tokens and estimating their impact on future reasoning\. These signals are then used for fine\-grained advantage allocation to concentrate the optimization strength on more valuable perception and reasoning tokens\. The main contributions of our method are summarized as follows:

- •We identify a key optimization bottleneck in multimodal RLVR: valuable visual reasoning signals are diluted across both trajectory sampling and token\-level optimization, making it difficult for the policy to consistently reinforce visually grounded reasoning behaviors\.
- •We propose PIVOT, a unified dual\-level framework that preserves high\-value visual reasoning experiences as retrospective anchors and reinforces optimization on visually grounded tokens via fine\-grained advantage modulation\.
- •Extensive experiments on various data and model scales demonstrate that PIVOT consistently improves multimodal reasoning performance over baselines\.

## 2Related Works

RLVR for Multimodal Reasoning\.Given the success of RLVR in the text domain, recent works have begun to explore its application to LVLMs\. From data perspective, methods like Vision\-R1\([Huang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib5)\)and MM\-EUREKA\([Meng et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib24)\)construct high\-quality CoT datasets, while NoisyRollout\([Liu et al\., 2025a](https://arxiv.org/html/2609.18057#bib.bib15)\)injects perceptual diversity through data augmentation\. For reward design, Perception\-R1\([Yu et al\., 2025a](https://arxiv.org/html/2609.18057#bib.bib36)\)designs specialized reward mechanisms for different visual tasks, and Vision\-SR1\([Li et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib11)\)decomposes visual perception and language reasoning to formulate a self\-reward mechanism\. For optimization objectives, PAPO\([Wang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib32)\)introduces an additional KL loss to improve perceptual capabilities\. Unlike prior work that mainly improves multimodal RL through stronger supervision signals, we revisit the optimization process and study how high\-value visual reasoning signals can be more effectively preserved and exploited during policy learning\.

Experience\-based RL\.Experience replay\([Lin, 1992](https://arxiv.org/html/2609.18057#bib.bib13)\)has been extensively studied in classical RL tasks\([Wang et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib30)\)\. Recent studies have extended it to LLMs\. For instance, RLEP\([Zhang et al\., 2025a](https://arxiv.org/html/2609.18057#bib.bib40)\)introduces a two\-stage framework of collection followed by replay; ReMix\([Liang et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib12)\)formulates a stable optimization objective for off\-policy training; and ExGRPO\([Zhan et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib39)\)prioritizes the replay of high\-value trajectories based on systematic metric analysis\. For LVLMs, EFRame\([Wang et al\., 2025a](https://arxiv.org/html/2609.18057#bib.bib27)\)replays successful trajectories for hard prompts, while VL\-Rethinker[Wang et al\. \(2025b\)](https://arxiv.org/html/2609.18057#bib.bib28)mitigates the vanishing advantage problem via selective sample replay\. In this work, we emphasize visual perception quality during experience collection while treating historical experience as a reference anchor for policy updates rather than a direct optimization objective, which yields better results\. CalibRL\([Huang et al\., 2026c](https://arxiv.org/html/2609.18057#bib.bib6)\)adopts a similar optimization objective to ours, but it relies on external expert demonstrations instead of self\-generated experience\.

Token Optimization in RLVR\.Recent works have revealed the heterogeneity and sparsity of the token optimization signal in RLVR updates\([Wang et al\., 2025c](https://arxiv.org/html/2609.18057#bib.bib29)\)\. In LVLMs, prior works assess token\-level visual dependence using counterfactual output divergence\([Ye et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib35);[Huang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib4)\), hidden state similarities\([Wang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib31)\), or visual attention scores\([Jiao et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib8);[Luo et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib21)\)to reinforce tokens with strong perceptual grounding\. Methods like PEPO\([Li et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib9)\)and ToR\([Lu et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib17)\)further integrate this with high\-entropy reasoning tokens\. Building upon counterfactual support for local visual dependence, we further formulate an entropy\-gated future visual support to capture downstream impact, enabling fine\-grained advantage allocation\.

## 3Preliminaries

### 3\.1Multimodal RLVR

Consider a multimodal reasoning problem where each inputxxconsists of a textual queryqqand an imageII, denoted byx=\(q,I\)x=\(q,I\)\. An LVLM parameterized byπθ\\pi\_\{\\theta\}generates a textual response sequence𝐲=\(y1,…,yT\)\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{T\}\)autoregressively according to

πθ​\(𝐲∣x\)=∏t=1Tπθ​\(yt∣q,I,y<t\)\.\\pi\_\{\\theta\}\(\\mathbf\{y\}\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid q,I,y\_\{<t\}\)\.\(1\)For reasoning tasks, the generated sequence𝐲\\mathbf\{y\}typically consists of an intermediate multi\-step reasoning chain followed by the final answer\. In the RLVR framework, a rule\-based verifier assigns a binary rewardR∈\{0,1\}R\\in\\\{0,1\\\}to each response based solely on whether its final extracted answer matches the ground truthaa\.

DAPO\([Yu et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib37)\)is a widely used RLVR algorithm for reasoning tasks\. It samples a group ofGGcandidate responses\{𝐲i\}i=1G\\\{\\mathbf\{y\}\_\{i\}\\\}\_\{i=1\}^\{G\}from old policyπθold\\pi\_\{\\theta\_\{\\text\{old\}\}\}and computes the advantageA^i\\hat\{A\}\_\{i\}for theii\-th candidate response𝐲i\\mathbf\{y\}\_\{i\}by

A^i=Ri−mean​\(\{Rk\}k=1G\)std​\(\{Rk\}k=1G\)\.\\hat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\text\{mean\}\(\\\{R\_\{k\}\\\}\_\{k=1\}^\{G\}\)\}\{\\text\{std\}\(\\\{R\_\{k\}\\\}\_\{k=1\}^\{G\}\)\}\.\(2\)The policy parametersθ\\thetaare then updated by minimizing the following clipping objective:

ℒ⁡\(θ\)\\displaystyle\\mathcal\{L\}\(\\theta\)=−𝔼\[1∑i=1G\|𝐲i\|∑i=1G∑t=1\|𝐲i\|min\(ρi,t\(θ\)A^i,t,\\displaystyle=\-\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{\\sum\_\{i=1\}^\{G\}\|\\mathbf\{y\}\_\{i\}\|\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|\\mathbf\{y\}\_\{i\}\|\}\\min\\Big\(\\rho\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i,t\},\(3\)clip\(ρi,t\(θ\),1−ϵl,1\+ϵh\)A^i,t\)\],\\displaystyle\\text\{clip\}\\big\(\\rho\_\{i,t\}\(\\theta\),1\-\\epsilon\_\{l\},1\+\\epsilon\_\{h\}\\big\)\\hat\{A\}\_\{i,t\}\\Big\)\\Bigg\],whereρi,t​\(θ\)=πθ​\(yi,t\|q,I,𝐲i,<t\)πθold​\(yi,t\|q,I,𝐲i,<t\)\\rho\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{i,t\}\|q,I,\\mathbf\{y\}\_\{i,<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{i,t\}\|q,I,\\mathbf\{y\}\_\{i,<t\}\)\}is the importance sampling ratio,ϵl\\epsilon\_\{l\}andϵh\\epsilon\_\{h\}are the clipping hyperparameters\.

### 3\.2Trajectory Quality Estimation

To evaluate the quality of generated trajectories in multimodal RLVR, we consider two complementary signals that capture different aspects of trajectory reliability\.

We first consider the visual dependency of a trajectory\. Following previous work\([Huang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib4)\), we quantify it by computing the KL divergence between the predictive distribution of the policy conditioned on the original image and a corrupted version\. Given the multimodal input\(q,I\)\(q,I\)and the generated trajectory𝐲\\mathbf\{y\}, we will perform counterfactual interventions \(such as random patch masking\([Wang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib32)\)\) on the original imageIIto obtain a corrupted imageI~\\widetilde\{I\}, then we define the visual dependency as:

V⁡\(𝐲\)\\displaystyle V\(\\mathbf\{y\}\)=1T∑t=1TDKL\(πθ\(⋅∣q,I,𝐲<t\)\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}D\_\{\\mathrm\{KL\}\}\\Big\(\\pi\_\{\\theta\}\(\\cdot\\mid q,I,\\mathbf\{y\}\_\{<t\}\)\(4\)∥πθ\(⋅∣q,I~,𝐲<t\)\)\.\\displaystyle\\\|\\;\\pi\_\{\\theta\}\(\\cdot\\mid q,\\widetilde\{I\},\\mathbf\{y\}\_\{<t\}\)\\Big\)\.A higherV⁡\(𝐲\)V\(\\mathbf\{y\}\)indicates that the trajectory is more grounded in visual evidence rather than relying solely on language priors\. Then, we consider the stability of the reasoning process and use the average trajectory entropy as its proxy:

H\(𝐲\)=1T∑t=1Tℋ\(πθ\(⋅∣q,I,𝐲<t\)\),H\(\\mathbf\{y\}\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathcal\{H\}\\big\(\\pi\_\{\\theta\}\(\\cdot\\mid q,I,\\mathbf\{y\}\_\{<t\}\)\\big\),\(5\)whereℋ\(p\)=−∑xp\(x\)logp\(x\)\\mathcal\{H\}\(p\)=\-\\sum\_\{x\}p\(x\)\\log p\(x\)denotes the entropy of the token distribution\. ExGRPO\([Zhan et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib39)\)has observed that lower\-entropy trajectories tend to exhibit more stable and higher\-quality reasoning behaviors under RLVR\. Together,V⁡\(𝐲\)V\(\\mathbf\{y\}\)andH⁡\(𝐲\)H\(\\mathbf\{y\}\)provide complementary views of trajectory quality: the former captures grounding in visual evidence, while the latter reflects the stability of the reasoning process\. These signals are used as complementary criteria for selecting high\-quality trajectories in experience replay\.

## 4Method

In this section, we introduce PIVOT, a dual\-level learning framework designed to preserve and amplify sparse yet informative learning signals for visually\-grounded reasoning\. Our approach extends the standard RLVR paradigm with two synergistic components: \(1\) a self\-calibrated experience replay mechanism that collects and reuses high\-quality past experiences to guide the policy optimization, and \(2\) a vision\-guided advantage allocation strategy that reallocate token advantages to prioritize visually critical reasoning steps\. These components work in tandem to transform rare successes into dense and informative signals, allowing the model to more effectively internalize visually grounded reasoning patterns\. Figure[1](https://arxiv.org/html/2609.18057#S4.F1)provides a schematic overview of our framework\.

![Refer to caption](https://arxiv.org/html/2609.18057v1/method_overview.png)Figure 1:Overview of the PIVOT framework\. Standard RLVR is augmented by two synergistic modules: \(1\)Self\-Calibrated Experience Replaymodule selectively collects and replays valuable prompts alongside high\-quality experiences to guide policy optimization through a self\-calibration loss; and \(2\)Vision\-Guided Advantage Allocationmodule allocates token advantages by combining local counterfactual visual support with entropy\-gated future influence to reinforce pivotal perception or reasoning steps\.### 4\.1Self\-Calibrated Experience Replay

This mechanism primarily addresses the issue of high\-quality trajectories being discarded during on\-policy sampling, which results in wasted optimization signals\. It achieves this by collecting and selectively replaying high\-quality trajectories and using them as reference anchors to guide policy optimization\.

#### Experience Pool Curation\.

During the rollout phase, the model generatesGGcandidate responses\{𝐲i\}i=1G\\\{\\mathbf\{y\}\_\{i\}\\\}\_\{i=1\}^\{G\}for a given multimodal inputx∗=\(q∗,I∗\)x^\{\*\}=\(q^\{\*\},I^\{\*\}\), which are then evaluated by a rule\-based verifier\. We denote the subset ofkksuccessful responses as𝒴\+=\{y\+∗\}\\mathcal\{Y\}^\{\+\}=\\\{y\_\{\+\}^\{\*\}\\\}, and estimate the empirical success probability for this input asp^​\(x∗\)=k/G\\hat\{p\}\(x^\{\*\}\)=k/G\. To efficiently manage these experiences, we store them in an experience bufferℬ\\mathcal\{B\}as a structured mappingx∗↦\{p^​\(x∗\),𝒴\+\}x^\{\*\}\\mapsto\\\{\\hat\{p\}\(x^\{\*\}\),\\mathcal\{Y\}^\{\+\}\\\}\. This design uniquely associates each input with its empirical success probability and a set of successful responses\.

#### Selective Experience Replay\.

A naive replay strategy involves randomly selecting prompts and successful trajectories from the experience pool\. However, previous work\([Zhan et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib39)\)has shown that this approach is suboptimal\. We argue that prompts at thefrontier of the model’s capabilitiesprovide the most informative training signal\. And we identify these prompts by estimating their reward variance, which has been shown to be a good proxy both theoretically and empirically\([Foster et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib1);[Jiang et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib7)\)\. For a multimodal promptx∗x^\{\*\}in the experience buffer, the variance can be estimated byp^​\(x∗\)​\(1−p^​\(x∗\)\)\\hat\{p\}\(x^\{\*\}\)\(1\-\\hat\{p\}\(x^\{\*\}\)\)\. When constructing a training batch of sizeNN, we sampleNoffN\_\{\\text\{off\}\}prompts from the experience buffer with a probability proportional to their estimated variance:

P⁡\(sample prompt​x∗\)∝p^​\(x∗\)​\(1−p^​\(x∗\)\)P\(\\text\{sample prompt \}x^\{\*\}\)\\propto\\hat\{p\}\(x^\{\*\}\)\(1\-\\hat\{p\}\(x^\{\*\}\)\)\(6\)This variance\-driven sampling naturally creates a curriculum that focuses the replay mechanism on partially solved problems\.

For a selected prompt, the experience buffer may contain multiple successful historical trajectories\. Instead of random selection, we will select a trajectory that is both visually grounded and stable under the current model\. Specifically, based on the metrics defined in Section[3\.2](https://arxiv.org/html/2609.18057#S3.SS2), we use the visual dependency scoreV⁡\(𝐲\)V\(\\mathbf\{y\}\)to quantify how strongly a trajectory relies on visual information, and the trajectory entropyH⁡\(𝐲\)H\(\\mathbf\{y\}\)to measure the stability of the reasoning process\. We employ a two\-stage filtering process to select the optimal one from the candidate set𝒴\+\\mathcal\{Y\}^\{\+\}\. First, we rank all trajectories in𝒴\+\\mathcal\{Y\}^\{\+\}by their entropyH⁡\(𝐲\)H\(\\mathbf\{y\}\)in ascending order, and retain the topα%\\alpha\\%\(e\.g\.,α=50\\alpha=50\) with the lowest entropy to form a stable subset𝒴∗\\mathcal\{Y\}^\{\*\}\. Subsequently, among these remaining high\-quality candidates, we select the one that maximizes the visual dependency score as our trajectory\-level anchor𝐲exp\\mathbf\{y\}^\{\\text\{exp\}\}:

𝐲exp=arg⁡max𝐲∈𝒴∗⁡V⁡\(𝐲\)\\mathbf\{y\}^\{\\text\{exp\}\}=\\mathop\{\\arg\\max\}\_\{\\mathbf\{y\}\\in\\mathcal\{Y\}^\{\*\}\}V\(\\mathbf\{y\}\)\(7\)

#### Experience Calibration Loss\.

After the experience sampling process described above, we have constructed a batch ofNNprompts\. Among them,NoffN\_\{\\text\{off\}\}prompts are selectively sampled from the experience buffer, each paired with its optimal historical response𝐲exp\\mathbf\{y\}^\{\\text\{exp\}\}\. The model will perform a standard rollout phase on this batch, and each prompt will get a group ofGGresponses, denoted as\{𝐲icur\}i=1G\\\{\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}\\\}\_\{i=1\}^\{G\}\.

For prompts replayed from the experience buffer, we introduce an additional self\-calibration loss to uncover valuable signals from historical experiences\. Rather than treating the historical experiences as a direct optimization target, we utilize them as stable reference anchors for self\-calibration\. We measure the policy’s relative preference between its newly generated trajectory𝐲icur\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}and the historical experience𝐲exp\\mathbf\{y\}^\{\\text\{exp\}\}by defining the log\-confidence gap:

Δ​ℓi=log⁡πθ​\(𝐲icur∣q,I\)−sg​\[log⁡πθ​\(𝐲exp∣q,I\)\]\\Delta\\ell\_\{i\}=\\log\\pi\_\{\\theta\}\(\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}\\mid q,I\)\-\\text\{sg\}\\left\[\\log\\pi\_\{\\theta\}\(\\mathbf\{y\}^\{\\text\{exp\}\}\\mid q,I\)\\right\]\(8\)wheresg​\[⋅\]\\text\{sg\}\[\\cdot\]denotes a stop\-gradient operator, ensuring𝐲exp\\mathbf\{y\}^\{\\text\{exp\}\}serves as a stable reference anchor\. To dynamically guide the policy based on the correctness of its current exploration, we formulate a contrastive margin loss:

ℒexp=\|Ai\|⋅softplus\(−s⋅Δℓi\)\\mathcal\{L\}\_\{\\text\{exp\}\}=\|A\_\{i\}\|\\cdot\\operatorname\{softplus\}\\left\(\-s\\cdot\\Delta\\ell\_\{i\}\\right\)\(9\)where\|Ai\|\|A\_\{i\}\|denotes the magnitude of the group\-wise advantage for𝐲icur\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}, andssis an indicator variable set to\+1\+1if𝐲icur\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}is correct and−1\-1if it is incorrect\. In practice, we implement this objective at the token level and normalize it over all valid response tokens, providing more fine\-grained calibration signals\.

This objective acts as a dynamic self\-calibration mechanism\. For successful explorations \(s=\+1s=\+1\), it increases the policy’s confidence in𝐲icur\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}, while the softplus operator naturally decays gradients onceΔ​ℓi\>0\\Delta\\ell\_\{i\}\>0to prevent over\-optimization\. Conversely, for incorrect trajectories \(s=−1s=\-1\), the historical anchor acts as a dynamic threshold: the loss penalizes the flawed𝐲icur\\mathbf\{y\}^\{\\text\{cur\}\}\_\{i\}until its confidence drops safely below the proven baseline\.

### 4\.2Vision\-Guided Advantage Allocation

Although the self\-calibrated experience replay mechanism preserves and leverages high\-quality historical experiences by introducing an additional calibration loss, standard RLVR still broadcasts a uniform advantage to all tokens within a trajectory, resulting in the dilution of optimization signals at the token level\. We address this by modulating the token advantage based on its visual utility\.

#### Quantifying Token Visual Utility\.

We begin by estimating the direct counterfactual visual support for each generated token\. This is achieved by comparing the model’s predictive probability under the original imageIIagainst the counterfactually corrupted imageI~\\widetilde\{I\}\(see Section[3\.2](https://arxiv.org/html/2609.18057#S3.SS2)\)\. We formulate this metric as:

ct=1−πθ​\(yt∣q,I~,𝐲<t\)πθ​\(yt∣q,I,𝐲<t\)\.c\_\{t\}=1\-\\frac\{\\pi\_\{\\theta\}\(y\_\{t\}\\mid q,\\widetilde\{I\},\\mathbf\{y\}\_\{<t\}\)\}\{\\pi\_\{\\theta\}\(y\_\{t\}\\mid q,I,\\mathbf\{y\}\_\{<t\}\)\}\.\(10\)This formulation captures the relative change in the model’s support for the sampled token after visual perturbation\. A positivectc\_\{t\}indicates that the sampled token receives positive support from the visual evidence, suggesting it is more likely to be a perception\-critical token\. Conversely,ct≈0c\_\{t\}\\approx 0orct<0c\_\{t\}<0implies a lack of visual support, indicating that the token might be generated from language priors or visual hallucinations\. Detailed justification is provided in Appendix[A](https://arxiv.org/html/2609.18057#A1)\.

However, using only local counterfactual support as visual utilities may overlook the impact of tokens on future visual reasoning processes\. To capture this impact, we define the future\-discounted visual support score:

Ft=∑k\>tγk−t−1​ck∑k\>tγk−t−1,F\_\{t\}=\\frac\{\\sum\_\{k\>t\}\\gamma^\{k\-t\-1\}c\_\{k\}\}\{\\sum\_\{k\>t\}\\gamma^\{k\-t\-1\}\},\(11\)whereγ∈\[0,1\]\\gamma\\in\[0,1\]is a discount factor\. In implementation, the summation is truncated to a finite window of sizeWW, which removes noisy long\-tail effects from distant tokens\.

To avoid indiscriminately propagating future visual credit to all preceding tokens, we further introduce an entropy\-based gate to highlight reasoning\-pivot tokens\([Wang et al\., 2025c](https://arxiv.org/html/2609.18057#bib.bib29)\)\. LetHtH\_\{t\}denote the token entropy andH¯\\bar\{H\}the average entropy over the batch\. We define

ut=1−exp⁡\(−HtH¯\)\.u\_\{t\}=1\-\\exp\\left\(\-\\frac\{H\_\{t\}\}\{\\bar\{H\}\}\\right\)\.\(12\)This gate assigns larger weights to high\-entropy positions, which are more likely to be reasoning decision points\. The final token visual utility is

Ut=ct\+λ⋅Detrending⁡\(ut​Ft\),U\_\{t\}=c\_\{t\}\+\\lambda\\cdot\\operatorname\{Detrending\}\(u\_\{t\}F\_\{t\}\),\(13\)whereλ\\lambdabalances local visual support and future\-aware influence\.Detrending⁡\(⋅\)\\operatorname\{Detrending\}\(\\cdot\)is a standard OLS detrending operator\([Wu et al\., 2007](https://arxiv.org/html/2609.18057#bib.bib33)\)to remove potential systematic position bias; implementation details are provided in Appendix[B](https://arxiv.org/html/2609.18057#A2)\.

#### Advantage Modification\.

The final token visual utility scoreUtU\_\{t\}not only measures local visual support but also captures its impact on downstream visual reasoning\. We then use it to modulate the original advantageA^i,t\\hat\{A\}\_\{i,t\}\(which is uniform across tokens within the trajectory\):

A^i,t′=A^i,t\+β​\|A^i,t\|​Ut,\\hat\{A\}^\{\\prime\}\_\{i,t\}=\\hat\{A\}\_\{i,t\}\+\\beta\|\\hat\{A\}\_\{i,t\}\|U\_\{t\},\(14\)whereβ\\betais a scaling factor\. Then we apply the following sign protection operation toA^i,t′\\hat\{A\}^\{\\prime\}\_\{i,t\}:

A^i,tfinal=\{max⁡\(A^i,t′,0\),if​Ri=1,min⁡\(A^i,t′,0\),if​Ri=0\.\\hat\{A\}\_\{i,t\}^\{\\text\{final\}\}=\\begin\{cases\}\\max\(\\hat\{A\}^\{\\prime\}\_\{i,t\},0\),&\\text\{if \}R\_\{i\}=1,\\\\ \\\\ \\min\(\\hat\{A\}^\{\\prime\}\_\{i,t\},0\),&\\text\{if \}R\_\{i\}=0\.\\end\{cases\}\(15\)This operation prevents the modulation from reversing the optimization directions of positive and negative samples\. All computations within this module are detached from the gradient graph, as they are only utilized to modulate the scalar token advantages\. The policy gradient is then computed using the refined token advantages\.

### 4\.3Final Training Objective

Our dual\-level learning framework is integrated into the standard policy optimization process\. The final optimization objective combines the RLVR loss with our modulated advantagesA^i,tfinal\\hat\{A\}^\{\\text\{final\}\}\_\{i,t\}, and the self\-calibration experience lossℒexp\\mathcal\{L\}\_\{\\text\{exp\}\}:

ℒfinal​\(θ\)=ℒRLVR​\(θ,A^i,tfinal\)\+λexp​ℒexp​\(θ\),\\mathcal\{L\}\_\{\\text\{final\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\text\{RLVR\}\}\(\\theta;\\hat\{A\}^\{\\text\{final\}\}\_\{i,t\}\)\+\\lambda\_\{\\text\{exp\}\}\\mathcal\{L\}\_\{\\text\{exp\}\}\(\\theta\),\(16\)whereℒRLVR​\(θ,A^i,tfinal\)\\mathcal\{L\}\_\{\\text\{RLVR\}\}\(\\theta;\\hat\{A\}^\{\\text\{final\}\}\_\{i,t\}\)is the standard RLVR loss using the modified advantageA^i,tfinal\\hat\{A\}^\{\\text\{final\}\}\_\{i,t\}, andλexp\\lambda\_\{\\text\{exp\}\}is a coefficient balancing the two components\. This objective not only preserves the on\-policy learning signal but also anchors policy optimization to high\-quality historical experiences and visually critical tokens, improving the utilization efficiency of high\-value optimization signals in RLVR\.

## 5Experiments

### 5\.1Experimental Setup

#### Models and Baselines\.

We adopt Qwen\-2\.5\-VL\-3B and Qwen\-2\.5\-VL\-7B as our base models\. To evaluate data scalability, we first conducted experiments on the Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2609.18057#bib.bib19)\)dataset and further extended the training to the larger\-scale VIRL39K dataset\([Wang et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib28)\)\. We reproduce the algorithms GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib26)\), DAPO\([Yu et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib37)\), PAPO\([Wang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib32)\), and VPPO\([Huang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib4)\)across different model scales and datasets\.

#### Evaluation\.

The evaluation benchmarks include MathVista\([Lu et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib18)\), MathVerse\([Zhang et al\., 2025d](https://arxiv.org/html/2609.18057#bib.bib43)\), We\-Math\([Qiao et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib25)\), MMK12\([Meng et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib24)\)and Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2609.18057#bib.bib19)\), LogicVista\([Xiao et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib34)\), SuperClevr\-Counting\([Li et al\., 2023](https://arxiv.org/html/2609.18057#bib.bib10)\), MMMU\-Pro\([Yue et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib38)\)and MathVerse\-V\([Zhang et al\., 2025d](https://arxiv.org/html/2609.18057#bib.bib43)\)\. We also test on ScienceQA\([Lu et al\., 2022](https://arxiv.org/html/2609.18057#bib.bib20)\), HallusionBench\([Guan et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib2)\), ChartQAPro\([Masry et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib22)\), InfographicVQA\([Mathew et al\., 2022](https://arxiv.org/html/2609.18057#bib.bib23)\)and RealWorldQA from lmms\-eval\([Zhang et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib41)\)to evaluate out\-of\-domain performance\. To reduce reliance on LLM\-as\-a\-judge systems, we adopt an exact\-match scoring protocol and report the average accuracy@8 with an inference temperature of 1\.0 following PAPO\([Wang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib32)\)\. More details are provided in Appendix[C\.1](https://arxiv.org/html/2609.18057#A3.SS1)\.

#### Implementation Details\.

Our training framework follows the DAPO\([Yu et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib37)\)recipe with a learning rate of 1e\-6, a rollout batch size of 384, and a maximum response length of 2048\. Models are trained for 15 epochs on Geo3k and 2 epochs on VIRL39K\. We set the rollout group sizen=5n=5for the 3B model, andn=8n=8for the 7B models\. Following VPPO\([Huang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib4)\), we apply a small entropy penalty \(coefficient 0\.06\) to all models during training to ensure training stability and fair comparison\. More details are provided in Appendix[C\.2](https://arxiv.org/html/2609.18057#A3.SS2)\.

ModelMathematical & Geometric ReasoningVision\-Dependent ReasoningAvg\.Geo3KtestMathVerseMathVistaWeMathMMK12LogicVistaCountingMathVerseVVQwen2\.5\-VL\-3B18\.9533\.3346\.4930\.1036\.4729\.4237\.2530\.5532\.82\+ GRPO35\.8842\.9252\.4347\.2540\.5736\.6647\.2539\.2842\.78\+ DAPO41\.6046\.8855\.1953\.0040\.5337\.1150\.7543\.3846\.06\+ PAPOD43\.6850\.4655\.5454\.8740\.9238\.1756\.3147\.5948\.44\+ VPPO43\.1651\.8954\.2858\.0542\.1437\.7855\.3146\.7848\.67\+ PIVOT \(Ours\)43\.0753\.9056\.2560\.0743\.0538\.2065\.6349\.8351\.25Qwen2\.5\-VL\-7B35\.7339\.0263\.5045\.8043\.1142\.8466\.7534\.5846\.42\+ GRPO47\.4047\.8466\.4656\.6145\.5243\.8272\.6943\.5652\.99\+ DAPO51\.7343\.7765\.5649\.7343\.7446\.0980\.5037\.4152\.32\+ PAPOD51\.6648\.7065\.9654\.2744\.7345\.6984\.5643\.1454\.84\+ VPPO52\.0651\.0368\.9455\.7847\.1144\.9780\.1344\.2455\.53\+ PIVOT \(Ours\)53\.1454\.4468\.3860\.8849\.6446\.2882\.5047\.5257\.85Table 1:Performance comparison of PIVOT against various baselines\. Models are trained on the Geometry3K dataset using Qwen2\.5\-VL\-3B and 7B as backbones\.Boldandunderlinedindicate the best and second\-best results\.Table 2:Performance evaluation of PIVOT and baselines trained on the larger\-scale VIRL39K dataset using Qwen2\.5\-VL\-3B as base model\.Boldandunderlinedindicate the best and second\-best results\.

### 5\.2Main Results

#### Multimodal Reasoning Performance\.

Table[1](https://arxiv.org/html/2609.18057#S5.T1)and Table[2](https://arxiv.org/html/2609.18057#S5.T2)present the performance of PIVOT compared to other baselines\. When trained on the Geometry3K dataset, PIVOT consistently achieves the highest average accuracy for both the 3B and 7B models, reaching 51\.25 \(11\.27% relative gains over DAPO\) and 57\.85 \(10\.57% relative gains over DAPO\), respectively\. The performance improvements are observed in both general mathematical and vision\-dependent reasoning tasks\. To validate data scalability, we further extend the training of the 3B model to the larger\-scale VIRL39K dataset\. As shown in Table[2](https://arxiv.org/html/2609.18057#S5.T2), PIVOT maintains its superiority, yielding the highest average score of 55\.68 and outperforming competitive baselines across most evaluated benchmarks\. These results demonstrate the effectiveness of PIVOT in enhancing the multimodal reasoning capabilities of LVLMs\.

#### Training Dynamics\.

Figure[2](https://arxiv.org/html/2609.18057#S5.F2)illustrates the training dynamics of the 7B model on Geo3k dataset\. As shown in Figure[2](https://arxiv.org/html/2609.18057#S5.F2)[2\(a\)](https://arxiv.org/html/2609.18057#S5.F2.sf1), PIVOT demonstrates superior learning efficiency, achieving higher training accuracy rewards compared to other baselines\. Furthermore, the corresponding validation accuracy \(Figure[2](https://arxiv.org/html/2609.18057#S5.F2)[2\(b\)](https://arxiv.org/html/2609.18057#S5.F2.sf2)\) confirms that this efficient optimization directly translates into better generalization results on unseen validation data\. These improvements demonstrate that our proposed method provides a more robust and effective optimization process\.

\(a\)Accuracy Rewards\.\(b\)Validation Accuracy\.
Figure 2:Training dynamics of Qwen2\.5\-VL\-7B on the Geometry3k dataset: \(a\) training accuracy rewards, and \(b\) validation accuracy on the Geometry3k@test data\([Lu et al\., 2021](https://arxiv.org/html/2609.18057#bib.bib19)\)\.

### 5\.3Quantitative Analysis

#### Ablation Study\.

Table 3:Ablation of Self\-calibrated Experience Replay \(SER\) module and Vision\-guided Advantage Allocation \(VAA\) module\. Their combination yields the best results, confirming the effectiveness of our dual\-level framework\.To isolate the contributions of different modules in our framework, we conduct ablation studies using the Qwen2\.5\-VL\-3B model trained on the Geo3k dataset\. PIVOT consists of two core modules: Self\-calibrated Experience Replay \(SER\) and Vision\-guided Advantage Allocation \(VAA\)\. We ablate each module to verify its influence\. Moreover, we further conduct ablations on the replay and token\-utility designs\. Specifically,Off\-PG\+VAAreuses successful experiences via off policy gradient updates, whereasOff\-SFT\+VAAuses them as supervised fine\-tuning targets\. In addition,SER\+VAAlocalremoves the entropy\-gated future visual utility from VAA, isolating the effect of local visual utility alone\. As shown in Table[3](https://arxiv.org/html/2609.18057#S5.T3), both SER\-only and VAA\-only yields noticeable performance gains over the baseline, and their combination yields the best average performance across evaluation benchmarks\.Off\-PG\+VAAandOff\-SFT\+VAAimprove over baseline but lag behind full PIVOT, indicating the advantage of using high\-quality experiences as calibration anchors\. The gap betweenSER\+VAAlocaland full PIVOT further validates the contribution of entropy\-gated future visual utility\. It is worth noting that PIVOT slightly sacrifices the in\-domain Geo3K test set performance relative to several ablations while improving broader generalization\. This arises from SER’s regularization against source\-domain over\-specialization and VAA’s modeling of downstream visual impact beyond local token–image correspondence\. Together, these designs balance exploitation and exploration, enabling broader generalization\.

In addition, to isolate the specific benefits of our self\-generated visually grounded anchors, we also conducted experiments evaluating both CalibRL and CalibRL\+VAA\. As shown in Table[3](https://arxiv.org/html/2609.18057#S5.T3), PIVOT consistently outperforms CalibRL and the CalibRL\+VAA variant on average and across almost all benchmarks\. This superiority can be attributed to two main factors: \(1\) PIVOT’s self\-generated anchors, which are inherently compatible with the evolving policy to ensure better\-calibrated confidence comparisons, and \(2\) the proposed entropy and visual\-dependency filtering mechanisms\. We also ablate different visual intervention strategies; details are shown in Appendix[D\.1](https://arxiv.org/html/2609.18057#A4.SS1)\.

#### Out\-of\-Domain Generalization\.

To ensure that our model is not simply over\-optimized for math and geometry benchmarks, we evaluate its Out\-of\-Domain generalization capabilities on ScienceQA, HallusionBench, ChartQAPro, InfographicVQA, and RealWorldQA\. As shown in Table[4](https://arxiv.org/html/2609.18057#S5.T4), our method consistently achieves improvements over baselines across different OOD datasets\. These results suggest that the visual reasoning capabilities acquired through our training paradigm can generalize to broader multimodal tasks and help mitigate multimodal hallucinations\.

Table 4:Out\-of\-Domain generalization results\. Though trained only on the Geometry3K dataset, our method shows strong generalization to out\-of\-domain multimodal tasks\.Table 5:Results of extending PIVOT to the GRPO algorithm\.
#### Sensitivity Analysis

To evaluate the robustness of our method, we conduct sensitivity analyses on two key hyperparameters: the advantage scaling factorβ\\betaand the loss balancing coefficientλexp\\lambda\_\{\\text\{exp\}\}\. As shown in Figure[3](https://arxiv.org/html/2609.18057#S5.F3), both hyperparameters exhibit a similar pattern: moderate values lead to the best performance\. This is consistent with our design intuition\. A smallβ\\betaweakens the effect of visual advantage modulation, whereas a largeβ\\betamay over\-amplify visual signals and destabilize optimization\. Similarly,λexp\\lambda\_\{\\text\{exp\}\}needs to balance the experience calibration loss with the main RL objective: insufficient weighting limits its benefit, while excessive weighting may interfere with on\-policy learning\.

Figure 3:Sensitivity Analysis ofβ\\betaandλexp\\lambda\_\{\\text\{exp\}\}\.
#### Generalize to GRPO

To verify the generalizability of PIVOT, we extended our training recipe to Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib26)\), which serves as another widely adopted baseline in the RLVR paradigm\. Specifically, we seamlessly integrated our self\-calibrated experience replay and vision\-guided advantage modulation into the standard GRPO optimization loop\.

As shown in Table[5](https://arxiv.org/html/2609.18057#S5.T5), incorporating PIVOT consistently improves standard GRPO’s performance across multimodal reasoning tasks\. This consistent improvement indicates that our dual\-level learning framework is highly orthogonal to the choice of the specific RLVR algorithms\. Rather than being confined to a specific algorithm, PIVOT functions as a general framework that effectively enhances the visual grounding capability of LVLMs during reinforcement learning\.

### 5\.4Additional Results

We also provide several additional results to complement the main experiments\. Specifically, Appendix[D\.2](https://arxiv.org/html/2609.18057#A4.SS2)provides the token visual utility visualizations, which demonstrate that PIVOT concentrates the RL optimization signal on visually grounded perception tokens and reasoning pivots that are more relevant to solving the problem\. Appendix[D\.3](https://arxiv.org/html/2609.18057#A4.SS3)presents a layer\-wise attention analysis, which provides complementary evidence that PIVOT encourages the model to attend more to visual tokens when performing multimodal reasoning\. Appendix[D\.4](https://arxiv.org/html/2609.18057#A4.SS4)reports the computational overhead of the proposed framework\. These results offer further insights into the behavior and practical cost of PIVOT\.

## 6Conclusion

In this paper, we proposed PIVOT, a dual\-level learning framework that addresses key optimization bottlenecks in multimodal RLVR by preserving and amplifying visual reasoning signals\. The framework collects and selectively replays high\-quality experiences as optimization anchors through a self\-calibration loss\. Building upon this, PIVOT further allocates fine\-grained visual advantages to tokens based on their local visual support and downstream impact on reasoning\. Extensive experiments demonstrate that PIVOT consistently improves multimodal reasoning capabilities and out\-of\-domain generalization of LVLMs\. Future work could consider extending this dual\-level learning framework to broader multimodal agent tasks\.

## Limitations

Despite the promising performance of PIVOT, several limitations remain to be addressed in future work\. First, the self\-calibrated experience replay module and the counterfactual intervention strategy inevitably introduce additional computational and storage overhead during training\. Although these overheads are acceptable, exploring more resource\-efficient designs and lightweight perturbation strategies remains an important direction for future optimization\. Secondly, due to constrained computational resources, we primarily evaluated our framework on vision\-language models up to the 7B parameter scale\. While PIVOT demonstrates consistent improvements on these models, validating our framework on much larger scales, such as 32B variants, would be beneficial to fully verify its scalability\. Finally, our current selection of high\-quality trajectories relies on empirical proxies, specifically visual dependency and trajectory entropy\. While these metrics are effective in practice, they may not fully capture the semantic correctness of complex multi\-step reasoning\. Future work could integrate structured, rubric\-based evaluation criteria\([Gunjal et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib3)\)to define and filter high\-quality visual reasoning paths more robustly\.

## Ethical Considerations

This work focuses on improving multimodal reasoning capabilities of large vision\-language models\. All experiments are conducted on public datasets, without using private user data, personally identifiable information, or human\-subject data\. Nevertheless, models trained with our method may still produce incorrect or biased outputs, especially under distribution shifts or in high\-stakes real\-world scenarios\. Thus, such models should be carefully validated and used with human oversight before deployment in domains such as medicine, law, or safety\-critical inspection\. AI\-assisted tools were used only for language polishing and coding assistance\.

## Acknowledgments

This work is jointly supported by the National Key R&D Program of China \(Grant No\. 2024YFF0505703\) and Beijing Municipal Natural Science Foundation \(Grant No\. L257009\)\.

## References

- Foster et al\. \(2025\)Thomas Foster, Anya Sims, Johannes Forkel, and Jakob Foerster\. 2025\.[Lilo: Learning to reason at the frontier of learnability](https://doi.org/10.52202/085713-1634)\.In*Advances in Neural Information Processing Systems*, volume 38, Main Conference, pages 48941–48974\. Curran Associates, Inc\.
- Guan et al\. \(2024\)Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou\. 2024\.[Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models](https://doi.org/10.1109/CVPR52733.2024.01363)\.In*2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 14375–14385\.
- Gunjal et al\. \(2026\)Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean M\. Hendryx\. 2026\.[Rubrics as rewards: Reinforcement learning beyond verifiable domains](https://openreview.net/forum?id=c1bTcrDmt4)\.In*The Fourteenth International Conference on Learning Representations*\.
- Huang et al\. \(2026a\)Siyuan Huang, Xiaoye Qu, Yafu Li, Yun Luo, Zefeng He, Daizong Liu, and Yu Cheng\. 2026a\.[Spotlight on token perception for multimodal reinforcement learning](https://openreview.net/forum?id=bRA4lVWJVQ)\.In*The Fourteenth International Conference on Learning Representations*\.
- Huang et al\. \(2026b\)Wenxuan Huang, Bohan Jia, Shaosheng Cao, Zheyu Ye, Fei zhao, Zhe Xu, Yao Hu, and Shaohui Lin\. 2026b\.[Vision\-r1: Incentivizing reasoning capability in multimodal large language models](https://openreview.net/forum?id=UZIjskfbfU)\.In*The Fourteenth International Conference on Learning Representations*\.
- Huang et al\. \(2026c\)Zhuoxu Huang, Mengxi Jia, Hao Sun, Xuelong Li, and Jungong Han\. 2026c\.[Controllable exploration in hybrid\-policy RLVR for multi\-modal reasoning](https://openreview.net/forum?id=5wxyCidRsK)\.In*The Fourteenth International Conference on Learning Representations*\.
- Jiang et al\. \(2025\)Guochao Jiang, Wenfeng Feng, Guofeng Quan, Chuzhan Hao, Yuewei Zhang, Guohua Liu, and Hao Wang\. 2025\.Vcrl: Variance\-based curriculum reinforcement learning for large language models\.*arXiv preprint arXiv:2509\.19803*\.
- Jiao et al\. \(2026\)Zhengbo Jiao, Shaobo Wang, Zifan Zhang, Wei Wang, Bing Zhao, Hu Wei, and Linfeng Zhang\. 2026\.Credit where it is due: Cross\-modality connectivity drives precise reinforcement learning for mllm reasoning\.*arXiv preprint arXiv:2602\.11455*\.
- Li et al\. \(2026a\)Yunheng Li, Hangyi Kuang, Hengrui Zhang, Jiangxia Cao, Zhaojie Liu, Qibin Hou, and Ming\-Ming Cheng\. 2026a\.Rethinking token\-level policy optimization for multimodal chain\-of\-thought\.*arXiv preprint arXiv:2603\.22847*\.
- Li et al\. \(2023\)Zhuowan Li, Xingrui Wang, Elias Stengel\-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille\. 2023\.Super\-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 14963–14973\.
- Li et al\. \(2026b\)Zongxia Li, Wenhao Yu, Chengsong Huang, Zhenwen Liang, Rui Liu, Fuxiao Liu, Jingxi Chen, Dian Yu, Jordan Lee Boyd\-Graber, Haitao Mi, and Dong Yu\. 2026b\.[Vision\-SR1: Self\-rewarding vision\-language model via reasoning decomposition and multi\-reward policy optimization](https://openreview.net/forum?id=C1M4ETatgM)\.In*The Fourteenth International Conference on Learning Representations*\.
- Liang et al\. \(2026\)Jing Liang, Jinyi Liu, Yi Ma, Hongyao Tang, YAN ZHENG, Shuyue Hu, LEI BAI, and Jianye HAO\. 2026\.[Squeeze the soaked sponge: Efficient off\-policy RFT for large language model](https://openreview.net/forum?id=quBjNSJMrC)\.In*The Fourteenth International Conference on Learning Representations*\.
- Lin \(1992\)Long\-Ji Lin\. 1992\.Self\-improving reactive agents based on reinforcement learning, planning and teaching\.*Machine learning*, 8\(3\):293–321\.
- Liu et al\. \(2026\)Shih\-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min\-Hung Chen, Hongxu Yin, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov\. 2026\.[GDPO: Group reward\-decoupled normalization policy optimization for multi\-reward RL optimization](https://openreview.net/forum?id=Y1rHxA58If)\.In*Forty\-third International Conference on Machine Learning*\.
- Liu et al\. \(2025a\)Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh\. 2025a\.[Noisyrollout: Reinforcing visual reasoning with data augmentation](https://openreview.net/forum?id=9zD2i7YRot)\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*\.
- Liu et al\. \(2025b\)Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang\. 2025b\.Visual\-rft: Visual reinforcement fine\-tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 2034–2044\.
- Lu et al\. \(2026\)Jinda Lu, Junkang Wu, Jinghan Li, Kexin Huang, Shuo Yang, Guoyin Wang, Jiancan Wu, Xiang Wang, and Xiangnan He\. 2026\.Bridging perception and reasoning: Token reweighting for rlvr in multimodal llms\.*arXiv preprint arXiv:2603\.25077*\.
- Lu et al\. \(2024\)Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai\-Wei Chang, Michel Galley, and Jianfeng Gao\. 2024\.[Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://openreview.net/forum?id=KUNzEQMWU7)\.In*The Twelfth International Conference on Learning Representations*\.
- Lu et al\. \(2021\)Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song\-Chun Zhu\. 2021\.[Inter\-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning](https://doi.org/10.18653/v1/2021.acl-long.528)\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 6774–6786, Online\. Association for Computational Linguistics\.
- Lu et al\. \(2022\)Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai\-Wei Chang, Song\-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan\. 2022\.[Learn to explain: Multimodal reasoning via thought chains for science question answering](https://openreview.net/forum?id=HjwK-Tc_Bc)\.In*Advances in Neural Information Processing Systems*\.
- Luo et al\. \(2026\)Ruilin Luo, Chufan Shi, Yizhen Zhang, Cheng Yang, Songtao Jiang, Tongkun Guan, Ruizhe Chen, Ruihang Chu, Peng Wang, Mingkun Yang, Lei Wang, Yujiu Yang, Junyang Lin, and Zhibo Yang\. 2026\.[From narrow to panoramic vision: Attention\-guided cold\-start reshapes multimodal reasoning](https://openreview.net/forum?id=4tsfY0lI1w)\.In*The Fourteenth International Conference on Learning Representations*\.
- Masry et al\. \(2025\)Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj, Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmohammadi, Megh Thakkar, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty\. 2025\.[ChartQAPro: A more diverse and challenging benchmark for chart question answering](https://doi.org/10.18653/v1/2025.findings-acl.978)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 19123–19151, Vienna, Austria\. Association for Computational Linguistics\.
- Mathew et al\. \(2022\)Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar\. 2022\.Infographicvqa\.In*2022 IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*, pages 2582–2591\. IEEE\.
- Meng et al\. \(2025\)Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao\. 2025\.[Mm\-eureka: Exploring the frontiers of multimodal reasoning with rule\-based reinforcement learning](https://arxiv.org/abs/2503.07365)\.*Preprint*, arXiv:2503\.07365\.
- Qiao et al\. \(2025\)Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Jiapeng Wang, Zhuoma GongQue, Shanglin Lei, YiFan Zhang, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Xiao Zong, Yida Xu, Peiqing Yang, Zhimin Bao, Muxi Diao, Chen Li, and Honggang Zhang\. 2025\.[We\-math: Does your large multimodal model achieve human\-like mathematical reasoning?](https://doi.org/10.18653/v1/2025.acl-long.983)In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 20023–20070, Vienna, Austria\. Association for Computational Linguistics\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://arxiv.org/abs/2402.03300)\.*Preprint*, arXiv:2402\.03300\.
- Wang et al\. \(2025a\)Chen Wang, Lai Wei, Yanzhi Zhang, Chenyang Shao, Zedong Dan, Weiran Huang, Yuzhi Zhang, and Yue Wang\. 2025a\.Eframe: Deeper reasoning via exploration\-filter\-replay reinforcement learning framework\.*arXiv preprint arXiv:2506\.22200*\.
- Wang et al\. \(2025b\)Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen\. 2025b\.[Vl\-rethinker: Incentivizing self\-reflection of vision\-language models with reinforcement learning](https://doi.org/10.52202/085713-1037)\.In*Advances in Neural Information Processing Systems*, volume 38, Main Conference, pages 30865–30891\. Curran Associates, Inc\.
- Wang et al\. \(2025c\)Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xiong\-Hui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin\. 2025c\.[Beyond the 80/20 rule: High\-entropy minority tokens drive effective reinforcement learning for llm reasoning](https://doi.org/10.52202/085713-3850)\.In*Advances in Neural Information Processing Systems*, volume 38, Main Conference, pages 115452–115486\. Curran Associates, Inc\.
- Wang et al\. \(2024\)Xu Wang, Sen Wang, Xingxing Liang, Dawei Zhao, Jincai Huang, Xin Xu, Bin Dai, and Qiguang Miao\. 2024\.[Deep reinforcement learning: A survey](https://doi.org/10.1109/TNNLS.2022.3207346)\.*IEEE Transactions on Neural Networks and Learning Systems*, 35\(4\):5064–5078\.
- Wang et al\. \(2026a\)Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, and Xiangxiang Chu\. 2026a\.Visually\-guided policy optimization for multimodal reasoning\.*arXiv preprint arXiv:2604\.09349*\.
- Wang et al\. \(2026b\)Zhenhailong Wang, Xuehang Guo, Sofia Stoica, Haiyang Xu, Hongru WANG, Hyeonjeong Ha, Xiusi Chen, Yangyi Chen, Ming Yan, Fei Huang, and Heng Ji\. 2026b\.[Perception\-aware policy optimization for multimodal reasoning](https://openreview.net/forum?id=izbBqTL8vb)\.In*The Fourteenth International Conference on Learning Representations*\.
- Wu et al\. \(2007\)Zhaohua Wu, Norden E\. Huang, Steven R\. Long, and Chung\-Kang Peng\. 2007\.[On the trend, detrending, and variability of nonlinear and nonstationary time series](https://doi.org/10.1073/pnas.0701020104)\.*Proceedings of the National Academy of Sciences*, 104\(38\):14889–14894\.
- Xiao et al\. \(2024\)Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang\. 2024\.Logicvista: Multimodal llm logical reasoning benchmark in visual contexts\.*arXiv preprint arXiv:2407\.04973*\.
- Ye et al\. \(2026\)Zekai Ye, Qiming Li, Xiaocheng Feng, Ruihan Chen, Ziming Li, Haoyu Ren, Kun Chen, Dandan Tu, and Bing Qin\. 2026\.Not all tokens see equally: Perception\-grounded policy optimization for large vision\-language models\.*arXiv preprint arXiv:2604\.01840*\.
- Yu et al\. \(2025a\)En Yu, Kangheng Lin, Liang Zhao, jisheng yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, Xiangyu Zhang, Daxin Jiang, Jingyu Wang, and Wenbing Tao\. 2025a\.[Perception\-r1: Pioneering perception policy with reinforcement learning](https://doi.org/10.52202/085713-3170)\.In*Advances in Neural Information Processing Systems*, volume 38, Main Conference, pages 94827–94853\. Curran Associates, Inc\.
- Yu et al\. \(2025b\)Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, juncai liu, LingJun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, and 17 others\. 2025b\.[Dapo: An open\-source llm reinforcement learning system at scale](https://doi.org/10.52202/085713-3775)\.In*Advances in Neural Information Processing Systems*, volume 38, Main Conference, pages 113222–113244\. Curran Associates, Inc\.
- Yue et al\. \(2025\)Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig\. 2025\.[MMMU\-pro: A more robust multi\-discipline multimodal understanding benchmark](https://doi.org/10.18653/v1/2025.acl-long.736)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 15134–15186, Vienna, Austria\. Association for Computational Linguistics\.
- Zhan et al\. \(2026\)Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F\. Wong, and Yu Cheng\. 2026\.[ExGRPO: Learning to reason from experience](https://openreview.net/forum?id=701tjQXWVk)\.In*The Fourteenth International Conference on Learning Representations*\.
- Zhang et al\. \(2025a\)Hongzhi Zhang, Jia Fu, Jingyuan Zhang, Kai Fu, Qi Wang, Fuzheng Zhang, and Guorui Zhou\. 2025a\.Rlep: Reinforcement learning with experience replay for llm reasoning\.*arXiv preprint arXiv:2507\.07451*\.
- Zhang et al\. \(2025b\)Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu\. 2025b\.[LMMs\-eval: Reality check on the evaluation of large multimodal models](https://doi.org/10.18653/v1/2025.findings-naacl.51)\.In*Findings of the Association for Computational Linguistics: NAACL 2025*, pages 881–916, Albuquerque, New Mexico\. Association for Computational Linguistics\.
- Zhang et al\. \(2025c\)Kaiyan Zhang, Yuxin Zuo, Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, Yu Fu, Xingtai Lv, Yuchen Zhang, Sihang Zeng, Shang Qu, Haozhan Li, Shijie Wang, Yuru Wang, Xinwei Long, and 20 others\. 2025c\.[A survey of reinforcement learning for large reasoning models](https://arxiv.org/abs/2509.08827)\.*Preprint*, arXiv:2509\.08827\.
- Zhang et al\. \(2025d\)Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai\-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li\. 2025d\.Mathverse: Does your multi\-modal llm truly see the diagrams in visual math problems?In*Computer Vision – ECCV 2024*, pages 169–186, Cham\. Springer Nature Switzerland\.

## Appendix AJustification of the Counterfactual Visual Support Score

In Section[4\.2](https://arxiv.org/html/2609.18057#S4.SS2), we define the direct counterfactual support score as a probability ratio rather than a standard log\-likelihood difference\. Here, we provide the theoretical justification for this design choice\.

For simplicity of notation at generation steptt, letpt​\(y\)=πθ​\(yt=y∣q,I,𝐲<t\)p\_\{t\}\(y\)=\\pi\_\{\\theta\}\(y\_\{t\}=y\\mid q,I,\\mathbf\{y\}\_\{<t\}\)denote the token distribution conditioned on the original image, andqt​\(y\)=πθ​\(yt=y∣q,I~,𝐲<t\)q\_\{t\}\(y\)=\\pi\_\{\\theta\}\(y\_\{t\}=y\\mid q,\\widetilde\{I\},\\mathbf\{y\}\_\{<t\}\)denote the distribution conditioned on the corrupted image\. For a sampled tokenyi,ty\_\{i,t\}, our metric is defined as:

ct=1−qt​\(yt\)pt​\(yt\)\.c\_\{t\}=1\-\\frac\{q\_\{t\}\(y\_\{t\}\)\}\{p\_\{t\}\(y\_\{t\}\)\}\.\(17\)And this computation is detached from the gradient graph\. Intuitively, this quantity measures whether the sampled token receives more probability support from the original\-image condition than from the corrupted\-image condition\. Ifpt​\(yt\)\>qt​\(yt\)p\_\{t\}\(y\_\{t\}\)\>q\_\{t\}\(y\_\{t\}\), thenct\>0c\_\{t\}\>0, meaning that the token is more likely to be generated when the correct visual evidence is available\. Such a token can be regarded as visually supported\. Ifpt​\(yt\)≈qt​\(yt\)p\_\{t\}\(y\_\{t\}\)\\approx q\_\{t\}\(y\_\{t\}\), thenct≈0c\_\{t\}\\approx 0, indicating that the token can be similarly explained with or without the original image, and is therefore more likely to correspond to language priors, formatting tokens, or generic connectors\. Ifpt​\(yt\)<qt​\(yt\)p\_\{t\}\(y\_\{t\}\)<q\_\{t\}\(y\_\{t\}\), thenct<0c\_\{t\}<0, suggesting that the token is even more favored under the corrupted\-image condition, these tokens may correspond to visual hallucinations or some completely unrelated tokens and should be suppressed\.

The key advantage of this form is that it estimates a signed distributional difference between the original\-image and corrupted\-image policies\. Specifically, for any vocabulary tokenaa, we have

𝔼y∼pt\[ct\(y\)𝟏\{y=a\}\]\\displaystyle\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[c\_\{t\}\(y\)\\mathbf\{1\}\\\{y=a\\\}\\right\]=pt​\(a\)​\(1−qt​\(a\)pt​\(a\)\)\\displaystyle=p\_\{t\}\(a\)\\left\(1\-\\frac\{q\_\{t\}\(a\)\}\{p\_\{t\}\(a\)\}\\right\)\(18\)=pt​\(a\)−qt​\(a\)\.\\displaystyle=p\_\{t\}\(a\)\-q\_\{t\}\(a\)\.Thus, under sampling from the original\-image policy, the residual provides an unbiased estimate of the signed probability\-mass differencept−qtp\_\{t\}\-q\_\{t\}\.

This property also explains why the score is preferable to directly using a log\-likelihood gap such as

log⁡pt​\(yt\)−log⁡qt​\(yt\)\.\\log p\_\{t\}\(y\_\{t\}\)\-\\log q\_\{t\}\(y\_\{t\}\)\.\(19\)Although the log\-likelihood gap is intuitive, its expectation underptp\_\{t\}is always non\-negative:

𝔼y∼pt\[logpt​\(y\)qt​\(y\)\]=DKL\(pt∥qt\)≥0\.\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[\\log\\frac\{p\_\{t\}\(y\)\}\{q\_\{t\}\(y\)\}\\right\]=D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|q\_\{t\}\)\\geq 0\.\(20\)Therefore, if it is directly added to the token\-level advantage, it tends to introduce a positive bonus on average, which may encourage longer responses or over\-reward tokens from prefixes with large distributional divergence\. By contrast, the proposed residual satisfies

𝔼y∼pt​\[1−qt​\(y\)pt​\(y\)\]\\displaystyle\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[1\-\\frac\{q\_\{t\}\(y\)\}\{p\_\{t\}\(y\)\}\\right\]=∑ypt​\(y\)−∑yqt​\(y\)\\displaystyle=\\sum\_\{y\}p\_\{t\}\(y\)\-\\sum\_\{y\}q\_\{t\}\(y\)\(21\)=0\.\\displaystyle=0\.Hence,ctc\_\{t\}is a zero\-mean signed residual under the original\-image distribution\. It redistributes optimization strength from tokens more favored by the corrupted\-image or language\-prior condition to tokens more favored by the original\-image condition, rather than injecting an additional uniformly positive reward\.

When used as an additive token\-level advantage correction,

A~i,t=Ai\+η​rc,t,\\widetilde\{A\}\_\{i,t\}=A\_\{i\}\+\\eta r\_\{c,t\},\(22\)the score induces a policy\-gradient correction that pushes the model toward the excess probability mass of the original\-image distribution\. To see this, consider the expected correction term for the logit of a vocabulary tokenaa\. Since

𝔼y∼pt​\[ct​\(y\)\]=0,\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\[c\_\{t\}\(y\)\]=0,\(23\)we have

𝔼y∼pt​\[ct​\(y\)​∇zt​\(a\)​log⁡pt​\(y\)\]\\displaystyle\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[c\_\{t\}\(y\)\\nabla\_\{z\_\{t\}\(a\)\}\\log p\_\{t\}\(y\)\\right\]\(24\)=𝔼y∼pt\[ct\(y\)\(𝟏\{y=a\}−pt\(a\)\)\]\\displaystyle=\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[c\_\{t\}\(y\)\\big\(\\mathbf\{1\}\\\{y=a\\\}\-p\_\{t\}\(a\)\\big\)\\right\]=𝔼y∼pt\[ct\(y\)𝟏\{y=a\}\]−pt\(a\)𝔼y∼pt\[ct\(y\)\]\\displaystyle=\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\\left\[c\_\{t\}\(y\)\\mathbf\{1\}\\\{y=a\\\}\\right\]\-p\_\{t\}\(a\)\\mathbb\{E\}\_\{y\\sim p\_\{t\}\}\[c\_\{t\}\(y\)\]=pt​\(a\)−qt​\(a\)\.\\displaystyle=p\_\{t\}\(a\)\-q\_\{t\}\(a\)\.Therefore, this correction increases the logits of tokens whose probability mass is higher under the original\-image condition and decreases those whose probability mass is higher under the corrupted\-image condition\. This givesci,tc\_\{i,t\}a clear optimization meaning: it encourages the model to reinforce the probability mass specifically contributed by visual evidence, while suppressing probability mass supported by language priors or visual hallucinations\.

## Appendix BDetails of the Detrending Operator

During preliminary experiments, we find that the future\-aware visual score may introduce a mild length\-related bias\. Although the future score is computed within a finite window, tokens near different relative positions can still have systematically different future\-support statistics due to sequence boundaries, response length variation, and heterogeneous reasoning structures\. In some cases, this bias encourages overly long responses\. To mitigate this issue, we apply a lightweight detrending step to the entropy\-gated future visual score\. Letai,t=ui,t​Fi,ta\_\{i,t\}=u\_\{i,t\}F\_\{i,t\}denote the raw entropy\-gated future visual score for tokenttin responseii\. LetEiE\_\{i\}be the set of valid tokens in responseiiandni=\|Ei\|n\_\{i\}=\|E\_\{i\}\|\. For each tokent∈Eit\\in E\_\{i\}, we define its relative position as

ri,t=rankEi⁡\(t\)max⁡\(ni−1,1\),r\_\{i,t\}=\\frac\{\\operatorname\{rank\}\_\{E\_\{i\}\}\(t\)\}\{\\max\(n\_\{i\}\-1,1\)\},\(25\)whererankEi⁡\(t\)∈\{0,…,ni−1\}\\operatorname\{rank\}\_\{E\_\{i\}\}\(t\)\\in\\\{0,\\ldots,n\_\{i\}\-1\\\}is the rank of tokenttamong valid tokens\. Thus,ri,t∈\[0,1\]r\_\{i,t\}\\in\[0,1\]normalizes token position within each response\.

For each response, we fit a response\-specific linear trend between the raw future score and the relative position:

\(αi,βi\)=arg⁡min⁡∑t∈Eiα,β⁡\(ai,t−α−β​ri,t\)2\.\(\\alpha\_\{i\},\\beta\_\{i\}\)=\\arg\\min\_\{\\alpha,\\beta\}\\sum\_\{t\\in E\_\{i\}\}\\left\(a\_\{i,t\}\-\\alpha\-\\beta r\_\{i,t\}\\right\)^\{2\}\.\(26\)The detrending operator is then defined as:

Detrending⁡\(ai,t\)=ai,t−\(αi\+βi​ri,t\)\.\\operatorname\{Detrending\}\(a\_\{i,t\}\)=a\_\{i,t\}\-\(\\alpha\_\{i\}\+\\beta\_\{i\}r\_\{i,t\}\)\.\(27\)
This operation can be viewed as response\-wise position detrending\. It removes the component of the future\-aware score that can be explained by relative token position, and preserves the token\-specific deviation beyond this trend\. Therefore, the retained signal does not simply reward a token for appearing in a favorable position of the response, but reflects whether the token has stronger future visual support than expected from its location\.

## Appendix CExperimental Settings

### C\.1Data and Evaluation

#### Training Datasets\.

To examine data scalability, we conduct reinforcement learning on two training sets with different scales and data coverage\.

- •Geometry3K\([Lu et al\., 2021](https://arxiv.org/html/2609.18057#bib.bib19)\): a geometry problem\-solving dataset containing 3,002 multiple\-choice geometry problems\. The dataset requires models to jointly parse geometric diagrams, understand problem statements, and perform symbolic geometric reasoning\. We follow the standard split, using the 2,101 training problems for training and the 601 test problems for validation\.
- •ViRL39K\([Wang et al\., 2025b](https://arxiv.org/html/2609.18057#bib.bib28)\): a larger\-scale vision\-language reinforcement learning dataset consisting of 38,870 verifiable question\-answering instances\. ViRL39K is constructed from newly collected problems and existing multimodal reasoning datasets through cleaning, reformatting, rephrasing, and rule\-based verification\. Compared with Geometry3K, it covers a broader range of visual reasoning scenarios, including grade\-school problems, STEM and social topics, charts, diagrams, tables, documents, and spatial reasoning\.

#### Evaluation Benchmarks\.

We conduct evaluation on diverse benchmarks to evaluate multimodal reasoning performance, details are as follows:

- •MathVista\([Lu et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib18)\): a visual mathematical reasoning benchmark with 6,141 examples collected from 28 existing multimodal datasets and three newly created datasets\. It evaluates mathematical reasoning in diverse visual contexts and requires fine\-grained visual understanding as well as compositional reasoning\.
- •MathVerse\([Zhang et al\., 2025d](https://arxiv.org/html/2609.18057#bib.bib43)\): it contains 2,612 visual math problems, each reformulated into multiple modality variants with different amounts of textual and visual information\. This design allows the benchmark to diagnose whether a model truly uses diagrams rather than relying only on textual cues\.
- •We\-Math\([Qiao et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib25)\): it consists of 6\.5K visual math problems, spanning 67 hierarchical knowledge concepts and five levels of knowledge granularity\. It is designed to evaluate not only final\-answer accuracy but also the underlying knowledge acquisition and generalization behavior of multimodal models\.
- •MMK12\([Meng et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib24)\): it is a K12 multimodal reasoning dataset that covers mathematics, physics, chemistry, and biology\. Its evaluation set contains 2,000 multiple\-choice questions, with 500 questions for each subject, and is designed to assess multidisciplinary visual reasoning under human\-verified solutions\.
- •LogicVista\([Xiao et al\., 2024](https://arxiv.org/html/2609.18057#bib.bib34)\): it is a visual logical reasoning benchmark containing 448 human\-annotated multiple\-choice questions\.
- •SuperCLEVR\-Counting\([Li et al\., 2023](https://arxiv.org/html/2609.18057#bib.bib10)\): it is derived from Super\-CLEVR, a controllable visual reasoning benchmark designed to diagnose domain robustness under factors such as visual complexity, question redundancy and concept compositionality\. We use its counting subset to evaluate fine\-grained object perception and numerical reasoning\.
- •MMMU\-Pro\([Yue et al\., 2025](https://arxiv.org/html/2609.18057#bib.bib38)\): it is a more robust version of MMMU that filters out questions answerable by text\-only models, augments answer choices from four to ten, and introduces a vision\-only setting where questions are embedded into screenshots or photos\.
- •MathVerse\-V\([Zhang et al\., 2025d](https://arxiv.org/html/2609.18057#bib.bib43)\): it is a vision\-centric subset of MathVerse, where solving the problem requires substantial information from the visual input\.

### C\.2Implementation Details\.

We trained Qwen2\.5\-VL\-3B and Qwen2\.5\-VL\-7B models on the Geometry3K and VIRL39K datasets\. All experiments were conducted utilizing PyTorch 2\.6\.0 and CUDA 12\.4\. Following GDPO\([Liu et al\., 2026](https://arxiv.org/html/2609.18057#bib.bib14)\), we add a small group decoupled format advantage with an advantage weight of 0\.1 to regularize the format of model responses\. To perform intervention on images, we perform random patch masking \(patch size 14, probability 0\.6\) on the input images following PAPO\([Wang et al\., 2026b](https://arxiv.org/html/2609.18057#bib.bib32)\)\. For future visual score computation, we use a discounted future window sizeW=32W=32, a discount factorγ=0\.8\\gamma=0\.8, and a future coefficientλ=0\.5\\lambda=0\.5\. For training stability, the self\-calibrated experience replay module is activated once the task solved ratio exceeds 0\.45 or after a maximum number of warmup steps\. Upon activation, 50% of each training batch is sampled from the experience buffer\. We retain the lowest\-entropy half of the candidates per prompt \(retaining at least one\), and then select the final experience that maximizes the trajectory dependency\. For all training and evaluation experiments, we used the single, standardized prompt template shown below\.

Reasoning TemplateSYSTEM: You are a helpful assistant\.USER: \{question\}You first think through the reasoning process as an internal monologue, enclosed within <think\> </think\> tags\. Then, provide your final answer enclosed within \\boxed\{ \}\.

## Appendix DAdditional Results

### D\.1Ablation on Intervention Strategy

Table 6:Comparison of three visual intervention strategies\.†Our final choice\.We compare three visual intervention strategies for computing visual support scores, including complete grey masking, additive Gaussian noise, and random patch masking following VPPO\([Huang et al\., 2026a](https://arxiv.org/html/2609.18057#bib.bib4)\)\.

- •Complete masking replaces the entire image with a neutral grey canvas with RGB value\(128,128,128\)\(128,128,128\), removing all visual information\.
- •Gaussian noise adds pixel\-wise noise with a standard deviation of189189, which is calibrated such that each pixel has approximately a50%50\\%probability of being saturated to its maximum or minimum value\.
- •Random patch masking follows the ViT\-style patch structure of the base LVLM: the image is divided into14×1414\\times 14patches, and each patch is independently blackened with probability0\.60\.6\.

Table[6](https://arxiv.org/html/2609.18057#A4.T6)shows that all intervention strategies outperform the DAPO baseline, confirming the usefulness of counterfactual visual perturbation for improving visual reasoning\. Among them, random patch masking achieves the best average performance\. Compared with complete masking, it avoids overly coarse removal of the whole image; compared with Gaussian noise, it introduces more severe visual corruption without severe pixel\-level artifacts\. Therefore, we adopt random patch masking as the default intervention strategy in all main experiments\.

### D\.2Visualization of Token Utilities

To better understand how VAA reallocates token\-level optimization signals, we visualize the estimated token visual utilities in Figure[4](https://arxiv.org/html/2609.18057#A4.F4)\. The left side shows the input image and question, while the right side displays the generated response with each token colored according to its visual utility\. Warmer colors indicate higher utility values and thus stronger contribution to visually grounded reasoning\. The visualization shows that VAA assigns higher utilities to tokens that are closely associated with visual perception and key reasoning operations\. For example, tokens such as “interior”, “quadrilateral”, “angles”, “70”, “56”, and “marks” correspond to information that must be extracted from the image\. Meanwhile, reasoning tokens such as “Given”, “Since”, and “divide” also receive high utility, since they connect the perceived visual facts to the final geometric derivation\. In contrast, routine function words and formatting tokens generally receive lower utility\. This suggests that VAA does not uniformly amplify all response tokens, but instead concentrates the RL optimization signal on visually grounded perception tokens and reasoning pivots that are more relevant to solving the problem\.

![Refer to caption](https://arxiv.org/html/2609.18057v1/token_utility_visualization.png)Figure 4:Visualization of token visual utilities\. Warmer colors indicate higher utility scores\.Table 7:Computational overhead comparison\.
### D\.3Layer\-wise Attention Analysis

To further examine whether PIVOT improves the model’s reliance on visual information, we conduct a layer\-wise attention analysis in Figure[5](https://arxiv.org/html/2609.18057#A4.F5)\. Specifically, we measure the attention mass assigned by response tokens to image tokens at each transformer layer, and compare PIVOT with the DAPO baseline and the original Qwen2\.5\-VL\-7B\-Instruct model\. The results show that PIVOT consistently assigns higher attention mass to image tokens across most layers, especially in the middle and later layers where multimodal reasoning is more actively integrated\. Compared with the original base model, DAPO already increases visual attention to some extent, suggesting that RLVR encourages the model to exploit visual evidence for solving reasoning tasks\. PIVOT further strengthens this trend, indicating that trajectory\-level anchoring and token\-level visual advantage allocation promote stronger visual grounding during response generation\. Although attention alone cannot fully characterize the reasoning process, this analysis provides complementary evidence that PIVOT encourages the model to attend more to visual tokens when performing multimodal reasoning\.

Figure 5:Layer\-wise attention analysis\. We report the average attention mass assigned by response tokens to image tokens across transformer layers\.
### D\.4Computational Overhead of PIVOT

We further analyze the computational overhead introduced by PIVOT in Table[7](https://arxiv.org/html/2609.18057#A4.T7)\. All measurements are conducted with Qwen2\.5\-VL\-7B on 4 NVIDIA A100 80GB GPUs\. Compared with DAPO, VPPO increases the average training time per step from698\.8698\.8s to830\.4830\.4s, mainly due to the additional computation required for visual perturbation\-based signal estimation\. PIVOT further increases the average time per step to860\.8860\.8s, corresponding to a23\.2%23\.2\\%overhead over DAPO and only a modest additional overhead over VPPO\. The extra cost of PIVOT mainly comes from two sources: maintaining the buffer for experience replay and estimating token visual utilities for advantage allocation\. The memory footprint of the experience buffer is lightweight, requiring only21\.121\.1MB of additional RAM in our setting \(training on Geometry3K dataset\)\. Therefore, although PIVOT introduces extra computation during training, its memory overhead remains negligible, and the overall training cost is still practical for LVLM reinforcement learning\.