Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

# Aligning Large Vision–Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach
Source: [https://arxiv.org/html/2608.03204](https://arxiv.org/html/2608.03204)
\(2026\)

###### Abstract\.

Post\-training reinforcement learning \(RL\) algorithms are commonly used to align large vision\-language models \(LVLMs\) with human intent and the requirements of visual reasoning tasks\. However, existing RL\-based alignment methods are often resource\-intensive and encounter mismatches between training objectives and inference\-time distributions\. To bridge this gap, we propose a novel test\-time alignment approach that leverages trajectory\-guided structured sampling for dynamic inference\-time refinement, achieving better alignment with visual grounding and ensuring logical consistency\. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns\. It subsequently accomplishes inference\-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo \(MCMC\) algorithm for localized multi\-objective refinement of the reasoning trace\. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead\. These results establish trajectory\-guided test\-time sampling as a scalable and effective alternative to traditional post\-training alignment, particularly for complex visual reasoning tasks\.

Test\-Time Alignment, Large Vision\-Language Model

††copyright:cc††journalyear:2026††doi:10\.1145/3767308\.3835984††conference:34th ACM International Conference on Multimedia; November 10–14, 2026; Rio de Janeiro, Brazil††isbn:979\-8\-4007\-2213\-4/2026/11††ccs:Computing methodologies Knowledge representation and reasoning††ccs:Computing methodologies Natural language generation## 1\.Introduction

Despite their impressive performance across many tasks\(Baiet al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib1); Zhuet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib30)\), large vision–language models \(LVLMs\) remain unreliable on complex visual reasoning tasks that require both strong visual grounding and multi\-step inference\(Zhanget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib46); Yaoet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib32)\)\. In such settings, even after post\-training, they often fail to sustain visually grounded and logically coherent intermediate reasoning at inference time, drifting from visual evidence and producing unstable reasoning trajectories\. Existing alignment approaches, such as reinforcement learning from human feedback \(RLHF\)\(Yuet al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib3)\), direct preference optimization \(DPO\)\(Liuet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib4)\), and reinforcement learning with verifiable rewards \(RLVR\), primarily address this problem through offline parameter updates\. However, such training\-based methods are computationally expensive and offer limited control over inference\-time reasoning, especially under new or unforeseen input distributions\.

![Refer to caption](https://arxiv.org/html/2608.03204v1/x1.png)Figure 1\.Standard autoregressive decoding makes locally optimal choices that may lead to errors\. In contrast, power\-scaled sampling reweights full sequences via a power\-transformed probabilitypαp^\{\\alpha\}\(α\>1\\alpha\>1\)\.Aligning LVLMs at inference time without modifying their underlying weights emerges as a promising direction for addressing the limitations of training\-based alignment methods\(Linet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib9); Xuet al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib38)\), such as ensemble\-based hypothesis reweighting\(Leeet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib37)\), and predictive planning\-based alignment\(Wanget al\.,[2025c](https://arxiv.org/html/2608.03204#bib.bib39)\)\. Within this paradigm, sampling\-based test\-time alignment offers an attractive trade\-off by using additional decoding\-time compute to search for outputs that better satisfy task objectives under distribution shift\. In practice, this is often achieved by reshaping the model’s output distribution, for example through reward model guided reweighting\(Linet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib9)\), importance\-weighted perturbations\(Kanaiet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib10)\), or iterative refinement\. More recently,Karan and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)demonstrate that sampling from the power\-scaled distributionpαp^\{\\alpha\}via Metropolis–Hastings \(MH\) yields training\-free reasoning improvements that are comparable to those obtained with RLVR\. As shown in Figure[1](https://arxiv.org/html/2608.03204#S1.F1), sequence\-level reweighting in power\-scaled sampling avoids misleading local token choices and better preserves visual evidence than autoregressive decoding\.

Despite its promise, applying MH\-based power sampling to LVLMs for visual reasoning remains challenging\. First, power scaling may amplify language priors\(Wanget al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib44)\)and visually unsupported details when visual evidence is weak, exacerbating hallucinations\. Second, complex visual reasoning requires long\-horizon, multi\-step deduction with multiple plausible solution paths\. For example, solving the question in Figure[1](https://arxiv.org/html/2608.03204#S1.F1)requires the model to identify the derivatives at bothx=2x=2andx=5x=5before comparing them\. Sampling can easily drift into high\-entropy, unstable reasoning branches\. Finally, for long\-horizon generation, the induced Markov chain often mixes slowly in a high\-dimensional token space\(Brown and Rosenthal,[2025](https://arxiv.org/html/2608.03204#bib.bib11)\), resulting in low acceptance and many expensive proposal rollouts, which substantially increases inference latency\.

To address these limitations, we propose a training\-free test\-time alignment framework that allocates computation to targeted, structured refinement during decoding\. We first build a reasoning memory bank through our trajectory learning algorithm, where each trajectory encodes the solution process for a complex question as an ordered sequence of high\-level reasoning patterns\. Given a query, we retrieve the top\-kkmost similar examples from the bank and aggregate their trajectories via majority voting to obtain a single guidance trajectory\. This trajectory provides a global prior over the reasoning structure and step order, while the LVLM is responsible for instantiating each pattern into task\-specific reasoning content\. Based on this decomposition, our structured sampling algorithm initializes one reasoning pattern per iteration and performs MCMC refinement within a sliding window of neighboring reasoning patterns\. This localized refinement yields low\-variance proposals and mitigates slow mixing, making MCMC practical for long\-form LVLM generation\. On top of this sampler, we define three complementary objectives to shape the power\-scaled target distribution: vision\-aware distribution sharpening to facilitate visual grounding, an entropy regularizer to favor reliable trajectories, and a linguistic control term to suppress degenerative generation\. In summary, the integration of trajectory\-guided structured proposals, iterative Metropolis–Hastings inference, and our synergistic objectives enables effective test\-time alignment for complex visual reasoning\. Our main contributions are three\-folds:

- •We propose an automatic trajectory learning algorithm rooted in an agentic framework, designed to construct and store candidate trajectories, thereby building a continuous memory of reasoning processes\.
- •We develop a trajectory\-guided structured sampling method that retrieves trajectories from memory as a structural reasoning prior, followed by iterative MCMC trajectory refinement that promotes visual grounding and stable generation through vision\-aware distribution sharpening, entropy regularization, and linguistic degeneration control, thereby ensuring alignment during inference\.
- •Extensive experiments across multiple reasoning benchmarks demonstrate consistent gains in both accuracy and sampling efficiency, establishing our training\-free test\-time sampling approach as a highly effective alternative to RLVR\.

## 2\.Preliminaries

Power\-Scaled Sampling\. LetVVandXXdenote the visual and textual inputs, respectively, and let𝐘=⟨y0,…,yT⟩\\mathbf\{Y\}=\\langle y\_\{0\},\\ldots,y\_\{T\}\\ranglebe a finite output token sequence withyt∈𝒱y\_\{t\}\\in\\mathcal\{V\}, where𝒱\\mathcal\{V\}is the model vocabulary\. An LVLM induces a normalized autoregressive distribution over𝒴\\mathcal\{Y\}:

\(1\)P​\(𝐘∣V,X\)=∏t=0TP​\(yt∣V,X,y<t\),P\(\\mathbf\{Y\}\\mid V,X\)=\\prod\_\{t=0\}^\{T\}P\(y\_\{t\}\\mid V,X,y\_\{<t\}\),where𝒴\\mathcal\{Y\}denotes the set of all finite sequences\. Recent work shows that strong reasoning gains can be obtained by sampling from a power\-transformed sequence distribution\(Karan and Du,[2025](https://arxiv.org/html/2608.03204#bib.bib5)\)\. Given an exponentα\>1\\alpha\>1, the corresponding unnormalized target is

\(2\)pα​\(𝐘∣V,X\)∝P​\(𝐘∣V,X\)α,p\_\{\\alpha\}\(\\mathbf\{Y\}\\mid V,X\)\\ \\propto\\ P\(\\mathbf\{Y\}\\mid V,X\)^\{\\alpha\},which amplifies differences in sequence likelihood and concentrates probability mass on high\-likelihood trajectories under the base model\. Power sampling operates at the sequence level by reweighting complete continuations𝐘\\mathbf\{Y\}, rather than making greedy or temperature\-scaled decisions at each step, as illustrated in Figure[1](https://arxiv.org/html/2608.03204#S1.F1)\. Sampling from the power distribution implicitly favors longer\-horizon planning by prioritizing globally coherent, high\-likelihood trajectories and mitigating failures induced bypivotal tokens\(Abdinet al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib45)\)that steer generation toward low\-likelihood continuations\.

![Refer to caption](https://arxiv.org/html/2608.03204v1/x2.png)Figure 2\.Overview of our approach\.We begin by constructing a reasoning memory bank based on a trajectory learning algorithm\. At inference time, retrieved trajectories provide a global structural reasoning prior, while MCMC performs localized updates within selected reasoning segments to iteratively steer generation toward the alignment target\.
## 3\.Methodology

In this section, we present our test\-time alignment approach for visual reasoning with LVLMs \(Figure[2](https://arxiv.org/html/2608.03204#S2.F2)\)\. We first introduce the automatic trajectory memory curation pipeline \(Section[3\.1](https://arxiv.org/html/2608.03204#S3.SS1)\), which uses an agentic framework to search for effective reasoning paths under predefined reasoning patterns\. We then describe the structured sampling algorithm \(Section[3\.2](https://arxiv.org/html/2608.03204#S3.SS2)\), where trajectory guidance provides a structural prior for LVLM reasoning and Metropolis–Hastings is used to perform iterative refinement within a local window of adjacent reasoning patterns\. Finally, we introduce our refinement target \(Section[3\.3](https://arxiv.org/html/2608.03204#S3.SS3)\), which combines vision\-aware distribution sharpening, entropy regularization, and linguistic degeneration control\.

![Refer to caption](https://arxiv.org/html/2608.03204v1/x3.png)Figure 3\.Automatic trajectory memory curation\.After defining reasoning patterns and preparing the seed dataset, the reasoning memory bank is built through a two\-stage reasoning memory curation process\.### 3\.1\.Automatic Trajectory Memory Curation

Complex multimodal questions often require high\-level reasoning behaviors, such as decomposition\(Liet al\.,[2025c](https://arxiv.org/html/2608.03204#bib.bib12)\), knowledge grounding, and self\-verification\(Wanget al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib13)\)\. However, standard LVLM decoding may fail to activate these behaviors reliably, leading to brittle trajectories on challenging instances\. We therefore introduce an automated trajectory discovery pipeline that decomposes the reasoning process into discrete patterns, curating a reusable memory bank of structured reasoning traces to facilitate test\-time guidance \(Figure[3](https://arxiv.org/html/2608.03204#S3.F3)\)\.

Based on insights from prior works\(Wuet al\.,[2026](https://arxiv.org/html/2608.03204#bib.bib6); Liet al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib8)\), we define seven reasoning patterns, as shown in Figure[3](https://arxiv.org/html/2608.03204#S3.F3)\. Rather than relying on heuristic prompts or taxonomies of reasoning actions\(Liet al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib14)\), we describe each pattern in a pseudo\-code style , making offline reasoning experience explicit and reusable during inference\. We construct the seed set using 34k de\-duplicated question–image pairs from ThinkLite\-70k\(Wanget al\.,[2025d](https://arxiv.org/html/2608.03204#bib.bib34)\), covering mathematical reasoning, natural image understanding, and chart understanding\.

Trajectory Learning Algorithm\.Motivated byLuet al\.\([2025](https://arxiv.org/html/2608.03204#bib.bib7)\), we develop an agentic framework that automatically learns trajectories through iterative collaboration among a planner, an executor, and a manager, as shown in Figure[3](https://arxiv.org/html/2608.03204#S3.F3)\. A standard chain\-of\-thought \(CoT\) can be represented as𝐘=\{r0,r1,…,rT\}\\mathbf\{Y\}=\\\{r\_\{0\},r\_\{1\},\\dots,r\_\{T\}\\\}, wherertr\_\{t\}denotes thett\-th rationale andrTr\_\{T\}contains the final answer\. To make a reasoning trajectory executable and editable at the step level, we represent each intermediate state using \(i\) a task list𝐓\\mathbf\{T\}, which specifies the remaining tasks, and \(ii\) an information list𝐈\\mathbf\{I\}, which stores the intermediate results and supporting evidence collected so far\. This state representation also provides explicit inputs and outputs for each reasoning pattern\. Accordingly, the CoT is reformulated as:

\(3\)𝐘=\{\(𝐓0,𝐈0\),\(𝐓1,𝐈1\),…,\(𝐓T,𝐈T\)\},\\mathbf\{Y\}=\\\{\(\\mathbf\{T\}\_\{0\},\\mathbf\{I\}\_\{0\}\),\(\\mathbf\{T\}\_\{1\},\\mathbf\{I\}\_\{1\}\),\\dots,\(\\mathbf\{T\}\_\{T\},\\mathbf\{I\}\_\{T\}\)\\\},where𝐈T\\mathbf\{I\}\_\{T\}contains the final answer\. Given an imageVVand a questionXX, the planner LVLMMθM\_\{\\theta\}selects the next reasoning patternata\_\{t\}at steptt\. Its decision is conditioned on the first pending task𝐓t\(0\)\\mathbf\{T\}\_\{t\}^\{\(0\)\}and the reasoning historyHtH\_\{t\}, which records the previously selected patterns and their associated thinking contexts:

\(4\)at=Mθ​\(V,X,𝐓t\(0\),Ht\),t≥0\.a\_\{t\}=M\_\{\\theta\}\(V,X,\\mathbf\{T\}\_\{t\}^\{\(0\)\},H\_\{t\}\),\\quad t\\geq 0\.To facilitate exploration, we rank patterns by their first\-token probabilities and sample from the top candidates\. The executor LVLM then applies the selected pattern to generate an intermediate thinking contextctc\_\{t\}and the corresponding pattern\-specific outputsoto\_\{t\}:

\(5\)ct,ot=Mθ​\(V,X,a≤t,𝐓t,𝐈t\)\.c\_\{t\},o\_\{t\}=M\_\{\\theta\}\(V,X,a\_\{\\leq t\},\\mathbf\{T\}\_\{t\},\\mathbf\{I\}\_\{t\}\)\.Finally, the manager LVLM usesctc\_\{t\}andoto\_\{t\}to transform the current state\(𝐓t,𝐈t\)\(\\mathbf\{T\}\_\{t\},\\mathbf\{I\}\_\{t\}\)into the next state\(𝐓t\+1,𝐈t\+1\)\(\\mathbf\{T\}\_\{t\+1\},\\mathbf\{I\}\_\{t\+1\}\)\. Specifically, it removes completed tasks from𝐓t\\mathbf\{T\}\_\{t\}and updates𝐈t\\mathbf\{I\}\_\{t\}by adding newly obtained information or revising existing entries\. The updated lists then provide a clear starting state for the next reasoning step\.

Reasoning Memory Curation\.We use depth\-first search to collect candidate trajectories\. At each step, we retain the highest\-ranked candidate patterns until their cumulative probability reaches a thresholdτ∈\(0,1\)\\tau\\in\(0,1\), and backtrack when a terminal pattern is reached\. To estimate reliability, we roll out each candidate trajectorykktimes and calculate its answer correctness rate, retaining the most reliable traces\. For downstream retrieval, we index the question embeddings in a vector database and store the best\-performing trajectory for each question as retrievable metadata\.

### 3\.2\.Trajectory\-Guided Structured Sampling

At inference, we retrieve the top\-kkquestions most similar to input\(V,X\)\(V,X\)\. Candidates are ranked by an entropy\-weighted score combining image and text embedding similarities withnn\-gram overlap, where lower\-entropy views receive larger weights\. The associated trajectories form:𝒯​\(V,X\)=\{𝝉\(j\)\}j=1k,\\mathcal\{T\}\(V,X\)=\\\{\\boldsymbol\{\\tau\}^\{\(j\)\}\\\}\_\{j=1\}^\{k\},where each trajectory is an ordered sequence of predefined reasoning patterns\. Given the retrieved set, we construct a guidance trajectory by majority voting at each aligned position:

\(6\)𝝉∗=⟨a1∗,…,aL∗⟩\.\\boldsymbol\{\\tau\}^\{\*\}=\\langle a^\{\*\}\_\{1\},\\ldots,a^\{\*\}\_\{L\}\\rangle\.Starting from all retrieved trajectories, at each steptt, we select the most frequent pattern asat∗a\_\{t\}^\{\*\}and retain only trajectories matching the selected prefix⟨a1∗,…,at∗⟩\\langle a\_\{1\}^\{\*\},\\ldots,a\_\{t\}^\{\*\}\\ranglefor the next step\.

Metropolis–Hastings\(Metropoliset al\.,[1953](https://arxiv.org/html/2608.03204#bib.bib17)\)is a classic Markov Chain Monte Carlo \(MCMC\) algorithm that approximately samples from an unnormalized target distribution via iterative proposals and a stochastic accept–reject rule\. Let𝐘0:T0\\mathbf\{Y\}^\{0\}\_\{0:T\}denote the current sequence being refined, and letppropp\_\{\\text\{prop\}\}be the proposal LVLM\. At iterationii, we sample a boundary indext∼𝒰​\{0,…,T\}t\\sim\\mathcal\{U\}\\\{0,\\dots,T\\\}and propose𝐘′\\mathbf\{Y\}^\{\\prime\}by resampling the suffix𝐘t:Ti\\mathbf\{Y\}^\{i\}\_\{t:T\}withppropp\_\{\\text\{prop\}\}while keeping the prefix𝐘0:t−1i\\mathbf\{Y\}^\{i\}\_\{0:t\-1\}fixed\. The proposal is accepted with probability

\(7\)A​\(𝐘i,𝐘′\)=min⁡\{1,pt​\(𝐘′\)​q​\(𝐘i∣𝐘′\)pt​\(𝐘i\)​q​\(𝐘′∣𝐘i\)\},A\(\\mathbf\{Y\}^\{i\},\\mathbf\{Y\}^\{\\prime\}\)=\\min\\left\\\{1,\\frac\{p\_\{t\}\(\\mathbf\{Y\}^\{\\prime\}\)\\,q\(\\mathbf\{Y\}^\{i\}\\mid\\mathbf\{Y\}^\{\\prime\}\)\}\{p\_\{t\}\(\\mathbf\{Y\}^\{i\}\)\\,q\(\\mathbf\{Y\}^\{\\prime\}\\mid\\mathbf\{Y\}^\{i\}\)\}\\right\\\},whereptp\_\{t\}is the unnormalized target density over sequences andqqis the proposal transition\. Repeating forNNiterations yields a chain\{𝐘0,…,𝐘N\}\\\{\\mathbf\{Y\}^\{0\},\\ldots,\\mathbf\{Y\}^\{N\}\\\}that approximately samples from the target\.

Given the trajectory guidance, we first use a rule\-based estimator that takes the aggregated trajectory as input to route trivial questions to full\-sequence MCMC refinement in order to reduce overthinking and improve efficiency\. For non\-trivial questions, we prompt the LVLM to reason by following the guided trajectory and do iterative sampling as shown in Algorithm[1](https://arxiv.org/html/2608.03204#alg1)\. We highlight three design choices that are central to our approach\.

Algorithm 1Trajectory\-Guided Structured Sampling1:proposal LVLM

ppropp\_\{\\text\{prop\}\}, inputs

\(V,X\)\(V,X\), max length

TT, initial temperature

τ0\\tau\_\{0\}, initial sharpening exponent

α0\\alpha\_\{0\}, decay threshold

κ\\kappa, decay factor

γ\\gamma, MCMC steps

NMCMCN\_\{\\text\{MCMC\}\}
2:

ℬ←\[\]\\mathcal\{B\}\\leftarrow\[\\,\]⊳\\trianglerightInitialize block boundaries;ℬ\\mathcal\{B\}stores start indices of reasoning blocks

3:

τ←τ0\\tau\\leftarrow\\tau\_\{0\},

α←α0\\alpha\\leftarrow\\alpha\_\{0\}
4:while

\|𝐘\|<T\|\\mathbf\{Y\}\|<Tdo

5:Annealed Sampling Schedule:

6:

τ←max\(0\.1,τ⋅γ𝕀​\[\|ℬ\|\>κ\]\\tau\\leftarrow\\text\{max\}\(0\.1,\\,\\tau\\cdot\\gamma^\{\\mathbb\{I\}\[\|\\mathcal\{B\}\|\>\\kappa\]\}\),

α←τ−1\\alpha\\leftarrow\\tau^\{\-1\}
7:Pattern\-wise Initialization:

8:

U←\|ℬ\|\+1U\\leftarrow\|\\mathcal\{B\}\|\+1⊳\\trianglerightthe max numbers of pattern to generate

9:

𝐘←Generation​\(pprop,𝐘,U,τ,α\)\\mathbf\{Y\}\\leftarrow\\textsc\{Generation\}\(p\_\{\\text\{prop\}\},\\,\\mathbf\{Y\},\\,U,\\tau,\\,\\alpha\)
10:Sliding\-Window Sampling:perform MCMC\-based refinement within the newest two reasoning patterns\.

11:

𝐘←MCMC​\(pprop,𝐘,NMCMC,τ​α\)\\mathbf\{Y\}\\leftarrow\\textsc\{MCMC\}\(p\_\{\\text\{prop\}\},\\mathbf\{Y\},N\_\{\\text\{MCMC\}\},\\,\\tau\\,\\alpha\)
12:Update

ℬ\\mathcal\{B\}according to updated sequence

𝐘\\mathbf\{Y\}
13:ifHasTerminal\(

𝐘\\mathbf\{Y\}\)then break

14:endwhile

15:return

𝐘\\mathbf\{Y\}

\(i\) Pattern\-wise Initialization\. Each iteration adds at most one new reasoning pattern, thus limiting proposal drift and improving acceptance in high\-dimensional sequence spaces\(Brown and Rosenthal,[2025](https://arxiv.org/html/2608.03204#bib.bib11)\)\. This pattern\-level granularity also makes the refinement process interpretable\.

\(ii\) Sliding\-Window Sampling\. Full\-sequence resampling is costly for long outputs\. Instead, we update only a local suffix over the most recent two reasoning patterns, reducing inference cost while retaining the benefits of sequence\-level refinement\.

\(iii\) Annealed Sampling Schedule\. Since windowed MH refines recent patterns on top of a mostly fixed prefix, excessive temperature can destabilize the existing context\. We therefore decrease the sampling temperatureτ\\tauand increase the sharpening exponentα\\alphaonly when the number of reasoning patterns\|𝒫\|\|\\mathcal\{P\}\|exceeds a thresholdκ\\kappa, yielding a smooth transition from exploration to exploitation\.

### 3\.3\.Multi\-Objective Trajectory Refinement

Trajectory guidance provides a global prior over desirable reasoning patterns, but local realizations can still drift from visual evidence or fall into decoding pathologies\. In this section, we define our test\-time alignment target that augments power\-based sharpening with step\-level alignment objectives, so that alignment can be improved via MCMC refinement without updating model parameters\.

#### 3\.3\.1\.Vision\-Aware Distribution Sharpening

Naively sharpening the sequence distribution can overemphasize language priors under weak visual evidence, leading to increased hallucinations\. Inspired by contrastive visual decoding\(Lenget al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib18); Yinet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib19)\), we introduce a vision\-aware calibration mechanism that reweights next\-token probabilities against a ungrounded baseline\. Specifically, the ungrounded branch is constructed by masking visual tokens during prefill, while subsequent decoding steps are synchronized with the visual\-grounded branch through teacher\-forcing\. Letyty\_\{t\}be the token chosen by the grounded branch at steptt, we thereby define the alignment ratiortr\_\{t\}as:

\(8\)rt=Pvisual\-grounded​\(yt\)\+δPungrounded​\(yt\)\+δ,r\_\{t\}=\\frac\{P\_\{\\text\{visual\-grounded\}\}\(y\_\{t\}\)\+\\delta\}\{P\_\{\\text\{ungrounded\}\}\(y\_\{t\}\)\+\\delta\},whereδ\\deltais a small smoothing constant for numerical stability\. Intuitively, a largerrtr\_\{t\}suggests that the current token is more visually grounded\. We then convertrtr\_\{t\}into a bounded calibration weights​\(rt\)s\(r\_\{t\}\), following adaptive calibration\(Huoet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib20); Huet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib21)\):

\(9\)s​\(rt\)=\{exp⁡\(−β​\[softplus⁡\(−ln⁡rt−ϵ\)−softplus⁡\(−ϵ\)\]\),rt<1,1,rt≥1,s\(r\_\{t\}\)=\\begin\{cases\}\\exp\\\!\\Big\(\-\\beta\\big\[\\operatorname\{softplus\}\(\-\\ln r\_\{t\}\-\\epsilon\)\-\\operatorname\{softplus\}\(\-\\epsilon\)\\big\]\\Big\),&r\_\{t\}<1,\\\\ 1,&r\_\{t\}\\geq 1,\\end\{cases\}whereβ\\betacontrols suppression strength,ϵ=10−3\\epsilon=10^\{\-3\}is a small margin, andsoftplus⁡\(x\)=ln⁡\(1\+ex\)\\operatorname\{softplus\}\(x\)=\\ln\(1\+e^\{x\}\)\. The subtraction term makes the penalty zero at the boundary, ensuring continuity asrt→1−r\_\{t\}\\to 1^\{\-\}\. Next\-token probabilities are calibrated withs​\(rt\)s\(r\_\{t\}\)as

\(10\)P~​\(yt∣V,X,y<t\)=P​\(yt∣V,X,y<t\)⋅s​\(rt\),\\tilde\{P\}\(y\_\{t\}\\mid V,X,y\_\{<t\}\)\\;=\\;P\(y\_\{t\}\\mid V,X,y\_\{<t\}\)\\cdot s\(r\_\{t\}\),resulting in a vision\-aware sharpened sequence likelihood:

\(11\)P~​\(𝐘∣V,X\)α=\(∏t=0TP~​\(yt∣V,X,y<t\)\)α\.\\tilde\{P\}\(\\mathbf\{Y\}\\mid V,X\)^\{\\alpha\}\\;=\\;\(\\prod\_\{t=0\}^\{T\}\\tilde\{P\}\(y\_\{t\}\\mid V,X,y\_\{<t\}\)\)^\{\\alpha\}\.

#### 3\.3\.2\.Entropy Regularization

Recent evidence suggests that RL\-style alignment is often accompanied by reduced predictive entropy\(Cuiet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib22)\)\. Consistently, our preliminary experiment shows that correct generations tend to exhibit lower predictive entropy under the base model\. We therefore exponentially downweight trajectories according to their cumulative token\-level predictive entropy:

\(12\)RH\(𝐘\)=exp\(−λH∑t=0TH\[P\(⋅∣V,X,y<t\)\]\),R\_\{H\}\(\\mathbf\{Y\}\)=\\exp\\\!\\left\(\-\\lambda\_\{H\}\\sum\_\{t=0\}^\{T\}H\\\!\\left\[P\(\\cdot\\mid V,X,y\_\{<t\}\)\\right\]\\right\),whereλH\\lambda\_\{H\}controls the strength of the entropy penalty\.

#### 3\.3\.3\.Linguistic Degeneration Control

To reduce premature stopping while discouraging repetitive loops, we introduce a linguistic control term that couples effective length with an explicit repetition signal\. Letℓ​\(𝐘\)\\ell\(\\mathbf\{Y\}\)be the sequence length and letρ​\(𝐘\)∈\[0,1\]\\rho\(\\mathbf\{Y\}\)\\in\[0,1\]denote annn\-gram repetition ratio\. We define the effective length score as:

\(13\)SL​\(𝐘\)=ℓ​\(𝐘\)1\+ln⁡\(ℓ​\(𝐘\)\)​ρ​\(𝐘\),S\_\{L\}\(\\mathbf\{Y\}\)\\;=\\;\\frac\{\\ell\(\\mathbf\{Y\}\)\}\{1\+\\ln\(\\ell\(\\mathbf\{Y\}\)\)\\,\\rho\(\\mathbf\{Y\}\)\},which grows with length but is increasingly discounted as repetition increases, especially for long outputs through theln⁡\(ℓ​\(𝐘\)\)\\ln\(\\ell\(\\mathbf\{Y\}\)\)factor\. We incorporate a linguistic degeneration control factor by exponentiating a shaped effective\-length reward:

\(14\)RL​\(𝐘\)=exp⁡\(L⋅\(1−exp⁡\(−a⋅SL​\(𝐘\)b\)\)\),R\_\{L\}\(\\mathbf\{Y\}\)\\;=\\;\\exp\\\!\\left\(L\\cdot\\Big\(1\-\\exp\\big\(\-a\\cdot S\_\{L\}\(\\mathbf\{Y\}\)^\{\\,b\}\\big\)\\Big\)\\right\),whereLLsets the saturation level,aacontrols the sensitivity to changes inSL​\(𝐘\)S\_\{L\}\(\\mathbf\{Y\}\), andbbshapes the degree of diminishing returns\.

#### 3\.3\.4\.Synergistic Joint Alignment Target

By integrating these objectives, we define an unnormalized alignment target over complete sequences, withα\>1\\alpha\>1controls the sharpening strength:

\(15\)pt​\(𝐘∣V,X\)∝P~​\(𝐘∣V,X\)α⋅RH​\(𝐘\)⋅RL​\(𝐘\)\.p\_\{t\}\(\\mathbf\{Y\}\\mid V,X\)\\ \\propto\\ \\tilde\{P\}\(\\mathbf\{Y\}\\mid V,X\)^\{\\alpha\}\\cdot R\_\{H\}\(\\mathbf\{Y\}\)\\cdot R\_\{L\}\(\\mathbf\{Y\}\)\.This synergistic formulation yields a balanced energy landscape\. The vision\-aware term first filters out visually unsupported reasoning branches, after which entropy regularization biases MCMC toward reliable, confident trajectories\. Crucially, while entropy regularization alone may favor confidently wrong outputs, its coupling with visual calibration ensures that only visually consistent paths are sharpened\. Finally, the linguistic control term prevents collapse into repetitive loops or premature termination\. Together, these objectives turn test\-time sampling into a robust optimization process that yields concise and logically coherent reasoning trajectories\.

## 4\.Experiments

Table 1\.Performance comparison on five datasets\.MethodMathVistaMathVisionMathVerseMMMUMMStarAvg\.General Vision\-Language ModelsInternVL2\.5\-8B\(Chenet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib28)\)64\.422\.039\.554\.962\.848\.7Qwen2\.5\-VL\-7B\(Baiet al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib1)\)68\.2\-49\.258\.663\.9\-InternVL3\-8B\(Zhuet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib30)\)71\.629\.346\.362\.768\.255\.6Qwen3\-VL\-8B\(Baiet al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib29)\)77\.2\-62\.169\.670\.9\-RLVR\-Trained Vision\-Language ModelsOpenVLThinker\-7B\(Denget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib31)\)70\.229\.650\.351\.963\.253\.0VL\-Rethinker\-7B\(Wanget al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib13)\)74\.932\.354\.256\.7\-\-Vision\-R1\-7B\(Huanget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib36)\)73\.530\.752\.450\.560\.253\.4MM\-EUREKA\-7B\(Menget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib35)\)73\.031\.950\.352\.364\.154\.3ThinkLite\-VL\-7B\(Wanget al\.,[2025d](https://arxiv.org/html/2608.03204#bib.bib34)\)75\.132\.952\.155\.565\.056\.1Sampling\-based Test Time Alingment \(Qwen2\.5\-VL\-7B\)Qwen2\.5\-VL\-7B \(Reproduced\)70\.027\.347\.154\.061\.352\.0\\rowcolormyblue \+ Multi\-Objective Sampling73\.2↑\\uparrow3\.230\.9↑\\uparrow3\.650\.0↑\\uparrow2\.954\.9↑\\uparrow0\.964\.0↑\\uparrow2\.754\.6↑\\uparrow2\.6\\rowcolormyblue \+ TG Structured Sampling72\.7↑\\uparrow2\.730\.3↑\\uparrow3\.049\.5↑\\uparrow2\.454\.7↑\\uparrow0\.764\.5↑\\uparrow3\.254\.3↑\\uparrow2\.3### 4\.1\.Experiment Settings

#### 4\.1\.1\.Datasets

We evaluate our framework on diverse multimodal benchmarks requiring multi\-hop reasoning, allowing us to assess test\-time alignment under long\-horizon inference\. We cover multimodal math reasoning through MathVista\(Luet al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib23)\), MathVerse\(Zhanget al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib27)\), and MathVision\(Wanget al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib24)\), while targeting college\-level multi\-disciplinary tasks in MMMU\(Yueet al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib25)\)and general LVLM robustness in MMStar\(Chenet al\.,[2024](https://arxiv.org/html/2608.03204#bib.bib26)\)\.

#### 4\.1\.2\.Baselines

We compare against two categories of models:

- •Strong general\-purpose LVLMs\.We include widely used open\-source backbones with strong general multimodal performance across model generations: InternVL2\.5\-8B\(Chenet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib28)\), Qwen2\.5\-VL\-7B\(Baiet al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib1)\), InternVL3\-8B\(Zhuet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib30)\), and Qwen3\-VL\-8B\(Baiet al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib29)\)\.
- •RLVR\-trained LVLMsWe include representative reasoning\-oriented LVLMs that improve multi\-step inference through RL with verifiable rewards \(RLVR\),such as OpenVLThinker\-7B\(Denget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib31)\), VL\-Rethinker\-7B\(Wanget al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib13)\), Vision\-R1\-7B\(Huanget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib36)\), MM\-EUREKA\-7B\(Menget al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib35)\), ThinkLite\-VL\-7B\(Wanget al\.,[2025d](https://arxiv.org/html/2608.03204#bib.bib34)\)\. All the baselines share the same Qwen2\.5\-VL\-7B backbone for fair comparison\.

#### 4\.1\.3\.Implementation details

For trajectory discovery pipeline, we adopt Qwen3\-VL\-8B as the planner, executor, and manager\. After verification, we retain 24k question–trajectory pairs and index the questions in a vector database\. In our main experiments, we report results for two variants: \(i\)Multi\-Objective Samplingwithout trajectory guidance, where we prompt the model to produce intermediate reasoning before answering and perform full\-sequence MCMC refinement, as inKaran and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)\. and \(ii\)Trajectory\-Guided \(TG\) Structured Sampling, which further incorporates trajectory guidance and block\-wise refinement for non\-trivial questions\. We set the sharpening exponentα=4\\alpha=4and the generation temperatureτ=0\.25\\tau=0\.25\. For the alignment objectives, we use visual calibration weightβ=2\.0\\beta=2\.0, entropy weightλH=1\.0\\lambda\_\{H\}=1\.0\. For linguistic control term, we seta=0\.05a=0\.05,b=0\.8b=0\.8and linguistic control saturation levelL=128L=128\. For TG structured sampling, we set the decay thresholdκ=4\\kappa=4and decay factorγ=0\.5\\gamma=0\.5\. We evaluate our methods on Qwen2\.5\-VL\-7B, a widely used LVLM that has shown strong potential in prior RL\-based studies\.

Table 2\.Ablation on trajectory\-guided structured sampling\.Table 3\.Ablation study on alignment target\. We sample directly from Qwen2\.5\-VL\-7B under different MCMC target distributions\.Table 4\.Performance on MMStar and MathVision with InternVL2\.5\-8B and Qwen3\-VL\-8B\.

### 4\.2\.Main Results

Table[1](https://arxiv.org/html/2608.03204#S4.T1)reports results on five challenging multimodal datasets\. Overall, our methods consistently improves the reproduced Qwen2\.5\-VL\-7B baseline across all datasets\. Multi\-Objective Sampling yields the best average accuracy \(54\.654\.6\), while TG Structured Sampling remains highly competitive \(54\.354\.3\)\. Three takeaways stand out\.

Training\-free test\-time alignment is competitive with RLVR\-trained reasoning models\. Without any post\-training, our sampling\-based refinement closes much of the gap to RLVR\-aligned LVLMs and outperforms several RLVR\-trained baselines on the reported suite, suggesting that reallocating compute to inference\-time refinement is an effective way to improve alignment\.

Trajectory\-guided structured sampling preserves accuracy and improves efficiency\. Compared to full\-sequence refinement, TG Structured Sampling achieves similar accuracy while using step\-aligned, block\-wise proposals guided by a voted trajectory prior, reducing unnecessary resampling and making long\-horizon refinement more practical \(efficiency results in Section[4\.7](https://arxiv.org/html/2608.03204#S4.SS7)\)\.

General\-purpose trajectory priors help most on general dataset\. Since the trajectory memory is built from a general\-purpose seed set \(ThinkLite\-70k\), trajectory guidance is particularly effective on the general dataset MMStar\. On more reasoning\-intensive math datasets, gains remain consistent but can be slightly smaller, suggesting room for more domain\-specialized trajectory libraries\.

### 4\.3\.Ablation Study on Structured Sampling

Table[2](https://arxiv.org/html/2608.03204#S4.T2)ablates two key components of trajectory\-guided structured sampling\. Removing annealed sampling schedule consistently degrades performance across all evaluated benchmarks, indicating that a gradual shift from exploration to exploitation is important for stable long\-horizon refinement\. Disabling trajectory guidance leads to a larger drop, especially on MathVision, suggesting that the retrieved\-and\-voted trajectory prior provides useful global structure that improves the quality of block\-level proposals and subsequent MCMC refinement\. Notably, this sensitivity to temperature is amplified by our relatively strong alignment target: overly aggressive exploration can cause block\-wise updates to overfit local edits that score well under the target while drifting from a globally coherent solution, trapping the chain in suboptimal regions and reducing final accuracy\. Overall, both components contribute to the final accuracy, with trajectory guidance being the primary driver and annealed sampling schedule offering complementary stabilization\.

### 4\.4\.Ablation Study on Alignment Target

Table[3](https://arxiv.org/html/2608.03204#S4.T3)ablates our test\-time alignment target \(Eq\.[15](https://arxiv.org/html/2608.03204#S3.E15)\)\. Power sampling improves upon the reproduced Qwen2\.5\-VL\-7B baseline across several datasets, indicating that sharpening the sequence distribution is a strong training\-free lever\. Incorporating the vision\-aware term yields further gains over power sampling on every dataset except MathVista, aligning with its role in suppressing language\-prior amplification under weak or ambiguous visual evidence\.

Adding entropy regularization alone consistently underperforms power sampling, suggesting a failure mode in which low\-entropy trajectories are favored even when they encode confident mistakes, thereby limiting error correction\. Coupling it with the vision\-aware term alleviates this issue by filtering unsupported tokens and stabilizing visually grounded refinement\. Among all ablations, the joint objective closely matches the best setting\.

Adding the linguistic control term yields further gains, most notably on MathVista\. Results on MMStar \(typically short\) and MathVision \(often long\) suggest that a fixed length prior is not universally optimal, and the parametersLLshould instead be calibrated to the length regime of the target dataset\.

### 4\.5\.Generalization of Alignment Target

To further assess the generalizability of our refinement target, we evaluate it on two additional LVLMs, InternVL2\.5\-8B and Qwen3\-VL\-8B, with results summarized in Table[4](https://arxiv.org/html/2608.03204#S4.T4)\. We use the same core experimental settings as for Qwen2\.5\-VL\-7B to test whether the refinement target transfers across model families\. In particular, both the visual calibration weight \(β=2\.0\\beta=2\.0\) and the entropy weight \(λH=1\.0\\lambda\_\{H\}=1\.0\) remain effective without retuning\. The only parameter that requires mild model\-specific adjustment is the linguistic control saturation levelLL, likely due to differences in linguistic priors and response verbosity\. Even so, our method produces consistent improvements across base architectures\. Overall, these findings indicate that the proposed synergistic alignment target generalizes well across models and serves as a practical plug\-and\-play component for improving LVLM reasoning\.

![Refer to caption](https://arxiv.org/html/2608.03204v1/x4.png)Figure 4\.Likelihood and entropy of Qwen2\.5\-VL\-7B responses on MathVista, averaged by output length\.Table 5\.Response length and token consumption on three datasets\.Multipliers \(shown after×\\times\) report token consumption relative to greedy decoding \(×1\.0\\times 1\.0\)\.MethodMathVistaMMStarMathVisionResponse LengthGreedy Decoding242184566Multi\-Objective Sampling216164401TG Structured Sampling235187528Token ConsumptionGreedy Decoding2421\.0×\\times1841\.0×\\times5661\.0×\\timesMulti\-Objective Sampling5,66523\.4×\\times4,28223\.3×\\times12,84022\.3×\\timesTG Structured Sampling3,71015\.3×\\times2,96416\.2×\\times9,90417\.5×\\times
### 4\.6\.Reasoning Trace Likelihood and Entropy

Figure[4](https://arxiv.org/html/2608.03204#S4.F4)shows the kernel density estimates of average token entropy and log\-likelihood for Qwen2\.5\-VL\-7B responses on MathVista under power\-scaled sampling \(Power\), our full refinement target \(Ours\), our target without the linguistic term \(Ours w/o L\), and vanilla decoding \(Base\)\. By design, both power\-scaled sampling and our method shift samples toward higher\-likelihood, lower\-entropy regions of the base model, favoring more confident continuations\. “Ours \(w/o L\)” achieves the lowest entropy with likelihood comparable to power sampling, highlighting the effect of visual sharpening and entropy regularization\. Compared with pure power sampling, “Ours” retains slightly higher entropy and slightly lower sequence likelihood, reflecting the intended multi\-objective trade\-off\.

![Refer to caption](https://arxiv.org/html/2608.03204v1/x5.png)Figure 5\.Illustrative case study\.
### 4\.7\.Output Length and Inference Cost

Response length\.We do not observe a tendency toward longer responses under multi\-objective sampling, as suggested byKaran and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)\. Instead, we observe a negative correlation where response length decreases as the alignment objective becomes stronger\. This trend indicates that enhanced alignment effectively suppresses linguistic redundancy and filler tokens inherent in vanilla decoding\. Rather than relying on verbosity, our sampling objective directs the model toward the most discriminative reasoning steps, thereby increasing information density while preserving performance\. By comparison, trajectory\-guided structure sampling tends to produce longer responses than multi\-objective sampling\.

Token consumption\.We define token consumption as the average number of output tokens generated per query\. With visual calibration enabled, each output token incurs two forward passes, and we therefore count each token twice when computing token consumption\. As shown in Table[5](https://arxiv.org/html/2608.03204#S4.T5), sampling from the unnormalized target distribution incurs substantial inference\-time compute: multi\-objective sampling increases token consumption by about2222–24×24\\timesover greedy decoding\. In contrast, trajectory\-guided structured sampling reduces token consumption to1515–18×18\\timesand achieves a1\.31\.3–1\.5×1\.5\\timesreduction relative to multi\- objective sampling, while exhibiting longer response length\.

### 4\.8\.Case Study

Figure[5](https://arxiv.org/html/2608.03204#S4.F5)visualizes the initial reasoning state at each refinement iteration\. Since iterationiiinherits the accepted refinement from the previous cycle, consecutive panels reveal how the trajectory evolves through local updates\. Iteration 1 establishes the task, while Iteration 2 extends the trajectory with Visual Detection but omits the target brown cylinder, leaving the object inventory incomplete\. The initial state of Iteration 3 restores the missing object while preserving the earlier task context\. However, its Logical Reasoning remains incomplete because the subtraction does not account for the brown cylinder\. Iteration 4 retains the corrected detection and refines the reasoning into explicit steps,7−1=67\-1=6and6−1=56\-1=5, ultimately yielding the correct answer\. This illustrates the capacity of our MCMC\-based refinement to incrementally correct local inconsistencies while preserving the overall reasoning context\.

## 5\.Related Work

### 5\.1\.Test\-Time Alignment Methods

Test\-time alignment adapts a frozen model to new objectives, distribution shifts, or evolving user preferences at inference time rather than updating parameters, encompassing a broad range of methods unified by post\-hoc output steering\. First, hypothesis reweighting\(Leeet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib37)\)leverages a pre\-trained diverse ensemble of model heads and dynamically fits mixture weights at inference time using a small target\-domain adaptation set\. Second, reward\-guided decoding\(Xuet al\.,[2025a](https://arxiv.org/html/2608.03204#bib.bib38)\)achieves efficient test\-time alignment by reshaping token\-level probabilities with an autoregressive reward model\. Recent advances\(Linet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib9)\)further extend this paradigm by employing a single preference\-conditioned model, which overcomes the inefficiency of model ensembling and allows different alignment criteria to be dynamically balanced on the fly\. Third, control and planning approaches cast decoding as an inference\-time optimization problem, using sampling\-based control in pre\-logit space\(Kanaiet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib10)\)or subgoal\-guided predictive planning to maintain long\-horizon consistency\(Wanget al\.,[2025c](https://arxiv.org/html/2608.03204#bib.bib39)\)\. Motivated by this view, we propose a sampling\-based test\-time alignment method that refines LVLM generations via trajectory\-guided structured MCMC sampling\.

### 5\.2\.MCMC\-based Autoregressive Sampling

Several works have revisited classical Monte Carlo inference as a test\-time alternative to training\-time alignment, combining MCMC\-style updates with autoregressive generation\. For instance,Zhaoet al\.\([2024](https://arxiv.org/html/2608.03204#bib.bib42)\)cast controlled generation as probabilistic inference and introduce twisted Sequential Monte Carlo, learning prefix\-level twists to allocate particles toward high\-reward continuations\. Alternatively,Fariaet al\.\([2024](https://arxiv.org/html/2608.03204#bib.bib43)\)employ Metropolis–Hastings decoding to target a metric\-induced Gibbs distribution via iterative accept\-reject refinement with autoregressive proposals\. More recently,Karan and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)propose sampling from a power\-scaled distribution defined directly by the base model itself\. Our work is the first to investigate these principles for LVLMs\. We introduce an efficient block\-wise Metropolis–Hastings refinement steered by retrieved trajectories, significantly reducing the overhead of long\-horizon generation\. To ensure generation quality, we further enhance the sampling target with a synergistic objective\.

## 6\.Conclusion

This paper presents a sampling\-based test\-time alignment framework for LVLMs\. Our approach is based on an automated, agent\-driven trajectory learning algorithm to build a reasoning memory bank, where each trajectory captures high\-level reasoning patterns for solving complex multimodal problems\. To address the computational challenges of long\-horizon generation, we propose trajectory\-guided structured sampling, which enables localized refinement instead of costly full\-sequence resampling\. The retrieved guidance trajectory provides a global prior over reasoning structure and step order, making local refinement practical while preserving overall coherence\. On top of this sampler, our synergistic alignment objectives steer generation toward visually grounded, reliable, and non\-degenerate outputs\.

Experiments on five challenging datasets demonstrate that the framework achieves consistent performance gains without parameter updates, while substantially reducing token consumption relative to full\-sequence resampling\. Overall, our results position trajectory\-guided test\-time sampling as a practical and efficient paradigm for alignment in complex multimodal reasoning\.

###### Acknowledgements\.

This work was supported by the New Generation Artificial Intelligence National Science and Technology Major Project \(Grant No\. 2025ZD0123402\), the Computational Biology Program \(Grant No\. 25JS2830402\) of Science and Technology Commission of Shanghai Municipality \(STCSM\), and the Shanghai Municipal Science and Technology Major Project \(Grant No\. 2025SHZDZX025G06\)\.

## References

- M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann, J\. R\. Lee, Y\. T\. Lee, Y\. Li, W\. Liu, C\. C\. T\. Mendes, A\. Nguyen, E\. Price, G\. de Rosa, O\. Saarikivi, A\. Salim, S\. Shah, X\. Wang, R\. Ward, Y\. Wu, D\. Yu, C\. Zhang, and Y\. Zhang \(2024\)Phi\-4 technical report\.External Links:2412\.08905,[Link](https://arxiv.org/abs/2412.08905)Cited by:[§2](https://arxiv.org/html/2608.03204#S2.p1.9)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025a\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[1st item](https://arxiv.org/html/2608.03204#S4.I1.i1.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.18.6.1.1.1),[Table 4](https://arxiv.org/html/2608.03204#S4.T4.4.7.2.1.1.1)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025b\)Qwen2\.5\-vl technical report\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1),[1st item](https://arxiv.org/html/2608.03204#S4.I1.i1.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.16.4.1.1.1)\.
- A\. Brown and J\. S\. Rosenthal \(2025\)Upper and lower bounds on the subgeometric convergence of adaptive markov chain monte carlo\.External Links:2411\.17084,[Link](https://arxiv.org/abs/2411.17084)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p3.2),[§3\.2](https://arxiv.org/html/2608.03204#S3.SS2.p4.1)\.
- L\. Chen, J\. Li, X\. Dong, P\. Zhang, Y\. Zang, Z\. Chen, H\. Duan, J\. Wang, Y\. Qiao, D\. Lin, and F\. Zhao \(2024\)Are we on the right way for evaluating large vision\-language models?\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=evP9mxNNxJ)Cited by:[§4\.1\.1](https://arxiv.org/html/2608.03204#S4.SS1.SSS1.p1.1)\.
- Z\. Chen, W\. Wang, Y\. Cao, Y\. Liu, Z\. Gao, E\. Cui, J\. Zhu, S\. Ye, H\. Tian, Z\. Liu, L\. Gu, X\. Wang, Q\. Li, Y\. Ren, Z\. Chen, J\. Luo, J\. Wang, T\. Jiang, B\. Wang, C\. He, B\. Shi, X\. Zhang, H\. Lv, Y\. Wang, W\. Shao, P\. Chu, Z\. Tu, T\. He, Z\. Wu, H\. Deng, J\. Ge, K\. Chen, K\. Zhang, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang \(2025\)Expanding performance boundaries of open\-source multimodal models with model, data, and test\-time scaling\.External Links:2412\.05271,[Link](https://arxiv.org/abs/2412.05271)Cited by:[1st item](https://arxiv.org/html/2608.03204#S4.I1.i1.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.15.3.1.1.1),[Table 4](https://arxiv.org/html/2608.03204#S4.T4.4.6.1.1.1.1)\.
- G\. Cui, Y\. Zhang, J\. Chen, L\. Yuan, Z\. Wang, Y\. Zuo, H\. Li, Y\. Fan, H\. Chen, W\. Chen,et al\.\(2025\)The entropy mechanism of reinforcement learning for reasoning language models\.arXiv preprint arXiv:2505\.22617\.Cited by:[§3\.3\.2](https://arxiv.org/html/2608.03204#S3.SS3.SSS2.p1.2)\.
- Y\. Deng, H\. Bansal, F\. Yin, N\. Peng, W\. Wang, and K\. Chang \(2025\)OpenVLThinker: complex vision\-language reasoning via iterative sft\-rl cycles\.External Links:2503\.17352,[Link](https://arxiv.org/abs/2503.17352)Cited by:[2nd item](https://arxiv.org/html/2608.03204#S4.I1.i2.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.20.8.1.1.1)\.
- G\. Faria, S\. Agrawal, A\. Farinhas, R\. Rei, J\. G\. C\. de Souza, and A\. Martins \(2024\)QUEST: quality\-aware metropolis\-hastings sampling for machine translation\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=dLnduWGTB4)Cited by:[§5\.2](https://arxiv.org/html/2608.03204#S5.SS2.p1.1)\.
- N\. Hu, X\. Duan, J\. Zhang, and G\. Kang \(2025\)Enhancing visual reliance in text generation: a bayesian perspective on mitigating hallucination in large vision\-language models\.InProceedings of the 33rd ACM International Conference on Multimedia,MM ’25,New York, NY, USA,pp\. 4778–4787\.External Links:ISBN 9798400720352,[Link](https://doi.org/10.1145/3746027.3755606),[Document](https://dx.doi.org/10.1145/3746027.3755606)Cited by:[§3\.3\.1](https://arxiv.org/html/2608.03204#S3.SS3.SSS1.p1.7)\.
- W\. Huang, B\. Jia, Z\. Zhai, S\. Cao, Z\. Ye, F\. Zhao, Z\. Xu, Y\. Hu, and S\. Lin \(2025\)Vision\-r1: incentivizing reasoning capability in multimodal large language models\.External Links:2503\.06749,[Link](https://arxiv.org/abs/2503.06749)Cited by:[2nd item](https://arxiv.org/html/2608.03204#S4.I1.i2.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.22.10.1.1.1)\.
- F\. Huo, W\. Xu, Z\. Zhang, H\. Wang, Z\. Chen, and P\. Zhao \(2025\)Self\-introspective decoding: alleviating hallucinations for large vision\-language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rsZwwjYHuD)Cited by:[§3\.3\.1](https://arxiv.org/html/2608.03204#S3.SS3.SSS1.p1.7)\.
- S\. Kanai, T\. Yoshida, H\. Takahashi, H\. Kuroki, and K\. Hashimoto \(2025\)Test\-time alignment of llms via sampling\-based optimal control in pre\-logit space\.External Links:2510\.26219,[Link](https://arxiv.org/abs/2510.26219)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03204#S5.SS1.p1.1)\.
- A\. Karan and Y\. Du \(2025\)Reasoning with sampling: your base model is smarter than you think\.External Links:2510\.14901,[Link](https://arxiv.org/abs/2510.14901)Cited by:[§A\.4](https://arxiv.org/html/2608.03204#A1.SS4.p1.6),[§B\.2](https://arxiv.org/html/2608.03204#A2.SS2.p1.1),[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§2](https://arxiv.org/html/2608.03204#S2.p1.8),[§4\.1\.3](https://arxiv.org/html/2608.03204#S4.SS1.SSS3.p1.9),[§4\.7](https://arxiv.org/html/2608.03204#S4.SS7.p1.1),[§5\.2](https://arxiv.org/html/2608.03204#S5.SS2.p1.1)\.
- Y\. Lee, J\. Williams, H\. Marklund, A\. Sharma, E\. Mitchell, A\. Singh, and C\. Finn \(2025\)Inference\-time alignment via hypothesis reweighting\.InSecond Workshop on Test\-Time Adaptation: Putting Updates to the Test\! at ICML 2025,External Links:[Link](https://openreview.net/forum?id=tl2nXqQSQJ)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03204#S5.SS1.p1.1)\.
- S\. Leng, H\. Zhang, G\. Chen, X\. Li, S\. Lu, C\. Miao, and L\. Bing \(2024\)Mitigating Object Hallucinations in Large Vision\-Language Models through Visual Contrastive Decoding\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,Los Alamitos, CA, USA,pp\. 13872–13882\.External Links:ISSN,[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01316),[Link](https://doi.ieeecomputersociety.org/10.1109/CVPR52733.2024.01316)Cited by:[§3\.3\.1](https://arxiv.org/html/2608.03204#S3.SS3.SSS1.p1.3)\.
- M\. Li, N\. Zhang, C\. Fan, H\. Jiao, Y\. Fu, S\. Peters, Q\. Xu, R\. Lissitz, and T\. Zhou \(2025a\)Understanding the thinking process of reasoning models: a perspective from schoenfeld’s episode theory\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 18278–18299\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.922/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.922),ISBN 979\-8\-89176\-332\-6Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p2.1)\.
- S\. Li, Z\. Lin, S\. Yang, J\. Zhao, and W\. Chen \(2025b\)ThinkPilot: steering reasoning models via automated think\-prefixes optimization\.External Links:2510\.12063,[Link](https://arxiv.org/abs/2510.12063)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p2.1)\.
- Z\. Li, W\. Yu, C\. Huang, R\. Liu, Z\. Liang, F\. Liu, J\. Che, D\. Yu, J\. Boyd\-Graber, H\. Mi, and D\. Yu \(2025c\)Self\-rewarding vision\-language model via reasoning decomposition\.External Links:2508\.19652,[Link](https://arxiv.org/abs/2508.19652)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p1.1)\.
- B\. Lin, W\. Jiang, Y\. Xu, H\. Chen, and Y\. Chen \(2025\)PARM: multi\-objective test\-time alignment via preference\-aware autoregressive reward model\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=zm53HtGiXN)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03204#S5.SS1.p1.1)\.
- Z\. Liu, Y\. Zang, X\. Dong, P\. Zhang, Y\. Cao, H\. Duan, C\. He, Y\. Xiong, D\. Lin, and J\. Wang \(2025\)MIA\-DPO: multi\-image augmented direct preference optimization for large vision\-language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=f7WBRSuf9l)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1)\.
- P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KUNzEQMWU7)Cited by:[§4\.1\.1](https://arxiv.org/html/2608.03204#S4.SS1.SSS1.p1.1)\.
- P\. Lu, B\. Chen, S\. Liu, R\. Thapa, J\. Boen, and J\. Zou \(2025\)OctoTools: an agentic framework with extensible tools for complex reasoning\.External Links:2502\.11271,[Link](https://arxiv.org/abs/2502.11271)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p3.6)\.
- F\. Meng, L\. Du, Z\. Liu, Z\. Zhou, Q\. Lu, D\. Fu, T\. Han, B\. Shi, W\. Wang, J\. He, K\. Zhang, P\. Luo, Y\. Qiao, Q\. Zhang, and W\. Shao \(2025\)MM\-eureka: exploring the frontiers of multimodal reasoning with rule\-based reinforcement learning\.External Links:2503\.07365,[Link](https://arxiv.org/abs/2503.07365)Cited by:[2nd item](https://arxiv.org/html/2608.03204#S4.I1.i2.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.23.11.1.1.1)\.
- N\. Metropolis, A\. W\. Rosenbluth, M\. N\. Rosenbluth, A\. H\. Teller, and E\. Teller \(1953\)Equation of state calculations by fast computing machines\.Journal of Chemical Physics21\(6\),pp\. 1087–1092\.Cited by:[§3\.2](https://arxiv.org/html/2608.03204#S3.SS2.p2.8)\.
- X\. Tian, S\. Zou, Z\. Yang, M\. He, F\. Waschkowski, L\. Wesemann, P\. Tu, and J\. Zhang \(2025\)More thought, less accuracy? on the dual nature of reasoning in vision\-language models\.External Links:2509\.25848,[Link](https://arxiv.org/abs/2509.25848)Cited by:[§B\.2](https://arxiv.org/html/2608.03204#A2.SS2.p1.1)\.
- C\. Wang, X\. Chen, N\. Zhang, B\. Tian, H\. Xu, S\. Deng, and H\. Chen \(2025a\)MLLM can see? dynamic correction decoding for hallucination mitigation\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4z3IguA4Zg)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p3.2)\.
- H\. Wang, C\. Qu, Z\. Huang, W\. Chu, F\. Lin, and W\. Chen \(2025b\)VL\-rethinker: incentivizing self\-reflection of vision\-language models with reinforcement learning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=4oYxzssbVg)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p1.1),[2nd item](https://arxiv.org/html/2608.03204#S4.I1.i2.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.21.9.1.1.1)\.
- K\. Wang, J\. Pan, W\. Shi, Z\. Lu, H\. Ren, A\. Zhou, M\. Zhan, and H\. Li \(2024\)Measuring multimodal mathematical reasoning with MATH\-vision dataset\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=QWTCcxMpPA)Cited by:[§4\.1\.1](https://arxiv.org/html/2608.03204#S4.SS1.SSS1.p1.1)\.
- K\. Wang, T\. Chen, Y\. H\. Hung, G\. Ko, S\. Ding, Y\. Wu, Y\. F\. Wang, C\. H\. Yang, W\. Peng, and P\. Hsieh \(2025c\)Test\-time alignment for large language models via textual model predictive control\.External Links:2502\.20795,[Link](https://arxiv.org/abs/2502.20795)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03204#S5.SS1.p1.1)\.
- X\. Wang, Z\. Yang, C\. Feng, H\. Lu, L\. Li, C\. Lin, K\. Lin, F\. Huang, and L\. Wang \(2025d\)SoTA with less: mcts\-guided sample selection for data\-efficient visual reasoning self\-improvement\.External Links:2504\.07934,[Link](https://arxiv.org/abs/2504.07934)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p2.1),[2nd item](https://arxiv.org/html/2608.03204#S4.I1.i2.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.24.12.1.1.1)\.
- J\. Wu, M\. Feng, G\. Zhai, S\. Zhang, Z\. Lian, F\. Lv, P\. Shao, R\. Jin, Z\. Wen, and J\. Tao \(2026\)AStar: boosting multimodal reasoning with automated structured thinking\.External Links:2502\.02339,[Link](https://arxiv.org/abs/2502.02339)Cited by:[§3\.1](https://arxiv.org/html/2608.03204#S3.SS1.p2.1)\.
- Y\. Xu, U\. M\. Sehwag, A\. Koppel, S\. Zhu, B\. An, F\. Huang, and S\. Ganesh \(2025a\)GenARM: reward guided generation with autoregressive reward model for test\-time alignment\.External Links:2410\.08193,[Link](https://arxiv.org/abs/2410.08193)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p2.1),[§5\.1](https://arxiv.org/html/2608.03204#S5.SS1.p1.1)\.
- Z\. Xu, C\. Liu, Q\. Wei, J\. Wu, J\. Zou, X\. E\. Wang, Y\. Zhou, and S\. Liu \(2025b\)More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=KzU33wR875)Cited by:[§B\.2](https://arxiv.org/html/2608.03204#A2.SS2.p1.1)\.
- H\. Yao, J\. Huang, W\. Wu, J\. Zhang, Y\. Wang, S\. Liu, Y\. Wang, Y\. Song, H\. Feng, L\. Shen, and D\. Tao \(2025\)Mulberry: empowering MLLM with o1\-like reasoning and reflection via collective monte carlo tree search\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=lwOV2ACEK9)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1)\.
- H\. Yin, G\. Si, and Z\. Wang \(2025\)ClearSight: visual signal enhancement for object hallucination mitigation in multimodal large language models\.pp\. 14625–14634\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.01363)Cited by:[§3\.3\.1](https://arxiv.org/html/2608.03204#S3.SS3.SSS1.p1.3)\.
- T\. Yu, Y\. Yao, H\. Zhang, T\. He, Y\. Han, G\. Cui, J\. Hu, Z\. Liu, H\. Zheng, and M\. Sun \(2024\)RLHF\-v: towards trustworthy mllms via behavior alignment from fine\-grained correctional human feedback\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 13807–13816\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01310)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen \(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of CVPR,Cited by:[§4\.1\.1](https://arxiv.org/html/2608.03204#S4.SS1.SSS1.p1.1)\.
- R\. Zhang, D\. Jiang, Y\. Zhang, H\. Lin, Z\. Guo, P\. Qiu, A\. Zhou, P\. Lu, K\. Chang, P\. Gao,et al\.\(2024\)MathVerse: does your multi\-modal llm truly see the diagrams in visual math problems?\.arXiv preprint arXiv:2403\.14624\.Cited by:[§4\.1\.1](https://arxiv.org/html/2608.03204#S4.SS1.SSS1.p1.1)\.
- R\. Zhang, B\. Zhang, Y\. Li, H\. Zhang, Z\. Sun, Z\. Gan, Y\. Yang, R\. Pang, and Y\. Yang \(2025\)Improve vision language model chain\-of\-thought reasoning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 1631–1662\.External Links:[Link](https://aclanthology.org/2025.acl-long.82/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.82),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1)\.
- S\. Zhao, R\. Brekelmans, A\. Makhzani, and R\. Grosse \(2024\)Probabilistic inference in language models via twisted sequential monte carlo\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§5\.2](https://arxiv.org/html/2608.03204#S5.SS2.p1.1)\.
- J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao, Z\. Gao, E\. Cui, X\. Wang, Y\. Cao, Y\. Liu, X\. Wei, H\. Zhang, H\. Wang, W\. Xu, H\. Li, J\. Wang, N\. Deng, S\. Li, Y\. He, T\. Jiang, J\. Luo, Y\. Wang, C\. He, B\. Shi, X\. Zhang, W\. Shao, J\. He, Y\. Xiong, W\. Qu, P\. Sun, P\. Jiao, H\. Lv, L\. Wu, K\. Zhang, H\. Deng, J\. Ge, K\. Chen, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang \(2025\)InternVL3: exploring advanced training and test\-time recipes for open\-source multimodal models\.External Links:2504\.10479,[Link](https://arxiv.org/abs/2504.10479)Cited by:[§1](https://arxiv.org/html/2608.03204#S1.p1.1),[1st item](https://arxiv.org/html/2608.03204#S4.I1.i1.p1.1),[Table 1](https://arxiv.org/html/2608.03204#S4.T1.12.17.5.1.1.1)\.

## Appendix AImplementation Details

### A\.1\.Definition of Reasoning Patterns

Table 6\.Reasoning patterns with explicit interfaces\.Table[6](https://arxiv.org/html/2608.03204#A1.T6)defines seven reasoning patterns as step functions with explicit input–output interfaces\. Each step consumes either the questionXX, a single task, or an accumulated information list, and produces a task \(or subtask list\), an information list, or the final answer\. Notably, while the table specifies the formal I/O signatures, the actual execution of each pattern also involves internal reasoning processes that transform inputs into outputs\. This design turns free\-form chain\-of\-thought into composable, well\-structured trajectories, making complex behaviors such as decomposition and grounding easier to elicit and reuse\.

Such pseudo\-code\-style definitions also support our downstream pipeline\. They make trajectories comparable at the step level for retrieval and voting, enable sliding\-window sampling by regenerating selected patterns while keeping others fixed, and simplify library curation through lightweight checks on structured outputs\. Moreover, when scaling to more patterns, these interfaces make it straightforward to introduce rule\-based behavior filters that accept, reject, or reroute pattern executions based on their inputs/outputs, steering reasoning toward desired behaviors\. Overall, the pattern interfaces yield interpretable trajectories that are easier to construct and are more amenable to step\-aligned refinement\.

### A\.2\.Entropy\-Weighted Multi\-View Retrieval

For a query instance\(V,X\)\(V,X\), we initially retrieve a candidate pool of sizeM=100M\{=\}100via question\-only embedding similarity\. To account for the heterogeneous information across modalities, we propose an adaptive re\-ranking mechanism based on entropy\-weighted multi\-view fusion\. We consider three complementary perspectives: textual semantic similaritystexts\_\{\\text\{text\}\}, visual consistencysimgs\_\{\\text\{img\}\}, and lexicalnn\-gram overlapsngs\_\{\\text\{ng\}\}\. To dynamically modulate the importance of each view, we utilize an information\-theoretic approach to quantify their respective utilities\. Specifically, for each viewii, we define the probability distributionpip\_\{i\}over the candidate pool as:

\(16\)pi\(j\)=si\(j\)\+ϵ∑j′=1M\(si\(j′\)\+ϵ\)\.p\_\{i\}^\{\(j\)\}=\\frac\{s\_\{i\}^\{\(j\)\}\+\\epsilon\}\{\\sum\_\{j^\{\\prime\}=1\}^\{M\}\\left\(s\_\{i\}^\{\(j^\{\\prime\}\)\}\+\\epsilon\\right\)\}\.The reliability of each modality is then inversely proportional to its normalized entropyHiH\_\{i\}:

\(17\)Hi=−1log⁡M​∑j=1Mpi\(j\)​log⁡pi\(j\)\.H\_\{i\}=\-\\frac\{1\}\{\\log M\}\\sum\_\{j=1\}^\{M\}p\_\{i\}^\{\(j\)\}\\log p\_\{i\}^\{\(j\)\}\.Intuitively, a view that yields a highly skewed distribution \(low entropy\) is more informative for ranking than one with a uniform distribution \(high entropy\)\. The final retrieval scoreS\(j\)S^\{\(j\)\}is computed as a weighted sum∑iwi​si\(j\)\\sum\_\{i\}w\_\{i\}s\_\{i\}^\{\(j\)\}, wherewi∝\(1−Hi\)w\_\{i\}\\propto\(1\-H\_\{i\}\)\. This ensures that the top\-kkresults are selected based on the most reliable and discriminative signals for the given query\. We use Qwen3\-Embedding\-0\.6B as our text embedding model and clip\-vit\-base\-patch32 as our image embedding model\.

### A\.3\.Design of Difficulty Estimator

After obtaining the guidance trajectory, we apply a simple rule\-based estimator to route instances by difficulty\. We first remove behaviors that do not directly advance the reasoning process, includingTask Initialization,Self\-Verification, andFinal Conclusion\. If the remaining trajectory contains more than one action, we treat the instance as non\-trivial, suggesting that it likely requires multi\-hop reasoning; we then perform block\-wise sampling under trajectory guidance\. Otherwise, we treat it as trivial and apply full\-sequence MCMC without trajectory guidance, which avoids overthinking while improving inference efficiency\.

### A\.4\.MCMC for Full\-Sequence Sampling

For full\-sequence sampling, we employ the autoregressive MH sampler proposed byKaran and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)\. To ensure sampling efficiency within high\-dimensional spaces, a sequence of maximum lengthTTis partitioned intoKKblocks, each consisting ofT/KT/Ktokens\. The refinement process followsKKsequential cycles, during which the number of active blocks eligible for resampling and MH updates increases incrementally\. The procedure terminates early if an end\-of\-sequence \(EOS\) token is generated\. In our implementation, we set the maximum sequence lengthT=3072T=3072and the number of blocksK=16K=16\.

## Appendix BMore Experiments

### B\.1\.Generation Entropy and Answer Accuracy

![Refer to caption](https://arxiv.org/html/2608.03204v1/x6.png)Figure 6\.Average smoothed entropy over timestep\.In Figure[6](https://arxiv.org/html/2608.03204#A2.F6), we plot the average smoothed token\-level output entropyHHover decoding trajectories on the ThinkLite\-70k dataset\. A consistent gap is observed throughout generation: trajectories that lead to correct answers exhibit markedly lower entropy than incorrect ones\. While this trend suggests that entropy can serve as a proximal reliability signal for test\-time refinement, it is not an infallible metric\. The model may occasionally succumb to being confidently wrong, yielding low\-entropy but factually incorrect outputs that stifle beneficial exploration\. To mitigate this, our framework employs entropy as a regularization term coupled with a vision\-aware calibration mechanism\. This dual approach effectively filters out low\-entropy hallucinations, leveraging entropy to stabilize the refinement process rather than as a binary criterion for correctness\.

### B\.2\.Analysis on Response Length

Table 7\.Response length of different sampling objectives on MathVista and MMStar with Qwen2\.5\-VL\-7B\.![Refer to caption](https://arxiv.org/html/2608.03204v1/x7.png)Figure 7\.Effect of MCMC refinement steps on MathVista and MMStar with Qwen2\.5\-VL\-7B\.![Refer to caption](https://arxiv.org/html/2608.03204v1/x8.png)Figure 8\.Case study of different sampling variants\.Table[7](https://arxiv.org/html/2608.03204#A2.T7)reports the average response length across three sampling configurations\. We observe a consistent trend where the introduction of additional alignment objectives leads to a reduction in average response length\. First, naive power sampling leads to shorter responses, which contrasts with the findings ofKaran and Du \([2025](https://arxiv.org/html/2608.03204#bib.bib5)\)\. We attribute this to the tendency of LVLMs to lose attention to visual inputs as generation length increases\(Xuet al\.,[2025b](https://arxiv.org/html/2608.03204#bib.bib40); Tianet al\.,[2025](https://arxiv.org/html/2608.03204#bib.bib41)\), leading to performance degradation\. We leave a detailed study for future work\. Vanilla decoding is frequently bottlenecked by linguistic redundancy, where high\-probability but uninformative sequences dominate the output\. By coupling visual evidence with entropy\-based weighting, our method actively penalizes such filler tokens, redirecting the generation toward critical, discriminative reasoning steps\. As a result, the refined trajectories deliver higher accuracy with remarkable conciseness, fundamentally enhancing the information density of the underlying reasoning\. A qualitative case study is provided in Figure[8](https://arxiv.org/html/2608.03204#A2.F8)\. Specifically, greedy decoding and power\-scaled sampling tend to generate either irrelevant contextual fillers or hallucinatory descriptions that contradict the visual evidence, ultimately culminating in incorrect conclusions\. In contrast, by filtering out these linguistically dominant but visually inconsistent paths, our method directs the model to converge on the correct answer with fewer tokens\. This case illustrates that our approach effectively prunes redundant or divergent reasoning branches, ensuring that the generated trajectory remains both concise and strictly grounded in the provided image\.

### B\.3\.Scaling Law on MCMC Steps

NMCMCN\_\{\\text\{MCMC\}\}controls test\-time scaling in our framework by determining the number of Metropolis–Hastings refinement steps performed under the alignment target\. Each step corresponds to one propose–accept update, so increasingNMCMCN\_\{\\text\{MCMC\}\}allocates more decoding\-time compute to sequence\-level correction, while smallerNMCMCN\_\{\\text\{MCMC\}\}favors lower latency\. Figure[7](https://arxiv.org/html/2608.03204#A2.F7)shows that increasingNMCMCN\_\{\\text\{MCMC\}\}generally improves performance on both MMStar and MathVista, although the gains are not strictly monotonic at intermediate steps\. On MMStar, accuracy rises from 61\.8 atNMCMC=0N\_\{\\text\{MCMC\}\}=0to 64\.2 atNMCMC=8N\_\{\\text\{MCMC\}\}=8, with a slight drop to 64\.0 atNMCMC=10N\_\{\\text\{MCMC\}\}=10, suggesting that most of the benefit is already realized within a moderate refinement budget\. On MathVista, the trend is more variable at small step counts but becomes clearly positive at larger budgets, improving from 70\.5 atNMCMC=0N\_\{\\text\{MCMC\}\}=0to 73\.2 atNMCMC=10N\_\{\\text\{MCMC\}\}=10\. Overall, these results suggest a favorable test\-time scaling trend with respect toNMCMCN\_\{\\text\{MCMC\}\}: allocating more MCMC refinement steps generally improves reasoning accuracy, although the returns become less smooth and dataset\-dependent at intermediate budgets\.

### B\.4\.Evaluation on Hallucination Benchmarks

Table 8\.Evaluation of hallucination on Qwen2\.5\-VL\-7B\. We report theF1F\_\{1\}score for POPE and CHAIR scores \(lower scores indicate fewer hallucinations\. \)\.To evaluate the effectiveness of vision\-aware distribution sharpening in mitigating hallucinations, we perform a quantitative analysis across two standard benchmarks, as summarized in Table[8](https://arxiv.org/html/2608.03204#A2.T8)\. We evaluate object hallucinations using POPE\(li\-etal\-2023\-evaluating\), a VQA\-based probing protocol, and CHAIR\(rohrbach\-etal\-2018\-object\), which quantifies hallucinations in image captioning by cross\-referencing generated tokens with ground\-truth objects\. As formulated in Equation \([18](https://arxiv.org/html/2608.03204#A2.E18)\) and \([19](https://arxiv.org/html/2608.03204#A2.E19)\), CHAIR measures hallucination at both the instance \(CHAIRI\\text\{CHAIR\}\_\{I\}\) and sentence \(CHAIRS\\text\{CHAIR\}\_\{S\}\) levels:

\(18\)CHAIRI=\|\{hallucinated objects\}\|all mentioned objects,\\text\{CHAIR\}\_\{I\}=\\frac\{\|\\\{\\text\{hallucinated objects\}\\\}\|\}\{\\text\{all mentioned objects\}\},
\(19\)CHAIRS=\|\{captions with hallucinated objects\}\|all captions\.\\text\{CHAIR\}\_\{S\}=\\frac\{\|\\\{\\text\{captions with hallucinated objects\}\\\}\|\}\{\\text\{all captions\}\}\.
As reported in Table[8](https://arxiv.org/html/2608.03204#A2.T8), power sampling with visual calibration yields marginal yet consistent improvements across the evaluated benchmarks\. Specifically, our method achieves a POPEF1F\_\{1\}score of 86\.1, slightly surpassing the greedy decoding baseline\. Notably, while vanilla power sampling exhibits a slight increase in hallucination rates \(e\.g\., 39\.2CHAIRS\\text\{CHAIR\}\_\{S\}\), the integration of visual priors successfully mitigates this regression, reaching the lowestCHAIRS\\text\{CHAIR\}\_\{S\}of 38\.6\. These results suggest that our approach helps maintain factual alignment with the visual input\.

### B\.5\.Hyperparameter sensitivity

![Refer to caption](https://arxiv.org/html/2608.03204v1/x9.png)Figure 9\.Hyperparameter sensitivity analysis on the MMStar dataset with Qwen2\.5\-VL\-7B and InternVL2\.5\-8B\.Figure[9](https://arxiv.org/html/2608.03204#A2.F9)examines the sensitivity of Qwen2\.5\-VL\-7B and InternVL2\.5\-8B to the calibration weightβ\\betaand entropy weightλH\\lambda\_\{H\}on MMStar\. Both models attain their highest accuracy at approximatelyβ=2\.0\\beta=2\.0andλH=1\.0\\lambda\_\{H\}=1\.0\. Performance remains relatively stable under moderate deviations from these values, indicating that the method is not overly sensitive to either hyperparameter, although more extreme settings cause noticeable degradation\. Importantly, all evaluated configurations consistently outperform their corresponding greedy baselines, demonstrating the robustness of the proposed method across a broad range of hyperparameter choices\.

## Appendix CVisualization

![Refer to caption](https://arxiv.org/html/2608.03204v1/x10.png)Figure 10\.Visualization of vision\-aware calibration weights​\(rt\)s\(r\_\{t\}\)and length rewardRL​\(𝐘\)R\_\{L\}\(\\mathbf\{Y\}\)Figure[10](https://arxiv.org/html/2608.03204#A3.F10)visualizes the two shaping functions used in our alignment target\. The top panel shows the effective\-length rewardRL​\(𝐘\)R\_\{L\}\(\\mathbf\{Y\}\)as a function ofSL​\(𝐘\)S\_\{L\}\(\\mathbf\{Y\}\)\. In all cases, the reward increases monotonically with effective length and gradually saturates at the target levelLL, encouraging sufficiently long reasoning traces while avoiding unbounded growth\. The parametersaaandbbcontrol the growth rate and saturation behavior: larger values lead to faster early growth and earlier saturation, while smaller values produce a smoother increase\. In our experiments, we seta=0\.05a=0\.05,b=0\.8b=0\.8for all settings\.

The bottom panel shows the vision\-aware calibration weights​\(rt\)s\(r\_\{t\}\)as a function of the alignment ratiortr\_\{t\}\. The weight is bounded in\[0,1\]\[0,1\], increases monotonically withrtr\_\{t\}, and reaches11whenrt≥1r\_\{t\}\\geq 1, so tokens that are better supported by visual evidence receive full sharpening\. The parameterβ\\betacontrols the selectivity of this calibration: smallerβ\\betayields a smoother transition, whereas largerβ\\betasuppresses weakly aligned tokens more aggressively\. We useβ=2\.0\\beta=2\.0in all experimental settings\.