Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

# Reasoning Instructions Can Break Answer Decoding in Vision–Language Models
Source: [https://arxiv.org/html/2609.29278](https://arxiv.org/html/2609.29278)
###### Abstract

Chain\-of\-thought \(CoT\) instructions can distort multiple\-choice VLM evaluation when a scorer appends a reasoning cue but reads answer\-label logits before the model generates any rationale\. We call this CoT\-prefix scoring\. On ScienceQA, Qwen2\.5\-VL\-7B drops from 80\.76% to 45\.48%, and across five option\-content permutations 93\.54% of CoT\-prefix predictions select the first slot\. Condition\-matched linear probes recover 78\.94% from the same hidden states, while free generation restores 75\.24%, showing that the answer often survives the prefix and the immediate readout fails\. Vocabulary and layer diagnostics explain the mismatch: probability mass moves toward continuation tokens, while answer information remains linearly accessible in late layers\. The effect recurs with varying severity across datasets and models, though not universally\. These results show that CoT\-prefix scoring can confound model knowledge with an evaluation\-interface mismatch and should be avoided unless the requested and scored output events are aligned\.

###### Index Terms:

vision–language models, chain of thought, multiple\-choice evaluation, selection bias

††address:1Shanghai Jiao Tong University## 1Introduction

Chain\-of\-thought \(CoT\) prompting lets a model generate intermediate steps before it answers, and this idea has been widely adopted in multimodal reasoning\[[18](https://arxiv.org/html/2609.29278#bib.bib1),[11](https://arxiv.org/html/2609.29278#bib.bib2),[15](https://arxiv.org/html/2609.29278#bib.bib14)\]\. In multiple\-choice evaluation, however, a common shortcut skips the reasoning\. The evaluator appends an instruction such as “Let me think step by step\.” to the question, then reads the answer directly from the next\-token logits on the option labels, without waiting for the model to generate any explanation\. The prompt asks the model to start explaining, while the scorer treats its first token as the final answer\. We refer to this procedure as*CoT\-prefix scoring*\. It is a scoring convention that has been widely used in practice, but it has not been systematically examined\.

We find that this shortcut can severely understate a strong model\. On ScienceQA\[[11](https://arxiv.org/html/2609.29278#bib.bib2)\], Qwen2\.5\-VL\-7B\[[3](https://arxiv.org/html/2609.29278#bib.bib13)\]loses more than thirty points of accuracy under CoT\-prefix scoring, and the failure is content\-blind\. Across five permutations of the option contents, the large majority of predictions land on the first option regardless of what that option contains\. Nothing about the questions has changed, only the suffix\. Either the suffix has erased the answer from the model, or the answer remains inside and the one\-step readout cannot reach it\.

![Refer to caption](https://arxiv.org/html/2609.29278v1/fig1_interface_mismatch_concise.png)Figure 1:A single suffix creates an event mismatch\. Direct prompting requests and scores B; the CoT prefix requests a rationale, so immediate label scoring can return A even when free generation and a matched probe recover B\.Distinguishing answer loss from readout failure matters because multiple\-choice accuracy can shift for reasons unrelated to model knowledge\. Prompt format, answer priors, option order, and the surface form being scored all affect the measured result\[[20](https://arxiv.org/html/2609.29278#bib.bib17),[6](https://arxiv.org/html/2609.29278#bib.bib18),[21](https://arxiv.org/html/2609.29278#bib.bib7)\]\. Prior work has benchmarked LVLM selection bias and logit correction\[[2](https://arxiv.org/html/2609.29278#bib.bib8),[16](https://arxiv.org/html/2609.29278#bib.bib9)\], analyzed position effects\[[17](https://arxiv.org/html/2609.29278#bib.bib15),[14](https://arxiv.org/html/2609.29278#bib.bib16)\], and documented prompt\-format flaws in multiple\-choice VQA\[[13](https://arxiv.org/html/2609.29278#bib.bib10)\]\. What these studies do not isolate is the specific event mismatch considered here: a continuation\-inducing suffix is appended, but the evaluator immediately restricts scoring to answer labels before any rationale token is generated\. We therefore use condition\-matched probes trained on the official training split, selected on validation, and evaluated on a locked test set, together with four controls: option\-content permutation, free reasoning before answer extraction, alternative scored output events, and replication across datasets and model families\.

The results support readout failure rather than answer loss\. A matched\-capacity probe recovers most of the lost points from the same final hidden state, and letting the model actually generate the requested reasoning restores most of the direct\-answer performance\. The collapse recurs, sometimes more severely, on other datasets and models, while one counterexample shows that the suffix can help instead\. Figure[1](https://arxiv.org/html/2609.29278#S1.F1)summarizes this*CoT\-prefix decodability gap*\. The important distinction is that a low immediate label score need not imply that the model has lost the answer; the requested continuation can change which event the native readout is prepared to emit\.

Our contributions are threefold\. First, we identify and name CoT\-prefix scoring as a common evaluation shortcut\. Second, we show through condition\-matched probes and generation controls that the answer survives the prefix and the failure is localized to the immediate readout\. Third, we trace the failure to a shift of probability mass onto continuation tokens, and we recommend that evaluations align the requested output event with the scored output event\.

## 2Evaluation Interfaces and Diagnostic Protocol

We compare three ways of obtaining an answer to the same multiple\-choice item\. Every item contains an image when available, a question, and two to five lettered choices\.Directends the prompt after the choices and scores the native logits of the valid answer tokens\.CoT\-prefixappends “Let me think step by step\.” yet still scores the answer tokens at the very next position, without generating any text\.Free CoTlets the model produce the requested continuation and extracts an answer only after generation\. The first two conditions differ only in the terminal text, so their contrast isolates the suffix\. The third also changes the output event, so its contrast with the second isolates generation itself\.

LetPc​\(x\)P\_\{c\}\(x\)denote the complete prompt under conditionc∈\{D,C\}c\\in\\\{D,C\\\},hc​\(x\)=fθ​\(Pc​\(x\)\)h\_\{c\}\(x\)=f\_\{\\theta\}\(P\_\{c\}\(x\)\)the hidden state at the final input position, andEvE\_\{v\}the output embedding of vocabulary tokenvv\. The model’s next\-token distribution is

pc​\(v∣x\)=exp⁡\(Ev⊤​hc​\(x\)\)∑u∈𝒱exp⁡\(Eu⊤​hc​\(x\)\)\.p\_\{c\}\(v\\mid x\)=\\frac\{\\exp\(E\_\{v\}^\{\\top\}h\_\{c\}\(x\)\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\(E\_\{u\}^\{\\top\}h\_\{c\}\(x\)\)\}\.\(1\)Let𝒜⁡\(x\)\\mathcal\{A\}\(x\)be the valid answer\-label tokens\. Immediate scoring and free reasoning query two different conditional events\. The first reads the label distribution at the next position,

y^imm\(c\)=arg⁡maxa∈𝒜⁡\(x\)​pc​\(a∣x\),\\hat\{y\}\_\{\\mathrm\{imm\}\}^\{\(c\)\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\(x\)\}p\_\{c\}\(a\\mid x\),\(2\)while the second queries the multi\-token event of generating a rationale and then an answer,

y^free\(C\)=Extract⁡\(arg⁡max⁡∑t=1Ty1,…,yT⁡log⁡pθ​\(yt∣PC​\(x\),y<t\)\)\.\\hat\{y\}\_\{\\mathrm\{free\}\}^\{\(C\)\}=\\operatorname\{Extract\}\\\!\\left\(\\arg\\max\_\{y\_\{1\},\\dots,y\_\{T\}\}\\sum\_\{t=1\}^\{T\}\\log p\_\{\\theta\}\(y\_\{t\}\\mid P\_\{C\}\(x\),y\_\{<t\}\)\\right\)\.\(3\)CoT\-prefix scoring applies Eq\. \([2](https://arxiv.org/html/2609.29278#S2.E2)\) withc=Cc=Ceven thoughPCP\_\{C\}requests the event in Eq\. \([3](https://arxiv.org/html/2609.29278#S2.E3)\)\. We study this event mismatch rather than CoT generation itself\. Restricted\-token scores are a common task readout\[[12](https://arxiv.org/html/2609.29278#bib.bib3)\], but here they are read at a position primed for rationale generation\.

The probe replaces the label rows of the vocabulary projection with a learned matrixWc∈ℝ5×3584W\_\{c\}\\in\\mathbb\{R\}^\{5\\times 3584\}, where the five rows cover the maximum number of displayed choices and invalid rows are masked at scoring time,

y^probe\(c\)=arg⁡maxa∈𝒜⁡\(x\)⁡\(Wc​hc​\(x\)\)a\.\\hat\{y\}\_\{\\mathrm\{probe\}\}^\{\(c\)\}=\\arg\\max\_\{a\\in\\mathcal\{A\}\(x\)\}\(W\_\{c\}h\_\{c\}\(x\)\)\_\{a\}\.\(4\)For each condition, we extract the final\-layer state at the last input position and train a separate linear map from its 3,584 dimensions to five choice logits, with invalid choices masked and the VLM frozen\. Training uses cross\-entropy with AdamW, at most 50 epochs, and retains the best validation checkpoint\. We summarize recovery by

ΔC\\displaystyle\\Delta\_\{C\}=Acc⁡\(y^probe\(C\)\)−Acc⁡\(y^imm\(C\)\),\\displaystyle=\\operatorname\{Acc\}\(\\hat\{y\}\_\{\\mathrm\{probe\}\}^\{\(C\)\}\)\-\\operatorname\{Acc\}\(\\hat\{y\}\_\{\\mathrm\{imm\}\}^\{\(C\)\}\),\(5\)ΔD\\displaystyle\\Delta\_\{D\}=Acc⁡\(y^probe\(D\)\)−Acc⁡\(y^imm\(D\)\),\\displaystyle=\\operatorname\{Acc\}\(\\hat\{y\}\_\{\\mathrm\{probe\}\}^\{\(D\)\}\)\-\\operatorname\{Acc\}\(\\hat\{y\}\_\{\\mathrm\{imm\}\}^\{\(D\)\}\),and define the condition gap asΓ=ΔC−ΔD\\Gamma=\\Delta\_\{C\}\-\\Delta\_\{D\}\. A largeΓ\\Gammaindicates condition\-specific recovery by equal\-capacity readouts\. Probes of this kind are a standard test of linear accessibility\[[1](https://arxiv.org/html/2609.29278#bib.bib21)\]\. A supervised probe measures what can be read out of a state, not what the model itself uses\[[5](https://arxiv.org/html/2609.29278#bib.bib19),[4](https://arxiv.org/html/2609.29278#bib.bib20)\], and a largeΓ\\Gammadoes not imply that the two hidden states are identical\.

We use the public Qwen2\.5\-VL\-7B\-Instruct checkpoint in evaluation mode and all official ScienceQA test questions, including image\-present and text\-only items\. Qwen and LLaVA represent the LVLM family that connects pretrained visual representations with autoregressive language models\[[3](https://arxiv.org/html/2609.29278#bib.bib13),[9](https://arxiv.org/html/2609.29278#bib.bib12)\]\. Inputs share the chat template, preprocessing, question, hint, and choices, and only the terminal instruction differs\. Each probe has roughly eighteen thousand parameters, and both conditions use the same capacity and selection rule\.

We compute native accuracy as a micro\-average over items with invalid answer labels masked\. Free\-CoT extraction happens after generation, and outputs without a recoverable choice count as errors\. Each item keeps its natural number of choices, and we also include a full\-string control that scores option text instead of short labels\. The evaluation is locked as follows\. Probes are fit on the image\-present portion of the official ScienceQA training split, checkpoints are chosen on the corresponding validation portion, and all reported numbers come from the complete official test split, including text\-only items\. The test split is never used to fit or select the primary decoder\.

To test whether the failure survives changes in answer content, we permute the option contents with five pre\-fixed seeds and remap the gold answer\. Because letter labels are regenerated in display order, the first slot always remains A\. For seedss, letyi\(s\)y\_\{i\}^\{\(s\)\}be the remapped gold label\. We report

Ac\(s\)\\displaystyle A\_\{c\}^\{\(s\)\}=1N∑i=1N𝟏\[y^i,c\(s\)=yi\(s\)\],\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\[\\hat\{y\}\_\{i,c\}^\{\(s\)\}=y\_\{i\}^\{\(s\)\}\],\(6\)Bc\(s\)\\displaystyle B\_\{c\}^\{\(s\)\}=1N∑i=1N𝟏\[y^i,c\(s\)=A\],\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\[\\hat\{y\}\_\{i,c\}^\{\(s\)\}=\\mathrm\{A\}\],whereBc\(s\)B\_\{c\}^\{\(s\)\}is the coupled A/first\-slot rate\. A highBC\(s\)B\_\{C\}^\{\(s\)\}after content shuffling indicates a stable default toward this coupled label–position event\. This control alone cannot separate position bias from label\-token bias\. Separating them would require independently counterbalanced labels\.

## 3Results

We first report native accuracy under the two scoring conditions\. CoT\-prefix scoring reduces native accuracy by more than thirty points, from 80\.76% to 45\.48%, while condition\-matched probes recover nearly all of the lost accuracy from the same hidden states\. The probe reaches 84\.27% for Direct and 78\.94% for CoT\-prefix, so the native/probe gaps are 3\.51 and 33\.46 points respectively\. Across five option\-content permutations, Direct accuracy stays at80\.26±0\.1380\.26\\pm 0\.13% and CoT\-prefix accuracy at45\.78±0\.6645\.78\\pm 0\.66%, with A/first\-slot rates of51\.71±1\.0651\.71\\pm 1\.06% and93\.54±0\.4393\.54\\pm 0\.43%\. Under uniform permutation the remapped gold label falls in the first slot on 40\.0% of items, since 52\.5% of the split has two choices\. Shuffled CoT\-prefix accuracy therefore exceeds the pure\-first\-slot base rate by only 5\.8 points\. The default is induced by the condition, and it does not depend on the option contents\.

Three controls separate the interface mismatch from a genuine loss of reasoning ability, and the same pattern holds on the 2,017\-example image\-present subset\. Letting the model generate the requested chain and then parsing its final choice raises CoT accuracy to 75\.24% on all 4,241 items, with 215 parse failures counted as errors, while the A/first\-slot rate falls from 94\.08% to 42\.09%\. On the image\-present subset, Direct scoring reaches 83\.49%, CoT\-prefix falls to 45\.27%, and generation restores 80\.61%, recovering 35\.34 of the 38\.22 lost points despite 121 unparseable outputs\. Forced one\-token generation closely reproduces the native accuracies, showing that the failure is tied to the next\-token event rather than the generation API\. Conversely, scoring length\-normalized likelihoods of the full option strings reverses the ordering, 52\.32% for Direct and 55\.41% for CoT\-prefix\. Recovery also does not depend on one exact suffix: across eleven endings, generated\-answer accuracy ranges from 76\.75% to 87\.46%, while answer\-eliciting endings such as “Answer:” perform well under immediate scoring and continuation\-inducing phrases perform substantially worse\. The collapse is therefore specific to scoring short answer tokens immediately after a continuation request\.

To test whether the failure is localized to the readout, we retrain only the readout\. Rank\-32 LoRA adapters\[[7](https://arxiv.org/html/2609.29278#bib.bib4)\]trained on the vocabulary projection alone, using 3,000 examples, frozen transformer layers, ten epochs, and no validation selection, raise CoT\-prefix accuracy from 45\.48% to 70\.90%, while Direct stays flat at 80\.76% to 80\.60%\. This confirms that the failure is localized to the readout\. Table[1](https://arxiv.org/html/2609.29278#S3.T1)repeats the native comparison on further datasets and architectures, with no fitted readouts\. These include AI2D\[[8](https://arxiv.org/html/2609.29278#bib.bib6)\], a ten\-subject MMMU subset\[[19](https://arxiv.org/html/2609.29278#bib.bib5)\], MMBench\[[10](https://arxiv.org/html/2609.29278#bib.bib11)\], both Qwen sizes, and LLaVA\-1\.5\-7B\. The collapse is severe but not universal\. It reaches 54\.02 points on AI2D, yet reverses on MMBench\. On AI2D the prefix raises the A/first\-slot rate from 24\.19% to 94\.66% for Q7 and from 24\.61% to 66\.71% for Q3\. Among examples that flip from Direct\-correct to CoT\-prefix\-wrong, 99\.17% for Q7 and 86\.43% for Q3 land on A\.

Table 1:Cross\-dataset/model checks\. Q7/Q3 denote Qwen2\.5\-VL\-7B/3B, L7 denotes LLaVA\-1\.5\-7B, and SQA\* is the image\-present subset\. A/first is the CoT\-prefix first\-slot rate and is omitted when fixed letter labels are unavailable\.A further boundary condition comes from the number of displayed choices for Q3 on the ScienceQA image subset\. For two\-choice items, Direct and CoT\-prefix accuracy is 85\.1% and 58\.2%, and CoT\-prefix selects A 86\.3% of the time\. For three choices, the values are 76\.7% and 48\.3%, with 69\.9% A selection\. For four choices, the values are 80\.6% and 66\.4%, with 42\.1% A selection\. The coupled default weakens as more choices are displayed, and the accuracy gap narrows from 26\.9 to 14\.2 points\. The five\-choice slice contains only 38 items, too few to interpret\. Taken together, these results show that CoT\-prefix scoring systematically diverts the immediate readout toward a coupled default, and that the effect is strongest when few choices are displayed\.

## 4Mechanistic and Exploratory Diagnostics

Having established that the failure is localized to the readout, we now examine what the prefix changes inside the model\. The question is not only whether the answer is still present, but where it remains accessible and how the prefix redirects the readout away from it\. On a fixed 200\-item subset \(Fig\.[2](https://arxiv.org/html/2609.29278#S4.F2)\), we first measure the position\-conditioned probability shift

Ij\\displaystyle I\_\{j\}=\{i:ri​\(yi\)=j\},\\displaystyle=\\\{i:r\_\{i\}\(y\_\{i\}\)=j\\\},\(7\)δj\\displaystyle\\delta\_\{j\}=1\|Ij\|​∑i∈Ij\[pC​\(yi∣xi\)−pD​\(yi∣xi\)\],\\displaystyle=\\frac\{1\}\{\|I\_\{j\}\|\}\\sum\_\{i\\in I\_\{j\}\}\\left\[p\_\{C\}\(y\_\{i\}\\mid x\_\{i\}\)\-p\_\{D\}\(y\_\{i\}\\mid x\_\{i\}\)\\right\],wherejjindexes the displayed position of the gold label,pDp\_\{D\}andpCp\_\{C\}denote the probability assigned to the correct label under Direct and CoT\-prefix scoring, and the average is taken over items whose correct label appears at positionjj\. The statistic therefore isolates how much the suffix changes the correct\-label probability at each displayed position, without conflating items with different label placements\.

Direct scores concentrate near the correct label, whereas CoT\-prefix creates a second mode near zero\. The measuredδ1\\delta\_\{1\}is near zero, whileδj\\delta\_\{j\}is strongly negative for later positions, which means the suffix suppresses the correct label almost exclusively when that label is not in the first slot\. This matches the content\-shuffle result above and explains why the A/first\-slot rate rises so sharply under CoT\-prefix\. A representative item illustrates the effect\. Direct scoring assigns 0\.904 probability to the correct option C, whereas CoT\-prefix assigns 0\.980 to the A/first\-slot option\. Free reasoning and the condition\-matched probe both recover C, and the probe assigns it probability 0\.9996\. The requested reasoning succeeds even though the immediate label event fails, so the loss is confined to the readout rather than the representation\.

Table[2](https://arxiv.org/html/2609.29278#S4.T2)tests continuation strength directly on the full test split\. Four explicit step\-by\-step requests cause large drops and A/first\-slot concentration, while the minimal cue “Reasoning:” remains close to Direct\. The effect therefore tracks continuation strength rather than one exact string\. A phrase that clearly invites a long continuation suppresses the label readout, whereas a short cue that does not commit the model to generate much leaves the label distribution largely intact\.

Figure 2:Probability diagnostics on 200 fixed items\. CoT\-prefix preserves correct\-label probability in the first slot but suppresses later positions\.Figure 3:Layer\-wise diagnostics on 400 fixed items\. Probe accuracy rises late, where the Direct–CoT\-prefix gap also widens\.Table 2:Trigger comparison on the ScienceQA test split\.Δ\\Deltaacc is relative to Direct; probe columns report condition\-matched accuracy and A/first rates \(%\)\.Letℛ\\mathcal\{R\}be the fixed set of common reasoning\-continuation tokens, such as “First”, “Let”, and “The”\. The unrestricted\-vocabulary masses, not renormalized over the valid labels, are

M𝒜\(c\)​\(x\)=∑a∈𝒜⁡\(x\)pc​\(a∣x\),Mℛ\(c\)​\(x\)=∑v∈ℛpc​\(v∣x\)\.M\_\{\\mathcal\{A\}\}^\{\(c\)\}\(x\)=\\sum\_\{a\\in\\mathcal\{A\}\(x\)\}p\_\{c\}\(a\\mid x\),\\qquad M\_\{\\mathcal\{R\}\}^\{\(c\)\}\(x\)=\\sum\_\{v\\in\\mathcal\{R\}\}p\_\{c\}\(v\\mid x\)\.\(8\)Table[3](https://arxiv.org/html/2609.29278#S4.T3)separates two effects that would otherwise be conflated\. The first is expected once the requested output event changes\. Probability mass moves from answer labels to reasoning\-continuation tokens:M𝒜M\_\{\\mathcal\{A\}\}falls by about fourteen\-fold,MℛM\_\{\\mathcal\{R\}\}rises four\-fold, and the mean rank of the correct answer token worsens by more than twenty thousand positions\. This shift alone is not a failure; a model prompted to explain should prefer prose tokens over a bare label\. The second effect is the failure studied here\. Even after restricting attention to the valid label set, predictions concentrate on the coupled A/first\-slot label, and correct\-label probability is preserved mainly when the gold option occupies that slot \(Fig\.[2](https://arxiv.org/html/2609.29278#S4.F2)\)\. Thus the suffix does more than move probability mass out of the answer vocabulary\. It also changes the conditional distribution within the answer labels, which explains why simply renormalizing the valid choices does not recover the original decision\.

Table 3:Full\-vocabulary competition on the ScienceQA image subset \(N=2,017N=2\{,\}017\); Change compares CoT\-prefix with Direct\.To locate where the answer becomes decodable, we fit the same linear readout to the layer\-ℓ\\ellstateshc\(ℓ\)h\_\{c\}^\{\(\\ell\)\}and compute

y^i,c\(ℓ\)\\displaystyle\\hat\{y\}\_\{i,c\}^\{\(\\ell\)\}=arg⁡maxa∈𝒜⁡\(xi\)⁡\(Wc\(ℓ\)​hi,c\(ℓ\)\)a,\\displaystyle=\\arg\\max\_\{a\\in\\mathcal\{A\}\(x\_\{i\}\)\}\(W\_\{c\}^\{\(\\ell\)\}h\_\{i,c\}^\{\(\\ell\)\}\)\_\{a\},\(9\)Ac\(ℓ\)\\displaystyle A\_\{c\}^\{\(\\ell\)\}=1Nte∑i𝟏\[y^i,c\(ℓ\)=yi\],\\displaystyle=\\frac\{1\}\{N\_\{\\mathrm\{te\}\}\}\\sum\_\{i\}\\mathbf\{1\}\[\\hat\{y\}\_\{i,c\}^\{\(\\ell\)\}=y\_\{i\}\],G\(ℓ\)\\displaystyle G^\{\(\\ell\)\}=AD\(ℓ\)−AC\(ℓ\)\.\\displaystyle=A\_\{D\}^\{\(\\ell\)\}\-A\_\{C\}^\{\(\\ell\)\}\.In the fixed 400\-item run \(Fig\.[3](https://arxiv.org/html/2609.29278#S4.F3)\), bothAD\(ℓ\)A\_\{D\}^\{\(\\ell\)\}andAC\(ℓ\)A\_\{C\}^\{\(\\ell\)\}remain near chance through early and middle layers, then rise sharply\. The gapG\(ℓ\)G^\{\(\\ell\)\}widens only in the late layers, where answer information becomes linearly organized\. This layer profile is consistent with the view that answer identity is computed late in the network and that the readout depends on a late\-layer representation\. The prefix does not remove this representation, but it changes how the final position aggregates it\.

A shuffle control confirms that the probes read content rather than label frequencies\. Permuting hidden states across examples drops linear accuracy from 86\.25% to 33\.75%, near the empirical random\-choice baseline of 32\.65%, and drops the full decoder from 86\.25% to 38\.75%\. Probe recovery therefore depends on example\-specific state information\. Taken together, these diagnostics show that the answer remains linearly organized in the late layers, while the prefix redirects the immediate readout toward continuation tokens\. The information is still present, but the scoring interface no longer exposes it\. This is why a probe or a generation step can recover the answer while the native label logits cannot\.

## 5Conclusion

We identify CoT\-prefix scoring, where an evaluator appends a reasoning instruction but reads answer\-label logits before any rationale is generated\. On ScienceQA, this convention lowers Qwen2\.5\-VL\-7B accuracy by more than 35 points and drives predictions toward the first slot regardless of its content\. Condition\-matched probes recover most of the answer from the same hidden states, while free generation restores most of the direct\-answer performance\. Vocabulary, probability, and layer\-wise diagnostics locate the mismatch: the prefix redirects probability mass toward continuation tokens, yet answer identity remains linearly organized in late layers\. The failure therefore lies in the alignment between the requested continuation and the immediate scoring event, not simply in whether the representation contains the answer\. Its severity varies across datasets and models and is not universal\.

Practically, our results suggest a simple evaluation checklist\. If a prompt asks for a rationale, the scoring rule should read the token event that corresponds to that request \(e\.g\., the generated answer after the rationale, or a dedicated answer field\), rather than the immediate label logits at the first post\-prefix position\. When label\-logit scoring is required for compatibility, the prompt should avoid continuation\-seeking instructions, or explicitly constrain the next token to be an answer label\. Reporting should also specify the exact interface \(labels vs\. free\-form generation\), the label tokenization, and whether decoding is conditioned on an intermediate rationale, since these choices can dominate measured accuracy\.

Our study is limited to a set of multiple\-choice benchmarks and two VLM families\. Linear probes establish decodability rather than causal use by the native decoder, and content shuffling cannot fully separate position bias from label\-token bias without independently counterbalanced labels\. We also study only a limited set of reasoning instructions and answer interfaces\. These limits do not change the main evaluation lesson: when a prompt requests a rationale, the scorer should evaluate the event the model is actually asked to produce\. Future work should test broader model families, open\-ended tasks, and training or decoding interventions \(e\.g\., interface\-aware instruction tuning or constrained decoding\) that retain the benefits of reasoning prompts without breaking immediate answer readout\. More broadly, evaluation protocols should treat the prompt\-and\-scorer pair as part of the method specification, not an interchangeable implementation detail\.

## References

- \[1\]G\. Alain and Y\. Bengio\(2016\)Understanding intermediate layers using linear classifier probes\.arXiv preprint arXiv:1610\.01644\.Cited by:[§2](https://arxiv.org/html/2609.29278#S2.p3.4)\.
- \[2\]Md\. Atabuzzaman, A\. Asgarov, and C\. Thomas\(2025\)Benchmarking and mitigating MCQA selection bias of large vision\-language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 33548–33562\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1703)Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[3\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\)Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p2.1),[§2](https://arxiv.org/html/2609.29278#S2.p4.1)\.
- \[4\]Y\. Belinkov\(2022\)Probing classifiers: promises, shortcomings, and advances\.Computational Linguistics48\(1\),pp\. 207–219\.External Links:[Document](https://dx.doi.org/10.1162/coli%5Fa%5F00422)Cited by:[§2](https://arxiv.org/html/2609.29278#S2.p3.4)\.
- \[5\]J\. Hewitt and P\. Liang\(2019\)Designing and interpreting probes with control tasks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 2733–2743\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1275)Cited by:[§2](https://arxiv.org/html/2609.29278#S2.p3.4)\.
- \[6\]A\. Holtzman, P\. West, V\. Shwartz, Y\. Choi, and L\. Zettlemoyer\(2021\)Surface form competition: why the highest probability answer isn’t always right\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 7038–7051\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.564)Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[7\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§3](https://arxiv.org/html/2609.29278#S3.p3.1)\.
- \[8\]A\. Kembhavi, M\. Salvato, E\. Kolve, M\. Seo, H\. Hajishirzi, and A\. Farhadi\(2016\)A diagram is worth a dozen images\.InEuropean Conference on Computer Vision,pp\. 235–251\.Cited by:[§3](https://arxiv.org/html/2609.29278#S3.p3.1)\.
- \[9\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 34892–34916\.Cited by:[§2](https://arxiv.org/html/2609.29278#S2.p4.1)\.
- \[10\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. Lin\(2024\)MMBench: is your multi\-modal model an all\-around player?\.InEuropean Conference on Computer Vision,pp\. 216–233\.Cited by:[§3](https://arxiv.org/html/2609.29278#S3.p3.1)\.
- \[11\]P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan\(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 2507–2521\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p1.1),[§1](https://arxiv.org/html/2609.29278#S1.p2.1)\.
- \[12\]F\. Petroni, T\. Rocktäschel, S\. Riedel, P\. Lewis, A\. Bakhtin, Y\. Wu, and A\. Miller\(2019\)Language models as knowledge bases?\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 2463–2473\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1250)Cited by:[§2](https://arxiv.org/html/2609.29278#S2.p2.4)\.
- \[13\]F\. Rosenthal, S\. Schmidt, T\. Graf, T\. Bagodonat, S\. Günnemann, and L\. Schwinn\(2025\)Unexplored flaws in multiple\-choice vqa evaluations\.arXiv preprint arXiv:2511\.22341\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[14\]L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. Vosoughi\(2025\)Judging the judges: a systematic study of position bias in LLM\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,Mumbai, India,pp\. 292–314\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.ijcnlp-long.18)Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[15\]X\. Tian, S\. Zou, Z\. Yang, M\. He, F\. Waschkowski, L\. Wesemann, P\. Tu, and J\. Zhang\(2026\)More thought, less accuracy? on the dual nature of reasoning in vision\-language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2509\.25848Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p1.1)\.
- \[16\]X\. Tian, S\. Zou, Z\. Yang, and J\. Zhang\(2025\)Identifying and mitigating position bias of multi\-image vision\-language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10599–10609\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[17\]Z\. Wang, H\. Zhang, X\. Li, K\. Huang, C\. Han, S\. Ji, S\. M\. Kakade, H\. Peng, and H\. Ji\(2025\)Eliminating position bias of language models: a mechanistic approach\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[18\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p1.1)\.
- \[19\]X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun, C\. Wei, B\. Yu, R\. Yuan, R\. Sun, M\. Yin, B\. Zheng, Z\. Yang, Y\. Liu, W\. Huang, H\. Sun, Y\. Su, and W\. Chen\(2024\)MMMU: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9556–9567\.Cited by:[§3](https://arxiv.org/html/2609.29278#S3.p3.1)\.
- \[20\]Z\. Zhao, E\. Wallace, S\. Feng, D\. Klein, and S\. Singh\(2021\)Calibrate before use: improving few\-shot performance of language models\.InProceedings of the 38th International Conference on Machine Learning,Vol\.139,pp\. 12697–12706\.Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.
- \[21\]C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. Huang\(2024\)Large language models are not robust multiple choice selectors\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.29278#S1.p3.1)\.