Detecting Errors in Large Language Model Responses: The Incremental Predictive Value of Hidden States

# Detecting Errors in Large Language Model Responses:The Incremental Predictive Value of Hidden States
Source: [https://arxiv.org/html/2610.27455](https://arxiv.org/html/2610.27455)
\[ BoldFont=texgyretermes\-bold\.otf, ItalicFont=texgyretermes\-italic\.otf, BoldItalicFont=texgyretermes\-bolditalic\.otf\] \[ ItalicFont=lmmono10\-italic\.otf, BoldFont=lmmonolt10\-bold\.otf, BoldItalicFont=lmmonolt10\-boldoblique\.otf\]

Xiaojun MaXiaofei WangAffiliation:Beijing Institute of Computer Technology and ApplicationAffiliation:Beijing, ChinaEmail:[inobrain0\.0@gmail\.com](mailto:)

###### Abstract

Although large language models support a wide range of tasks, hallucinations remain a major obstacle to user trust\. When a model generates an answer, both its internal processing and final output vary with the question\. Internal states are often expected to offer richer signals for hallucination detection\. However, studies using these states typically report their standalone discrimination performance, while output\-side comparisons often use simple baselines with weaker modeling capacity\. We integrate output\-side information available at inference time into a combined baseline and compare detection performance before and after adding hidden states under identical data splits and evaluation rules\. Experiments cover open\-ended generation and constrained multiple\-choice tasks across different models\. Adding hidden states improves AUROC by approximately 0\.004 and 0\.039 on the two tasks, respectively\. Hidden states thus provide additional predictive information, but strong standalone detection performance can still yield only limited additional gains\.

## 1Introduction

Large language models \(LLMs\) have become widely used as interactive assistants\([Chatterji et al\., 2025](https://arxiv.org/html/2610.27455#bib.bib7)\)\. Users obtain knowledge and information through natural\-language questions, and retrieval\-based systems generate answers supported by web sources\([Nakano et al\., 2021](https://arxiv.org/html/2610.27455#bib.bib24)\)\. A central requirement is that these models answer accurately and reliably without producing false information\([Lin et al\., 2022](https://arxiv.org/html/2610.27455#bib.bib21);[Nakano et al\., 2021](https://arxiv.org/html/2610.27455#bib.bib24)\)\. Yet fluent responses can depart from facts or contradict the given context\. This phenomenon is commonly called hallucination\. For example, a model may fabricate a reference or add unsupported information when summarizing a document\. Fluency alone therefore provides an unreliable basis for assessing factuality, motivating hallucination detection\.

Prior work detects hallucinations using information available at the model’s output and hidden states recorded during generation\([Snyder et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib32);[Azaria and Mitchell, 2023](https://arxiv.org/html/2610.27455#bib.bib1)\)\. When decoding scores are accessible, logits and token probabilities can be retained at each generation step; statistics derived from them have been used to estimate uncertainty\([Snyder et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib32);[Vashurin et al\., 2025](https://arxiv.org/html/2610.27455#bib.bib38)\)\. Agreement among responses sampled for the same question also provides evidence of reliability\([Manakul et al\., 2023](https://arxiv.org/html/2610.27455#bib.bib22);[Kadavath et al\., 2022](https://arxiv.org/html/2610.27455#bib.bib18)\)\. Semantic entropy further accounts for differences in meaning to estimate uncertainty and detect some factual hallucinations\([Kuhn et al\., 2023](https://arxiv.org/html/2610.27455#bib.bib20);[Farquhar et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib12)\)\. These findings show that output\-side information available at inference time can help predict hallucinations\. Hidden states, in turn, encode internal representations during answer generation and have been used to distinguish true from false statements or detect factual errors in generated answers\([Azaria and Mitchell, 2023](https://arxiv.org/html/2610.27455#bib.bib1);[Snyder et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib32)\)\. However, probe performance depends on classifier capacity and control settings, so high predictive performance alone does not establish the role of an internal representation\([Hewitt and Liang, 2019](https://arxiv.org/html/2610.27455#bib.bib16);[Belinkov, 2022](https://arxiv.org/html/2610.27455#bib.bib2)\)\. Richer hidden representations need not yield larger improvements once output\-side information is used\. Noise may impede access to useful signals, while overlap with output\-side information means that some predictive capability is already available\. Thus, even when hidden states distinguish correct from incorrect answers well, their standalone performance does not reveal which capabilities remain useful beyond existing output\-side information\. This motivates our question: after using the available output\-side information, can hidden states further improve error detection, and how large is the additional gain?

We jointly model several types of output\-side information as a combined baseline, check whether it preserves the predictive performance of each component block, and compare detection performance before and after adding hidden states\. The difference measures their incremental predictive value relative to this baseline\. Experiments on open\-ended generation and constrained multiple\-choice tasks examine this value under different output conditions\. On open\-ended generation, Qwen’s standalone\-statement adaptation yields a gain of 0\.0043, while Llama gains 0\.0040 over the combined baseline\. On multiple\-choice questions, Llama retains a gain of 0\.0387 after jointly including the selected option text and the full response in the output baseline, with a 95% confidence interval of\[0\.0264,0\.0515\]\[0\.0264,0\.0515\]\. Hidden states can therefore further improve error detection, but strong standalone performance does not necessarily yield a large additional gain\. Standalone detection performance is consequently insufficient to establish their added value over output\-side information\.

Our contributions are:

1. 1\.We formulate the question of conditional gain from hidden states, shifting attention from their standalone detection performance to their added predictive value over available non\-activation information\.
2. 2\.We develop and apply an experimental and identification protocol fixed before analysis, using a combined baseline and blockwise comparisons to establish how incremental results should be interpreted\.
3. 3\.Cross\-model and readout comparisons show that hidden states provide gains beyond a combined output\-side baseline, while strong standalone performance need not translate into large additional improvements\.111[https://github\.com/in0brain/attention\_k](https://github.com/in0brain/attention_k)

## 2Related Work

[Chakraborty et al\. \(2025\)](https://arxiv.org/html/2610.27455#bib.bib6)classify hallucination detectors as black\-box, gray\-box, or white\-box according to their access to the generating model\. Black\-box methods use generated text; gray\-box methods also use decoding signals such as token probabilities and sequence likelihoods; white\-box methods access internal representations such as hidden states and attention\. Within approaches that do not rely on external knowledge bases or auxiliary evaluation models, information used by the first two categories can be grouped as output\-side information\. We therefore compare output\-side information with hidden states, focusing primarily on factual errors\.

##### Output\-side detection\.

Output\-side detectors assess reliability through decoding confidence and generation behavior\. Token probabilities, sequence likelihoods, and generation entropy capture uncertainty and have been used for confidence estimation and hallucination detection\([Kadavath et al\., 2022](https://arxiv.org/html/2610.27455#bib.bib18);[Fadeeva et al\., 2023](https://arxiv.org/html/2610.27455#bib.bib11);[Vashurin et al\., 2025](https://arxiv.org/html/2610.27455#bib.bib38)\)\. Signals from a single decoding pass provide limited evidence about stability across possible responses\. SelfCheckGPT addresses this by checking factual consistency between a target response and repeatedly sampled responses\([Manakul et al\., 2023](https://arxiv.org/html/2610.27455#bib.bib22)\)\. Since different phrasings may express the same meaning, semantic entropy groups semantically equivalent responses before measuring dispersion across meanings\([Farquhar et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib12)\)\. Beyond probabilities and consistency,[Kadavath et al\. \(2022\)](https://arxiv.org/html/2610.27455#bib.bib18)study models’ direct assessments of answer correctness, and[Pedapati et al\. \(2025\)](https://arxiv.org/html/2610.27455#bib.bib27)investigate confidence estimation under black\-box access\. These studies primarily compare output\-side measures, including response consistency and semantic uncertainty, and their detection performance\([Manakul et al\., 2023](https://arxiv.org/html/2610.27455#bib.bib22);[Farquhar et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib12);[Vashurin et al\., 2025](https://arxiv.org/html/2610.27455#bib.bib38)\)\. They do not directly estimate the conditional gain from hidden states over a baseline that jointly models multiple types of output information\.

##### Hidden\-state detection\.

SAPLMA trains a multilayer perceptron \(MLP\) on intermediate activations for true and false statements, demonstrating that hidden states can distinguish the two\([Azaria and Mitchell, 2023](https://arxiv.org/html/2610.27455#bib.bib1)\)\.[CH\-Wang et al\. \(2024\)](https://arxiv.org/html/2610.27455#bib.bib5)further compare probe performance across layers and readout methods\. Because supervised probes require labeled examples, later work explores unsupervised detection\. MIND exploits statistical patterns in hidden states, while HaloScope learns useful internal representations from unlabeled model generations\([Su et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib35);[Du et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib9)\)\. Other work extends consistency analysis to internal representations\. INSIDE measures reliability through differences among hidden representations from repeated generations\([Chen et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib8)\); DRIFT uses internal representational inconsistency to assess factuality\([Bhatnagar et al\., 2026](https://arxiv.org/html/2610.27455#bib.bib4)\); and semantic entropy probes predict semantic uncertainty directly from hidden states, reducing the computation required for repeated sampling\([Kossen et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib19)\)\.[Han et al\. \(2025\)](https://arxiv.org/html/2610.27455#bib.bib15)apply lightweight probes to long\-form factuality detection and compare their performance and computational cost with methods requiring repeated sampling\.[Orgad et al\. \(2025\)](https://arxiv.org/html/2610.27455#bib.bib26)examine cross\-dataset generalization and find limited additional generalization over an output\-logit baseline\. The former compares against repeated\-sampling detectors, and the latter controls for a single output\-side signal\. Neither design establishes how much predictive information hidden states add after multiple output\-side sources are jointly modeled\.

Some studies evaluate several detection signals within a common framework\.[Snyder et al\. \(2024\)](https://arxiv.org/html/2610.27455#bib.bib32)use a unified classifier to compare output probabilities, input attributions, self\-attention, and feed\-forward activations for factual error detection\. LLM\-Check uses an auxiliary language model’s hidden states, attention, and output probabilities to detect hallucinations in a single response\([Sriramanan et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib33)\)\. Probe performance nevertheless depends on classifier capacity, training, and controls\([Hewitt and Liang, 2019](https://arxiv.org/html/2610.27455#bib.bib16);[Voita and Titov, 2020](https://arxiv.org/html/2610.27455#bib.bib39);[Pimentel et al\., 2020](https://arxiv.org/html/2610.27455#bib.bib28);[Belinkov, 2022](https://arxiv.org/html/2610.27455#bib.bib2)\), and non\-target dataset cues can also produce high detection performance\([Hussain and Kantarcioglu, 2026](https://arxiv.org/html/2610.27455#bib.bib17)\)\. Strong performance from hidden states, alone or combined with other signals, is therefore insufficient to establish their added value\. We use a combined baseline of multiple output\-side sources to assess whether hidden states provide additional predictive gains\.

## 3Method

### 3\.1Overall Design and Conditional Gain

We evaluate the predictive value that hidden states add to existing output\-side information\. To account for the predictive capabilities of multiple output\-side sources, we construct a combined baseline and use blockwise comparisons to check whether it preserves each block’s performance\. We then hold the data splits, learning procedure, and output\-side prediction scores fixed, compare detection performance before and after adding hidden states, and check whether the fused model outperforms each block\. Because additional gains also depend on output\-side coverage and hidden\-state readout, we vary these settings separately to examine both dependencies\.

Standalone prediction from hidden states evaluates their detection capability; the performance difference between the combined baseline and the fused model evaluates their added value\.

LetY∈\{0,1\}Y\\in\\\{0,1\\\}denote the error label of a generated response, and letSSdenote the non\-activation information allowed by the protocol, including prompts, generated outputs, repeated\-generation statistics, and accessible decoding signals\. We use*output\-side information*as an umbrella term that also includes prompt\-side controls\. LetHHdenote hidden states extracted using a predetermined layer and pooling rule\.

We measure error prediction performance by the area under the receiver operating characteristic curve \(AUROC\)\. The combined baselineS∗S^\{\\ast\}is constructed fromSS, and adding the hidden\-state readout yields the fused modelS∗\+HS^\{\\ast\}\+H\. We define the conditional gain as

ΔH=AUROC⁡\(S∗\+H\)−AUROC⁡\(S∗\)\.\\Delta\_\{H\}=\\operatorname\{AUROC\}\(S^\{\\ast\}\+H\)\-\\operatorname\{AUROC\}\(S^\{\\ast\}\)\.\(1\)Under the current experimental setting,ΔH\>0\\Delta\_\{H\}\>0indicates that adding hidden states improves error detection over the combined output\-side baseline\.

[Table1](https://arxiv.org/html/2610.27455#S3.T1)summarizes the notation\.

Table 1:Main notation\.Both models use this conditional comparison;[Figure1](https://arxiv.org/html/2610.27455#S3.F1)illustrates the main procedure\.

Two tasks H1: open\-ended recall MCQ: multiple choice Sec\.[4\.1](https://arxiv.org/html/2610.27455#S4.SS1)Shared outer splits Prespecified groups Sec\.[4\.4](https://arxiv.org/html/2610.27455#S4.SS4)Output\-side informationSSB1B\_\{1\}Answer identityB2B\_\{2\}Length and formatB3B\_\{3\}Output textB4B\_\{4\}Scores and agreementB5B\_\{5\}Prompt contextHidden statesHH Fixed layer and pooling Sec\.[4\.2](https://arxiv.org/html/2610.27455#S4.SS2)Fit each block Select learners and parameters in training foldsHidden\-state readout Same splits and selection rulesShared output scores One column per block Cross\-fitted for trainingHidden\-state scores Additional input to fusionCombined baselineS∗S^\{\\ast\} Meta\-learner: five scores AUROC⁡\(S∗\)\\operatorname\{AUROC\}\(S^\{\\ast\}\)Fused modelS∗\+HS^\{\\ast\}\+H Meta\-learner: six scores AUROC⁡\(S∗\+H\)\\operatorname\{AUROC\}\(S^\{\\ast\}\{\+\}H\)Conditional gainΔH\\Delta\_\{H\} AUROC⁡\(S∗\+H\)−AUROC⁡\(S∗\)\\operatorname\{AUROC\}\(S^\{\\ast\}\{\+\}H\)\-\\operatorname\{AUROC\}\(S^\{\\ast\}\) Definition: Sec\.[3\.1](https://arxiv.org/html/2610.27455#S3.SS1) Results: Sec\.[5\.1](https://arxiv.org/html/2610.27455#S5.SS1)Baseline vs\. individual blocks:ΔS,k\\Delta\_\{S,k\} Check preservation of each block’s predictive performance Criteria: Sec\.[3\.3](https://arxiv.org/html/2610.27455#S3.SS3)Fusion vs\. individual blocks:ΔD,k\\Delta\_\{D,k\} Check whether fusion outperforms each block Criteria: Sec\.[3\.3](https://arxiv.org/html/2610.27455#S3.SS3)Factors affecting gainsOutput coverage; hidden\-state readout and representation \(Sec\.[4\.3](https://arxiv.org/html/2610.27455#S4.SS3)\) Numerical handling and grouping sensitivity \(Sec\.[4\.4](https://arxiv.org/html/2610.27455#S4.SS4)\)Inputs and splitsInformation sources\([3\.2](https://arxiv.org/html/2610.27455#S3.SS2),[4\.2](https://arxiv.org/html/2610.27455#S4.SS2)\)Within\-fold fitting\([3\.2](https://arxiv.org/html/2610.27455#S3.SS2)\)Shared prediction scores\([3\.2](https://arxiv.org/html/2610.27455#S3.SS2)\)Stacking and testingPrediction and evaluationBlockwise checks

Figure 1:Framework for evaluating conditional gain\.
### 3\.2Combined Baseline and Hidden\-State Fusion

##### Information blocks\.

Different output\-side sources offer different cues for error prediction\. To evaluate hidden states after jointly using these cues, the baseline contains five predefined blocks covering the answer, generation behavior, and prompt context\. Both tasks use the same information categories, with representations adapted to their output formats:

S∗=g⁡\(B1,B2,B3,B4,B5\)\.S^\{\\ast\}=g\(B\_\{1\},B\_\{2\},B\_\{3\},B\_\{4\},B\_\{5\}\)\.\(2\)[Table2](https://arxiv.org/html/2610.27455#S3.T2)specifies the blocks for each task\.

Table 2:Five information blocks in the main\-analysis baseline\.
##### Two\-level stacking\.

Because the blocks have different representations, we first fit a base learner to each block and then combine their prediction scores through two\-level stacking\([Wolpert, 1992](https://arxiv.org/html/2610.27455#bib.bib40)\)\. The baseline and fused model share the output\-side scores; the fused model receives only one additional column of hidden\-state scores\. This evaluates improvement over the same output\-side predictions\.

The meta\-learner is trained on cross\-fitted scores within each outer training fold, avoiding combinations learned from base learners’ in\-sample fitted scores\. At test time, the base learners score the shared outer test fold, and the meta\-learner produces the final predictions\. All preprocessing and hyperparameter selection are confined to training folds\([Varma and Simon, 2006](https://arxiv.org/html/2610.27455#bib.bib37)\)\. The*Within\-fold fitting*and*Stacking and testing*stages of[Figure1](https://arxiv.org/html/2610.27455#S3.F1)illustrate training and prediction\.

### 3\.3Blockwise Comparisons and Criteria

Joint modeling requires checking whether individual blocks’ predictive capabilities are preserved\. If the combined baseline performs worse than a single block, an improvement from fusion may still leave the fused model below that block\. We therefore assess baseline preservation and fused\-model improvement relative to each block separately\.

To check whether the baseline preserves each block’s performance, we first compute its difference from the single\-block predictorBkB\_\{k\}:

ΔS,k=AUROC⁡\(S∗\)−AUROC⁡\(Bk\)\.\\Delta\_\{S,k\}=\\operatorname\{AUROC\}\(S^\{\\ast\}\)\-\\operatorname\{AUROC\}\(B\_\{k\}\)\.\(3\)The non\-inferiority check allows an AUROC decrease ofεN​I=0\.02\\varepsilon\_\{NI\}=0\.02relative to each block and requires every one\-sided lower confidence bound forΔS,k\\Delta\_\{S,k\}to exceed−εN​I\-\\varepsilon\_\{NI\}\. This tolerance was fixed before the formal analysis first accessed the true labels\.

To determine whether adding hidden states yields performance above each block, we also compute

ΔD,k=AUROC⁡\(S∗\+H\)−AUROC⁡\(Bk\)\.\\Delta\_\{D,k\}=\\operatorname\{AUROC\}\(S^\{\\ast\}\+H\)\-\\operatorname\{AUROC\}\(B\_\{k\}\)\.\(4\)The direct blockwise comparison requires every one\-sided lower bound forΔD,k\\Delta\_\{D,k\}to exceed zero\. Passing all comparisons establishes that the fused model outperforms every block included in the comparison\.

Non\-inferiority allows a small baseline loss relative to individual blocks\. To assess whether the hidden\-state gain exceeds this allowed loss, we add a more conservative consistency check: the baseline must pass non\-inferiority, and the lower confidence bound forΔH\\Delta\_\{H\}must exceedεN​I\\varepsilon\_\{NI\}\.

## 4Experimental Setup

### 4\.1Tasks and Data

We examine open\-ended recall \(H1\) and constrained multiple\-choice questions \(MCQ\) to assess the added value of hidden states under different task settings\. H1 generates text containing identifiers; MCQ asks the model to select a given option and return its answer letter\.

Both tasks require explicit correctness criteria\. Cybersecurity provides standardized identifier systems and question\-answering datasets with reference answers\. We therefore construct H1 from MITRE ATT&CK and CWE entries\([Strom et al\., 2020](https://arxiv.org/html/2610.27455#bib.bib34);[MITRE, 2026](https://arxiv.org/html/2610.27455#bib.bib23)\)and use CyberMetric for MCQ\([Tihanyi et al\., 2024](https://arxiv.org/html/2610.27455#bib.bib36)\)\.

##### Open\-ended recall \(H1\)\.

The model generates an identifier from the name and description of a technique or weakness\. The task detects fabricated identifiers by checking their existence in a fixed knowledge base\. After excluding identifiers copied verbatim from the prompt, a response is labeled erroneous if any remaining ATT&CK or CWE identifier is absent from the knowledge base\. Each generation record is one analysis unit\.

##### Constrained multiple choice \(MCQ\)\.

The model selects an answer from the given options\. A selection that differs from the reference answer is labeled erroneous\. Each question is one analysis unit\.

For both models, only samples with valid labels and hidden states are analyzed; the two analysis pipelines are called the H1 and MCQ arms\. Qwen includes 8,664 H1 and 1,750 MCQ records; Llama includes 8,810 and 1,746, respectively\*\. H1 generations from the same prompt remain in the same group, while each MCQ question forms its own group\. Training and test sets are split by group, as shown in the*Inputs and splits*stage of[Figure1](https://arxiv.org/html/2610.27455#S3.F1)\.

### 4\.2Models and Training

We use Qwen2\.5\-7B\-Instruct\([Qwen Team, 2024](https://arxiv.org/html/2610.27455#bib.bib30)\)and Llama\-3\.1\-8B\-Instruct, both with a local 8\-bit quantized backend\. Each model uses the same learning procedure and sampling parameters across tasks\. To apply a common layer\-selection rule across architectures, we select the hidden\-state layer at a fixed fraction of total model depth, set in advance to0\.70\.7\. H1 averages hidden vectors over all eligible identifier tokens; MCQ uses the vector at the answer\-letter position\.

Text blocks use word\- and character\-level term frequency–inverse document frequency \(TF\-IDF\) features; dense blocks and hidden states are standardized\. Base learners and meta\-learners use L2\-regularized logistic regression\. Base\-learner regularization is selected by grouped cross\-validation within each outer training fold\. Feature extraction and predictor training correspond to*Information sources*and*Within\-fold fitting*in[Figure1](https://arxiv.org/html/2610.27455#S3.F1)\.

### 4\.3Output Coverage and Readout Comparisons

The output\-coverage comparison tests whether hidden states remain useful when more response text is incorporated\. Within each model, we hold the analyzable samples, splits, hidden states, and statistical protocol fixed and vary only the MCQ text blockB3B\_\{3\}\. The three conditions use the selected option text, the full model response, and their concatenation\. Text representations and predictors are refitted within training folds for each condition\.

Readout comparisons test whether different learners can make fuller use of hidden\-state signals\. Using Qwen’s existing vectors and output\-side scores, we compare L2 tuning ranges, MLPs, L1 logistic regression, and XGBoost\. H1 representation adaptation converts each distinct identifier into a standalone statement about its existence in the catalog, extracts new statement representations, trains on statement labels, and aggregates predictions to generation records using a prespecified maximum score\.

### 4\.4Evaluation and Sensitivity Comparisons

##### Primary evaluation\.

The combined baseline and fused model are evaluated on the same outer test folds to compare performance on identical samples\. Both models use the grouping described above, with five outer folds, three inner folds, and three repetitions\. We compute the out\-of\-fold AUROC difference within each repetition and then average the differences\.

We estimate 95% confidence intervals using a paired grouped bootstrap\([Field and Welsh, 2007](https://arxiv.org/html/2610.27455#bib.bib13)\), comparing predictors on the same resampled records\. Each MCQ question contributes one evaluated response, so sampling a question includes that response\. Each H1 prompt has multiple responses; sampling a prompt group includes all its responses to preserve their within\-group dependence\. A question or prompt group may be sampled more than once\.

##### Information and numerical controls\.

To examine dependence on generation and sampling signals, both models compare path A, which retains all five blocks, with path C, which removesB4B\_\{4\}from both the baseline and fused model\. Qwen’s H1B4B\_\{4\}has a non\-finite\-value handling issue\. Path Btherefore reconstructs the affected features while holding hidden states fixed to examine the effect of numerical handling\.

##### Grouping sensitivity\.

The main H1 split allows identifiers to be shared between training and test sets\. To examine dependence on such sharing, Qwen is also evaluated using groups defined by identifiers, vocabulary blocks, and joint prompt–fabricated\-identifier relationships\. The vocabulary\-block comparison redefines labels and hidden states by block; the other comparisons retain the original prediction task\.

## 5Results

### 5\.1Additional Predictive Gains from Hidden States

To assess whether hidden states improve error detection beyond output\-side information, we compare the combined baseline, standalone hidden\-state predictor, and fused model for Qwen and Llama\.[Table3](https://arxiv.org/html/2610.27455#S5.T3)reports these results alongside readout and representation comparisons\.

Table 3:Error detection for both models, with readout and representation comparisons\.ModelTaskS∗S^\{\\ast\}HHS∗\+HS^\{\\ast\}\+HΔH\\Delta\_\{H\}\(95% CI\)QwenH10\.98890\.94380\.9905\+0\.0016\+0\.0016\[\+0\.0006,\+0\.0028\]\[\+0\.0006,\+0\.0028\]MCQ0\.81980\.82920\.8487\+0\.0289\+0\.0289\[\+0\.0189,\+0\.0389\]\[\+0\.0189,\+0\.0389\]LlamaH10\.97120\.91480\.9752\+0\.0040\+0\.0040\[\+0\.0014,\+0\.0075\]\[\+0\.0014,\+0\.0075\]MCQ0\.74650\.79160\.8061\+0\.0596\+0\.0596\[\+0\.0447,\+0\.0751\]\[\+0\.0447,\+0\.0751\]TaskReadout or representationS∗S^\{\\ast\}HHS∗\+HS^\{\\ast\}\+HΔH\\Delta\_\{H\}\(95% CI\)H1L2, wide grid0\.98890\.94820\.9910\+0\.0021\+0\.0021\[\+0\.0008,\+0\.0037\]\[\+0\.0008,\+0\.0037\]MLP adaptation0\.93300\.9913\+0\.0024\+0\.0024\[\+0\.0009,\+0\.0043\]\[\+0\.0009,\+0\.0043\]L1 adaptation0\.93280\.9906\+0\.0017\+0\.0017\[\+0\.0006,\+0\.0029\]\[\+0\.0006,\+0\.0029\]XGBoost adaptation0\.95860\.9910\+0\.0021\+0\.0021\[\+0\.0009,\+0\.0035\]\[\+0\.0009,\+0\.0035\]L1, wide grid0\.91930\.9876−0\.0013\-0\.0013\[−0\.0039,\+0\.0011\]\[\-0\.0039,\+0\.0011\]Standalone statements0\.97800\.9931\+0\.0043\+0\.0043\[\+0\.0020,\+0\.0070\]\[\+0\.0020,\+0\.0070\]MCQL2, wide grid0\.81980\.82920\.8487\+0\.0289\+0\.0289\[\+0\.0189,\+0\.0389\]\[\+0\.0189,\+0\.0389\]MLP adaptation0\.78260\.8242\+0\.0045\+0\.0045\[−0\.0005,\+0\.0095\]\[\-0\.0005,\+0\.0095\]L1 adaptation0\.80300\.8445\+0\.0247\+0\.0247\[\+0\.0151,\+0\.0345\]\[\+0\.0151,\+0\.0345\]XGBoost adaptation0\.80850\.8426\+0\.0228\+0\.0228\[\+0\.0134,\+0\.0326\]\[\+0\.0134,\+0\.0326\]L1, wide grid0\.80300\.8445\+0\.0247\+0\.0247\[\+0\.0151,\+0\.0345\]\[\+0\.0151,\+0\.0345\]
Model comparisons use the original L2 grid; readout and representation comparisons use Qwen\. The baseline contains five output\-side blocks\. The first three numerical columns report AUROC averaged over three repetitions\. Readout comparisons share output\-side scores; statement adaptation aggregates statement predictions to generation records before evaluation\.

Both models improve on both tasks\. On H1, AUROC increases by0\.00160\.0016for Qwen and0\.00400\.0040for Llama; on MCQ, the gains are0\.02890\.0289and0\.05960\.0596, respectively\*\. All four main\-result confidence intervals lie above zero, showing that hidden states provide additional predictive information\.

Both models exhibit the same pattern: standalone hidden\-state detection is stronger on H1 than on MCQ, but its improvement over the output baseline is smaller\. H1 labels depend on identifier existence in a fixed knowledge base\. Using identifier identity alone, Qwen already achieves an AUROC of0\.98260\.9826\*, indicating strong predictive cues from identity\. The combined baseline approaches the AUROC ceiling, leaving little room for absolute improvement\. Strong standalone detection can therefore yield only limited additional gains\.

Both models pass baseline non\-inferiority on both tasks, and their fused models outperform every compared block\. The more conservative consistency check passes only for Llama’s main MCQ result: its gain interval has a lower bound of0\.04470\.0447, above0\.020\.02\. The other three settings do not reach this threshold\*\.

### 5\.2Effect of Output\-Side Information

To explain how conditional gain changes with the output baseline, we first compare retaining and removing generation and sampling signals\. For both models, removing these signals increases the conditional gain while lowering fused\-model performance\.[Figure2](https://arxiv.org/html/2610.27455#S5.F2)shows the Qwen comparison: removingB4B\_\{4\}reduces baseline performance more than fused\-model performance on both tasks, widening their gap\. The larger gain thus reflects a weaker output baseline, while final detection performance decreases\.

Figure 2:Absolute performance and conditional gain with and withoutB4B\_\{4\}\. Each panel has its own vertical scale; points and connecting lines show point estimates only\.In the numerical control, reconstructing the affected features yields Qwen gains of0\.00280\.0028on H1 and0\.02730\.0273on MCQ, preserving the positive direction and respective magnitudes\.

We next jointly include the selected option text and the full response in the MCQ baseline \([Table4](https://arxiv.org/html/2610.27455#S5.T4)\)\. Qwen typically returns an answer letter, whereas Llama produces explanatory text; joint encoding therefore adds more previously uncovered text for Llama\. Its baseline AUROC rises from0\.74650\.7465to0\.76890\.7689, while fused performance changes from0\.80610\.8061to0\.80750\.8075\. The larger baseline improvement reduces the hidden\-state gain from0\.05960\.0596to0\.03870\.0387\. Qwen’s gain changes only slightly, from0\.02890\.0289to0\.02770\.0277\.

Table 4:Output coverage of the MCQ text block\.Samples, splits, andHHare fixed within each model\. The first three numerical columns report AUROC; intervals are 95% confidence intervals\.

Both models retain positive gains after the full response is added, while improvements to the output baseline reduce the additional benefit from hidden states\. The gain thus depends both on the improvement contributed by hidden states and on how fully existing output\-side cues are used\.

### 5\.3Effect of Readout and Representation

We examine the readout and representation comparisons in[Table3](https://arxiv.org/html/2610.27455#S5.T3)to assess whether better access to hidden\-state signals increases their added value\. With alternative readouts and wider regularization searches, H1 gain estimates range from−0\.0013\-0\.0013to0\.00240\.0024\. Standalone\-statement adaptation raises hidden\-state AUROC from0\.94380\.9438to0\.97800\.9780, but increases the gain over the output baseline only from0\.00160\.0016to0\.00430\.0043\. The improvement in standalone detection does not translate into an equally large additional gain, as the strong output baseline continues to limit absolute headroom\.

MCQ gains are more sensitive to readout choice\. L2, L1, and XGBoost yield positive improvements, whereas the MLP gain is0\.00450\.0045with a confidence interval containing zero\. These differences occur under the same output baseline, indicating that the readout affects how effectively the additional predictive capability of hidden states is used\.

## 6Conclusion and Discussion

We evaluate the added predictive value of hidden states using a combined baseline of multiple output\-side sources and blockwise comparisons\. Hidden states provide predictive information beyond the available output\-side information, but strong standalone detection does not necessarily yield large additional gains\. On H1, the combined baseline already performs well, improvements from hidden states are small, and the tested readout and representation adaptations preserve this pattern\. On MCQ, hidden states continue to add predictive value over the combined output\-side baseline\. Standalone detection performance is therefore insufficient to establish their added value over existing output\-side information\.

Baseline construction and information coverage affect gain estimates\. Removing generation and sampling signals can increase the gain while reducing fused\-model performance\. Evaluations should report baseline performance, standalone hidden\-state performance, fused performance, and conditional gain together, and specify the output information being compared\. These conclusions concern predictive gains under this study’s tasks, models, backends, and learning procedures\.

## Limitations

This study has limitations in numerical integrity, parsing conventions, and probe training coverage\. The current data do not support an estimate of generalization to unseen identifiers, and the fused model’s advantage over individual blocks varies with grouping\*\. AUROC measures ranking performance for error prediction; practical utility also depends on detection thresholds, probability calibration, and readout cost\. Future work can assess whether predictive gains translate into practical benefits under explicitly specified conditions\.

## References

- Azaria and Mitchell \(2023\)Amos Azaria and Tom Mitchell\. 2023\.[The internal state of an LLM knows when it is lying](https://doi.org/10.18653/v1/2023.findings-emnlp.68)\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 967–976\.
- Belinkov \(2022\)Yonatan Belinkov\. 2022\.[Probing classifiers: Promises, shortcomings, and advances](https://doi.org/10.1162/coli_a_00422)\.*Computational Linguistics*, 48\(1\):207–219\.
- Berger and Hsu \(1996\)Roger L\. Berger and Jason C\. Hsu\. 1996\.[Bioequivalence trials, intersection–union tests and equivalence confidence sets](https://doi.org/10.1214/ss/1032280304)\.*Statistical Science*, 11\(4\):283–319\.
- Bhatnagar et al\. \(2026\)Rohan Bhatnagar, Youran Sun, Chi Andrew Zhang, Yixin Wen, and Haizhao Yang\. 2026\.[DRIFT: Detecting representational inconsistencies for factual truthfulness](https://doi.org/10.48550/arXiv.2601.14210)\.*arXiv preprint arXiv:2601\.14210*\.Preprint\.
- CH\-Wang et al\. \(2024\)Sky CH\-Wang, Benjamin Van Durme, Jason Eisner, and Chris Kedzie\. 2024\.[Do androids know they’re only dreaming of electric sheep?](https://doi.org/10.18653/v1/2024.findings-acl.260)In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 4401–4420\.
- Chakraborty et al\. \(2025\)Neeloy Chakraborty, Melkior Ornik, and Katherine Driggs\-Campbell\. 2025\.[Hallucination detection in foundation models for decision\-making: A flexible definition and review of the state of the art](https://doi.org/10.1145/3716846)\.*ACM Computing Surveys*, 57\(7\):1–35\.
- Chatterji et al\. \(2025\)Aaron Chatterji, Thomas Cunningham, David J\. Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman\. 2025\.[How people use ChatGPT](https://doi.org/10.3386/w34255)\.Working Paper 34255, National Bureau of Economic Research\.
- Chen et al\. \(2024\)Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao, Zhihang Fu, and Jieping Ye\. 2024\.[INSIDE: LLMs’ internal states retain the power of hallucination detection](https://proceedings.iclr.cc/paper_files/paper/2024/hash/0d1986a61e30e5fa408c81216a616e20-Abstract-Conference.html)\.In*International Conference on Learning Representations*\.
- Du et al\. \(2024\)Xuefeng Du, Chaowei Xiao, and Yixuan Li\. 2024\.[HaloScope: Harnessing unlabeled LLM generations for hallucination detection](https://doi.org/10.52202/079017-3270)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS 2024\)*\.
- Elangovan et al\. \(2021\)Aparna Elangovan, Jiayuan He, and Karin Verspoor\. 2021\.[Memorization vs\. generalization: Quantifying data leakage in NLP performance evaluation](https://doi.org/10.18653/v1/2021.eacl-main.113)\.In*Proceedings of EACL*, pages 1325–1335\.
- Fadeeva et al\. \(2023\)Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, Timothy Baldwin, and Artem Shelmanov\. 2023\.[LM\-Polygraph: Uncertainty estimation for language models](https://doi.org/10.18653/v1/2023.emnlp-demo.41)\.In*Proceedings of EMNLP 2023: System Demonstrations*, pages 446–461\.
- Farquhar et al\. \(2024\)Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal\. 2024\.[Detecting hallucinations in large language models using semantic entropy](https://doi.org/10.1038/s41586-024-07421-0)\.*Nature*, 630:625–630\.
- Field and Welsh \(2007\)C\. A\. Field and A\. H\. Welsh\. 2007\.[Bootstrapping clustered data](https://doi.org/10.1111/j.1467-9868.2007.00593.x)\.*Journal of the Royal Statistical Society: Series B \(Statistical Methodology\)*, 69\(3\):369–390\.
- Gururangan et al\. \(2018\)Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R\. Bowman, and Noah A\. Smith\. 2018\.[Annotation artifacts in natural language inference data](https://doi.org/10.18653/v1/N18-2017)\.In*Proceedings of NAACL\-HLT*, pages 107–112\.
- Han et al\. \(2025\)Jiatong Han, Neil Band, Muhammed Razzak, Jannik Kossen, Tim G\. J\. Rudner, and Yarin Gal\. 2025\.[Simple factuality probes detect hallucinations in long\-form natural language generation](https://doi.org/10.18653/v1/2025.findings-emnlp.880)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 16209–16226\.
- Hewitt and Liang \(2019\)John Hewitt and Percy Liang\. 2019\.[Designing and interpreting probes with control tasks](https://doi.org/10.18653/v1/D19-1275)\.In*Proceedings of EMNLP\-IJCNLP*, pages 2733–2743\.
- Hussain and Kantarcioglu \(2026\)Khizar Hussain and Murat Kantarcioglu\. 2026\.[PARALLAX: Separating genuine hallucination detection from benchmark construction artifacts](https://doi.org/10.48550/arXiv.2605.17028)\.*arXiv preprint arXiv:2605\.17028*\.Preprint\.
- Kadavath et al\. \(2022\)Saurav Kadavath, Tom Conerly, Amanda Askell, et al\. 2022\.[Language models \(mostly\) know what they know](https://doi.org/10.48550/arXiv.2207.05221)\.*arXiv preprint arXiv:2207\.05221*\.
- Kossen et al\. \(2024\)Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal\. 2024\.[Semantic entropy probes: Robust and cheap hallucination detection in LLMs](https://doi.org/10.48550/arXiv.2406.15927)\.*arXiv preprint arXiv:2406\.15927*\.
- Kuhn et al\. \(2023\)Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar\. 2023\.[Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation](https://openreview.net/forum?id=VD-AYtP0dve)\.In*International Conference on Learning Representations*\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.[TruthfulQA: Measuring how models mimic human falsehoods](https://doi.org/10.18653/v1/2022.acl-long.229)\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3214–3252\.
- Manakul et al\. \(2023\)Potsawee Manakul, Adian Liusie, and Mark J\. F\. Gales\. 2023\.[SelfCheckGPT: Zero\-resource black\-box hallucination detection for generative large language models](https://doi.org/10.18653/v1/2023.emnlp-main.557)\.In*Proceedings of EMNLP*, pages 9004–9017\.
- MITRE \(2026\)MITRE\. 2026\.[Common weakness enumeration](https://cwe.mitre.org/)\.Accessed 2026\-07\-30\.
- Nakano et al\. \(2021\)Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman\. 2021\.[WebGPT: Browser\-assisted question\-answering with human feedback](https://doi.org/10.48550/arXiv.2112.09332)\.*arXiv preprint arXiv:2112\.09332*\.
- Nosek et al\. \(2018\)Brian A\. Nosek, Charles R\. Ebersole, Alexander C\. DeHaven, and David T\. Mellor\. 2018\.[The preregistration revolution](https://doi.org/10.1073/pnas.1708274114)\.*Proceedings of the National Academy of Sciences*, 115\(11\):2600–2606\.
- Orgad et al\. \(2025\)Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov\. 2025\.[LLMs know more than they show: On the intrinsic representation of LLM hallucinations](https://proceedings.iclr.cc/paper_files/paper/2025/hash/a712d461e57201efe35d429a6f1731c1-Abstract-Conference.html)\.In*International Conference on Learning Representations*\.
- Pedapati et al\. \(2025\)Tejaswini Pedapati, Amit Dhurandhar, Soumya Ghosh, Soham Dan, and Prasanna Sattigeri\. 2025\.[Large language model confidence estimation via black\-box access](https://openreview.net/forum?id=WrWYChkyRI)\.*Transactions on Machine Learning Research*\.
- Pimentel et al\. \(2020\)Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell\. 2020\.[Information\-theoretic probing for linguistic structure](https://doi.org/10.18653/v1/2020.acl-main.420)\.In*Proceedings of ACL*, pages 4609–4622\.
- Poliak et al\. \(2018\)Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme\. 2018\.[Hypothesis only baselines in natural language inference](https://doi.org/10.18653/v1/S18-2023)\.In*Proceedings of \*SEM*, pages 180–191\.
- Qwen Team \(2024\)Qwen Team\. 2024\.[Qwen2\.5 technical report](https://doi.org/10.48550/arXiv.2412.15115)\.Technical Report arXiv:2412\.15115, Alibaba Cloud\.
- Roberts et al\. \(2017\)David R\. Roberts, Volker Bahn, Simone Ciuti, Mark S\. Boyce, Jane Elith, Gurutzeta Guillera\-Arroita, Severin Hauenstein, José J\. Lahoz\-Monfort, Boris Schröder, Wilfried Thuiller, David I\. Warton, Brendan A\. Wintle, Florian Hartig, and Carsten F\. Dormann\. 2017\.[Cross\-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure](https://doi.org/10.1111/ecog.02881)\.*Ecography*, 40\(8\):913–929\.
- Snyder et al\. \(2024\)Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar\. 2024\.[On early detection of hallucinations in factual question answering](https://doi.org/10.1145/3637528.3671796)\.In*Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD ’24\)*, pages 2721–2732\.
- Sriramanan et al\. \(2024\)Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi\. 2024\.[LLM\-Check: Investigating detection of hallucinations in large language models](https://doi.org/10.52202/079017-1077)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS 2024\)*\.
- Strom et al\. \(2020\)Blake Strom, Andy Applebaum, Douglas P\. Miller, Kathryn Nickels, Adam Pennington, and Cody Thomas\. 2020\.[MITRE ATT&CK: Design and philosophy](https://www.mitre.org/news-insights/publication/mitre-attck-design-and-philosophy)\.Technical report, MITRE\.
- Su et al\. \(2024\)Weihang Su, Changyue Wang, Qingyao Ai, Yiran Hu, Zhijing Wu, Yujia Zhou, and Yiqun Liu\. 2024\.[Unsupervised real\-time hallucination detection based on the internal states of large language models](https://doi.org/10.18653/v1/2024.findings-acl.854)\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 14379–14391\.
- Tihanyi et al\. \(2024\)Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah\. 2024\.[CyberMetric: A benchmark dataset based on retrieval\-augmented generation for evaluating LLMs in cybersecurity knowledge](https://doi.org/10.48550/arXiv.2402.07688)\.*arXiv preprint arXiv:2402\.07688*\.
- Varma and Simon \(2006\)Sudhir Varma and Richard Simon\. 2006\.[Bias in error estimation when using cross\-validation for model selection](https://doi.org/10.1186/1471-2105-7-91)\.*BMC Bioinformatics*, 7:91\.
- Vashurin et al\. \(2025\)Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, Maxim Panov, and Artem Shelmanov\. 2025\.[Benchmarking uncertainty quantification methods for large language models with LM\-Polygraph](https://doi.org/10.1162/tacl_a_00737)\.*Transactions of the Association for Computational Linguistics*, 13:220–248\.
- Voita and Titov \(2020\)Elena Voita and Ivan Titov\. 2020\.[Information\-theoretic probing with minimum description length](https://doi.org/10.18653/v1/2020.emnlp-main.14)\.In*Proceedings of EMNLP*, pages 183–196\.
- Wolpert \(1992\)David H\. Wolpert\. 1992\.[Stacked generalization](https://doi.org/10.1016/S0893-6080(05)80023-1)\.*Neural Networks*, 5\(2\):241–259\.

## Appendix AMethodological Details

### A\.1Five Non\-Activation Blocks in Each Arm

The table below specifies the fields and dimensions of the five output\-side blocks for H1 and MCQ\.

Table 5:Implementation ofB1B\_\{1\}–B5B\_\{5\}in H1 and MCQ\.B1B\_\{1\}–B5B\_\{5\}form the combined baselineS∗S^\{\\ast\}; adding the hidden\-state block yieldsS∗\+HS^\{\\ast\}\+H\. Path CremovesB4B\_\{4\}from both arms, so its baseline uses the remaining four blocks and its fused model addsHHto those blocks\.

### A\.2Within\-Fold Preprocessing and Nested Stacking

The baseline and fused model share within\-fold preprocessing and base\-learner scores\. We detail the feature transformations, cross\-fitting procedure, and learner settings below\.

Text blocks use word 1–2\-gram and word\-boundary character 3–5\-gram TF\-IDF features\. Both vocabularies usemin\_df=2, at most 50,000 features each, and sublinear term frequency\. The two representations are concatenated and each row is L2\-normalized\. Vocabularies, inverse document frequencies, and all transformations are fitted only on the current training fold and then applied to the corresponding outer test fold\.

Dense blocks are standardized using dimension\-wise means and standard deviations from the training fold, replacing zero standard deviations with 1\. The standardized matrix is divided byd\\sqrt\{d\}, whereddis the block dimension, to give dense blocks a scale comparable to row\-normalized text features\. Test folds do not contribute to means, standard deviations, vocabularies, or hyperparameter estimates\. Hidden states use the same dense preprocessing\.

S∗S^\{\\ast\}stacks five blocks, andS∗\+HS^\{\\ast\}\+Hstacks six\. Within each outer training fold, a base learner is fitted independently to each block\. Cross\-fitted training scores are used to train the meta\-learner\. The base learners are then refitted on the complete outer training fold and score the outer test fold; the meta\-learner produces final predictions from these scores\. Both models share the same base\-learner cache\. Within a repetition, they differ only in whether the meta\-learner receives the column ofHHscores\.[Table6](https://arxiv.org/html/2610.27455#A1.T6)lists fixed learning and resampling settings\.

Table 6:Prespecified learning and resampling parameters\.
### A\.3M1b, iv\-a, iv\-b, and Task\-Construct Checks

The protocol denotes baseline non\-inferiority, direct blockwise comparison, and the consistency check as M1b, iv\-a, and iv\-b, respectively\. It also includes a task\-construct check for identity and format cues\.

For each blockBkB\_\{k\}, define

ΔS,k\\displaystyle\\Delta\_\{S,k\}=AUROC⁡\(S∗\)−AUROC⁡\(Bk\),\\displaystyle=\\operatorname\{AUROC\}\(S^\{\\ast\}\)\-\\operatorname\{AUROC\}\(B\_\{k\}\),\(5\)ΔD,k\\displaystyle\\Delta\_\{D,k\}=AUROC⁡\(S∗\+H\)−AUROC⁡\(Bk\)\.\\displaystyle=\\operatorname\{AUROC\}\(S^\{\\ast\}\+H\)\-\\operatorname\{AUROC\}\(B\_\{k\}\)\.\(6\)The non\-inferiority tolerance is fixed atεN​I=0\.02\\varepsilon\_\{NI\}=0\.02\. This operational threshold specifies the allowed baseline loss and has not been externally validated\. M1b checks whether each marginal one\-sided 95% lower bound forΔS,k\\Delta\_\{S,k\}exceeds−εN​I\-\\varepsilon\_\{NI\}and combines the checks through an intersection–union test\([Berger and Hsu, 1996](https://arxiv.org/html/2610.27455#bib.bib3)\)\. The primary procedure uses no multiplicity correction\. Simultaneous lower bounds based on the maximum statistic are reported descriptively as secondary results; their nominal coverage relies on asymptotic approximation\. M1b is determined by the intersection–union rule applied to the marginal checks\.

iv\-a checks whether every marginal one\-sided lower bound forΔD,k\\Delta\_\{D,k\}exceeds zero, assessing whether the fused model outperforms every compared block\. The more conservative iv\-b check requires M1b to pass and the lower confidence bound forΔH\\Delta\_\{H\}to exceedεN​I\\varepsilon\_\{NI\}\. The two checks are reported separately under their respective conditions\.

Within a prespecified branch, the fused model is judged superior to all compared blocks only if every direct blockwise comparison passes\. This is an intersection–union test\([Berger and Hsu, 1996](https://arxiv.org/html/2610.27455#bib.bib3)\): if any component null hypothesis holds, the probability of incorrectly passing all comparisons is no greater than that component test’s significance level\. Thus, valid component tests require no further adjustment for the joint decision within a branch\. Reported bounds are marginal bounds for individual blockwise comparisons\. The six branches are reported separately without cross\-branch multiplicity correction, retaining the prespecified primary path\.

The task\-construct check compares the full output\-text blockB3B\_\{3\}with identity and format blocksB1B\_\{1\}andB2B\_\{2\}:

Δtask,k\\displaystyle\\Delta\_\{\\mathrm\{task\},k\}=AUROC⁡\(B3\)−AUROC⁡\(Bk\),\\displaystyle=\\operatorname\{AUROC\}\(B\_\{3\}\)\-\\operatorname\{AUROC\}\(B\_\{k\}\),\(7\)k∈\{1,2\}\.\\displaystyle k\\in\\\{1,2\\\}\.The primary analysis uses marginal one\-sided upper bounds for conservative screening\. An upper bound below0\.020\.02indicates that the full\-text improvement over identity or format cues does not reach the prespecified threshold\. Results from triggered branches describe predictive performance under the current label definition\.

## Appendix BData and Experimental Details

### B\.1Model, Hidden States, and Eligibility

We specify Qwen’s model, hidden\-state extraction, sample eligibility, and final analysis populations below\.

Both arms of the Qwen confirmatory analysis use Qwen2\.5\-7B\-Instruct with a local 8\-bit quantized backend\. The model has 28 decoder blocks and a hidden dimension of 3,584\. Relative depth0\.70\.7selects the output of block 19 \(zero\-indexed\), corresponding tohidden\_states\[20\]in the model interface\. Model, tokenizer, extraction layer, temperature, nucleus\-sampling settings, and random seeds are identical across arms\. Maximum generation lengths were set by task to 384 tokens for H1 and 16 for MCQ\.

H1 enters hidden\-state analysis only when a response contains at least one ATT&CK or CWE identifier not copied from the prompt\. The vectorHHaverages target\-layer states over all eligible identifier tokens, counting each token position once\. MCQ is eligible only when the answer can be normalized to an option in A–D;HHis the state of the single answer\-letter token\. Both arms store float32 vectors of shape\(3584,\)\(3584,\)\. H1 has 9,264 final records, of which 8,664 are eligible; MCQ has 1,751 final records, of which 1,750 are eligible\.[Table7](https://arxiv.org/html/2610.27455#A2.T7)provides the full accounting\.

The 8,664 analyzable H1 records come from 1,508 prompt groups, 345 of which contain at least one error\. Another 600 records remain in the final population but are ineligible: 477 only copy prompt identifiers, 122 contain no identifier, and 1 contains only a CVE identifier\. These lack the primary identifier label and corresponding hidden states\. The nominal MCQ population contains 1,760 questions: 9 are excluded as whole groups after runtime technical failures and 1 is lost during normalization or parsing, leaving 1,750 for analysis\. This arm generated 10,520 raw units, 10,506 belonging to retained questions and 14 failed attempts kept only for auditing\. Excluded questions are not replaced, and missing hidden states are not imputed\. The analyzable populations contain 8,664 and 1,750 hidden\-state vectors, respectively\.

Table 7:Sample accounting for Qwen\.GGis the number of resampling groups: prompts for H1 and questions for MCQ\. In H1, 345 prompt groups contain at least one error\.

### B\.2Llama Comparability and Sample Accounting

The cross\-model Llama comparison is non\-confirmatory and uses the same type of local 8\-bit backend to estimate paths A and C on both tasks\. Its MCQ generation budget is 512 tokens, compared with 16 for Qwen\. Llama uses a corrected parser that remained unchanged during formal production\. Gains are estimated separately under each model’s generation budget, parser version, and analyzable population\.

Both models share the same finalized H1 list of 9,264 generation records\. Qwen’s analyzable population has 8,664 records, 693 positives, and 1,508 groups; Llama’s has 8,810 records, 339 positives, and 1,516 groups\. The intersection contains 8,450 records\. Qwen excludes 477 prompt\-copying responses, 122 with no identifier, and 1 containing only CVE; Llama excludes 376 prompt\-copying responses and 78 with no identifier\.

Qwen’s finalized MCQ list contains 1,751 questions after 9 runtime technical exclusions\. Excluding 1 failed greedy\-response parse leaves 1,750 analyzable questions and 318 positives\. Llama finalizes all 1,760 questions without these technical exclusions; removing 14 failed greedy\-response parses leaves 1,746 questions and 328 positives\. The analyzable intersection contains 1,737 questions\. These intersections are used only to audit comparability; no models were retrained on common\-sample subsets\.

Llama did not run tail\-reconstruction path B or vocabulary\-block analysis E3, and itsB4B\_\{4\}numerical integrity remains unaudited\. The initial A/C run retained summaries without complete per\-sample predictions\. The joint full\-response MCQ condition was later rerun in a separate directory, retaining eight columns of out\-of\-fold predictions for three repetitions, fold mappings, and per\-fold audits\. Its summary matched the initial run\.

### B\.3MCQ Output Coverage and Parsing Denominators

The MCQ output\-coverage comparison is non\-confirmatory\. The originalB3B\_\{3\}uses selected option text and does not directly include the full model response\. Median analyzable response length is 1 character for Qwen and 886 for Llama, while the originalB3B\_\{3\}has a median length of approximately 32 characters for both\. The comparison holds samples and splits fixed within each model, changes only the text source, and refits the text block and fused model\. Llama’s joint\-condition interval was reproduced from saved predictions under the original resampling plan\. This check uses the same statistical library and verifies the computation from saved predictions to metrics and intervals\. Intervals for differences between conditions were not computed\.

Sampling\-consistency features retain successfully parsed labels from the six responses per question and use the number of successful parses as the denominator\. Among Llama’s 10,560 raw responses, 168 fail parsing: 14 greedy responses and 154 sampled responses\. The numbers of questions with 6, 5, 4, and 3 successful parses are 1,618, 120, 18, and 4, respectively\. After excluding greedy failures, 128 of the 1,746 analysis records use a denominator below 6\. None of Qwen’s 1,750 analysis records has this issue\.

Both consistency features can be recomputed record by record from the raw data under the current rules and match their stored values\. However, the original protocol fixed six responses per question without specifying how parsing failures affect the denominator\. Using successful parses is a deterministic implementation choice\. The effect of alternative denominators on predictive performance and hidden\-state gain has not been measured\.

### B\.4Readouts, Training Budgets, and Scoring

The readout comparisons and standalone\-statement adaptation are non\-confirmatory\. The first five readout conditions were fixed before that run; the wide\-grid L1 condition was added after the first run, and statement adaptation was conducted separately\.

To examine dependence on hidden\-state readout, these analyses reuse Qwen’s existing vectors, samples, and outer splits\. The original L2 grid isC∈\{0\.01,0\.1,1,10\}C\\in\\\{0\.01,0\.1,1,10\\\}; the wide grid has eight logarithmically spaced values from10−310^\{\-3\}to10410^\{4\}\. The original L1 candidates are\{0\.01,0\.1,0\.5,1,10\}\\\{0\.01,0\.1,0\.5,1,10\\\}, with100,1000,10000100,1000,10000added after the initial run\. XGBoost combines depths\{3,6\}\\\{3,6\\\}with\{200,1000\}\\\{200,1000\\\}trees at learning rate0\.10\.1\. All candidate selection is confined to training folds\.

Following the SAPLMA architecture, the MLP has three hidden layers of widths256,128,64256,128,64, with ReLU and Adam\([Azaria and Mitchell, 2023](https://arxiv.org/html/2610.27455#bib.bib1)\)\. Candidate epoch limits are\{5,50\}\\\{5,50\\\}and regularization coefficients are\{10−5,10−3,10−1\}\\\{10^\{\-5\},10^\{\-3\},10^\{\-1\}\\\}\. L1 and XGBoost reuse classifier construction from the authors’ implementation of[Han et al\. \(2025\)](https://arxiv.org/html/2610.27455#bib.bib15), adapted to this study’s single\-layer input, standardization, and grouped evaluation\.

Across the six readout conditions, logistic regression outputs decision scores directly\. MLP and XGBoost probabilities for the positive class are clipped to\[10−6,1−10−6\]\[10^\{\-6\},1\-10^\{\-6\}\]and converted to log\-odds before fusion\. This may create ties among extreme scores; the fusion layer receives the clipped and transformed scores\. The full combination of raw activations, complete author\-specified settings, and native scoring was not evaluated\.

For H1, the original L2 grid selects its upper bound in all 60 selections; the wider grid selectsC=100C=100in 54\. Widening L2 changes neither the selected parameters nor predictions on MCQ\. Wide\-grid L1 selects largerCCvalues at a few H1 stages but reduces the overall gain estimate\.

For final outer\-fold MLP fits, H1 selects the 50\-epoch limit in 15/15 cases; MCQ selects 5 epochs in 12/15 and 50 in 3/15\. These are selected iteration limits; complete final\-fit iteration logs were not retained\. Among 180 validation records for 50\-epoch candidates, 165 on H1 and 67 on MCQ contain at least one inner fit that does not satisfy the convergence criterion\. Counts are at the candidate\-validation\-record level\. MLP conclusions apply to this training budget and scoring convention\.

### B\.5Standalone\-Statement Adaptation for H1

This adaptation follows the use of final\-token representations of standalone statements\. Each distinct primary identifier is converted into a fixed\-template statement about its existence in the catalog, with an existence label from the fixed catalog\. There are 10,168 statement records, 870 distinct texts, and 8,664 original H1 evaluation records\. Encoding deduplicates identical texts, but training keeps each statement record as a separate row without combining weights\.

Outer splits are first materialized on the original generation records; statements inherit their parent record’s fold and prompt group\. On the training side, statements are cross\-fitted within inner folds, and the prespecified maximum raw score aggregates them back to generation records for fusion training\. Test predictions use the same aggregation\. The maximum raw score serves as a record\-level feature for the fusion layer\. Hyperparameters are selected using AUROC after record\-level aggregation\. This condition changes representation, training unit, and aggregation together\.

Across the three repetitions, approximately93\.4%93\.4\\%–93\.8%93\.8\\%of test statement records have identical text in the corresponding training fold\. Results therefore describe the current prompt\-grouped population, which allows repeated identifiers\. All 60/60 selections reach the original grid’s upper bound; other tuning ranges were not evaluated\. Statement\-level raw scores, fusion\-training columns, and some final\-fit logs were not retained\.

All saved target\-layer vectors and upstream checks are finite, but all 870/870 distinct texts produce non\-finite values in the returned final\-layer states and output scores\. Hooks directly locate the anomaly within the final decoder block for two inputs; equivalent internal localization was not completed for the remaining texts\.

### B\.6Repetition\-Averaged Estimand and Uncertainty

Performance differences are averaged over three repetitions, with intervals estimated by paired grouped bootstrap\. Numerical error due to finite resampling is estimated separately\.

Each repetition produces out\-of\-fold scores forS∗S^\{\\ast\},HH, andS∗\+HS^\{\\ast\}\+H, with AUROC first calculated within that repetition\. The primary estimand is the mean AUROC difference across three repetitions:

ΔH\\displaystyle\\Delta\_\{H\}=13​∑r=13ΔH,r,\\displaystyle=\\frac\{1\}\{3\}\\sum\_\{r=1\}^\{3\}\\Delta\_\{H,r\},\(8\)ΔH,r\\displaystyle\\Delta\_\{H,r\}=AUROCr⁡\(S∗\+H\)−AUROCr⁡\(S∗\)\.\\displaystyle=\\operatorname\{AUROC\}\_\{r\}\(S^\{\\ast\}\+H\)\-\\operatorname\{AUROC\}\_\{r\}\(S^\{\\ast\}\)\.
Intervals use a grouped bootstrap\. The same group sample is shared across all three repetitions to preserve pairing\. H1 resamples 1,508 prompt groups; MCQ resamples 1,750 questions\. Every E1 branch has 20,000 valid bootstrap samples, with none discarded for containing only one class\. Two\-sided 95% intervals use the 2\.5th and 97\.5th percentiles of the pooled bootstrap distribution\. Marginal one\-sided lower and upper bounds use the 5th and 95th percentiles\.

Each bootstrap sample draws the original number of groups with replacement and concatenates all records in the selected groups, retaining repetitions of groups\. H1’s 693 error records are sampled with their groups, preserving within\-group label composition\. The same resampled records are used for all predictors and repetitions to estimate paired AUROC differences\. Based on fixed out\-of\-fold predictions, the intervals quantify variation induced by group resampling\. Their endpoints are the corresponding distribution percentiles, and the group count denotes the number of sampling units\. Interpretation assumes independence between groups\.

The 20,000 samples are split into 20 batches\. Monte Carlo error in interval endpoints and widths is estimated primarily by a delete\-one\-batch jackknife and checked secondarily by the batch\-means method\. Width error bands are centered on the interval width from the pooled distribution and uset0\.975,19=2\.093t\_\{0\.975,19\}=2\.093\. These bands quantify numerical error from finite resampling\. A 95% interval width above 0\.04 forΔH\\Delta\_\{H\}is classified as insufficient precision\.

### B\.7Numerical Paths, Analysis Freeze, and Unblinding

The numerical control holds hidden states fixed and reconstructs affected generation features\. We document the checks, path settings, and data and rules fixed before formal analysis below\.

Path Byields an MCQ gain of0\.02730\.0273\(\[0\.0175,0\.0372\]\[0\.0175,0\.0372\]\), close to path A, and an H1 gain of0\.00280\.0028\(\[0\.0014,0\.0046\]\[0\.0014,0\.0046\]\), still on the order of a few thousandths\. This numerical check preserves the direction of E1 gains and their respective magnitudes\.

The original H1B4B\_\{4\}computation filters non\-finite token log\-probabilities, making features depend on a record\-specific subset of valid tokens; this issue affects almost all records\. Verification of path Bconfirms that H1 E3 block\-level hidden states match the original cache elementwise and that reconstructed fields are finite in both arms\. Numerical handling preserves the main E1 direction while changing some E3 criteria\. Path Bmeasures sensitivity to numerical handling in the original implementation\.

Samples, features, splits, and interpretation rules were fixed before the formal analysis accessed true labels\([Nosek et al\., 2018](https://arxiv.org/html/2610.27455#bib.bib25)\); analyses modified after unblinding are identified separately\. We report protocol hashes and the unblinding time below\. Appendix[C\.4](https://arxiv.org/html/2610.27455#A3.SS4)describes the cross\-arm contrast and confidence\-stratified analysis that did not yield valid estimates\.

Path Afollows the original implementation fixed before unblinding\. Path Bpreserves hidden states read from block 19 and reconstructsB4B\_\{4\}using a stabilized tail in the same forward pass\. The 8,664 H1 block\-level hidden\-state files are elementwise identical to the existing canonical repooling cache\. Path CremovesB4B\_\{4\}from both arms to examine dependence on that block\. The set and order of paths and their interpretation were fixed before accessing true labels\. Ais the primary result; Band Creport sensitivity to numerical handling and block dependence\.

The one\-time confirmatory analysis invokes both arms, all three paths, and all estimand branches in a prespecified order\. Validity of each branch’s estimate follows the stated criteria and audit records\. The protocol SHA\-256 begins withd483914c; the finalized\-record hash begins with38a20718; and the analysis\-input hash begins with2bad2870\. Unblinding occurred at2026\-08\-23T05:47:11\.928\-04:00\. The confirmatory result SHA\-256 isba467c67…9eeff5a\. Full hashes are retained in the analysis\-input and result\-lineage manifests\. Analyses added after unblinding are identified and reported separately as non\-confirmatory\.

### B\.8Estimands and Group Definitions

The primary estimand E1, entity generalization E2, vocabulary\-block generalization E3, and label\-dependent grouping M2c use the following split rules\.

- •E1 \(primary estimand\)\.Outer splits keep prompt groups disjoint to estimate conditional predictive gain when training and test sets share no prompt groups\. H1 has six generation records per prompt, all assigned to the same fold\. Each MCQ question forms a separate group\.
- •E2 \(entity generalization\)\.H1 is split by connected components of the identifier co\-occurrence graph so that an identifier never crosses folds\. E2 is non\-estimable if the largest component covers more than20%20\\%of records or there are fewer than5​F=255F=25components; only graph structure is then reported\. HereF=5F=5is the fixed number of outer folds, and 25 components ensure at least five per fold on average\. These estimability conditions were prespecified for this grouping and cross\-validation design\.
- •E3 \(vocabulary\-block generalization\)\.H1’s identifier vocabulary is partitioned into five fixed blocks without labels; the blocks serve as outer test folds\. Labels use only primary identifier mentions within the current vocabulary block, and hidden states are pooled only over token positions corresponding to that block\.
- •M2c \(label\-dependent grouping sensitivity\)\.H1 is split by connected components formed from prompts and fabricated identifiers\. Because grouping uses outcome labels, M2c is reported as a grouping\-sensitivity analysis\.

E2, E3, and M2c apply only to H1\. MCQ lacks the corresponding identifier\-entity and vocabulary\-block structures, so all three are marked not applicable for MCQ\.

Conditional gain can vary with baseline information, numerical handling, and grouping\. To examine these dependencies, we remove information blocks, reconstruct affected features, and change sample grouping\. Identity and format controls examine task constructs\([Hewitt and Liang, 2019](https://arxiv.org/html/2610.27455#bib.bib16);[Gururangan et al\., 2018](https://arxiv.org/html/2610.27455#bib.bib14);[Poliak et al\., 2018](https://arxiv.org/html/2610.27455#bib.bib29)\); identifier and vocabulary grouping examine performance under different sharing conditions\([Roberts et al\., 2017](https://arxiv.org/html/2610.27455#bib.bib31);[Elangovan et al\., 2021](https://arxiv.org/html/2610.27455#bib.bib10)\)\.[Table8](https://arxiv.org/html/2610.27455#A2.T8)summarizes these comparisons and their analytical roles\.

Table 8:Purposes and settings of additional comparisons\.

## Appendix CFull Results and Checks

### C\.1Prediction Results for All Paths

Qwen paths A, B, and C are confirmatory, with values from the August 23, 2026 confirmatory results\. Llama paths A and C are non\-confirmatory\.[Table9](https://arxiv.org/html/2610.27455#A3.T9)reports all Qwen estimates\. The left panel of[Figure3](https://arxiv.org/html/2610.27455#A3.F3)compares AUROC forS∗S^\{\\ast\},HH, andS∗\+HS^\{\\ast\}\+H; the right panel shows the corresponding gains and 95% confidence intervals\. Gray shading identifies path C, and a broken axis accommodates its MCQ gain\.[Table10](https://arxiv.org/html/2610.27455#A3.T10)reports interval widths and their Monte Carlo error bands\.

Table 9:Confirmatory estimates for the six E1 branches\.nnis the number of records, andGGis the number of groups resampled by the grouped bootstrap\.

Effect of numerical handling\.Path Breconstructs affected generation features while preserving hidden states to test whether numerical handling changes the gain\. The resulting gains are0\.00280\.0028on H1 and0\.02730\.0273on MCQ, preserving path A’s direction and magnitude\.[Tables9](https://arxiv.org/html/2610.27455#A3.T9)and[3](https://arxiv.org/html/2610.27455#A3.F3)show that H1 gains are on the order of thousandths and MCQ gains on the order of hundredths under paths A and B, with all intervals excluding zero\. Predictive improvements on both tasks remain stable after this reconstruction\.

The C/MCQ gain is substantially larger but accompanies a weaker combined baseline\. Its interval width is0\.059690\.05969, exceeding the prespecified limit of0\.040\.04\.

Figure 3:AUROC and conditional gain across E1 paths\.Table 10:E1 confidence interval widths and Monte Carlo error bands\.Jackknife standard errors of the endpoints range from7\.6×10−67\.6\\times 10^\{\-6\}to4\.2×10−44\.2\\times 10^\{\-4\}, indicating that 20,000 bootstrap samples reliably distinguish the precision classifications in this table\. Monte Carlo error captures numerical variation from finite resampling; confidence intervals capture sample variation estimated through group resampling\.

[Table11](https://arxiv.org/html/2610.27455#A3.T11)reports Llama’s paths A and C\.

Table 11:Llama detection performance and conditional gains on paths A and C\.The first three numerical columns report AUROC\. H1 has 8,810 records and 1,516 groups; MCQ has 1,746 records and 1,746 groups\. A retains all five blocks; C removesB4B\_\{4\}\. C/MCQ fails baseline non\-inferiority and has insufficient interval precision\.

### C\.2Blockwise Performance and Baseline Comparisons

To assess preservation of individual blocks’ predictive performance,[Table12](https://arxiv.org/html/2610.27455#A3.T12)reports their AUROC values and the baseline’s differences from them\. These comparisons characterize H1’s main predictive cues and MCQ’s loss from combination\.

Table 12:Single\-block AUROC and baseline point\-estimate differences on path A\.Identifier identity versus text content\.To assess whether H1’s high detection performance can already be explained by identifier identity, we compareB1B\_\{1\}with full\-text blockB3B\_\{3\}\. Their AUROC values in[Table12](https://arxiv.org/html/2610.27455#A3.T12)are0\.98260\.9826and0\.92860\.9286, respectively: identity alone performs better\. Identifier identity therefore supplies strong error\-prediction cues\. The AUROC gap betweenB1B\_\{1\}and the combined baseline is0\.00630\.0063\.

On MCQ,S∗S^\{\\ast\}falls 0\.0059 belowB4B\_\{4\}\. This difference is within the prespecified non\-inferiority tolerance, and M1b passes\. The value 0\.0059 is a descriptive difference between point estimates; its confidence interval was not separately estimated\.ΔH=0\.02893\\Delta\_\{H\}=0\.02893measures the total improvement over the five\-block stacked baseline\.

### C\.3Numerical Results for the Criteria

This section reports criterion outcomes, the most constraining blocks, and lower confidence bounds for the six E1 branches\. Direct blockwise improvement and the more conservative consistency requirement are distinguished, using frozen confirmatory results throughout\. iv\-b requires M1b to pass and the lower confidence bound forΔH\\Delta\_\{H\}to exceedεN​I=0\.02\\varepsilon\_\{NI\}=0\.02\.

Table 13:Blockwise criteria for the six E1 branches\.The iv\-a column gives the smallest marginal one\-sided 95% lower bound for the fused model’s difference from individual block predictors\.

In[Table13](https://arxiv.org/html/2610.27455#A3.T13), paths A and B and C/H1 fail iv\-b because their gain lower bounds do not exceed0\.020\.02\. Although C/MCQ has a lower bound of0\.22700\.2270, the baseline fails non\-inferiority againstB5B\_\{5\}, so it also fails the conjunction required by iv\-b\.

For both tasks on Qwen path A, the fused model’s differences from all individual blocks have positive lower confidence bounds, supporting superiority over every compared block\. The following path\-A bounds show the M1b and iv\-a calculations, with thresholds of−0\.02\-0\.02and00, respectively\.

Table 14:Marginal one\-sided 95% lower bounds for M1b and iv\-a on path A\.[Figure4](https://arxiv.org/html/2610.27455#A3.F4)subtracts each criterion’s threshold from the marginal one\-sided 95% lower bounds on the primary path\. The non\-inferiority panels show lower bounds plus0\.020\.02; the direct\-comparison panels show the lower bounds themselves\. All points lie to the right of zero, so every corresponding blockwise check passes\. Blocks closer to zero constrain the joint decision more strongly\.

Figure 4:Distance between blockwise lower bounds and criterion thresholds on primary path A\. Points show the marginal one\-sided 95% lower bound minus its threshold; positive values satisfy the corresponding criterion\.On path A, both arms fail iv\-b\. The MCQ lower bound forΔH\\Delta\_\{H\}is 0\.018941, falling 0\.00106 below 0\.02\. H1’s bound is 0\.000562, an order of magnitude below the threshold\. Both arms pass the direct blockwise comparison iv\-a but fail the more conservative conjunction in iv\-b\.

The path\-A task\-construct check is triggered on H1 byB1B\_\{1\}:Δtask,1=−0\.0539\\Delta\_\{\\mathrm\{task\},1\}=\-0\.0539, with marginal one\-sided upper bound−0\.0437\-0\.0437\. TheB2B\_\{2\}difference is 0\.2624 and does not trigger the check\. Neither MCQ comparison triggers it\. H1 thus shows strong prediction from identity cues under the current label definition; conditional gain is still reported under the same evaluation protocol\.

Both Llama tasks on path A pass M1b and iv\-a\. H1’s smallest non\-inferiority and direct\-comparison lower bounds are0\.00170\.0017and0\.00570\.0057, both constrained byB1B\_\{1\}\. MCQ’s corresponding bounds are−0\.0165\-0\.0165and0\.03720\.0372, both constrained byB4B\_\{4\}and exceeding−0\.02\-0\.02and zero, respectively\. H1’s gain lower bound is0\.00140\.0014, so iv\-b fails; MCQ’s is0\.04470\.0447, so it passes\. With joint option\-text and full\-response encoding, Llama MCQ still passes M1b and iv\-a, and its gain lower bound of0\.02640\.0264also passes iv\-b\.

### C\.4Additional Estimands and Incomplete Analyses

We report generalization and grouping\-sensitivity results, followed by the reasons why the cross\-arm contrast and RQ2 did not yield valid estimates\.

#### C\.4\.1Generalization and Grouping Sensitivity

To examine generalization to unseen identifiers, E2 connects records sharing a prompt or identifier and uses connected components to define disjoint training and test groups\. H1’s entity co\-occurrence graph contains 889 identifier identities and 10,494 identity edges, forming 17 components, fewer than the required 25\. The largest component covers 8,595/8,664 records \(99\.20%99\.20\\%\), exceeding the20%20\\%limit\. Both prespecified structural conditions fail, so E2 is non\-estimable and the analysis stops at structural statistics\.

E3 assigns 1,445 prompt groups to five fixed vocabulary blocks\. No target block can be assigned to 161 records, and another 2,418 records lack eligible labels and hidden states within their own test block, leaving 6,085 analyzable records\. All five test folds and their training complements contain both classes\.

Table 15:Confirmatory E3 estimates and criteria\.In[Table15](https://arxiv.org/html/2610.27455#A3.T15), all three E3 gain intervals exclude zero and all pass direct comparison iv\-a, but the remaining criteria differ\. Path A fails baseline non\-inferiority againstB4B\_\{4\}\. Path B passes non\-inferiority, but its gain lower bound remains below0\.020\.02\. Only path C satisfies both conditions and passes iv\-b\. E3 estimates conditional gain after redefining labels and hidden states by vocabulary block\.

M2c uses 238 fabricated identifiers and 1,124 identity edges, producing 1,283 components with a largest\-component share of 0\.0464\. Gains across the three paths are 0\.0142, 0\.0205, and 0\.0236, respectively, but all fail M1b againstB1B\_\{1\}and all fail iv\-a\. This grouping prevents only fabricated identifiers from crossing folds; known correct identifiers may still be shared\. Grouping also depends on labels, with an unknown direction of bias\. These gain estimates characterize sensitivity to label\-dependent grouping\. Superiority of the fused model over every compared block is sensitive to the grouping rule\.

[Table16](https://arxiv.org/html/2610.27455#A3.T16)reports the E3 and M2c criterion bounds and thresholds, identifying the blocks or conditions that constrain these decisions\.

Table 16:Vocabulary\-block generalization \(E3\) and grouping sensitivity \(M2c\)\.nnis the record count\. Criterion columns give lower bound / threshold; parentheses identify the block with the smallest bound\. Consistency requires non\-inferiority and a gain lower bound above0\.020\.02\.

#### C\.4\.2Cross\-Arm Contrast and Stratified Analysis

The prespecified cross\-arm contrast isD=SM​C​Q−SH​1D=S\_\{MCQ\}\-S\_\{H1\}and requires out\-of\-fold predictions from the output\-side observable predictorOO, which combines generation scores and visible output text\. The implementation mistakenly supplied out\-of\-fold predictions from the fused modelS∗\+HS^\{\\ast\}\+H, so the prespecifiedDDwas not validly estimated\. The frozen procedure returned nominal estimates of−0\.2835\-0\.2835,−0\.2781\-0\.2781, and−0\.3280\-0\.3280, with nominal 95% intervals\[−0\.3262,−0\.2413\]\[\-0\.3262,\-0\.2413\],\[−0\.3210,−0\.2351\]\[\-0\.3210,\-0\.2351\], and\[−0\.3731,−0\.2836\]\[\-0\.3731,\-0\.2836\], respectively\. These values are retained only as audit records of the input mismatch and cannot determine the prespecified cross\-arm gate or compare the tasks\. Recalculation under the original definition would be a post hoc recovery analysis\.

RQ2 was intended to stratify by identifier\-token log\-probability and describe conditional gain at different observable confidence levels\. Its prespecified primary stratifier depends on the H1B4B\_\{4\}field affected by non\-finite\-value filtering\. Following the branch fixed before unblinding, the primary analysis marks it uninterpretable and stops computation\. The stratifier was not replaced with tail\-stabilized fields, so no confirmatory RQ2 estimates are reported\.

## Appendix DDiscussion and Limitations

### D\.1Additional Discussion

Output\-baseline performance, task labels, and sample grouping jointly shape the interpretation of hidden\-state gains\.

Hidden states and output\-side information may share predictive cues\. Conditional gain measures the additional predictive capability obtained by the current learning procedure beyond existing output information\. H1’s output baseline approaches the AUROC ceiling, limiting absolute headroom\. Qwen’s baseline is0\.98890\.9889: even a fused AUROC of11would leave only about0\.01110\.0111of improvement at the point\-estimate level\. Llama H1 and the tested Qwen readout adaptations likewise show small gains under the current labels, learners, and evaluation populations\.

MCQ retains more pronounced gains\. Llama improves by0\.03870\.0387after joint full\-response encoding, showing that positive gains persist with broader output\-text coverage\. The same result illustrates coverage dependence: adding the full response changes both baseline and fused performance and reduces the gain estimate\.

Blockwise checks define the baseline’s comparison scope\. Non\-inferiority assesses the combined baseline against the selected blocks within a specified tolerance\. Passing direct comparisons supports fused\-model superiority over those blocks; consistency is a separate, more conservative condition\. All six Qwen E1 branches fail the consistency check, whereas Llama MCQ path A and the joint full\-response condition pass\. The lower\-bound threshold for this study’s consistency check is0\.020\.02\.

Reducing baseline information can increase the gain while decreasing final detection performance\. This occurs on MCQ path C for both models\. Ranking readout methods solely by gain would therefore overlook changes in the reference baseline\. Studies should report the combined baseline, standalone hidden\-state performance, fused performance, and conditional gain together, specifying the output sources and blockwise baseline comparisons\.

Future output\-side comparisons could include semantic entropy, SelfCheckGPT, and other text encoders, separately estimating changes in baseline and fused performance\. The effect of the parsing\-failure denominator also requires re\-estimating both\.

H1 labels depend on identifier existence in a fixed knowledge base, directly linking identity to the label\. Identity alone supplies strong error\-prediction cues\. In standalone\-statement adaptation, about93\.4%93\.4\\%–93\.8%93\.8\\%of test\-record texts occur in the corresponding training fold\. Identifier memorization is one possible explanation, but its contribution has not been separated from generalization\. E2 is non\-estimable, leaving generalization to unseen identifiers without a valid estimate\. E3 concerns a prediction task with labels and hidden states redefined by vocabulary block\. M2c retains positive gains but fails direct blockwise comparison, further limiting generalization across grouping schemes\.

### D\.2Limitations and Unevaluated Comparisons

The experiments leave the following conditions and data\-processing limitations unresolved:

1. 1\.Models, tasks, and analysis status\.We study Qwen2\.5\-7B\-Instruct and Llama\-3\.1\-8B\-Instruct\. Qwen A/B/C analyses are confirmatory; Llama A/C analyses are non\-confirmatory\. Both use local 8\-bit quantized backends; other precisions and domains are not covered\. Gains are estimated under each model’s generation budget, parser version, and analyzable population\. H1 labels verify whether generated identifiers exist in a fixed knowledge base\.
2. 2\.Information access and output baselines\.The study uses decoding scores and repeated\-sampling statistics, requiring access to those signals\. Output controls cover five block types and TF\-IDF text encoding; other output\-side methods and encoders remain to be compared\. Blockwise non\-inferiority applies to the selected blocks and prespecified tolerance\.
3. 3\.Readout settings and adaptation scope\.The main analysis uses a fixed layer and linear readout, while additional experiments cover limited MLP, L1, and XGBoost settings\. MLP candidate training budgets are limited, and probability clipping in the earlier readout conditions may change extreme scores and fusion inputs\. Actual iteration counts for selected models and complete execution traces were not retained\. H1 representation adaptation still has upper\-bound tuning selections, extensive cross\-fold text duplication, and missing intermediate logs\. Results concern these representations, budgets, and scoring settings\.
4. 4\.Numerical integrity and parsing conventions\.Non\-finite\-value filtering in Qwen’s original H1B4B\_\{4\}continues to constrain related conclusions\. Llama’sB4B\_\{4\}has not undergone an independent numerical integrity audit\. Although all target\-layer vectors in Qwen statement extraction are finite, downstream layers and output scores widely contain non\-finite values\. MCQ consistency features use successful parses as the denominator, affecting 128/1,746 Llama analysis rows through failure exclusion\. Recalculation agrees with the current implementation, but the failure\-denominator rule was not specified in the original protocol\. Its directional effect on predictive performance and gain has not been measured\.
5. 5\.Absolute gain and practical utility\.A baseline near the AUROC ceiling limits absolute headroom\. Label composition and baseline levels across tasks and models also affect comparisons\. This study evaluates ranking performance through AUROC gain and uses an operational tolerance of0\.020\.02\. Decisions at fixed thresholds, probability calibration, and readout cost remain unevaluated\.