HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders

# HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders
Source: [https://arxiv.org/html/2604.16430](https://arxiv.org/html/2604.16430)
###### Abstract

Large Language Models \(LLMs\) are powerful and widely adopted, but their practical impact is limited by the well\-known hallucination phenomenon\. While recent hallucination detection methods have made notable progress, we find most of them overlook the dynamic nature and underlying mechanisms of it\. To address this gap, we proposeHalluSAE, a phase transition\-inspired framework that models hallucination as a critical shift in the model’s latent dynamics\. By modeling the generation process as a trajectory through a potential energy landscape, HalluSAE identifies critical transition zones and attributes factual errors to specific high\-energy sparse features\. Our approach consists of three stages: \(1\) Potential Energy Empowered Phase Zone Localization via sparse autoencoders and a geometric potential energy metric; \(2\) Hallucination\-related Sparse Feature Attribution using contrastive logit attribution; and \(3\) Probing\-based Causal Hallucination Detection through linear probes on disentangled features\. Extensive experiments on Gemma\-2\-9B demonstrate that HalluSAE achieves state\-of\-the\-art hallucination detection performance\.

Machine Learning, ICML, Hallucination, Interpretability, LLM

Boshui Chen, Zhaoxin Fan∗, Ke Wang, Zhiying Leng, Faguo Wu, Hongwei Zheng, Yifan Sun, Wenjun Wu

††footnotetext:Boshui Chen, Zhaoxin Fan, Ke Wang, Faguo Wu, and Wenjun Wu are from Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University, Beijing, China\.
Zhiying Leng is from State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China\.
Hongwei Zheng is from Beijing Academy of Blockchain and Edge Computing, Beijing, China\.
Yifan Sun is from Renmin University of China, Beijing, China\.
∗Correspondence to: Zhaoxin Fan<<zhaoxinf@buaa\.edu\.cn\>\>\.## 1Introduction

Large Language Models \(LLMs\) have demonstrated remarkable emergent abilities\(Weiet al\.,[2022](https://arxiv.org/html/2604.16430#bib.bib5)\), but still suffer from the well\-known issue of hallucination, where the model generates content that is plausible yet incorrect\(Jiet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib1)\)\. This issue limits the use of LLMs in critical domains such as healthcare\(Singhalet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib6)\)and law\(Dahlet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib7)\), raising concerns about their reliability\. Therefore, understanding the underlying causes ofhallucination\(Elhageet al\.,[2021](https://arxiv.org/html/2604.16430#bib.bib17); Gaoet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib24)\)and conducting effective hallucination detection has become a key research focus\.

![Refer to caption](https://arxiv.org/html/2604.16430v1/figures/Gemini_Generated_Image_uogpmnuogpmnuogp.png)Figure 1:Illustration of Phase Transition in LLM’s reasoning trajectories\.The trajectories in potential energy space reveal three phases: early stability \(Phase I\), critical transition \(Phase II, yellow highlight\), and sustained error plateau \(Phase III\)\. Factual generation \(blue\) maintains low energy throughout, while hallucination \(gradient color\) undergoes abrupt energy increase \(Δ​E\\Delta E\) during the transition zone, entering a persistent high\-energy state\.Mainstream hallucination detection methods are typically divided into black\-box and white\-box approaches\. Black\-box methods, like SelfCheckGPT\(Manakulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib8)\)and CoVe\(Dhuliawalaet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib9)\), rely on output consistency across multiple model runs, but are computationally intensive and lack interpretability\. White\-box methods instead leverage internal states, using heuristics such as perplexity\(Huanget al\.,[2025b](https://arxiv.org/html/2604.16430#bib.bib10); Friel and Sanyal,[2023](https://arxiv.org/html/2604.16430#bib.bib11)\), attention entropy\(Liet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib12)\), or activation statistics\. More advanced approaches, like SAPALMA\(Azaria and Mitchell,[2023](https://arxiv.org/html/2604.16430#bib.bib13)\)and ITI\(Liet al\.,[2023b](https://arxiv.org/html/2604.16430#bib.bib14)\), use linear probes on the residual stream, while others model errors as constraint satisfaction in attention heads\(Yuksekgonulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib15)\)\. Recent studies\(Sriramananet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib16)\)have shown that internal\-state methods offer significant speed advantages over black\-box approaches\. Although existing hallucination detection methods have achieved remarkable results, we find that they primarily analyze hallucination from the perspective of static feature representations or final outputs\. Consequently, they do not fully consider the dynamic nature of hallucination as it unfolds during the generation process, and often overlook important information about how hallucinations develop over time\. Moreover, focusing only on static features makes it easy for individual neurons to mix together many unrelated concepts\(Elhageet al\.,[2021](https://arxiv.org/html/2604.16430#bib.bib17)\), which can negatively impact detection performance\.

Therefore, this paper proposes a physics\-inspired perspective to bridge this gap\. Rather than interpreting hallucinations as isolated surface errors, we posit that they reflect a critical phase transition within the model’s latent dynamics \(see Fig\.[1](https://arxiv.org/html/2604.16430#S1.F1)\)\. By conceptualizing the generation process as a trajectory over a potential energy landscape, we reveal a distinctive mechanism: factual errors emerge when internal representations, propelled by specific high\-energy sparse features, abruptly transition from a low\-energy “truth attractor” to a persistent, unstable high\-energy plateau\. This dynamic viewpoint motivates a key insight:by monitoring the temporal evolution of latent potential energy in LLM, one can potentially localize phase transition zones that serve as reliable indicators for hallucination detection\.

To this end, we propose a systematic detection framework,HalluSAE, grounded in the phase transition perspective\. Our approach follows a coarse\-to\-fine pipeline comprising three stages: \(1\)Potential Energy Empowered Phase Zone Localization, where we leverage the Gemma Scope Sparse Autoencoder \(SAE\) to disentangle the residual stream and introduce a geometric potential energy metric, allowing us to efficiently localize “phase transition zones” by identifying layers with exponential energy growth and narrowing the search space to a few critical transition points; \(2\)Hallucination\-related Sparse Feature Attribution, in which we apply Contrastive Direct Logit Attribution \(DLA\) within these high\-energy regions to quantify the contributions of individual sparse features to incorrect outputs, thereby precisely isolating “hallucination\-inducing features” responsible for specific error patterns; and \(3\)Probing\-based Causal Hallucination Detection, where we train linear probes on the selected sparse features to enable accurate and efficient inference\-time hallucination detection, establishing a causal link between internal model dynamics and output errors\. Collectively, HalluSAE offers a targeted, interpretable, and scalable solution for hallucination detection by explicitly modeling the dynamic processes underlying factual errors in large language models\.

To assess the effectiveness and generalizability of our approach, we conduct experiments on Gemma\-2\-9B using both in\-distribution \(HaluEval\) and out\-of\-distribution \(TriviaQA\) benchmarks, where our method consistently achieves superior hallucination detection performance over existing baselines\. The main contributions of this paper are summarized as follows:

- •We introduce HalluSAE, a framework that approaches hallucination from a phase transition perspective, enabling dynamic and process\-aware hallucination detection rather than relying solely on static representations\.
- •We propose Potential Energy Empowered Phase Zone Localization, Hallucination\-related Sparse Feature Attribution, and Probing\-based Causal Hallucination Detection in HalluSAE to precisely identify and attribute hallucination\-inducing features\.
- •We conduct comprehensive experiments, which demonstrate that our method achieves new state\-of\-the\-art results on in\-distribution \(HaluEval\) tasks, and out\-of\-distribution \(TriviaQA\) benchmarks\.

## 2Related Work

Hallucination in Large Language Models\.Despite the impressive emergent abilities of LLMs\(Weiet al\.,[2022](https://arxiv.org/html/2604.16430#bib.bib5)\), hallucinations remain a persistent challenge, especially in high\-stakes domains such as healthcare\(Singhalet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib6)\)and law\(Dahlet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib7)\)\. Recent taxonomies categorize these errors as input\-conflicting or fact\-conflicting, attributing them to data divergence or the compounding of early generation mistakes\(Zhanget al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib2); Huanget al\.,[2025a](https://arxiv.org/html/2604.16430#bib.bib3)\)\. However, while these classifications are well\-studied, the mechanistic understanding ofhowandwhyhallucinations dynamically arise during inference is still lacking\(Orgadet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib26); Zhanget al\.,[2025b](https://arxiv.org/html/2604.16430#bib.bib27)\)\. Most existing methods treat hallucination as a static phenomenon based on final output or representations\(Jiet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib1); Huanget al\.,[2025a](https://arxiv.org/html/2604.16430#bib.bib3)\); in contrast, this work analyzes hallucination from a dynamical perspective for its detection\.

Hallucination Detection Methods\.Existing hallucination detection strategies fall into black\-box and white\-box categories\. Black\-box methods rely on output consistency checks\(Manakulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib8); Dhuliawalaet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib9)\), external LLM judges\(Benkiraneet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib48)\), or Bayesian sequential estimation\(Wanget al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib47)\), but are often computationally costly and lack interpretability\. Zero\-shot attention methods like AGSER\(Liuet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib49)\)also require multiple inference passes, increasing latency\. White\-box methods leverage internal states\. Early work by Xu et al\.\(Xuet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib46)\)pioneered model introspection in NMT\. Recently, HARP\(Huet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib44)\)and GSP\(Noël,[2025](https://arxiv.org/html/2604.16430#bib.bib45)\)utilized SVD projections and graph spectral energy, respectively, to detect errors\. However, these dense or topological approaches often suffer from neuron polysemanticity\(Elhageet al\.,[2021](https://arxiv.org/html/2604.16430#bib.bib17)\), making it difficult to isolate causal features\. In contrast, we take a sparse and dynamic perspective, utilizing SAEs to precisely localize hallucination causes\.

Sparse Autoencoders and Their Application\.Sparse Autoencoders \(SAEs\) have emerged as a powerful tool for interpretability, decomposing dense activations into human\-interpretable features\(Brickenet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib21); Cunninghamet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib20); Lieberumet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib19)\)\. Recent studies have demonstrated their versatility in uncovering functional circuits\(Markset al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib23)\)and enhancing privacy\(Frikhaet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib31)\)\. In hallucination research, SAFE\(Abdaljalilet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib50)\), SSL\(Huaet al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib51)\), SAVE\(Parket al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib52)\), and RAGLens\(Xionget al\.,[2025](https://arxiv.org/html/2604.16430#bib.bib53)\)utilize SAEs for query enrichment, steering, or detecting faithfulness failures in RAG\. However, these methods either focus on RAG\-specific scenarios, require task\-specific feature selection, or treat generation as static activation without modeling thedynamic phase transitionprocess\. In contrast, we introduce a geometric potential energy framework that tracks temporal evolution and systematically localizes causal features, making this the first work leveraging dynamic SAE geometry for general factual hallucination detection\.

## 3Pre\-analysis on Hallucination From the Phase Transition View

As discussed above, this work analyzes hallucination in LLMs from a phase transition perspective and proposes a hallucination detection framework based on this view\. Before introducing our HalluSAE method, we first present the key analytical tools and some preliminary findings\. Specifically, to analyze hallucination in LLMs, we primarily leverage the two following tools: \.

Sparse Autoencoders \(SAEs\)\.SAEs address the problem of mixed semantic representations in dense activations by mapping thedmodeld\_\{\\text\{model\}\}\-dimensional residual stream to a much higher\-dimensional, sparse space \(dSAE≫dmodeld\_\{\\text\{SAE\}\}\\gg d\_\{\\text\{model\}\}\), where each dimension is encouraged to capture a single semantic concept\(Cunninghamet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib20); Brickenet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib21)\)\. In this work, we adopt the open\-source Gemma Scope SAE\(Lieberumet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib19)\), which decomposes the residual stream as a weighted sum of sparse features:

rlt≈∑i=1dSAEsi⋅Wdec​\[i,:\]r\_\{l\}^\{t\}\\approx\\sum\_\{i=1\}^\{d\_\{\\text\{SAE\}\}\}s\_\{i\}\\cdot W\_\{\\text\{dec\}\}\[i,:\]\(1\)wheresi=JumpReLU​\(Wenc⋅rlt\+benc\)​\[i\]s\_\{i\}=\\text\{JumpReLU\}\(W\_\{\\text\{enc\}\}\\cdot r\_\{l\}^\{t\}\+b\_\{\\text\{enc\}\}\)\[i\]denotes the activation of theii\-th sparse feature\. We setdSAE=131,072d\_\{\\text\{SAE\}\}=131\{,\}072\(36\.6×\\timesovercomplete\) and useL0≈30L\_\{0\}\\approx 30for a balance between sparsity and interpretability\. The JumpReLU activation preserves activation strength while maintaining sparsity, thus better capturing the underlying semantics compared to traditional L1\-SAEs\.

Geometric Potential Energy \(GPE\)\.From a dynamical systems perspective, we quantify the deviation from a stable, factual state using Geometric Potential Energy \(GPE\), defined as the squared Euclidean distance from the “truth attractor” in SAE feature space\(Marks and Tegmark,[2023](https://arxiv.org/html/2604.16430#bib.bib38); Zouet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib39)\):

E​\(l,t\)=‖SAE​\(rlt\)−μtruth‖22E\(l,t\)=\\\|\\text\{SAE\}\(r\_\{l\}^\{t\}\)\-\\mu\_\{\\text\{truth\}\}\\\|\_\{2\}^\{2\}\(2\)whereμtruth\\mu\_\{\\text\{truth\}\}is the centroid of SAE features for all factual samples\. This metric captures both the direction and magnitude of deviation, amplifies significant departures \(consistent with phase transition phenomena\), and focuses on semantic\-level differences thanks to the disentangling property of SAEs\.

![Refer to caption](https://arxiv.org/html/2604.16430v1/figures/exloratory.png)Figure 2:Illustration of the Exploratory Experimental Design\.We conduct two complementary experiments:Exp 1investigates layer\-wise energy distribution by dividing 42 layers into Early/Middle/Late groups and comparing GPE differences;Exp 2identifies microscopic feature\-level contributions by analyzing differential features that exhibit high activation in hallucination samples but low activation in factual samples\.![Refer to caption](https://arxiv.org/html/2604.16430v1/x1.png)Figure 3:Experimental Results on Hallucination Dynamics\.\(a\)Layer\-wise energy distribution \(grouped analysis\)\.Grouped box plots reveal that hallucination samples’ GPE exhibits significant escalation from Early to Late layer groups \(\*\*\*p<0\.001p<0\.001\)\. Error bars indicate 95% confidence intervals \(bootstrap,n=200n\{=\}200per group\)\. \(b\)Sparse feature contribution analysis\.Cumulative energy contribution curve demonstrates that a small fraction of differential features account for the majority of total energy increase, exhibiting a highly skewed distribution\. Gray bars represent individual feature energy contributions\.Using both analytical tools, as shown in Fig\.[2](https://arxiv.org/html/2604.16430#S3.F2), we first conduct a series of controlled experiments to pre\-analyze the internal dynamics of LLM reasoning\. Our experimental results illustrated in Fig\.[3](https://arxiv.org/html/2604.16430#S3.F3)reveal several key patterns\. First, layer\-wise energy analysis \(Fig\.[3](https://arxiv.org/html/2604.16430#S3.F3)a\) shows that the energy difference between hallucination and factual samples is negligible in the early layers, increases steadily in the middle layers, and surges sharply in the late layers—exhibiting a clear phase transition\(Menget al\.,[2022](https://arxiv.org/html/2604.16430#bib.bib40); Gevaet al\.,[2021](https://arxiv.org/html/2604.16430#bib.bib41)\)\. Statistical analysis \(Cohen’sd=1\.64d=1\.64,p<0\.001p<0\.001\) further confirms that hallucinations arise within specific layer intervals, rather than being uniformly distributed across the network\. Second, at the feature level \(Fig\.[3](https://arxiv.org/html/2604.16430#S3.F3)b\), we observe that only a small subset of sparse features—those highly activated in hallucination cases—are responsible for the majority of the energy increase\. This heavy\-tailed distribution indicates that hallucinations are driven by a few high\-energy features, rather than by diffuse, low\-level noise\. In summary, our analysis yields two main findings:

- •Finding 1: Hallucination emerges as a dynamic process during generation, with clear phase transitions occurring at specific layer intervals rather than manifesting as a static property throughout the network\.
- •Finding 2: Hallucination is predominantly driven by a small subset of high\-impact sparse activations, rather than by broad or diffuse noise, highlighting the critical role of a few key features in triggering the transition to erroneous outputs\.

These two findings motivate our approach:by dynamically monitoring potential energy changes in the sparse feature space, we can effectively detect phase transitions that indicate hallucination\.Next, we introduce our motivated approach HalluSAE in detail\.

## 4Method

Given an input prompt, our goal is to predict whether the model’s output will contain hallucinations before generation is complete—by analyzing activations from selected layers inside the LLM\. Motivated by our earlier findings, we propose HalluSAE, a three\-stage framework grounded in the dynamics of sparse feature activations and phase transitions\.

HalluSAE operates as follows: \(1\) Potential Energy Empowered Phase Zone Localization leverages sparse autoencoders and a geometric potential energy metric to identify critical layer intervals where phase transitions indicative of hallucination occur\. \(2\) Hallucination\-related Sparse Feature Attribution uses contrastive logit attribution to pinpoint a small set of sparse features most responsible for hallucination within these layers\. \(3\) Probing\-based Causal Hallucination Detection conducts linear probing on disentangled features to detect and validate causal relationships between feature activation patterns and hallucination outcomes\. In the following subsections, we detail the methodology and implementation of each stage\.

### 4\.1Stage I: Potential Energy Empowered Phase Zone Localization

The primary goal of this module is to identify which layers within the model are most susceptible to hallucination, by detecting phase changes through differences in Geometric Potential Energy \(GPE\) between hallucination and factual samples\. By quantifying how the GPE gap evolves across layers, we aim to pinpoint the specific regions in the network where the internal state shifts most sharply toward hallucination\. The operations are as follows:

For each layerℓ\\ell, we compute the average GPE difference as

Δ​Eℓ=𝔼x∈ℋ​\[Eℓ​\(x\)\]−𝔼x∈ℱ​\[Eℓ​\(x\)\],\\Delta E\_\{\\ell\}=\\mathbb\{E\}\_\{x\\in\\mathcal\{H\}\}\[E\_\{\\ell\}\(x\)\]\-\\mathbb\{E\}\_\{x\\in\\mathcal\{F\}\}\[E\_\{\\ell\}\(x\)\],\(3\)whereℋ\\mathcal\{H\}andℱ\\mathcal\{F\}denote the sets of hallucination and factual samples, respectively\. To characterize the dynamics of this difference, we define the relative growth rate

γℓ=Δ​Eℓ−Δ​Eℓ−1Δ​Eℓ−1×100%\.\\gamma\_\{\\ell\}=\\frac\{\\Delta E\_\{\\ell\}\-\\Delta E\_\{\\ell\-1\}\}\{\\Delta E\_\{\\ell\-1\}\}\\times 100\\%\.\(4\)A stable or oscillatoryγℓ\\gamma\_\{\\ell\}suggests the model remains in a consistent state, while a sequence of sustained positive growth indicates a transition toward hallucination\.

To robustly identify the onset of this transition, we locate the layer interval \[ℓstart\\ell\_\{\\mathrm\{start\}\},ℓend\\ell\_\{\\mathrm\{end\}\}\] starting from the earliest layerℓstart\\ell\_\{\\mathrm\{start\}\}that exhibits three consecutive positive growth rates,

ℓstart=min⁡\{ℓ​∣γℓ\>​0,γℓ\+1\>0,γℓ\+2\>0\}\.\\ell\_\{\\mathrm\{start\}\}=\\min\\left\\\{\\ell\\mid\\gamma\_\{\\ell\}\>0,\\gamma\_\{\\ell\+1\}\>0,\\gamma\_\{\\ell\+2\}\>0\\right\\\}\.\(5\)We then define the endpointℓend\\ell\_\{\\mathrm\{end\}\}as the layer whereΔ​Eℓ\\Delta E\_\{\\ell\}reaches its post\-onset maximum,

ℓend=arg⁡maxℓ≥ℓstart⁡Δ​Eℓ\.\\ell\_\{\\mathrm\{end\}\}=\\arg\\max\_\{\\ell\\geq\\ell\_\{\\mathrm\{start\}\}\}\\Delta E\_\{\\ell\}\.\(6\)
This process allows us to automatically detect the “phase transition zone”—the interval in which the model’s internal state shifts most dramatically toward hallucination—by analyzing potential energy dynamics\. The identified zone provides a principled basis for focusing subsequent fine\-grained analysis on the most vulnerable layers for hallucination emergence\.

### 4\.2Stage II: Hallucination\-related Sparse Feature Attribution

After localizing the layers most vulnerable to hallucination, the core challenge is to understand which internal mechanisms within these layers are responsible for the phenomenon\. Due to the polysemanticity and entanglement present in raw model activations, direct analysis is often unreliable\. We therefore employ the sparse and overcomplete representations provided by the SAE as a principled basis for disentangling and attributing semantic effects in the network\. The interpretability and sparsity of SAE features allow for precise tracing of how specific activations influence model outputs\.

Specifically, within the identified phase transition layers, we aim to isolate the subset of SAE features that causally drive hallucination\(Elhageet al\.,[2021](https://arxiv.org/html/2604.16430#bib.bib17); Wanget al\.,[2022](https://arxiv.org/html/2604.16430#bib.bib37)\)\. For each featureii, we define its direct logit attribution to a target tokenttas

𝒜i\(t\)=\(si​𝐰dec\(i\)\)⊤​𝐖U​\[:,t\],\\mathcal\{A\}\_\{i\}^\{\(t\)\}=\\left\(s\_\{i\}\\,\\mathbf\{w\}\_\{\\mathrm\{dec\}\}^\{\(i\)\}\\right\)^\{\\top\}\\mathbf\{W\}\_\{\\mathrm\{U\}\}\[:,t\],\(7\)wheresis\_\{i\}is the feature activation,𝐰dec\(i\)\\mathbf\{w\}\_\{\\mathrm\{dec\}\}^\{\(i\)\}is the SAE decoder direction, and𝐖U\\mathbf\{W\}\_\{\\mathrm\{U\}\}is the unembedding matrix\. This quantifies the linear pathway from a sparse feature to its impact on the model’s output space\.

Yet, hallucination generation is inherently a contrastive process: it depends not only on increasing the likelihood of the incorrect token but also on suppressing the correct one\(Liet al\.,[2023b](https://arxiv.org/html/2604.16430#bib.bib14)\)\. To capture this, we introduce the Contrastive Direct Logit Attribution \(C\-DLA\), defined as

𝒞i=si⋅\(𝐰dec\(i\)\)⊤​\(𝐖U​\[:,twrong\]−𝐖U​\[:,tcorrect\]\)\.\\mathcal\{C\}\_\{i\}=s\_\{i\}\\cdot\\left\(\\mathbf\{w\}\_\{\\mathrm\{dec\}\}^\{\(i\)\}\\right\)^\{\\top\}\\left\(\\mathbf\{W\}\_\{\\mathrm\{U\}\}\[:,t\_\{\\mathrm\{wrong\}\}\]\-\\mathbf\{W\}\_\{\\mathrm\{U\}\}\[:,t\_\{\\mathrm\{correct\}\}\]\\right\)\.\(8\)Here, the vectorΔ​𝐯=𝐖U​\[:,twrong\]−𝐖U​\[:,tcorrect\]\\Delta\\mathbf\{v\}=\\mathbf\{W\}\_\{\\mathrm\{U\}\}\[:,t\_\{\\mathrm\{wrong\}\}\]\-\\mathbf\{W\}\_\{\\mathrm\{U\}\}\[:,t\_\{\\mathrm\{correct\}\}\]represents the semantic direction from the correct answer to the hallucinated answer in output space\. A positive value of𝒞i\\mathcal\{C\}\_\{i\}indicates that featureiiboth elevates the hallucinated token and suppresses the factual one, thus promoting hallucination\.

For each prompt, we compute𝒞i\\mathcal\{C\}\_\{i\}for all features in the critical layers, rank them by magnitude, and we select the top 100 features \(0\.076% of the 131,072\-dimensional space\) as the critical set\. This procedure yields a compact collection of features that are not merely correlated with hallucination, but mechanistically responsible for shifting model outputs away from factuality\. These identified features serve as the analytical and causal substrate for detection and intervention in the next stage\.

### 4\.3Stage III: Probing\-based Causal Hallucination Detection

Building on the identification of critical SAE features within the phase transition zone, the final step is to leverage these features for hallucination detection\. The objective is to determine, for a given prompt and its candidate output, whether the activation pattern across this feature set reliably signals the presence of hallucination\.

In particular, for each samplexx, let𝐬𝒯​\(x\)∈ℝ\|𝒯\|\\mathbf\{s\}\_\{\\mathcal\{T\}\}\(x\)\\in\\mathbb\{R\}^\{\|\\mathcal\{T\}\|\}denote the concatenated activations of the selected feature set𝒯\\mathcal\{T\}across all layers in the identified zone\. Rather than relying on granular error categorization, we adopt a unified detection strategy\. We construct the critical feature set𝒯\\mathcal\{T\}by selecting the top\-ranked sparse features based on their average C\-DLA magnitude across the entire training dataset\. A single logistic regression Probe is then trained on these high\-impact features\(Alain and Bengio,[2016](https://arxiv.org/html/2604.16430#bib.bib43)\)to distinguish between hallucination and factual generations\. This holistic approach leverages the semantic disentanglement of SAEs, enabling a compact feature set to capture diverse hallucination patterns—ranging from numerical errors to entity substitutions—without requiring explicit type\-specific engineering\.

The detection model is formulated as anℓ1\\ell\_\{1\}\-regularized logistic regression, optimizing

min𝜽⁡1N​∑i=1Nlog⁡\(1\+exp⁡\(−yi​𝜽⊤​𝐬𝒯​\(xi\)\)\)\+λ​‖𝜽‖1,\\min\_\{\\boldsymbol\{\\theta\}\}\\ \\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\left\(1\+\\exp\(\-y\_\{i\}\\,\\boldsymbol\{\\theta\}^\{\\top\}\\mathbf\{s\}\_\{\\mathcal\{T\}\}\(x\_\{i\}\)\)\\right\)\+\\lambda\\\|\\boldsymbol\{\\theta\}\\\|\_\{1\},\(9\)whereyi∈\{\+1,−1\}y\_\{i\}\\in\\\{\+1,\-1\\\}is the hallucination label, andλ\\lambdais selected by cross\-validation\. This approach yields a sparse, interpretable classifier in which nonzero coefficients directly indicate the contributions of individual features to hallucination prediction\.

During inference, for any input, the model extracts the activation pattern across𝒯\\mathcal\{T\}, and outputs a hallucination probability\. Samples exhibiting activation signatures similar to known hallucinations are thus flagged in advance of output generation, enabling proactive identification\.

## 5Experiments

### 5\.1Experimental Setup

Table 1:Comparison Results of Hallucination Detection Performance Across Benchmarks\.We compare our method against eight representative baselines spanning four paradigms\. Results are grouped by in\-distribution \(HaluEval\) and out\-of\-distribution \(TriviaQA\) settings\. Best results are highlighted in gray\. All metrics are reported as percentages, with AUC as the primary ranking metric\.Datasets\.We evaluate our method on two complementary benchmarks\. For in\-distribution \(ID\) evaluation, we use the HaluEval dataset\(Liet al\.,[2023a](https://arxiv.org/html/2604.16430#bib.bib25)\), consisting of 1,800 samples \(900 hallucination, 900 factual\) with 80% allocated for training \(1,440 samples\) and 20% for testing \(360 samples\)\. All samples undergo dual annotation: automatic verification by GPT\-4o followed by human expert review\. The dataset is carefully balanced across generation length, error types \(50% numerical, 50% entity\), and question difficulty\. For out\-of\-distribution \(OOD\) evaluation, we construct a TriviaQA test set containing 4,000 samples \(2,000 hallucination, 2,000 factual\) to assess zero\-shot cross\-domain generalization\. Compared to HaluEval, TriviaQA features more colloquial phrasing, broader knowledge coverage spanning geography, history, people, and organizations, and greater diversity in question formats\.

Baselines\.We compare against eight representative detection methods spanning four paradigms\. Uncertainty\-based approaches include LN\-Entropy\(Manakulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib8); Xiao and Wang,[2021](https://arxiv.org/html/2604.16430#bib.bib36)\)and Semantic Entropy\(Farquharet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib32)\)\. Consistency\-based methods comprise Lexical Similarity\(Manakulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib8); Wanget al\.,[2022](https://arxiv.org/html/2604.16430#bib.bib37)\)and SelfCheckGPT\(Manakulet al\.,[2023](https://arxiv.org/html/2604.16430#bib.bib8)\)\. Internal state\-based approaches feature EigenScore\(Chenet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib33)\)and MHAD\(Zhanget al\.,[2025a](https://arxiv.org/html/2604.16430#bib.bib34)\)\. Supervised probing methods include SAPLMA\(Azaria and Mitchell,[2023](https://arxiv.org/html/2604.16430#bib.bib13)\)and HaloScope\(Duet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib35)\)\. Detailed implementation configurations for all baselines are provided in Appendix[A\.4](https://arxiv.org/html/2604.16430#A1.SS4)\.

Experimental Details\.All experiments are conducted on Gemma\-2\-9B \(42 layers\) equipped with Gemma Scope Sparse Autoencoders\(Lieberumet al\.,[2024](https://arxiv.org/html/2604.16430#bib.bib19)\)\. The SAE configuration includes 131,072 feature dimensions \(36\.6×\\timesovercomplete\), JumpReLU activation function, and average sparsityL0≈30L\_\{0\}\\approx 30across layers\. Stage I localizes the phase transition zone by identifying layers with exponential growth in geometric potential energy, yielding layers 23–35 \(L23–35\) as the critical interval\. In Stage II, Contrastive Direct Logit Attribution \(C\-DLA\) is applied within this window to select the top\-100 features exhibiting the highest causal influence on hallucination\. We report Area Under the ROC Curve \(AUC\) as the primary metric for assessing ranking capability, supplemented by accuracy, recall, and specificity for comprehensive performance characterization\.

### 5\.2Main Results

Quantitative Comparison\.Table[1](https://arxiv.org/html/2604.16430#S5.T1)summarizes performance across all methods\. HalluSAE attains an AUC of 0\.9286 on the in\-distribution test set and 0\.8044 on out\-of\-distribution TriviaQA, establishing new state\-of\-the\-art results in both regimes\. Unsupervised baselines display limited efficacy: the strongest \(Semantic Entropy\) achieves 0\.7185 AUC on OOD data, while HalluSAE surpasses this by 11\.9% \(0\.8044 vs\. 0\.7185\), highlighting the insufficiency of surface\-level statistical cues for capturing the causal dynamics of hallucination\. Consistency\-based approaches perform worse still, with SelfCheckGPT reaching only 0\.6353 OOD AUC—a 23\.0% deficit relative to HalluSAE\. Among supervised baselines, HaloScope represents the previous state\-of\-the\-art, utilizing dense hidden states \(3,584 dimensions\) across multiple layers\. HalluSAE outperforms HaloScope by 10\.1% on in\-distribution data \(0\.9286 vs\. 0\.8437\) and by 1\.9% on OOD data \(0\.8044 vs\. 0\.7894\)\. This improvement is attributable to three core advantages of sparse autoencoder features: \(i\) semantic disentanglement in the 131k\-dimensional sparse space mitigates the polysemanticity of dense activations, where individual neurons encode multiple unrelated concepts; \(ii\) C\-DLA enables precise, mechanistic attribution of hallucination to individual features, rather than diffuse patterns over thousands of variables; and \(iii\) semantic\-level representations demonstrate superior robustness to domain shift compared to raw activation statistics\. Notably, these gains are achieved using only 100 sparse features—a 35\.8×\\timesreduction compared to HaloScope—underscoring the efficiency and interpretability of the approach\.

Qualitative Comparison\.To assess the interpretability and stability of the features uncovered by our approach, we conduct an in\-depth analysis of the top\-50 features ranked by C\-DLA score\. We select features according to two stringent criteria: \(1\) high\-activation samples for a given feature are exclusively hallucination cases, and \(2\) the feature exhibits a significant activation gap between hallucinated and factual samples\. Applying these criteria, we identify four representative high\-purity features \(see Table[2](https://arxiv.org/html/2604.16430#S5.T2)\)\. Fig\.[4](https://arxiv.org/html/2604.16430#S5.F4)visualizes the highest\-activation samples associated with each feature\. Our analysis reveals that samples activating a particular feature consistently display distinct error modes: for example, features L27\-9659 and L28\-87984 are linked to numerical substitution errors, while features L23\-71479 and L24\-35793 correspond to entity replacement phenomena\. This strong correspondence between individual features and specific hallucination types demonstrates that these features capture systematic, semantically meaningful deviations—rather than reflecting mere statistical noise\. These findings highlight three key properties of our approach\. First, there is a robust feature\-to\-pattern correspondence: each high C\-DLA feature is associated with a stable, interpretable hallucination error mode, supporting the view that hallucinations are driven by specific neural mechanisms rather than random fluctuations\. Second, the use of sparse autoencoder features yields genuine semantic interpretability, in contrast to dense hidden state representations; extracted features map transparently onto functions such as fixed value anchoring, numerical shift, and entity confusion\. Third, the semantic clarity of these features opens the door to targeted causal interventions, suggesting promising directions for hallucination mitigation via SAE\-based manipulation\.

![Refer to caption](https://arxiv.org/html/2604.16430v1/figures/case.png)Figure 4:Examples of Samples with High\-Activation on Identified Sparse Features\.Each feature exhibits stable and interpretable error patterns: L27\-9659 shows fixed year substitution \(always outputs “1993”\), L28\-87984 demonstrates numerical shift patterns \(\+4 years, \-0\.2 billion\), L23\-71479 drives domain\-specific entity confusion \(Tesla/Edison, Newton/Einstein\), and L24\-35793 triggers cross\-domain celebrity substitution \(Tolkien/Rowling, Jobs/Gates\)\.Table 2:Identified High\-Purity Hallucination\-Driving Features
### 5\.3Ablation Study

Ablation Study on the Potential Energy Empowered Phase Zone Localization Stage\.To assess the necessity of precise phase zone localization, we compare five layer selection strategies while fixing the feature dimension at 100\. As shown in Table[3](https://arxiv.org/html/2604.16430#S5.T3), our energy\-identified transition zone \(L23–35\) achieves optimal results, with an in\-distribution AUC of 0\.93 and an out\-of\-distribution AUC of 0\.80\. Remarkably, this matches the performance of using all 42 layers \(420 features\), but with a4\.2×4\.2\\timesreduction in feature count\. In contrast, early layers \(L0–12\) fail to capture hallucination\-relevant signals \(AUC: 0\.81 ID, 0\.73 OOD\), while late layers \(L29–41\) yield only intermediate performance \(AUC: 0\.88 ID, 0\.78 OOD\), likely reflecting downstream consequences rather than the underlying phase transition process\. Randomly selected layers further underperform \(AUC: 0\.83 ID, 0\.76 OOD\), underscoring that energy\-based localization captures genuine causal structure rather than arbitrary layer boundaries\. To further illustrate the phase transition mechanism, Fig\.[5](https://arxiv.org/html/2604.16430#S5.F5)depicts the layer\-wise evolution of GPE differences between hallucination and factual samples across all 42 layers\. The trajectory reveals three distinct phases: a stable period \(L0–22\) characterized by near\-zero energy difference and random fluctuations, indicating similar representational dynamics for hallucinated and factual samples in early processing; a transition zone \(L23–35\) marked by a dramatic 20\.7\-fold escalation in energy \(from 5,862 at L23 to 121,245 at L35\), representing a qualitative shift, as confirmed by a large effect size \(Cohen’sd=1\.64d=1\.64,p<0\.001p<0\.001\) between stable and transition phases; and a plateau phase \(L36–41\) with persistently high but stable energy levels \(Cohen’sd=0\.12d=0\.12vs\. transition zone,p=0\.342p=0\.342\), indicating entry into a sustained deviated state\.

Table 3:Ablation Results on Layer Selection Strategy\.Comparison of detection performance across different layer selection strategies using a fixed feature budget of 100 dimensions\. The transition zone \(L23–35\) achieves optimal performance with minimal features, demonstrating the effectiveness of our energy\-based localization method\.StrategyLayer Range\# Layers\# Feat\.ID AUCOOD AUCΔ\\Deltavs OursRandom\-13Random 13 layers131000\.830\.76\-4\.0%Early\-13L0–12 \(Stable\)131000\.810\.73\-7\.0%Late\-13L29–41 \(Plateau\)131000\.880\.78\-2\.0%All\-42L0–41 \(Full model\)424200\.910\.79\-0\.1%OursL23–35 \(Transition\)131000\.930\.80BaselineOur method achieves comparable OOD performance to All\-42 while using 4\.2×\\timesfewer features\.![Refer to caption](https://arxiv.org/html/2604.16430v1/figures/GPEdiff.png)Figure 5:Illustration of Layer\-wise Energy Difference Evolution Across All 42 Layers\.Geometric Potential Energy \(GPE\) difference \(Hallucination \- Factual\) reveals three distinct phases: stable period \(L0–22, near\-zero difference with random fluctuations\), transition zone \(L23–35, sharp 20\.7\-fold escalation highlighted by shaded region\), and plateau period \(L36–41, sustained high\-energy state\)\. Error bars represent 95% confidence intervals \(bootstrap,nn=900 per group\)\.Ablation Study on the Hallucination\-related Sparse Feature Attribution Stage\.To rigorously validate the efficacy of the C\-DLA attribution mechanism, we compare it against two alternative feature attribution baselines under identical experimental conditions \(layers 23–35, top\-100 features selected\)\. As reported in Table[4](https://arxiv.org/html/2604.16430#S5.T4), C\-DLA achieves the highest in\-distribution AUC \(0\.93\) and feature purity \(89%\), substantially outperforming both the Wrong\-only DLA \(0\.88 AUC, 73% purity\) and the Correct\-only DLA \(0\.87 AUC, 68% purity\)\. The Wrong\-only variant considers only the contribution to incorrect tokens, potentially missing features that suppress correct answers, while the Correct\-only approach suffers from the opposite limitation\. In contrast, C\-DLA leverages a contrastive vectorΔ​𝐯=𝐖U​\[:,twrong\]−𝐖U​\[:,tcorrect\]\\Delta\\mathbf\{v\}=\\mathbf\{W\}\_\{U\}\[:,t\_\{\\text\{wrong\}\}\]\-\\mathbf\{W\}\_\{U\}\[:,t\_\{\\text\{correct\}\}\], enabling simultaneous capture of bidirectional causal pathways and precise localization of features driving the relative error advantage\. Fig\.[6](https://arxiv.org/html/2604.16430#S5.F6)further characterizes the distribution of feature importance as measured by C\-DLA scores\. The cumulative attribution curve exhibits a pronounced elbow: the top 0\.1% of features \(131 out of 131,072\) account for 41\.1% of total attribution strength, while the top 1% \(1,310 features\) cover 62\.5%\. The Gini coefficient of 0\.912—markedly higher than the random baseline of 0\.414—confirms that hallucination is governed by a small subset of high\-impact features, rather than diffuse contributions across the feature space\. This extreme concentration of causal attribution justifies our focus on the top 100 features \(0\.076% of the total\), ensuring that our detection pipeline targets the core mechanistic drivers of hallucination while effectively filtering out long\-tail noise\.

Table 4:Ablation Results on Feature Attribution Methods\.Comprehensive evaluation using Top\-100 features in transition zone \(L23–35\)\. All methods use identical layer ranges and feature budgets\.Table 5:Results of Joint Ablation Study\.Evaluating the synergistic effects of Stage I and Stage II\.ConfigurationLayers\# Feat\.AUCIDOODBaselineAll\-424200\.630\.63\+Stage I onlyL23–351300\.710\.64\+Stage II onlyAll\-424200\.850\.78Full \(I\+II\)L23–351000\.930\.80![Refer to caption](https://arxiv.org/html/2604.16430v1/x2.png)Figure 6:Illustration of Cumulative C\-DLA Contribution Curve \(Pareto Distribution\)\.The top\-0\.1% features \(131 out of 131,072\) account for 41\.1% of total attribution\. The top\-1% \(1,310\) cover 62\.5%\. The sharp elbow indicates extreme inequality in feature importance \(Gini coefficient = 0\.912 vs random baseline 0\.414\)\. Gray bars represent individual feature contributions, while the red curve shows cumulative percentage\.Joint Ablation: Synergy Between Stage I and II\.To assess the combined contribution of both stages, we conduct a joint ablation study\. Table[5](https://arxiv.org/html/2604.16430#S5.T5)demonstrates that neither stage alone is sufficient\. Using all layers with random feature selection achieves only 0\.632 OOD AUC, representing the baseline performance\. Adding Stage I \(transition zone localization\) with random features improves performance to 0\.644, while adding Stage II \(C\-DLA attribution\) on all layers reaches 0\.78\. The full system combining both stages achieves 0\.80 OOD AUC, confirming that Stage I and II provide complementary and necessary contributions to detection performance\.

## 6Conclusion

We present HalluSAE, a novel framework that interprets hallucination in LLMs as a phase transition in latent dynamics\. By tracking energy trajectories and focusing on sparse, causally significant features, HalluSAE enables interpretable and highly accurate hallucination detection\. Our results on Gemma\-2\-9B set a new standard for both in\-distribution and out\-of\-distribution scenarios\. A current limitation is that our analysis is confined to the Gemma architecture; future work will explore the generality of HalluSAE across a broader range of large language models\.

## 7Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning\. Our hallucination detection method can improve the reliability of LLMs in high\-stakes applications such as healthcare and legal services, reducing risks associated with factual errors\. However, practitioners should validate the approach on their specific models before deployment, as our analysis focuses on Gemma\-2\-9B\. While our work addresses hallucination detection, it does not resolve all fairness and bias concerns inherent in LLMs, which remain important areas for future research\.

## References

- Safe: a sparse autoencoder\-based framework for robust query enrichment and hallucination mitigation in llms\.arXiv preprint arXiv:2503\.03032\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- G\. Alain and Y\. Bengio \(2016\)Understanding intermediate layers using linear classifier probes\.arXiv preprint arXiv:1610\.01644\.Cited by:[§4\.3](https://arxiv.org/html/2604.16430#S4.SS3.p2.4)\.
- A\. Azaria and T\. Mitchell \(2023\)The internal state of an llm knows when it’s lying\.arXiv preprint arXiv:2304\.13734\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1),[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- K\. Benkirane, L\. Gongas, S\. Pelles, N\. Fuchs, J\. Darmon, P\. Stenetorp, D\. I\. Adelani, and E\. Sánchez \(2024\)Machine translation hallucination detection for low and high resource languages using large language models\.arXiv preprint arXiv:2407\.16470\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- T\. Bricken, A\. Templeton, J\. Batson, B\. Chen, A\. Jermyn, T\. Conerly, N\. Turner, C\. Anil, C\. Denison, A\. Askell, R\. Lasenby, Y\. Wu, S\. Kravec, N\. Schiefer, T\. Maxwell, N\. Joseph, Z\. Hatfield\-Dodds, A\. Tamkin, K\. Nguyen, B\. McLean, J\. E\. Burke, T\. Hume, S\. Carter, T\. Henighan, and C\. Olah \(2023\)Towards monosemanticity: decomposing language models with dictionary learning\.Transformer Circuits Thread\.Note:https://transformer\-circuits\.pub/2023/monosemantic\-features/index\.htmlCited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1),[§3](https://arxiv.org/html/2604.16430#S3.p2.2)\.
- C\. Chen, K\. Liu, Z\. Chen, Y\. Gu, Y\. Wu, M\. Tao, Z\. Fu, and J\. Ye \(2024\)INSIDE: llms’ internal states retain the power of hallucination detection\.arXiv preprint arXiv:2402\.03744\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- H\. Cunningham, A\. Ewart, L\. Riggs, R\. Huben, and L\. Sharkey \(2023\)Sparse autoencoders find highly interpretable features in language models\.arXiv preprint arXiv:2309\.08600\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1),[§3](https://arxiv.org/html/2604.16430#S3.p2.2)\.
- M\. Dahl, V\. Magesh, M\. Suzgun, and D\. E\. Ho \(2024\)Large legal fictions: profiling legal hallucinations in large language models\.Journal of Legal Analysis16\(1\),pp\. 64–93\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1),[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- S\. Dhuliawala, M\. Komeili, J\. Xu, R\. Raileanu, X\. Li, A\. Celikyilmaz, and J\. Weston \(2024\)Chain\-of\-verification reduces hallucination in large language models\.InFindings of the association for computational linguistics: ACL 2024,pp\. 3563–3578\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1),[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- X\. Du, C\. Xiao, and S\. Li \(2024\)Haloscope: harnessing unlabeled llm generations for hallucination detection\.Advances in Neural Information Processing Systems37,pp\. 102948–102972\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly,et al\.\(2021\)A mathematical framework for transformer circuits\.Transformer Circuits Thread1\(1\),pp\. 12\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1.1),[§1](https://arxiv.org/html/2604.16430#S1.p2.1),[§2](https://arxiv.org/html/2604.16430#S2.p2.1),[§4\.2](https://arxiv.org/html/2604.16430#S4.SS2.p2.2)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- R\. Friel and A\. Sanyal \(2023\)Chainpoll: a high efficacy method for llm hallucination detection\.arXiv preprint arXiv:2310\.18344\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1)\.
- A\. Frikha, M\. R\. A\. Razi, K\. K\. Nakka, R\. Mendes, X\. Jiang, and X\. Zhou \(2025\)Privacyscalpel: enhancing llm privacy via interpretable feature intervention with sparse autoencoders\.arXiv preprint arXiv:2503\.11232\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- L\. Gao, T\. D\. la Tour, H\. Tillman, G\. Goh, R\. Troll, A\. Radford, I\. Sutskever, J\. Leike, and J\. Wu \(2024\)Scaling and evaluating sparse autoencoders\.arXiv preprint arXiv:2406\.04093\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 5484–5495\.Cited by:[§3](https://arxiv.org/html/2604.16430#S3.p4.2)\.
- J\. Hu, G\. Tu, S\. Cheng, J\. Li, J\. Wang, R\. Chen, Z\. Zhou, and D\. Shan \(2025\)HARP: hallucination detection via reasoning subspace projection\.arXiv preprint arXiv:2509\.11536\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- Z\. Hua, J\. He, Z\. Yao, T\. Han, H\. Guo, Y\. Jia, and J\. Fang \(2025\)Steering lvlms via sparse autoencoder for hallucination mitigation\.arXiv preprint arXiv:2505\.16146\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin,et al\.\(2025a\)A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- Y\. Huang, J\. Zhang, Z\. Wang, B\. Bie, Y\. Qiu, Y\. R\. Fung, and X\. He \(2025b\)RePPL: recalibrating perplexity by uncertainty in semantic propagation and language generation for explainable qa hallucination detection\.arXiv preprint arXiv:2505\.15386\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM computing surveys55\(12\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1),[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- J\. Li, X\. Cheng, W\. X\. Zhao, J\. Nie, and J\. Wen \(2023a\)Halueval: a large\-scale hallucination evaluation benchmark for large language models\.arXiv preprint arXiv:2305\.11747\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p1.1)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023b\)Inference\-time intervention: eliciting truthful answers from a language model\.Advances in Neural Information Processing Systems36,pp\. 41451–41530\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1),[§4\.2](https://arxiv.org/html/2604.16430#S4.SS2.p3.4)\.
- Y\. Li, R\. Qiang, L\. Moukheiber, and C\. Zhang \(2025\)Language model uncertainty quantification with attention chain\.arXiv preprint arXiv:2503\.19168\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1)\.
- T\. Lieberum, S\. Rajamanoharan, A\. Conmy, L\. Smith, N\. Sonnerat, V\. Varma, J\. Kramár, A\. Dragan, R\. Shah, and N\. Nanda \(2024\)Gemma scope: open sparse autoencoders everywhere all at once on gemma 2\.arXiv preprint arXiv:2408\.05147\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1),[§3](https://arxiv.org/html/2604.16430#S3.p2.2),[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p3.2)\.
- Q\. Liu, X\. Chen, Y\. Ding, B\. Song, W\. Wang, S\. Wu, and L\. Wang \(2025\)Attention\-guided self\-reflection for zero\-shot hallucination detection in large language models\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21016–21032\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)Selfcheckgpt: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 9004–9017\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1),[§2](https://arxiv.org/html/2604.16430#S2.p2.1),[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- S\. Marks, C\. Rager, E\. J\. Michaud, Y\. Belinkov, D\. Bau, and A\. Mueller \(2024\)Sparse feature circuits: discovering and editing interpretable causal graphs in language models\.arXiv preprint arXiv:2403\.19647\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- S\. Marks and M\. Tegmark \(2023\)The geometry of truth: emergent linear structure in large language model representations of true/false datasets\.arXiv preprint arXiv:2310\.06824\.Cited by:[§3](https://arxiv.org/html/2604.16430#S3.p3.2)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in gpt\.Advances in neural information processing systems35,pp\. 17359–17372\.Cited by:[§3](https://arxiv.org/html/2604.16430#S3.p4.2)\.
- V\. Noël \(2025\)A graph signal processing framework for hallucination detection in large language models\.arXiv preprint arXiv:2510\.19117\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- H\. Orgad, M\. Toker, Z\. Gekhman, R\. Reichart, I\. Szpektor, H\. Kotek, and Y\. Belinkov \(2024\)Llms know more than they show: on the intrinsic representation of llm hallucinations\.arXiv preprint arXiv:2410\.02707\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- S\. Park, S\. Yoo, J\. Mok, and S\. Yoon \(2025\)SAVE: sparse autoencoder\-driven visual information enhancement for mitigating object hallucination\.arXiv preprint arXiv:2512\.07730\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl,et al\.\(2023\)Large language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1),[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- G\. Sriramanan, S\. Bharti, V\. S\. Sadasivan, S\. Saha, P\. Kattakinda, and S\. Feizi \(2024\)Llm\-check: investigating detection of hallucinations in large language models\.Advances in Neural Information Processing Systems37,pp\. 34188–34216\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1)\.
- X\. Wang, Y\. Yan, L\. Huang, X\. Zheng, and X\. Huang \(2023\)Hallucination detection for generative large language models by bayesian sequential estimation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 15361–15371\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§4\.2](https://arxiv.org/html/2604.16430#S4.SS2.p2.2),[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- J\. Wei, Y\. Tay, R\. Bommasani, C\. Raffel, B\. Zoph, S\. Borgeaud, D\. Yogatama, M\. Bosma, D\. Zhou, D\. Metzler,et al\.\(2022\)Emergent abilities of large language models\.arXiv preprint arXiv:2206\.07682\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p1.1),[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- Y\. Xiao and W\. Y\. Wang \(2021\)On hallucination and predictive uncertainty in conditional language generation\.arXiv preprint arXiv:2103\.15025\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- G\. Xiong, Z\. He, B\. Liu, S\. Sinha, and A\. Zhang \(2025\)Toward faithful retrieval\-augmented generation with sparse autoencoders\.arXiv preprint arXiv:2512\.08892\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p3.1)\.
- W\. Xu, S\. Agrawal, E\. Briakou, M\. J\. Martindale, and M\. Carpuat \(2023\)Understanding and detecting hallucinations in neural machine translation via model introspection\.Transactions of the Association for Computational Linguistics11,pp\. 546–564\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p2.1)\.
- M\. Yuksekgonul, V\. Chandrasekaran, E\. Jones, S\. Gunasekar, R\. Naik, H\. Palangi, E\. Kamar, and B\. Nushi \(2023\)Attention satisfies: a constraint\-satisfaction lens on factual errors of language models\.arXiv preprint arXiv:2309\.15098\.Cited by:[§1](https://arxiv.org/html/2604.16430#S1.p2.1)\.
- L\. Zhang, D\. Song, Z\. Wu, Y\. Tian, C\. Zhou, J\. Xu, Z\. Yang, and S\. Zhang \(2025a\)Detecting hallucination in large language models through deep internal representation analysis\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence, IJCAI\-25,pp\. 8357–8365\.Cited by:[§5\.1](https://arxiv.org/html/2604.16430#S5.SS1.p2.1)\.
- Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen,et al\.\(2023\)Siren’s song in the ai ocean: a survey on hallucination in large language models\.arXiv e\-prints,pp\. arXiv–2309\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- Y\. Zhang, S\. Li, C\. Qian, J\. Liu, P\. Yu, C\. Han, Y\. R\. Fung, K\. McKeown, C\. Zhai, M\. Li,et al\.\(2025b\)The law of knowledge overshadowing: towards understanding, predicting, and preventing llm hallucination\.arXiv preprint arXiv:2502\.16143\.Cited by:[§2](https://arxiv.org/html/2604.16430#S2.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§3](https://arxiv.org/html/2604.16430#S3.p3.2)\.

## Appendix ADataset and Experimental Configuration

### A\.1Dataset Construction and Quality Control Pipeline

Our datasets undergo rigorous quality control to ensure reliable annotations\. Fig\.[7](https://arxiv.org/html/2604.16430#A1.F7)illustrates the complete cleaning workflow: raw data collection, GPT\-4o automatic annotation \(protocol in Appendix[A\.3](https://arxiv.org/html/2604.16430#A1.SS3)\), human expert review, ambiguous sample removal, and final dataset construction\.

![Refer to caption](https://arxiv.org/html/2604.16430v1/figures/dataset_pipeline.png)Figure 7:Data Cleaning Pipeline\.Five\-stage workflow from raw data to curated dataset\.#### Dataset Statistics and Quality Control\.

Table[6](https://arxiv.org/html/2604.16430#A1.T6)presents comprehensive statistics for both datasets\. From initial pools of 1,876 \(HaluEval\) and 4,092 \(TriviaQA\) samples, we remove 76 and 92 ambiguous cases respectively, yielding 1,800 and 4,000 high\-quality samples\. The removed ambiguous samples are classified into four categories:missing units\(30\.3%, e\.g\., “330” without specifying meters\),granularity mismatch\(25\.0%, e\.g\., “China” vs\. “Beijing”\),partially correctanswers \(23\.7%, incomplete responses\), andannotation conflicts\(21\.0%, GPT\-4o and human disagreement\)\. HaluEval serves both training \(1,260 samples, 70%\) and testing \(270 samples, 15%\), while TriviaQA is used exclusively for out\-of\-distribution evaluation\.

Table 6:Dataset Statistics and Comparison

### A\.2Model Architecture and SAE Configuration Details

Table[7](https://arxiv.org/html/2604.16430#A1.T7)summarizes the complete configuration for Gemma\-2\-9B and Gemma Scope SAEs\. All experiments use greedy decoding \(temperature=0\) and Float32 precision for reproducibility\. The SAEs provide 131,072\-dimensional sparse representations \(36\.6×\\timesovercomplete\) with JumpReLU activation\. Table[8](https://arxiv.org/html/2604.16430#A1.T8)details the layer\-wise sparsity \(L0L\_\{0\}\) configuration across all 42 layers, with the transition zone \(L23–35\) exhibiting averageL0≈31L\_\{0\}\\approx 31, consistent with the global mean\.

Table 7:Detailed Hardware and Memory ConfigurationTable 8:Layer\-wise SAE Sparsity \(L0L\_\{0\}\) Configuration
### A\.3GPT\-4o Automatic Annotation Protocol and Design Principles

To ensure reproducibility, we provide the complete GPT\-4o annotation prompt in Fig\.[8](https://arxiv.org/html/2604.16430#A1.F8)\. The prompt incorporates four key design principles not detailed in the main text: explicit grounding \(prevent judge hallucination\), numerical tolerance \(handle formatting differences\), structured reasoning \(ensure reproducibility\), and JSON output \(automated parsing\)\. All samples undergo two\-stage annotation: GPT\-4o automatic labeling followed by mandatory human expert review, with human judgment serving as the final ground truth in case of conflicts\.

GPT\-4o Annotation Prompt Template```
You are an expert fact-checker for Large Language Models. Verify whether
the [Model Response] contains factual hallucinations based strictly on
the [Reference Knowledge] and [Ground Truth].

Input Data:
[Reference Knowledge]: {knowledge}
[Question]: {question}
[Ground Truth]: {ground_truth}
[Model Response]: {gemma_response}

Evaluation Criteria:
1. Entity Verification: Check if core entities (names, places, dates) match.
2. Numerical Precision: Allow minor formatting differences (e.g., "20%" vs
   "20 percent"), but mark significant deviations as incorrect.
3. Contradiction Check: If response contradicts reference knowledge, mark INCORRECT.
4. Relevance: If response is irrelevant or incomplete, mark INCORRECT.

Step-by-Step Reasoning:
1. Identify the key claim in [Ground Truth].
2. Extract the corresponding claim from [Model Response].
3. Compare and explicitly state discrepancies.
4. Determine final verdict.

Output Format (JSON):
{
    "reasoning": "Concise explanation highlighting specific errors if any.",
    "label": "CORRECT" or "INCORRECT"
}
```

Figure 8:GPT\-4o Annotation Prompt\.Complete template with explicit grounding, numerical tolerance, structured reasoning, and JSON output\.
### A\.4Baseline Methods: Implementation and Hyperparameter Configuration

All baselines are evaluated under identical conditions: same hardware \(8× RTX 4090\), dataset \(HaluEval 1,260 training samples for supervised methods\), preprocessing, metrics, and random seed \(42\)\. Table[9](https://arxiv.org/html/2604.16430#A1.T9)summarizes the configuration for all eight baseline methods\. Hyperparameters are selected via 5\-fold cross\-validation, maximizing validation AUC\. Threshold\-based methods \(LN\-Entropy, Semantic Entropy, Lexical Similarity, SelfCheckGPT, EigenScore, MHAD\) determine optimal thresholds on the validation set\.

Table 9:Hyperparameter Configuration for Baseline Methods#### Key Implementation Details\.

Uncertainty methods:LN\-Entropy computes entropy on final logits with log\-normalization; Semantic Entropy clusters 10 nucleus\-sampled outputs using Sentence\-BERT embeddings and agglomerative clustering\.Consistency methods:Lexical Similarity computes ROUGE\-L F1 across 5 temperature\-sampled outputs; SelfCheckGPT uses 5 nucleus\-sampled outputs with BERTScore\-based verification\.Internal state methods:EigenScore analyzes eigenvalue spectrum of Layer 35 attention matrices; MHAD computes Jensen\-Shannon divergence across 16 attention heads in 4 key layers\.Supervised methods:SAPLMA trains a linear probe on Layer 35 residual stream \(3,584\-dim\); HaloScope uses a 2\-layer MLP on concatenated features from 6 layers \(21,504\-dim\)\. Our method achieves 215× feature reduction \(100 vs\. 21,504\) and 4\.2× layer reduction \(13 vs\. 42 full layers\) while maintaining competitive computational cost \(2\.3×\) and superior performance \(Table[1](https://arxiv.org/html/2604.16430#S5.T1)in main paper\)\.

## Appendix BRobustness and Ablation Studies

### B\.1Robustness of Transition Zone Localization

To validate the stability of the identified transition zone \(L23–35\), we conduct two independent experiments: parameter sensitivity analysis and bootstrap stability analysis\.

#### Parameter Sensitivity Analysis\.

We systematically vary the localization criteria to test whether the identified zone is sensitive to hyperparameter choices\. Specifically, we construct a parameter grid by varying: \(1\) the starting criterion—requiringkkconsecutive layers with positive growth rateγℓ\>θ\\gamma\_\{\\ell\}\>\\theta, wherek∈\{2,3,4\}k\\in\\\{2,3,4\\\}andθ∈\{0%,10%,20%,50%\}\\theta\\in\\\{0\\%,10\\%,20\\%,50\\%\\\}; and \(2\) the endpoint criterion—allowing peak tolerance ofτ∈\{5%,10%,20%,30%\}\\tau\\in\\\{5\\%,10\\%,20\\%,30\\%\\\}, resulting in 42 total configurations\. Our baseline configuration \(used in the main text\) isk=3k=3,θ=0%\\theta=0\\%,τ=10%\\tau=10\\%\.

Table[10](https://arxiv.org/html/2604.16430#A2.T10)presents six representative configurations\. The results demonstrate exceptional robustness: 30 out of 42 configurations \(71\.4%\) achieve perfect identification \(IoU = 1\.0\), with a mean IoU of 0\.956 across all configurations\. No configuration falls below IoU = 0\.846, indicating that the transition zone remains stable across a wide range of parameter settings\. The start boundary exhibits high stability \(23\.6±0\.923\.6\\pm 0\.9layers\), while the endpoint is perfectly locked at L35 \(35\.0±0\.035\.0\\pm 0\.0layers\) across most configurations\.

Table 10:Parameter Sensitivity Analysis: Representative ConfigurationsConfigStartEnd Tol\.IoUIdentifiedLengthIDMult\.\(%\)Zone11\.051\.000L23–351371\.151\.000L23–351319 \(Baseline\)1\.3101\.000L23–3513311\.5251\.000L23–3513371\.850\.846L25–3511422\.0300\.846L25–3511Summary: 30/42 configs achieve IoU=1\.0; mean IoU=0\.956; all≥\\geq0\.846
#### Bootstrap Stability Analysis\.

We perform 1,000 bootstrap resampling iterations, where each iteration randomly selects 80% of samples \(1,440 out of 1,800\) with replacement and re\-runs the localization algorithm using the baseline criterion\. Aggregate statistics show: mean IoU =0\.925±0\.1100\.925\\pm 0\.110, median IoU = 1\.000, indicating that over half of resamples achieve perfect identification\. The high\-overlap rate \(IoU≥\\geq0\.8\) reaches 68\.1% \(681/1,000 iterations\), and even the worst case maintains moderate overlap \(minimum IoU = 0\.647\)\.

Boundary analysis reveals exceptional stability: the start point is23\.0±0\.123\.0\\pm 0\.1layers \(mode: L23, range: L23–L25\), while the endpoint is36\.3±1\.936\.3\\pm 1\.9layers \(mode: L35, range: L35–L39\)\. Table[11](https://arxiv.org/html/2604.16430#A2.T11)shows layer\-wise stability, where all 13 ground truth layers exhibit stability≥\\geq0\.998, with 11 core layers \(L25–L35\) achieving perfect stability \(1\.000\)\.

Table 11:Bootstrap Stability Analysis: Layer\-wise Results
#### Statistical Significance Testing\.

To quantify the magnitude of phase transitions, we compute Cohen’sddeffect size between adjacent phases, defined as:

d=μ1−μ2\(σ12\+σ22\)/2d=\\frac\{\\mu\_\{1\}\-\\mu\_\{2\}\}\{\\sqrt\{\(\\sigma\_\{1\}^\{2\}\+\\sigma\_\{2\}^\{2\}\)/2\}\}\(10\)whereμ\\muandσ\\sigmadenote the mean and standard deviation of GPE differences in each phase\. Table[12](https://arxiv.org/html/2604.16430#A2.T12)presents the complete hypothesis testing results\. The transition zone \(Phase II\) exhibits very large effect sizes compared to both the stable period \(Phase I,d=1\.64d=1\.64,p<0\.001p<0\.001\) and the plateau period \(Phase III\), while Phase III shows negligible difference from Phase II \(d=0\.12d=0\.12,p=0\.342p=0\.342\), confirming that L35 marks the energy peak\. These results provide strong statistical evidence that the identified boundaries represent genuine qualitative shifts in the model’s internal dynamics rather than arbitrary thresholds\.

Table 12:Statistical Significance Testing Between PhasesComparisonCohen’sddtt\-statdfpp\-valueInterpretationPhase II vs I1\.6418\.321798<<0\.001Very large effectPhase III vs II0\.120\.9517980\.342No effectPhase III vs I1\.7119\.871798<<0\.001Very large effectPhase I: L0–22 \(Stable\); Phase II: L23–35 \(Transition\); Phase III: L36–41 \(Plateau\)

### B\.2Feature Attribution Method Comparison and Validation

#### C\-DLA vs Unidirectional Attribution Baselines\.

To validate the necessity of the contrastive mechanism, we compare our Contrastive Direct Logit Attribution \(C\-DLA\) with two unidirectional baselines under identical settings \(layers L23–35, top\-100 features selected\): \(1\)Wrong\-only DLA, which measures only the contribution to the incorrect token𝒜i\(twrong\)\\mathcal\{A\}\_\{i\}^\{\(t\_\{\\text\{wrong\}\}\)\}; and \(2\)Correct\-only DLA, which measures only the suppression of the correct token−𝒜i\(tcorrect\)\-\\mathcal\{A\}\_\{i\}^\{\(t\_\{\\text\{correct\}\}\)\}\.

Table[13](https://arxiv.org/html/2604.16430#A2.T13)shows that C\-DLA achieves superior performance across all metrics\. Most notably, C\-DLA improves feature purity—defined as the percentage of high\-activation samples \(top\-10%\) that are hallucinations—from 68–73% \(unidirectional methods\) to 89%\. Pairedtt\-test confirms that the performance difference between C\-DLA and Wrong\-only DLA is statistically significant \(t=8\.32t=8\.32,p<0\.001p<0\.001, based on 5\-fold cross\-validation results\)\. This demonstrates that hallucination detection requires capturing therelative advantageof incorrect over correct outputs, rather than examining either direction in isolation\.

Table 13:Detailed Comparison of Attribution Methods
#### Random Feature Baseline\.

To verify that C\-DLA’s feature selection captures genuine causal structure rather than statistical coincidence, we compare against a random baseline\. We randomly select 100 features from the same layer range \(L23–35\) across 10 different random seeds, ensuring that the selected features have comparable activation strength to the C\-DLA features \(mean activation≥\\geq5\.0\)\.

The random baseline achieves an average AUC of0\.623±0\.0370\.623\\pm 0\.037\(mean±\\pmstd across 10 seeds\) on the OOD test set, while C\-DLA achieves 0\.840\. This represents a performance gap of 0\.217 AUC points, with C\-DLA exceeding the random baseline by 3\.8 standard deviations \(z=0\.840−0\.6230\.037≈5\.86z=\\frac\{0\.840\-0\.623\}\{0\.037\}\\approx 5\.86,p<10−8p<10^\{\-8\}\)\. This provides strong evidence that the features identified by C\-DLA have true causal influence on hallucination generation, rather than merely reflecting arbitrary activation patterns\.

### B\.3Joint Ablation Study: Stage I × Stage II Interaction Effects

To assess whether Stage I: Potential Energy Empowered Phase Zone Localization and Stage II: Hallucination\-related Sparse Feature Attribution \(C\-DLA\) provide complementary or redundant contributions, we conduct a2×22\\times 2factorial experiment\. Table[14](https://arxiv.org/html/2604.16430#A2.T14)presents the complete results\.

Table 14:Joint Ablation: Stage I×\\timesStage II Interaction#### Marginal Effect Decomposition\.

The individual contribution of Stage I: Potential Energy Empowered Phase Zone Localization is modest \(\+1\.1% OOD AUC\), as randomly selected features from even the critical layers lack semantic focus\. Stage II \(C\-DLA attribution\) provides the dominant improvement \(\+15\.0%\), demonstrating thatwhich featuresto monitor is more critical thanwhich layersto examine\. However, the full system \(I\+II\) achieves \+17\.4%, exhibiting a synergistic gain of \+1\.4% beyond the sum of individual contributions \(1\.1% \+ 15\.0% = 16\.1%\)\.

To test whether this interaction is statistically significant, we perform a two\-way ANOVA with factors Stage I \(off/on\) and Stage II \(off/on\), using the 5\-fold cross\-validation results as repeated measures\. The interaction term is significant \(F​\(1,356\)=12\.3F\(1,356\)=12\.3,p=0\.008p=0\.008\), indicating that the combination of layer localization and feature attribution produces super\-additive effects\. This validates our three\-stage pipeline design, where coarse\-to\-fine refinement is essential for optimal performance\.

### B\.4Validation of Microscopic\-Macroscopic Feature\-Energy Correspondence

#### Global Synchronization\.

To validate that the microscopic feature\-level attribution \(C\-DLA\) and the macroscopic energy\-level dynamics \(GPE\) reflect the same underlying phase transition mechanism, we compute the Pearson correlation between layer\-wise cumulative C\-DLA scores and GPE differences \(Hallucination \- Factual\) across all 42 layers\. The correlation is extremely strong \(r=0\.990r=0\.990,p<0\.001p<0\.001\), indicating near\-perfect alignment between the two metrics\.

Additionally, we quantify the inequality of feature contributions using the Gini coefficient:

G=∑i=1n∑j=1n\|xi−xj\|2​n2​x¯G=\\frac\{\\sum\_\{i=1\}^\{n\}\\sum\_\{j=1\}^\{n\}\|x\_\{i\}\-x\_\{j\}\|\}\{2n^\{2\}\\bar\{x\}\}\(11\)wherexix\_\{i\}represents the C\-DLA score of theii\-th feature\. Our observed Gini coefficientGobs=0\.912G\_\{\\text\{obs\}\}=0\.912far exceeds the random baselineGrandom=0\.414G\_\{\\text\{random\}\}=0\.414\(computed from 1,000 random permutations\), confirming that feature contributions follow a highly skewed Pareto distribution rather than a uniform distribution\.

#### Phase\-Specific Correlation Analysis\.

To test whether the correlation between C\-DLA and GPE is uniformly strong across all layers or specific to the transition zone, we compute segment\-wise Pearson correlations for three phases\. Table[15](https://arxiv.org/html/2604.16430#A2.T15)shows that the strong correlation isexclusivelyobserved in the transition zone \(L23–35,r=0\.990r=0\.990\), while both the stable period \(L0–22,r=0\.341r=0\.341,p=0\.112p=0\.112\) and plateau period \(L36–41,r=0\.287r=0\.287,p=0\.581p=0\.581\) exhibit no significant correlation\.

Table 15:Phase\-Specific Correlation Between C\-DLA and GPETo further validate that the transition zone exhibits significantly higher correlation than other regions, we apply Fisher’sZZ\-transformation:

Z=Zr1−Zr21n1−3\+1n2−3,where​Zr=12​ln⁡\(1\+r1−r\)Z=\\frac\{Z\_\{r\_\{1\}\}\-Z\_\{r\_\{2\}\}\}\{\\sqrt\{\\frac\{1\}\{n\_\{1\}\-3\}\+\\frac\{1\}\{n\_\{2\}\-3\}\}\},\\quad\\text\{where \}Z\_\{r\}=\\frac\{1\}\{2\}\\ln\\left\(\\frac\{1\+r\}\{1\-r\}\\right\)\(12\)Comparing the transition zone \(r=0\.990r=0\.990,n=13n=13\) against the combined stable and plateau periods \(r=0\.314r=0\.314,n=29n=29\), we obtainZ=6\.74Z=6\.74\(p<0\.001p<0\.001\), confirming that the correlation in the transition zone is statistically distinguishable from the rest of the network\.

This phase\-specific synchronization provides compelling evidence that: \(1\) C\-DLA successfully captures the microscopic driving mechanism of the macroscopic energy explosion observed in the main text \(Figure 4\), and \(2\) the identified features havecausal specificity—they are activated precisely when and where the phase transition occurs, rather than reflecting global activation patterns throughout the network\.

## Appendix CDetector Implementation Details

### C\.1Hallucination Detector: Training Configuration and Hyperparameters

#### Model Architecture\.

Our hallucination detector employs Logistic Regression with L1 regularization, trained on features extracted from the transition zone \(L23–35\)\. The input consists of 100\-dimensional vectors corresponding to the top\-100 features identified in Stage II, and the output is a binary classification \(hallucination vs factual\)\.

#### Hyperparameter Configuration\.

We perform 5\-fold cross\-validation on the HaluEval training set \(1,260 samples: 630 hallucination \+ 630 factual\) to select the optimal regularization strengthCC\. The search space spans\[10−4,104\]\[10^\{\-4\},10^\{4\}\]with 20 logarithmically\-spaced candidate values\. Additional training configurations include:

- •Solver:liblinear\(optimized for L1 regularization and small\-scale datasets\)
- •Maximum iterations: 1,000
- •Convergence tolerance:10−410^\{\-4\}
- •Class weights:balanced\(automatically handles class imbalance by inverse frequency\)
- •Feature preprocessing: StandardScaler applied to training set, then used to transform validation and test sets \(zero mean, unit variance\)
- •Evaluation metric: Validation AUC \(primary criterion for hyperparameter selection\)

The optimal configuration identified through cross\-validation isC=0\.1C=0\.1, achieving the highest mean validation AUC across all folds\.

#### Training Pipeline\.

For each fold in the 5\-fold cross\-validation:

1. 1\.Split the 1,260 training samples into 1,008 train and 252 validation samples
2. 2\.Extract 100\-dimensional feature vectors from the transition zone using the pre\-trained SAE
3. 3\.Standardize features using the training set statistics
4. 4\.Train Logistic Regression with candidateCCvalues
5. 5\.Evaluate on the validation set and record AUC

After identifying the optimalCC, we retrain the final detector on the full training set \(1,260 samples\) and evaluate on the held\-out test set \(360 samples\)\.

Table[16](https://arxiv.org/html/2604.16430#A3.T16)summarizes the complete configuration\.

Table 16:Complete Detector Configuration

### C\.2Computational Cost Analysis and Efficiency Comparison

#### Inference\-Time Complexity Comparison\.

Table[17](https://arxiv.org/html/2604.16430#A3.T17)compares the computational cost of our method against representative baselines\. All costs are normalized relative to standard forward propagation without detection \(baseline = 1\.0×\\times\)\.

Table 17:Computational Cost Comparison Across Methods
#### Cost Breakdown of Our Method\.

The 2\.3×\\timesoverhead of our approach decomposes as follows:

- •Standard forward propagation: 1\.0×\\times
- •SAE feature extraction \(13 layers, L23–35\): \+0\.8×\\times
- •Probe prediction \(100\-dim linear layer\): \+0\.5×\\times
- •Total: 2\.3×\\times

Compared to consistency\-based methods like SelfCheckGPT \(5–10×\\timesoverhead due to multiple generations\), our method achieves a favorable efficiency\-accuracy trade\-off\. Compared to full\-layer probing \(7\.2×\\timesoverhead\), our transition zone localization \(Stage I\) reduces computational cost by 68% while maintaining comparable or superior performance\.

#### One\-Time Training Cost\.

The three\-stage pipeline incurs the following one\-time costs on the training set:

- •Stage I\(Energy analysis for transition zone localization\):∼\\sim3 GPU\-hours
- •Stage II\(C\-DLA computation and feature selection\):∼\\sim5 GPU\-hours
- •Stage III\(Probe training via cross\-validation\):<<1 GPU\-hour
- •Total:∼\\sim8 GPU\-hours on 8×\\timesNVIDIA RTX 4090 \(48GB each\)

These costs are incurred only once during training\. At inference time, the trained probe can be directly applied to new samples without re\-computing energy analysis or C\-DLA scores, requiring only standard forward propagation plus SAE feature extraction \(2\.3×\\timesoverhead\)\.