Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

# Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization
Source: [https://arxiv.org/html/2606.03022](https://arxiv.org/html/2606.03022)
Mingkuan Zhao1,Wentao Hu1,211footnotemark:1,Tianchen Huang4,Yuheng Min5,Suquan Chen1, Yide Gao1,Yanbo Zhai1,Shuangyong Song2,Xuelong Li3 1Xi’an Jiaotong University,2Xingchen AGI Lab, China Telecom AI Technology \(Beijing\) Co\., Ltd\.,3Institute of Artificial Intelligence, China Telecom,4University of Science and Technology of China,5Tsinghua University \{mingkuanzhao, wentao\_hu, yanbozhai, suquanchen, yidegao\}@stu\.xjtu\.edu\.cn,tchuang@mail\.ustc\.edu\.cn,minyh24@mails\.tsinghua\.edu\.cn,songshy@chinatelecom\.cn,xuelong\_li@ieee\.org

###### Abstract

Hallucination in Large Language Models \(LLMs\)—characterized by the generation of content inconsistent with contextual facts or logical constraints—remains a persistent challenge for reliable deployment\. In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis\. We propose that hallucinations manifest asorthogonal noiserelative to the semantic manifold of the residual stream\. Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation\.

Based on this formulation, we introduceDynamic Contextual Orthogonalization \(DCO\), an inference\-time intervention method\. DCO utilizes the input residual stream as a dynamic context anchor to perform orthogonal decomposition on attention head outputs\. To distinguish between context\-aligned semantic updates and divergent noise, DCO employs a layer\-wise Z\-score suppression mechanism that selectively attenuates outlier orthogonal components based on statistical distributions\.

Evaluations on Llama\-3\-8B and 70B across benchmarks such as XSum, NQ\-Swap, and IFEval demonstrate that DCO achieves superior contextual faithfulness compared to state\-of\-the\-art intervention baselines\. Furthermore, DCO maintains high performance on knowledge\-intensive tasks like TriviaQA and TruthfulQA, effectively mitigating the trade\-off between hallucination suppression and parametric knowledge retention often observed in existing methods\. Our findings validate the geometric interpretation of hallucinations and establish DCO as a computationally efficient approach for enforcing manifold alignment\. Our code is available at[https://github\.com/Harry\-Miral/DCO](https://github.com/Harry-Miral/DCO)\.

Hallucinations as Orthogonal Noise: Inference\-Time Manifold Alignment via Dynamic Contextual Orthogonalization

## 1Introduction

The deployment of Large Language Models \(LLMs\) in critical domains is hindered by the persistence of hallucinations, where generated content diverges from contextual facts or logical constraints\. While current mitigation strategies such as Retrieval\-Augmented Generation \(RAG\) or contrastive decoding effectively modulate output probabilities, they often operate without a mechanistic view of the internal representation dynamics\. We posit that hallucinations are not merely surface\-level statistical anomalies but manifestations of geometric misalignment within the model’s latent space, specifically when the model processes conflicting or ambiguous contexts\.

This study approaches the hallucination problem through theLinear Representation Hypothesis\(Park et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib28)\), which suggests that semantic concepts are encoded as linear directions in high\-dimensional space\(Gurnee and Tegmark,[2024](https://arxiv.org/html/2606.03022#bib.bib9)\)\. Under this framework, we formalize the faithfulness of generation as a manifold alignment task\. Ideally, attention heads should propagate information that lies parallel to the subspace defined by the context\. We hypothesize that hallucinations arise fromOrthogonal Noise Injection—a phenomenon where specific attention heads introduce components orthogonal to the established context manifold, thereby steering the latent state evolution away from the factual trajectory\.

![Refer to caption](https://arxiv.org/html/2606.03022v1/figure1.png)Figure 1:Conceptual geometry of Dynamic Contextual Orthogonalization \(DCO\)\.A hallucinating attention head produces an output vector𝐨h\\mathbf\{o\}\_\{h\}that diverges from the contextual semantic manifoldℳ\\mathcal\{M\}\. DCO decomposes𝐨h\\mathbf\{o\}\_\{h\}into a context\-aligned resonance component𝐨h∥\\mathbf\{o\}\_\{h\\parallel\}and an orthogonal noise component𝐨h⟂\\mathbf\{o\}\_\{h\\perp\}\. By dynamically attenuating𝐨h⟂\\mathbf\{o\}\_\{h\\perp\}based on layer\-wise statistical distributions, DCO realigns the latent representation with the established manifold\.Based on this geometric formulation, we introduceDynamic Contextual Orthogonalization \(DCO\), an inference\-time intervention method\. Unlike static steering vectors that apply a fixed direction, DCO utilizes the input residual stream of each layer as a dynamiccontext anchor\. The method performs orthogonal decomposition on attention head outputs to isolate components that deviate from the global semantic consensus\. Crucially, to distinguish between necessary information diversification and harmful noise, DCO employs a layer\-wise Z\-Score suppression mechanism\. This adaptive filtering is derived from the observation that hallucinatory signals manifest as statistical outliers in the orthogonality distribution, necessitating a dynamic rather than static threshold to preserve the model’s parametric knowledge retrieval capabilities\.

We evaluate DCO on the Llama\-3\-8B and 70B models across a diverse set of benchmarks\. Empirical results indicate that DCO achieves superior performance compared to state\-of\-the\-art intervention methods, including Inference\-Time Intervention \(ITI\) and Decoding by Contrasting Layers \(DoLa\)\. Specifically, DCO significantly enhances contextual faithfulness on XSum and IFEval while maintaining robustness on knowledge\-intensive tasks such as TriviaQA and MuSiQue\. These findings demonstrate that enforcing geometric constraints in the residual stream offers a computationally efficient and mechanistically grounded approach to hallucination mitigation, achieving an optimal trade\-off between faithfulness and general capability\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2606.03022v1/figure4.png)Figure 2:Mechanistic comparison of intervention paradigms\.\(A\) Static Intervention \(ITI\):Injects a fixed steering vector derived from offline probing, applying indiscriminate directionality to the residual stream regardless of context\.\(B\) Logit Contrast \(DoLa/CAD\):Operates at the output layer by subtracting premature/amateur logits from mature layer logits, enhancing sharpness but failing to correct internal representation dynamics\.\(C\) Geometric Alignment \(DCO, Ours\):Performs dynamic intervention within the latent space\. It treats hallucination as an orthogonal noise component𝐨h⟂\\mathbf\{o\}\_\{h\\perp\}diverging from the context manifoldℳ\\mathcal\{M\}and projects the attention output𝐨h\\mathbf\{o\}\_\{h\}to align with the valid semantic subspace𝐨h∥\\mathbf\{o\}\_\{h\\parallel\}\.#### Mechanistic Interpretability of Hallucinations\.

Understanding the computational underpinnings of Transformer models is essential for diagnosing generation failures\. Grounded in the view of the residual stream as a communication channel modulated by independent attention heads\(Dar et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib3)\), mechanistic interpretability has identified specific components governing information flow\. Key findings include the characterization of induction heads\(Olsson et al\.,[2022](https://arxiv.org/html/2606.03022#bib.bib27)\)and “retrieval heads” that extract factual content from long contexts\(Wu et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib33)\)\. Moreover, research on knowledge localization\(Meng et al\.,[2022](https://arxiv.org/html/2606.03022#bib.bib24); Geva et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib7)\)elucidates how factual associations are retrieved via the interplay between MLP layers and attention mechanisms\. Recent work byNanda et al\. \([2023](https://arxiv.org/html/2606.03022#bib.bib25)\)andYu et al\. \([2024](https://arxiv.org/html/2606.03022#bib.bib36)\)further distinguishes between early\-layer knowledge gaps and upper\-layer retrieval failures as distinct causes of hallucination\. Complementary to this,Zhao et al\. \([2025b](https://arxiv.org/html/2606.03022#bib.bib39)\)demonstrate that attention heads exhibit heterogeneous importance, showing that sparse attention patterns can be enforced without a speed\-performance trade\-off—an observation that supports our hypothesis of differential head behavior in the context of hallucinatory noise\. Our work operationalizes these insights, translating the theoretical understanding of attention head dynamics into a direct intervention method within the residual stream\.

#### Inference\-Time Hallucination Mitigation\.

Various training\-free decoding strategies have been developed to intervene during inference\. Methods such as Contrastive Decoding \(CD\)\(Li et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib18)\)and its variants, including DoLa\(Chuang et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib2)\)and Autocontrastive Decoding\(Gera et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib6)\), adjust next\-token probabilities by contrasting distributions across model layers or scales\. Addressing contextual faithfulness specifically,Shi et al\. \([2024](https://arxiv.org/html/2606.03022#bib.bib29)\)proposed Context\-Aware Decoding \(CAD\) to amplify evidence\-based logits, whileChen et al\. \([2024](https://arxiv.org/html/2606.03022#bib.bib1)\)utilized in\-context sharpness as a detection signal\. More recently,Gema et al\. \([2024](https://arxiv.org/html/2606.03022#bib.bib5)\)introduced DeCoRe to mask retrieval heads based on static importance scores\. Crucially, these approaches predominantly operate at the output logit level or employ static interventions\(as illustrated in Figure[2](https://arxiv.org/html/2606.03022#S2.F2)\)\. In contrast, guided by theLinear Representation Hypothesis\(Park et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib28); Hernandez et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib11)\), DCO implements adynamicorthogonalization mechanism\. Unlike Inference\-Time Intervention \(ITI\)\(Li et al\.,[2024a](https://arxiv.org/html/2606.03022#bib.bib15)\), which applies fixed steering vectors derived from offline probing, DCO dynamically calibrates the intervention based on the instantaneous geometry of the residual stream at each step\.

#### Geometric Perspectives on Semantic Consistency\.

We conceptualize generative faithfulness as a manifold alignment problem\. In contexts involving knowledge conflicts or long\-range dependencies, parametric memory often interferes with contextual extraction\(Longpre et al\.,[2021](https://arxiv.org/html/2606.03022#bib.bib23); Liu et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib21)\)\. While Representation Engineering\(Zou et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib41)\)demonstrates the utility of top\-down linear control, the geometric interpretation of hallucinations as orthogonal interference remains under\-explored\. DCO addresses this by reformulating factual consistency—evaluated via metrics such as FactKB\(Feng et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib4)\)—as the filtering of orthogonal noise\. By integrating challenges identified in summarization benchmarks like XSum\(Narayan et al\.,[2018](https://arxiv.org/html/2606.03022#bib.bib26)\)into this framework, we provide a unified geometric approach that preserves model integrity while suppressing noise, offering a mechanistic alternative to probability\-based corrections\.

#### Large Language Model Development and Downstream Applications\.

The rapid scaling of LLMs, as exemplified by bilingual model series such as TeleChat\(He et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib10); Wang et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib31)\)and their successors TeleChat2 and TeleChat2\.5\(Wang et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib32)\), as well as the Mixture\-of\-Experts architecture TeleChat3\-MoE\(Liu et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib22)\), highlights the increasing complexity and diversity of deployed language systems\. Similarly, the Tele\-FLM series\(Li et al\.,[2024c](https://arxiv.org/html/2606.03022#bib.bib17),[b](https://arxiv.org/html/2606.03022#bib.bib16)\)demonstrates the empirical lessons learned when scaling from tens to hundreds of billions of parameters\. Across these architectures, hallucination persists as a critical reliability bottleneck\. This challenge is particularly pronounced in high\-stakes downstream applications, including mathematical reasoning\(Zhao et al\.,[2025a](https://arxiv.org/html/2606.03022#bib.bib38)\), structured table reasoning\(Xiong et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib35)\), process\-reward\-guided structured data construction\(Xing et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib34)\), and universal information extraction via reinforcement learning\(Li et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib19)\), where factual deviations carry direct consequences\. Furthermore, model compression research such as Mosaic Pruning for Mixture\-of\-Experts models\(Hu et al\.,[2025](https://arxiv.org/html/2606.03022#bib.bib12)\)underscores the need to maintain representational integrity under efficiency constraints—a concern closely aligned with hallucination\-free generation\. Our work addresses this fundamental reliability gap through a training\-free geometric intervention, providing a principled solution broadly applicable across the aforementioned deployment contexts\.

## 3Methodology

The computational architecture of the DCO intervention is illustrated inFigure[3](https://arxiv.org/html/2606.03022#S3.F3)\. The mechanism of Dynamic Contextual Orthogonalization \(DCO\) functions by applying linear projection operators to constrain the state evolution of latent representations during inference\. This operation targets the output of the Multi\-Head Attention \(MHA\) module in layerLL, where vectors produced by individual attention heads are orthogonally decomposed to identify and attenuate components that exhibit statistically significant deviation from the contextual semantic manifold\.

### 3\.1Context Anchor Construction

The initial phase involves constructing a context anchor,𝐜\\mathbf\{c\}, which quantitatively defines the direction of semantic consistency at the current logical step\. We posit that the input residual stream𝐱i​nL∈ℝdm​o​d​e​l\\mathbf\{x\}\_\{in\}^\{L\}\\in\\mathbb\{R\}^\{d\_\{model\}\}at layerLLrepresents the accumulated semantic consensus derived from the context processed up to layerL−1L\-1\. To focus on the directional alignment rather than magnitude, we apply Root Mean Square Normalization \(RMSNorm\) to the residual vector\. The normalized context anchor is defined as:

𝐜=RMSNorm​\(𝐱i​nL\)‖RMSNorm​\(𝐱i​nL\)‖2\+ϵ\\mathbf\{c\}=\\frac\{\\text\{RMSNorm\}\(\\mathbf\{x\}\_\{in\}^\{L\}\)\}\{\\\|\\text\{RMSNorm\}\(\\mathbf\{x\}\_\{in\}^\{L\}\)\\\|\_\{2\}\+\\epsilon\}\(1\)whereϵ\\epsilonis a small stability constant\. Geometrically,𝐜\\mathbf\{c\}serves as the reference unit vector on the hypersphere of the latent space, delineating the principal direction of the ongoing generative process\.

![Refer to caption](https://arxiv.org/html/2606.03022v1/figure2.png)Figure 3:Mechanistic pipeline of Dynamic Contextual Orthogonalization \(DCO\)\.The process begins by deriving a context anchor𝐜\\mathbf\{c\}from the input residual stream𝐱i​nL\\mathbf\{x\}\_\{in\}^\{L\}\. Each attention head output𝐨h\\mathbf\{o\}\_\{h\}is then orthogonally decomposed into acontext\-aligned component𝐨h∥\\mathbf\{o\}\_\{h\\parallel\}and an orthogonal component𝐨h⟂\\mathbf\{o\}\_\{h\\perp\}\. The aligned path bypasses intervention to preserve logical continuity, while the orthogonal path is modulated by a dynamic suppression coefficientλh\\lambda\_\{h\}, computed via layer\-wise Z\-score statistics \(μ,σ\\mu,\\sigma\) of the orthogonality metricρ\\rho\. This ensures that only outlier heads—introducing divergent noise relative to the semantic manifold—are attenuated\.
### 3\.2Subspace\-based Orthogonal Decomposition

The second phase executes a fine\-grained decomposition of attention head outputs\. LetHHdenote the number of attention heads in layerLL\. For thehh\-th attention head, let𝐨h∈ℝdm​o​d​e​l\\mathbf\{o\}\_\{h\}\\in\\mathbb\{R\}^\{d\_\{model\}\}be its output vector\. Following the principles of linear projection, we partition𝐨h\\mathbf\{o\}\_\{h\}into acontext\-aligned component𝐨h∥\\mathbf\{o\}\_\{h\\parallel\}and acontext\-orthogonal component𝐨h⟂\\mathbf\{o\}\_\{h\\perp\}\.

The context\-aligned component, representing information congruent with the context anchor, is computed via projection:

𝐨h∥=\(𝐨h⋅𝐜\)⋅𝐜\\mathbf\{o\}\_\{h\\parallel\}=\(\\mathbf\{o\}\_\{h\}\\cdot\\mathbf\{c\}\)\\cdot\\mathbf\{c\}\(2\)The orthogonal component is defined as the vector rejection:

𝐨h⟂=𝐨h−𝐨h∥\\mathbf\{o\}\_\{h\\perp\}=\\mathbf\{o\}\_\{h\}\-\\mathbf\{o\}\_\{h\\parallel\}\(3\)To quantify the divergence of each head relative to the context, we define the orthogonality metricρh\\rho\_\{h\}as the ratio of the orthogonal component’s magnitude to the total magnitude:

ρh=‖𝐨h⟂‖2‖𝐨h‖2\+ϵ\\rho\_\{h\}=\\frac\{\\\|\\mathbf\{o\}\_\{h\\perp\}\\\|\_\{2\}\}\{\\\|\\mathbf\{o\}\_\{h\}\\\|\_\{2\}\+\\epsilon\}\(4\)Asρh→0\\rho\_\{h\}\\to 0, the head reinforces the existing context; asρh→1\\rho\_\{h\}\\to 1, the head introduces information linearly independent of the established semantic manifold\.

### 3\.3Z\-Score Based Dynamic Suppression

To achieve adaptive intervention, DCO relies on the statistical distribution ofρ\\rhowithin the current layer rather than static thresholds\. We observe that in high\-dimensional semantic spaces, valid information diversification and hallucinatory noise often overlap in absolute magnitude but differ in their relative statistical deviation\.

We first calculate the meanμρ\\mu\_\{\\rho\}and standard deviationσρ\\sigma\_\{\\rho\}across allHHheads:

μρ=1H​∑h=1Hρh,σρ=1H​∑h=1H\(ρh−μρ\)2\\mu\_\{\\rho\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\rho\_\{h\},\\quad\\sigma\_\{\\rho\}=\\sqrt\{\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\(\\rho\_\{h\}\-\\mu\_\{\\rho\}\)^\{2\}\}\(5\)The standard score \(Z\-Score\)zhz\_\{h\}for each head is derived to measure relative deviation:

zh=ρh−μρσρ\+ϵz\_\{h\}=\\frac\{\\rho\_\{h\}\-\\mu\_\{\\rho\}\}\{\\sigma\_\{\\rho\}\+\\epsilon\}\(6\)As illustrated in Figure[4](https://arxiv.org/html/2606.03022#S3.F4), we hypothesize that hallucinating heads manifest as statistical outliers in the long tail of the orthogonality distribution\. To softly attenuate these outliers while preserving gradient continuity, we employ a non\-linear Sigmoid gating function to determine the suppression coefficientλh∈\(0,1\]\\lambda\_\{h\}\\in\(0,1\]:

λh=11\+exp⁡\(β⋅\(zh−τ\)\)\\lambda\_\{h\}=\\frac\{1\}\{1\+\\exp\(\\beta\\cdot\(z\_\{h\}\-\\tau\)\)\}\(7\)
![Refer to caption](https://arxiv.org/html/2606.03022v1/figure3.jpg)Figure 4:Schematic of the Z\-Score dynamic suppression mechanism\.We model the distribution of orthogonality metricsρ\\rhoacross attention heads\. Most heads cluster within the context\-aligned region \(blue peak\), while hallucinating heads manifest as statistical outliers in the high\-orthogonality tail \(red region\)\. DCO dynamically sets a thresholdτ\\tauderived from layer\-wise statistics to selectively suppress these divergent components via a Sigmoid gate\.Here,τ\\taurepresents the tolerance threshold in standard deviation units, andβ\\betais the temperature coefficient controlling the steepness of the suppression curve\. The final reconstructed output of the attention module at layerLLis the summation of the intervened head vectors:

AttentionOutputL=∑h=1H\(𝐨h∥\+λh⋅𝐨h⟂\)\\text\{AttentionOutput\}\_\{L\}=\\sum\_\{h=1\}^\{H\}\(\\mathbf\{o\}\_\{h\\parallel\}\+\\lambda\_\{h\}\\cdot\\mathbf\{o\}\_\{h\\perp\}\)\(8\)This formulation ensures that heads exhibiting normative behavior \(zh<τz\_\{h\}<\\tau\) retain their orthogonal contributions \(λh≈1\\lambda\_\{h\}\\approx 1\), thereby preserving necessary semantic diversity\.

### 3\.4Implementation Strategy

DCO is deployed using a layer\-selective strategy, targeting the middle\-to\-late layers defined byL∈\[Ls​t​a​r​t,Le​n​d\]L\\in\[L\_\{start\},L\_\{end\}\]\. This design is grounded in mechanistic findings that early layers \(L<Ls​t​a​r​tL<L\_\{start\}\) are critical for parsing local syntax and often exhibit high intrinsic orthogonality required for structural composition\.

The computational overhead of DCO is negligible\. For a model with hidden dimensiondm​o​d​e​ld\_\{model\}andHHheads, the complexity isO​\(L⋅H⋅dm​o​d​e​l\)O\(L\\cdot H\\cdot d\_\{model\}\)\. Sincedm​o​d​e​ld\_\{model\}is constant andH⋅dm​o​d​e​lH\\cdot d\_\{model\}scales linearly with the model size, this operation is significantly more efficient than the quadratic attention mechanism \(O​\(N2⋅dm​o​d​e​l\)O\(N^\{2\}\\cdot d\_\{model\}\)\), ensuring minimal latency increase in production environments\.

![Refer to caption](https://arxiv.org/html/2606.03022v1/figure5.jpg)Figure 5:Visualization of latent manifold dynamics\.The blue ribbon represents theContext Manifoldℳ\\mathcal\{M\}\. Without intervention, the generative trajectory \(Red dashed line\) suffers fromorthogonal drift, diverging into hallucination\. DCO \(Blue solid line\) applies continuous orthogonal projection \(Green arrow\) at each steptt, constraining the latent state evolution within the safe semantic subspace\.

## 4Results and Analysis

We evaluate the efficacy of Dynamic Contextual Orthogonalization \(DCO\) across a spectrum of benchmarks to assess its impact on faithfulness, factuality, and reasoning stability\. All experiments are conducted using the Llama\-3\-8B\-Instruct and Llama\-3\-70B\-Instruct models\(Grattafiori et al\.,[2024](https://arxiv.org/html/2606.03022#bib.bib8)\)\.

We evaluate contextual faithfulness and instruction following using IFEval\(Zhou et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib40)\), XSum, and NQ\-Swap\. For factuality and knowledge retention, we employ TruthfulQA\(Lin et al\.,[2022](https://arxiv.org/html/2606.03022#bib.bib20)\), TriviaQA\(Joshi et al\.,[2017](https://arxiv.org/html/2606.03022#bib.bib13)\), and Natural Questions \(NQ\-Open\)\(Kwiatkowski et al\.,[2019](https://arxiv.org/html/2606.03022#bib.bib14)\)\. Complex reasoning stability is assessed via MuSiQue\(Trivedi et al\.,[2022](https://arxiv.org/html/2606.03022#bib.bib30)\)\.

Table 1:Performance comparison on faithfulness evaluation tasks\. We report ROUGE\-L \(R\-L\), BERTScore\(Zhang et al\.,[2020](https://arxiv.org/html/2606.03022#bib.bib37)\), and FactKB\(Feng et al\.,[2023](https://arxiv.org/html/2606.03022#bib.bib4)\)scores\. Note thatNQ\-Openhere is evaluated in anOpen\-Booksetting \(with retrieved context\)\.### 4\.1Experimental Configuration

To maintain intervention stability across diverse semantic landscapes, we adopt a calibration strategy based onsemantic densityandreasoning depth\. The intervention is primarily applied to the middle\-to\-late layers of the residual stream, aligning with mechanistic findings that these layers govern factual retrieval and integration\.

The hyperparameters—tolerance thresholdτ\\tauand temperature coefficientβ\\beta—are configured to reflect the geometric constraints of specific tasks\. For counterfactual tasks requiring strict adherence to provided evidence \(e\.g\., NQ\-Swap\), we apply tighter projection constraints to filter parametric bias\. Conversely, for multi\-hop reasoning \(e\.g\., MuSiQue\), constraints are relaxed to accommodate the necessary semantic expansion required for logic chaining\. This task\-dependent configuration ensures that DCO adapts to the specific manifold geometry of downstream applications\.

Table 2:Performance comparison on factuality benchmarks\. We focus on TruthfulQA \(MC\), TriviaQA, and NQ\-Open in the closed\-book setting\.
### 4\.2Performance on Contextual Faithfulness

Table[1](https://arxiv.org/html/2606.03022#S4.T1)details the performance of DCO on benchmarks evaluating adherence to external contexts and instructions\. DCO demonstrates statistically significant improvements over baselines, particularly in instruction following and context\-grounded retrieval tasks\.

On the IFEval benchmark, DCO achieves state\-of\-the\-art results across model scales\. For Llama\-3\-70B, DCO elevates the Prompt\-level strict accuracy to 85\.77% and Instruction\-level accuracy to 90\.29%, representing a substantial margin over the greedy baseline of 77\.45% and 84\.41%, respectively, and exceeding existing intervention methods such as DeCoRe\. Similarly, in the NQ\-Open Open\-Book setting, DCO outperforms all baselines, reaching 79\.51% on the 70B model\. These results suggest that enforcing geometric alignment within the residual stream effectively enhances the capacity of the model to prioritize contextual evidence over internal parametric noise\.

Regarding the XSum summarization task, DCO achieves the highest ROUGE\-L of 20\.39 on the 8B model\. Crucially, when compared to Inference\-Time Intervention \(ITI\), which utilizes static steering vectors, DCO exhibits superior robustness\. While ITI suffers a severe performance regression, dropping to 13\.25 R\-L, DCO maintains high linguistic quality\. This disparity validates the hypothesis that dynamic, context\-dependent orthogonalization is essential for maintaining the integrity of the latent manifold\.

In the NQ\-Swap counterfactual task, while Context\-Aware Decoding \(CAD\) retains the highest EM scores through logit\-level contrastive amplification, DCO serves as a computationally efficient alternative\. By achieving competitive alignment of 75\.33% EM on the 70B model via a single\-forward projection, DCO avoids the dual\-decoding latency overhead inherent in contrastive methods while still effectively suppressing parametric interference\.

Table 3:Multi\-hop reasoning performance on MuSiQue\.
### 4\.3Pareto Frontier: Knowledge Preservation

An essential consideration in hallucination mitigation is the preservation of the model’s general capabilities, as aggressive intervention often leads to a degradation of parametric knowledge\. Table[2](https://arxiv.org/html/2606.03022#S4.T2)illustrates this performance trade\-off\. While methods such as ITI and CD report higher scores on TruthfulQA, they frequently cause a significant performance regression in factual retrieval\. For instance, on the Llama\-3\-8B model, ITI results in a decrease in TriviaQA accuracy from 56\.58% to 48\.41%, indicating that static interventions may inadvertently suppress valid factual retrieval pathways alongside hallucinatory signals\.

In contrast, DCO optimizes the Pareto frontier by maintaining or enhancing performance on knowledge\-intensive benchmarks\. On the Llama\-3\-70B model, DCO achieves the highest performance among the evaluated methods for both TriviaQA \(76\.12%\) and NQ\-Open Closed\-book \(43\.62%\) settings\. This robustness is attributed to the adaptive Z\-score statistical filtering mechanism\. By dynamically defining suppression thresholds based on layer\-wise distributions, DCO distinguishes between the normative statistical signals of factual retrieval and the outlier components characteristic of hallucinations\. These results provide empirical evidence that dynamic orthogonalization effectively minimizes interference with standard knowledge retrieval processes while maintaining competitive truthfulness scores, such as the 40\.39% MC1 achieved on the 8B model\.

### 4\.4Reasoning Stability in Complex Chains

Table[3](https://arxiv.org/html/2606.03022#S4.T3)evaluates the performance of various intervention methods on the MuSiQue multi\-hop reasoning dataset\. On the Llama\-3\-70B model, DCO achieves the highest performance in the Closed\-book setting when utilized with Chain\-of\-Thought \(CoT\), reaching an Exact Match \(EM\) score of 21\.14%\. This represents an improvement over both the greedy baseline \(20\.15%\) and head\-level intervention methods such as DeCoRe \(20\.60%\)\.

The observed improvement is posited to stem from the mitigation of compounding deviations within the residual stream during multi\-step inference\. In complex reasoning chains, minor orthogonal components introduced in initial logical steps may accumulate, leading to semantic drift where the latent trajectory diverges from the factual manifold\. By enforcing manifold alignment at each layer, DCO attenuates these divergent components before they propagate through the network\. Furthermore, the competitive performance of DCO relative to head\-masking approaches like DeCoRe suggests that the errors associated with multi\-hop reasoning may be distributive in nature\. Consequently, these errors appear to be more effectively addressed via global statistical constraints on the residual stream rather than the isolation of individual attention heads\. While DoLa and DeCoRe maintain a slight advantage in certain Open\-book scenarios, DCO demonstrates robust stability in internalizing reasoning paths without external evidence\.

![Refer to caption](https://arxiv.org/html/2606.03022v1/figure6.jpg)Figure 6:Qualitative comparison under knowledge conflict\.Top: The input context contains counterfactual entities \(“Banana Inc\.”\)\. Middle: The baseline model \(Red\) ignores the context and hallucinates real\-world entities \(“Apple”, “iPhone”\) due to parametric memory bias\. Bottom: DCO \(Green\) successfully suppresses this orthogonal noise, aligning the generation with the counterfactual context\.
### 4\.5Qualitative Analysis

To examine the microscopic mechanism of DCO, we analyze a "knowledge conflict" scenario in Figure[6](https://arxiv.org/html/2606.03022#S4.F6), where fictitious context contradicts the model’s parametric priors\. In the baseline model, specific attention heads dominated by pre\-trained associations generate real\-world entities \(e\.g\., "Apple"\), effectively ignoring the provided context\. Geometrically, these heads produce output vectors that are linearly independent of the context anchor𝐜\\mathbf\{c\}, representing orthogonal drift from the semantic manifold\.

DCO identifies these divergent components as statistical outliers in the Z\-score distribution of the orthogonality metricρ\\rho\. By applying the dynamic gating mechanism, DCO attenuates these anomalous orthogonal vectors while preserving components collinear with the context anchor\. This operation effectively projects the divergent generative trajectory back onto the context manifoldℳ\\mathcal\{M\}without disrupting syntactic coherence\. These findings provide empirical evidence for the hypothesis of hallucinations as orthogonal noise and demonstrate DCO’s efficacy in enforcing contextual alignment under conflict\.

## 5Conclusion

In this work, we have presented Dynamic Contextual Orthogonalization \(DCO\), a framework that operationalizes the geometric hypothesis of hallucinations asorthogonal noise injection\. By formalizing factual deviations as vector components that diverge from the contextual semantic manifold, we established a mechanistic basis for intervention\. DCO utilizes the instantaneous residual stream as a dynamic anchor and employs a layer\-wise Z\-Score suppression mechanism\. This design addresses the rigidity of static steering vectors by adapting to the local statistical distribution of attention outputs, thereby distinguishing between necessary semantic diversification and anomalous noise\.

Empirical validation across the Llama\-3 family confirms that DCO effectively resolves the persistent trade\-off between contextual faithfulness and parametric knowledge retention\. Unlike traditional decoding strategies that often degrade general reasoning capabilities to enforce context adherence, DCO’s outlier\-based attenuation preserves the model’s underlying logical structure while significantly enhancing performance on faithfulness benchmarks such as XSum and IFEval\. The method’s robustness on TriviaQA and MuSiQue further validates that enforcing geometric constraints is a non\-destructive pathway to reliable generation\.

In summary, DCO offers a computationally efficient, training\-free mechanism for alignment during inference\. It underscores the necessity of dynamic, representation\-level intervention for suppressing hallucinations in complex semantic spaces\. Future research will focus on developing meta\-learning frameworks to automate the calibration of geometric constraints, aiming to achieve self\-adaptive manifold control that generalizes across varying semantic densities\.

## 6Limitations

While Dynamic Contextual Orthogonalization \(DCO\) demonstrates significant efficacy in enhancing faithfulness and mitigating hallucinations, we identify specific boundary conditions and design choices inherent to the framework\.

The implementation of DCO introduces additional linear projection and statistical normalization operations at each decoding step\. While this theoretically increases the floating\-point operations \(FLOPs\) per layer, the added complexity scales linearly with the model dimension \(O​\(dm​o​d​e​l\)O\(d\_\{model\}\)\)\. This is negligible compared to the quadratic complexity of the self\-attention mechanism \(O​\(N2\)O\(N^\{2\}\)\) inherent to Transformer architectures\. Furthermore, unlike contrastive decoding paradigms \(e\.g\., CAD, DoLa\) which necessitate multiple forward passes or parallel decoding of different model states to compute logit differences, DCO operates strictly within a single inference pass\. Consequently, in production environments prioritizing throughput and latency, DCO maintains a distinct efficiency advantage over multi\-pass intervention strategies\.

The geometric definition of the semantic manifold in DCO relies on the extraction of a context anchor from the input residual stream\. This design is explicitly optimized for context\-grounded generation, summarization, and context\-grounded reasoning, where an external reference exists\. In scenarios entirely devoid of context \(pure closed\-book generation\), the definition of orthogonality becomes less distinct\. However, our empirical results on knowledge\-intensive benchmarks \(e\.g\., TriviaQA, TruthfulQA\) indicate that the dynamic Z\-score filtering mechanism is robust enough to distinguish between valid parametric retrieval and stochastic noise even in lower\-context settings\. This suggests that DCO successfully mitigates the trade\-off between hallucination suppression and general capability degradation, a common failure mode in static intervention methods\.

Our framework is predicated on the Linear Representation Hypothesis, positing that semantic deviations manifest as orthogonal vectors in the residual stream\. While the underlying neural architecture involves non\-linear activation functions, the effectiveness of DCO supports the utility of linear approximations for manipulating high\-level semantic features\. This geometric simplification provides a mechanistically interpretable control method, offering a more transparent alternative to black\-box modifications of output probability distributions\.

## References

- Chen et al\. \(2024\)Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He\. 2024\.In\-context sharpness as alerts: An inner representation perspective for hallucination mitigation\.*arXiv preprint arXiv:2403\.01548*\.
- Chuang et al\. \(2023\)Yung\-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He\. 2023\.Dola: Decoding by contrasting layers improves factuality in large language models\.*arXiv preprint arXiv:2309\.03883*\.
- Dar et al\. \(2023\)Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant\. 2023\.[Analyzing transformers in embedding space](https://doi.org/10.18653/v1/2023.acl-long.893)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 16124–16170, Toronto, Canada\. Association for Computational Linguistics\.
- Feng et al\. \(2023\)Shangbin Feng, Vidhisha Balachandran, Yuyang Bai, and Yulia Tsvetkov\. 2023\.[FactKB: Generalizable factuality evaluation using language models enhanced with factual knowledge](https://doi.org/10.18653/v1/2023.emnlp-main.59)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 933–952, Singapore\. Association for Computational Linguistics\.
- Gema et al\. \(2024\)Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Teare, Beatrice Alex, Pasquale Minervini, and Amrutha Saseendran\. 2024\.Decore: Decoding by contrasting retrieval heads to mitigate hallucinations\.*arXiv preprint arXiv:2410\.18860*\.
- Gera et al\. \(2023\)Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim, and Eyal Shnarch\. 2023\.[The benefits of bad advice: Autocontrastive decoding across model layers](https://doi.org/10.18653/v1/2023.acl-long.580)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10406–10420, Toronto, Canada\. Association for Computational Linguistics\.
- Geva et al\. \(2023\)Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson\. 2023\.[Dissecting recall of factual associations in auto\-regressive language models](https://doi.org/10.18653/v1/2023.emnlp-main.751)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12216–12235, Singapore\. Association for Computational Linguistics\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others\. 2024\.[The llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*Preprint*, arXiv:2407\.21783\.
- Gurnee and Tegmark \(2024\)Wes Gurnee and Max Tegmark\. 2024\.[Language models represent space and time](https://arxiv.org/abs/2310.02207)\.*Preprint*, arXiv:2310\.02207\.
- He et al\. \(2024\)Zhongjiang He, Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, Yuyao Huang, Xuelong Li, Yongxiang Li, Zhonghao Che, Zhaoxi Zhang, Yan Wang, Xin Wang, Luwen Pu, Huinan Xu, Ruiyu Fang, Yu Zhao, Jie Zhang, Xiaomeng Huang, Zhilong Lu, and 17 others\. 2024\.[Telechat technical report](https://arxiv.org/abs/2401.03804)\.*Preprint*, arXiv:2401\.03804\.
- Hernandez et al\. \(2024\)Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau\. 2024\.[Linearity of relation decoding in transformer language models](https://arxiv.org/abs/2308.09124)\.*Preprint*, arXiv:2308\.09124\.
- Hu et al\. \(2025\)Wentao Hu, Mingkuan Zhao, Shuangyong Song, Xiaoyan Zhu, Xin Lai, and Jiayin Wang\. 2025\.[Mosaic pruning: A hierarchical framework for generalizable pruning of mixture\-of\-experts models](https://arxiv.org/abs/2511.19822)\.*Preprint*, arXiv:2511\.19822\.
- Joshi et al\. \(2017\)Mandar Joshi, Eunsol Choi, Daniel S\. Weld, and Luke Zettlemoyer\. 2017\.[Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension](https://arxiv.org/abs/1705.03551)\.*Preprint*, arXiv:1705\.03551\.
- Kwiatkowski et al\. \(2019\)Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming\-Wei Chang, Andrew M\. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov\. 2019\.[Natural questions: A benchmark for question answering research](https://doi.org/10.1162/tacl_a_00276)\.*Transactions of the Association for Computational Linguistics*, 7:452–466\.
- Li et al\. \(2024a\)Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg\. 2024a\.[Inference\-time intervention: Eliciting truthful answers from a language model](https://arxiv.org/abs/2306.03341)\.*Preprint*, arXiv:2306\.03341\.
- Li et al\. \(2024b\)Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, and Tiejun Huang\. 2024b\.[52b to 1t: Lessons learned via tele\-flm series](https://arxiv.org/abs/2407.02783)\.*Preprint*, arXiv:2407\.02783\.
- Li et al\. \(2024c\)Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Chao Wang, Xinzhang Liu, Zihan Wang, Yu Zhao, Xin Wang, Yuyao Huang, Shuangyong Song, Yongxiang Li, Zheng Zhang, Bo Zhao, Aixin Sun, Yequan Wang, Zhongjiang He, Zhongyuan Wang, Xuelong Li, and Tiejun Huang\. 2024c\.[Tele\-flm technical report](https://arxiv.org/abs/2404.16645)\.*Preprint*, arXiv:2404\.16645\.
- Li et al\. \(2023\)Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis\. 2023\.[Contrastive decoding: Open\-ended text generation as optimization](https://doi.org/10.18653/v1/2023.acl-long.687)\.In*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 12286–12312, Toronto, Canada\. Association for Computational Linguistics\.
- Li et al\. \(2025\)Zhongqiu Li, Shiquan Wang, Ruiyu Fang, Mengjiao Bao, Zhenhe Wu, Shuangyong Song, Yongxiang Li, and Zhongjiang He\. 2025\.[Mr\-uie: Multi\-perspective reasoning with reinforcement learning for universal information extraction](https://arxiv.org/abs/2509.09082)\.*Preprint*, arXiv:2509\.09082\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\. 2022\.[Truthfulqa: Measuring how models mimic human falsehoods](https://arxiv.org/abs/2109.07958)\.*Preprint*, arXiv:2109\.07958\.
- Liu et al\. \(2024\)Nelson F\. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang\. 2024\.[Lost in the middle: How language models use long contexts](https://doi.org/10.1162/tacl_a_00638)\.*Transactions of the Association for Computational Linguistics*, 12:157–173\.
- Liu et al\. \(2025\)Xinzhang Liu, Chao Wang, Zhihao Yang, Zhuo Jiang, Xuncheng Zhao, Haoran Wang, Lei Li, Dongdong He, Luobin Liu, Kaizhe Yuan, Han Gao, Zihan Wang, Yitong Yao, Sishi Xiong, Wenmin Deng, Haowei He, Kaidong Yu, Yu Zhao, Ruiyu Fang, and 35 others\. 2025\.[Training report of telechat3\-moe](https://arxiv.org/abs/2512.24157)\.*Preprint*, arXiv:2512\.24157\.
- Longpre et al\. \(2021\)Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh\. 2021\.[Entity\-based knowledge conflicts in question answering](https://doi.org/10.18653/v1/2021.emnlp-main.565)\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 7052–7063, Online and Punta Cana, Dominican Republic\. Association for Computational Linguistics\.
- Meng et al\. \(2022\)Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov\. 2022\.Locating and editing factual associations in gpt\.In*Proceedings of the 36th International Conference on Neural Information Processing Systems*, NIPS ’22, Red Hook, NY, USA\. Curran Associates Inc\.
- Nanda et al\. \(2023\)Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt\. 2023\.[Progress measures for grokking via mechanistic interpretability](https://arxiv.org/abs/2301.05217)\.*Preprint*, arXiv:2301\.05217\.
- Narayan et al\. \(2018\)Shashi Narayan, Shay B\. Cohen, and Mirella Lapata\. 2018\.[Don’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization](https://doi.org/10.18653/v1/D18-1206)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 1797–1807, Brussels, Belgium\. Association for Computational Linguistics\.
- Olsson et al\. \(2022\)Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield\-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, and 7 others\. 2022\.[In\-context learning and induction heads](https://arxiv.org/abs/2209.11895)\.*Preprint*, arXiv:2209\.11895\.
- Park et al\. \(2024\)Kiho Park, Yo Joong Choe, and Victor Veitch\. 2024\.[The linear representation hypothesis and the geometry of large language models](https://arxiv.org/abs/2311.03658)\.*Preprint*, arXiv:2311\.03658\.
- Shi et al\. \(2024\)Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen\-tau Yih\. 2024\.Trusting your evidence: Hallucinate less with context\-aware decoding\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\)*, pages 783–791\.
- Trivedi et al\. \(2022\)Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal\. 2022\.[Musique: Multihop questions via single\-hop question composition](https://arxiv.org/abs/2108.00573)\.*Preprint*, arXiv:2108\.00573\.
- Wang et al\. \(2024\)Zihan Wang, XinZhang Liu, Shixuan Liu, Yitong Yao, Yunyao Huang, Mengxiang Li, Zhongjiang He, Yongxian Li, Luwen Pu, Huinan Xu, Chao Wang, and Shuangyong Song\. 2024\.[TeleChat: An open\-source billingual large language model](https://aclanthology.org/2024.sighan-1.2/)\.In*Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing \(SIGHAN\-10\)*, pages 10–20, Bangkok, Thailand\. Association for Computational Linguistics\.
- Wang et al\. \(2025\)Zihan Wang, Xinzhang Liu, Yitong Yao, Chao Wang, Yu Zhao, Zhihao Yang, Wenmin Deng, Kaipeng Jia, Jiaxin Peng, Yuyao Huang, Sishi Xiong, Zhuo Jiang, Kaidong Yu, Xiaohui Hu, Fubei Yao, Ruiyu Fang, Zhuoru Jiang, Ruiting Song, Qiyi Xie, and 19 others\. 2025\.[Technical report of telechat2, telechat2\.5 and t1](https://arxiv.org/abs/2507.18013)\.*Preprint*, arXiv:2507\.18013\.
- Wu et al\. \(2024\)Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu\. 2024\.[Retrieval head mechanistically explains long\-context factuality](https://arxiv.org/abs/2404.15574)\.*Preprint*, arXiv:2404\.15574\.
- Xing et al\. \(2025\)Hongrui Xing, Xinzhang Liu, Zhuo Jiang, Zhihao Yang, Yitong Yao, Zihan Wang, Wenmin Deng, Chao Wang, Shuangyong Song, Wang Yang, Zhongjiang He, and Yongxiang Li\. 2025\.[LLMSR@XLLM25: A language model\-based pipeline for structured reasoning data construction](https://doi.org/10.18653/v1/2025.xllm-1.31)\.In*Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling \(XLLM 2025\)*, pages 342–350, Vienna, Austria\. Association for Computational Linguistics\.
- Xiong et al\. \(2025\)Sishi Xiong, Dakai Wang, Yu Zhao, Jie Zhang, Changzai Pan, Haowei He, Xiangyu Li, Wenhan Chang, Zhongjiang He, Shuangyong Song, and Yongxiang Li\. 2025\.[Tablereasoner: Advancing table reasoning framework with large language models](https://arxiv.org/abs/2507.08046)\.*Preprint*, arXiv:2507\.08046\.
- Yu et al\. \(2024\)Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong\. 2024\.[Mechanistic understanding and mitigation of language model non\-factual hallucinations](https://doi.org/10.18653/v1/2024.findings-emnlp.466)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 7943–7956, Miami, Florida, USA\. Association for Computational Linguistics\.
- Zhang et al\. \(2020\)Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q\. Weinberger, and Yoav Artzi\. 2020\.[Bertscore: Evaluating text generation with bert](https://arxiv.org/abs/1904.09675)\.*Preprint*, arXiv:1904\.09675\.
- Zhao et al\. \(2025a\)Deji Zhao, Donghong Han, Jia Wu, Zhongjiang He, Bo Ning, Ye Yuan, Yongxiang Li, Chao Wang, and Shuangyong Song\. 2025a\.Enhancing math reasoning ability of large language models via computation logic graphs\.*Knowledge\-Based Systems*, 325:113905\.
- Zhao et al\. \(2025b\)Mingkuan Zhao, Wentao Hu, Jiayin Wang, Xin Lai, Tianchen Huang, Yuheng Min, Rui Yan, and Xiaoyan Zhu\. 2025b\.[Making every head count: Sparse attention without the speed\-performance trade\-off](https://arxiv.org/abs/2511.09596)\.*Preprint*, arXiv:2511\.09596\.
- Zhou et al\. \(2023\)Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou\. 2023\.[Instruction\-following evaluation for large language models](https://api.semanticscholar.org/CorpusID:265157752)\.*ArXiv*, abs/2311\.07911\.
- Zou et al\. \(2025\)Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann\-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J\. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others\. 2025\.[Representation engineering: A top\-down approach to ai transparency](https://arxiv.org/abs/2310.01405)\.*Preprint*, arXiv:2310\.01405\.