Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models

# Self-Improving is Often Sudden: Enlightenment-style Finetuning for Large-Scale Models
Source: [https://arxiv.org/html/2607.13395](https://arxiv.org/html/2607.13395)
11institutetext:1Frontier of Artificial Network Lab, Department of Data Science, City University of Hong Kong, Hong Kong SAR, China22institutetext:2The Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen, China, The Chinese University of Hong Kong\-Shenzhen, Shenzhen, China33institutetext:3School of Computer Science and Artificial Intelligence, Guangdong University of Education, Guangzhou 510303, China44institutetext:4School of Instrumentation Science and Engineering, Harbin Institute of Technology, Harbin, China\.
⋆\\starCorresponding author\.Tianwei Zhang2Yu\-Hao Jiang1Feifei Zhang3Hang\-Cheng Dong4Feng\-Lei Fan1,⋆\\star

\(Received: date / Accepted: date\)

###### Abstract

The pursuit of autonomously self\-improving models has attracted growing interest in the era of large\-scale foundation models\. Drawing inspiration from the concept of “enlightenment” or “aha moment” in human brain, we hypothesize that large models exhibit an analogous enlightenment phenomenon — a latent capacity for sudden capability boost\. Then, we proposeEnlightenment, a novel training\-free post\-tuning paradigm for large\-scale models\. Our approach modifies shortcuts for key modules/layers without weight updates, while existing training\-free ones predominantly manipulate attention weights\. We introduce two architecture\-specific instantiations: i\) For large language models, we propose attention head\-mixing shortcuts that recalibrate attention weights by linking the initial attention head’s output to all other target heads, modulated by an adaptive scaling factor initialization strategy\. ii\) For vision\-language models, we apply a lightweight scalar\-modulated factor to residual connections in the decoder layers, regulating information flow\. Extensive experiments show thatEnlighten\-mentefficiently unlocks the latent potential of pre\-trained networks, yielding remarkable performance improvements across diverse benchmarks and models\. We have open\-sourced our code at[Enlightenment](https://github.com/asdvfghg/Enlightenment_project)for readers’ free check\.

## 1Introduction

Recently, enabling a large\-scale model to improve itself has gained large traction in the AI communityWuet al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib53)\); Quet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib54)\); Huanget al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib55)\)\. This venture actually can be found across the history of AI—from early symbolic self‑modification to the self‑play dynamics that powered breakthroughs in games like chess, and beyondBillard and Diday \([2002](https://arxiv.org/html/2607.13395#bib.bib97)\); Udrescu and Tegmark \([2020](https://arxiv.org/html/2607.13395#bib.bib96)\)\. With the rapid advancement of large\-scale pretrained models, realizing their effective self\-improvement has emerged as a pivotal challenge on the path toward artificial general intelligence \(AGI\), promising to endow these models with an unprecedented degree of adaptability and generalizability\. However, for this moment, self\-improvement is a research area whose definition and boundary are not crystal clear\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x1.png)Figure 1:The overall concept of Enlightenment\. Just as sudden comprehension arises from the formation of new neuronal connections in the human brain, we modify shortcuts within pre\-trained models, enabling them to self\-improve during inference without additional post\-training\.To the best of our knowledge, self\-improvement in contemporary research mainly concerns four schools: i\) Continual learning provides the machinery for models to adapt to new tasks without catastrophic forgetting, employing elastic weight consolidation, dynamic architecture expansion, or episodic memory replay so that each time of learning builds upon previous knowledge rather than erasing itNinget al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib56)\); ii\) Self\-distillation leverages the model as its own oracle: the model is iteratively retrained on softened distributions produced by its own earlier checkpoints, which progressively sharpens predictions without external ground truthGouet al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib57)\); Jianget al\.\([2026b](https://arxiv.org/html/2607.13395#bib.bib58)\); iii\) Test\-time adaptation, in contrast, updates model statistics or a small subset of parameters on a per\-sample or per\-distribution basis using unsupervised objectives like entropy minimization, pseudo\-labeling, or self\-supervised rotation prediction, which allows the model to rapidly conforming to individual user patterns after deploymentLianget al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib59)\); iv\) Training\-free finetuning explores token\-level attention modulation for improving foundation model performance without any parameter modificationsHanet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib27)\); Xiaoet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib28)\)\.

It can be seen that a diverse spectrum of approaches are related to model self\-improvement\. Yet, no single methodology has yet crystallized into a dominant trajectory\. Our intention here is not to reconcile or unify these existing routes\. Rather, given the multi\-faceted nature of self\-improvement, we take a step back and design a new mechanism in a principled way, thereby offering a perspective that is inherently distinct from existing lines\. We turn our attention to the roots of brain intelligence and seek inspiration from our brain\. This aligns with the idea of “NeuroAI”Zadoret al\.\([2022](https://arxiv.org/html/2607.13395#bib.bib52)\); Sadeh and Clopath \([2025](https://arxiv.org/html/2607.13395#bib.bib50)\); Fanet al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib51)\)which argues that insights from neuroscience can greatly accelerate the development of next‑generation AI\. This claim is well grounded: the brain remains the most intelligent system known, and artificial neural networks can be viewed as simplified abstractions of it\. Thus, mechanisms of neuroscience can more or less shed light on the challenges in AI\.

Motivated by the NeuroAI paradigm, it is noted that human’s self\-improving is often sudden, manifesting as the so\-called “aha moment” or enlightenment\. In the human brain, enlightenment is not merely subjective flashes of clarity\. Neuroimaging and electrophysiological studies suggest that such insights often involve the abrupt recruitment of the anterior superior temporal gyrus and the anterior cingulate cortex, accompanied by a transient burst of gamma‑band activityArikha \([2006](https://arxiv.org/html/2607.13395#bib.bib60)\)\. In addition, enlightenment often appears during resting time when active learning or training has ceased\. This resting\-time‑driven plasticity change can sometimes establish entirely new microcircuits, thereby enabling a qualitative leap in performance rather than incremental adjustmentMonsivaiset al\.\([2017](https://arxiv.org/html/2607.13395#bib.bib92)\)\. Given the compelling characteristics of enlightenment,can we introduce the related mechanism into large models and examine the associated computational benefits in self\-improvement?

Translated to artificial systems, enlightenment suggests that a new operational phase in manipulating models beyond training and testing, referred to as retrospection, is important, which tunes models in the resting time and not necessarily relying on data\. More importantly, the model may wire new circuits to improve itself during retrospectionChen and Krajbich \([2017](https://arxiv.org/html/2607.13395#bib.bib93)\)\. Furthermore, we posit that enlightenment in artificial networks should fulfill the following conditions: i\) no gradient is computed, since we highlight the sudden change; ii\) no data is involved, since we highlight the resting\-time adjustment that is before testing and after training; and iii\) no weight update is needed, since we highlight modifying circuits\. Per these conditions, we proposeEnlightenment, a novel and highly effective training\-free retrospection method for large models\. As Figure[1](https://arxiv.org/html/2607.13395#S1.F1)shows, we focus on modifying the network structure, with an emphasis on shortcuts, and we even inject connections that do not exist previously into large\-scale models\. Table[1](https://arxiv.org/html/2607.13395#S1.T1)shows that the proposedEnlightenmentoccupies a unique niche relative to other techniques\. Particularly, the existing training\-free finetuning methods focus on plasticity adjustments, while our method emphasizes internal structures\.

Table 1:Comparisons of different self\-improvement paradigms\.Specifically, guided by the principle of modifying shortcuts, we implementEnlightenmentfor VLMs and LLMs, respectively\. For VLMs, we apply a single shared scalar modulator to every residual connection within the Transformer decoder layers, covering both the attention and feed\-forward network \(FFN\) sub\-blocks\. For LLMs, we take a bolder step: inserting new shortcuts to aggregate information across per\-head attention outputs from the head to the rest\. The efficacy of such intra\-layer shortcuts is corroborated by prior workFanet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib94)\); Zhanget al\.\([2024c](https://arxiv.org/html/2607.13395#bib.bib95)\)\. Furthermore, to complement these architectural augmentations, shortcuts are assigned with dynamic scaling hyperparameters, providing a per\-head level recalibration\.Enlightenmentexpands the representational capacity of large models without altering pre\-existing weights\. Meanwhile, our method introduces only a small number of hyperparameters, and the effect of self\-improvement is insensitive to most of them\. In brief, we summarize our contributions as threefold:

- 1\)Inspired by the neuroscience of enlightenment, we introduceEnlightenment, a training‑free tuning paradigm that operates in a novel retrospection phase between training and inference\. This paradigm focuses on shortcuts instead of weights\.
- 2\)To instantiate this paradigm, we propose a lightweight architectural intervention strategy that modulates shortcut connections via scaling hyperparameters and even inserting cross‑head shortcuts, thereby expanding representational capacity without post‑training\.
- 3\)Extensive experiments on diverse multimodal and reasoning benchmarks demonstrate thatEnlightenmentconsistently delivers salient performance improvements across three widely\-used VLM and three mainstream LLM variants, while requiring no additional data, no gradient computation, and no weight updates\.

## 2Related Work

### 2\.1Continual Learning

Continual learning \(CL\), also referred to as lifelong learning, addresses the challenge of enabling LLMs to acquire new knowledge without forgetting previously learned tasksZhouet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib83)\); Chenet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib82)\)\. Continual learning can be regarded as a form of self\-improvement\. Recently, continual instruction tuning \(CIT\), which refines a large model’s instruction\-following abilityChenet al\.\([2024a](https://arxiv.org/html/2607.13395#bib.bib84)\); Heet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib85)\), has been intensively investigated\.

i\) Task\-incremental CIT fine\-tunes LLMs on sequential task\-specific instructions, with methods such as Progressive Prompts[Razdaibiedinaet al\.](https://arxiv.org/html/2607.13395#bib.bib86)freezing most parameters and learning per\-task prompt tokens to reduce computation and forgetting, while the scalable language modelPenget al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib87)\)incorporates vector space retrieval for knowledge expansion\. ii\) Domain\-incremental CIT adapts LLMs to evolving domains\. Task\-adaptive pre\-training \(TAPT\)Gururanganet al\.\([2020](https://arxiv.org/html/2607.13395#bib.bib88)\)employs a data selection strategy that retrieves in\-domain unlabeled text for fine\-tuning, and ConPETSonget al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib89)\)combines parameter\-efficient tuning with dynamic replay to prevent forgetting\. iii\) Tool\-incremental CIT represents a particularly promising direction, extending LLMs beyond text generation to real\-world tool use\. ToolLLMQinet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib90)\)demonstrates that LLMs can master over 16,000 real\-world APIs through CIT, enabling appropriate tool selection and invocation for diverse tasks\. ToolkenGPTHaoet al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib91)\)takes a complementary approach by augmenting frozen language models with tool embeddings, updating only tool\-specific parameters to eliminate catastrophic forgetting while continuously expanding tool capabilities\.

### 2\.2Self\-Distillation

Self\-Distillation \(SD\) transfers knowledge from a model or its historical states, enabling self\-evolution without an external teacher networkXuet al\.\([2026a](https://arxiv.org/html/2607.13395#bib.bib76)\)\. In the instruction\-following domain, Self\-InstructWanget al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib78)\)establishes a closed\-loop pipeline for autonomously generating, filtering, and refining high\-quality fine\-tuning data\. For preference alignment, self\-play fine\-tuning \(SPIN\)Chenet al\.\([2024b](https://arxiv.org/html/2607.13395#bib.bib79)\)casts the process as an adversarial self\-play game between pseudo\-responses and gold\-standard SFT trajectories under a direct preference optimization \(DPO\) objective, breaking reliance on third\-party evaluators\. Self\-distillation fine\-tuning \(SDFT\)Shenfeldet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib80)\)further addresses manifold contraction from multi\-stage fine\-tuning by using an undegraded historical model state as a distillation anchor, recovering broad semantic capabilities without sacrificing task\-specific performance\.

In reasoning domains, SD has enabled breakthroughs in long\-chain logical reasoning and code generation\. On\-policy self\-correction \(OPSD\)Zhaoet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib77)\)trains a single LLM in dual roles—a blind student and a privileged teacher—applying KL or Jensen\-Shannon divergence as a probabilistic penalty over on\-policy solution trajectories\. Simple self\-distillation \(SSD\)Zhanget al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib81)\)takes a simpler route, using the model’s own unverified temperature\-sampled outputs as cross\-entropy targets to reshape the token\-level decoding state, suppress long\-tail trajectory drift in chain\-of\-thought generation, and consolidate both diversity and generalization\.

### 2\.3Test\-Time Adaptation

Test\-Time Adaptation \(TTA\) updates model parameters during inference to mitigate distributional shifts without costly offline retrainingLianget al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib59)\), fundamentally blurring the boundary between training and deploymentParket al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib69)\)\. Early TTA methods address the absence of ground\-truth labels through self\-supervised objectives, such as token\-level entropy minimizationNooriet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib70)\)and next\-token likelihood optimizationThrampoulidis \([2024](https://arxiv.org/html/2607.13395#bib.bib71)\), combined with selective parameter\-efficient strategies including isolated layer updates, sparse weight adjustments, and low\-rank parameter\-efficient fine\-tuning \(PEFT\) to scale gradient updates to billion\-parameter models while preserving foundational linguistic capabilitiesXuet al\.\([2026b](https://arxiv.org/html/2607.13395#bib.bib72)\)\.

The optimization landscape of test\-time fine\-tuning increases risks of catastrophic forgetting and localized overfitting on single instances or small batchesLuoet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib75)\)\. To mitigate these instabilities, recent work increasingly frames TTA through the lens of meta\-learning and online optimization, leveraging principles from model\-agnostic meta\-learningLiang and Chen \([2025](https://arxiv.org/html/2607.13395#bib.bib73)\)and ReptileNicholet al\.\([2018](https://arxiv.org/html/2607.13395#bib.bib74)\)to optimize initial pre\-trained states for rapid downstream adaptation\.

### 2\.4Training\-Free Finetuning for Large Models

Training\-free tuning methods have garnered considerable attention as effective strategies for enhancing the inference\-time performance of LLMs and VLMs without incurring additional computational overhead from parameter optimization\. This kind of methods mainly explores token\-level attention modulation for improving foundation model performance without any parameter modifications\. These methods are based on the well\-documented “attention sink” phenomenon, wherein a disproportionately large share of attention weight is systematically concentrated on the initial Beginning\-of\-Sequence \(BOS\) token\(Hanet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib27); Xiaoet al\.,[2024](https://arxiv.org/html/2607.13395#bib.bib28); Kaulet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib29); Guet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib31); Barberoet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib30)\)\. Attention sink motivates weight re\-distribution techniques\. Representative approaches such as Post\-hoc Attention Steering \(PASTA,Zhanget al\.\([2024a](https://arxiv.org/html/2607.13395#bib.bib64)\)\) and AutoPASTAZhanget al\.\([2024b](https://arxiv.org/html/2607.13395#bib.bib65)\)identify and selectively amplify the attention weights assigned to task\-relevant tokens, while Attention Calibration \(ACT\) suppresses non\-initial sink tokens to redistribute attention more effectivelyYuet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib66)\)\. This principle has also been extended to vision\-language models, where attention re\-weighting toward visual tokens has been shown to effectively mitigate hallucinationsLiuet al\.\([2024a](https://arxiv.org/html/2607.13395#bib.bib67)\); Zhuet al\.\([2025](https://arxiv.org/html/2607.13395#bib.bib68)\)\. More recently, Han et al\. introduced ZeroTuning that applies head\-specific attention adjustments to the BOS token, demonstrating consistent and generalizable performance improvements across foundation LLMsHanet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib27)\)\.

As Table[1](https://arxiv.org/html/2607.13395#S1.T1)shows,Enlightenmentrepresents a new paradigm for training\-free finetuning methods\. Our opinion is that weights and architecture are the two cornerstones of a model\. While existing training\-free adaptations predominantly manipulate attention weights,our work reveals that we can also re\-allocate weight distribution from an architectural perspective\. This architectural view opens up an orthogonal yet equally promising direction, which is to treat model distribution as a global structural property rather than a parametric one\.To the best of our knowledge, this view goes beyond weight\-centric finetuning paradi\- gms, which was not explored before\.

## 3Methodology

The overall framework ofEnlightenmentis illustrated in Figure[1](https://arxiv.org/html/2607.13395#S1.F1)\. The motivation behind our approach draws a loose conceptual analogy to neuroscience, where sudden insights in human learning are sometimes associated with rapid reorganization of neural pathways\. We adopt this only as a high\-level intuition, without claiming a direct biological basis\. To reflect this notion in machine learning, we modify structures by either adjusting existing or establishing new shortcuts within pre\-trained models, enabling them to self\-improve their performance during inference without requiring additional post\-training\.

The unique characteristics ofEnlightenmentare that it requires no training data, no gradient computation, and no weight updates\. It is built upon a unified principle:modifying shortcuts for key modules/layersby exerting scaling factors over them, while establishing new shortcuts can be thought as turning the scaling factor from 0 to 1 for otherwise absent paths\. The standard residual connection offers no mechanism to control the*mixing*between the residual stream and the sublayer output at inference time\. Specifically, shortcuts are assigned with dynamic scaling parameters, ensuring the network can dynamically calibrate its attention to specific layers and more precisely govern the propagation strength of features\. As a result,Enlightenmentcan enrich the representational capacity of large models without altering or tuning their pre\-trained weights\. What’s more favorable is that this enhanced expressivity is generalizable across diverse tasks and model families\.

While the underlying principle is shared, the integration points and companion modules regarding shortcut modification could differ across different model families\. From an information\-flow perspective, multimodal networks require residual damping to suppress visual noise, while text\-only models require attention modulation to redistribute linguistic focusJianget al\.\([2026a](https://arxiv.org/html/2607.13395#bib.bib98)\); Kimet al\.\([2016](https://arxiv.org/html/2607.13395#bib.bib99)\); Xiaoet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib28)\)\. Now, we elaborate on how we perform shortcut modification on VLMs and LLMs, respectively\.

### 3\.1Enlightenment for LLM

LLMs frequently exhibit an “attention sink” phenomenon, wherein disproportional attention weight is systematically allocated to the initial BOS token\(Hanet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib27); Xiaoet al\.,[2024](https://arxiv.org/html/2607.13395#bib.bib28); Kaulet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib29); Guet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib31); Barberoet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib30)\)\. Beyond attention weights modification, our proposed method recalibrates BOS attention behavior by mixing heads with scaled shortcuts\. This provides a per\-head level recalibration rather than globally suppressing BOS attention\. In this section, we first examine the basic mechanics of the attention in LLMs before detailing theEnlightenmentintegration\.

Attention mechanism\. Modern LLMs rely on the multi\-head attention \(MHA\) mechanism to capture complex dependencies across tokenized sequences\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.13395#bib.bib33)\)\. We define an input sequence as a matrix𝐗∈ℝL×dmodel\\mathbf\{X\}\\in\\mathbb\{R\}^\{L\\times d\_\{\\text\{model\}\}\}, whereLLdenotes the sequence length \(total number of tokens\) anddmodeld\_\{\\text\{model\}\}represents the hidden state dimension\.

To process information across distinct representation subspaces, the hidden dimensiondmodeld\_\{\\text\{model\}\}is split intoHHparallel attention heads, each operating on a lower\-dimensional feature spacedh=dmodel/Hd\_\{h\}=d\_\{\\text\{model\}\}/H\. For a given attention headh∈\{1,…,H\}h\\in\\\{1,\\dots,H\\\}, the input activation sequence𝐗\\mathbf\{X\}is linearly projected into query \(𝐐h\\mathbf\{Q\}\_\{h\}\), key \(𝐊h\\mathbf\{K\}\_\{h\}\), and value \(𝐕h\\mathbf\{V\}\_\{h\}\) matrices using learned projection weight matrix:

𝐐h=𝐗𝐖qh,𝐊h=𝐗𝐖kh,𝐕h=𝐗𝐖vh\.\\mathbf\{Q\}\_\{h\}=\\mathbf\{X\}\\mathbf\{W\}\_\{q\}^\{h\},\\quad\\mathbf\{K\}\_\{h\}=\\mathbf\{X\}\\mathbf\{W\}\_\{k\}^\{h\},\\quad\\mathbf\{V\}\_\{h\}=\\mathbf\{X\}\\mathbf\{W\}\_\{v\}^\{h\}\.\(1\)Here,𝐖qh,𝐖kh,𝐖vh∈ℝdmodel×dh\\mathbf\{W\}\_\{q\}^\{h\},\\mathbf\{W\}\_\{k\}^\{h\},\\mathbf\{W\}\_\{v\}^\{h\}\\in\\mathbb\{R\}^\{d\_\{\\text\{model\}\}\\times d\_\{h\}\}are head\-specific projection matrices\. The scaled dot\-product attention for thehh\-th head maps the queries and keys to a valid probability distribution via the softmax operator, weighting the corresponding values accordingly:

𝐀h\\displaystyle\\mathbf\{A\}\_\{h\}=softmax​\(𝐐h​𝐊h⊤dh\)∈ℝL×L\\displaystyle=\\mathrm\{softmax\}\\left\(\\frac\{\\mathbf\{Q\}\_\{h\}\\mathbf\{K\}\_\{h\}^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\\in\\mathbb\{R\}^\{L\\times L\}\(2\)𝐎h\\displaystyle\\mathbf\{O\}\_\{h\}=AttnOutput​\(𝐐h,𝐊h,𝐕h\)=𝐀h​𝐕h∈ℝL×dh\.\\displaystyle=\\mathrm\{AttnOutput\}\(\\mathbf\{Q\}\_\{h\},\\mathbf\{K\}\_\{h\},\\mathbf\{V\}\_\{h\}\)=\\mathbf\{A\}\_\{h\}\\mathbf\{V\}\_\{h\}\\in\\mathbb\{R\}^\{L\\times d\_\{h\}\}\.In this formulation,𝐀h\\mathbf\{A\}\_\{h\}represents the post\-softmax attention weight matrix, where each elementAh,q,kA\_\{h,q,k\}defines the normalized alignment score between theqq\-th query token and thekk\-th key token\. The factordh\\sqrt\{d\_\{h\}\}acts as a regularizing scaling variance to alleviate vanishing gradients during backpropagation\.

The individual head outputs𝐎h\\mathbf\{O\}\_\{h\}are stacked across the head dimension to a tensor𝐎∈ℝH×L×dh\\mathbf\{O\}\\in\\mathbb\{R\}^\{H\\times L\\times d\_\{h\}\}\(or batched asℝB×H×L×dh\\mathbb\{R\}^\{B\\times H\\times L\\times d\_\{h\}\}\)\. In a vanilla transformer, these heads are aggregated via a learned linear combination using the output projection weights𝐖oh∈ℝdh×dmodel\\mathbf\{W\}\_\{o\}^\{h\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times d\_\{\\text\{model\}\}\}:

MHA​\(𝐗\)=∑h=0H𝐎h​𝐖oh\.\\mathrm\{MHA\}\(\\mathbf\{X\}\)=\\sum\_\{h=0\}^\{H\}\\mathbf\{O\}\_\{h\}\\mathbf\{W\}\_\{o\}^\{h\}\.\(3\)
Enlightenmentmodifies the attention head𝐀h\\mathbf\{A\}\_\{h\}across a designated subset of target layersℒ\\mathcal\{L\}and headsℋ\\mathcal\{H\}through head\-to\-head information mixing\. By coupling the recalibration with the scattering shortcuts, the model successfully redistributes its linguistic focus\. The overall framework is depicted in Figure[2](https://arxiv.org/html/2607.13395#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x2.png)Figure 2:The architectural framework of Enlightenment for LLMs\.Attention head mixing by inserting new shortcuts\. First, letAb,h,q,kA\_\{b,h,q,k\}denote the attention weight that theqq\-th query position assigns to thekk\-th key position in thehh\-th head\. For a target layerℓ∈ℒ\\ell\\in\\mathcal\{L\}and headh∈ℋh\\in\\mathcal\{H\}, leveraging the attention sink, we assign a scaling parameterγℓ,h\>0\\gamma\_\{\\ell,h\}\>0to the initial attention matrix:

\{A¯b,h,q,0=γh⋅Ab,h,q,0A¯b,h,q,≥1=Ab,h,q,≥1,\\begin\{cases\}&\\bar\{A\}\_\{b,h,q,0\}=\\gamma\_\{h\}\\cdot A\_\{b,h,q,0\}\\\\ &\\bar\{A\}\_\{b,h,q,\\geq 1\}=A\_\{b,h,q,\\geq 1\},\\end\{cases\}\(4\)and normalize the values of newly scaled attention matricesA¯\\bar\{A\}across all positions:

A~b,h,q,:=A¯b,h,q,:∑kA¯b,h,q,k\.\\tilde\{\{A\}\}\_\{b,h,q,:\}=\\frac\{\{\\bar\{A\}\}\_\{b,h,q,:\}\}\{\\sum\_\{k\}\\bar\{A\}\_\{b,h,q,k\}\}\.\(5\)According to Eq\. \([2](https://arxiv.org/html/2607.13395#S3.E2)\), the output of thehh\-th attention head is

𝐎h=𝐀~h​𝐕h=\[𝐀~h,0​𝐕h,0,𝐀~h,1​𝐕h,1,⋯,𝐀~h,n​𝐕h,n\]\.\{\\mathbf\{O\}\}\_\{h\}=\\tilde\{\\mathbf\{A\}\}\_\{h\}\\mathbf\{V\}\_\{h\}=\[\\tilde\{\\mathbf\{A\}\}\_\{h,0\}\\mathbf\{V\}\_\{h,0\},\\tilde\{\\mathbf\{A\}\}\_\{h,1\}\\mathbf\{V\}\_\{h,1\},\\cdots,\\tilde\{\\mathbf\{A\}\}\_\{h,n\}\\mathbf\{V\}\_\{h,n\}\]\.\(6\)
After that, we establish new shortcuts that link the output of head 0 to all other target heads\. For each edge from source head 0 to a target destination headhh, we set a mixing rateα\\alphato control the additive injection\. The matrix combining all heads’ output is

𝐎~=\[𝐀~0​𝐕0,α​𝐀~0​𝐕0\+𝐀~1​𝐕1,⋯,α​𝐀~0​𝐕0\+𝐀~H​𝐕H\]\.\\tilde\{\\mathbf\{O\}\}=\[\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\},\\alpha\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\}\+\\tilde\{\\mathbf\{A\}\}\_\{1\}\\mathbf\{V\}\_\{1\},\\cdots,\\alpha\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\}\+\\tilde\{\\mathbf\{A\}\}\_\{H\}\\mathbf\{V\}\_\{H\}\]\.\(7\)Whenα=1\\alpha=1, the attention module remains unmodified\. This process enriches the representation of thehh\-th head with contextual contributions from the initial head, which is a feature reuse\.

Finally, based on Eq\. \([3](https://arxiv.org/html/2607.13395#S3.E3)\), the output of the MHA is

MHA​\(𝐗\)\\displaystyle\\mathrm\{MHA\}\(\\mathbf\{X\}\)=𝐀~0​𝐕0​𝐖0\+\(α​𝐀~0​𝐕0\+𝐀~1​𝐕1\)​𝐖1\+\\displaystyle=\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\}\\mathbf\{W\}\_\{0\}\+\(\\alpha\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\}\+\\tilde\{\\mathbf\{A\}\}\_\{1\}\\mathbf\{V\}\_\{1\}\)\\mathbf\{W\}\_\{1\}\+\(8\)⋯\+\(α​𝐀~0​𝐕0\+𝐀~H​𝐕H\)​𝐖H\.\\displaystyle\\cdots\+\(\\alpha\\tilde\{\\mathbf\{A\}\}\_\{0\}\\mathbf\{V\}\_\{0\}\+\\tilde\{\\mathbf\{A\}\}\_\{H\}\\mathbf\{V\}\_\{H\}\)\\mathbf\{W\}\_\{H\}\.These new shortcuts make the LLM dynamically redistribute linguistic focus from the initial attention block to other heads without introducing additional weights or training overhead\.

Adaptive hyperparameter modulation\. HyperparametersΓ=\[γ1,γ2,⋯,γH\]\\Gamma=\[\\gamma\_\{1\},\\gamma\_\{2\},\\cdots,\\gamma\_\{H\}\]andα\\alphaare introduced inEnlight\-enment\. Among them,α\\alphais a global parameter whose value is easy to determine by grid search\. In contrast, in previous works, to alleviate the attention sink, initial heads exhibiting high attention values should be attenuated, while those with low attention should be amplified\(Hanet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib27); Xiaoet al\.,[2024](https://arxiv.org/html/2607.13395#bib.bib28); Kaulet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib29); Guet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib31); Barberoet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib30)\)\. Therefore,Γ\\Gammashould be carefully calibrated for each head across layers to conform to this phenomenon\. To this end, we introduce an adaptive strategy:

∙\\small\\bulletFirst, we calculate the value of the initial attention matrix of each head:

a0\(h\)=1B⋅Q​∑b=1B∑q=1QAb,h,q,0,a\_\{0\}^\{\(h\)\}=\\frac\{1\}\{B\\cdot Q\}\\sum\_\{b=1\}^\{B\}\\sum\_\{q=1\}^\{Q\}A\_\{b,h,q,0\},\(9\)which serves as a runtime indicator of how strongly thehh\-th head attends to the initial position\. We normalizea0\(h\)a\_\{0\}^\{\(h\)\}to\[−1,1\]\[\-1,1\]via

a^0\(h\)=2⋅a0\(h\)−aminamax−amin−1,\\hat\{a\}\_\{0\}^\{\(h\)\}=2\\cdot\\frac\{a\_\{0\}^\{\(h\)\}\-a\_\{\\min\}\}\{a\_\{\\max\}\-a\_\{\\min\}\}\-1,\(10\)whereamaxa\_\{\\max\}andamina\_\{\\min\}are the extreme values ina\(h\)a^\{\(h\)\}\.

∙\\small\\bulletSecond, we aim at constructing a functionγ=f​\(a^0\)\\gamma=f\(\\hat\{a\}\_\{0\}\)\. The general principle is that this function should be smooth over mid\-range values, large for small attention values, and small for large attention values, thereby producing a more balanced scaling for attention values\. To achieve this, we define four anchor pairs to ensure a basic shape forff\. Each pair maps a high attention value to a lowerγ\\gamma, while a low initial attention value is amplified:

\(a0^,γ\)∈\{\(0\.05,1\.5\),\(0\.15,1\.1\),\(0\.30,1\),\(0\.50,0\.7\)\}\.\(\\hat\{a\_\{0\}\},\\gamma\)\\in\\\{\(0\.05,1\.5\),\(0\.15,1\.1\),\(0\.30,1\),\(0\.50,0\.7\)\\\}\.\(11\)We fit these four anchor points to the Chebyshev polynomial function:

γh=Td​\(xh;𝐜\)=∑j=0dcj​Tj​\(xh\),\\gamma\_\{h\}=T\_\{d\}\(x\_\{h\};\\mathbf\{c\}\)=\\sum\_\{j=0\}^\{d\}c\_\{j\}\\,T\_\{j\}\(x\_\{h\}\),\(12\)with𝐜=\[c0,…,cd\]\\mathbf\{c\}=\[c\_\{0\},\\dots,c\_\{d\}\]are coefficients, andTjT\_\{j\}represents thejj\-th Chebyshev polynomial\(Islamet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib32)\)satisfying

T0​\(x\)=1,T1​\(x\)=x,Tj​\(x\)=2​x​Tj−1​\(x\)−Tj−2​\(x\)\.T\_\{0\}\(x\)=1,T\_\{1\}\(x\)=x,T\_\{j\}\(x\)=2xT\_\{j\-1\}\(x\)\-T\_\{j\-2\}\(x\)\.\(13\)
Figure[3](https://arxiv.org/html/2607.13395#S3.F3)presents a 3rd\-order Chebyshev polynomial function\.γh\\gamma\_\{h\}of each head can be determined by this function\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x3.png)Figure 3:The fitted 3\-order Chebyshev polynomial curve\.Furthermore, we setβ\\betaas a task\-dependent factor, which scales the Chebyshev polynomialTdT\_\{d\}horizontally\.

γh=β⋅Td​\(xh;𝐜\)\.\\gamma\_\{h\}=\\beta\\cdot T\_\{d\}\(x\_\{h\};\\mathbf\{c\}\)\.\(14\)Adjustingβ\\betacan effectively handle domain/task shift\.

Table[2](https://arxiv.org/html/2607.13395#S3.T2)presents all the hyperparameters and their default values utilized inEnlightenmentfor LLMs\.

Table 2:Default hyper\-parameters of the Enlightenment\.
### 3\.2Enlightenment for VLM

The core idea is to apply a single shared scalar to every residual connection within the Transformer decoder layers\. Pre\-trained sublayers, such as attention and Feed\-Forward Networks, frequently become overly active during inferenceYouet al\.\([2026](https://arxiv.org/html/2607.13395#bib.bib100)\); Korthikantiet al\.\([2023](https://arxiv.org/html/2607.13395#bib.bib101)\)\. This excessive activity leads to excessive information, introducing visual noise that degrades downstream predictive performance\. By uniformly scaling down these sublayer outputs, the method effectively applies residual damping\. This mechanism acts as a regularizer, suppressing superfluous sublayer activity while securely preserving the critical residual pathway that carries the core embedding information\.

This method integrates scalar\-modulated residual into the linear operations across layers, where a scaling hyperparameter can be adjusted to control contributions of these connections\. The overall framework is illustrated in Figure[4](https://arxiv.org/html/2607.13395#S3.F4)\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x4.png)Figure 4:The architectural framework of Enlightenment for VLMs\.Scalar\-modulated residuals\. The standard transformer decoder layer computesxℓ\+1=xℓ\+Fℓ​\(LN​\(xℓ\)\)x\_\{\\ell\+1\}=x\_\{\\ell\}\+F\_\{\\ell\}\(\\text\{LN\}\(x\_\{\\ell\}\)\), whereFℓF\_\{\\ell\}is either self\-attention or an FFN\. We replace this with a single scalar\-modulated variant:

xℓ\+1=xℓ\+η⋅Fℓ​\(LN​\(xℓ\)\),x\_\{\\ell\+1\}=x\_\{\\ell\}\+\\eta\\cdot F\_\{\\ell\}\(\\text\{LN\}\(x\_\{\\ell\}\)\),\(15\)
whereη∈\(0,1\]\\eta\\in\(0,1\]is a scalar shared across allLLdecoder layers and both sublayer types \(attention and FFN\)\. Atη=1\\eta=1, the model operates in its unmodified state\. Atη=0\.5\\eta=0\.5, the sublayer outputs are uniformly halved, introducing a form of*residual damping*that mitigates noise while preserving the primary residual pathway carrying the original embedding information\.

Table 3:Structural comparison of VLM and LLM variants by Enlightenment\.The computational overhead is minimal, requiring only one scalar hyperparameter and one additional multiplication operation per layer\. Consequently,Enlightenmentrequires no training data, no gradient computation, no weight updates\. Our ablation studies demonstrate thatη\\etacan be easily tuned within the interval\[0\.1,1\]\[0\.1,1\]\. Regarding the target layers, applyingEnlightenmentacross all layers serves as a robust default strategy that consistently yields performance improvements\.

In summary, Table[3](https://arxiv.org/html/2607.13395#S3.T3)compares the structural modifications applied to VLMs and LLMs by the proposedEnlight\-enmentframework\.

## 4Experiments on LLMs

### 4\.1Experimental Setup

Models\.We evaluateEnlightenmenton three models: Qwen3\.5\-9B\-Base\(Qwen Team,[2026](https://arxiv.org/html/2607.13395#bib.bib37)\), DeepSeek\-R1\-Distill\-Qwen\-14B\(DeepSeek\-AI,[2025](https://arxiv.org/html/2607.13395#bib.bib38)\), and Mimo\-7B\-RLXiaomi \([2025](https://arxiv.org/html/2607.13395#bib.bib47)\)\. Specifically, Qwen3\.5 uses hybrid attention with 8 full\-attention layers\. DeepSeek\-R1\-14B is based on Qwen2 with 40 layers\. Mimo\-7B is a Qwen2\-based model with 36 layers, GQA \(32 query heads, 8 KV heads\), 4096 hidden size, and SiLU activation, trained with RL\. All results are test on 3 seeds \(11, 22, 33\) and compute the average values, with all decoder layers applied scalar\-modulated factor forEnlightenment\.

Table 4:Accuracy \(%\) of LLMs comparison\. Best results per row in bold\.Qwen3\.5\-9BDeepSeek\-R1\-14BMimo\-7B\-RLDatasetBaseZTEnlightenmentBaseZTEnlightenmentBaseZTEnlightenmentSST\-2\\cellcolor\[HTML\]DAE8FC92\.2\\cellcolor\[HTML\]DAE8FC92\.3\\cellcolor\[HTML\]FFFFE493\.4\\cellcolor\[HTML\]DAE8FC53\.4\\cellcolor\[HTML\]DAE8FC54\.3\\cellcolor\[HTML\]FFFFE459\.2\\cellcolor\[HTML\]DAE8FC90\.7\\cellcolor\[HTML\]DAE8FC90\.6\\cellcolor\[HTML\]FFFFE491\.7CB\\cellcolor\[HTML\]DAE8FC91\.1\\cellcolor\[HTML\]DAE8FC92\.9\\cellcolor\[HTML\]FFFFE493\.9\\cellcolor\[HTML\]DAE8FC79\.2\\cellcolor\[HTML\]DAE8FC80\.4\\cellcolor\[HTML\]FFFFE478\.6\\cellcolor\[HTML\]DAE8FC80\.4\\cellcolor\[HTML\]DAE8FC80\.4\\cellcolor\[HTML\]FFFFE482\.4BoolQ\\cellcolor\[HTML\]DAE8FC88\.2\\cellcolor\[HTML\]DAE8FC89\.3\\cellcolor\[HTML\]FFFFE490\.1\\cellcolor\[HTML\]DAE8FC58\.3\\cellcolor\[HTML\]DAE8FC57\.4\\cellcolor\[HTML\]FFFFE465\.8\\cellcolor\[HTML\]DAE8FC71\.5\\cellcolor\[HTML\]DAE8FC70\.6\\cellcolor\[HTML\]FFFFE472\.1ARCE\\cellcolor\[HTML\]DAE8FC94\.0\\cellcolor\[HTML\]DAE8FC94\.1\\cellcolor\[HTML\]FFFFE494\.3\\cellcolor\[HTML\]DAE8FC92\.6\\cellcolor\[HTML\]DAE8FC92\.7\\cellcolor\[HTML\]FFFFE492\.7\\cellcolor\[HTML\]DAE8FC89\.5\\cellcolor\[HTML\]DAE8FC89\.6\\cellcolor\[HTML\]FFFFE490\.8ARCC\\cellcolor\[HTML\]DAE8FC93\.0\\cellcolor\[HTML\]DAE8FC93\.0\\cellcolor\[HTML\]FFFFE494\.3\\cellcolor\[HTML\]DAE8FC87\.3\\cellcolor\[HTML\]DAE8FC87\.3\\cellcolor\[HTML\]FFFFE487\.0\\cellcolor\[HTML\]DAE8FC84\.6\\cellcolor\[HTML\]DAE8FC84\.6\\cellcolor\[HTML\]FFFFE486\.6CQA\\cellcolor\[HTML\]DAE8FC54\.7\\cellcolor\[HTML\]DAE8FC54\.7\\cellcolor\[HTML\]FFFFE455\.5\\cellcolor\[HTML\]DAE8FC75\.9\\cellcolor\[HTML\]DAE8FC77\.3\\cellcolor\[HTML\]FFFFE476\.7\\cellcolor\[HTML\]DAE8FC76\.6\\cellcolor\[HTML\]DAE8FC76\.6\\cellcolor\[HTML\]FFFFE477\.8PIQA\\cellcolor\[HTML\]DAE8FC49\.3\\cellcolor\[HTML\]DAE8FC50\.6\\cellcolor\[HTML\]FFFFE451\.4\\cellcolor\[HTML\]DAE8FC85\.1\\cellcolor\[HTML\]DAE8FC86\.3\\cellcolor\[HTML\]FFFFE485\.0\\cellcolor\[HTML\]DAE8FC82\.6\\cellcolor\[HTML\]DAE8FC83\.5\\cellcolor\[HTML\]FFFFE483\.9Avg\.\\cellcolor\[HTML\]DAE8FC80\.4\\cellcolor\[HTML\]DAE8FC81\.0\\cellcolor\[HTML\]FFFFE481\.8\\cellcolor\[HTML\]DAE8FC76\.0\\cellcolor\[HTML\]DAE8FC76\.5\\cellcolor\[HTML\]FFFFE477\.9\\cellcolor\[HTML\]DAE8FC82\.3\\cellcolor\[HTML\]DAE8FC82\.3\\cellcolor\[HTML\]FFFFE483\.6Avg\.Δ\\Deltavs base\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.6\\cellcolor\[HTML\]FFFFE4\+1\.5\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.6\\cellcolor\[HTML\]FFFFE4\+1\.9\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.0\\cellcolor\[HTML\]FFFFE4\+1\.3In addition, model compression is increasingly useful in the era of LLMs\. In the field of model compression, the performance degradation needs to be controlled carefully\. Therefore, we also evaluate ifEnlightenmentworks for the compressed LLM\. Now, in industry, the mainstream model compression is based on quantization\. We evaluate theW4A4C2quantization on the DeepSeek\-R1\-Distill\-Qwen\-1\.5B checkpoint\. Specifically, the base model is a full\-precision FP16 baseline, while theW4A4C2quantization model uses W4 weight dequantization \(4\-bit restored\), A4 activation fake\-quantization via LobCQ \(4\-bit, group size 128, outlier ratio 0\.01\), and C2 KV\-cache vector quantization \(2\-bit K/V, group size 32, residual length 32\)\.

Datasets\.Our evaluation spans several standard benchmarks\. For classification, we include SST\-2 \(Stanford Sentiment Treebank; sentiment analysis; 2 classes\)\(Socheret al\.,[2013](https://arxiv.org/html/2607.13395#bib.bib40)\), CB \(CommitmentBank; natural language inference; 3 classes\)\(de Marneffeet al\.,[2019](https://arxiv.org/html/2607.13395#bib.bib41)\), and BoolQ \(Boolean Questions; yes/no QA; 2 classes\)\(Clarket al\.,[2019](https://arxiv.org/html/2607.13395#bib.bib42)\)\. For multiple\-choice and multi\-class tasks, we use ARCE \(AI2 Reasoning Challenge Easy; multiple\-choice science QA; 4 classes\)\(Clarket al\.,[2018](https://arxiv.org/html/2607.13395#bib.bib43)\), ARCC \(AI2 Reasoning Challenge Challenge; multiple\-choice science QA; 4 classes\)\(Clarket al\.,[2018](https://arxiv.org/html/2607.13395#bib.bib43)\), CQA \(CommonsenseQA; multiple\-choice commonsense QA; 5 classes\)\(Talmoret al\.,[2019](https://arxiv.org/html/2607.13395#bib.bib44)\), and PIQA \(Physical Interaction QA; multiple\-choice physical commonsense QA; 2 classes\)\(Bisket al\.,[2020](https://arxiv.org/html/2607.13395#bib.bib45)\)\. Across all datasets, we adopt perplexity\-based scoring over label templates\. For compressed models, all results use the vendored lm\-evaluation\-harness111[https://github\.com/EleutherAI/lm\-evaluation\-harness](https://github.com/EleutherAI/lm-evaluation-harness)\. Tasks are evaluated in a zero\-shot, log\-likelihood\-based multiple\-choice setting with batch size 8, max length 2048, and no bootstrapping\.

Baselines\.We compareEnlightenmentagainst ZeroTuning \(ZT\)\(Hanet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib27)\)\. This method is the most recent SOTA training\-free tuning method for LLMs, serving as an updated version of earlier methodsZhanget al\.\([2024a](https://arxiv.org/html/2607.13395#bib.bib64),[b](https://arxiv.org/html/2607.13395#bib.bib65)\); Yuet al\.\([2024](https://arxiv.org/html/2607.13395#bib.bib66)\)\. We therefore adopt it as the sole baseline for comparison\. ForEnlightenment, we apply attention head mixing with uniform per\-head rates\. All methods include no training and are evaluated at inference time\. Specifically, for the W4A4C2 quantization model, we apply theEnlightenmentattention head mixing to all layers and all heads at a base rate ofα=0\.1,β=0\.5\\alpha=0\.1,\\beta=0\.5\.

### 4\.2Main Results

Table[4](https://arxiv.org/html/2607.13395#S4.T4)compares the accuracy of three LLMs across seven benchmarks\. Overall,Enlightenmentachieves the best average performance for all three models, improving Qwen3\.5\-9B from 80\.4% to 81\.8%, DeepSeek\-R1\-14B from 76\.0% to 77\.9%, and Mimo\-7B\-RL from 82\.3% to 83\.6%\. Compared with the base setting,Enlightenmentyields average gains of \+1\.5, \+1\.9, and \+1\.3 points, respectively, while ZT provides smaller or negligible improvements\. At the dataset level, Enlightenment obtains the highest accuracy in most cases, particularly showing clear gains on SST\-2, BoolQ, and ARCC\. Although ZT slightly outperforms Enlightenment on some tasks for DeepSeek\-R1\-14B, such as CB, CQA, and PIQA, overall,Enlightenmentprovides a more consistent and stronger improvement across models and datasets\.

Table 5:Using Enlightenment \(all layers/heads\) after quantization LLM\.Table 6:Comparison of perplexity\-based marginal improvement\. “Margin” =log⁡P​\(correct\)−log⁡P​\(wrong\)\\log P\(\\text\{correct\}\)\-\\log P\(\\text\{wrong\}\), higher is better\.Table 7:Full output of a CommonsenseQA example\.Table[5](https://arxiv.org/html/2607.13395#S4.T5)demonstrates thatEnlightenmentconsistently improves the performance of the quantized LLM across all evaluated tasks\. Under the aggressive W4A4C2 quantization setting, the average score drops substantially from 0\.580 in FP16 to 0\.396, indicating a clear degradation caused by low\-bit quantization\. ApplyingEnlightenmentafter quantization raises the average score to 0\.441, yielding a \+0\.045 absolute improvement over W4A4C2\. The gain is especially pronounced on BoolQ, where performance increases from 0\.378 to 0\.567, recovering a large portion of the accuracy lost during quantization\. Improvements are also observed on RTE, WinoGrande, ARCE, PIQA, and MMLU, showing that the benefit is not limited to a single benchmark\. These results suggest thatEnlightenmentcan effectively mitigate quantization\-induced degradation and enhance the robustness of low\-bit LLM inference without restoring the model to full precision\.

We also provide qualitative examples comparing baseline andEnlightenmenton the SST\-2 sentiment classification task using Qwen3\.5\-9B\-Base\. The model is prompted with

Sentence: \.\.\.

Answer: \.\.\.

While individual greedy\-decoded outputs are often identical, the improvement from Enlightenment operates on the log\-probability margin between the correct and incorrect label\. Table[6](https://arxiv.org/html/2607.13395#S4.T6)shows four examples from the SST\-2 validation set where Enlightenment increases the log\-probability margin for the correct label\. The “Margin” column shows the differencelog⁡P​\(correct\)−log⁡P​\(wrong\)\\log P\(\\text\{correct\}\)\-\\log P\(\\text\{wrong\}\); larger is better\.

In each case, the Enlightenment suppresses the log\-probab\- ility of the wrong label \(more negative\) while marginally increasing or maintaining the correct label probability, widening the margin by\+0\.19\+0\.19to\+0\.31\+0\.31log\-probability units\.

Moreover, Table[7](https://arxiv.org/html/2607.13395#S4.T7)shows a CommonsenseQA example\. CommonsenseQA is a 5\-way multiple\-choice task\. The evaluation computeslog⁡P​\(answer letter\)\\log P\(\\text\{answer letter\}\)for each of the five choices and selects the highest\. Under the baseline, the model assigns nearly identical log\-probabilities to choices A and B \(both−5\.97\-5\.97\), and the tie is resolved in favor of the wrong answer\. Enlightenment shifts the distribution: choice B’s score rises relative to choice A \(AAfrom−5\.97\-5\.97to−6\.06\-6\.06,BBfrom−5\.97\-5\.97to−6\.00\-6\.00\), producing a clear margin of0\.060\.06nats in favor of the correct answer\. The remaining choices \(C, D, E\) are unaffected because the BOS scaling targets only the top\-ranking candidates\.

### 4\.3Hyperparameter Sensitivity

We evaluate the sensitivity of hyperparameters in theEnligh\-tenmentby varying each hyperparameter in isolation while keeping all others at their defaults\. The evaluation is based on the BoolQ task \(100 samples, seed 11\)\. Table[8](https://arxiv.org/html/2607.13395#S4.T8)reports accuracy across two hyperparameters, polynomial degree \(dd\) and mixing strength \(α\\alpha\)\. As the task\-dependent factorβ\\betacan be used to finetune the model in different tasks, we set it to 1 in this experiment\.

Table 8:Hyperparameter sensitivity on BoolQ \(Qwen3\.5\-9B, 100×\\times1\)\. Default values are bold\-faced\.SectionValueAcc\. \(%\)\\rowcolor\[HTML\]DAE8FC\\cellcolor\[HTML\]DAE8FC394\.0\\rowcolor\[HTML\]DAE8FC\\cellcolor\[HTML\]DAE8FC192\.0\\rowcolor\[HTML\]DAE8FC\\cellcolor\[HTML\]DAE8FC293\.0\\rowcolor\[HTML\]DAE8FC\\cellcolor\[HTML\]DAE8FCPloynomial degreedd594\.0\\rowcolor\[HTML\]FFFFE4\\cellcolor\[HTML\]FFFFE40\.194\.0\\rowcolor\[HTML\]FFFFE4\\cellcolor\[HTML\]FFFFE41 \(disabled\)91\.0\\rowcolor\[HTML\]FFFFE4\\cellcolor\[HTML\]FFFFE40\.0594\.0\\rowcolor\[HTML\]FFFFE4\\cellcolor\[HTML\]FFFFE40\.294\.0\\rowcolor\[HTML\]FFFFE4\\cellcolor\[HTML\]FFFFE4Mixingα\\alpha0\.594\.0The hyperparameter sensitivity results on BoolQ show that the method is generally robust to changes in both the polynomial degreeddand the mixing coefficientα\\alpha\. Using the default settingd=3d=3achieves the best accuracy of 94\.0%, while lower degrees lead to slight drops, withd=1d=1reaching 92\.0% andd=2d=2reaching 93\.0%\. This suggests that a moderate polynomial degree is sufficient to capture the desired effect, and increasing it further does not provide additional gains\. For the mixing coefficient, the default valueα=0\.1\\alpha=0\.1also achieves 94\.0% accuracy, and performance remains unchanged across different settings\. In contrast, disabling mixing by settinga​l​p​h​a=1alpha=1reduces accuracy to 91\.0%, indicating that the mixing mechanism is important for maintaining strong performance\.

### 4\.4Ablation Study

Adaptive hyperparameter modulation\.We use Qwen3\.5\-9B to conduct the ablation study\. Table[9](https://arxiv.org/html/2607.13395#S4.T9)evaluates the individual and combined impact of our proposed mechanisms by isolating the head mixing component \(HeadMix only\) and applying adaptive hyperparameter modulation \(Adap\.\+ HeadMix\)\. First, applying HeadMix yields highly inconsistent results across tasks, leading to a minor average regression to 80\.3% compared to the Base model’s 80\.4%\. This instability indicates that while head mixing can capture specialized attention patterns, it introduces optimization variability\. In contrast, the joint configuration \(Adap\.\+HeadMix\) successfully mitigates these vulnerabilities, securing the highest performance on six out of seven datasets and achieving a superior overall average accuracy of 81\.8%\. The adaptive hyperparameter modulation framework acts as a vital stabilizer, reversing the isolated regressions of HeadMix on SST\-2 and CQA to produce net gains over Base \(\+1\.2% and \+0\.8%, respectively\)\. This strong synergy is particularly pronounced on reasoning\-heavy datasets like CB \(\+2\.8%\) and PIQA \(\+2\.1%\)\.

Table 9:Accuracy \(%\) of component ablation on Qwen3\.5\-9B\.Different layer and head ranges\.We conduct experiments to analyze the impact of applyingEnlightenmentin different layer and head ranges\. The results are presented in Table[10](https://arxiv.org/html/2607.13395#S4.T10)\. For target layersℒ\\mathcal\{L\}, it indicates that our method is highly robust to the choice of target layers, since all tested layer configurations, ranging from full\-layer intervention to sparse subsets, consistently achieve94\.0%94\.0\\%accuracy\. This suggests that a small number of selected layers is already sufficient to capture the relevant effect\. For target headsℋ\\mathcal\{H\}, the trend is similarly stable: most partial head selections yield94\.0%94\.0\\%, while using all heads provides a slightly higher accuracy of95\.0%95\.0\\%\. Overall, these findings show that the approach is not sensitive to precise layer/head selection\.

Table 10:Changing Enlightenment layers and heads on BoolQ \(Qwen3\.5\-9B, 100×\\times1\)\.Table 11:Accuracy \(%\) of compared post\-tuning methods across 5 benchmarks\. Best results per row in bold\.MiMo\-VL\-7BQwen2\-VL\-7BQwen3\-VL\-8BDataset\# SamplesBaseZTEnlightenmentBaseZTEnlightenmentBaseZTEnlightenmentMMBench300\\cellcolor\[HTML\]DAE8FC84\.33\\cellcolor\[HTML\]DAE8FC84\.35\\cellcolor\[HTML\]FFFFE485\.67\\cellcolor\[HTML\]DAE8FC79\.33\\cellcolor\[HTML\]DAE8FC79\.67\\cellcolor\[HTML\]FFFFE485\.33\\cellcolor\[HTML\]DAE8FC92\.00\\cellcolor\[HTML\]DAE8FC91\.67\\cellcolor\[HTML\]FFFFE492\.00MME1,187\\cellcolor\[HTML\]DAE8FC43\.22\\cellcolor\[HTML\]DAE8FC43\.23\\cellcolor\[HTML\]FFFFE451\.22\\cellcolor\[HTML\]DAE8FC88\.21\\cellcolor\[HTML\]DAE8FC88\.29\\cellcolor\[HTML\]FFFFE489\.13\\cellcolor\[HTML\]DAE8FC84\.08\\cellcolor\[HTML\]DAE8FC84\.33\\cellcolor\[HTML\]FFFFE485\.15AI2D3,088\\cellcolor\[HTML\]DAE8FC78\.59\\cellcolor\[HTML\]DAE8FC78\.61\\cellcolor\[HTML\]FFFFE479\.05\\cellcolor\[HTML\]DAE8FC50\.87\\cellcolor\[HTML\]DAE8FC50\.55\\cellcolor\[HTML\]FFFFE453\.04\\cellcolor\[HTML\]DAE8FC33\.55\\cellcolor\[HTML\]DAE8FC33\.55\\cellcolor\[HTML\]FFFFE445\.53RealworldQA765\\cellcolor\[HTML\]DAE8FC72\.55\\cellcolor\[HTML\]DAE8FC72\.51\\cellcolor\[HTML\]FFFFE473\.20\\cellcolor\[HTML\]DAE8FC66\.41\\cellcolor\[HTML\]DAE8FC66\.41\\cellcolor\[HTML\]FFFFE466\.27\\cellcolor\[HTML\]DAE8FC69\.02\\cellcolor\[HTML\]DAE8FC70\.20\\cellcolor\[HTML\]FFFFE469\.80MMMU541\\cellcolor\[HTML\]DAE8FC26\.80\\cellcolor\[HTML\]DAE8FC26\.81\\cellcolor\[HTML\]FFFFE427\.54\\cellcolor\[HTML\]DAE8FC26\.25\\cellcolor\[HTML\]DAE8FC26\.62\\cellcolor\[HTML\]FFFFE428\.10\\cellcolor\[HTML\]DAE8FC34\.01\\cellcolor\[HTML\]DAE8FC34\.20\\cellcolor\[HTML\]FFFFE435\.49Avg\.\\cellcolor\[HTML\]DAE8FC61\.10\\cellcolor\[HTML\]DAE8FC61\.10\\cellcolor\[HTML\]FFFFE463\.34\\cellcolor\[HTML\]DAE8FC62\.21\\cellcolor\[HTML\]DAE8FC62\.31\\cellcolor\[HTML\]FFFFE464\.37\\cellcolor\[HTML\]DAE8FC62\.53\\cellcolor\[HTML\]DAE8FC62\.79\\cellcolor\[HTML\]FFFFE465\.59Avg\.Δ\\Deltavs base\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.00\\cellcolor\[HTML\]FFFFE4\+2\.24\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.09\\cellcolor\[HTML\]FFFFE4\+2\.16\\cellcolor\[HTML\]DAE8FC—\\cellcolor\[HTML\]DAE8FC\+0\.26\\cellcolor\[HTML\]FFFFE4\+3\.06

## 5Experiments on VLMs

### 5\.1Experimental Setup

Models: We evaluate 3 VLMs: MiMo\-VL\-7B\(Xiaomi,[2025](https://arxiv.org/html/2607.13395#bib.bib47)\), Qwen2\-VL\-7B\(Wanget al\.,[2024](https://arxiv.org/html/2607.13395#bib.bib1)\), Qwen3\-VL\-8B\(Baiet al\.,[2025](https://arxiv.org/html/2607.13395#bib.bib34)\)that were widely used in recent two years\. All models are loaded at BF16 precision with eager attention\. The experiments are conducted on NVIDIA RTX PRO 6000 GPUs \(96 GiB VRAM each\)\. For each evaluation task, a fresh copy of the model is loaded, and the compared method is applied before running the benchmark\.

Benchmarks: We use 5 benchmarks that cover the mainstream multimodal tasks\. MMBench\(Liuet al\.,[2024b](https://arxiv.org/html/2607.13395#bib.bib36)\)\(4,876 natural\-image MCQ\), MME\(Fuet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib7)\)\(2,374 Yes/No perception\), AI2D\(Kembhaviet al\.,[2016](https://arxiv.org/html/2607.13395#bib.bib13)\)\(3,088 diagram MCQ\), RealworldQA\(xAI,[2024](https://arxiv.org/html/2607.13395#bib.bib12)\)\(765 spatial reasoning MCQ\), MMMU \(541 Architecture & Engineering MCQ via HuggingFace datasets\)\(Yueet al\.,[2024](https://arxiv.org/html/2607.13395#bib.bib48)\)\.

Methods\. Similar to LLM experiments, we compare our method with the ZeroTuning \(ZT\)\(Hanet al\.,[2026](https://arxiv.org/html/2607.13395#bib.bib27)\)that uses the original uniform BOS scaling\. ForEnlightenment, we apply a single shared scalar modulatorγ=0\.5\\gamma=0\.5to all residual connections \(x\+γ⋅Fx\+\\gamma\\cdot F\), patching both attention and MLP sublayers on every decoder layer\.

Inference details\. All benchmarks use greedy decoding\. For standard models, we setMax New Tokens=10 for MCQ \(MMBench, AI2D, MMMU\) and Yes/No \(MME\),2020for free\-form RealworldQA\. For MiMo\-VL\-7B—a reasoning model that generates chain\-of\-thought via<think\>tags, we setMax New Tokens=256 for MCQ and6464for free\-form to accommodate deliberation\. Answer extraction for MCQ checks the first option letter appearing in the response for Qwen models or the last option letter after</think\>for the MiMo model\. MME matches response substrings against “yes”/“no” ground truth\.

### 5\.2Main Results

Table[11](https://arxiv.org/html/2607.13395#S4.T11)compares the accuracy of three settings, Base, ZT, andEnlightenment, across five multimodal benchmarks on MiMo\-VL\-7B, Qwen2\-VL\-7B, and Qwen3\-VL\-8B\. Overall,Enlightenmentconsistently achieves the strongest performance, yielding the highest average accuracy across all three models\. Compared with the Base,Enlightenmentimproves average accuracy by \+2\.24% for MiMo\-VL\-7B, \+1\.92% for Qwen2\-VL\-7B, and \+3\.06% for Qwen3\-VL\-8B\. These gains are particularly pronounced on challenging benchmarks, such as MME for MiMo\-VL\-7B, where accuracy increases over 8%, and AI2D for Qwen3\-VL\-8B, where performance rises substantially from 33\.55% to 45\.53%\. In contrast, ZT provides only marginal improvement\. AlthoughEnlightenmentis not the top performer in every individual case, it achieves the best or tied\-best results on most datasets and consistently improves the overall average\. These results demonstrate thatEnlightenmentis a more effective and robust post\-tuning strategy than ZT, delivering clear and stable gains across different VLM architectures and benchmark types\.

Furthermore, we conduct per\-category analysis\. The primary evaluation results demonstrate thatEnlightenmentachieves competitive performance on the MMBench dataset when integrated with the Qwen2\-VL\-7B model\. Since MMBench is a comprehensive benchmark, we conduct a per\-category analysis to characterize the specific strengths and limitations of this method\. Table[12](https://arxiv.org/html/2607.13395#S5.T12)breaks down the MMBench performance gains by L3 category \(spanning 20 categories across the full 1,292\-sample set\)\. Notably,Enlight\-enmentyields the most substantial improvements on perception and recognition tasks: image emotion \(\+16\.67%\+16\.67\\%\), attribute recognition \(\+6\.67%\+6\.67\\%\), and image style \(\+6\.45%\+6\.45\\%\)\. At the L2 aggregation level, “coarse perception” improves by\+6\.08%\+6\.08\\%, while “fine\-grained perception \(instance\-level\)” increases by\+4\.50%\+4\.50\\%\. Conversely, performance declines in relational reasoning categories: nature\_relation \(−11\.29%\-11\.29\\%\), future\_prediction \(−4\.00%\-4\.00\\%\), and the L2 aggregate “relation reasoning” drops by1\.15%1\.15\\%\. This category\-level performance profile explains different benchmark\-level generalization abilities ofEnlightenment: it benefits MMBench because the dataset is dominated by perception tasks, whereas showing a slight degrades performance on RealworldQA due to its inherent focus on relational reasoning\.

Table 12:Per\-category MMBench accuracy deltas with Enlightenment on Qwen2\-VL\-7B \(1292 samples\)\. Top 5 gainers and losers\.CategoryBaselineEnlightenment𝚫\\boldsymbol\{\\Delta\}Top gainers \(L2: coarse perception\+6\.08\+6\.08, finegrained instance\+4\.50\+4\.50\)image\_emotion\\cellcolor\[HTML\]DAE8FC68\.33%\\cellcolor\[HTML\]FFFFE485\.00%\\cellcolor\[HTML\]FFCCC9\+16\.67\+16\.67attribute\_recognition\\cellcolor\[HTML\]DAE8FC88\.33%\\cellcolor\[HTML\]FFFFE495\.00%\\cellcolor\[HTML\]FFCCC9\+6\.67\+6\.67image\_style\\cellcolor\[HTML\]DAE8FC87\.10%\\cellcolor\[HTML\]FFFFE493\.55%\\cellcolor\[HTML\]FFCCC9\+6\.45\+6\.45social\_relation\\cellcolor\[HTML\]DAE8FC85\.48%\\cellcolor\[HTML\]FFFFE491\.94%\\cellcolor\[HTML\]FFCCC9\+6\.45\+6\.45celebrity\_recognition\\cellcolor\[HTML\]DAE8FC92\.93%\\cellcolor\[HTML\]FFFFE498\.99%\\cellcolor\[HTML\]FFCCC9\+6\.06\+6\.06Top losers \(L2: relation reasoning−1\.15\-1\.15\)function\_reasoning\\cellcolor\[HTML\]DAE8FC91\.67%\\cellcolor\[HTML\]FFFFE490\.00%\\cellcolor\[HTML\]FFCCC9−1\.67\-1\.67attribute\_comparison\\cellcolor\[HTML\]DAE8FC78\.43%\\cellcolor\[HTML\]FFFFE476\.47%\\cellcolor\[HTML\]FFCCC9−1\.96\-1\.96future\_prediction\\cellcolor\[HTML\]DAE8FC80\.00%\\cellcolor\[HTML\]FFFFE476\.00%\\cellcolor\[HTML\]FFCCC9−4\.00\-4\.00nature\_relation\\cellcolor\[HTML\]DAE8FC88\.71%\\cellcolor\[HTML\]FFFFE477\.42%\\cellcolor\[HTML\]FFCCC9−11\.29\-11\.29

We also provide qualitative examples comparing Baseline andEnlightenmenton the MMBench dataset\. The model is Qwen2\-VL\-7B\-Instruct \(Qwen2VL architecture, 28 decoder layers, hidden dim 3584, 8\.29B parameters\), and loaded at BF16 precision, greedy decoding, 10 generated tokens\.

Question:Two magnets are placed as shown in the Figure[5](https://arxiv.org/html/2607.13395#S5.F5)\(both sets of arrows point in the same direction inside each magnet, indicating the same pole orientation\)\. Will these magnets attract or repel each other?

![Refer to caption](https://arxiv.org/html/2607.13395v1/figures/sample29.png)Figure 5:Two magnets placed with the same poles facing each other\.Options:

- •A\. Repel\.
- •B\. Attract\.

Correct answer: A \(Repel\.\)

The model output is presented in Table[13](https://arxiv.org/html/2607.13395#S5.T13)\. The baseline model incorrectly predicts “Attract” despite the poles being aligned in the same direction\.Enlightenment’s residual damping suppresses the over\-active sublayer outputs that contribute to this visual reasoning error, allowing the model to correctly identify the physical relationship\. This example is representative of the pattern we observe across the MMBench dataset:Enlightenmentimproves perceptual and physical reasoning accuracy while preserving the model’s overall knowledge\.

Table 13:Baseline vs\. Enlightenment output\.
### 5\.3Sensitivity Analysis

Hyperparameters\.Enlightenmentintroduces only a single hyperparameterη\\eta\. To evaluate its impact, we conduct a sensitivity analysis ofη\\etaon MMBench\-300, as summarized in Figure[6](https://arxiv.org/html/2607.13395#S5.F6)\. Most models exhibit stable performance across different values ofη\\eta\. Qwen2\-VL\-7B is the sole exception, displaying a clear inverted\-U pattern that peaks atη=0\.5\\eta=0\.5with a 6% performance gain\. In contrast, the performance of Qwen2\-VL\-2B, Qwen3\-VL\-8B, and MiMo\-VL\-7B are subjected to slight changes across the intervalη∈\[0\.4,0\.8\]\\eta\\in\[0\.4,0\.8\]\. This highlights thatEnlightenmentis highly robust to hyperparameter selection, making it an ideal plug\-and\-play, zero\-tuning approach to boost the performance of base VLMs\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/figures/sweep20260715T102041.png)Figure 6:Sweepη\\etaon MMBench\-300 across 4 primary models\.η=1\.0\\eta=1\.0is the baseline\.Different layer ranges\. We conduct experiments to analyze the impact of applyingEnlightenmentin different layer ranges\. These results are summarized in Table[14](https://arxiv.org/html/2607.13395#S5.T14)\. First, the MMBench results show that gating any single layer range in isolation degrades performance; early layers are particularly sensitive \(\-12%\), while late layers also experience a decline \(\-7%\)\. Conversely, results on the AI2D dataset reveal a different pattern: gating only the middle layers \(L10–19\) improves accuracy by 4\.9%, suggesting that visual diagrammatic reasoning benefits from mid\-layer residual damping even in isolation\. On the other hand, late\-layer gating on AI2D causes the most severe performance drop among all configurations, indicating that diagram comprehension relies heavily on late\-layer representations\. Uniformly gating all layers yields an improvement of 1\.49% on AI2D, consistent with the full\-benchmark evaluation\. Overall, an all\-layer application is necessary to achieve general improvements across different benchmarks, and targeted layer selection offers a promising avenue for task\-specific fine\-tuning to achieve optimal performance\.

Table 14:Apply Enlightenment to different layer\-range on Qwen2\-VL\-7B withη=0\.5\\eta=0\.5\.Δ\\Deltafrom baseline\.
### 5\.4Ablation Study

Our method usesEnlightenmenton both attention and FFN layers\. We conduct experiments to show their individual contributions\.

Table 15:Attention vs\. FFN gating on Qwen2\-VL\-7B\. Best per benchmark inbold\.Δ\\Deltais from baseline\.Table[15](https://arxiv.org/html/2607.13395#S5.T15)reveals that neither attention\-only nor FFN\-only gating works in isolation\. Scaling only attention degrades MMBench by approximately \-18%, and scaling only FFN degrades MMBench by about \-25%\. Only when both sublayer types are suppressed together does the accuracy improves on MMBench\. This demonstrates that applyingEnlightenmentto attention and FFN residuals forms a coupled system: the scalar modulation’s effectiveness derives from uniform suppression of both sublayer types simultaneously\.

## 6Discussion

### 6\.1Swapping Enlightenment Strategies Between VLM and LLM

In this section, we examine why distinctEnlightenmentstrategies are needed for VLMs and LLMs\. We evaluate both HeadMix and scalar\-modulated shortcuts on two baseline models: Qwen2\-VL\-7B \(a VLM\) and Mimo\-7B\-RL \(an LLM\)\. The results are reported in Table[16](https://arxiv.org/html/2607.13395#S6.T16)\.

For the VLM, HeadMix degrades performance, lowering MMBench accuracy from 79\.3% to 76\.5%\. In contrast, the scalar\-modulated residual achieves 85\.3%, a gain of 6\.0 percentage points over the baseline\. For the LLM, the pattern reverses: scalar\-modulated residuals cause accuracy to drop to 89\.5%, indicating that this mechanism, despite its effectiveness in multimodal contexts, does not benefit text\-only reasoning\. HeadMix raises accuracy to 91\.7%, confirming its alignment with the structural characteristics of LLMs\.

Table 16:HeadMix vs\. Scalar\-Modulated Shortcuts on VLM and LLM\.Taken together, these findings reveal a clear cross\-architec\-ture asymmetry: the twoEnlightenmentstrategies exhibit pronouncedmodel\-type specificity\. HeadMix introduces new scaling and additive computation within the attention module, which proves beneficial for LLMs yet disrupts the inherently more complex multimodal attention patterns that jointly process text and visual tokens, thereby degrading VLM performance\. Conversely, scalar\-modulated residuals regulate feature flow through a lightweight gating mechanism that selectively attenuates extraneous signals entering the transformer decoder, reducing cross\-modal noise and yielding stable, substantial gains for VLMs\. This reciprocal behavior underscores the necessity of tailoringEnlighten\-mentstrategies to each model family\.

### 6\.2Output Distributions

We use the Qwen3\.5\-9B\-Base model with the prompt“Classify the sentence into one of the following categories: positive or negative\. Sentence: A breathtaking masterpiece of modern cinema\. Answer: …”\. A forward hook was registered on the output self\-attention module of the 15thlayer \(one of the target attention layers\) to capture the post\-softmax attention weight tensor of the shape\[1,16,28,28\]\[1,16,28,28\]\(batch, head groups, query length, key length\)\. 16 head groups correspond to the KV\-head dimension after GQA expansion in Qwen3\.5 \(32 query heads, 8 KV heads, with the query heads grouped into 16 tracked slices by the patched attention function\)\. The “BOS attention” for each head group is computed as1Q​∑q=0Q−1𝐀h,q,0\\frac\{1\}\{Q\}\\sum\_\{q=0\}^\{Q\-1\}\\mathbf\{A\}\_\{h,q,0\},i\.e\., the mean attention weight that each query position assigns to the BOS key position \(k=0k=0\), averaged over allQ=28Q=28query positions\.

Specifically, we note that generic scaling rate assigns0\.68,0\.68,0\.66,0\.86,0\.510\.68,0\.68,0\.66,0\.86,0\.51to the 3rd, 7th, 15th, 23th, and 31stlayers\. All five rates are below1\.01\.0, meaning that suppressing the initial attention at target layers yields better accuracy on tasks\. This is consistent with the “attention sink” phenomenon, where pretrained transformers allocate excessive attention to the initial token, and reducing this baseline improves task\-specific focus\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x5.png)Figure 7:The BOS attention by the head at the 15thlayer\. Enlightenment suppresses BOS attention only at heads 0 and 8\.Moreover, Figure[7](https://arxiv.org/html/2607.13395#S6.F7)illustrates the effect ofEnlighten\-menton BOS attention across heads in a targeted layer\. Unlike previous methods that globally reduce initial attention values,Enlightenmentselectively decreases BOS attention only on heads 0 and 8, while other heads remain slightly decreased or nearly unchanged\. This phenomenon indicates that the injection of new shortcuts operates through a mechanism distinct from attention\-based modification\. Our method can dynamically control attention across heads rather than globally suppressing BOS attention\.

### 6\.3Activation Analysis

To directly verify the over\-active sublayer hypothesis in VLM, we compare the hidden state norms of all 28 decoder layers on the same input\. First, Figure[8](https://arxiv.org/html/2607.13395#S6.F8)present the L2 Norm of decoder layers and actication reduction byEnlightenment\. We have the following findings\. \(1\) Baseline activations explode in late layers\. The L2 norm grows from 47 at layer 0 to 371 at layer 27, which is a nearly 8×\\timesincrease\. This suggests that sublayer outputs in deep layers accumulate redundant information rather than refining the residual stream\. \(2\)Enlightenmentsuppresses late layers proportionally more\. Layers 0–2 see only55–10%10\\%reduction, while layers 20–27 see4040–47%47\\%reduction, consistent with the baseline trend that over\-activation concentrates at depth\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x6.png)Figure 8:Comparison of the baseline and Enlightenment VLM decoder activation outputs\.Moreover, we present the VLM decoder activation outputs correspondence to token position in Figure[9](https://arxiv.org/html/2607.13395#S6.F9)\. The results confirm the “attention sink” phenomenon in the baseline model, where the initial token dominates the attention values\. In contrast, our scalar\-modulated method successfully suppresses this effect, substantially reducing the initial token’s outsized activation\. Finally, the activation profile directly supports the mechanism: pretrained sublayer outputs become increasingly noisy at depth, and a singleη\\etasuppresses that excess while preserving the residual path\.

![Refer to caption](https://arxiv.org/html/2607.13395v1/x7.png)Figure 9:Comparison of the baseline and Enlightenment VLM decoder activation outputs correspondence to token position\. Note that, for the baseline, we visualize only the preceding 100\+ token positions because most attention values are concentrated at the initial token position\.

## 7Conclusion

In this paper, we have introducedEnlightenment, a novel, training\-free finetuning paradigm designed to facilitate zero\-shot self\-improvement in large\-scale models\. Drawing inspiration from the neurological phenomenon of enlightenment during resting states, our method modifies shortcuts within pre\-trained transformers without requiring any data, gradient computation, or weight update\.Enlightenmenteffectively regularizes over\-active sublayer outputs and optimizes token\-level attention allocation\. Extensive empirical evaluations across multiple VLMs and LLMs demonstrate thatEnlightenmentconsistently elevates baseline performance across a wide array of multimodal, perception, and reasoning benchmarks\. Moreover, the method exhibits a markedly low sensitivity for hyperparameters\. Our work confirms the promising trajectory of “NeuroAI” principles\. Future work will be digging deeper into the brain’s mechanisms to boost the self\-improvement of large models\.

## References

- N\. Arikha \(2006\)Form and function in the early enlightenment\.Perspectives on Science14\(2\),pp\. 153–188\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p4.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p1.1)\.
- F\. Barbero, A\. Arroyo, X\. Gu, C\. Perivolaropoulos, M\. Bronstein, P\. Veličković, and R\. Pascanu \(2025\)Why do llms attend to the first token?\.arXiv preprint arXiv:2504\.02732\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p9.4)\.
- L\. Billard and E\. Diday \(2002\)Symbolic regression analysis\.InClassification, clustering, and data analysis: recent advances and applications,pp\. 281–288\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p1.1)\.
- Y\. Bisk, R\. Zellers, J\. Gao, and Y\. Choi \(2020\)Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- C\. Chen, J\. Zhu, X\. Luo, H\. T\. Shen, J\. Song, and L\. Gao \(2024a\)Coin: a benchmark of continual instruction tuning for multimodel large language models\.Advances in neural information processing systems37,pp\. 57817–57840\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p1.1)\.
- H\. Chen, Z\. Sun, H\. Ye, K\. Li, and X\. Lin \(2026\)Continual learning in large language models: methods, challenges, and opportunities\.arXiv preprint arXiv:2603\.12658\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p1.1)\.
- W\. J\. Chen and I\. Krajbich \(2017\)Computational modeling of epiphany learning\.Proceedings of the National Academy of Sciences114\(18\),pp\. 4637–4642\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p5.1)\.
- Z\. Chen, Y\. Deng, H\. Yuan, K\. Ji, and Q\. Gu \(2024b\)Self\-play fine\-tuning converts weak language models to strong language models\.InInternational Conference on Machine Learning,pp\. 6621–6642\.Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p1.1)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2924–2936\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1300),[Link](https://aclanthology.org/N19-1300)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- M\. de Marneffe, M\. Simons, and J\. Tonhauser \(2019\)The commitmentbank: investigating projection in naturally occurring discourse\.InProceedings of Sinn und Bedeutung 23,Vol\.2,pp\. 107–124\.Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p1.1)\.
- F\. Fan, Y\. Li, T\. Zeng, F\. Wang, and H\. Peng \(2025\)Towards neuroai: introducing neuronal diversity into artificial neural networks\.Med\-X3\(1\),pp\. 2\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p3.1)\.
- F\. Fan, Z\. Li, H\. Xiong, and T\. Zeng \(2026\)On expressivity of height in neural networks\.IEEE Transactions on Artificial Intelligence7\(3\),pp\. 1223–1237\.External Links:[Document](https://dx.doi.org/10.1109/TAI.2025.3592162)Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p6.1)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun,et al\.\(2026\)Mme: a comprehensive evaluation benchmark for multimodal large language models\.Advances in Neural Information Processing Systems38\.Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p2.1)\.
- J\. Gou, X\. Xiong, B\. Yu, L\. Du, Y\. Zhan, and D\. Tao \(2023\)Multi\-target knowledge distillation via student self\-reflection\.International Journal of Computer Vision131\(7\),pp\. 1857–1874\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1)\.
- X\. Gu, T\. Pang, C\. Du, Q\. Liu, F\. Zhang, C\. Du, Y\. Wang, and M\. Lin \(2025\)When attention sink emerges in language models: an empirical view\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 97114–97144\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p9.4)\.
- S\. Gururangan, A\. Marasović, S\. Swayamdipta, K\. Lo, I\. Beltagy, D\. Downey, and N\. A\. Smith \(2020\)Don’t stop pretraining: adapt language models to domains and tasks\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 8342–8360\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- F\. Han, X\. Yu, J\. Tang, D\. Rao, W\. Du, and L\. Ungar \(2026\)ZeroTuning: unlocking the initial token’s power to enhance large language models without training\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EdkQ14FiiO)Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1),[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p9.4),[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p4.1),[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p3.2)\.
- S\. Hao, T\. Liu, Z\. Wang, and Z\. Hu \(2023\)Toolkengpt: augmenting frozen language models with massive tools via tool embeddings\.Advances in neural information processing systems36,pp\. 45870–45894\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- J\. He, H\. Guo, K\. Zhu, M\. Tang, and J\. Wang \(2026\)Continual instruction tuning for large multimodal models\.IEEE Transactions on Image Processing\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p1.1)\.
- J\. Huang, S\. Gu, L\. Hou, Y\. Wu, X\. Wang, H\. Yu, and J\. Han \(2023\)Large language models can self\-improve\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 1051–1068\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p1.1)\.
- A\. Islam, A\. Bouzerdoum, and S\. B\. Belhaouari \(2026\)Bio\-inspired adaptive neurons for dynamic weighting in artificial neural networks\.AI Open\.Cited by:[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p11.7)\.
- K\. Jiang, R\. Zhu, S\. Huang, H\. Zhang, and X\. Li \(2026a\)Latent noise mask for reducing visual redundancy in multimodal large language models\.arXiv preprint arXiv:2606\.30168\.Cited by:[§3](https://arxiv.org/html/2607.13395#S3.p3.1)\.
- S\. Jiang, Y\. Fang, H\. Zhang, Q\. Liu, Y\. Qi, Y\. Yang, and P\. Wang \(2026b\)Teacher agent: a knowledge distillation\-free framework for rehearsal\-based video incremental learning\.International Journal of Computer Vision134\(4\),pp\. 190\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1)\.
- P\. Kaul, C\. Ma, I\. Elezi, and J\. Deng \(2025\)From attention to activation: unraveling the enigmas of large language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 34531–34582\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p9.4)\.
- A\. Kembhavi, M\. Salvato, E\. Kolve, M\. J\. Seo, H\. Hajishirzi, and A\. Farhadi \(2016\)A diagram is worth a dozen images\.InEuropean Conference on Computer Vision \(ECCV\),Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p2.1)\.
- J\. Kim, S\. Lee, D\. Kwak, M\. Heo, J\. Kim, J\. Ha, and B\. Zhang \(2016\)Multimodal residual learning for visual qa\.Advances in neural information processing systems29\.Cited by:[§3](https://arxiv.org/html/2607.13395#S3.p3.1)\.
- V\. A\. Korthikanti, J\. Casper, S\. Lym, L\. McAfee, M\. Andersch, M\. Shoeybi, and B\. Catanzaro \(2023\)Reducing activation recomputation in large transformer models\.Proceedings of Machine Learning and Systems5,pp\. 341–353\.Cited by:[§3\.2](https://arxiv.org/html/2607.13395#S3.SS2.p1.1)\.
- J\. Liang, R\. He, and T\. Tan \(2025\)A comprehensive survey on test\-time adaptation under distribution shifts\.International Journal of Computer Vision133\(1\),pp\. 31–64\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p1.1)\.
- Y\. Liang and K\. Chen \(2025\)Automatic test\-time adaptation for heterogeneous contexts in meta\-learning\.Neural Computing and Applications37\(18\),pp\. 12631–12652\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p2.1)\.
- S\. Liu, K\. Zheng, and W\. Chen \(2024a\)Paying more attention to image: a training\-free method for alleviating hallucination in lvlms\.InEuropean Conference on Computer Vision,pp\. 125–140\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024b\)Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p2.1)\.
- J\. Luo, Z\. Ni, W\. Liu, X\. Zhu, and Y\. Feng \(2026\)DPC\-tta: dual perturbation and correction network guided test\-time adaptation\.Knowledge\-Based Systems,pp\. 115570\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p2.1)\.
- D\. Monsivais, K\. Bhattacharya, A\. Ghosh, R\. I\. Dunbar, and K\. Kaski \(2017\)Seasonal and geographical impact on human resting periods\.Scientific reports7\(1\),pp\. 10717\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p4.1)\.
- A\. Nichol, J\. Achiam, and J\. Schulman \(2018\)On first\-order meta\-learning algorithms\.arXiv preprint arXiv:1803\.02999\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p2.1)\.
- K\. Ning, H\. Ke, Y\. Liu, J\. Yao, Y\. Tian, and L\. Yuan \(2026\)Sparse orthogonal parameters tuning for continual learning\.International Journal of Computer Vision134\(6\),pp\. 311\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1)\.
- M\. Noori, D\. Osowiechi, G\. Vargas Hakim, A\. Bahri, M\. Yazdanpanah, S\. Dastani, F\. Beizaee, I\. Ayed, and C\. Desrosiers \(2026\)Test\-time adaptation of vision\-language models for open\-vocabulary semantic segmentation\.Advances in Neural Information Processing Systems38,pp\. 74741–74778\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p1.1)\.
- U\. Park, S\. Jeong, Y\. Jang, G\. Park, and J\. H\. Ko \(2025\)Test\-time fine\-tuning of image compression models for multi\-task adaptability\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 4430–4440\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p1.1)\.
- B\. Peng, Z\. Tian, S\. Liu, M\. Yang, and J\. Jia \(2024\)Scalable language model with generalized continual learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 28412–28434\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)Toolllm: facilitating large language models to master 16000\+ real\-world apis\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 9695–9717\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- Y\. Qu, T\. Zhang, N\. Garg, and A\. Kumar \(2024\)Recursive introspection: teaching language model agents how to self\-improve\.Advances in Neural Information Processing Systems37,pp\. 55249–55285\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p1.1)\.
- Qwen Team \(2026\)Qwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p1.1)\.
- \[45\]A\. Razdaibiedina, Y\. Mao, R\. Hou, M\. Khabsa, M\. Lewis, and A\. AlmahairiProgressive prompts: continual learning for language models\.InThe Eleventh International Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- S\. Sadeh and C\. Clopath \(2025\)The emergence of neuroai: bridging neuroscience and artificial intelligence\.Nature Reviews Neuroscience26\(10\),pp\. 583–584\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p3.1)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.arXiv preprint arXiv:2601\.19897\.Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p1.1)\.
- R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Ng, and C\. Potts \(2013\)Recursive deep models for semantic compositionality over a sentiment treebank\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,Seattle, Washington, USA,pp\. 1631–1642\.External Links:[Link](https://www.aclweb.org/anthology/D13-1170)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- C\. Song, X\. Han, Z\. Zeng, K\. Li, C\. Chen, Z\. Liu, M\. Sun, and T\. Yang \(2023\)Conpet: continual parameter\-efficient tuning for large language models\.arXiv preprint arXiv:2309\.14763\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p2.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p3.1)\.
- C\. Thrampoulidis \(2024\)Implicit optimization bias of next\-token prediction in linear models\.Advances in Neural Information Processing Systems37,pp\. 22624–22656\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p1.1)\.
- S\. Udrescu and M\. Tegmark \(2020\)AI feynman: a physics\-inspired method for symbolic regression\.Science advances6\(16\),pp\. eaay2631\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p2.3)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, Y\. Fan, K\. Dang, M\. Du, X\. Ren, R\. Men, D\. Liu, C\. Zhou, J\. Zhou, and J\. Lin \(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p1.1)\.
- Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. Hajishirzi \(2023\)Self\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p1.1)\.
- T\. Wu, W\. Yuan, O\. Golovneva, J\. Xu, Y\. Tian, J\. Jiao, J\. E\. Weston, and S\. Sukhbaatar \(2025\)Meta\-rewarding language models: self\-improving alignment with llm\-as\-a\-meta\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 11548–11565\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p1.1)\.
- xAI \(2024\)RealWorldQA: a benchmark for real\-world spatial understanding\.Note:[https://huggingface\.co/datasets/xai\-org/RealworldQA](https://huggingface.co/datasets/xai-org/RealworldQA)Accessed: 2025\-04\-26Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p2.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21875–21895\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p2.1),[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2607.13395#S3.SS1.p9.4),[§3](https://arxiv.org/html/2607.13395#S3.p3.1)\.
- L\. Xiaomi \(2025\)MiMo: unlocking the reasoning potential of language model – from pretraining to posttraining\.External Links:2505\.07608,[Link](https://arxiv.org/abs/2505.07608)Cited by:[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p1.1)\.
- K\. Xu, L\. Wang, H\. Zhang, and B\. Yin \(2026a\)Self\-Knowledge Distillation and Its Application: A Survey\.IEEE Transactions on Knowledge & Data Engineering38\(07\),pp\. 4674–4693\.External Links:ISSN 1558\-2191,[Document](https://dx.doi.org/10.1109/TKDE.2026.3675811)Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p1.1)\.
- L\. Xu, H\. Xie, S\. J\. Qin, X\. Tao, and F\. L\. Wang \(2026b\)Parameter\-efficient fine\-tuning methods for pretrained language models: a critical review and assessment\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§2\.3](https://arxiv.org/html/2607.13395#S2.SS3.p1.1)\.
- C\. You, K\. Wu, Z\. Jia, L\. Chen, S\. Bhojanapalli, J\. Guo, U\. Evci, J\. Wassenberg, P\. Netrapalli, J\. Willcock,et al\.\(2026\)Spark transformer: reactivating sparsity in transformer ffn and attention\.Advances in Neural Information Processing Systems38,pp\. 23706–23741\.Cited by:[§3\.2](https://arxiv.org/html/2607.13395#S3.SS2.p1.1)\.
- Z\. Yu, Z\. Wang, Y\. Fu, H\. Shi, K\. Shaikh, and Y\. C\. Lin \(2024\)Unveiling and harnessing hidden attention sinks: enhancing large language models without training through attention calibration\.arXiv preprint arXiv:2406\.15765\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p4.1)\.
- X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 9556–9567\.Cited by:[§5\.1](https://arxiv.org/html/2607.13395#S5.SS1.p2.1)\.
- A\. Zador, S\. Escola, B\. Richards, Y\. Bengio, K\. Boahen, M\. Botvinick, D\. Chklovskii, A\. Churchland, C\. Clopath, J\. DiCarlo,et al\.\(2022\)Toward next\-generation artificial intelligence: catalyzing the neuroai revolution\.arXiv preprint arXiv:2210\.08340\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p3.1)\.
- Q\. Zhang, C\. Singh, L\. Liu, X\. Liu, B\. Yu, J\. Gao, and T\. Zhao \(2024a\)Tell your model where to attend: post\-hoc attention steering for llms\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 42411–42430\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p4.1)\.
- Q\. Zhang, X\. Yu, C\. Singh, X\. Liu, L\. Liu, J\. Gao, T\. Zhao, D\. Roth, and H\. Cheng \(2024b\)Model tells itself where to attend: faithfulness meets automatic attention steering\.arXiv preprint arXiv:2409\.10790\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1),[§4\.1](https://arxiv.org/html/2607.13395#S4.SS1.p4.1)\.
- R\. Zhang, R\. H\. Bai, H\. Zheng, N\. Jaitly, R\. Collobert, and Y\. Zhang \(2026\)Embarrassingly simple self\-distillation improves code generation\.arXiv preprint arXiv:2604\.01193\.Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p2.1)\.
- S\. Zhang, J\. Chen, J\. Wu, G\. Zhang, H\. Xiong, B\. Gu, and Z\. Zhou \(2024c\)On the intrinsic structures of spiking neural networks\.Journal of Machine Learning Research25\(194\),pp\. 1–74\.Cited by:[§1](https://arxiv.org/html/2607.13395#S1.p6.1)\.
- S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. Grover \(2026\)Self\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§2\.2](https://arxiv.org/html/2607.13395#S2.SS2.p2.1)\.
- D\. Zhou, H\. Sun, J\. Ning, H\. Ye, and D\. Zhan \(2024\)Continual learning with pre\-trained models: a survey\.InProceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence,pp\. 8363–8371\.Cited by:[§2\.1](https://arxiv.org/html/2607.13395#S2.SS1.p1.1)\.
- L\. Zhu, D\. Ji, T\. Chen, P\. Xu, J\. Ye, and J\. Liu \(2025\)Ibd: alleviating hallucinations in large vision\-language models via image\-biased decoding\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 1624–1633\.Cited by:[§2\.4](https://arxiv.org/html/2607.13395#S2.SS4.p1.1)\.