Hybrid Gated Attention

# Hybrid Gated Attention
Source: [https://arxiv.org/html/2608.11805](https://arxiv.org/html/2608.11805)
###### Abstract

Gated attention is an effective approach to mitigate attention sinks and enhance the representational capacity of attention\. To further extend its effectiveness\-efficiency Pareto frontier, we propose a Hybrid Gated Attention \(HyGA\) framework that contains three types of gating strategies\. Specifically, these gates leverage diverse information from multiple stages of attention, and collaboratively build element\-wise/head\-wise gating from multiple perspectives, capturing intra\-head and cross\-head information interactions\. Through our hybrid gating components, HyGA could provide multi\-source modulation signals, enabling more comprehensive control over information flow and improving the representational capacity of attention\. We also introduce low\-rank matrix decomposition and learnable attention sink to further enhance training efficiency and stability\. In experiments, we evaluate HyGA on widely\-used benchmarks based on different backbones\. The experimental results show that our HyGA comprehensively improves both training loss and various downstream performances compared with Gated attention\. HyGA has also been verified to achieve the best performance at different computation costs, with comprehensive model analyses for better understanding\. The proposed HyGA sheds light on a more effective, efficient, and stable attention mechanism\.

## 1Introduction

Multi\-head attention\([37](https://arxiv.org/html/2608.11805#bib.bib1)\)has become a core component of LLMs for contextual information interaction\. Consequently, the information\-processing capability of attention directly affects an LLM’s contextual understanding and information integration ability, and thus plays a critical role in determining the model’s representational capacity\.

In recent years, a growing body of work has sought to improve the attention mechanism\. One major research direction focuses on enhancing the efficiency of attention and reducing the KV\-cache overhead, leading to representative approaches such as linear attention\([42](https://arxiv.org/html/2608.11805#bib.bib16);[41](https://arxiv.org/html/2608.11805#bib.bib17)\), SSMs\([10](https://arxiv.org/html/2608.11805#bib.bib39)\), sparse attention\([27](https://arxiv.org/html/2608.11805#bib.bib42);[39](https://arxiv.org/html/2608.11805#bib.bib43);[43](https://arxiv.org/html/2608.11805#bib.bib44)\), and KV\-sharing or compression mechanisms such as GQA\([2](https://arxiv.org/html/2608.11805#bib.bib2)\)and MLA\([26](https://arxiv.org/html/2608.11805#bib.bib3)\)\.

Another important direction aims to address the performance and stability issues associated with attention, among which the attention\-sink phenomenon has been extensively studied\. Attention sink\([38](https://arxiv.org/html/2608.11805#bib.bib5);[15](https://arxiv.org/html/2608.11805#bib.bib35);[4](https://arxiv.org/html/2608.11805#bib.bib36);[35](https://arxiv.org/html/2608.11805#bib.bib34);[36](https://arxiv.org/html/2608.11805#bib.bib45)\)refers to the phenomenon in which an attention module assigns a disproportionately large amount of attention mass to a small number of tokens with limited semantic relevance, such as the BOS token\. These tokens act as attention sinks by absorbing excess attention, effectively allowing certain attention heads to approximate a no\-op operation\. However, excessively concentrated attention may reduce the effective utilization of contextual information, which should be considered\.

Figure 1:We propose HyGA with three gates, i\.e\., X\-gate, H\-gate, and C\-gate, to enhance the attention’s gating strategy with more informative inputs and diverse control\. It further incorporates low\-rank matrix decomposition and learnable attention sinks for more efficient and stable training\.Gated attention is a representative approach for mitigating attention sinks and improving attention ability\([31](https://arxiv.org/html/2608.11805#bib.bib4)\)\. It applies element\-wise gating to Scaled Dot\-Product Attention \(SDPA\) output, introducing non\-linearity and sparsity to alleviate attention sinks and enhance model expressiveness\. Nevertheless, several limitations still remain: \(a\) the original gated attention relies on the raw input X for gating, ignoring other information that is a good supplement to the current gating strategy\. \(b\) Element\-wise gating introduces considerable additional parameters and computational overhead, leaving substantial room for improving gating efficiency\. And \(c\) existing gated attention still exhibits a non\-negligible BOS\-token sink ratio, motivating us to further mitigate the attention\-sink phenomenon for stable training\.

Motivated by the above observations, we substantially extend gated attention and propose ourHybrid Gated Attention \(HyGA\)\. Specifically, our HyGA contains three well\-coordinated attention gates to smartly adjust the output volume of SDPA from diverse aspects\. Besides the classical attention gate controlled by the input X \(*X\-gate*\), we first introduce an element\-wise attention gate that takes the SDPA outputsHiH\_\{i\}as the head\-specific gate controller, noted as the*H\-gate*, which contains richer contextual information after attention and thereby provides additional information gains\. Since the two gates are conditioned on representations from different stages of attention, they can provide complementary modulation signals, enabling more comprehensive control over information flow and consequently improving the representational capacity of attention\. To achieve a favorable effectiveness–efficiency Pareto frontier, we apply low\-rank matrix factorization to the projections of both gates\. In this case, HyGA could achieve comparable training loss and downstream performance with substantially fewer attention parameters than classical Gated attention\.

Besides the H\-gate that focus on the intra\-head interactions to generate element\-wise gating scores, we further design the cross\-head gate \(*C\-gate*\) to capture inter\-head correlations\. It jointly adopts all heads’ SDPA outputs to calculate head\-wise gating scores, which functions as a good supplement to the above fine\-grained element\-wise gating\. All three gates cooperate well with each other, enriching the attention gating strategy with more comprehensive considerations on diverse features\. To prevent potential training instability and output vanishing caused by the stacked multiplication of three gates, we adopt a gate fusion strategy\. We also find that HyGA could be more stable armed with learnable attention sink\([1](https://arxiv.org/html/2608.11805#bib.bib8)\), which seems to have potential functional overlap with gated attention\.

In experiments, we evaluate HyGA across different models on widely\-used benchmarks, where the results demonstrate that HyGA consistently outperforms the original Gated attention\. Through appropriate low\-rank compression settings, HyGA could achieve slightly better performance with only26%26\\%of Gated attention’s gating computation cost\. The contributions are summarized as follows:

- •We propose HyGA, which jointly adopts three hybrid gating strategies to provide both head\-wise and element\-wise gate scoring calculated from different factors\. Equipped with learnable sink, HyGA achieves more stable training\.
- •We explore different low\-rank matrix factorization settings in our hybrid gates to extend the effectiveness\-efficiency Pareto frontier of Gated attention methods\.
- •Overall, HyGA achieves significant improvement compared to baselines with different backbones and model settings, shedding light on more effective, efficient, and stable gated attention modules in practice\.

## 2Preliminary

In this section, we first give a brief introduction and formulation on classical attention mechanisms used in this work\.

Grouped\-Query Attention \(GQA\)\.GQA is widely\-used in popular LLMs\. Conventional multi\-head attention mechanisms adopt query \(Q\), key \(K\), and value \(V\) to compute contextualized token representations, where each query head attends to its corresponding key\-value head independently\. GQA allows the queries from multiple attention heads to share a common set of key\-value projections\. Let𝑯i\\bm\{H\}\_\{i\}denote the attention output of the i\-th head, the formulas are:

𝑯i=Softmax⁡\(𝑸i​𝑲g⁡\(i\)⊤dh\)​𝑽g⁡\(i\),\\bm\{H\}\_\{i\}=\\operatorname\{Softmax\}\\left\(\\frac\{\\bm\{Q\}\_\{i\}\\bm\{K\}\_\{g\(i\)\}^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\\right\)\\bm\{V\}\_\{g\(i\)\},\(1\)whereg⁡\(i\)g\(i\)maps the i\-th query head to its corresponding key\-value group\. Query heads within the same group share the same key and value representations, thereby substantially reducing the memory footprint of the KV cache while preserving most of the model’s inference capability\.

Multi\-latent Attention \(MLA\)\.MLA improves GQA by compressing𝑲\\bm\{K\}and𝑽\\bm\{V\}into a low\-dimensional latent space through down\-projection\. Only the low\-dimensional latent representation is cached in the KV cache, and the complete𝑲\\bm\{K\}and𝑽\\bm\{V\}matrices are up\-projected during attention computation\. The specific formulas are as follows:𝒄tK​V=𝑾D​K​V​𝒉t,𝒌tC=𝑾U​K​𝒄tK​V,𝒗tC=𝑾U​V​𝒄tK​V\\bm\{c\}\_\{t\}^\{KV\}=\\bm\{W\}^\{DKV\}\\bm\{h\}\_\{t\},\\bm\{k\}\_\{t\}^\{C\}=\\bm\{W\}^\{UK\}\\bm\{c\}\_\{t\}^\{KV\},\\bm\{v\}\_\{t\}^\{C\}=\\bm\{W\}^\{UV\}\\bm\{c\}\_\{t\}^\{KV\}, where𝑾D​K​V\\bm\{W\}^\{DKV\}is the down\-projection matrix and𝒄tK​V\\bm\{c\}\_\{t\}^\{KV\}is the compressed latent vector shared by keys and values\. Compared with GQA, MLA achieves stronger KV cache compression and can save more memory in long\-context inference\.

Gated Attention\.Gated Attention\([31](https://arxiv.org/html/2608.11805#bib.bib4)\)refers to applying an element\-wise gating operation to the output matrix after the original per\-head attention\. The typical gated attention function is adopted after SDPA output:

𝑯i′=𝑯i⊙σ⁡\(𝑿​𝑾i\),𝑶=𝑯′​𝑾O,𝑾i∈ℝdmodel×d\.\\bm\{H\}^\{\\prime\}\_\{i\}=\\bm\{H\}\_\{i\}\\odot\\sigma\(\\bm\{X\}\\bm\{W\}\_\{i\}\),\\;\\bm\{O\}=\\bm\{H\}^\{\\prime\}\\bm\{W\}\_\{O\},\\;\\bm\{W\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\}\.\(2\)whereddis the head dimension,dmodeld\_\{\\mathrm\{model\}\}is the model dimension,𝑿\\bm\{X\}is the input matrix and𝑯′\\bm\{H\}^\{\\prime\}is the gated output before𝑾O\\bm\{W\}\_\{O\}\.𝑾i\\bm\{W\}\_\{i\}represents the projection matrix for the i\-th head, providing element\-wise gating for d dimensions\. It simultaneously provides nonlinear expressiveness and suppresses attention sink and massive activation, and thus could further improve the performance of both GQA and MLA\.

## 3Method

In this work, we propose Hybrid Gated Attention, which attempts to further optimize the effectiveness, efficiency, and training stability of gated attention\.

Figure 2:Overview of the proposed HyGA, which contains three gates: X\-gate \(𝑮X\\bm\{G\}\_\{X\}\), H\-gate \(𝑮H\\bm\{G\}\_\{H\}\), and C\-gate \(𝑮C\\bm\{G\}\_\{C\}\), armed with low\-rank matrix decomposition \(in H\-gate and X\-gate\) and learnable attention sink \(LS in SDPA\)\.### 3\.1Overall Framework

As shown in Fig\.[2](https://arxiv.org/html/2608.11805#S3.F2), HyGA mainly extends the original Gated attention with three gating mechanisms: \(a\) the original gate that takes the raw input𝑿\\bm\{X\}before the attention layer as input \(X\-gate\), \(b\) the proposed gate that takes the output𝑯\\bm\{H\}after SDPA as input \(H\-gate\), and \(c\) the proposed cross\-head gate \(C\-gate\), which captures the inter\-head connections and provides head\-level reweighting\. We find that these three attention gates could cooperate well and jointly improve the performance\. Next, we adopt a gate fusion strategy to get our final hybrid gated attention output, avoiding undesired dominating gates to ensure smooth training\. Besides, to improve training stability, we implement HyGA with learnable attention sink to provide a double guarantee for training stability\. Based on the above mechanisms, our HyGA effectively improves both model capability and training stability\.

### 3\.2Hybrid Gating Strategy

#### Hybrid Gating Inputs and Functions

The original gated attention adopts the raw input𝑿\\bm\{X\}to control the element\-wise gate as stated in Eq\. \([2](https://arxiv.org/html/2608.11805#S2.E2)\)\. Through an analysis of the original gated attention, we observe that the output𝑯\\bm\{H\}contains richer token\-interaction information after attention, maybe providing additional information gain besides𝑿\\bm\{X\}\. Motivated by this observation, we propose to use the output after SDPA𝑯\\bm\{H\}as the input to control the attention gate as a supplement of𝑿\\bm\{X\}\. We have:

𝑯i′=𝑯i⊙σ⁡\(𝑿​𝑾i\+SiLU​\(𝑯i​𝑾id\)​𝑾iu\)\.\\bm\{H\}\_\{i\}^\{\\prime\}=\\bm\{H\}\_\{i\}\\odot\\sigma\(\\bm\{X\}\\bm\{W\}\_\{i\}\+\\text\{SiLU\}\(\\bm\{H\}\_\{i\}\\bm\{W\}\_\{i\}^\{d\}\)\\bm\{W\}\_\{i\}^\{u\}\)\.\(3\)For the newly added H\-gate, we adopt a 2\-layer MLP form withSiLU\([12](https://arxiv.org/html/2608.11805#bib.bib6)\)to enhance nonlinear modulation\. This is because the above form can more conveniently perform low\-rank parameter compression by adjusting the intermediate dimension, preparing for the efficiency improvements discussed below\. Here,𝑾id∈ℝd×dint\\bm\{W\}\_\{i\}^\{d\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{int\}\}\}and𝑾iu∈ℝdint×d\\bm\{W\}\_\{i\}^\{u\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{int\}\}\\times d\}are the down\-projection and up\-projection gating matrices for the i\-th head, respectively, wheredintd\_\{\\mathrm\{int\}\}denotes the intermediate dimension that controls the bottleneck of the down\-then\-up projection\.

As can be seen from the above formulation, the original X\-gate based on𝑿\\bm\{X\}and the added H\-gate based on𝑯\\bm\{H\}exhibit strong functional complementarity\. The former mainly captures the intrinsic features of input tokens before attention, while the latter mainly captures the features after contextual interaction through attention\. Therefore, the information modeled by the two gates is different to some extent, and the two gates modulate information at different positions\. Their joint effect can thus provide a more comprehensive capture and modeling of the information in attention output, enabling the model to better control the information flow through element\-wise gating\.

#### Fused or separate X\-/H\- gates\.

Different from the fused form in Eq\. \([3](https://arxiv.org/html/2608.11805#S3.E3)\), the hybrid X\-/H\- gates can also be separate:

𝑯i′=𝑯i⊙σ⁡\(𝑿​𝑾i\)⊙σ⁡\(SiLU​\(𝑯i​𝑾id\)​𝑾iu\),\\bm\{H\}\_\{i\}^\{\\prime\}=\\bm\{H\}\_\{i\}\\odot\\sigma\(\\bm\{X\}\\bm\{W\}\_\{i\}\)\\odot\\sigma\(\\text\{SiLU\}\(\\bm\{H\}\_\{i\}\\bm\{W\}\_\{i\}^\{d\}\)\\bm\{W\}\_\{i\}^\{u\}\),\(4\)However, we choose the additive coupling form in Eq\. \([3](https://arxiv.org/html/2608.11805#S3.E3)\) rather than that in Eq\. \([4](https://arxiv.org/html/2608.11805#S3.E4)\)\. This is because the multiplicative coupling form in Eq\. \([4](https://arxiv.org/html/2608.11805#S3.E4)\) multiplies two gates together\. Since the sigmoid activation function has a value range of\(0,1\)\(0,1\), each gate generally acts as a suppressive modulator of the information flow\. Multiplying too many gates may therefore lead to overly strong suppression, which may weaken gradient propagation and limit effective representation learning \(especially with the cross\-head gate in Sec\.[3\.3](https://arxiv.org/html/2608.11805#S3.SS3)\)\. In contrast, the fused gate in Eq\. \([3](https://arxiv.org/html/2608.11805#S3.E3)\) first adds the pre\-activation gate logits and then applies the activation function, allowing different factors to jointly control the information flow rather than to independently modulate it\. This could lead to a more stable information flow and stronger expressive capacity\.

#### Approaching the Effectiveness\-Efficiency Pareto Frontier

While pursuing improved model expressiveness, we also attempt to minimize the computation cost to extend the Pareto frontier between effectiveness and efficiency\. Although the two gate mechanisms described above could cooperate well, there may still exist partial functional overlap between them, enabling them to withstand greater information compression\. Therefore, we can use low\-rank factorization to reduce computation and parameter cost while preserving most of the expressive capacity\.

Specifically, we further decompose the X\-based gate using the same low\-rank matrix factorization adopted for our H\-gate, also with aSiLUactivation function in the middle to provide nonlinearity\. By adjusting the intermediate dimension of𝑾d\\bm\{W\}^\{d\}/𝑾u\\bm\{W\}^\{u\}in the two gates, we can control the maximum ranks of the two projections to balance effectiveness and efficiency\. The formulation is given as follows:

𝑯i′=𝑯i⊙σ⁡\(SiLU​\(𝑿​𝑾¯id\)​𝑾¯iu\+SiLU​\(𝑯i​𝑾id\)​𝑾iu\),\\begin\{split\}\\bm\{H\}\_\{i\}^\{\\prime\}=\\bm\{H\}\_\{i\}\\odot\\sigma\(\\text\{SiLU\}\(\\bm\{X\}\\bm\{\\bar\{W\}\}\_\{i\}^\{d\}\)\\bm\{\\bar\{W\}\}\_\{i\}^\{u\}\+\\text\{SiLU\}\(\\bm\{H\}\_\{i\}\\bm\{W\}\_\{i\}^\{d\}\)\\bm\{W\}\_\{i\}^\{u\}\),\\end\{split\}\(5\)where𝑾id\\bm\{W\}\_\{i\}^\{d\},𝑾iu\\bm\{W\}\_\{i\}^\{u\}and𝑾¯id\\bm\{\\bar\{W\}\}\_\{i\}^\{d\},𝑾¯iu\\bm\{\\bar\{W\}\}\_\{i\}^\{u\}are different groups of weighting matrices\. The maximum rank of each gating matrix can be controlled by adjusting the intermediate dimension\. We find that appropriately reducing the ranks of both groups of matrices could better reduce the number of parameters and computational cost while largely preserving representational capacity, compared to the original gated attention\. The resulting effectiveness\-efficiency Pareto frontier is reported in the detailed experimental results in Section[4\.4](https://arxiv.org/html/2608.11805#S4.SS4)\.

### 3\.3Cross\-Head Gating Strategy

The H\-gate introduced above mainly works together with the original X\-gate to regulate the element\-wise information flow more finely and improve expressiveness\. Although it enables a more fine\-grained, hybrid, and efficient attention gating, the gating strategy still depends on the intra\-head element interactions for each head’s output separately, while inter\-heads mutual interactions are relatively neglected\. In multi\-head attention, different heads do not express information completely independently\. Instead, information from different heads can interact with each other and be expressed collaboratively\. Therefore, we hypothesize that the gating score of one head should also consider the opinions of other heads for reference\. Based on this hypothesis, we further propose the cross\-head gating module \(i\.e\., C\-gate\)\.

Specifically, we have the concatenated attention outputs of all heads𝑯=concat⁡\{𝑯1,𝑯2,…,𝑯h\}\\bm\{H\}=\\mathrm\{concat\}\\\{\\bm\{H\}\_\{1\},\\bm\{H\}\_\{2\},\.\.\.,\\bm\{H\}\_\{h\}\\\}\. We use𝑯\\bm\{H\}as the input and apply a matrix transformation to obtain our head\-wise gating as a supplement, formulated as:

𝑯i′=𝑯i⊙σ⁡\(SiLU​\(𝑿​𝑾¯id\)​𝑾¯iu\+SiLU​\(𝑯i​𝑾id\)​𝑾iu\)⊙σ⁡\(Broadcast⁡\(\(𝑯​𝑾c\)i\)\)\.\\begin\{split\}\\bm\{H\}\_\{i\}^\{\\prime\}=\\bm\{H\}\_\{i\}&\\odot\\sigma\(\\text\{SiLU\}\(\\bm\{X\}\\bm\{\\bar\{W\}\}\_\{i\}^\{d\}\)\\bm\{\\bar\{W\}\}\_\{i\}^\{u\}\+\\text\{SiLU\}\(\\bm\{H\}\_\{i\}\\bm\{W\}\_\{i\}^\{d\}\)\\bm\{W\}\_\{i\}^\{u\}\)\\\\ &\\odot\\sigma\\left\(\\mathrm\{Broadcast\}\(\(\\bm\{H\}\\bm\{W\}\_\{c\}\)\_\{i\}\)\\right\)\.\\end\{split\}\(6\)Here,𝑾c∈ℝh​d×h\\bm\{W\}\_\{c\}\\in\\mathbb\{R\}^\{hd\\times h\}denotes the transformation matrix for computing the cross\-head gate for h heads based on all heads’ elements\. Note that𝑯​𝑾c\\bm\{H\}\\bm\{W\}\_\{c\}is an h\-dimensional vector, andBroadcast⁡\(\(𝑯​𝑾c\)i\)\\mathrm\{Broadcast\}\(\(\\bm\{H\}\\bm\{W\}\_\{c\}\)\_\{i\}\)denotes broadcasting its i\-th dimension to the shape of𝑯i\\bm\{H\}\_\{i\}, indicating that this gate provides one gate score for all elements of one head\. The reasons are: a\) the C\-gate is supplementary to the above element\-wise X\-gate and H\-gate and thus should not bring in much additional computation, and b\) the cross\-head interactions are supposed to provide coarse\-grained inter\-head reweighting\.

The cross\-head gating strategy integrates information across heads and performs head\-wise gating based on the integrated input\. Since it can perceive the global head state during gating, it can dynamically adjust the relative contributions of different heads and encourage collaborative information modeling across heads\. Specifically, this cross\-head mechanism may suppress heads with lower contribution under the current input, emphasize more important heads, and dynamically allocate information flow across heads according to the current context\. In conclusion, our C\-gate mainly regulates coarse\-grained head\-wise information across heads, while the X\-/H\- gates mainly capture fine\-grained element\-wise information within each head from different aspects\. In conclusion, all hybrid gates capture distinct modulation signals and exhibit limited functional overlap\. Our experiments show that the cross\-head gate can achieve further better performance\.

### 3\.4HyGA with Learnable Attention Sink

The above methods improve the expressiveness of the gated attention module through the combination of multiple mechanisms\. However, beyond representational expressiveness, training stability is also a critical concern in industrial\-scale training of large language models\. Attention sinks\([38](https://arxiv.org/html/2608.11805#bib.bib5)\)and massive activations are important factors that affect training stability\. In the original work on Gated attention, the authors also mention that the head\-specific gating mechanism of gated attention can introduce sparsity and mitigate attention sinks and massive activations\.

However, our experimental results show that, when using gated attention alone, its mitigation of the sink ratio is still not perfect, and massive activations occasional occur in production\-scale training\. Therefore, inspired by GPT\-OSS\([1](https://arxiv.org/html/2608.11805#bib.bib8)\), we implement HyGA with learnable attention sinks to provide an additional safeguard for training stability\. Through experiments, we find that adding learnable attention sinks on top of HyGA can further reduce the sink ratio and effectively alleviate massive activations, even with slight loss advantages\. In our observations, the hidden states become significantly smoother, and massive activations are substantially reduced\.

## 4Experiments

In this section, we conduct a series of experiments to verify the effectiveness of our HyGA with three research questions: \(RQ1\) Does HyGA outperform the original gated attention on different backbones and benchmarks \(in Sec\.[4\.2](https://arxiv.org/html/2608.11805#S4.SS2)\)? \(RQ2\) Are all components of HyGA effective \(in Sec\.[4\.3](https://arxiv.org/html/2608.11805#S4.SS3)\)? \(RQ3\) Can HyGA achieve the effectiveness\-efficiency Pareto frontier with stable training \(in Sec\.[4\.4](https://arxiv.org/html/2608.11805#S4.SS4)and[4\.5](https://arxiv.org/html/2608.11805#S4.SS5)\)?

Table 1:Results on 14 widely\-used benchmarks of MoE\-5B \(MLA\) trained on around 500B tokens\.![Refer to caption](https://arxiv.org/html/2608.11805v1/mainresult_mla.png)Figure 3:Training loss trends of the original Gated attention and our HyGA of MoE\-5B trained on 500B token\.![Refer to caption](https://arxiv.org/html/2608.11805v1/ablation_result_1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.11805v1/ablation_result_2.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.11805v1/ablation_result_3.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.11805v1/ablation_result_4.png)\(d\)

Figure 4:Ablation results for different variants of HyGA\. Adding learnable sink, H\-gate, C\-gate and gate fusion sequentially does bring in benefits on 14 benchmarks \(average performance gains: \+0\.24%→\\rightarrow\+1\.21%→\\rightarrow\+1\.53%\)\.### 4\.1Experimental Setups

#### Datasets\.

We evaluate models on 14 popular benchmarks, including CEval\([19](https://arxiv.org/html/2608.11805#bib.bib22)\), CMMLU\([24](https://arxiv.org/html/2608.11805#bib.bib20)\), MMLU\([16](https://arxiv.org/html/2608.11805#bib.bib19)\), AGIEval\([45](https://arxiv.org/html/2608.11805#bib.bib31)\), ARC\([8](https://arxiv.org/html/2608.11805#bib.bib26)\), GPQA\-Diamond\([32](https://arxiv.org/html/2608.11805#bib.bib28)\), GSM8K\([9](https://arxiv.org/html/2608.11805#bib.bib23)\), MATH\([17](https://arxiv.org/html/2608.11805#bib.bib27)\), MBPP\+\([3](https://arxiv.org/html/2608.11805#bib.bib24);[28](https://arxiv.org/html/2608.11805#bib.bib25)\), HellaSwag\([44](https://arxiv.org/html/2608.11805#bib.bib21)\), PIQA\([5](https://arxiv.org/html/2608.11805#bib.bib29)\), SIQA\([33](https://arxiv.org/html/2608.11805#bib.bib33)\), Natural Questions\([23](https://arxiv.org/html/2608.11805#bib.bib30)\), and TriviaQA\([22](https://arxiv.org/html/2608.11805#bib.bib32)\)\.

#### Model Settings and Competitors\.

We mainly conduct experiments on an MoE\([20](https://arxiv.org/html/2608.11805#bib.bib9)\)model with nearly 1B activated parameters and 5B total parameters \(noted as MoE\-5B\)\. The model adopts classical Transformer structure with MLA\([26](https://arxiv.org/html/2608.11805#bib.bib3)\)used in the attention module, incorporating element\-wise gated attention\([31](https://arxiv.org/html/2608.11805#bib.bib4)\)\. It has 64 experts and 4 activated experts, armed with the Muon optimizer\([21](https://arxiv.org/html/2608.11805#bib.bib12)\)and trained on 500B tokens\. Besides, we also adopt another backbone of Qwen3\-0\.6B\([40](https://arxiv.org/html/2608.11805#bib.bib10)\), which is a dense model with GQA\([2](https://arxiv.org/html/2608.11805#bib.bib2)\)\. It is trained with around 200B tokens via AdamW\([29](https://arxiv.org/html/2608.11805#bib.bib11)\)\. We implement gated attention and our HyGA on it for further comparisons on different LLM structures\.

### 4\.2Main Results \(RQ1\)

The experimental results of MoE\-5B are shown in Table[1](https://arxiv.org/html/2608.11805#S4.T1), with the training loss trends of MoE\-5B variants given in Figure[3](https://arxiv.org/html/2608.11805#S4.F3)\. The results of Qwen3\-0\.6B are illustrated in Table[2](https://arxiv.org/html/2608.11805#S4.T2)\. From these results, we can observe that:

\(a\) As evidenced by the evaluation results and loss curves of MoE\-5B, our HyGA achieves consistently improvements over Gated attention across most metrics and training steps, and the overall improvement is significant\. Note that we do not apply low\-rank compression for effectiveness in the main experiments\. For the loss trend in Fig\.[3](https://arxiv.org/html/2608.11805#S4.F3), HyGA consistently maintains a lower loss throughout training, with the gap gradually widening \(the loss advantage is approximately 0\.012 at 60k step\)\. These indicate the effectiveness of HyGA on MoE or MLA structures\.

\(b\) For Qwen3\-0\.6B, its relatively small model size and trained token size make the results on some of the aforementioned fourteen benchmarks less reliable\. We therefore select six relatively reliable benchmarks and report the performance gaps\. We can also find that HyGA achieves the overall better performance compared to gated attention based on GQA and dense model\. Besides, after training on 200B tokens, HyGA attains a training loss approximately 0\.008 lower than that of Gated GQA\. Taken together, these results show that HyGA remains effective when applied to different backbones and model settings, implying the generalization ability of HyGA\.

Table 2:Results on 6 widely\-used benchmarks of Qwen3\-0\.6B \(GQA\) trained on 200B tokens\.
### 4\.3Ablation Study \(RQ2\)

To demonstrate that each module in HyGA makes a distinct contribution and is not fully functionally redundant with others modules, we conduct ablation studies on essential components with the same settings in the main experiment \(i\.e\., MoE\-5B, MLA, 500B trained tokens\)\. The detailed results of ablation versions on 14 benchmarks are in Figure[4](https://arxiv.org/html/2608.11805#S4.F4)\.

#### Effectiveness of H\-gate and Learnable Attention Sink

We first conduct an ablation study on the H\-based gating module: the gated attention baseline versus the gated attention with H\-gate\. The results show that introducing H\-gate yields clear improvements in most benchmarks covering different capabilities, achieving significantly better average performance\. It suggests that the proposed H\-gate could provide additional information fromHiH\_\{i\}that is not captured by the original Gated attention, which merely contains the original raw inputXX\. As discussed in Section[3\.2](https://arxiv.org/html/2608.11805#S3.SS2), compared withXX, the post\-attention outputHiH\_\{i\}contains richer contextual interaction information\. Consequently, H\-gate can provide a different view to control the element\-wise gating, cooperating well with the original X\-gate in gated attention\. Besides, we also find that the cooperation with learnable attention sink could also bring in a slight improvement on the average score\. In\-depth analysis on the stability advantages of learnable sink will be discussed in Sec\.[4\.5](https://arxiv.org/html/2608.11805#S4.SS5)\.

#### Effectiveness of Cross\-head Gate with Gate Fusion

Next, we validate the effectiveness of C\-gate with the help of gate fusion\. We find that adding C\-gate based on X\+H gates further reduces the final training loss by 0\.004\. However, we discover that multiplicatively applying three gates will excessively suppress the output activations, as stated in Sec\.[3\.2](https://arxiv.org/html/2608.11805#S3.SS2)\. Therefore, we include the gate fusion strategy as the final HyGA version, achieving a more numeric\-healthy gating mechanism\. Comparing HyGA with HyGA \(learnable sink\+H\-gate\), we know that HyGA consistently outperforms the H\-gate variant on most downstream benchmarks, and its lower final training loss further supports this result\. These findings suggest that jointly applying our C\-gate and the gate\-fusion mechanism improves HyGA’s performance\.

### 4\.4Balancing Effectiveness and Efficiency \(RQ3\)

In this subsection, we discuss the balance between effectiveness and efficiency of HyGA, which could be flexibly adjusted by the low\-rank matrix decomposition technique introduced in Sec\.[3\.2](https://arxiv.org/html/2608.11805#S3.SS2)\. Specifically, we implement HyGA with different intermediate dimension sizesdintd\_\{\\mathrm\{int\}\}of X\-gate and H\-gate \(i\.e\., controlling the output dimensions of𝑾id,𝑾¯id∈ℝd×dint\\bm\{W\}^\{d\}\_\{i\},\\bm\{\\bar\{W\}\}^\{d\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{int\}\}\}, so as to constrain the maximum rank of the projection of both gating\)\.

Taking the H\-gateσ⁡\(SiLU​\(𝑯i​𝑾id\)​𝑾iu\)\\sigma\(\\text\{SiLU\}\(\\bm\{H\}\_\{i\}\\bm\{\{W\}\}\_\{i\}^\{d\}\)\\bm\{\{W\}\}\_\{i\}^\{u\}\)of MoE\-5B as an example, in the main experiments, we setdint=d=192d\_\{\\mathrm\{int\}\}=d=192to maximize model capacity, resulting in a full\-width gating structure\. In this experiment, we evaluate three reduced dimensions,dint∈\{16,32,64\}d\_\{\\mathrm\{int\}\}\\in\\\{16,32,64\\\}, and apply the same low\-rank compression scheme to the X\-gate\. To assess computational efficiency and identify the effectiveness–efficiency Pareto frontier, we calculate the computation costs of attention gating modules under differentdintd\_\{\\mathrm\{int\}\}\. Since the C\-gate provides head\-wise gating, its does not bring in many additional parameters \(nearly 3%\)\. Hence, we remove the C\-gate in the following experiments to more explicitly show the impact of low\-rank compression on H\-gate and X\-gate\. We adopt the same training and evaluation settings as in the main experiments, and report the results in Fig\.[5](https://arxiv.org/html/2608.11805#S4.F5)\.

![Refer to caption](https://arxiv.org/html/2608.11805v1/low_rank_pareto_frontier.png)Figure 5:HyGA’s variants with different low\-rank compressed matrices in H\-gate and X\-gate could extend the effectiveness\-efficiency Pareto frontier of gated attention\.We can observe that: HyGA achieves lower losses with largerdintd\_\{\\mathrm\{int\}\}\(indicating less compression rates and more computation costs\)\. Whendint=32d\_\{\\mathrm\{int\}\}=32, HyGA utilizes only approximately26%26\\%of the gating parameters required by the original Gated attention baseline, while achieving slightly better performance on training loss\. Moreover, we verify that HyGA with low\-rank matrix factorization \(dint=32d\_\{\\mathrm\{int\}\}=32\) also achieves better overall performance on downstream tasks\. It demonstrates that HyGA does extend the effectiveness\-efficiency Pareto frontier of the original gated attention\. We also attempt to set differentdintd\_\{\\mathrm\{int\}\}for X\-gate and H\-gate, which seldom brings in further loss advantages\. In practice, we could flexibly set appropriate intermediate dimensions according to the computation constraints\.

### 4\.5Training Stability \(RQ3\)

Although gated attention can largely alleviate the attention\-sink phenomenon, we find that its attention score assigned to the BOS token still remains relatively high compared to other tokens, indicating that attention sinks still persist\. Motivated by this observation, we introduce learnable attention sinks into gated attention to further reduce sink ratios and improve training stability\. As shown in Figure[6](https://arxiv.org/html/2608.11805#S4.F6), HyGA with learnable sink substantially reduces the BOS\-token’s attention score \(especially in the last few layers\)\. We also observe a marked reduction in massive activations, suggesting that the learnable attention sink further improves training stability\. Therefore, we incorporate the learnable sink into all subsequent HyGA experiments\. We further compare the BOS\-token attention scores of gated attention \+ learnable attention sink and HyGA\. HyGA achieves consistently low attention sink ratios in all layers, indicating that other components of HyGA further contribute to the training stability\.

![Refer to caption](https://arxiv.org/html/2608.11805v1/bos_score_layer_averages.png)\(a\)Layer\-wise average BOS token scores\.
![Refer to caption](https://arxiv.org/html/2608.11805v1/bos_score_gt_0p3_ratio_layer_averages.png)\(b\)Layer\-wise ratios of BOS token scores greater than 0\.3\.

Figure 6:Layer\-wise BOS tokens’ attention score statistics of MoE\-5B models: \(a\) average BOS token scores and \(b\) ratios of BOS token scores greater than 0\.3\.For Qwen3, as in Figure[7](https://arxiv.org/html/2608.11805#S4.F7), both the baseline and Gated attention exhibit obvious loss spikes in training, whereas HyGA does not under the same recommended learning rate\. These observations suggest that HyGA is more robust to larger learning rates and provides more stable optimization\.

![Refer to caption](https://arxiv.org/html/2608.11805v1/qwen_loss_spike.png)Figure 7:Training loss curves of the Qwen3 baseline, Gated GQA, and HyGA during the early stage of training\. HyGA exhibits more stable optimization without apparent loss spikes\.

## 5Related Works

##### Gating Method in Neural Networks

Gating is a long\-standing method for controlling information flow in neural networks\. Early models such as LSTMs\([18](https://arxiv.org/html/2608.11805#bib.bib13)\)and GRUs\([7](https://arxiv.org/html/2608.11805#bib.bib14)\)use gates to regulate memory updates, while later works such as GLU\([11](https://arxiv.org/html/2608.11805#bib.bib15)\)and SwiGLU\([34](https://arxiv.org/html/2608.11805#bib.bib7)\)further introduce gating into feed\-forward activations\. Gating also plays an important role in modern recurrent and state\-space models, including S4, Mamba, and Mamba\-2\([14](https://arxiv.org/html/2608.11805#bib.bib37);[13](https://arxiv.org/html/2608.11805#bib.bib38);[10](https://arxiv.org/html/2608.11805#bib.bib39)\), where it controls state updates and memory retention\. These examples suggest that gating is not merely an auxiliary technique, but a general architectural principle for improving selectivity and expressive capacity\.

##### Gated Attention and Its Variants

Gated attention\([31](https://arxiv.org/html/2608.11805#bib.bib4)\)introduces a head\-specific sigmoid gate after the scaled dot\-product attention output, showing that this simple modification improves performance and training stability\. This work provides the foundation for studying gating as a structural enhancement to attention rather than as a peripheral fusion module\. Since Gated attention, some related studies have explored several directions related to gated attention\. Some works place gates at different locations in the attention computation, such as applying a forget gate to unnormalized attention scores\([25](https://arxiv.org/html/2608.11805#bib.bib41)\)or computing gates from value states to mitigate extreme\-token phenomena\([6](https://arxiv.org/html/2608.11805#bib.bib18)\)\. Another significant direction is low\-rank gate, such as gated norm\([30](https://arxiv.org/html/2608.11805#bib.bib40)\)\. While these studies demonstrate the usefulness and generality of gating, they typically investigate individual design dimensions, such as gate placement or low\-rank parameterization\. In contrast, our work focuses on the architecture of the gate itself by jointly considering its conditioning pathways, coupling mechanism, parameterization, and cross\-head interaction, aiming to improve the model performance and training stability\.

## 6Conclusion and Future Work

In this work, we introduced HyGA with three gates depend on different inputs and granularities: X\-gate, H\-gate, and C\-gate\. Besides, we also adopt low\-rank matrix decomposition and learnable sink for efficiency and training stability\. Through more comprehensive information interaction capability, HyGA achieves improved performance\.

In the future, we plan to scale HyGA to larger models and evaluate its effectiveness at greater scales\. We will also investigate its generalizability across different attention backbones, such as linear/sparse attention architectures\.

## References

- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p6.1),[§3\.4](https://arxiv.org/html/2608.11805#S3.SS4.p2.1)\.
- Ainslieet al\.\(2023\)J\. Ainslie, J\. Lee\-Thorp, M\. De Jong, Y\. Zemlyanskiy, F\. Lebrón, and S\. SanghaiGqa: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of EMNLP,Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Barberoet al\.\(2025\)F\. Barbero, A\. Arroyo, X\. Gu, C\. Perivolaropoulos, M\. Bronstein, P\. Veličković, and R\. PascanuWhy do llms attend to the first token?\.arXiv preprint arXiv:2504\.02732\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p3.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.Piqa: reasoning about physical commonsense in natural language\.InProceedings of AAAI,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Buet al\.\(2025\)R\. Bu, H\. Zhong, W\. Chen, and Y\. LiValue\-state gated attention for mitigating extreme\-token phenomena in transformers\.arXiv preprint arXiv:2510\.09017\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px2.p1.1)\.
- Choet al\.\(2014\)K\. Cho, B\. Van Merriënboer, Ç\. Gulçehre, D\. Bahdanau, F\. Bougares, H\. Schwenk, and Y\. BengioLearning phrase representations using rnn encoder–decoder for statistical machine translation\.InProceedings of EMNLP,Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Dao and Gu \(2024\)T\. Dao and A\. GuTransformers are ssms: generalized models and efficient algorithms through structured state space duality\.arXiv preprint arXiv:2405\.21060\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1),[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Dauphinet al\.\(2017\)Y\. N\. Dauphin, A\. Fan, M\. Auli, and D\. GrangierLanguage modeling with gated convolutional networks\.InICML,Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Elfwinget al\.\(2018\)S\. Elfwing, E\. Uchibe, and K\. DoyaSigmoid\-weighted linear units for neural network function approximation in reinforcement learning\.Neural networks\.Cited by:[§3\.2](https://arxiv.org/html/2608.11805#S3.SS2.SSSx1.p1.2)\.
- Gu and Dao \(2023\)A\. Gu and T\. DaoMamba: linear\-time sequence modeling with selective state spaces\.arXiv preprint arXiv:2312\.00752\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2021\)A\. Gu, K\. Goel, and C\. RéEfficiently modeling long sequences with structured state spaces\.arXiv preprint arXiv:2111\.00396\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Guet al\.\(2025\)X\. Gu, T\. Pang, C\. Du, Q\. Liu, F\. Zhang, C\. Du, Y\. Wang, and M\. LinWhen attention sink emerges in language models: an empirical view\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p3.1)\.
- Hendryckset al\.\(2020\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Hochreiter and Schmidhuber \(1997\)S\. Hochreiter and J\. SchmidhuberLong short\-term memory\.Neural computation\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Huanget al\.\(2023\)Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, Y\. Fu,et al\.C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.Advances in NIPS\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaqa: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of ACL,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.Natural questions: a benchmark for question answering research\.Transactions of the ACL\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Liet al\.\(2024\)H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. BaldwinCmmlu: measuring massive multitask language understanding in chinese\.InFindings of ACL,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Linet al\.\(2025\)Z\. Lin, E\. Nikishin, X\. He, and A\. CourvilleForgetting transformer: softmax attention with a forget gate\.InICLR,Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo,et al\.Deepseek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Liuet al\.\(2025\)A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong,et al\.Deepseek\-v3\. 2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1)\.
- Liuet al\.\(2023\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZhangIs your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in NIPS\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Loshchilov and Hutter \(2017\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.arXiv preprint arXiv:1711\.05101\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Qiuet al\.\(2026a\)Z\. Qiu, Z\. Huang, K\. Wen, P\. Jin, B\. Zheng, Y\. Zhou, H\. Huang, Z\. Wang, X\. Li, H\. Zhang,et al\.A unified view of attention and residual sinks: outlier\-driven rescaling is essential for transformer training\.arXiv preprint arXiv:2601\.22966\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2026b\)Z\. Qiu, Z\. Wang, B\. Zheng, Z\. Huang, K\. Wen, S\. Yang, R\. Men, L\. Yu, F\. Huang, S\. Huang,et al\.Gated attention for large language models: non\-linearity, sparsity, and attention\-sink\-free\.Advances in NIPS\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p4.1),[§2](https://arxiv.org/html/2608.11805#S2.p4.1),[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1),[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px2.p1.1)\.
- Reinet al\.\(2023\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGpqa: a graduate\-level google\-proof q&a benchmark\.arXiv preprint arXiv:2311\.12022\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. ChoiSocial iqa: commonsense reasoning about social interactions\.InProceedings of EMNLP\-IJCNLP,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Shazeer \(2020\)N\. ShazeerGlu variants improve transformer\.arXiv preprint arXiv:2002\.05202\.Cited by:[§5](https://arxiv.org/html/2608.11805#S5.SS0.SSS0.Px1.p1.1)\.
- Suet al\.\(2026\)Z\. Su, H\. Zhang, W\. Wu, Y\. Zhang, Y\. Liu, H\. Xiao, Q\. Yang, Y\. Sun, R\. Yang, C\. Zhang,et al\.Attention sink in transformers: a survey on utilization, interpretation, and mitigation\.arXiv preprint arXiv:2604\.10098\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p3.1)\.
- Sunet al\.\(2026\)S\. Sun, A\. Canziani, Y\. LeCun, and J\. ZhuThe spike, the sparse and the sink: anatomy of massive activations and attention sinks\.arXiv preprint arXiv:2603\.05498\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p3.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in NIPS\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p1.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InProceedings of ICLR,Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p3.1),[§3\.4](https://arxiv.org/html/2608.11805#S3.SS4.p1.1)\.
- Xuet al\.\(2026\)A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx2.p1.1)\.
- Yanget al\.\(2025b\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving mamba2 with delta rule\.InICLR,Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1)\.
- Yanget al\.\(2023\)S\. Yang, B\. Wang, Y\. Shen, R\. Panda, and Y\. KimGated linear attention transformers with hardware\-efficient training\.arXiv preprint arXiv:2312\.06635\.Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1)\.
- Yuanet al\.\(2025\)J\. Yuan, H\. Gao, D\. Dai, J\. Luo, L\. Zhao, Z\. Zhang, Z\. Xie, Y\. Wei, L\. Wang, Z\. Xiao,et al\.Native sparse attention: hardware\-aligned and natively trainable sparse attention\.InProceedings of ACL,Cited by:[§1](https://arxiv.org/html/2608.11805#S1.p2.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaswag: can a machine really finish your sentence?\.InProceedings of ACL,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.
- Zhonget al\.\(2024\)W\. Zhong, R\. Cui, Y\. Guo, Y\. Liang, S\. Lu, Y\. Wang, A\. Saied, W\. Chen, and N\. DuanAgieval: a human\-centric benchmark for evaluating foundation models\.InFindings of NAACL,Cited by:[§4\.1](https://arxiv.org/html/2608.11805#S4.SS1.SSSx1.p1.1)\.