GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference

# GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference
Source: [https://arxiv.org/html/2607.24788](https://arxiv.org/html/2607.24788)
###### Abstract

As Large Language Models scale to increasingly long contexts, the memory I/O and computational overhead of the Key\-Value \(KV\) cache during decoding emerges as the primary throughput bottleneck\. To address this, we proposeGlide, a Guided Layerwise Hybrid Attention that strategically integrates sliding\-window softmax attention with linear recurrent aggregation\.Glideis motivated by layer\-wise heterogeneity: early layers exhibit high sensitivity to softmax removal, while deeper layers demonstrate redundancy and tolerate aggressive replacement by linear alternatives\. Leveraging this insight,Glideintroduces alayer\-wise adaptive mechanismwherein each layer balances an efficient linear recurrence with a variable\-sized softmax window\. Unlike uniform hybrid approaches,Glidenon\-uniformly compresses the softmax footprint across the model, reducingaggregate KV cache I/Owhile preserving expressive power where most vital\. Empirical evaluations demonstrate theGlideachieves superior performance\-efficiency tradeoffs, reducing end\-to\-end latency for long\-context generation without compromising quality\.

## IIntroduction

Long\-context Large Language Model \(LLM\) inference is increasingly limited not by compute alone, but by data movement during autoregressive decoding\. As the KV\-cache grows with sequence length, each generated token demands retrieval of an expanding volume of cached key\-value states, shifting execution from compute\-bound to memory\-bandwidth\-bound regimes\[[8](https://arxiv.org/html/2607.24788#bib.bib8)\]\. This bottleneck is particularly acute in long\-context generation, where KV\-cache growth inflates both memory footprint and off\-chip data transfer, degrading hardware utilization and overall throughput\. Such constraints pose a critical barrier to scalable deployment on memory\-constrained accelerators and in latency\-sensitive serving environments\[[16](https://arxiv.org/html/2607.24788#bib.bib35)\]\. This motivates a central question:How can long\-context decoding achieve lower latency and reduced memory I/O while maintaining generation quality?

Recent work has approached efficient long\-context inference by optimizing KV\-cache management through two main strategies: a\)Eviction\-basedmethods reduce memory and compute by selectively discarding cached tokens deemed less relevant\[[26](https://arxiv.org/html/2607.24788#bib.bib9),[3](https://arxiv.org/html/2607.24788#bib.bib5),[17](https://arxiv.org/html/2607.24788#bib.bib6)\], while b\)Retention\-basedmethods preserve contextual information through hybrid attention mechanisms that combine local softmax attention with efficient recurrent\-style aggregation\. Retention\-based hybrid architectures \(also see Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)such as LoLCats\[[24](https://arxiv.org/html/2607.24788#bib.bib25)\]and Liger\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]\) offer a particularly promising design point: they apply parameter\-efficient fine\-tuning \(PEFT\) to distill pretrained softmax attention into linear recurrent forms, improving decoding efficiency without explicitly evicting context from the KV\-cache\. However, existing retention\-based methods apply a uniform hybridization policy across all layers, replacing softmax attention with linear alternatives at fixed positions throughout the network\. This one\-size\-fits\-all strategy overlooks a critical observation: attention layers vary significantly in their sensitivity to linearization\. Early layers with foundational token representations may rely more heavily on precise softmax attention, while deeper layers operating on increasingly abstract features may tolerate aggressive approximation with minimal performance degradation\.

Can a non\-uniform, layer\-wise hybrid attention policy guided by depth\-dependent sensitivity achieve a superior efficiency–accuracy Pareto frontier compared to uniform hybridization?

![Refer to caption](https://arxiv.org/html/2607.24788v1/x1.png)

\(\(a\)\)
![Refer to caption](https://arxiv.org/html/2607.24788v1/x2.png)\(\(b\)\)

Figure 1:Layer\-Wise Adaptive Attention Strategy ofGlide\.\(a\) Attention Mechanisms:Glidestrategically transitions from softmax attention \(red\) in early layers to linear attention \(green\) in deeper layers, concentrating expensive computation where most critical, unlike vanilla softmax, sliding window attention \(SWA\), or pure linear approaches\.\(b\) Pareto Analysis:Glideconfigurations achieve45×45\\times–62×62\\timeslower KV cache I/O than baseline LLaMA while retaining92%92\\%–96%96\\%accuracy, outperforming uniform alternatives \(SWA, Hybrid Model\) by exploiting layer\-wise heterogeneity\.As a first step towards answering this question, we conduct an empirical study in which we systematically vary the ratio of softmax\-to\-linear attention across transformer layers and measure the resulting impact on downstream task accuracy\. Our analysis reveals a clear layerwise heterogeneity in softmax importance: earlier layers responsible for establishing low\-level token representations and local syntactic structure benefit significantly from retaining softmax\-based attention, whereas deeper layers, which operate on more abstract semantic features, tolerate aggressive linearization with minimal impact on generation quality\. This finding suggests that uniform softmax allocation across layers is fundamentally suboptimal, leaving substantial efficiency gains unrealized\. Building on this insight, we introduceGlide\(GuidedLayerwise Hybrid Attention for EfficientDecoding\), a retention\-based architecture that allocates softmax attention non\-uniformly across layers according to their measured sensitivity\.

As illustrated in Fig\.[1](https://arxiv.org/html/2607.24788#S1.F1),Glideexploits layer\-wise sensitivity patterns to achieve superior efficiency\-accuracy trade\-offs compared to various attention schemes\. Fig\.[1](https://arxiv.org/html/2607.24788#S1.F1)\(a\) contrasts four attention mechanisms: vanilla softmax attention \(red\) applies full quadratic attention uniformly across all layers, achieving optimal performance but incurring prohibitive memory costs; sliding window attention \(red with evicted tokens\) reduces memory by discarding distant context, sacrificing accuracy; linear attention \(green\) eliminates KV cache overhead entirely but suffers severe degradation; andGlidestrategically transitions from softmax attention in early layers to linear attention in deeper layers, concentrating expensive computation where most critical\. This layer\-wise allocation directly reduces KV cache memory traffic during autoregressive decoding, the dominant bottleneck in long\-context generation, while preserving representational fidelity in sensitive layers\. Fig\.[1](https://arxiv.org/html/2607.24788#S1.F1)\(b\) demonstrates the resulting Pareto frontier:Glideconfigurations occupy the favorable top\-left region, achieving45×45\\times–62×62\\timeslower KV cache I/O than baseline while retaining92%92\\%–96%96\\%accuracy, substantially outperforming uniform alternatives including hybrid models and sliding window attention which fail to exploit layer\-wise heterogeneity\. By enabling strategic allocation of softmax versus linear attention across depth,Glideunlocks previously inaccessible operating points that balance memory bandwidth, latency, and task performance\. Our main contributions are as follows:

- •We conduct a systematic study of hybrid attention sensitivity across transformer depth, revealing that softmax attention importance is strongly layer\-dependent: early layers are markedly more sensitive to linearization, with full linearization causing catastrophic accuracy collapse \(36% avg\.\), while late layers tolerate complete linearization with minimal degradation \(Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)\)\.
- •We introduceGlide, a guided layerwise hybrid attention framework that allocates softmax attention non\-uniformly across transformer blocks based on observed layer\-wise sensitivity\.Glidepartitions the network into early, middle, and late blocks, assigning full softmax \(δ=0\\delta=0\) to early layers, partial linearization \(δ=α⋅w\\delta=\\alpha\\cdot w,α∈\[0,1\]\\alpha\\in\[0,1\]\) to middle layers, and full linearization \(δ=w\\delta=w\) to late layers\. This reduces theδ\\delta\-configuration search space to a single scalarα\\alphathat continuously interpolates between full softmax and full linearization \(Section[III\-B](https://arxiv.org/html/2607.24788#S3.SS2)\)\.
- •We demonstrate thatGlideintegrates seamlessly with existing retention\-based architectures \(Liger, LoLCats\), achieving up to62×\\timesreduction in KV\-cache I/O and3\.3×\\timesdecoding speedup while retaining 92–94% of baseline accuracy across six reasoning benchmarks on Llama\-3\-8B and Mistral\-7B \(Section[IV](https://arxiv.org/html/2607.24788#S4)\)\.

## IIBackground

This section gives an overview of attention and KV\-cache mechanisms in autoregressive Transformer inference, reviews prior approaches for improving long\-context efficiency, and highlights the gaps that motivatesGlide\.

### II\-ANotations

We consider anLL\-layer Transformer model withHHattention heads per layer\. Letdmodeld\_\{\\text\{model\}\}denote the model dimension andd=dmodel/Hd=d\_\{\\text\{model\}\}/Hdenotes the per\-head dimension\. For autoregressive generation at positionii, we denote the query vector as𝐪i∈ℝd\\mathbf\{q\}\_\{i\}\\in\\mathbb\{R\}^\{d\}and the key\-value pairs as\{𝐤j,𝐯j\}j=1i\\\{\\mathbf\{k\}\_\{j\},\\mathbf\{v\}\_\{j\}\\\}\_\{j=1\}^\{i\}with𝐤j,𝐯j∈ℝd\\mathbf\{k\}\_\{j\},\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{d\}\. The attention weights are denotedαi​j\\alpha\_\{ij\}, representing the normalized similarity between query𝐪i\\mathbf\{q\}\_\{i\}and key𝐤j\\mathbf\{k\}\_\{j\}\. The attention output at positioniiis written as𝐎i\\mathbf\{O\}\_\{i\}, with superscripts distinguishing variants:𝐎iSWA\\mathbf\{O\}\_\{i\}^\{\\text\{SWA\}\}for sliding window attention,𝐎iLinear\\mathbf\{O\}\_\{i\}^\{\\text\{Linear\}\}for linear attention, and𝐎iGlide\\mathbf\{O\}\_\{i\}^\{\\text\{Glide\}\}forGlideattention\. We usewwto denote the sliding window size andδ\\deltato control the extent of linearization within the window, where𝜹ℓ\\boldsymbol\{\\delta\_\{\\ell\}\}specifies the layer\-wise allocation for layerℓ∈\[1,L\]\\ell\\in\[1,L\]\. In linear attention,ϕ​\(⋅\)\\phi\(\\cdot\)denotes the separable kernel \(feature map\) applied to queries and keys, while𝐒i∈ℝd×d\\mathbf\{S\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times d\}and𝐙i∈ℝd\\mathbf\{Z\}\_\{i\}\\in\\mathbb\{R\}^\{d\}represent the recurrent state and normalization factor, respectively\. The operator⊕\\oplusdenotes the weighted combination of linear and softmax attention outputs based on their respective normalizing denominators\. For block\-wise allocation, the model is partitioned into early, middle, and late segments containingLeL\_\{e\},LmL\_\{m\}, andLlL\_\{l\}layers, and the configuration tuple𝜹b=\(δb1,δb2,δb3\)\\boldsymbol\{\\delta\}\_\{b\}=\(\\delta\_\{b\_\{1\}\},\\delta\_\{b\_\{2\}\},\\delta\_\{b\_\{3\}\}\)specifies theδ\\deltavalues assigned to each segment, where theδ=w\.α\\delta=w\.\\alphais generic definition ofδ\\deltaand we distinguish the attention scoreα𝐢∈ℝn\\mathbf\{\\alpha\_\{i\}\}\\in\\mathbb\{R\}^\{n\}from the cache sparsity factorα∈\[0,1\]\\alpha\\in\[0,1\]\.

### II\-BPreliminaries on Attention and KV Cache

Overview of Self\-Attention\.In autoregressive next\-token prediction at positionii, the query vector𝐪i∈ℝd\\mathbf\{q\}\_\{i\}\\in\\mathbb\{R\}^\{d\}\(whered=dmodel/Hd=d\_\{\\text\{model\}\}/HforHHattention heads\) attends to all previously generated keys\{𝐤j\}j=1i\\\{\\mathbf\{k\}\_\{j\}\\\}\_\{j=1\}^\{i\}via scaled dot\-product similarity, producing softmax\-normalized attention weights that quantify relevance\. The attention outputOiO\_\{i\}is then obtained as the weighted aggregation of the corresponding value vectors\{𝐯j\}j=1i\\\{\\mathbf\{v\}\_\{j\}\\\}\_\{j=1\}^\{i\}and subsequently linearly projected throughWoW\_\{o\}to form the final head contribution\.Consequently, predicting the next token requires access to the entire set of prior key–value pairs\{𝐤j,𝐯j∣j≤i\}\\\{\\mathbf\{k\}\_\{j\},\\mathbf\{v\}\_\{j\}\\mid j\\leq i\\\}, resulting in a quadratic computational complexity with respect to the sequence length during full attention computation\.

Linearized Attention\.Linear attention\[[14](https://arxiv.org/html/2607.24788#bib.bib21)\]approximates softmax attention by replacing the exponential similarity kernel with separable feature mappingsϕ​\(⋅\)\\phi\(\\cdot\)applied independently to queries and keys:αi​j≈ϕ​\(𝐪i\)​ϕ​\(𝐤j\)⊤∑j=1iϕ​\(𝐪i\)​ϕ​\(𝐤j\)⊤\\alpha\_\{ij\}\\approx\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}\}\{\\sum\_\{j=1\}^\{i\}\\phi\(\\mathbf\{q\}\_\{i\}\)\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}\}\. This enables constant\-space recurrent computation:OiLinear=ϕ​\(𝐪i\)​𝐒iϕ​\(𝐪i\)​𝐙iO\_\{i\}^\{\\text\{Linear\}\}=\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\mathbf\{S\}\_\{i\}\}\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\mathbf\{Z\}\_\{i\}\}, where cumulative states𝐒i=𝐒i−1\+ϕ​\(𝐤i\)⊤​𝐯i\\mathbf\{S\}\_\{i\}=\\mathbf\{S\}\_\{i\-1\}\+\\phi\(\\mathbf\{k\}\_\{i\}\)^\{\\top\}\\mathbf\{v\}\_\{i\}and𝐙i=𝐙i−1\+ϕ​\(𝐤i\)⊤\\mathbf\{Z\}\_\{i\}=\\mathbf\{Z\}\_\{i\-1\}\+\\phi\(\\mathbf\{k\}\_\{i\}\)^\{\\top\}accumulate key\-value interactions\. Memory complexity is reduced fromO​\(n\)O\(n\)toO​\(1\)O\(1\)with respect to context length, and KV cache I/O is eliminated entirely\.However, kernel\-based linear attention sacrifices expressiveness\[[25](https://arxiv.org/html/2607.24788#bib.bib23),[10](https://arxiv.org/html/2607.24788#bib.bib11)\]: the separable approximation degrades attention quality, particularly in early transformer layers where precise token interactions are critical\.

KV Cache\.To avoid recomputing past states during autoregressive inference, KV caching stores generated key–value pairs\(𝐤i,𝐯i\)\(\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{i\}\)for reuse\. The process involves aprefillstage that initializes the cache from the prompt, followed by adecodestage where tokens are generated sequentially by attending to cached entries and appending new pairs\.Although this removes redundant computation, the cache grows linearly with context lengthnn, and each decoding step requires memory access, which grows linearly with respective tonn\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/x3.png)

\(\(a\)\)
![Refer to caption](https://arxiv.org/html/2607.24788v1/x4.png)\(\(b\)\)

Figure 2:Analysis of Zero\-Shot Accuracy Under Varying Softmax Attention Intensity \(Non Fine\-Tuned\)\.\(a\) Theδ\\deltaparameter controls the Linear:Softmax ratio within each layer’s sliding windowww, trading efficiency for representational fidelity\. \(b\) Layer\-wise sensitivity analysis across Early, Middle, and Late layer groups, evaluated on LM\-Eval benchmarks \(PiQA, ARC\-e, ARC\-c, HellaSwag, WinoG\) withw=64w=64andd=4096d=4096\. GLA \(Gated Linear Attention\) replaces softmax within the window using a fixed\-size recurrent state, preserving context beyond the sliding window; the red bar \(δ=w\\delta=w\) denotes full linearization\.
### II\-CPolicies of Sliding Window Attention

Sliding Window Attention\.To mitigate the linear memory growth and per\-token attention cost of full KV cache, prior work proposes Sliding Window Attention \(SWA\)\[[2](https://arxiv.org/html/2607.24788#bib.bib3)\], where the query𝐪i\\mathbf\{q\}\_\{i\}attends only to the most recentwwkey–value pairs\. Concretely, attention weights are computed over the restricted rangej∈\[i−w\+1,i\]j\\in\[i\-w\+1,i\], i\.e\.,αi​jSWA=exp⁡\(𝐪i​𝐤j⊤\)∑j′=i−w\+1iexp⁡\(𝐪i​𝐤j′⊤\)\\alpha\_\{ij\}^\{\\text\{SWA\}\}=\\frac\{\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\top\}\)\}\{\\sum\_\{j^\{\\prime\}=i\-w\+1\}^\{i\}\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j^\{\\prime\}\}^\{\\top\}\)\}, and the output isOiSWA=∑j=i−w\+1iαi​jSWA​𝐯jO\_\{i\}^\{\\text\{SWA\}\}=\\sum\_\{j=i\-w\+1\}^\{i\}\\alpha\_\{ij\}^\{\\text\{SWA\}\}\\mathbf\{v\}\_\{j\}\. This reduces the per\-token memory and compute complexity toO​\(w\)O\(w\)withw<nw<n\.However, SWA follows a first\-in\-first\-out eviction policy, whereby older tokens are discarded as new ones enter the window, potentially removing semantically important context and degrading recall over long\-range dependencies\.We categorize prior work addressing the limitations of recency\-based eviction and the performance degradation caused by constrained memory into two groups\.

\(a\) Eviction\-based Policies\.Several works attempt to mitigate the limitations of recency\-based eviction in SWA including, AdaKV\[[8](https://arxiv.org/html/2607.24788#bib.bib8)\]introduces an adaptive, plug\-and\-play KV\-budget allocation mechanism that integrates with existing cache management policies\. MSWA\[[23](https://arxiv.org/html/2607.24788#bib.bib13)\]instead varies the attention window sizewwacross heads and layers to improve computational efficiency while maintaining model performance\.Although saliency\-based eviction mitigates the limitations of recency\-based policies, it often incurs noticeable performance degradation as a consequence of discarding cached entries\.

\(b\) Retention\-based Policies\.These works address the performance degradation caused by eviction\-based policies under constrained memory by incorporating a linear recurrent state\. LESS\[[6](https://arxiv.org/html/2607.24788#bib.bib10)\]introduces a sparse KV\-cache strategy where discarded KV pairs are absorbed into a recurrent state via linear attention with an MLP\-basedϕ\\phi\-kernel\. BASED\[[1](https://arxiv.org/html/2607.24788#bib.bib4)\]combines sliding\-window attention with a degree\-2 Taylor approximation of the softmax to realize linear attention while preserving global context\. Infini\-Transformer\[[18](https://arxiv.org/html/2607.24788#bib.bib14)\]processes inputs in segments while maintaining a recurrent state using ELU\-based linear attention, enabling effectively unbounded context under limited memory\. EdgeInfinite\[[4](https://arxiv.org/html/2607.24788#bib.bib15)\]follows a similar segmented design but introduces a learnable gate to regulate the contribution of ELU\-based linear attention\.Despite the performance enhancement via constant recurrent state, these policies often requires extensive re\-training\.

### II\-DParameter\-Efficient Hybrid Architecture

The inefficiency introduced by the complex training procedures in prior hybrid architectures\[[1](https://arxiv.org/html/2607.24788#bib.bib4),[6](https://arxiv.org/html/2607.24788#bib.bib10)\]can be mitigated by adopting parameter\-efficient fine\-tuning \(PEFT\) strategies\. In particular, Low\-Rank Adaptation \(LoRA\)\[[12](https://arxiv.org/html/2607.24788#bib.bib26)\]enables efficient adaptation by injecting trainable low\-rank matrices into frozen pretrained weights, dramatically reducing memory and training cost\. LoLCats\[[24](https://arxiv.org/html/2607.24788#bib.bib25)\]employs this LoRA\-based approach to distill pretrained softmax attention into learnableϕ\\phi\-kernel linear attention, while Liger\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]applies the same strategy for non\-learnableϕ\\phi\-kernel linear attention, establishing a practical path for hybrid architecture adaptation without full retraining\. However, despite prior works on adaptive KV\-cache budgeting\[[8](https://arxiv.org/html/2607.24788#bib.bib8),[23](https://arxiv.org/html/2607.24788#bib.bib13),[3](https://arxiv.org/html/2607.24788#bib.bib5)\], we observe that most hybrid architectures still employ a uniform hybridization strategy across layers\.This raises a fundamental question: is uniform layer\-wise hybridization optimal, or can non\-uniformδ\\delta\-allocation exploit depth\-dependent linearization tolerance?

## IIIMethodology

In this section, we presentGlide, aGuidedLayerwise Hybrid Attention that strategically integrates sliding\-window softmax attention with linear recurrent aggregation\. We first present our motivating observations \(Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)\) through an empirical study in which we systematically vary the ratio of softmax\-to\-linear attention across transformer layers and measure the resulting impact on downstream task accuracy, which reveal the layer\-wise heterogeneity in softmax importance and tolerance to linearization\. We then present theGlideframework in detail \(Section[III\-B](https://arxiv.org/html/2607.24788#S3.SS2)\) as well as the configuration strategies \(Section[III\-C](https://arxiv.org/html/2607.24788#S3.SS3)\) to exploit the layer\-wise heterogeneity\.

### III\-AMotivating Observations

Prior work has shown that the quality degradation inherent to attention linearization can be partially recovered through selective softmax augmentation\[[24](https://arxiv.org/html/2607.24788#bib.bib25)\]\. However, the optimal allocation of softmax computation remains an open problem: excessive softmax reintroduces the KV cache bottleneck that linearization seeks to eliminate, while insufficient softmax compromises model expressivity\. This raises a fundamental question:What is the minimal softmax budget required to maintain downstream quality under linearization, and is this requirement uniform across layers or depth\-dependent?

Accuracy Recovery from Linearization\.To understand how softmax attention affects model quality, we analyze the Liger\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]modelwith Llama\-3 8B architecture as backboneby progressively reintroducing softmax attention from a fully linearized baseline without any fine\-tuning, as shown in Fig\.[2](https://arxiv.org/html/2607.24788#S2.F2)\(a\)\. Fig\.[2](https://arxiv.org/html/2607.24788#S2.F2)\(b\), “All Layers” traces this recovery: zero\-shot accuracy starts at 36% under full linearization and steadily climbs as softmax is uniformly reintroduced, reaching 72% in the softmax\-only configuration\. Interestingly, the recovery curve shows diminishing returns, suggesting that only a sparse softmax budget may be needed to restore model quality while retaining most efficiency gains, and to assess the generalizability of this behavior, we also evaluate the Liger setup using a Mistral backbone; results are presented in Fig\.[8](https://arxiv.org/html/2607.24788#A0.F8)\(AppendixGLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference\)\. This leads to a natural question: if only a fraction of softmax computation is required, can we allocate it selectively rather than spreading it uniformly? We hypothesize that Transformers exhibit non\-uniform sensitivity to linearization along the depth axis, with early layers acting as critical representational bottlenecks while deeper layers possess enough redundancy to tolerate linear approximation\. To test this, we analyzed our baseline LLM using a pairwise isolated linearization setup, where a pair of layers from a specific region \(early, middle, or late\) is linearized while all other layers retain full softmax attention\. As Fig\.[2](https://arxiv.org/html/2607.24788#S2.F2)\(b\) reveals, linearizing early layers\{1,2\}\\\{1,2\\\}causes a catastrophic accuracy drop to roughly 36%, whereas linearizing middle\{16,17\}\\\{16,17\\\}and late\{31,32\}\\\{31,32\\\}layers produces performance nearly indistinguishable from the full\-softmax baseline\. This finding, that tolerance to linearization grows with layer depth, forms the foundation of theGlideframework\. Rather than applying a uniform policy,Glideselectively prioritizes softmax attention in early\-stage bottlenecks while maximizing linearization in deeper layers, preserving model utility while reducing KV\-cache overhead\.

Observation 1:Transformer layers exhibitdepth\-dependent sensitivity to attention linearization:early layers require high softmax intensity to preserve accuracy, while deeper layers tolerate aggressive linearization with minimal degradation\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/x5.png)Figure 3:Attention Latency Analysis Across Linearization Ratios and Baseline\.Profiling Llama\-3\-8B decoding from 20K\-token prefill across uniform linear:softmax ratios with the base window sizew=20​Kw=20\\text\{K\}and model dimensiond=4096d=4096\. \(a\) Baseline encounters OOM mid\-sequence while GLIDE configurations complete decoding with stable latency, achieving3\.3×3\.3\\timesspeedup at full linearization\. \(b\) Operator breakdown: softmax \(red\) dominates baseline cost and triggers OOM at 3K tokens; increasing linear ratio eliminates this bottleneck while non\-attention overhead \(gray\) remains constant\. \(c\) Attention\-only latency: linear layers contribute 7ms regardless of sequence length, yielding17\.4×17\.4\\timesreduction versus softmax\.![Refer to caption](https://arxiv.org/html/2607.24788v1/x6.png)Figure 4:KV Cache I/O Scales Linearly with Softmax Layer Proportion\.Per\-token KV cache memory traffic for Llama\-3\-8B at32,000t​h32,000^\{th\}token generation for the window sizew=20kw=\\text\{20k\}and the model dimensiond=4096d=\\text\{4096\}\. Softmax layers retrieve the full KV cache per token; linear layers maintain fixed\-size recurrent states with zero cache I/O\. Transitioning from pure softmax \(0:100\) to 75:25 linear:softmax yields a4×4\\timesreduction in memory bandwidth\.Low KV Cache I/O with Reduced FLOPs\.The structural non\-uniformity identified in our performance analysis \(Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)\) suggests a significant opportunity to optimize LLM inference by reducing both computational latency and memory footprint\. To quantify these potential efficiency gains, we evaluate a hybrid attention architecture across varying KV\-cache retrieval intensities uniformly distributed across layers\. Rather than the growing memory complexity inherent in standard Transformers, this selective retrieval strategy leverages the constant\-size recurrent states of linear layers to mitigate the I/O bottleneck\. As illustrated in Fig\.[4](https://arxiv.org/html/2607.24788#S3.F4), the per\-token KV\-cache I/O scales linearly with the proportion of softmax layers\. For the Llama\-3 8B baseline \(0:100 linear:softmax ratio\), memory movement peaks at approximately 4000 MB per token at 32,000 tokens of context\. Transitioning to a 25:75 configuration reduces this to 1875 MB, while a 75:25 ratio yields just 625 MB, achieving a 4×\\timesreduction in memory bandwidth\. In the extreme 100:0 linear configuration, the per\-token I/O overhead for historical context is eliminated entirely\. These results highlight that even modest increases in linearization yield substantial memory savings, motivating a depth\-aware allocation strategy that balances accuracy with I/O efficiency\.

Observation 2:Hybrid attention achievesmemory bandwidth reduction\. Even modest linearization \(25:75 linear:softmax\) yields substantial savings, motivating depth\-aware layer\-wise allocation\.

Improved Latency Across Configurations\.This reduction in I/O translates directly to improved temporal performance\. Fig\.[3](https://arxiv.org/html/2607.24788#S3.F3)illustrates the impact on inter\-token latency, driven by the compression of floating\-point operations within the attention mechanism\. The mean inter\-token latency drops from approximately 180ms in the baseline to roughly 55ms in the linearized setup, representing a 3\.3×\\timesspeedup\. A deeper breakdown of the attention component reveals that while the linear attention latency remains minimal and constant \(7–9ms\), the softmax overhead is the primary driver of execution time\. Notably, theGlideframework’s ability to selectively linearize high\-redundancy layers allows for a significant breach of the "memory wall" while preserving the representational fidelity of the early\-stage bottleneck layers\. By leveraging architectural insights into attention mechanisms and the efficiency characteristics revealed through observed geometric properties,Glideconstitutes an adaptive framework tailored specifically for hybrid attention architectures\. This formulation enables extended context processing while substantially reducing KV cache I/O traffic, consequently minimizing per\-token FLOP requirements and yielding lower inter\-token latency during the decode phase\.

Observation 3:Partial linearized layers can achieve a significantreduction in KV\-cache I/Oandimproves overall speedup\. This strategic retrieval yields a notable localized acceleration in attention computation while preserving critical early\-layer bottlenecks\.

### III\-BGLIDE Attention

Glideis designed around two key observations: \(1\) transformer layers exhibit non\-uniform sensitivity to linearization, and \(2\) significant efficiency gains are achievable across various retrieval rates,Glidefunctions as an adaptive module that augments existing hybrid attention mechanisms, facilitating extended context processing with improved computational efficiency relative to baseline architectures\.Glidemitigates information loss arising from selective retrieval by applying linear attention to theδ\\deltaregion within the sliding windowww\. This mechanism enables controlled linearization governed by the choice ofδ\\delta, concentrating local attention while reducing computational overhead with minimal impact on downstream performance\. The selection ofδ\\deltais guided by the observed behavior of the attention mechanism, permitting increased linearization in deeper layers where sensitivity to approximation is lower\. Furthermore, tokens evicted from the sliding window are seamlessly incorporated into the linear recurrent memory, preserving information without increasing memory footprint as the sequence length scales\.This mechanism underpins the ability ofGlideto process extended contexts efficiently with bounded memory requirements\.The proposed framework is a modular and adaptive framework designed for hybrid attention architectures that combine sliding window attention \(SWA\) with linear attention\. The framework is architecture\-agnostic, supporting a variety of linear attention variants including Taylor\-based kernels\[[1](https://arxiv.org/html/2607.24788#bib.bib4)\],\[[25](https://arxiv.org/html/2607.24788#bib.bib23)\], and GLA\-style gating mechanisms\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]\. Crucially,Glideoperates with any non\-learnableϕ\\phi\-kernel, enabling parameter\-efficient deployment without requiring additional fine\-tuning or learned projections\. This design choice ensures low\-latency inference while maintaining compatibility with pretrained transformer weights\.Given a queryqiq\_\{i\}at the instanceii, the output ofGlideattention can be defined as,

OiGlide=Oi−w−δlinear⊕Oi−w−δ\+1SWA,O^\{\\text\{Glide\}\}\_\{i\}=O^\{\\text\{linear\}\}\_\{i\-w\-\\delta\}\\;\\oplus\\;O^\{\\text\{SWA\}\}\_\{i\-w\-\\delta\+1\},\(1\)wherewwdenotes the constant KV cache size of Sliding Window Attention \(SWA\), andδ\\deltais the parameter controlling selective KV cache retrieval from this window\. The parameterδ\\deltaeffectively determines the degree of linearization applied at each layer: a largerδ\\deltaimplies greater reliance on linear attention, whileδ=w\\delta=wcorresponds to full linearization, completely bypassing KV cache retrieval\. In practice, settingδ<w\\delta<wreduces the effective KV I/O fromwwtow′=w−δw^\{\\prime\}=w\-\\delta, enabling fine\-grained control over the trade\-off between memory bandwidth and attention fidelity\. Further breaking down \([1](https://arxiv.org/html/2607.24788#S3.E1)\), the concentrated local attention computed by Sliding Window Attention \(SWA\) can be given as,

Oi−w−δ\+1SWA=∑j′=i−w−δ\+1iexp⁡\(qi​kj′T\)⋅vj′∑j′=i−w−δ\+1iexp⁡\(qi​kj′T\),O^\{\\text\{SWA\}\}\_\{i\-w\-\\delta\+1\}=\\frac\{\\sum\_\{j^\{\\prime\}=i\-w\-\\delta\+1\}^\{i\}\\exp\(q\_\{i\}k\_\{j^\{\\prime\}\}^\{T\}\)\\cdot v\_\{j^\{\\prime\}\}\}\{\\sum\_\{j^\{\\prime\}=i\-w\-\\delta\+1\}^\{i\}\\exp\(q\_\{i\}k\_\{j^\{\\prime\}\}^\{T\}\)\},\(2\)
Tokens evicted from the sliding window KV cache, together with theδ\\deltamost recent tokens within the window, are handled via linear attention rather than full softmax attention\. This design enables efficient global context modeling through a constant\-size recurrent state, eliminating the need to store or retrieve explicit key\-value pairs for tokens outside the active window\. The linear attention component can be formally defined as,

Oi−w−δLinear=∑j′=1i−w−δϕ​\(qi\)⋅ϕ​\(kj′\)T⋅vj′∑j′=1i−w−δϕ​\(qi\)⋅ϕ​\(kj′\)T,O\_\{i\-w\-\\delta\}^\{\\text\{Linear\}\}=\\frac\{\\sum\_\{j^\{\\prime\}=1\}^\{i\-w\-\\delta\}\\phi\(q\_\{i\}\)\\cdot\\phi\(k\_\{j^\{\\prime\}\}\)^\{T\}\\cdot v\_\{j^\{\\prime\}\}\}\{\\sum\_\{j^\{\\prime\}=1\}^\{i\-w\-\\delta\}\\phi\(q\_\{i\}\)\\cdot\\phi\(k\_\{j^\{\\prime\}\}\)^\{T\}\},\(3\)
whereϕ​\(⋅\)\\phi\(\\cdot\)denotes a non\-learnable separable kernel function that enables parameter\-efficient inference without requiring additional learned projections\. In our implementation, we adoptϕ​\(⋅\)=softmax​\(⋅\)\\phi\(\\cdot\)=\\text\{softmax\}\(\\cdot\)as the default kernel choice, following recent findings that softmax\-based feature maps preserve the spectral properties of the original attention mechanism more effectively than polynomial or RBF alternatives\. TheGlidearchitecture preserves the full representational capacity of softmax attention by combining two complementary components: adaptive local attention over the sliding window, which captures fine\-grained token interactions within a bounded context, and constant\-memory global attention via linear recurrence, which maintains a compressed summary of the entire prefix history\. The combination operator⊕\\oplusmerges these two attention outputs, weighted by their respective normalizing denominators, ensuring proper probability mass allocation across the full context and preventing either component from dominating the final output\. As the hybridization is generic,Glidesupports various formulations of the hybrid attention works, including BASED, LoLCats\[[1](https://arxiv.org/html/2607.24788#bib.bib4),[24](https://arxiv.org/html/2607.24788#bib.bib25)\]and Liger\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]\(see Appendix[\-A](https://arxiv.org/html/2607.24788#A0.SS1)\)\. This formulation guarantees two important boundary conditions: whenw→∞w\\to\\infty, the sliding window spans the entire sequence andGlideattention reduces to standard causal softmax attention, and whenw=0w=0, the mechanism collapses to pure linear attention with recurrent state updates, ensuring thatGlidegeneralizes both attention paradigms within a unified framework\. We formally define the \[Linear:Softmax\] ratio in our previous illustrations as aGlideδ\\delta\-allocation as below:

- •𝜹=\{0,0,…,0\}\\boldsymbol\{\\delta\}=\\\{0,0,\\ldots,0\\\}, i\.e\.,δℓ=0\\delta\_\{\\ell\}=0for allll: This corresponds to the baselineHybridconfiguration\. Specifically, within the sliding window of sizeww, all layers allocate0%0\\%to linear attention and100%100\\%to softmax attention\. The KV cache stores only softmax states\.
- •𝜹=\{w,w,…,w\}\\boldsymbol\{\\delta\}=\\\{w,w,\\ldots,w\\\}, i\.e\.,δl=w\\delta\_\{l\}=wfor allll: This corresponds to the baselineLinearconfiguration\. Within the sliding windowww, all layers allocate100%100\\%to linear attention and0%0\\%to softmax attention\. The KV cache stores only linear compressed states\.
- •𝜹=\{0,…,0\|α⋅w,…,α⋅w\|w,…,w\}\\boldsymbol\{\\delta\}=\\\{0,\\ldots,0~\|~\\alpha\\cdot w,\\ldots,\\alpha\\cdot w~\|~w,\\ldots,w\\\}: This corresponds to aGlideconfiguration discussed in the next section\. Here, different layer groups are assigned varying ratios of linear and softmax attention within the windowww\. The parameterα∈\[0,1\]\\alpha\\in\[0,1\]is defined as cache sparsity, where,δl=α⋅w\\delta\_\{l\}=\\alpha\\cdot wmeans the layer usesα⋅w\\alpha\\cdot w\(gliding\) tokens from cache for linear attention and remaining\(1−α\)​w\(1\-\\alpha\)wtokens for softmax attention incurring KV cache I/O overhead for the latter\. For instance,𝜹=\{0,…,0\|w/2,…,w/2\|w,…,w\}\\boldsymbol\{\\delta\}=\\\{0,\\ldots,0~\|~w/2,\\ldots,w/2~\|~w,\\ldots,w\\\}represents a three\-block configuration where early layers use pure softmax \(α=0\\alpha=0, no linear attention, no sparsity\), middle layers use equal mix \(α=0\.5\\alpha=0\.5, yielding50%50\\%linear and50%50\\%softmax\), and last layers use pure linear \(α=1\\alpha=1, no softmax, aggressive sparsity\)\. This configuration is denoted as\(0,w/2,w\)\(0,w/2,w\)\.

1

Input:Model

MM, window

ww
Output:Optimized model

M∗M^\{\*\}
2

3

\(B1,B2,B3\)←Split​\(M,3\)\(B\_\{1\},B\_\{2\},B\_\{3\}\)\\leftarrow\\textsc\{Split\}\(M,3\);

4Assign\(

B1,0B\_\{1\},0\);

5Assign\(

B3,wB\_\{3\},w\);

6

𝒮←\{0,w2,15​w16,w\}\\mathcal\{S\}\\leftarrow\\\{0,\\tfrac\{w\}\{2\},\\tfrac\{15w\}\{16\},w\\\};

7

8foreach*δ∈𝒮\\delta\\in\\mathcal\{S\}*do

9Assign\(

B2,δB\_\{2\},\\delta\);

10

11

12

δ∗←arg⁡maxδ∈𝒮⁡TaskAccEval​\(Mδ\)\\delta^\{\*\}\\leftarrow\\arg\\max\_\{\\delta\\in\\mathcal\{S\}\}\\textsc\{TaskAccEval\}\(M\_\{\\delta\}\);

13Assign\(

B2,δ∗B\_\{2\},\\delta^\{\*\}\);

14return

M∗M^\{\*\};

Algorithm 1Blockwiseδ\\deltaAllocationSub\-quadratic Efficiency\.Under autoregressive decoding, the resulting attention operatorOiGlideO^\{\\mathrm\{Glide\}\}\_\{i\}attains a per\-token time and space complexity of𝒪​\(\(δℓ\)​d\+d2\)\\mathcal\{O\}\(\(\\delta\_\{\\ell\}\)d\+d^\{2\}\), whereddis the head dimension andδℓ\\delta\_\{\\ell\}denotes the layer\-specific linearization parameter at layerℓ\\ell\. Sinceδℓ\\delta\_\{\\ell\}controls the extent of linearization within the window, larger values reduce the effective quadratic interaction region fromwwto\(w−δℓ\)\(w\-\\delta\_\{\\ell\}\), progressively replacing exact pairwise attention with linearized aggregation\. In the extreme caseδℓ=w\\delta\_\{\\ell\}=w, layerℓ\\ellbecomes fully linear with respect to the window, whereasδℓ=0\\delta\_\{\\ell\}=0recovers standard sliding\-window attention\. Consequently, for anLL\-layer model with layer\-wise allocations𝜹=\{δ1,δ2,…,δL\}\\boldsymbol\{\\delta\}=\\\{\\delta\_\{1\},\\delta\_\{2\},\\ldots,\\delta\_\{L\}\\\}, the total per\-token decoding complexity becomes𝒪​\(∑ℓ=1L\(w−δℓ\)​d\+L​d2\)\\mathcal\{O\}\\left\(\\sum\_\{\\ell=1\}^\{L\}\(w\-\\delta\_\{\\ell\}\)d\+Ld^\{2\}\\right\), which simplifies to𝒪​\(\(L​w−∑ℓ=1Lδℓ\)​d\+L​d2\)\\mathcal\{O\}\\left\(\(Lw\-\\sum\_\{\\ell=1\}^\{L\}\\delta\_\{\\ell\}\)d\+Ld^\{2\}\\right\)\. This formulation reveals thatGlideprovides a continuous and controllable interpolation between quadratic and linear regimes, enabling explicit accuracy–efficiency trade\-offs with sub\-quadratic decoding complexity\. Furthermore, leveraging non\-uniform allocation across layers, where deeper layers typically admit largerδℓ\\delta\_\{\\ell\}values due to reduced sensitivity to linearization,Glideachieves near\-minimal complexity in later layers while preserving representational fidelity in early layers where precise attention is critical\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/x7.png)Figure 5:Proposedδ\\delta\-Allocation Strategy forGlide\.Block\-wise allocation partitions the model into blocks of early, middle, and last layers based on our motivating Observation 1 \(in Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)\)\. Early layers retain full softmax attention \(δ=0\\delta=0\) where representational fidelity is critical, last layers use complete linearization \(δ=w\\delta=w\) for maximum efficiency, and middle layers exhibit glide attention withδ=α⋅w\\delta=\\alpha\\cdot wwhereα∈\[0,1\]\\alpha\\in\[0,1\], enabling fine\-grained control over the efficiency\-accuracy trade\-off\.TABLE I:Fine\-Tuned Accuracy Across Reasoning and Knowledge Benchmarks\.We evaluate Fine\-tuned GLIDE across standard benchmarks: PiQA, ARC\-Easy, ARC\-Challenge, HellaSwag, and WinoGrande in zero\-shot settings, with MMLU evaluated using 5\-shot prompting\. Each row corresponds to a block\-wiseδ\\delta\-configuration ranging from full softmax retention\(0,0,0\)\(0,0,0\)to complete linearization\(w,w,w\)\(w,w,w\)with the base window sizew=1204w=1204with model dimensiond=4096d=4096\. Avg\. denotes the mean across all six tasks\. Results show that moderate linearization retains competitive performance\.ModelConfigurationPiQA\(↑\\uparrow\)ARC\-E\(↑\\uparrow\)ARC\-C\(↑\\uparrow\)H\.Swag\(↑\\uparrow\)WinoG\.\(↑\\uparrow\)MMLU\(↑\\uparrow\)Avg\.\(↑\\uparrow\)KV I/O\(↓\\downarrow\)\(5\-shot\)\(MB/tok\)Llama\-3\-8BBaseline \(Vanilla SoftMax\)78\.8478\.8480\.9880\.9853\.9253\.9279\.1979\.1973\.3273\.3266\.6866\.6872\.1672\.1640004000Glide\(0,0,0\)≡Baseline \(Hybrid\)\(0,0,0\)\\equiv\\text\{Baseline \(Hybrid\)\}80\.2080\.2079\.8879\.8851\.7151\.7177\.4177\.4171\.1171\.1154\.3954\.3969\.1269\.12128128Glide\(0,0,w\)\(0,0,w\)78\.7378\.7378\.5878\.5849\.9149\.9176\.0376\.0370\.0970\.0953\.6853\.6867\.8467\.848888Glide\(0,w/2,w\)\(0,w/2,w\)78\.8978\.8978\.1178\.1150\.3450\.3475\.9375\.9370\.4870\.4852\.6252\.6267\.7367\.736464Glide\(0,15​w/16,w\)\(0,15w/16,w\)78\.2478\.2477\.6577\.6548\.5548\.5574\.5874\.5870\.4870\.4850\.9350\.9366\.7466\.744343Glide\(w,w,w\)≡Baseline \(Linear\)\(w,w,w\)\\equiv\\text\{Baseline \(Linear\)\}53\.1653\.1627\.5727\.5724\.2324\.2325\.2225\.2250\.5150\.5123\.0923\.0933\.9633\.960Mistral\-7BBaseline \(Vanilla SoftMax\)80\.7980\.7980\.2280\.2254\.0154\.0181\.1781\.1775\.3075\.3063\.7863\.7872\.5572\.5540004000Glide\(0,0,0\)≡Baseline \(Hybrid\)\(0,0,0\)\\equiv\\text\{Baseline \(Hybrid\)\}81\.3481\.3481\.7381\.7355\.1255\.1280\.7280\.7272\.6172\.6156\.3256\.3271\.3171\.31128128Glide\(0,0,w\)\(0,0,w\)80\.6380\.6379\.7679\.7651\.4551\.4578\.0078\.0073\.2473\.2455\.6955\.6969\.8069\.808888Glide\(0,w/2,w\)\(0,w/2,w\)80\.3080\.3080\.0580\.0551\.1951\.1977\.9877\.9873\.1673\.1653\.9353\.9369\.4469\.446464Glide\(0,15​w/16,w\)\(0,15w/16,w\)80\.4180\.4179\.9779\.9751\.2851\.2876\.0876\.0873\.1673\.1648\.4848\.4868\.2368\.234343Glide\(w,w,w\)≡Baseline \(Linear\)\(w,w,w\)\\equiv\\text\{Baseline \(Linear\)\}53\.2653\.2628\.4928\.4924\.1524\.1525\.9225\.9251\.1451\.1423\.0923\.0934\.3434\.340

### III\-CSensitivity\-Guided Layerwise Allocation

Glideattention supports fine\-grained layer\-wise allocation ofδℓ\\delta\_\{\\ell\}, enabling per\-layer control over the linearization–fidelity trade\-off to maximize throughput while preserving downstream task performance\. However, fully heterogeneous layer\-wise configurations can incur runtime overhead due to frequent kernel switching and reduced operator fusion opportunities\. To address this, we introduce block\-wiseδ\\deltaallocation, where contiguous groups ofBBlayers share a commonδ\\deltavalue, i\.e\.,δℓ=δb\\delta\_\{\\ell\}=\\delta\_\{b\}forℓ∈ℬb\\ell\\in\\mathcal\{B\}\_\{b\}\. This coarser granularity reduces configuration complexity while still capturing the depth\-dependent sensitivity patterns observed in Section[III\-A](https://arxiv.org/html/2607.24788#S3.SS1)namely, that early layers require smallerδ\\deltavalues for representational fidelity, while deeper layers tolerate aggressive linearization\. Block\-wise allocation also enables kernel\-aware pipelining and configuration reuse across layers, improving wall\-clock efficiency with minimal impact on model accuracy\.

We propose a simplified block\-wise allocation strategy described in Algorithm[1](https://arxiv.org/html/2607.24788#alg1)to reduce the search space\. Here we partition the model into blocks of early, middle, and last layers as illustrated in Fig\.[5](https://arxiv.org/html/2607.24788#S3.F5)andSPLITfunction in Algorithm[1](https://arxiv.org/html/2607.24788#alg1)\. The early block is assignedδb=1=0\\delta\_\{b=1\}=0\(full sliding window softmax attention\) and the last block is assignedδb=3=w\\delta\_\{b=3\}=w\(full linear attention\)\. The middle block allows us to allocateδb=2=α⋅w\\delta\_\{b=2\}=\\alpha\\cdot w, where,α∈\[0,1\]\\alpha\\in\[0,1\], is cache sparsity, and to explore the performance–efficiency tradeoffs across variousGlideconfigurations, we restrictα\\alphasuch thatδ∈\{0,w/2,15​w/16,w\}\\delta\\in\\\{0,\\,w/2,\\,15w/16,\\,w\\\}within the search space described in Algorithm[1](https://arxiv.org/html/2607.24788#alg1)\.

Speedup\.For three equally\-sized blocks, each comprisingL/3L/3layers, we achieve a theoretical computational speedup of

S=L​w​d\+L​d2\(L​w​d/3\+L​w​\(1−α\)​d/3\+L​d2\)=1\+1\+α\(2\+3​d/w\)−αS=\\frac\{Lwd\+Ld^\{2\}\}\{\(Lwd/3\+Lw\(1\-\\alpha\)d/3\+Ld^\{2\}\)\}=1\+\\frac\{1\+\\alpha\}\{\(2\+3d/w\)\-\\alpha\}for per\-token decoding with respect to baseline hybrid model, which is a specific configuration,Glide\(0,0,0\) withα=0\\alpha=0\. The theoretical speedupSSis highly context\-dependent, scaling up to a3×3\\timesmaximum in the long\-context regime \(w≫dw\\gg d\) where softmax attention acts as the primary computational bottleneck and is highly responsive to cache sparsityα\\alpha\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/x8.png)\(\(a\)\)
![Refer to caption](https://arxiv.org/html/2607.24788v1/x9.png)\(\(b\)\)

Figure 6:Performance vs\. KV Cache I/O Trade\-off\.Average task accuracy versus KV cache I/O per token for \(a\) Llama\-3\-8B and \(b\) Mistral\-7B\. Markers indicate: baseline \(▲\\blacktriangle\), uniform \(∙\\bullet\), non\-uniform \(■\\blacksquare\), and non\-uniform fine\-tuned \(◆\\blacklozenge\)\. Fine\-tuning recovers 6–8 accuracy points, achieving 96% baseline performance at 45× lower memory bandwidth while non\-uniform allocations consistently outperform uniform alternatives\.

## IVExperiments

This section validates ability ofGlideto exploit layer\-wise linearization for reducing KV cache I/O overhead and accelerating inference while preserving downstream task accuracy\. We systematically evaluate computational efficiency through two primary metrics: \(1\) KV cache I/O overhead, quantified as the required total KV cache movement, and \(2\) end\-to\-end inference latency, measured across sequence lengths\. Task performance is assessed on six standard reasoning and knowledge benchmarks from the language model evaluation harness: PiQA \(physical commonsense\), ARC\-Easy and ARC\-Challenge \(science reasoning\), HellaSwag \(commonsense inference\), WinoGrande \(coreference resolution\), and MMLU \(multitask language understanding\)\. To address potential accuracy degradation under aggressive linearization, we apply LoRA\-based parameter\-efficient fine\-tuning \(PEFT\) toGlideconfigurations, enabling recovery of performance while maintaining the reduced KV cache footprint of linearized layers\. Our experimental analysis addresses the following research question:CanGlide’s per\-layerδ\\delta\-allocation mechanism reduce KV cache I/O and computational cost while preserving downstream task accuracy through strategic layer\-wise linearization?

### IV\-AExperimental Setup

Implementation Configurations\.Our implementation builds upon Liger and Lolcats\[[15](https://arxiv.org/html/2607.24788#bib.bib27),[24](https://arxiv.org/html/2607.24788#bib.bib25)\]and leverages PyTorch’s FlexAttention with Flash Attention Backend\[[7](https://arxiv.org/html/2607.24788#bib.bib29)\]to realize an adaptive sliding\-window attention mechanism\. We conducted our experiments on a single NVIDIA Grace Hopper \(GH200 Superchip: Grace CPU \+ Hopper GPU\) 120GB, and we fine\-tuned the model using LoRA with a rank of 8 and a scaling coefficient of 8\. Training is performed for two epochs on 100K cleaned Alpaca instruction samples \(≈\\approx0\.02B tokens\) to support gate\-recurrent linearization\. All sequences are set to a maximum length of 1024 tokens in the LM eval benchmark framework\. Optimization is carried out with a micro\-batch size of 1, and an effective batch size of 8 is realized via gradient accumulation, consistent with the LoLCATs training setup\[[24](https://arxiv.org/html/2607.24788#bib.bib25)\]with base window sizew=1024w=1024\.

Models & Datasets\.To systematically quantify the layer\-wise sensitivity ofGlide, we perform a controlled paired\-replacement study across varyingδ\\deltaconfigurations, isolating the effect of selective linearization at each transformer layer\. Fine\-tuning is conducted for two epochs on 50,000 cleaned Alpaca\[[19](https://arxiv.org/html/2607.24788#bib.bib30)\]instruction\-following samples to ensure stable adaptation under limited supervision\. To demonstrate generality, we integrateGlideinto strong, widely adopted backbone models such as LLaMA 3 8B\[[9](https://arxiv.org/html/2607.24788#bib.bib31)\]and Mistral 7B\[[13](https://arxiv.org/html/2607.24788#bib.bib32)\]with baseline model dimensiond=4096d=4096, which serve as competitive baselines for comparative evaluation\. Following the sensitivity hierarchy illustrated in Fig\.[2\(b\)](https://arxiv.org/html/2607.24788#S2.F2.sf2), we partition both models \(L=32L=32, 0\-indexed\) into early \(layers11–1111,δ=0\\delta=0\), middle \(layers1212–2121,δ=α⋅w\\delta=\\alpha\\cdot w\), and late \(layers2222–3232,δ=w\\delta=w\) blocks, assigning conservative softmax budgets where sensitivity is highest and aggressive linearization where redundancy is established\.

### IV\-BResults and Discussion

To validate the efficiency\-performance trade\-off enabled byGlide, we evaluate the framework on six standard reasoning benchmarks: PiQA, ARC\-Easy, ARC\-Challenge, HellaSwag, WinoGrande, and MMLU\. We compareGlideconfigurations against three baselines: vanilla softmax attention, hybrid attention, and pure linear attention\. Efficiency gains are measured through two metrics: end\-to\-end latency and KV cache I/O volume across varying sequence lengths\. These results demonstrate the Pareto\-optimal configurations enabled byGlideframework\.

Recovering Performance through PEFT\.WhenGlideconfigurations are deployed as drop\-in replacements for standard attention with various non\-uniformδ\\delta\-allocations, we observe a degradation of5%5\\%to10%10\\%in downstream task performance relative to the baseline hybrid model\. This performance gap is visualized in Fig\.[6](https://arxiv.org/html/2607.24788#S3.F6)\(a\) and \(b\) as the vertical distance between the\(0,0,0\)\(0,0,0\)configuration \(∙\\bullet, baseline hybrid with full softmax in all blocks\) and the\(0,0,w\)\(0,0,w\)configuration \(■\\blacksquare, aggressive linearization in later blocks\)\. While the vanilla softmax baseline \(▲\\blacktriangle\) achieves the highest accuracy of approximately72%72\\%on both models, it incurs a prohibitive KV cache I/O cost of 4000 MB/token\. In contrast, the\(0,0,0\)\(0,0,0\)hybrid baseline achieves6969–71%71\\%accuracy at only 128 MB/token, representing a31×31\\timesreduction in memory bandwidth with minimal performance loss\. To recover the performance degradation observed in more aggressive linearization schemes while maintaining efficiency gains, we apply LoRA\-based parameter\-efficient fine\-tuning \(PEFT\) to theGlideconfigurations\. Fine\-tuned models are indicated by dark green diamond markers \(◆\\blacklozenge\) in Fig\.[6](https://arxiv.org/html/2607.24788#S3.F6)\. The results demonstrate that fine\-tuning substantially closes the accuracy gap: the fine\-tuned\(0,0,w\)\(0,0,w\)configuration recovers66to88accuracy points, reaching approximately6868–70%70\\%accuracy \(96% of baseline performance\) while operating at only 88 MB/token—a45×45\\timesreduction compared to vanilla softmax\. Similarly, the\(0,15​w/16,w\)\(0,15w/16,w\)configuration achieves6666–68%68\\%accuracy at just 43 MB/token, nearly93×93\\timesmore efficient than the vanilla baseline\.This pattern holds across all non\-uniformδ\\delta\-allocations, with fine\-tuned variants \(◆\\blacklozenge\) consistently outperforming their zero\-shot counterparts \(■\\blacksquare\) at equivalent KV cache I/O levels, demonstratingGlide’s ability to unlock a rich Pareto frontier of efficiency\-performance trade\-offs\.

Performance Trade\-offs AcrossGlideConfigurations\.Table[I](https://arxiv.org/html/2607.24788#S3.T1)presents fine\-tuned accuracy results for Llama\-3\-8B and Mistral\-7B across six reasoning and knowledge benchmarks\. The vanilla softmax baseline achieves 72\.16% average accuracy on Llama\-3\-8B and 72\.55% on Mistral\-7B, but requires 4000 MB/token KV cache I/O\. The hybrid baseline configuration\(0,0,0\)≡baseline \(Hybrid\)\(0,0,0\)\\equiv\\text\{baseline \(Hybrid\)\}achieves 69\.12% and 71\.31% respectively at only 128 MB/token—a31×31\\timesreduction in memory bandwidth while retaining 96–98% of baseline performance\. Progressive linearization reveals distinct per\-layer sensitivity patterns\. For Llama\-3\-8B, the\(0,0,w\)\(0,0,w\)configuration achieves 67\.84% accuracy at 88 MB/token\. Extending linearization to intermediate layers with\(0,w/2,w\)\(0,w/2,w\)maintains nearly identical performance at 67\.73%, indicating that mid\-layer linearization incurs negligible additional cost once the final block is linearized\. The aggressive\(0,15​w/16,w\)\(0,15w/16,w\)configuration yields 66\.74% at just 43 MB/token—a93×93\\timesreduction compared to vanilla softmax while preserving 92% of baseline accuracy\. However, full linearization\(w,w,w\)≡baseline \(Linear\)\(w,w,w\)\\equiv\\text\{baseline~\(Linear\)\}degrades catastrophically to 33\.96%, with severe failures on reasoning tasks such as HellaSwag \(25\.22%\) and ARC\-C \(24\.23%\)\. Mistral\-7B exhibits consistent trends with stronger absolute performance\. Progressive linearization through\(0,0,w\)\(0,0,w\),\(0,w/2,w\)\(0,w/2,w\), and\(0,15​w/16,w\)\(0,15w/16,w\)yields 69\.80%, 69\.44%, and 68\.23% respectively, demonstrating graceful degradation of approximately 1% per linearization step\. Full linearization again proves detrimental at 34\.34%, confirming that complete softmax elimination severely impacts model quality across architectures\.These results demonstrate that moderate linearization configurations achieve4545–93×93\\timesefficiency gains while preserving 90–96% of baseline accuracy after fine\-tuning\.

TABLE II:Block\-wiseδ\\delta\-Allocation Strategies\.End\-to\-end accumulated latency over sequence length \(S\) underGlide’s allocation schemes for Llama\-3\-8B and Mistral\-7B with base window sizew=1024w=1024and model dimensiond=4096d=4096\. Configurations exploit layer\-wise linearization tolerance, concentrating softmax in early layers\. Asδ\\deltaincreases, memory bandwidth decreases and inference accelerates\.Llama\-3\-8BConfigurationE2E Latency \(s\) over Seq\. Length4K8K16K32KBaseline \(Vanilla Softmax\)264\.63264\.63765\.92765\.92–\(OOM\)1–\(OOM\)1Glide\(0,0,0\)≡Baseline​\(Hybrid\)\(0,0,0\)\\equiv\\text\{Baseline\}~\(\\text\{Hybrid\}\)453\.68453\.68919\.66919\.661850\.921850\.923705\.723705\.72Glide\(0,0,w\)\(0,0,w\)227\.71227\.71453\.08453\.08973\.58973\.582638\.352638\.35Glide\(0,w/2,w\)\(0,w/2,w\)223\.58223\.58446\.23446\.23966\.50966\.502637\.422637\.42Glide\(0,15​w/16,w\)\(0,15w/16,w\)225\.47225\.47449\.67449\.67972\.20972\.202637\.542637\.54Glide\(w,w,w\)≡Baseline​\(Linear\)\(w,w,w\)\\equiv\\text\{Baseline\}~\(\\text\{Linear\}\)205\.12205\.12409\.11409\.11818\.59818\.591640\.751640\.75Mistral\-7BConfigurationE2E Latency \(s\) over Seq\. Length4K8K16K32KBaseline \(Vanilla Softmax\)305\.26305\.261123\.571123\.57–\(OOM\)1–\(OOM\)1Glide\(0,0,0\)≡Baseline​\(Hybrid\)\(0,0,0\)\\equiv\\text\{Baseline\}~\(\\text\{Hybrid\}\)707\.25707\.251396\.801396\.802789\.582789\.585589\.725589\.72Glide\(0,0,w\)\(0,0,w\)724\.06724\.061448\.751448\.752998\.592998\.595856\.515856\.51Glide\(0,w/2,w\)\(0,w/2,w\)723\.66723\.661476\.851476\.852854\.072854\.075770\.935770\.93Glide\(0,15​w/16,w\)\(0,15w/16,w\)679\.15679\.151354\.911354\.912711\.122711\.125444\.565444\.56Glide\(w,w,w\)≡Baseline​\(Linear\)\(w,w,w\)\\equiv\\text\{Baseline\}~\(\\text\{Linear\}\)710\.38710\.381415\.441415\.442824\.712824\.715652\.725652\.72
- 1–\(OOM\) denotes Out\-of\-Memory; longer sequence lengths are not supported\.

End\-To\-End Accumulated Latency Comparison\.Table[II](https://arxiv.org/html/2607.24788#S4.T2)presents end\-to\-end latency measurements for Llama\-3\-8B and Mistral\-7B across sequence lengths from 4K to 32K tokens\. Llama backbone, the vanilla softmax baseline achieves264\.63264\.63s at 4K but encounters out\-of\-memory failures beyond 8K tokens\. The\(0,0,0\)≡baseline \(hybrid\)\(0,0,0\)\\equiv\\text\{baseline~\(hybrid\)\}eliminates these constraints, scaling to3705\.723705\.72s at 32K tokens and enabling long\-context inference\.Progressive non\-uniform linearization yields substantial efficiency gains\. The\(0,0,w\)\(0,0,w\)configuration achieves227\.71227\.71s at 4K \(a 2\.0× speedup over the hybrid baseline\) and scales to2638\.352638\.35s at 32K\. Notably, configurations\(0,0,w\)\(0,0,w\),\(0,w/2,w\)\(0,w/2,w\), and\(0,15​w/16,w\)\(0,15w/16,w\)exhibit nearly identical latencies at 32K tokens, completing within26372637s to26382638s despite differingδ2\\delta\_\{2\}allocations\. This iso\-latency behavior arises because memory bandwidth, not computation, dominates the bottleneck\. The key differentiator becomes KV cache I/O: these configurations require 88, 64, and4343MB/token respectively, representing2×2\\timesto3×3\\timesmemory bandwidth reductions\. The fully linearized\(w,w,w\)≡baseline \(Linear\)\(w,w,w\)\\equiv\\text\{baseline~\(Linear\)\}achieves the lowest latency \(205\.12205\.12s at 4K,1640\.751640\.75s at 32K\) but suffers catastrophic accuracy degradation to33\.96%33\.96\\%\. Additionally, the Mistral\-7B backbone exhibits similar behavior underGlide, where accumulated end\-to\-end latency shows only minimal improvement acrossδ\\delta\-allocation configurations\. We attribute this to hardware\-level overheads inherent to hybrid attention execution, including warp divergence, irregular data access patterns, and kernel launch overhead\.Glideconfigurations therefore represent the optimal balance, achieving1\.4×1\.4\\timesto2×2\\timeslatency improvements and 2 to 3× memory bandwidth reductions while preserving92%92\\%to96%96\\%of baseline accuracy\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/x10.png)Figure 7:Accumulated KV I/O vs\. Sequence Length\.Glideconfigurations achieve sub\-linear KV cache growth compared to baseline’s linear scaling\. Progressive linearization from\(0,0,0\)\(0,0,0\)to\(0,15​w/16,w\)\(0,15w/16,w\)reduces memory I/O while maintaining accuracy, with\(0,15​w/16,w\)\(0,15w/16,w\)achieving5\.25\.2GB at128128K tokens versus baseline’s16\.316\.3GB\.KV I/O Efficient Reasoning Scalability\.A key challenge in scaling reasoning models to long contexts is the linear growth of KV cache I/O, which shifts inference from being compute\-bound to memory\-bandwidth\-bound\. As shown in Fig\.[7](https://arxiv.org/html/2607.24788#S4.F7), baseline models accumulate over 17\.5 GB of KV I/O at 128k tokens, creating a throughput bottleneck where each generated token must retrieve an increasingly large volume of cached states\. This memory pressure severely limits the number of concurrent users a single accelerator can serve, as KV cache consumption scales linearly with both sequence length and batch size\.Glidemitigates this by reducing aggregate KV retrieval through layer\-wise hybrid attention\. Non\-uniform configurations bring KV I/O down to 5\.5\-11\.0 GB at 128k tokens, a reduction of up to 3\.2×\\timescompared to baseline\. This reduction directly enables higher serving capacity: by compressing per\-request memory footprint without major performance degradation \(retaining 92\-96% accuracy\),Glideallows deployments to serve 2\-3×\\timesmore concurrent users on the same hardware\.The efficiency gains compound at scale, as reduced memory bandwidth contention improves overall throughput and reduces tail latency under load, making long\-context inference practically deployable in multi\-user serving environments\.

## VLimitations & Future Works

WhileGlidedemonstrates substantial improvements in decoding efficiency through selective KV cache retrieval and layer\-wise hybrid attention, several limitations remain\. Our experiments reveal that linear attention components exhibit information dilution as sequence length increases, where the compressed recurrent state progressively loses fine\-grained token\-level distinctions\. Although the sliding\-window softmax layers partially compensate by preserving local precision, this dilution contributes to the observed accuracy\-efficiency trade\-off on long\-context workloads\. Investigating improved state compression mechanisms or hybrid state management strategies that mitigate dilution while preserving memory efficiency remains an important direction\. From a systems perspective, the current implementation relies on attention kernels not specifically optimized for hybrid execution patterns\. We observed that smaller window sizes \(w≤128w\\leq 128\), despite reducing theoretical FLOPs, incur kernel launch and memory scheduling overhead that diminishes practical latency gains on GPU architectures\. Fused kernels co\-optimized for sliding\-window softmax and linear recurrent components, potentially leveraging persistent thread blocks or warp\-specialized execution, could unlock further performance improvements\. Finally, extendingGlideto distributed inference scenarios with tensor or pipeline parallelism, and evaluating its impact on reasoning\-intensive workloads where layer\-wise attention sensitivity may differ, represent promising avenues for future work\.

## VIConclusion

This paper introducesGlide\(Guided Layerwise Hybrid Attention for Efficient Decoding\), an adaptive framework that enables efficient long\-context generation in large language models while preserving downstream task accuracy\.Glideis built on a key empirical observation: transformer layers exhibit depth\-dependent sensitivity to attention linearization, with early layers requiring high\-fidelity softmax attention for representational quality, while deeper layers tolerate aggressive linearization with minimal performance impact\. Leveraging this insight,Glideselectively allocates softmax attention non\-uniformly across layers, concentrating expensive full\-attention computation where most critical while replacing approximation\-tolerant layers with memory\-efficient linear recurrence\. This layer\-wise allocation strategy substantially reduces KV cache I/O during autoregressive decoding, the primary bottleneck in long\-context generation, achieving up to 62×\\timesreduction in memory bandwidth while retaining 92\-96% of baseline accuracy across reasoning benchmarks\. Beyond single\-request efficiency,Glide’s compressed memory footprint directly translates to improved serving capacity: by reducing per\-request KV cache overhead, the framework enables resource\-constrained accelerators to handle 2\-3×\\timesmore concurrent users compared to vanilla softmax attention, making scalable multi\-user deployment of long\-context models practically feasible in memory\-limited environments\.

## References

- \[1\]S\. Arora, S\. Eyuboglu, M\. Zhang, A\. Timalsina, S\. Alberti, D\. Zinsley, J\. Zou, A\. Rudra, and C\. Ré\(2024\)Simple linear attention language models balance the recall\-throughput tradeoff\.arXiv preprint arXiv:2402\.18668\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.18668)Cited by:[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p3.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p1.7),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p6.6),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[2\]I\. Beltagy, M\. E\. Peters, and A\. Cohan\(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.External Links:[Link](https://doi.org/10.48550/arXiv.2004.05150)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p1.3),[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p1.7)\.
- \[3\]Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.02069)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p5.1),[§I](https://arxiv.org/html/2607.24788#S1.p2.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3)\.
- \[4\]J\. Chen, S\. Peng, D\. Luo, F\. Yang, R\. Wu, F\. Li, and X\. Chen\(2025\)Edgeinfinite: a memory\-efficient infinite\-context transformer for edge devices\.arXiv preprint arXiv:2503\.22196\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.22196)Cited by:[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p3.1),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[5\]J\. Dass, S\. Wu, H\. Shi, C\. Li, Z\. Ye, Z\. Wang, and Y\. Lin\(2023\)Vitality: unifying low\-rank and sparse approximation for vision transformer acceleration with a linear taylor attention\.In2023 IEEE International Symposium on High\-Performance Computer Architecture \(HPCA\),pp\. 415–428\.External Links:[Link](https://doi.org/10.48550/arXiv.2211.05109)Cited by:[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[6\]H\. Dong, X\. Yang, Z\. Zhang, Z\. Wang, Y\. Chi, and B\. Chen\(2024\)Get more with less: synthesizing recurrence with kv cache compression for efficient llm inference\.arXiv preprint arXiv:2402\.09398\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.09398)Cited by:[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p3.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[7\]J\. Dong, B\. FENG, D\. Guessous, Y\. Liang, and H\. He\(2025\)FlexAttention: a programming model for generating fused attention variants\.\.InEighth Conference on Machine Learning and Systems,External Links:[Link](https://doi.org/10.48550/arXiv.2412.05496)Cited by:[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p1.2)\.
- \[8\]Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou\(2024\)Ada\-kv: optimizing kv cache eviction by adaptive budget allocation for efficient llm inference\.arXiv preprint arXiv:2407\.11550\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.11550)Cited by:[§I](https://arxiv.org/html/2607.24788#S1.p1.1),[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p2.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3)\.
- \[9\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma\(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://doi.org/10.48550/arXiv.2407.21783)Cited by:[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p2.12)\.
- \[10\]D\. Han, Y\. Pu, Z\. Xia, Y\. Han, X\. Pan, X\. Li, J\. Lu, S\. Song, and G\. Huang\(2024\)Bridging the divide: reconsidering softmax and linear attention\.arXiv\.org\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2412.06590),[Link](https://doi.org/10.48550/arXiv.2412.06590)Cited by:[§II\-B](https://arxiv.org/html/2607.24788#S2.SS2.p2.7.2)\.
- \[11\]D\. Han, Y\. Pu, Z\. Xia, Y\. Han, X\. Pan, X\. Li, J\. Lu, S\. Song, and G\. Huang\(2024\)Bridging the divide: reconsidering softmax and linear attention\.Advances in Neural Information Processing Systems37,pp\. 79221–79245\.External Links:[Link](https://doi.org/10.48550/arXiv.2412.06590)Cited by:[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[12\]E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://doi.org/10.48550/arXiv.2106.09685)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p6.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[13\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed\(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://doi.org/10.48550/arXiv.2310.06825)Cited by:[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p2.12)\.
- \[14\]A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret\(2020\)Transformers are rnns: fast autoregressive transformers with linear attention\.InInternational conference on machine learning,pp\. 5156–5165\.External Links:[Link](https://doi.org/10.48550/arXiv.2006.16236)Cited by:[§II\-B](https://arxiv.org/html/2607.24788#S2.SS2.p2.7),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p6.5)\.
- \[15\]D\. Lan, W\. Sun, J\. Hu, J\. Du, and Y\. Cheng\(2025\)Liger: linearizing large language models to gated recurrent structures\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://doi.org/10.48550/arXiv.2503.01496)Cited by:[§I](https://arxiv.org/html/2607.24788#S1.p2.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3),[§III\-A](https://arxiv.org/html/2607.24788#S3.SS1.p2.3),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p1.7),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p6.6),[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p1.2),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[16\]H\. LI, Y\. Li, A\. Tian, T\. Tang, Z\. Xu, X\. Chen, N\. HU, W\. Dong, L\. Qing, and L\. Chen\(2025\)A survey on large language model acceleration based on KV cache management\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://doi.org/10.48550/arXiv.2412.19442)Cited by:[§I](https://arxiv.org/html/2607.24788#S1.p1.1)\.
- \[17\]Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen\(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.14469)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p5.1),[§I](https://arxiv.org/html/2607.24788#S1.p2.1)\.
- \[18\]T\. Munkhdalai, M\. Faruqui, and S\. Gopal\(2024\)Leave no context behind: efficient infinite context transformers with infini\-attention\.arXiv preprint arXiv:2404\.07143101\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.07143)Cited by:[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p3.1),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[19\]R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Stanford alpaca: an instruction\-following llama model\.GitHub\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p2.12)\.
- \[20\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.Advances in neural information processing systems30\.External Links:[Link](https://doi.org/10.48550/arXiv.1706.03762)Cited by:[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p4.10)\.
- \[21\]G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis\(2024\)Efficient streaming language models with attention sinks, 2024\.URL https://arxiv\. org/abs/2309\.174531\.External Links:[Link](https://doi.org/10.48550/arXiv.2309.17453)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p5.1)\.
- \[22\]Y\. Xu, C\. Li, D\. Li, X\. Sheng, F\. Jiang, L\. Tian, and E\. Barsoum\(2024\)QT\-vit: improving linear attention in vit with quadratic taylor expansion\.Advances in Neural Information Processing Systems37,pp\. 83048–83067\.External Links:[Link](https://doi.org/10.52202/079017-2642)Cited by:[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[23\]Y\. Xu, S\. Nag, D\. Li, L\. Tian, and E\. Barsoum\(2025\)MSWA: refining local attention with multi\-scalewindow attention\.arXiv preprint arXiv:2501\.01039\.External Links:[Link](https://doi.org/10.48550/arXiv.2501.01039)Cited by:[§II\-C](https://arxiv.org/html/2607.24788#S2.SS3.p2.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3)\.
- \[24\]M\. Zhang, S\. Arora, R\. Chalamala, B\. F\. Spector, A\. Wu, K\. Ramesh, A\. Singhal, and C\. Re\(2025\)LoLCATs: on low\-rank linearizing of large language models\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://doi.org/10.48550/arXiv.2410.10254)Cited by:[§I](https://arxiv.org/html/2607.24788#S1.p2.1),[§II\-D](https://arxiv.org/html/2607.24788#S2.SS4.p1.3),[§III\-A](https://arxiv.org/html/2607.24788#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p6.6),[§IV\-A](https://arxiv.org/html/2607.24788#S4.SS1.p1.2),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[25\]M\. Zhang, K\. Bhatia, H\. Kumbong, and C\. Ré\(2024\)The hedgehog & the porcupine: expressive linear attentions with softmax mimicry, 2024b\.URL https://arxiv\. org/abs/2402\.04347\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.04347)Cited by:[§II\-B](https://arxiv.org/html/2607.24788#S2.SS2.p2.7.2),[§III\-B](https://arxiv.org/html/2607.24788#S3.SS2.p1.7),[GLIDE: Guided Layerwise Hybrid Attention for Efficient LLM Inference](https://arxiv.org/html/2607.24788#p10.8)\.
- \[26\]Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.External Links:[Link](https://doi.org/10.48550/arXiv.2306.14048)Cited by:[§\-A](https://arxiv.org/html/2607.24788#A0.SS1.p5.1),[§I](https://arxiv.org/html/2607.24788#S1.p2.1)\.

Overview of Attention\.Consider the next token prediction using the current query𝐪i∈ℝd\\mathbf\{q\}\_\{i\}\\in\\mathbb\{R\}^\{d\}, whered=dmodel/Hd=d\_\{\\text\{model\}\}/HandHHdenote the number of heads present in the single multi\-head attention layer anddmodeld\_\{\\text\{model\}\}is the overall dimension of the model\. The attention scoreαi​j\\alpha\_\{ij\}captures the similarity between the current query𝐪i\\mathbf\{q\}\_\{i\}and the keys𝐤j∈ℝd\\mathbf\{k\}\_\{j\}\\in\\mathbb\{R\}^\{d\}, wherej∈\[1,i\]j\\in\[1,i\]defined as,

αi​j=sim​\(𝐪i,𝐤j\)∑j′=1isim​\(𝐪i,𝐤j′\),\\alpha\_\{ij\}=\\frac\{\\text\{sim\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j\}\)\}\{\\sum\_\{j^\{\\prime\}=1\}^\{i\}\\text\{sim\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j^\{\\prime\}\}\)\},\(4\)
Oi=∑j=1iαi​j​𝐯j,O\_\{i\}=\\sum\_\{j=1\}^\{i\}\\alpha\_\{ij\}\\mathbf\{v\}\_\{j\},\(5\)
where,sim​\(𝐪i,𝐤j\)=exp​\(𝐪i,𝐤jT/d\)\\text\{sim\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j\}\)=\\text\{exp\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j\}^\{T\}/\\sqrt\{d\}\)corresponds to softmax attention\[[20](https://arxiv.org/html/2607.24788#bib.bib1)\]\. The output of the attention blockOiO\_\{i\}defined in \([4](https://arxiv.org/html/2607.24788#A0.E4)\) is the attention\-weighted aggregation of value vectors𝐯j∈ℝd\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{d\}\. Further, the computed outputOiO\_\{i\}is transformed linearly by dot product with the projection matrixWo∈ℝn×dW\_\{o\}\\in\\mathbb\{R\}^\{n\\times d\}, wherenndenotes the context length of the model\. According to \([4](https://arxiv.org/html/2607.24788#A0.E4)\), predictingi\+1t​hi\+1^\{th\}token depends on the set of previously generated key & value pairs\{𝐤j,𝐯j∣j∈\[1,i\]\}\\\{\\,\\mathbf\{k\}\_\{j\},\\mathbf\{v\}\_\{j\}\\mid j\\in\[1,i\]\\,\\\}and computing those matrices at timettcould require expensiveO​\(n2\)O\(n^\{2\}\)computational cost\.

KV Cache\.The overhead involved in computing the past key and value pairs can be mitigated by caching each generated key𝐤𝐢\\mathbf\{k\_\{i\}\}and𝐯𝐢\\mathbf\{v\_\{i\}\}at every iteration\. KV caching involves two computational stages namely prefill and decode\.Prefill Stage:The stage during which the model computes self\-attention over all prompts, and populates the key–value \(KV\) cache by storing the key𝐤𝐢\\mathbf\{k\_\{i\}\}and value𝐯𝐢\\mathbf\{v\_\{i\}\}pairs for each token\.Decode Stage:The stage in which the model generates tokens auto\-regressively; at each step, self\-attention is computed using the previously stored key–value \(KV\) cache, and the key and value corresponding to the newly generated token are appended to the cache\. Despite minimizing redundant computes, the memory required for the KV cache scales linearly with the context lengthnn, and at each token prediction, attention incurs an expensiveO​\(n\)O\(n\)memory footprint\.

Linear Attention\.To understand how global context is preserved in retention\-based SWA, we formalize linear attention, which underlies these methods\. Considering \([4](https://arxiv.org/html/2607.24788#A0.E4)\), the similarity functions​i​m​\(⋅\)sim\(\\cdot\)in softmax attention can be approximated using separable kernel functionsϕ\\phi\[[14](https://arxiv.org/html/2607.24788#bib.bib21)\], applied independently to the query𝐪i\\mathbf\{q\}\_\{i\}and key𝐤j\\mathbf\{k\}\_\{j\}\. Under this formulation, the approximated attention scoreαi​j\\alpha\_\{ij\}can be expressed as,

sim\(𝐪i\.𝐤j\)∑j=1isim\(𝐪i\.𝐤j\)≈ϕ​\(𝐪i\)\.ϕ​\(𝐤j\)T∑j=1iϕ​\(𝐪i\)\.ϕ​\(𝐤j\)T,\\frac\{\\text\{sim\}\(\\mathbf\{q\}\_\{i\}\.\\mathbf\{k\}\_\{j\}\)\}\{\\sum\_\{j=1\}^\{i\}\\text\{sim\}\(\\mathbf\{q\}\_\{i\}\.\\mathbf\{k\}\_\{j\}\)\}\\approx\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\}\{\\sum\_\{j=1\}^\{i\}\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\},\(6\)
on computing theOiO\_\{i\}, the matrices can be rearranged from∑j=iiϕ​\(𝐪i\)\.ϕ​\(𝐤j\)T​𝐯j\\sum\_\{j=i\}^\{i\}\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\\mathbf\{v\}\_\{j\}toϕ​\(𝐪i\)\.\(∑j=iiϕ​\(𝐤j\)T​𝐯j\)\\phi\(\\mathbf\{q\}\_\{i\}\)\.\(\\sum\_\{j=i\}^\{i\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\\mathbf\{v\}\_\{j\}\)resulting in the constant recurrent memory ofd×dd\\times dand it can be expressed as,

OiLinear=ϕ\(𝐪i\)\.\(∑j=1iϕ\(𝐤j\)T\.𝐯j\)ϕ​\(𝐪i\)\.∑j=1iϕ​\(𝐤j\)T=ϕ​\(𝐪i\)\.𝐒iϕ​\(𝐪i\)\.𝐙i,O\_\{i\}^\{\\text\{Linear\}\}=\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\.\(\\sum\_\{j=1\}^\{i\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\.\\mathbf\{v\}\_\{j\}\)\}\{\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\sum\_\{j=1\}^\{i\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{T\}\}=\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\mathbf\{S\}\_\{i\}\}\{\\phi\(\\mathbf\{q\}\_\{i\}\)\.\\mathbf\{Z\}\_\{i\}\},\(7\)
where the recurrent state𝐒i\\mathbf\{S\}\_\{i\}is updated as𝐒i=𝐒i−1\+ϕ​\(𝐤j\)⊤​𝐯j∈ℝd×d\\mathbf\{S\}\_\{i\}=\\mathbf\{S\}\_\{i\-1\}\+\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}\\mathbf\{v\}\_\{j\}\\in\\mathbb\{R\}^\{d\\times d\}, and the corresponding normalization factor is updated as𝐙i=𝐙i−1\+∑j=1iϕ​\(𝐤j\)⊤∈ℝd\\mathbf\{Z\}\_\{i\}=\\mathbf\{Z\}\_\{i\-1\}\+\\sum\_\{j=1\}^\{i\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}\\in\\mathbb\{R\}^\{d\}\. Unlike softmax attention, linear attention incurs only anO​\(d\)O\(d\)data movement cost, and its memory usage remains constant with respect to the context lengthnn\. Vanilla softmax attention can be approximated using linear attention with different choices ofϕ\\phi\-kernels\. Commonly usedϕ\\phi\-kernels includeELU\+1\\mathrm\{ELU\}\+1\(EdgeInfinite\[[4](https://arxiv.org/html/2607.24788#bib.bib15)\], Infini\-Transformer\[[18](https://arxiv.org/html/2607.24788#bib.bib14)\]\), MLP\-based kernels \(LESS\[[6](https://arxiv.org/html/2607.24788#bib.bib10)\], Hedgehog\[[25](https://arxiv.org/html/2607.24788#bib.bib23)\], LoLCats\[[24](https://arxiv.org/html/2607.24788#bib.bib25)\], LOLA\[[12](https://arxiv.org/html/2607.24788#bib.bib26)\]\), ReLU \(InLine\[[11](https://arxiv.org/html/2607.24788#bib.bib22)\]\), and Taylor\-series expansions \(ViTaLity\[[5](https://arxiv.org/html/2607.24788#bib.bib16)\]\(degree\-1\), QT\-ViT\[[22](https://arxiv.org/html/2607.24788#bib.bib17)\]\(degree\-2\), BASED\[[1](https://arxiv.org/html/2607.24788#bib.bib4)\]\(degree 2\)\) and Softmax \(Liger\[[15](https://arxiv.org/html/2607.24788#bib.bib27)\]\)\.

### \-AExtended Study on Window Attention

Sliding Window Attention\.To alleviate the linear growth in memory footprint and per\-token attention cost induced by full KV caching, prior work proposes Sliding Window Attention\[[2](https://arxiv.org/html/2607.24788#bib.bib3)\]\. Under sliding window attention, self\-attention for query𝐪i\\mathbf\{q\}\_\{i\}is restricted to the most recent window sizewwkey–value pairs, and the attention scoreαi​j\\alpha\_\{ij\}is computed only within this window as,

αi​jSWA=exp​\(𝐪i,𝐤j\)∑j′=i−w\+1iexp​\(𝐪i,𝐤j′\),\\alpha\_\{ij\}^\{\\text\{SWA\}\}=\\frac\{\\text\{exp\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j\}\)\}\{\\sum\_\{j^\{\\prime\}=i\-w\+1\}^\{i\}\\text\{exp\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{k\}\_\{j^\{\\prime\}\}\)\},\(8\)
OiSWA=∑j=i−w\+1iαi​jSWA\.𝐯j,O\_\{i\}^\{\\text\{SWA\}\}=\\sum\_\{j=i\-w\+1\}^\{i\}\\alpha^\{\\text\{SWA\}\}\_\{ij\}\.\\mathbf\{v\}\_\{j\},\(9\)
where the attention incurs a reducedO​\(w\)O\(w\)memory footprint, with the window sizewwconstrained such thatw<nw<n\. In SWA, based on the window sizewwthe initial token within the window gets evicted as the new token comes in, this first\-in based eviction policy could remove important tokens from the window, leading to a degradation in the model’s recall rate, which measures the model’s ability to retrieve, attend to, and condition its generation on relevant information from earlier context tokens\.

\(a\) Eviction\-based SWA\.Eviction\-based approaches compute attention scores under two categories: a fixed KV cache and an adaptive KV cache budget\. Some of the fixed\-cache budget works are SnapKV\[[17](https://arxiv.org/html/2607.24788#bib.bib6)\]compresses the KV cache by retaining head\-specific salient tokens identified from attention patterns\. H2O\[[26](https://arxiv.org/html/2607.24788#bib.bib9)\]dynamically manages the KV cache by preserving both recent tokens and heavy hitters with high cumulative attention scores\. StreamingLLM\[[21](https://arxiv.org/html/2607.24788#bib.bib7)\]enables unbounded sequence generation by explicitly retaining attention sink tokens\. Beyond fixed\-budget eviction, several works introduce adaptive KV\-cache budget allocation strategies\. PyramidKV\[[3](https://arxiv.org/html/2607.24788#bib.bib5)\]proposes dynamic KV\-cache memory allocation across transformer layers, selectively retaining important KV pairs under a global budget constraint\.

![Refer to caption](https://arxiv.org/html/2607.24788v1/figures/GLIDE_Ablation_Mistral_Perf.png)Figure 8:Analysis of Zero\-Shot Accuracy Under Varying Softmax Attention Intensity \(Non Fine\-Tuned\)\.Layer\-wise sensitivity analysis: initial layers require softmax attention, while end layers tolerate full linearization\. We evaluate uniform configurations\{0,w/4,w/2,3​w/4,w\}\\\{0,\\,w/4,\\,w/2,\\,3w/4,\\,w\\\}with the base window sizew=64w=64and model dimensiond=4096d=4096across all transformer layers using LM\-Eval \(PiQA, ARC\-e, ARC\-c, HellaSwag, WinoG\), then examine recovery by applying varying softmax attention to isolated layer groups \(Early, Middle, Late\)\.\(b\) Retention\-based SWA\.Despite advances in local context optimization, eviction\-based methods inherently restrict access to global context, which can adversely affect recall\. This limitation can be mitigated by preserving global context as a recurrent state via linear attention mechanisms, enabling long\-range information retention under a constant memory budget\. Building on this framework, LOLA\[[12](https://arxiv.org/html/2607.24788#bib.bib26)\]introduces a sparse caching mechanism to reduce computational overhead while maintaining approximation quality\. Additionally, the prior works formulates attention hybridization as a two\-component output combining a linear recurrent with a local softmax term over the active window\. The various mathematical formation of hybrid attention can be given by,

Oihybrid=ϕ​\(𝐪i\)​𝐒i−wϕ​\(𝐪i\)​𝐙i−w\+∑j=i−w\+1iexp⁡\(𝐪i​𝐤j⊤\)​𝐯j∑j′=i−w\+1iexp⁡\(𝐪i​𝐤j′⊤\)O^\{\\text\{hybrid\}\}\_\{i\}=\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\,\\mathbf\{S\}\_\{i\-w\}\}\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\,\\mathbf\{Z\}\_\{i\-w\}\}\\;\+\\;\\frac\{\\sum\_\{j=i\-w\+1\}^\{i\}\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\top\}\)\\,\\mathbf\{v\}\_\{j\}\}\{\\sum\_\{j^\{\\prime\}=i\-w\+1\}^\{i\}\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j^\{\\prime\}\}^\{\\top\}\)\}\(10\)
where𝐒i−w=∑j=1i−wϕ​\(𝐤j\)⊤​𝐯j\\mathbf\{S\}\_\{i\-w\}=\\sum\_\{j=1\}^\{i\-w\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}\\mathbf\{v\}\_\{j\}and𝐙i−w=∑j=1i−wϕ​\(𝐤j\)⊤\\mathbf\{Z\}\_\{i\-w\}=\\sum\_\{j=1\}^\{i\-w\}\\phi\(\\mathbf\{k\}\_\{j\}\)^\{\\top\}are the recurrent state and normalization factor accumulating all tokens evicted from the sliding window\. Further, the another form of hybrid attention can be written as,

Oihybrid=ϕ​\(𝐪i\)​𝐒iϕ​\(𝐪i\)​𝐙i\+∑j=i−w\+1iexp⁡\(𝐪i​𝐤j⊤\)​𝐯j∑j′=i−w\+1iexp⁡\(𝐪i​𝐤j′⊤\)O^\{\\text\{hybrid\}\}\_\{i\}=\\frac\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\,\\mathbf\{S\}\_\{i\}\}\{\\phi\(\\mathbf\{q\}\_\{i\}\)\\,\\mathbf\{Z\}\_\{i\}\}\\;\+\\;\\frac\{\\sum\_\{j=i\-w\+1\}^\{i\}\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\top\}\)\\,\\mathbf\{v\}\_\{j\}\}\{\\sum\_\{j^\{\\prime\}=i\-w\+1\}^\{i\}\\exp\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j^\{\\prime\}\}^\{\\top\}\)\}\(11\)
where, the constant recurrent stateSi∈ℝd×dS\_\{i\}\\in\\mathbb\{R\}^\{d\\times d\}includes redundant information from the prefill or the instance before the sliding window attention starts evicting the tokens\.Glideadapts both forms of the hybrid attention mechanism, unified through the generic⊕\\oplusoperator\.