SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

# SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Source: [https://arxiv.org/html/2606.31145](https://arxiv.org/html/2606.31145)
Amirhossein Abaskohi1Giuseppe Carenini1Peter West1Yuhang He2 1University of British Columbia2Microsoft ResearchCorresponding author:aabaskoh@cs\.ubc\.ca\. Part of this work was done during the internship at Microsoft Research \(MSR\), Vancouver Lab\.

###### Abstract

Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression\. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation\. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token\-level detail from a compressed span once it becomes relevant during generation\. As a solution, we proposeSeKV, a resolution\-adaptive semantic KV cache that organizes context into entropy\-guided semantic spans and stores them across a GPU–CPU memory hierarchy without discarding information\. Each span keeps a lightweight summary vector on GPU for coarse routing and a low\-rank SVD basis on CPU for on\-demand token\-level reconstruction\. A trained zoom\-in mechanism selectively expands query\-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU\.SeKVenables adaptive token\-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0\.05% trainable parameters\. Across four benchmarks,SeKVimproves over the strongest semantic compression baseline by 5\.9% on average while reducing GPU memory by 53\.3% versus full KV caching at 128K context\.111Code is available on[GitHub\.](https://github.com/AmirAbaskohi/SeKV)\.

SeKV: Resolution\-Adaptive KV Cache with Hierarchical Semantic Memory for Long\-Context LLM Inference

Amirhossein Abaskohi1††thanks:Corresponding author:aabaskoh@cs\.ubc\.ca\. Part of this work was done during the internship at Microsoft Research \(MSR\), Vancouver Lab\.Giuseppe Carenini1Peter West1Yuhang He21University of British Columbia2Microsoft Research

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.31145v1/x1.png)Figure 1:\(a\)Existing token eviction methods discard semantically critical tokens from distant context, causing attention to pool at document boundaries while the answer region receives near\-zero attention, leading to hallucination;\(b\)SeKV organizes context into entropy\-guided semantic spans, preserving all information across a GPU/CPU memory hierarchy\. A trained zoom\-in mechanism dynamically expands the most query\-relevant span to full token\-level resolution, with anchor tokens always retained on GPU for precise retrieval\.Large language models \(LLMs\) are increasingly expected to reason over long contexts in applications such as document analysisLuoet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib15)\), multi\-turn dialogueZhanget al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib12)\), and agentic planningHuanget al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib14)\)\. While recent models advertise context windows of 128K tokens or more, using these contexts in practice remains expensive: the Key\-Value \(KV\) cache grows linearly with sequence length and must be retained throughout decoding\. As a result, full KV caching can become prohibitively memory\-intensive, making long\-context inference difficult to deploy without compression or memory\-efficient cache management\. This creates a practical challenge distinct from simply extending the context window: long\-context models must not only accept long inputs, but also preserve access to relevant evidence under tight memory constraints\.

Recent KV cache management methods reduce inference cost through token eviction, quantization, merging, retrieval, or offloadingLiet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib18)\); Caiet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib37)\); Wu and Tu \([2024](https://arxiv.org/html/2606.31145#bib.bib39)\); Chenet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib6)\); Zhaoet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib9)\)\. However, most methods either treat context as a flat sequence of token\-level states or apply semantic grouping with a static compression policy fixed at prefill timeLiuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\); Wuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\); Zhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\)\. Both choices are limiting for long\-context reasoning\. Token\-level compression can fragment coherent evidence or permanently discard information, while static semantic compression cannot recover token\-level detail when a compressed span becomes relevant later in decoding\. This is especially problematic when evidence that appears peripheral early becomes a critical logical anchor for a later query\. Therefore, in this paper, we argue that long\-context KV caching requires not only stronger compression, but a different memory organization: one that stores context at variable resolution and dynamically zooms into the appropriate level of detail based on query relevance\.

We introduceSeKV\(SemanticKVCache\), a resolution\-adaptive semantic KV cache for efficient long\-context LLM inference\.SeKVorganizes the KV cache as ahierarchical semantic memory\. First, the input context is segmented into semantically coherent spans usingtoken surprisalas a boundary signal, which is available from the model’s prefill forward pass and often peaks at semantic transitions such as topic shifts and entity introductionsZhaoet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib20)\)\. High\-surprisalanchor tokensat span boundaries are preserved at full resolution on GPU, as they carry concentrated semantic information useful for later routing and reconstruction\. Second, each span is represented by a lightweightsummary vectoron GPU \(used for efficient coarse routing\) and alow\-rank SVD basison CPU that compactly encodes the span’s token\-level structure for faithful reconstruction\. Third, atrained zoom\-in mechanismdynamically selects which spans to expand to token\-level resolution during decoding, triggering asynchronous CPU\-to\-GPU fetches of SVD bases for relevant spans while GPU computation proceeds in parallel\. Crucially,SeKVavoids irreversible token eviction: each span is represented by GPU\-resident summary vectors for coarse routing and CPU\-resident low\-rank bases that enable approximate token\-level reconstruction on demand\. Across four long\-context understanding benchmarks,SeKVachieves 5\.9% improvement over the strongest semantic compression baseline while reducing GPU memory consumption by 53\.3% compared to full KV caching at 128K context length\.

In summary, this work makes three contributions\.\(1\)we proposeSeKV, a resolution\-adaptive semantic KV cache that stores context at multiple resolutions and dynamically adjusts cache detail based on query relevance during decoding\.\(2\)We introduceentropy\-guided span segmentation, which uses token surprisal during prefilling to identify coherent semantic spans and preserve high\-surprisal anchor tokens at full resolution for reliable later retrieval\.\(3\)We design adual\-representation KV structurewith GPU\-resident span summaries for coarse routing, CPU\-resident low\-rank bases for approximate token\-level reconstruction, and a trained zoom\-in mechanism that selectively expands only the query\-relevant spans during decoding\.

## 2Related Work

KV cache compression and eviction\.The dominant approach to KV cache compression is token eviction, where tokens are permanently discarded based on attention statisticsZhanget al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib34)\); Liuet al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib35)\); Xiaoet al\.\([2024b](https://arxiv.org/html/2606.31145#bib.bib38)\); Liet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib18)\); Caiet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib37)\)\. Head\-aware methods distinguish retrieval heads from streaming heads to apply sparsity selectivelyXiaoet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib27)\); Tanget al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib40)\), while cross\-layer methods exploit the high similarity between adjacent layer KV states to merge or share representationsLiuet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib17)\); Wu and Tu \([2024](https://arxiv.org/html/2606.31145#bib.bib39)\); Sunet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib21)\); Huet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib8)\)\. To scale beyond GPU memory limits, retrieval\-based systems offload KV pairs to CPU and fetch relevant entries on demand using lightweight GPU\-resident proxiesXiaoet al\.\([2024a](https://arxiv.org/html/2606.31145#bib.bib19)\); Chenet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib6)\); Zhaoet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib9)\)\. DesireKVChenget al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib5)\)targets reasoning models specifically by jointly considering attention\-derived importance and quantization sensitivity to make differentiated per\-token compression decisions\. While collectively effective, these methods all fix compression decisions at prefill time, operate at the token level, and permanently discard information — properties that limit their ability to recover evidence that becomes relevant only during generation\.SeKVdiffers fundamentally: it organizes context into semantic spans, avoids irreversible token eviction through a GPU–CPU memory hierarchy, and dynamically recovers token\-level detail on demand with a trained zoom\-in mechanism\.

Semantic KV cache compression\.Recent methods introduce semantically structured KV compression by grouping tokens before compression\. ChunkKVLiuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\)preserves local coherence by selecting contiguous token chunks; SemantiCacheWuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\)segments context, clusters tokens by key similarity, and merges each cluster into representative KV pairs; and SentenceKVZhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\)stores sentence\-level semantic vectors on GPU and offloads selected token\-level KV pairs to CPU for dynamic retrieval during decoding\. These methods show that semantic structure can improve cache efficiency over flat token eviction\. However, their semantic units and retained cache representations are largely fixed during prefill, making it difficult to recover finer\-grained evidence outside the retained token pool when it becomes relevant later\.SeKVaddresses this limitation with entropy\-guided spans, GPU\-resident routing summaries, CPU\-resident low\-rank bases, and a trained zoom\-in mechanism that reconstructs query\-relevant spans on demand\.

## 3SeKV

![Refer to caption](https://arxiv.org/html/2606.31145v1/x2.png)Figure 2:Overview ofSeKV\.The input is segmented into entropy\-guided spans\. Anchor tokens and summary vectors reside on GPU for coarse routing, while SVD bases are stored on CPU\. At each decoding step, Stage 1 routing identifies relevant spans and triggers asynchronous fetching of their SVD bases\. Stage 2 reconstructs token\-level KV pairs for zoomed spans and computes fine\-grained attention, with outputs merged across both stages\.SeKVreorganizes the KV cache from a flat token\-level sequence into a hierarchical semantic memory\. As illustrated in Figure[2](https://arxiv.org/html/2606.31145#S3.F2), the context is segmented into semantically coherent spans, each represented at two resolutions simultaneously: a compact summary vector on GPU for efficient coarse routing, and a low\-rank SVD basis on CPU for faithful token\-level reconstruction on demand\. A trained zoom\-in mechanism with per\-head per\-layer routing projections, a learned per\-span rank budget, and learned thresholds dynamically selects which spans to expand during decoding and at what fidelity\. We describe each component below\.

### 3\.1Entropy\-Guided Span Segmentation

SeKVsegments the input context into semantically coherent spans using token\-level surprisal as a zero\-cost boundary signal\. During prefilling, the surprisal of the token at each positionttis computed as

Ht=−log⁡p​\(xt∣x<t\),H\_\{t\}=\-\\log p\\big\(x\_\{t\}\\mid x\_\{<t\}\\big\),\(1\)wherextx\_\{t\}is the token at positionttandx<t=\(x1,…,xt−1\)x\_\{<t\}=\(x\_\{1\},\\ldots,x\_\{t\-1\}\)denotes the preceding tokens\. This quantity is already produced by the prefill forward pass and requires no additional computation\. High\-surprisal tokens often mark semantic transitions such as topic shifts, entity introductions, and logical turnsZhaoet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib20)\)\. We therefore treat positions whereHt\>μ\+α​σH\_\{t\}\>\\mu\+\\alpha\\sigmaas span boundaries, whereμ\\muandσ\\sigmaare the mean and standard deviation of surprisal over the sequence andα\\alphacontrols boundary sensitivity\. These boundary tokens are retained asanchor tokensat full resolution on GPU, while the tokens between two neighboring anchors form a semantic spanSS\.

Compared with fixed\-size chunkingLiuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\)or delimiter\-based splittingWuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\), surprisal\-based segmentation is content\-adaptive and requires no external tagger, splitter, or additional model\. It allocates finer resolution to information\-dense regions, which produce shorter spans and more anchors, while compressing lower\-density text into longer spans\. The resulting spans are then encoded into the dual\-resolution representation described in Section[3\.2](https://arxiv.org/html/2606.31145#S3.SS2)\.

### 3\.2Dual\-Resolution Span Representation and Memory Hierarchy

Each semantic spanSSis represented at two resolutions: a GPU\-resident summary used for efficient span\-level routing, and a CPU\-resident low\-rank basis used for approximating token\-level reconstruction when the span is selected for zoom\-in\.

#### Summary vector \(GPU\)\.

For each tokent∈St\\in S, letHtH\_\{t\}denote its surprisal under the frozen LLM during prefilling\. We first convert token surprisals into normalized span\-local weights:

wt=Ht∑k∈SHk\.w\_\{t\}=\\frac\{H\_\{t\}\}\{\\sum\_\{k\\in S\}H\_\{k\}\}\.\(2\)This weights non\-anchor tokens by their residual surprisal\.

For each layerlland attention headhh, letKt\(l,h\)∈ℝdhK\_\{t\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}denote the key vector of tokentt, wheredhd\_\{h\}is the head dimension \(dh=128d\_\{h\}=128onLlama\-3\-8B\)\. We compute a compact summary keyK¯S\(l,h\)∈ℝd′\\bar\{K\}\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}, whered′≪dhd^\{\\prime\}\\ll d\_\{h\}is the routing dimension \(d′=32d^\{\\prime\}=32\), by projecting token keys with a learned per\-head, per\-layer projectionW\(l,h\)∈ℝd′×dhW^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\\times d\_\{h\}\}and taking a normalized surprisal\-weighted average:

K¯S\(l,h\)=∑t∈Swt​W\(l,h\)​Kt\(l,h\)‖∑t∈Swt​W\(l,h\)​Kt\(l,h\)‖2,\\bar\{K\}\_\{S\}^\{\(l,h\)\}=\\frac\{\\sum\_\{t\\in S\}w\_\{t\}W^\{\(l,h\)\}K\_\{t\}^\{\(l,h\)\}\}\{\\left\\\|\\sum\_\{t\\in S\}w\_\{t\}W^\{\(l,h\)\}K\_\{t\}^\{\(l,h\)\}\\right\\\|\_\{2\}\},\(3\)where the per\-head, per\-layer projectionW\(l,h\)W^\{\(l,h\)\}lets each head\-layer pair learn its own relevance space, since attention patterns vary across heads and layersVoitaet al\.\([2019](https://arxiv.org/html/2606.31145#bib.bib23)\)\. This summary is used only for routing\. We additionally keep on GPU the surprisal\-weighted mean key and value in the original space,k¯S\(l,h\)=∑t∈Swt​Kt\(l,h\)\\bar\{k\}\_\{S\}^\{\(l,h\)\}=\\sum\_\{t\\in S\}w\_\{t\}K\_\{t\}^\{\(l,h\)\}andv¯S\(l,h\)=∑t∈Swt​Vt\(l,h\)\\bar\{v\}\_\{S\}^\{\(l,h\)\}=\\sum\_\{t\\in S\}w\_\{t\}V\_\{t\}^\{\(l,h\)\}, both inℝdh\\mathbb\{R\}^\{d\_\{h\}\}, which serve as the coarse contribution of a non\-expanded span in the output attention \(Sec\.[3\.3](https://arxiv.org/html/2606.31145#S3.SS3)\)\.

At each decoding step, we project the current query into the same routing space as the span summaries:

q\(l,h\)=W\(l,h\)​qt\(l,h\)∈ℝd′\.q^\{\(l,h\)\}=W^\{\(l,h\)\}q\_\{t\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}\.\(4\)Since zoom\-in is an independent per\-span decision, we use a sigmoid relevance gate rather than a competitive softmax:

α~S\(l,h\)=sig​\(q\(l,h\)​\(K¯S\(l,h\)\)⊤d′\+log⁡\|S\|\),\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\}=\\mathrm\{sig\}\\\!\\left\(\\frac\{q^\{\(l,h\)\}\\left\(\\bar\{K\}\_\{S\}^\{\(l,h\)\}\\right\)^\{\\top\}\}\{\\sqrt\{d^\{\\prime\}\}\}\+\\log\|S\|\\right\),\(5\)wheresig​\(x\)=1/\(1\+e−x\)\\mathrm\{sig\}\(x\)=1/\(1\+e^\{\-x\}\)is the logistic function\. Thelog⁡\|S\|\\log\|S\|term acts as a monotone size prior, preventing longer spans from being systematically under\-selected\. Becauseα~S\(l,h\)\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\}is span\-independent, the threshold in Sec\.[3\.3](https://arxiv.org/html/2606.31145#S3.SS3)remains stable across context lengths\.

#### Low\-rank SVD basis with a learned rank budget \(CPU\)\.

To support token\-level reconstruction without keeping full KV pairs on GPU, we store a compact low\-rank representation of each span on CPU\. For each layerlland headhh, letKS\(l,h\)∈ℝ\|S\|×dhK\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{\|S\|\\times d\_\{h\}\}denote the key matrix formed by stacking the token keys in spanSS\. We compute its truncated SVD up to a maximum rankR=min⁡\(\|S\|,dh,Rmax\)R=\\min\(\|S\|,d\_\{h\},R\_\{\\max\}\), for a fixed capRmaxR\_\{\\max\},

KS\(l,h\)=∑i=1RσS,i\(l,h\)​uS,i\(l,h\)​\(vS,i\(l,h\)\)⊤,K\_\{S\}^\{\(l,h\)\}=\\sum\_\{i=1\}^\{R\}\\sigma\_\{S,i\}^\{\(l,h\)\}\\,u\_\{S,i\}^\{\(l,h\)\}\{\\left\(v\_\{S,i\}^\{\(l,h\)\}\\right\)\}^\{\\top\},\(6\)with left and right singular vectorsuS,i\(l,h\)∈ℝ\|S\|u\_\{S,i\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{\|S\|\}andvS,i\(l,h\)∈ℝdhv\_\{S,i\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\}\. Rather than truncating at a single global rank, a lightweight predictorgϕg\_\{\\phi\}assigns per\-component soft gates

mS,i\(l,h\)=sig​\(gϕ​\(σS,⋅\(l,h\),K¯S\(l,h\)\)i\)∈\(0,1\),m\_\{S,i\}^\{\(l,h\)\}=\\mathrm\{sig\}\\\!\\left\(g\_\{\\phi\}\\big\(\\sigma\_\{S,\\cdot\}^\{\(l,h\)\},\\bar\{K\}\_\{S\}^\{\(l,h\)\}\\big\)\_\{i\}\\right\)\\in\(0,1\),\(7\)computed from the singular spectrum and the span summary, and the gated reconstruction is

K^S\(l,h\)=∑i=1RmS,i\(l,h\)​σS,i\(l,h\)​uS,i\(l,h\)​\(vS,i\(l,h\)\)⊤\.\\hat\{K\}\_\{S\}^\{\(l,h\)\}=\\sum\_\{i=1\}^\{R\}m\_\{S,i\}^\{\(l,h\)\}\\,\\sigma\_\{S,i\}^\{\(l,h\)\}\\,u\_\{S,i\}^\{\(l,h\)\}\{\\left\(v\_\{S,i\}^\{\(l,h\)\}\\right\)\}^\{\\top\}\.\(8\)The effective rank of spanSSisrS\(l,h\)=∑imS,i\(l,h\)r\_\{S\}^\{\(l,h\)\}=\\sum\_\{i\}m\_\{S,i\}^\{\(l,h\)\}\. At deployment, components with negligible gates are dropped, so the stored and transferred factorsUS\(l,h\)∈ℝ\|S\|×rSU\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{\|S\|\\times r\_\{S\}\},ΣS\(l,h\)∈ℝrS×rS\\Sigma\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{r\_\{S\}\\times r\_\{S\}\}, andVS\(l,h\)∈ℝdh×rSV\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times r\_\{S\}\}adapt per span: information\-dense spans retain more components, redundant spans fewer\. We apply the same gated decomposition to the value matrix𝒱S\(l,h\)∈ℝ\|S\|×dh\\mathcal\{V\}\_\{S\}^\{\(l,h\)\}\\in\\mathbb\{R\}^\{\|S\|\\times d\_\{h\}\}, obtaining its own gatesmS,j𝒱,\(l,h\)m\_\{S,j\}^\{\\mathcal\{V\},\(l,h\)\}and factorsUS𝒱,\(l,h\),ΣS𝒱,\(l,h\),VS𝒱,\(l,h\)U\_\{S\}^\{\\mathcal\{V\},\(l,h\)\},\\Sigma\_\{S\}^\{\\mathcal\{V\},\(l,h\)\},V\_\{S\}^\{\\mathcal\{V\},\(l,h\)\}, which are likewise stored on CPU\. HereVS\(l,h\)V\_\{S\}^\{\(l,h\)\}andVS𝒱,\(l,h\)V\_\{S\}^\{\\mathcal\{V\},\(l,h\)\}denote the right singular vectors of the key and value matrices, while𝒱S\(l,h\)\\mathcal\{V\}\_\{S\}^\{\(l,h\)\}denotes the value matrix itself\.

Because entropy\-guided segmentation groups tokens into coherent spans, their KV matrices are often more compressible than arbitrary token blocks, so a small number of components can preserve most span information\. Storing a retained rank\-rSr\_\{S\}factorization costsrS​\(\|S\|\+dh\+1\)r\_\{S\}\(\|S\|\+d\_\{h\}\+1\)values per head and layer, compared with\|S\|​dh\|S\|d\_\{h\}for the full key matrix, yielding savings whenrS≪dhr\_\{S\}\\ll d\_\{h\}\. The representation is therefore compact in transfer volume, although the left factor still scales with span length\. To avoid unnecessary overhead, spans shorter thanLminL\_\{\\min\}are kept at full resolution on GPU\. Key and value decompositions are computed once during prefilling; randomized truncated SVD keeps this step parallelizable and amortizes its cost over decoding\.

Together, these representations define the memory hierarchy inSeKV\. Anchor tokens remain uncompressed on GPU and always participate in fine\-grained attention, while span summaries support coarse routing\. Low\-rank SVD bases for non\-anchor tokens reside on CPU and are fetched asynchronously only for selected spans\. During decoding,SeKVoverlaps GPU computation with CPU–GPU prefetching, following pipelined orchestration designsChenet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib6)\)\. Unlike retrieval\-based offloading methods that transfer full token\-level KV pairs from CPUXiaoet al\.\([2024a](https://arxiv.org/html/2606.31145#bib.bib19)\); Zhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\),SeKVtransfers compact low\-rank bases and reconstructs only zoomed spans, reducing transfer volume while avoiding irreversible token eviction\.

### 3\.3Trained Zoom\-In Mechanism

At each decoding step,SeKVfirst routes over span summary vectors\. For each layerlland headhh, the routing gate assigns a per\-span relevance probabilityα~S\(l,h\)\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\}, and a learned per\-head per\-layer thresholdτ\(l,h\)\\tau^\{\(l,h\)\}determines whether spanSSis expanded:

zS\(l,h\)=𝟏​\[α~S\(l,h\)\>τ\(l,h\)\]\.z\_\{S\}^\{\(l,h\)\}=\\mathbf\{1\}\\\!\\left\[\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\}\>\\tau^\{\(l,h\)\}\\right\]\.\(9\)Per\-head thresholds matter because heads have different selectivity: retrieval heads may expand many distant spans, while streaming heads should rarely trigger expansion\. Each decoding step then performs a single attention computation over a mixed\-resolution memory\. Anchor tokens always contribute full key–value pairs\. Non\-expanded spans contribute one coarse entry, given by their retained mean key and valuek¯S\(l,h\)\\bar\{k\}\_\{S\}^\{\(l,h\)\}andv¯S\(l,h\)\\bar\{v\}\_\{S\}^\{\(l,h\)\}, with alog⁡\|S\|\\log\|S\|correction added to the pre\-softmax score so that one summary competes proportionally to the\|S\|\|S\|tokens it representsMohtashami and Jaggi \([2023](https://arxiv.org/html/2606.31145#bib.bib22)\)\. Expanded spans fetch their low\-rank bases from CPU, reconstruct token\-level KV pairs via Eq\.[8](https://arxiv.org/html/2606.31145#S3.E8), and enter attention at full resolution\. Since coarse summaries and reconstructed tokens share the same softmax, zooming refines a span’s contribution rather than adding a separate attention path, and no span is dropped from context\.

For fair comparison with fixed\-budget baselines, we define the budget as the peak GPU\-resident KV during a decoding step, including persistent anchors, summaries, coarse means, and any spans reconstructed at that step\. The learned thresholdsτ\(l,h\)\\tau^\{\(l,h\)\}provide the default expansion rule\. If the selected spans exceed the budget, we keep spans in decreasing order ofα~S\(l,h\)\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\}until the budget is filled\. Thus, the threshold sets the operating point, while the cap enforces the matched memory budget\.

We train only the lightweight routing components while keeping the base LLM frozen, using a straight\-through estimatorBengioet al\.\([2013](https://arxiv.org/html/2606.31145#bib.bib2)\)for the binary decision in Eq\.[9](https://arxiv.org/html/2606.31145#S3.E9)\. The objective combines four terms:

ℒ=ℒdistill\+λ1​ℒzoom\+λ2​ℒrecon\+λ3​ℒbudget\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{distill\}\}\+\\lambda\_\{1\}\\mathcal\{L\}\_\{\\text\{zoom\}\}\+\\lambda\_\{2\}\\mathcal\{L\}\_\{\\text\{recon\}\}\+\\lambda\_\{3\}\\mathcal\{L\}\_\{\\text\{budget\}\}\.\(10\)
#### Distillation\.

The distillation loss alignsSeKVwith a full\-KV teacher and provides the end\-task signal that flows, through the straight\-through gate, to both the routing projections and the thresholds:

ℒdistill=DKL​\(pfull∥pSeKV\)\.\\mathcal\{L\}\_\{\\text\{distill\}\}=D\_\{\\text\{KL\}\}\\\!\\left\(p\_\{\\text\{full\}\}\\,\\\|\\,p\_\{\\text\{\{SeKV\}\}\}\\right\)\.\(11\)

#### Zoom supervision\.

We supervise the routing probability directly against a teacher\-derived target rather than against the thresholded decision, which keeps the loss well\-posed and free of the trainable threshold\. LetaS\(l,h\)=∑t∈Sαt\(l,h\),teachera\_\{S\}^\{\(l,h\)\}=\\sum\_\{t\\in S\}\\alpha\_\{t\}^\{\(l,h\),\\mathrm\{teacher\}\}be the teacher attention mass on spanSS, and let𝒞ρ\(l,h\)\\mathcal\{C\}\_\{\\rho\}^\{\(l,h\)\}be the smallest set of spans whose cumulative mass reaches a fixed fractionρ\\rho\. The fixed binary target is

yS\(l,h\)=𝟏​\[S∈𝒞ρ\(l,h\)\],y\_\{S\}^\{\(l,h\)\}=\\mathbf\{1\}\\\!\\left\[S\\in\\mathcal\{C\}\_\{\\rho\}^\{\(l,h\)\}\\right\],\(12\)which depends only on the teacher and not onτ\(l,h\)\\tau^\{\(l,h\)\}\. The zoom loss is a positive\-weighted binary cross\-entropy on the routing probability,

ℒzoom=∑l∑h∑SBCEw\+​\(α~S\(l,h\),yS\(l,h\)\),\\mathcal\{L\}\_\{\\text\{zoom\}\}=\\sum\_\{l\}\\sum\_\{h\}\\sum\_\{S\}\\mathrm\{BCE\}\_\{w^\{\+\}\}\\\!\\left\(\\tilde\{\\alpha\}\_\{S\}^\{\(l,h\)\},\\,y\_\{S\}^\{\(l,h\)\}\\right\),\(13\)where positive weightw\+w^\{\+\}compensates for the rarity of relevant spans\. Since the target is fixed and independent ofτ\(l,h\)\\tau^\{\(l,h\)\}, this loss trains the routing projections to separate relevant from irrelevant spans without letting the threshold define its own supervision\.

#### Reconstruction\.

The reconstruction loss supervises the per\-span rank budget through the per\-component soft gatesmS,i\(l,h\)∈\(0,1\)m\_\{S,i\}^\{\(l,h\)\}\\in\(0,1\)predicted bygϕg\_\{\\phi\}\(Eq\.[7](https://arxiv.org/html/2606.31145#S3.E7)\), and their value counterpartsmS,j𝒱,\(l,h\)m\_\{S,j\}^\{\\mathcal\{V\},\(l,h\)\}\. Since the singular components are orthonormal, the gated approximation error of a single factorization isℰ​\(m,σ\)=∑i\(1−mi\)2​σi2\\mathcal\{E\}\(m,\\sigma\)=\\sum\_\{i\}\(1\-m\_\{i\}\)^\{2\}\\sigma\_\{i\}^\{2\}\. Summing over keys and values, heads, layers, and spans gives

ℒrecon=∑l,h,Sℰ​\(mS\(l,h\),σS\(l,h\)\)\+∑l,h,Sℰ​\(mS𝒱,\(l,h\),σS𝒱,\(l,h\)\)\\begin\{split\}\\mathcal\{L\}\_\{\\text\{recon\}\}=\{\}&\\sum\_\{l,h,S\}\\mathcal\{E\}\\big\(m\_\{S\}^\{\(l,h\)\},\\,\\sigma\_\{S\}^\{\(l,h\)\}\\big\)\\\\ &\+\\sum\_\{l,h,S\}\\mathcal\{E\}\\big\(m\_\{S\}^\{\\mathcal\{V\},\(l,h\)\},\\,\\sigma\_\{S\}^\{\\mathcal\{V\},\(l,h\)\}\\big\)\\end\{split\}\(14\)whereσS\(l,h\)\\sigma\_\{S\}^\{\(l,h\)\}andσS𝒱,\(l,h\)\\sigma\_\{S\}^\{\\mathcal\{V\},\(l,h\)\}are the singular values of the key and value matrices of spanSS\. This is differentiable in the gates and therefore trains the rank predictorgϕg\_\{\\phi\}\. The singular values and vectors come from the frozen keys and values and are treated as constants \(no gradient flows through the SVD\), which avoids the instability of SVD backpropagation under near\-degenerate singular values\.

#### Budget\.

The reconstruction loss alone is minimized by retaining every component, and distillation likewise prefers more rank and more expansions, so a cost term is required to make the rank budget and the threshold meaningful:

ℒbudget=∑l∑h∑S\(zS\(l,h\)\+β​∑imS,i\(l,h\)\)\.\\mathcal\{L\}\_\{\\text\{budget\}\}=\\sum\_\{l\}\\sum\_\{h\}\\sum\_\{S\}\\left\(z\_\{S\}^\{\(l,h\)\}\+\\beta\\sum\_\{i\}m\_\{S,i\}^\{\(l,h\)\}\\right\)\.\(15\)The first term penalizes the number of expanded spans and, through the straight\-through gate, encourages higher thresholds when expansions are too costly\. The second term penalizes the total retained rank∑imS,i\(l,h\)\\sum\_\{i\}m\_\{S,i\}^\{\(l,h\)\}, balancingℒrecon\\mathcal\{L\}\_\{\\text\{recon\}\}so that rank is allocated only where it meaningfully reduces error\. The coefficientβ\>0\\beta\>0sets the relative cost of retained rank against span expansion, controlling how the budget is split between transferring more spans and reconstructing each at higher rank\.

#### Learnable parameters\.

SeKVkeeps the base LLM frozen and learns only the routing projections\{W\(l,h\)\}\\\{W^\{\(l,h\)\}\\\}, the zoom thresholds\{τ\(l,h\)\}∈ℝL×H\\\{\\tau^\{\(l,h\)\}\\\}\\in\\mathbb\{R\}^\{L\\times H\}, and the shared rank\-gate predictorgϕg\_\{\\phi\}\.Llama\-3\-8BLlama Team \([2024](https://arxiv.org/html/2606.31145#bib.bib26)\)uses grouped\-query attention withL=32L=32layers,H=32H=32query heads, and88key\-value heads\. Each query head routes against a summary built from its group’s shared key\-value head, so the projections are indexed per query head, with\{W\(l,h\)\}∈ℝL×H×d′×dh\\\{W^\{\(l,h\)\}\\\}\\in\\mathbb\{R\}^\{L\\times H\\times d^\{\\prime\}\\times d\_\{h\}\}\. Withdh=128d\_\{h\}=128andd′=32d^\{\\prime\}=32, the projections account for about4\.194\.19M parameters, the thresholds forL×H=1024L\\times H=1024, andgϕg\_\{\\phi\}for under0\.10\.1M, totaling approximately4\.34\.3M, about0\.05%0\.05\\%of the base model\.

## 4Experiments and Results

### 4\.1Experimental Setup

Benchmarks\.We evaluateSeKVon four long\-context benchmarks:LongBenchBaiet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib42)\),RULERHsiehet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib43)\),InfiniteBenchZhanget al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib52)\), andNeedle\-in\-a\-Haystack \(NIAH\)gkamradt \([2023](https://arxiv.org/html/2606.31145#bib.bib53)\)\. Together, these benchmarks cover realistic long\-document understanding, controlled synthetic retrieval and reasoning, ultra\-long context stress tests, and targeted retrieval fidelity up to 128K tokens\. We additionally evaluateGSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.31145#bib.bib54)\)in a many\-shot setting to test whether compression preserves step\-by\-step reasoning over long prompts\. Refer to Appendix[B](https://arxiv.org/html/2606.31145#A2)for more details\.

Table 1:Performance comparison of KV cache compression methods across four benchmarks and five models\. All compression methods use a GPU\-resident KV budget equal to 10% of the full KV cache\. Best compressed result inbold, second bestunderlined\.Baselines\.We compareSeKVwith full KV caching and representative KV\-cache compression baselines\. Token\-level methods include StreamingLLMXiaoet al\.\([2024b](https://arxiv.org/html/2606.31145#bib.bib38)\), H2OZhanget al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib34)\), SnapKVLiet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib18)\), and PyramidKVCaiet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib37)\); semantic or chunk\-level methods include ChunkKVLiuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\), SemantiCacheWuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\), and SentenceKVZhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\)\. All compression methods are evaluated under matched GPU\-resident KV memory budgets, with main results at 10% of full KV and additional 5%, 15%, and 20% budget results in the ablation study\. Further baseline details are provided in Appendix[B](https://arxiv.org/html/2606.31145#A2)\.

Backbone models\.We evaluateSeKVon five backbones:Llama\-3\.2\-3B\-Instruct,Llama\-3\-8B,Llama\-3\.1\-8B\-InstructLlama Team \([2024](https://arxiv.org/html/2606.31145#bib.bib26)\),Mistral\-7B\-Instruct\-v0\.3Jianget al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib25)\), andQwen2\.5\-14B\-InstructTeam \([2025](https://arxiv.org/html/2606.31145#bib.bib24)\)\. This suite covers different model scales and families, allowing us to test whetherSeKVgeneralizes across backbones\. Within each benchmark, all methods use the same backbone, tokenizer, decoding strategy, and maximum generation length\. For contexts beyond a model’s default limit, we enable its available long\-context configuration; for settings above 128K tokens, we useQwen2\.5\-14B\-Instruct\-1Mwhen full\-context evaluation is required\.

Tasks & Metrics\.We follow each benchmark’s official protocol, reporting average F1 on LongBench QA subsets, average accuracy on RULER, accuracy on the InfiniteBench Retrieve subset, retrieval accuracy on NIAH, and exact\-match accuracy on GSM8K\.

Implementation and inference details\.SeKVis implemented in PyTorch222[https://pytorch\.org/](https://pytorch.org/)with HuggingFace Transformers333[https://huggingface\.co/](https://huggingface.co/)\. The base LLM remains frozen; we train only the per\-head per\-layer routing projections, the zoom\-in thresholds, and the shared rank\-gate predictorgϕg\_\{\\phi\}\(Section[3\.3](https://arxiv.org/html/2606.31145#S3.SS3)\)\. These lightweight components are trained on long documents from RedPajamaWeberet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib57)\)\(arXiv, books, and code subsets\), with the zoom objective supervised only at mined query positions whose teacher attention concentrates beyond a local window ofw=512w=512tokens; no instruction or synthetic\-retrieval data is used, and the training corpus is disjoint from LongBench, RULER, InfiniteBench, NIAH, and GSM8K\. Training uses AdamWLoshchilov and Hutter \([2019](https://arxiv.org/html/2606.31145#bib.bib58)\)\(learning rate1×10−31\\times 10^\{\-3\}, weight decay0\.010\.01, cosine decay with10%10\\%warmup\) in bf16 for roughly33K steps over about0\.50\.5B tokens, with an88K→\\rightarrow3232K length curriculum and an effective batch of88sequences \(one per GPU\)\. We use a maximum SVD rankR=32R=32with the per\-span effective rank set bygϕg\_\{\\phi\}, summary dimensiond′=32d^\{\\prime\}=32, boundary sensitivityα=1\.0\\alpha=1\.0, permissive threshold initializationτ=0\.05\\tau=0\.05, and loss weightsλ1=1\.0\\lambda\_\{1\}=1\.0\(zoom\),λ2=0\.5\\lambda\_\{2\}=0\.5\(reconstruction\), andλ3=0\.1\\lambda\_\{3\}=0\.1\(budget, annealed from0over the first half of training\), with rank sub\-weightβ=0\.1\\beta=0\.1, positive class weightw\+=10w^\{\+\}=10, and coverage fractionρ=0\.9\\rho=0\.9\. Training a single backbone takes22to66hours on one8×8\\timesNVIDIA A100 \(80GB\) node, withQwen2\.5\-14B\-Instructat the upper end; the full hyperparameter list is given in Appendix[A](https://arxiv.org/html/2606.31145#A1)\. At inference time, we use greedy decoding with temperature0and benchmark\-specific maximum generation lengths\. Anchor tokens, span summaries, and coarse span means remain on GPU, while low\-rank SVD bases are stored on CPU and fetched asynchronously when zoom\-in is triggered; to match a target budget, spans are expanded in decreasing routing\-score order until the GPU\-resident KV budget is reached\. LongBench, RULER, NIAH, and GSM8K are evaluated on a single A100 80GB GPU whenever the method fits\. For InfiniteBench and full\-KV reference runs that exceed single\-device memory, we use tensor parallelism across up to88A100 80GB GPUs\. All efficiency comparisons among compression methods use the single\-GPU deployment setting\.

### 4\.2Long\-Context Reasoning

Long\-context benchmark performance\.Table[1](https://arxiv.org/html/2606.31145#S4.T1)shows thatSeKVis the strongest compressed\-cache method across all four benchmarks and all five backbone models\.SeKVwins every compressed setting: 20 out of 20 benchmark–model comparisons\.Averaged over all settings,SeKVimproves over the strongest baseline, SentenceKV, by5\.9%under the same 10% KV budget\. The gains are especially large on retrieval\-heavy benchmarks:\+5\.68 on NIAHand\+3\.63 on RULER, showing that adaptive zoom\-in is particularly useful when a small piece of evidence must be recovered from a long context\. On more heterogeneous benchmarks, the gains remain consistent:\+1\.48 on LongBenchand\+1\.85 on InfiniteBench\. These results support the central claim ofSeKV: static compression preserves coarse context, but dynamic resolution recovery is needed to retrieve fine\-grained evidence at generation time\. We further isolate the contribution of each SEKV component and analyze sensitivity to the GPU\-resident KV budget and key hyperparameters in Appendix[C](https://arxiv.org/html/2606.31145#A3)\.

![Refer to caption](https://arxiv.org/html/2606.31145v1/x3.png)Figure 3:Needle\-in\-a\-Haystack retrieval maps forLlama\-3\.1\-8Bwith KV cache size 128 and contexts up to 8K tokens\. Greener cells indicate higher retrieval success across needle depths and context lengths\.SeKVshows the most stable retrieval behavior, consistent with its strongest NIAH score in Table[1](https://arxiv.org/html/2606.31145#S4.T1)\.Comparison to Full KV and model scaling\.Despite using only 10% of the KV budget,SeKVremains close to Full KV, with average gaps of only0\.80 pointson LongBench and1\.23 pointson InfiniteBench\. The gap is larger on retrieval\-sensitive benchmarks such as RULER and NIAH, butSeKVstill substantially narrows it compared to prior methods\. Its gains over SentenceKV are also stable across backbones, ranging from\+2\.94to\+3\.83, with the largest improvement onQwen2\.5\-14B, suggesting that stronger models benefit more from resolution\-adaptive cache access\.

Needle retrieval behavior\.Figure[3](https://arxiv.org/html/2606.31145#S4.F3)shows NIAH heatmaps forLlama\-3\.1\-8Bwith a fixed KV cache size of 128\.SeKVmaintains strong retrieval across context lengths and needle depths, with most cells remaining high\-success\. Prior methods degrade more noticeably at longer contexts and deeper insertions: SentenceKV and ChunkKV remain competitive but show more failures, while token\-eviction methods such as SnapKV, PyramidKV, and StreamingLLM exhibit broader low\-success regions\. This indicates thatSeKV’s gains are consistent across the length–depth space rather than concentrated on easy needle positions\.

Table 2:Many\-shot GSM8K accuracy with 50\-shot prompts\. FullKV is the uncompressed reference; compressed methods use a 10% GPU\-resident KV budget\.Table 3:Runtime comparison onQwen2\.5\-14B\-Instructwith batch size 1\.SeKVachieves competitive latency and throughput while improving accuracy\.
### 4\.3Many\-Shot Reasoning

Table[2](https://arxiv.org/html/2606.31145#S4.T2)evaluates whether KV\-cache compression preserves reasoning behavior in a long\-prompt setting\. Unlike NIAH or RULER, GSM8K does not primarily test retrieval of a single hidden fact; instead, the model must preserve many in\-context demonstrations that define the reasoning pattern\.SeKVachieves the best compressed\-cache result across all five backbones, improving over the strongest baseline, SentenceKV, by\+2\.3 points on average\. The gap to FullKV also remains small:SeKVis within2\.4 pointsof FullKV on average, with particularly strong results onLlama\-3\.1\-8B\-InstructandQwen2\.5\-14B\-Instruct\. These results show thatSeKVpreserves both sparse retrieval and many\-shot reasoning structure\.

![Refer to caption](https://arxiv.org/html/2606.31145v1/x4.png)Figure 4:GPU memory scaling with context length onQwen2\.5\-14B\-Instructat batch size 1\.
### 4\.4Efficiency Analysis

Table[3](https://arxiv.org/html/2606.31145#S4.T3)compares latency and throughput across input–output lengths\.SeKVis consistently faster than FullKV, reducing latency from 43\.60s to 38\.05s in the 4K/1K setting and from 183\.42s to 166\.95s in the 8K/4K setting, while improving throughput from 105\.92 to 120\.11 T/S and from 55\.93 to 64\.21 T/S\. Although StreamingLLM is fastest due to aggressive eviction, it suffers substantial accuracy loss in Table[1](https://arxiv.org/html/2606.31145#S4.T1)\. Among semantic methods,SeKVoffers the best trade\-off: faster than SentenceKV, close to ChunkKV, and substantially more accurate\. Figure[4](https://arxiv.org/html/2606.31145#S4.F4)further shows thatSeKVkeeps GPU memory nearly flat as context length grows, increasing only from 31\.2GB to 34\.9GB between 8K and 128K, compared with FullKV’s growth from 36\.0GB to 74\.8GB\. Overall,SeKVimproves retrieval fidelity without prohibitive runtime or memory overhead\. A detailed memory and computational complexity analysis is provided in Appendix[D](https://arxiv.org/html/2606.31145#A4)\.

### 4\.5Zoom\-In Behavior Analysis

Figure[5](https://arxiv.org/html/2606.31145#S4.F5)shows thatSeKVexpands compressed spans selectively rather than uniformly\. Zoom\-in decisions are sparse and concentrated in specific heads, especially in mid\-to\-late layers, indicating that only a small subset of retrieval\-oriented heads frequently require token\-level reconstruction\. This supports the use of per\-head per\-layer thresholds and shows thatSeKVsaves memory by allocating fine\-grained resolution only where needed\. Additional heatmaps are provided in Appendix[E](https://arxiv.org/html/2606.31145#A5)\.

![Refer to caption](https://arxiv.org/html/2606.31145v1/x5.png)Figure 5:Average zoom\-in rate across layers and heads forSeKVon NIAH withQwen2\.5\-14B\-Instruct\. Brighter regions indicate more frequent reconstruction\.

## 5Conclusion

We introducedSeKV, a resolution\-adaptive semantic KV cache for efficient long\-context LLM inference\.SeKVorganizes context into entropy\-guided spans, keeps lightweight routing summaries on GPU, and reconstructs low\-rank CPU\-resident span bases on demand, enabling query\-adaptive cache resolution without modifying the base LLM\. Across four benchmarks,SeKVoutperforms prior KV compression methods, including token\-eviction approaches \(StreamingLLMXiaoet al\.\([2024b](https://arxiv.org/html/2606.31145#bib.bib38)\), H2OZhanget al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib34)\), SnapKVLiet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib18)\), PyramidKVCaiet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib37)\)\) and semantic or chunk\-level approaches \(ChunkKVLiuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\), SemantiCacheWuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\), SentenceKVZhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\)\), improving over the strongest semantic baseline, SentenceKV, by 5\.9% on average while reducing GPU memory by 53\.3% compared to Full KV at 128K context length\. These results show that efficient long\-context inference can be achieved by allocating fine\-grained resolution only where needed\.

## Limitations

SeKVhas some limitations worth noting\. First, its effectiveness depends on the quality of the semantic span construction\. SinceSeKVallocates cache resolution based on entropy\-guided semantic segmentation, noisy boundaries or poorly calibrated uncertainty estimates can lead to suboptimal memory allocation, especially in documents with highly fragmented discourse, dense tables, code, or unusual formatting\. Second,SeKVintroduces several design choices, including the entropy threshold for span formation, the number of retained anchor tokens, the rank of span\-level low\-rank bases, and the retrieval budget for CPU\-resident spans\. These parameters control the tradeoff between memory usage, reconstruction quality, and decoding latency, and may require validation when transferring to new models or context lengths\. Third, althoughSeKVreduces GPU memory by moving inactive span representations to CPU, this introduces dependence on host\-device bandwidth\. In settings with slow CPU–GPU transfer or highly adversarial queries that repeatedly activate many distant spans, the latency advantage may be reduced\. Finally,SeKVkeeps the base LLM frozen and only adds lightweight routing components, which improves deployability but may limit the ability of the model to fully adapt its internal attention behavior to the compressed memory structure\. Jointly training the routing module with more expressive span representations or adaptive reconstruction policies is an important direction for future work\.

## References

- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3119–3137\.External Links:[Link](https://aclanthology.org/2024.acl-long.172/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[§B\.1](https://arxiv.org/html/2606.31145#A2.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p1.1)\.
- Estimating or propagating gradients through stochastic neurons for conditional computation\.External Links:1308\.3432,[Link](https://arxiv.org/abs/1308.3432)Cited by:[§3\.3](https://arxiv.org/html/2606.31145#S3.SS3.p3.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=ayi7qezU87)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.18.18.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.7.7.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- Y\. Chen, J\. Zhang, B\. Lu, Q\. Zhang, C\. Zhang, J\. Luo, D\. Liu, H\. Jiang, Q\. Chen, J\. Liu, B\. Ding, X\. Yan, J\. Jiang, C\. Chen, M\. Zhang, Y\. Yang, F\. Yang, and M\. Yang \(2025\)RetroInfer: a vector\-storage approach for scalable long\-context llm inference\.External Links:2505\.02922,[Link](https://arxiv.org/abs/2505.02922)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§3\.2](https://arxiv.org/html/2606.31145#S3.SS2.SSS0.Px2.p3.1)\.
- P\. Cheng, J\. Wang, T\. Chen, B\. Liu, X\. Hou, and J\. Liu \(2026\)DesireKV: decoupling sensitivity and importance for reasoning\-aware kv cache compression\.Proceedings of the AAAI Conference on Artificial Intelligence40\(25\),pp\. 20518–20526\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/39187),[Document](https://dx.doi.org/10.1609/aaai.v40i25.39187)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§B\.1](https://arxiv.org/html/2606.31145#A2.SS1.SSS0.Px5.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p1.1)\.
- gkamradt \(2023\)LLMTest needle in a haystack \- pressure testing llms\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)Cited by:[§B\.1](https://arxiv.org/html/2606.31145#A2.SS1.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§B\.1](https://arxiv.org/html/2606.31145#A2.SS1.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p1.1)\.
- Y\. Hu, J\. Tan, J\. Zhang, W\. Zan, P\. Sun, Y\. Lu, Y\. Sun, Y\. Xie, X\. Cai, and J\. Zhang \(2025\)Optimizing native sparse attention with latent attention and local global alternating strategies\.External Links:2511\.00819,[Link](https://arxiv.org/abs/2511.00819)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- X\. Huang, W\. Liu, X\. Chen, X\. Wang, H\. Wang, D\. Lian, Y\. Wang, R\. Tang, and E\. Chen \(2024\)Understanding the planning of llm agents: a survey\.External Links:2402\.02716,[Link](https://arxiv.org/abs/2402.02716)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p3.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=poE54GOq2l)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.17.17.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.6.6.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- A\. Liu, J\. Liu, Z\. Pan, Y\. He, G\. Haffari, and B\. Zhuang \(2024\)MiniCache: KV cache compression in depth dimension for large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=sgVOjDqUMT)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- X\. Liu, Z\. Tang, P\. Dong, Z\. Li, Liuyue, B\. Li, X\. Hu, and X\. Chu \(2026\)ChunkKV: semantic\-preserving KV cache compression for efficient long\-context LLM inference\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=20JDhbJqn3)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p2.1),[§3\.1](https://arxiv.org/html/2606.31145#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.19.19.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.8.8.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 52342–52364\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/a452a7c6c463e4ae8fbdc614c6e983e6-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- A\. Llama Team \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.3](https://arxiv.org/html/2606.31145#S3.SS3.SSS0.Px5.p1.15),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p3.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[Appendix A](https://arxiv.org/html/2606.31145#A1.SS0.SSS0.Px2.p1.23),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p5.28)\.
- C\. Luo, Y\. Shen, Z\. Zhu, Q\. Zheng, Z\. Yu, and C\. Yao \(2024\)LayoutLLM: layout instruction tuning with large language models for document understanding\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Vol\.,pp\. 15630–15640\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01480)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p1.1)\.
- A\. Mohtashami and M\. Jaggi \(2023\)Random\-access infinite context length for transformers\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 54567–54585\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/ab05dc8bf36a9f66edbff6992ec86f56-Paper-Conference.pdf)Cited by:[§3\.3](https://arxiv.org/html/2606.31145#S3.SS3.p1.9)\.
- Y\. Sun, L\. Dong, Y\. Zhu, S\. Huang, W\. Wang, S\. Ma, Q\. Zhang, J\. Wang, and F\. Wei \(2024\)You only cache once: decoder\-decoder architectures for language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 7339–7361\.External Links:[Document](https://dx.doi.org/10.52202/079017-0235),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/0df38cd13520747e1e64e5b123a78ef8-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- H\. Tang, Y\. Lin, J\. Lin, Q\. Han, D\. Ke, S\. Hong, Y\. Yao, and G\. Wang \(2025\)RazorAttention: efficient kv cache compression through retrieval heads\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 16632–16646\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/2a98af4fea6a24b73af7b588ca95f755-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- Q\. Team \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p3.1)\.
- E\. Voita, D\. Talbot, F\. Moiseev, R\. Sennrich, and I\. Titov \(2019\)Analyzing multi\-head self\-attention: specialized heads do the heavy lifting, the rest can be pruned\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 5797–5808\.External Links:[Link](https://aclanthology.org/P19-1580/),[Document](https://dx.doi.org/10.18653/v1/P19-1580)Cited by:[§3\.2](https://arxiv.org/html/2606.31145#S3.SS2.SSS0.Px1.p2.14)\.
- M\. Weber, D\. Y\. Fu, Q\. G\. Anthony, Y\. Oren, S\. Adams, A\. Alexandrov, X\. Lyu, H\. Nguyen, X\. Yao, V\. Adams, B\. Athiwaratkun, R\. Chalamala, K\. Chen, M\. Ryabinin, T\. Dao, P\. Liang, C\. Re, I\. Rish, and C\. Zhang \(2024\)RedPajama: an open dataset for training large language models\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=lnuXaRpwvw)Cited by:[Appendix A](https://arxiv.org/html/2606.31145#A1.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p5.28)\.
- H\. Wu and K\. Tu \(2024\)Layer\-condensed KV cache for efficient inference of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11175–11188\.External Links:[Link](https://aclanthology.org/2024.acl-long.602/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.602)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- S\. Wu, H\. Lin, S\. Chen, T\. Lu, Y\. Zeng, S\. Zhan, H\. Zheng, and H\. Kim \(2026\)SemantiCache: efficient kv cache compression via semantic chunking and clustered merging\.External Links:2603\.14303,[Link](https://arxiv.org/abs/2603.14303)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px7.p1.1),[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p2.1),[§3\.1](https://arxiv.org/html/2606.31145#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.20.20.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.9.9.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- C\. Xiao, P\. Zhang, X\. Han, G\. Xiao, Y\. Lin, Z\. Zhang, Z\. Liu, and M\. Sun \(2024a\)InfLLM: training\-free long\-context extrapolation for LLMs with an efficient context memory\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=bTHFrqhASY)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§3\.2](https://arxiv.org/html/2606.31145#S3.SS2.SSS0.Px2.p3.1)\.
- G\. Xiao, J\. Tang, J\. Zuo, junxian guo, S\. Yang, H\. Tang, Y\. Fu, and S\. Han \(2025\)DuoAttention: efficient long\-context LLM inference with retrieval and streaming heads\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=cFu7ze7xUm)Cited by:[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024b\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21875–21895\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5e5fd18f863cbe6d8ae392a93fd271c9-Paper-Conference.pdf)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.15.15.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.4.4.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- C\. Zhang, X\. Dai, Y\. Wu, Q\. Yang, Y\. Wang, R\. Tang, and Y\. Liu \(2025\)A survey on multi\-turn interaction capabilities of large language models\.External Links:2501\.09959,[Link](https://arxiv.org/abs/2501.09959)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p1.1)\.
- X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. Hao, X\. Han, Z\. Thai, S\. Wang, Z\. Liu, and M\. Sun \(2024\)∞\\inftyBench: Extending long context evaluation beyond 100K tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15262–15277\.External Links:[Link](https://aclanthology.org/2024.acl-long.814)Cited by:[§B\.1](https://arxiv.org/html/2606.31145#A2.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Re, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RkRrPp7GKO)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.16.16.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.5.5.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.
- J\. Zhao, Z\. Ji, Y\. Feng, P\. Qi, S\. Niu, B\. Tang, F\. Xiong, and Z\. Li \(2025\)Meta\-chunking: learning text segmentation and semantic completion via logical perception\.External Links:2410\.12788,[Link](https://arxiv.org/abs/2410.12788)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p3.1),[§3\.1](https://arxiv.org/html/2606.31145#S3.SS1.p1.9)\.
- Z\. Zhao, B\. Lu, S\. Lin, Y\. Chen, J\. Liu, Y\. Zhang, Z\. Miao, M\. Yang, H\. Shen, Q\. Chen, and F\. Yang \(2026\)Unifying sparse attention with hierarchical memory for scalable long\-context llm serving\.External Links:2604\.26837,[Link](https://arxiv.org/abs/2604.26837)Cited by:[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p1.1)\.
- Y\. Zhu, A\. Falahati, D\. H\. Yang, and M\. M\. Amiri \(2025\)SentenceKV: efficient LLM inference via sentence\-level semantic KV caching\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=HyPeYU9JR6)Cited by:[§B\.2](https://arxiv.org/html/2606.31145#A2.SS2.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2606.31145#S1.p2.1),[§2](https://arxiv.org/html/2606.31145#S2.p2.1),[§3\.2](https://arxiv.org/html/2606.31145#S3.SS2.SSS0.Px2.p3.1),[§4\.1](https://arxiv.org/html/2606.31145#S4.SS1.p2.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.10.10.1),[Table 1](https://arxiv.org/html/2606.31145#S4.T1.1.1.21.21.1),[§5](https://arxiv.org/html/2606.31145#S5.p1.1)\.

## Appendix ATraining Details

#### Training data\.

BecauseSeKVkeeps the backbone frozen and trains only the lightweight routing components of Section[3\.3](https://arxiv.org/html/2606.31145#S3.SS3), it requires little data and a single language\-modeling corpus suffices\. We train on long documents from RedPajamaWeberet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib57)\)\(arXiv, books, and code subsets\)\. The distillation and reconstruction objectives use all positions, while the zoom objective is supervised only at mined query positions whose teacher attention is concentrated on non\-local spans; the attended spans form the coverage target𝒞ρ\\mathcal\{C\}\_\{\\rho\}\. This matches the router to the teacher’s own long\-range attention without any distribution shift from a QA template, and uses no synthetic retrieval or instruction data, so the training corpus is disjoint from every benchmark in Section[4\.1](https://arxiv.org/html/2606.31145#S4.SS1)\. A separate set of routing parameters is trained for each backbone\.

#### Training details\.

The teacher is the frozen backbone with a full KV cache\. We optimize the routing projections, thresholds, and rank\-gate predictor with AdamWLoshchilov and Hutter \([2019](https://arxiv.org/html/2606.31145#bib.bib58)\)\(learning rate1×10−31\\times 10^\{\-3\}, weight decay0\.010\.01, cosine decay,10%10\\%warmup\) for roughly33K steps over about0\.50\.5B tokens, at a maximum sequence length of3232K with an88K→\\rightarrow3232K length curriculum and early stopping on a held\-out split\. We anneal the budget weight from0to its target over the first half of training and initialize the thresholds permissively so that spans expand early and gradients flow through the straight\-through gate\. Teacher span\-attention targets are computed at a sampled subset of query positions to bound the cost of materializing attention weights\. Training a single backbone takes22to66hours on a single8×8\\timesA100 \(80GB\) node\. We use loss weightsλ1=1\\lambda\_\{1\}=1for the zoom term andλ2=0\.5\\lambda\_\{2\}=0\.5for the reconstruction term, with the budget weight annealed toλ3=0\.1\\lambda\_\{3\}=0\.1and rank sub\-weightβ=0\.1\\beta=0\.1; the zoom BCE uses a positive class weightw\+=10w^\{\+\}=10, and the remaining components use coverage fractionρ=0\.9\\rho=0\.9, segmentation sensitivityα=1\.0\\alpha=1\.0, minimum span lengthLmin=16L\_\{\\min\}=16, routing dimensiond′=32d^\{\\prime\}=32, and maximum rankR=32R=32\.

## Appendix BBenchmark and Baseline Details

### B\.1Benchmark Details

We evaluateSeKVon a diverse set of long\-context benchmarks that stress different aspects of KV\-cache compression: multi\-task long\-document understanding, controlled retrieval, ultra\-long\-context reasoning, needle retrieval, and many\-shot mathematical reasoning\. Table[4](https://arxiv.org/html/2606.31145#A2.T4)summarizes the evaluation datasets used in our experiments\.

Table 4:Dataset statistics for the benchmarks used in our experiments\. \# Train and \# Eval denote the number of training and evaluation examples when fixed by the benchmark\. For RULER and NIAH, the evaluation size depends on the chosen context lengths, tasks, and depth/grid settings\. GSM8K is used only for many\-shot evaluation; its training set is not used to trainSeKV\.#### LongBench

Baiet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib42)\)is a multi\-task benchmark designed to evaluate language models on long\-context inputs\. It contains 17 datasets covering single\-document question answering, multi\-document question answering, summarization, few\-shot learning, synthetic retrieval, and code completion\. The average input length ranges from roughly 1K to 18K tokens depending on the task\. LongBench is useful for evaluating whether KV\-cache compression preserves broad long\-document understanding rather than only synthetic retrieval behavior\. In our experiments, we report the official task\-specific average score, using F1, exact match, accuracy, or ROUGE\-L depending on the task\.

#### RULER

Hsiehet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib43)\)provides controlled long\-context tasks at configurable sequence lengths, commonly ranging from 4K to 128K tokens\. It includes synthetic retrieval and reasoning tasks designed to measure whether models can effectively use information placed deep inside long contexts\. Unlike LongBench, where document lengths and task formats vary naturally, RULER allows controlled scaling of context length and therefore directly probes effective context utilization under different KV\-cache budgets\. We report aggregate accuracy across the evaluated RULER tasks and context lengths\.

#### InfiniteBench

Zhanget al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib52)\)evaluates models on substantially longer inputs, often exceeding 100K tokens on average\. It includes tasks such as long\-document question answering, summarization, code debugging, and key\-value retrieval\. This benchmark is particularly important for KV\-cache compression because it reaches the regime where FullKV becomes expensive or infeasible on a single GPU\. We follow the official task\-level evaluation protocol and report the aggregate score across tasks\.

#### Needle\-in\-a\-Haystack \(NIAH

gkamradt \([2023](https://arxiv.org/html/2606.31145#bib.bib53)\)evaluates whether a model can retrieve a specific inserted fact from a long distractor context\. The needle is placed at different depth percentages and evaluated across increasing context lengths\. This benchmark directly measures retrieval fidelity under compression: if a method evicts or over\-compresses the wrong region, the model fails to recover the hidden needle\. We report retrieval accuracy averaged across context lengths and needle depths\.

#### GSM8K many\-shot setting

\(Cobbeet al\.,[2021](https://arxiv.org/html/2606.31145#bib.bib54)\)is a grade\-school mathematical reasoning benchmark\. We use it in a 50\-shot setting to test whether KV\-cache compression preserves the long\-context structure needed for in\-context reasoning\. Unlike NIAH or RULER, GSM8K does not primarily test retrieval of a single hidden fact\. Instead, the model must retain many demonstrations that define the reasoning pattern\. We report exact\-match accuracy after standard answer normalization\.

### B\.2Baseline Details

We compareSeKVwith representative KV\-cache compression methods spanning token eviction, importance\-based retention, structured chunk compression, and semantic cache compression\. FullKV is included as an uncompressed reference\.

#### FullKV

stores all key and value states for every token at every layer and head\. It provides the highest\-fidelity cache representation because no token\-level information is removed or compressed\. However, its memory cost grows linearly with context length, making it increasingly expensive and often infeasible at very long context lengths\. We therefore treat FullKV as an upper\-bound reference rather than a scalable baseline\.

#### StreamingLLM

Xiaoet al\.\([2024b](https://arxiv.org/html/2606.31145#bib.bib38)\)is a token\-retention method designed for streaming generation\. It keeps a small set of initial attention\-sink tokens together with a recent local window, discarding most middle\-context tokens\. This makes StreamingLLM fast and memory\-efficient, but it can fail on tasks requiring retrieval from distant parts of the context because discarded tokens cannot be recovered\.

#### H2O

Zhanget al\.\([2023](https://arxiv.org/html/2606.31145#bib.bib34)\)evicts KV states based on accumulated attention importance\. It identifies heavy\-hitter tokens that receive high attention over time and preserves them under a fixed cache budget\. Compared with simple recency\-based eviction, H2O better retains globally important tokens\. However, its decisions are still token\-level and irreversible, so evidence that appears unimportant early can be permanently removed\.

#### SnapKV

Liet al\.\([2024](https://arxiv.org/html/2606.31145#bib.bib18)\)compresses the KV cache by selecting important tokens based on attention patterns observed during prefilling\. It uses the prompt\-side attention distribution to identify tokens likely to be useful during generation\. SnapKV is effective and efficient for many long\-context tasks, but its compression decisions are fixed after prefill and cannot adapt when later decoding steps require previously compressed information\.

#### PyramidKV

Caiet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib37)\)applies a layer\-wise KV\-cache allocation strategy\. It observes that different layers contribute differently to long\-context retrieval and assigns cache budgets in a pyramid\-like manner across layers\. This improves over uniform token eviction, but the method still operates by retaining or evicting token\-level states and cannot reconstruct discarded evidence\.

#### ChunkKV

Liuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib4)\)moves beyond individual token eviction by selecting contiguous token chunks\. This preserves local coherence better than token\-level pruning because neighboring tokens often jointly encode meaningful evidence\. However, ChunkKV still makes static compression decisions and does not provide a mechanism to dynamically recover fine\-grained token\-level detail from a compressed chunk during decoding\.

#### SemantiCache

Wuet al\.\([2026](https://arxiv.org/html/2606.31145#bib.bib11)\)introduces semantic grouping for KV\-cache compression\. It segments text using linguistic boundaries, clusters tokens according to key\-vector similarity, and merges each cluster into a compact representative KV state\. It also applies proportional attention correction to reduce aggregation bias caused by representing multiple tokens with one compressed state\. SemantiCache is a strong semantic compression baseline, but its merged representations are fixed after prefill and cannot be expanded back into token\-level KV states on demand\.

#### SentenceKV

Zhuet al\.\([2025](https://arxiv.org/html/2606.31145#bib.bib16)\)compresses the cache at the sentence level\. It keeps compact sentence representations on GPU and retrieves relevant token\-level states during decoding\. This makes it the closest baseline toSeKVbecause it combines semantic units with GPU–CPU cache management\. However, SentenceKV relies on sentence\-level segmentation and static sentence representations, whereasSeKVuses entropy\-guided spans, GPU\-resident anchor tokens, low\-rank span bases, and a trained per\-head zoom\-in mechanism for adaptive token\-level reconstruction\.

## Appendix CAblation Studies

#### Component ablations\.

Table[5](https://arxiv.org/html/2606.31145#A3.T5)shows that each component contributes toSeKV’s performance\. The largest drops come from removing SVD reconstruction and trained zoom\-in, reducing NIAH from91\.17to83\.47and85\.96, respectively\. This confirms that span summaries alone are insufficient: the model must be able to recover token\-level detail when a span becomes relevant\. Replacing entropy\-guided segmentation with fixed chunks also hurts substantially, especially on RULER and NIAH, indicating that content\-adaptive span boundaries better preserve retrieval evidence\. Finally, removing anchor tokens, using mean pooling, or replacing per\-head thresholds with a global threshold produces smaller but consistent drops\.Together, these ablations show thatSeKV’s gains come from the combination of semantic span construction, learned zoom\-in, and recoverable fine\-grained KV reconstruction\.

Table 5:Ablation study onQwen2\.5\-14B\-Instructunder a 10% GPU\-resident KV budget\. We report LongBench, RULER, and NIAH scores\. Removing trained zoom\-in or SVD reconstruction causes the largest degradation, showing that adaptive span expansion and token\-level recovery are central toSeKV’s performance\.
#### KV\-budget sensitivity\.

Table[6](https://arxiv.org/html/2606.31145#A3.T6)shows thatSeKVis especially effective under tight memory budgets\. At only 5% KV budget,SeKVreaches86\.31NIAH accuracy, outperforming SentenceKV by\+7\.69points and ChunkKV by\+12\.00points\. Increasing the budget to 10% gives the largest jump, improvingSeKVto91\.17; beyond this point, gains saturate, with only\+1\.29improvement from 10% to 20%\. This suggests thatSeKVrecovers most of the useful long\-context evidence with a small GPU\-resident cache, while additional budget yields diminishing returns\. Beyond the KV budget,SeKVintroduces several implementation hyperparameters, including the SVD rankrr, the surprisal thresholdα\\alpha, and the local window sizeww\.

Table 6:KV\-budget sensitivity on NIAH usingQwen2\.5\-14B\-Instruct\. We vary the GPU\-resident KV budget and report retrieval accuracy\.SeKVconsistently outperforms token\-level and semantic compression baselines across all budgets, with the largest gains in the low\-budget regime\.
#### Hyperparameter Sensitivity\.

Table[7](https://arxiv.org/html/2606.31145#A3.T7)reports the sensitivity ofSeKVto its main hyperparameters across all four benchmarks\. BecauseSeKVlearns a per\-span rank rather than fixing one, this setting controls the maximum rankRRthat the gate predictor may allocate\. Raising the cap fromR=8R=8to3232improves performance across benchmarks, since a tight cap clips spans whose reconstruction needs more components than allowed\. The gains saturate beyondR=32R=32: increasing to6464yields only marginal changes, indicating that the learned gates rarely request more than about3232components, so this cap captures most of the useful span\-level KV structure while bounding memory and transfer overhead\. For entropy\-guided segmentation,α=1\.0\\alpha=1\.0performs best overall\. Smaller thresholds create too many short spans, reducing compression efficiency, while larger thresholds produce overly coarse spans that make zoom\-in less precise\. Finally, increasing the local window fromw=256w=256tow=512w=512improves performance, butw=1024w=1024gives only minor additional gains\. These trends support the default configuration used in the main experiments as a balanced setting across retrieval, reasoning, and long\-context understanding tasks\.

Table 7:Full hyperparameter sensitivity ofSeKVonQwen2\.5\-14B\-Instruct\. Unless varied, we use the default configuration: maximum rankRmax=32R\_\{\\max\}=32, surprisal thresholdα=1\.0\\alpha=1\.0, local window sizew=512w=512, and a 10% GPU\-resident KV budget\. Results are reported on all four long\-context benchmarks\.SeKVis robust across hyperparameter settings, with the default configuration providing a strong accuracy–efficiency trade\-off\.

## Appendix DMemory and Complexity Analysis

This section analyzes the memory and computational cost ofSeKVcompared to FullKV and prior KV\-cache compression methods\. We focus on the dominant inference\-time cost in long\-context generation: storing, accessing, and transferring key\-value states\.

#### FullKV memory cost\.

For a transformer withLLlayers,Hk​vH\_\{kv\}key\-value heads, head dimensiondhd\_\{h\}, sequence lengthnn, andbbbytes per scalar, the memory required by FullKV is

MemFullKV=2​L​Hk​v​dh​n​b,\\mathrm\{Mem\}\_\{\\mathrm\{FullKV\}\}=2LH\_\{kv\}d\_\{h\}nb,\(16\)where the factor of 2 accounts for storing both keys and values\. This cost grows linearly with the context lengthnn\. For example, for a model withL=48L=48,Hk​v=8H\_\{kv\}=8,dh=128d\_\{h\}=128, and bf16 KV states \(b=2b=2\), the KV cache requires approximately 192KB per token\. At 128K tokens, this corresponds to roughly 24GB of KV\-cache memory alone, excluding model weights, activations, temporary buffers, and runtime overhead\.

#### SeKVGPU memory cost\.

SeKVdecomposes the context into entropy\-guided spans and stores different components at different memory resolutions\. Let𝒮\\mathcal\{S\}denote the set of spans,\|𝒮\|=m\|\\mathcal\{S\}\|=m, and letaadenote the number of anchor tokens\. The GPU\-resident memory ofSeKVconsists of full\-resolution anchor KV states, compact span summaries, and the active local window\. The approximate GPU memory cost is

MemGPUSeKV≈2​L​Hk​v​dh​\(a\+w\)​b\+2​L​H​d′​m​b,\\mathrm\{Mem\}\_\{\\mathrm\{GPU\}\}^\{\\textsc\{SeKV\}\}\\approx 2LH\_\{kv\}d\_\{h\}\(a\+w\)b\+2LHd^\{\\prime\}mb,\(17\)wherewwis the local window size,HHis the number of attention heads used for routing, andd′d^\{\\prime\}is the projected summary dimension\. The first term corresponds to full\-resolution KV states for anchor tokens and local\-window tokens, while the second term corresponds to compact span summaries\. Sincea\+w≪na\+w\\ll nandd′≪dhd^\{\\prime\}\\ll d\_\{h\}, the GPU\-resident memory grows much more slowly than FullKV\.

#### SeKVCPU storage cost\.

The remaining non\-anchor token information is stored on CPU as low\-rank span bases\. For each spanSSof length\|S\|\|S\|,SeKVstores a rank\-rrapproximation of both key and value matrices\. The CPU storage cost per span is approximately

MemCPU​\(S\)=2​L​Hk​v​r​\(\|S\|\+dh\+1\)​b,\\mathrm\{Mem\}\_\{\\mathrm\{CPU\}\}\(S\)=2LH\_\{kv\}r\(\|S\|\+d\_\{h\}\+1\)b,\(18\)where the terms correspond toUSU\_\{S\},VSV\_\{S\}, andΣS\\Sigma\_\{S\}for both keys and values\. Summing over all spans gives

MemCPUSeKV=∑S∈𝒮2​L​Hk​v​r​\(\|S\|\+dh\+1\)​b\.\\mathrm\{Mem\}\_\{\\mathrm\{CPU\}\}^\{\\textsc\{SeKV\}\}=\\sum\_\{S\\in\\mathcal\{S\}\}2LH\_\{kv\}r\(\|S\|\+d\_\{h\}\+1\)b\.\(19\)This representation is approximate because truncated SVD retains only the toprrsingular directions\. Thus,SeKVavoids permanent token eviction, but it does not claim exact lossless storage of all full\-rank KV states\. Instead, it preserves recoverable low\-rank token\-level structure and reconstructs relevant spans on demand\.

#### CPU\-to\-GPU transfer cost\.

During decoding,SeKVfirst routes over GPU\-resident span summaries\. Only spans selected by the zoom\-in mechanism are fetched from CPU and reconstructed on GPU\. If𝒵t\\mathcal\{Z\}\_\{t\}is the set of zoomed spans at decoding steptt, the CPU\-to\-GPU transfer cost at that step is

Transfert=∑S∈𝒵t2​L​Hk​v​r​\(\|S\|\+dh\+1\)​b\.\\mathrm\{Transfer\}\_\{t\}=\\sum\_\{S\\in\\mathcal\{Z\}\_\{t\}\}2LH\_\{kv\}r\(\|S\|\+d\_\{h\}\+1\)b\.\(20\)This is substantially smaller than transferring full token\-level KV states whenr≪dhr\\ll d\_\{h\}\. Importantly, transfer cost depends on the number and size of zoomed spans, not on the full context length\. Since zoom\-in decisions are sparse, only a small fraction of spans are expanded at each decoding step\.

![Refer to caption](https://arxiv.org/html/2606.31145v1/x6.png)Figure 6:Additional zoom\-in behavior heatmaps forSeKVacross models and benchmarks\. Each heatmap shows the average zoom\-in rate across layers and attention heads\. Brighter cells indicate heads that more frequently trigger token\-level reconstruction from CPU\-resident span bases\. Across NIAH, RULER, and InfiniteBench, zoom\-in decisions remain sparse and concentrated in a subset of heads, especially in middle and later layers\. This pattern is consistent acrossQwen2\.5\-14B\-Instruct,Llama\-3\.1\-8B\-Instruct, andMistral\-7B\-Instruct, supporting the use of per\-head per\-layer routing thresholds\.
#### Routing and reconstruction complexity\.

The Stage 1 routing cost is proportional to the number of spans rather than the number of tokens\. For each layer and head, routing compares the current query withmmspan summaries:

𝒪​\(L​H​m​d′\)\.\\mathcal\{O\}\(LHmd^\{\\prime\}\)\.\(21\)Sincem≪nm\\ll nandd′≪dhd^\{\\prime\}\\ll d\_\{h\}, this coarse routing step is much cheaper than attending over the full token\-level context\. For zoomed spans,SeKVreconstructs token\-level KV states from low\-rank bases\. For a selected spanSS, reconstruction has approximate complexity

𝒪​\(L​Hk​v​\|S\|​r​dh\),\\mathcal\{O\}\(LH\_\{kv\}\|S\|rd\_\{h\}\),\(22\)up to a constant factor for reconstructing both keys and values\. This cost is incurred only for selected spans, so the full decoding overhead depends on the zoom\-in rate rather than the full context length\.

#### Comparison with prior compression methods\.

Token\-eviction methods such as StreamingLLM, H2O, SnapKV, and PyramidKV reduce GPU memory by keeping only a subset of token\-level KV states\. Their GPU memory is approximately

Memevict=2​L​Hk​v​dh​k​b,\\mathrm\{Mem\}\_\{\\mathrm\{evict\}\}=2LH\_\{kv\}d\_\{h\}kb,\(23\)wherekkis the number of retained tokens\. These methods are efficient because they reduce the effective cache length, but discarded KV states cannot be recovered later\. Semantic compression methods reduce memory by grouping tokens into chunks, clusters, or sentences\. Their memory cost depends on the number of semantic units retained on GPU and the representation used for each unit\.SeKVdiffers by separating routing and reconstruction: compact summaries remain on GPU for efficient span selection, while CPU\-resident low\-rank bases allow selected spans to be reconstructed at token\-level resolution\.

The main memory advantage ofSeKVcomes from replacing a full token\-level GPU cache with a variable\-resolution hierarchy\. FullKV stores every token at full resolution on GPU, giving memory cost linear innn\.SeKVstores only anchor tokens, local\-window tokens, and span summaries on GPU, while moving recoverable low\-rank span bases to CPU\. As a result, GPU memory grows primarily with the number of anchors, active local tokens, and spans, rather than with all tokens in the context\. This explains the memory\-scaling behavior observed in Figure[4](https://arxiv.org/html/2606.31145#S4.F4):SeKVavoids the steep GPU\-memory growth of FullKV while retaining the ability to dynamically recover fine\-grained token\-level information when needed\.

## Appendix EAdditional Zoom\-In Behavior Analysis

Figure[6](https://arxiv.org/html/2606.31145#A4.F6)provides additional zoom\-in behavior visualizations forSeKVacross three backbone models and three long\-context benchmarks\. Each heatmap reports the average zoom\-in rate across layers and attention heads, where brighter regions correspond to heads that more frequently trigger token\-level reconstruction\. The rows correspond to NIAH, RULER, and InfiniteBench, while the columns correspond toQwen2\.5\-14B\-Instruct,Llama\-3\.1\-8B\-Instruct, andMistral\-7B\-Instruct\.

The heatmaps show three consistent trends\. First, zoom\-in is sparse: most heads remain dark across most layers, indicating thatSeKVdoes not uniformly expand compressed spans during decoding\. Second, higher zoom\-in rates are concentrated in a limited number of heads, suggesting that only a subset of attention heads specialize in long\-range evidence recovery\. Third, zoom\-in activity is generally stronger in middle and later layers, where long\-range retrieval and semantic integration are more likely to occur\. These observations support the design of learned per\-head per\-layer thresholds\. A single global threshold would either over\-expand spans for mostly local heads or under\-expand spans for retrieval\-oriented heads, which is consistent with the performance drop observed in the fixed\-threshold ablation in Table[5](https://arxiv.org/html/2606.31145#A3.T5)\.

The benchmark\-level differences also align with the nature of each task\. NIAH produces sparse but sharp zoom\-in patterns because the task requires recovering a specific inserted fact\. RULER induces a broader routing pattern because it contains multiple controlled retrieval and reasoning tasks\. InfiniteBench shows the most distributed zoom\-in behavior, reflecting its longer and more heterogeneous inputs\. Overall, these visualizations confirm thatSeKVallocates token\-level resolution adaptively: it preserves efficiency by leaving most spans compressed, while selectively reconstructing the spans most relevant to the current decoding state\.