MLIPS：多层索引器池共享

# MLIPS: Multi-Layer Indexer Pool Sharing
Source: [https://arxiv.org/html/2610.27438](https://arxiv.org/html/2610.27438)
NVIDIA Corporation \{samkitj,spampana,shloka\}@nvidia\.com

###### Abstract

Sparse attention limits the keys used by each query, but finding those keys can still require a full\-prefix scan at every layer\. We introduceMLIPS \(Multi\-Layer Indexer Pool Sharing\), a training\-free method that shares candidate search while retaining each consumer’s own scoring and selection\. A producer publishes a top\-MMpool larger than the attention budgetKK; subsequent layers select their own top\-KKfrom that pool\. Measurements across three sparse\-attention model families show substantially higher coverage from broad pools than overlap between final selections\. We implement layerwise sharing, persistent decode reuse, and a coarse producer in vLLM for DeepSeek\-V4\-Flash\. Across four long\-context suites, three consumer layers per producer give1\.141\.14–1\.40×1\.40\\timesspeedups and14\.814\.8–27\.4%27\.4\\%lower GPU energy, with observed accuracy changes of−1\.49\-1\.49to\+0\.29\+0\.29percentage points in single runs\. More extensive sharing exposes task\-dependent quality losses\. Sampled prefill traces also reveal recurring selected KV positions across query groups and layers\. We characterize this shared working\-set structure and use it to motivate Top\-GG, an exploratory bounded pool construction for prefill\. Its deployment, task accuracy, and serving benefits remain open\.

## 1Introduction

Sparse attention makes long\-context language models more efficient by restricting each query to a small set of keys\. Yet selecting those keys can remain expensive\. In DeepSeek Sparse Attention, a Lightning Indexer scores the visible prefix before choosing the top\-KKpositions used by attention\([DeepSeek\-AI, 2025](https://arxiv.org/html/2610.27438#bib.bib5);[DeepSeek\-AI, 2026b](https://arxiv.org/html/2610.27438#bib.bib1)\)\. Attention then operates on a bounded selection, while the indexer’s search grows with context length\. Repeating that search across sparse layers and queries leaves substantial work to optimize\.

Sharing selections is one established response\. IndexCache lets intervening layers inherit a retained indexer’s top\-KK, with training\-free calibration or training\-aware distillation\([Bai et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib2)\)\. DeepSeek\-V4\.1\-Flash trains deeper indexers to re\-select from a larger pool produced by an earlier layer\([DeepSeek\-AI, 2026a](https://arxiv.org/html/2610.27438#bib.bib3)\)\. These approaches motivate a deployment question: how much candidate search can an existing checkpoint share without retraining its indexers or changing its attention operator?

Our starting observation is that*agreement on broad candidate sets is much stronger than agreement on final selections*\. On DeepSeek\-V4\-Flash traces spanning contexts up to one million tokens, an adjacent producer’s top\-512512contains2323–34%34\\%of a consumer’s independent top\-512512, depending on the suite\. A16,38416\{,\}384\-entry producer pool raises that coverage to7272–85%85\\%\. The same distinction appears in MiniMax\-M3 and GLM\-5\.1 at their native selection granularities \(Section[3](https://arxiv.org/html/2610.27438#S3)\)\. A consumer can therefore retain many of its preferred candidates without repeating the full scan, provided that it can rank the candidates for itself\.

We introduceMLIPS \(Multi\-Layer Indexer Pool Sharing\)\. A producer runs the full indexer, selects its own top\-KK, and publishes a wider top\-MMpool\. Each consumer scores those candidates with its own indexer and selects its own top\-KK\. Consumers preserve their attention computation and the checkpoint’s organization of KV state\. The restriction is in the search domain: a consumer can re\-rank pooled entries, but cannot recover an omitted one\. Layerwise sharing applies during prefill and decode\. Persistent decode reuse extends a pool across later generation steps; a coarse producer further reduces candidate\-generation work by scoring groups of keys\.

We implement these paths in vLLM and evaluate DeepSeek\-V4\-Flash on four long\-context suites\. With three consumer layers per producer and no reuse across decode steps, MLIPS gives1\.141\.14–1\.40×1\.40\\timesspeedups and14\.814\.8–27\.4%27\.4\\%lower GPU energy\. Observed accuracy changes range from−1\.49\-1\.49to\+0\.29\+0\.29points \(Table[1](https://arxiv.org/html/2610.27438#S5.T1)\)\.

The traces also expose a broader opportunity: nearby queries and sparse layers repeatedly select a shared working set of KV positions\. We characterize this structure and suggest Top\-GG, a bounded pool assembled from a query group’s producer selections \(Section[6](https://arxiv.org/html/2610.27438#S6)\)\. This is an empirical observation and an exploratory construction; turning it into a serving mechanism remains future work\.

Our contributions are:

- •Evidence for candidate reuse\.We distinguish final\-selection overlap from broad\-pool coverage across three model families and characterize its dependence on query position and sharing extent\.
- •An inference\-time sharing method and serving evaluation\.We develop MLIPS’s producer–consumer design, implement layerwise and persistent decode reuse with a coarse producer option, and measure accuracy, wall time, and GPU energy\.
- •Shared working\-set structure in prefill\.We measure recurring selected KV positions across queries and layers and outline Top\-GGas a way to exploit that structure\.

## 2Background and Related Work

#### Sparse attention and candidate search\.

An indexer assigns relevance scores to causally visible keys and selects the top\-KKindices for attention\([DeepSeek\-AI, 2025](https://arxiv.org/html/2610.27438#bib.bib5)\)\. The indexer scores and the attention weights are separate: attention computes its own weights after selection\. We denote the visible search domain byNNentries\. In DeepSeek\-V4\-Flash’s compressed sparse\-attention \(CSA\) layers, these are compressed entries, with approximatelyN=L/4N=L/4for context lengthLL, andK=512K=512\([DeepSeek\-AI, 2026b](https://arxiv.org/html/2610.27438#bib.bib1)\)\. A separate local\-attention window remains active\. A full indexer scansNNentries even though the global attention branch uses onlyKK\. This creates the opportunity to share candidate search\.

#### Pools, selections, and schedules\.

A producer publishesMMcandidate indices, whereK<M≪NK<M\\ll N; a consumer scores those candidates and selects its own top\-KK\. Selection inheritance instead directly adopts the producer’s top\-KK\. Pools contain indices, not shared attention scores or KV vectors\. We usehhfor the number of consumer sparse layers served by each producer andq​hqhfor the number of subsequent decode steps served before a rebuild\. Exact producer–consumer distance is denotedddwhen discussing an individual layer pair; sparse\-layer distance need not equal Transformer\-layer distance\. Schedule averages at differenthhcan include different producer placements and several distances\.

#### What coverage measures\.

Let𝒞K\\mathcal\{C\}\_\{K\}be a consumer’s top\-KKunder full search and let𝒫K\\mathcal\{P\}\_\{K\}and𝒫M\\mathcal\{P\}\_\{M\}be the producer’s selection and wider pool\. We measure

RK​K=\|𝒞K∩𝒫K\|K,RK​M=\|𝒞K∩𝒫M\|K\.R\_\{KK\}=\\frac\{\|\\mathcal\{C\}\_\{K\}\\cap\\mathcal\{P\}\_\{K\}\|\}\{K\},\\qquad R\_\{KM\}=\\frac\{\|\\mathcal\{C\}\_\{K\}\\cap\\mathcal\{P\}\_\{M\}\|\}\{K\}\.\(1\)Selection overlapRK​KR\_\{KK\}describes agreement between final selections\. Pool coverageRK​MR\_\{KM\}describes the consumer’s preferred entries available for re\-selection\. Nested producer sets implyRK​M≥RK​KR\_\{KM\}\\geq R\_\{KK\}\. Neither metric measures retained attention mass or downstream accuracy\. Across model families, counts use native selection units—compressed entries, blocks, or tokens—so equal pool sizes need not have equal cost\.

#### Cross\-layer reuse\.

IndexCache studies top\-KKinheritance, using calibration to choose retained indexers without weight updates or distillation to train them\([Bai et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib2)\)\. IndexShare and LongCat Sparse Attention also share selections through trained architectures\([Z\.ai, 2026b](https://arxiv.org/html/2610.27438#bib.bib6);[Meituan LongCat Team, 2026](https://arxiv.org/html/2610.27438#bib.bib16)\)\. DeepSeek\-V4\.1\-Flash is the closest pool\-based design: during decode, a full indexer constructs a wider candidate pool and deeper Reindex layers score that pool independently; the restriction is introduced during post\-training\([DeepSeek\-AI, 2026a](https://arxiv.org/html/2610.27438#bib.bib3)\)\. MLIPS evaluates pool re\-selection on existing checkpoints and extends reuse across successive decode steps\. It preserves the host checkpoint’s attention\-state organization\.

#### Coarse search and query sharing\.

HISA scores mean\-pooled block representatives before fine selection\([Xu et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib4)\)\. MLIPS’s coarse producer combines this type of filtering with pool sharing across layers\. PIVOT forms a proxy from jointly available queries and runs one shared prefix scan; its Refine variant re\-scores a broader candidate set for each query\([Liu et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib7)\)\. This establishes query\-group candidate search\. Our persistent decode pool instead serves queries that arrive at later steps\. The exploratory Top\-GGstudy examines producer\-selection memberships across query groups and layers, rather than constructing a proxy query\.

#### Routing and KV storage\.

Cross\-Layer Attention and YOCO share KV state across layers\([Brandon et al\., 2024](https://arxiv.org/html/2610.27438#bib.bib10);[Sun et al\., 2024](https://arxiv.org/html/2610.27438#bib.bib11)\); You Only Index Once combines shared routing with YOCO\([Sun et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib9)\)\. MLIPS shares candidate indices without requiring KV sharing\. Restricting the active working set may support offloading, but does not itself reduce total KV storage\. We analyze that opportunity separately in Appendix[F](https://arxiv.org/html/2610.27438#A6)\. Together, these distinctions position MLIPS as a study of training\-free candidate reuse and its measured quality–efficiency tradeoffs\.

## 3Evidence for Candidate Reuse

LB2RULERMRCR000\.250\.250\.50\.50\.750\.75110\.340\.340\.230\.230\.280\.280\.850\.850\.720\.720\.810\.81Index recallSelection overlapRK​KR\_\{KK\}Pool coverageRK​MR\_\{KM\}

\(a\) All sampled query positions in up\-to\-1M captures\.

0–200K200–500K500K–1M000\.250\.250\.50\.50\.750\.75110\.310\.310\.230\.230\.20\.20\.90\.90\.760\.760\.660\.66Index recallRK​KR\_\{KK\}RK​MR\_\{KM\}

\(b\) RULER\-1M, grouped by query position\.

Figure 1:A wider pool increases available consumer choices, but coverage declines at later positions\. DeepSeek\-V4\-Flash,h=1h=1,K=512K=512,M=16,384M=16\{,\}384; values are rounded from the capture summaries\. Bands refer to positions within prompts\. Full cross\-model results appear in Appendix[C](https://arxiv.org/html/2610.27438#A3)\.We first ask whether existing indexers provide enough shared candidate structure to justify restricting a consumer’s search\. We measure the index recalls in Equation[1](https://arxiv.org/html/2610.27438#S2.E1)on DeepSeek\-V4\-Flash, MiniMax\-M3, and GLM\-5\.1 captures\([DeepSeek\-AI, 2026b](https://arxiv.org/html/2610.27438#bib.bib1);[Lai et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib8);[Z\.ai, 2026a](https://arxiv.org/html/2610.27438#bib.bib17)\), using each family’s native selection unit\. Captures use the ten largest prompts from each of three suites\. A context band denotes*query positions within prompts*, rather than a separate prompt\-length condition\. Full heatmaps, capture details, and pooling conventions are in Appendix[C](https://arxiv.org/html/2610.27438#A3); Figure[1](https://arxiv.org/html/2610.27438#S3.F1)highlights the main trends\.

#### Broad pools retain more consumer choices\.

We use the consumer’s independently computed full\-search top\-KKas the reference\. Pool coverage measures the fraction of these indices present in the producer’s wider top\-MMpool; final\-selection overlap counts only those also in the producer’s top\-KK\. At query positions below200200K, broad pools retain8989–94%94\\%of the consumer’s choices for adjacent producer–consumer pairs across the three families and suites\. The producer’s final top\-KKretains only3131–65%65\\%\. The pools contain16,38416\{,\}384compressed entries for DeepSeek\-V4\-Flash,128128blocks for MiniMax\-M3, and16,38416\{,\}384tokens for GLM\-5\.1\.

Across query positions in up\-to\-11M captures, DeepSeek\-V4\-Flash’s pool coverage is7272–85%85\\%, compared with2323–34%34\\%final\-selection overlap\. MiniMax\-M3 gives7474–85%85\\%coverage, compared with4747–56%56\\%overlap\. Thus a wider pool makes many more of the consumer’s preferred indices available for re\-selection\. This measures candidate availability, not attention\-weight agreement or task accuracy; task\-level consequences are evaluated in Section[5](https://arxiv.org/html/2610.27438#S5)\.

#### Sharing has a context and depth boundary\.

Coverage declines toward later query positions\. On RULER\-1M, DeepSeek\-V4\-Flash’s adjacent\-layer pool coverage falls from0\.900\.90to0\.760\.76to0\.660\.66across the00–200200K,200200–500500K, and500500K–11M position bands\. Serving more consumer layers from one producer also generally lowers coverage, because the later consumers are farther from the producer\. However, each schedule averages over a different set of layer pairs\. With one consumer per producer \(h=1h=1\), for example, the third sparse layer is a producer; with two \(h=2h=2\), it consumes the first layer’s pool\. Such changes can produce local increases in the average even as sharing grows\. Pool width and sharing extent should therefore be evaluated together, checking individual layer pairs as well as the schedule average\.

#### Nearby queries share candidates too\.

We also hold a pool fixed and measure how many full\-search choices it contains for later queries in the same layer\. Across query positions in the RULER\-1M captures, DeepSeek\-V4\-Flash retains84\.6%84\.6\\%of these choices one query later and78\.5%78\.5\\%three queries later\. MiniMax\-M3 retains91\.8%91\.8\\%and89\.7%89\.7\\%at the same offsets\. For comparison, adjacent\-layer reuse at the same query gives72\.5%72\.5\\%coverage on DeepSeek\-V4\-Flash and74\.2%74\.2\\%on MiniMax\-M3\.

These measurements come from sampled bursts of four consecutive prompt positions during prefill\. They test reuse at offsets of one to three positions within each burst, not longer intervals or model\-generated continuations\. The results motivate short\-range temporal reuse, but do not show how a persistent pool affects generation quality; we test that separately in Section[5\.3](https://arxiv.org/html/2610.27438#S5.SS3)\. A stale pool can miss both new keys and older keys that become relevant\.

These findings support a method that shares broad candidates while preserving fresh consumer scoring\. The serving evaluation tests whether the resulting search restriction pays off in practice \(Section[5](https://arxiv.org/html/2610.27438#S5)\)\. Section[6](https://arxiv.org/html/2610.27438#S6)then asks how far the shared selected\-index structure extends across larger prefill query groups\.

## 4MLIPS

Producer: scoreNNkeysown top\-KK\+ candidate poolMMshared candidate indicesConsumerℓ\\ellown scorestop\-KK, own KVConsumerℓ\+1\\ell\+1own scorestop\-KK, own KV

\(a\) Share the search domain; re\-select independently\.

ℓ0\\ell\_\{0\}ℓ0\\ell\_\{0\}ℓ1\\ell\_\{1\}ℓ1\\ell\_\{1\}ℓ2\\ell\_\{2\}ℓ2\\ell\_\{2\}q\+0q\+0PCCCCCq\+1q\+1PCCCCCq\+2q\+2PCCCCCPLayer sharingPersistent decode reuse

\(b\) One pool serves a row or a tile of queries and layers\.

Figure 2:MLIPS producer–consumer design\. Orange cells build a pool; blue cells re\-score its candidates\. Dashed boxes show reuse ath=2h=2and, on the right,q​h=2qh=2\. Consumers retain independent selections, which may overlap\.### 4\.1Share candidates, retain independent selection

A producer runs the checkpoint’s indexer over the visible domain, keeps its own top\-KK, and publishes the top\-MMcandidate indices\. Each consumer gathers its own indexer keys at those positions, computes fresh scores, and selects its own top\-KKfor attention\. Only candidate addresses are shared: the consumer uses its own query, indexer keys, attention weights, and KV entries\. Figure[2](https://arxiv.org/html/2610.27438#S4.F2)summarizes the mechanism\.

The approximation is the restricted search domain\. Re\-scoring preserves the consumer’s ranking within the pool, but cannot recover an omitted key\. For nested producer sets,𝒫K⊆𝒫M\\mathcal\{P\}\_\{K\}\\subseteq\\mathcal\{P\}\_\{M\}impliesRK​M≥RK​KR\_\{KM\}\\geq R\_\{KK\}; this is a coverage property, not a guarantee of equal task accuracy\. Appendix[A](https://arxiv.org/html/2610.27438#A1)gives the formal argument and operation accounting\.

### 4\.2Layerwise and persistent decode reuse

#### Layer sharing \(1D MLIPS\)\.

We partition thennindexer\-equipped sparse layers into groups of at mosth\+1h\+1\. The first layer in each group produces a pool; the followinghhconsumers re\-select from it\. The producer count isp=⌈n/\(h\+1\)⌉p=\\lceil n/\(h\+1\)\\rceil, and the consumer count isc=n−pc=n\-p\. The final group may be shorter\. Pools are built independently for each query, so layer sharing applies during both prefill and decode\. Changinghhchanges producer placement as well as the maximum consumer distance; it is not simply a continuous increase in one fixed pair’s distance\.

#### Persistent decode pools \(2D MLIPS\)\.

A producer can retain its pool forq​hqhsubsequent decode steps\. It rebuilds once everyq​h\+1qh\+1steps and re\-scores the stored candidates in between; consumer layers also re\-score the pool at each step\. Each later query therefore receives fresh scores, although the candidate membership is older\. This persistence serves queries as they arrive, extending the reuse schedule along the query axis\.

A stored pool can omit older keys that become relevant and compressed keys created after the build\. The local\-attention window still provides access to recent tokens, but does not guarantee that restricted global selection matches full search\. Our measurements useq​h≤3qh\\leq 3and compression ratior=4r=4\. Admitting newly materialized keys and triggering adaptive refreshes are possible extensions, not evaluated corrections to these results\.

### 4\.3Indexer scoring cost

ForQQqueries andRRcandidates, the dominant dot\-product work per sequence is

I⁡\(Q,R\)=2​Hi​Di​Q​R,I\(Q,R\)=2H\_\{i\}D\_\{i\}QR,\(2\)whereHiH\_\{i\}andDiD\_\{i\}are the indexer head count and dimension\. Summing over thennindexer\-equipped layers, the full\-search baseline costsn​I​\(Q,N\)nI\(Q,N\)\. Layer sharing replaces this with

C1​D=p​I​\(Q,N\)\+c​I​\(Q,M\)\.C\_\{\\mathrm\{1D\}\}=pI\(Q,N\)\+cI\(Q,M\)\.\(3\)Averaging over one decode rebuild cycle gives the scoring\-work reduction

Rops​\(h,q​h\)=n​N​\(q​h\+1\)p​N\+\[p​q​h\+c⁡\(q​h\+1\)\]​M\.R\_\{\\mathrm\{ops\}\}\(h,qh\)=\\frac\{nN\(qh\+1\)\}\{pN\+\\left\[p\\,qh\+c\(qh\+1\)\\right\]M\}\.\(4\)For a concrete example, DeepSeek\-V4\-Flash hasn=21n=21indexer\-equipped CSA layers; this counts only the layers that perform indexed sparse selection, not all backbone layers\. Its indexer usesHi=64H\_\{i\}=64,Di=128D\_\{i\}=128, and compression ratior=4r=4\. AtL=220L=2^\{20\}tokens, the compressed domain therefore containsN=L/4=262,144N=L/4=262\{,\}144entries\. WithM=16,384M=16\{,\}384, Equation[4](https://arxiv.org/html/2610.27438#S4.E4)gives scoring\-work reductions of3\.03×3\.03\\timesfor\(h,q​h\)=\(3,0\)\(h,qh\)=\(3,0\)and3\.25×3\.25\\timesfor\(1,1\)\(1,1\)\. These ratios describe indexer dot products\. They exclude top\-KK/top\-MMselection, gathering, other score arithmetic, and memory traffic, and do not predict proportional end\-to\-end speedups\.

### 4\.4Coarse candidate generation

A*coarse producer*first searches groups of keys, then refines its selection within the retained groups\. We call this variant MLIPS with a Coarse Producer \(MLIPS\-CP\)\.

It reduces the producer scan by groupingPPconsecutive compressed keys and scoring one mean representative per group\. The producer keepsbbgroups withb​P=MbP=M, publishes their members as the candidate pool, and performs a fine selection for its own top\-KK\. Consumers perform their usual re\-selection within that pool\. This combines coarse\-to\-fine indexing\([Xu et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib4)\)with cross\-layer candidate sharing\. Its modeled reduction is

RCP​\(P,h,q​h\)=n​N​\(q​h\+1\)p​N/P\+n⁡\(q​h\+1\)​M\.R\_\{\\mathrm\{CP\}\}\(P,h,qh\)=\\frac\{nN\(qh\+1\)\}\{pN/P\+n\(qh\+1\)M\}\.\(5\)The main coarse configuration usesP=2P=2andb=8,192b=8\{,\}192\. Pooling can filter out useful groups; consumer refinement cannot restore them\. We evaluate this additional restriction separately from exact\-producer sharing\. Appendix[A](https://arxiv.org/html/2610.27438#A1)contains the parameter grids and an explanation of representative\-score dilution\.

Appendix[B](https://arxiv.org/html/2610.27438#A2)describes the vLLM integration, buffers, CUDA\-graph handling, and selection checks\.

## 5Evaluation

### 5\.1Setup

We evaluate four long\-context suites: LongBench\-v2 \(503503items\), RULER\-1M \(130130items, ten per task\), MRCR\-v2 8\-needle \(784784items\), and GraphWalks \(1,1501\{,\}150items\)\([Bai et al\., 2025](https://arxiv.org/html/2610.27438#bib.bib12);[Hsieh et al\., 2024](https://arxiv.org/html/2610.27438#bib.bib13);[OpenAI, 2025b](https://arxiv.org/html/2610.27438#bib.bib14);[OpenAI, 2025a](https://arxiv.org/html/2610.27438#bib.bib15)\)\. The serving stack uses DeepSeek\-V4\-Flash in vLLM on one NVIDIA GB300 node, with four GPUs at1,4001\{,\}400W each and tensor parallelism44\. The pool width isM=16,384M=16\{,\}384\. Chunked prefill uses an8,1928\{,\}192\-token budget, with at most2424concurrent sequences and2424requests in flight; prefix caching is off\. Expert parallelism and speculative decoding are disabled, and CUDA graphs are enabled\. Stock and MLIPS use the same checkpoint and software; configuration flags select the indexer path \(Appendix[B](https://arxiv.org/html/2610.27438#A2)\)\. Appendix[D\.1](https://arxiv.org/html/2610.27438#A4.SS1)gives precision, software, and measurement details\.

We report each suite’s task score on a00–100100scale, end\-to\-end workload speedup relative to stock, and GPU energy saved\. Energy integrates four\-GPU power samples taken every500500ms, from the end of warm\-up to workload completion; it excludes startup, compilation, and graph capture, as well as CPU, host\-memory, and network energy\. Every configuration is a single serving run\. Two stock boots differed by up to1\.3%1\.3\\%in time and0\.3%0\.3\\%in energy, with score differences of approximately one to two points or less across suites\. This limited comparison does not establish accuracy equivalence or statistical significance\. We report observed differences and retain each experiment campaign’s own baseline\.

### 5\.2Layer sharing: consistent serving gains

Table[1](https://arxiv.org/html/2610.27438#S5.T1)evaluatesh=1,2,3h=1,2,3with no persistent decode reuse \(q​h=0qh=0\)\. Every configuration is faster and uses less GPU energy than its stock run\. Ath=3h=3, speedups range from1\.139×1\.139\\timeson MRCR\-v2 to1\.398×1\.398\\timeson RULER\-1M, with energy savings of14\.814\.8–27\.4%27\.4\\%\. The corresponding score changes, in table order, are−0\.40\-0\.40,−0\.29\-0\.29,−1\.49\-1\.49, and\+0\.29\+0\.29points\. These observations support sharing a broad candidate pool while preserving each consumer’s independent selection; the search restriction still changes some task scores\.

Table 1:Layer sharing withM=16,384M=16\{,\}384andq​h=0qh=0\. Scores, speedups, and GPU energy savings use each suite’s stock run\. Every row is one serving run\.The size of the gain depends on how much work reaches the sharing path\. All RULER\-1M prompts exceed its roughly6565K\-token activation threshold, whereas about half of MRCR\-v2 prompts do not\. The speedup differences are consistent with that distinction, although the aggregate measurements do not isolate indexer time from other serving costs\. Measured speedups are therefore the relevant performance result; the operation model in Section[4](https://arxiv.org/html/2610.27438#S4)explains the opportunity rather than predicting end\-to\-end gains directly\.

Quality is not monotone inhh\. RULER\-1M loses1\.951\.95points ath=2h=2, but only0\.290\.29ath=3h=3\. GraphWalks also has a reported long\-context BFS dip ath=2h=2\(Appendix[D\.2](https://arxiv.org/html/2610.27438#A4.SS2)\); its cause remains unresolved\. Extending sharing toh=5,6h=5,6increases speed further, reaching1\.572×1\.572\\timeson RULER\-1M ath=6h=6, but its score then falls by4\.424\.42points\. The other three suites have smaller aggregate changes in these runs\. Thus the evidence favors modest sharing as the initial operating region, while deeper schedules require workload\-specific quality checks\. Appendix[D\.2](https://arxiv.org/html/2610.27438#A4.SS2)retains all extended configurations\.

Energy and latency also need separate measurements\. Ath=1h=1, mean GPU power falls by2\.22\.2–5\.2%5\.2\\%, so energy savings exceed the fraction of wall time saved\. At deeper schedules, mean power can rise even as the run finishes sooner\. Lower indexer work consequently does not imply a fixed power reduction\.

### 5\.3Decode reuse: a modest gain with quality risks

Persistent reuse across decode steps \(Section[4\.2](https://arxiv.org/html/2610.27438#S4.SS2.SSS0.Px2)\) adds little performance benefit on the four long\-context suites\. Withh=0h=0, allq​h=1,2,3qh=1,2,3runs achieve only0\.9600\.960–1\.000×1\.000\\timesstock speed; adding reuse to layer sharing changes time little while sometimes reducing task scores substantially\. At\(h,q​h\)=\(3,2\)\(h,qh\)=\(3,2\), score changes are−1\.79\-1\.79,−9\.68\-9\.68,−5\.29\-5\.29, and−3\.01\-3\.01points on LongBench\-v2, RULER\-1M, MRCR\-v2, and GraphWalks, respectively\. The full grid appears in Table[6](https://arxiv.org/html/2610.27438#A4.T6)and its continuations\. These results show that query locality alone does not justify a fixed refresh schedule for all workloads\.

To isolate decode, we use3030synthetic prompts of approximately0\.90\.9M tokens, offering2424sessions with prefix caching disabled\. We subtract the wall time of a one\-token generation from that of a40,00040\{,\}000\-token generation\. Table[2](https://arxiv.org/html/2610.27438#S5.T2)reports the mean of two draws after discarding a warm\-up draw, with one boot per arm andh=0h=0throughout\.

Table 2:Isolated decode ath=0h=0\. Changes are relative to stock\. This synthetic timing experiment does not measure answer quality\.Atq​h=3qh=3, reuse reduces the measured decode time by3\.3%3\.3\\%; atq​h=1qh=1, it increases time by1\.2%1\.2\\%\. This supports a limited performance opportunity for longer generations, but neither combined layer\-and\-query speedup nor generated\-answer quality is established by this instrument\. A practical refresh policy must address newly relevant existing keys as well as keys appended since the pool was built\.

### 5\.4Coarse producers: energy savings and failure cases

1122335566−4\-4−2\-20022Layer\-sharing schedulehhAccuracy change \(points\)LB2RULERMRCRGraphWalks

\(a\) Exact producer,q​h=0qh=0\.

24816−15\-15−10\-10−5\-500Coarse group sizePPAccuracy change \(points\)LB2RULERMRCRGraphWalks

\(b\) Coarse producer,h=1h=1,q​h=0qh=0, fixedMM\.

Figure 3:Observed quality limits of more extensive sharing and coarse filtering\. Each configuration is one serving run; changes use the corresponding campaign’s stock baseline\. Curves connect measured settings and are not uncertainty estimates\. Full timing, energy, and accuracy values are in Appendix[D\.2](https://arxiv.org/html/2610.27438#A4.SS2)\.The coarse producer replaces the dense producer scan with group representatives \(Section[4\.4](https://arxiv.org/html/2610.27438#S4.SS4)\)\. AtP=2P=2,b=8,192b=8\{,\}192,h=1h=1, andq​h=0qh=0, its four speedups are1\.0271\.027,1\.2291\.229,0\.9790\.979, and1\.087×1\.087\\times, with GPU energy savings of10\.210\.2,22\.522\.5,7\.77\.7, and16\.5%16\.5\\%, respectively\. Score changes against this campaign’s baseline are−0\.60\-0\.60,\+0\.28\+0\.28,−0\.75\-0\.75, and−0\.01\-0\.01points\. Energy falls even on MRCR\-v2, where latency worsens\. Relative to the separately measured exact producer, the coarse path is not consistently faster\.

More aggressive settings expose substantial failures\. AtP=2P=2, RULER\-1M loses5\.715\.71points ath=2h=2, and MRCR\-v2 loses3\.303\.30ath=3h=3\. IncreasingPPfrom22to1616at fixedh=1h=1andb​P=MbP=Mraises energy savings on every suite, but theP=16P=16score changes are\+0\.39\+0\.39,−14\.30\-14\.30,−3\.47\-3\.47, and−2\.52\-2\.52points\. Replacing mean representatives by elementwise maxima does not remove the RULER\-1M failure\. Tables[10](https://arxiv.org/html/2610.27438#A4.T10)–[12](https://arxiv.org/html/2610.27438#A4.T12)give all settings and raw measurements\. Figure[3](https://arxiv.org/html/2610.27438#S5.F3)shows the quality changes for deeper layer sharing and larger coarse groups\.

Coarse\-producer results use a separate campaign baseline and one run per configuration\. Attributing the observed losses to representative dilution requires verified matched controls and repeats\. Exact layer sharing is the clearest measured benefit; decode reuse and coarse production expose additional workload\-dependent tradeoffs\.

## 6Shared Candidate Working Sets

MLIPS shares a candidate pool across layers for each query\. The traces also suggest that nearby queries repeatedly select a common set of KV positions\. This raises a broader question: can a bounded pool represent the working set of a group of queries across several sparse layers? We investigate this structure through an exploratory*Top\-GG*construction\. The evidence concerns shared selected positions; it does not establish similarity among the corresponding KV vectors\.

Letℓ\\ellindex CSA layers, and letKqℓK\_\{q\}^\{\\ell\}andMqℓM\_\{q\}^\{\\ell\}be the producer’s Top\-KKand Top\-MMindex sets for queryqq\. For a group of recorded queriesCC, we construct one pool

𝒢ℓ,C⊆⋃q∈C\(Kqℓ∪Mqℓ\),\|𝒢ℓ,C\|≤G\.\\mathcal\{G\}\_\{\\ell,C\}\\subseteq\\bigcup\_\{q\\in C\}\(K\_\{q\}^\{\\ell\}\\cup M\_\{q\}^\{\\ell\}\),\\qquad\|\\mathcal\{G\}\_\{\\ell,C\}\|\\leq G\.\(6\)The constructor prioritizes producer Top\-KKmemberships from queries whose selections remain poorly covered, using residual Top\-MMmemberships to break ties\. This provides a concrete probe of shared structure without using selections from consumer layers\. Appendix[E](https://arxiv.org/html/2610.27438#A5)gives the algorithm and sampling protocol\. We freeze the pool and measure

Rℓ,q​\(d\)=\|Kqℓ\+d∩𝒢ℓ,C\|\|Kqℓ\+d\|,d≥1,R\_\{\\ell,q\}\(d\)=\\frac\{\|K\_\{q\}^\{\\ell\+d\}\\cap\\mathcal\{G\}\_\{\\ell,C\}\|\}\{\|K\_\{q\}^\{\\ell\+d\}\|\},\\qquad d\\geq 1,\(7\)whereddcounts subsequent CSA layers, separated by two physical model layers in these traces\.

Nominalquery windowSubsequent CSA\-layer distance123456Mean4,09616,38465,536Sequence82\.479\.479\.476\.974\.373\.577\.674\.171\.171\.268\.666\.065\.569\.457\.154\.154\.652\.049\.549\.852\.936\.534\.535\.532\.730\.631\.733\.6Top\-KKindex recall \(%\);G=32,000G=32\{,\}000

Figure 4:A shared pool covers many subsequent selections, with coverage decreasing as its query window grows\. Results use sampled queries from LB2 sequences 0–6; window sizes describe original positions, not the number of measured queries\. Mean averages distances 1–6\.The capture contains eight LB2 sequences and 21 CSA layers\. The primary results cover sequences 0–6, withK=512K=512, nominalM=16​KM=16\\mathrm\{K\},G=32,000G=32\{,\}000, and context length at leastGG\. Crucially, the trace records four queries per 256 original positions: a nominal 4,096\-position window contains 64 recorded queries, not 4,096 measured queries\.

Figure[4](https://arxiv.org/html/2610.27438#S6.F4)shows a shared working set across both axes\. For the smallest window, recall decreases from82\.4%82\.4\\%at the next CSA layer to73\.5%73\.5\\%six layers later, averaging77\.6%77\.6\\%\. Increasing the nominal window to 16,384 or 65,536 positions reduces mean recall to69\.4%69\.4\\%or52\.9%52\.9\\%; sequence\-wide sharing reaches33\.6%33\.6\\%\. Thus a fixed pool captures recurring selections but cannot absorb all query\-specific variation\. Per\-sequence measurements show the same window ordering \(Appendix[E](https://arxiv.org/html/2610.27438#A5)\)\.

These observations motivate consolidating producer selections into a shared Top\-GGpool, with each consumer scoring and selecting independently\. Shared index metadata and coordinated KV staging are possible uses\. Their benefits require measurement:G\>MG\>Mcan increase consumer scoring, and index coverage alone establishes neither task accuracy nor speedup\. Construction over a whole query group can also depend on later queries; a serving design must specify information availability and causal masks\. The present result is evidence for exploitable structure and a proposed way to represent it\. It leaves deployment and full\-query validation open\.

## 7Limitations and Future Work

#### Evidence and generalization\.

Serving results cover one checkpoint and hardware stack; the cross\-model evidence concerns index coverage\. Each serving configuration is a single run\. Two stock boots provide a limited variability reference, not an equivalence test or uncertainty estimate for every comparison\. Repeated paired evaluations and matched implementations of the closest reuse baselines are needed to establish robustness and comparative benefit\. The operation model also excludes selection, gathering, and memory costs; phase\-level profiling would clarify why scoring reductions translate unevenly into latency savings\.

#### Adaptive reuse\.

Uniform schedules expose weak layer pairs and unexplained quality changes, including theh=2h=2dip\. A useful next step is to select producer locations and pool widths using calibration, then evaluate the chosen schedule on held\-out workloads\. Persistent decode reuse needs refresh rules that balance pool age, new\-key admission, and shifts in query relevance\. Generated\-token fidelity and task accuracy should accompany timing results; prefill locality alone cannot settle those questions\.

#### From shared structure to Top\-GG\.

The prefill traces identify recurring selected positions, without establishing KV\-vector similarity or a deployed compression scheme\. Testing all query rows, additional prompts, and alternative constructors would show how robust the apparent working sets are\. A serving design must account for pool\-construction cost, causal information flow, per\-query masking, task accuracy, and the cost of retrieving pooled entries\. The current sampled recall measurements do not establish a serving speedup\.

#### Memory management\.

Candidate pools may bound a consumer’s immediate working set, but realizing lower GPU residency requires cache management and transfers\. The offloading analysis in Appendix[F](https://arxiv.org/html/2610.27438#A6)is therefore a capacity opportunity rather than a measured memory reduction\. Future evaluation should report peak resident memory and transfer overhead together with latency and quality\.

## 8Conclusion

MLIPS shares candidate search across sparse\-attention layers while retaining each consumer’s own scoring and selection\. Broad producer pools cover substantially more consumer choices than final\-selection overlap suggests across three model families\. On DeepSeek\-V4\-Flash, three consumer layers per producer give1\.141\.14–1\.40×1\.40\\timesserving speedups and14\.814\.8–27\.4%27\.4\\%lower GPU energy across four suites, with observed accuracy changes of−1\.49\-1\.49to\+0\.29\+0\.29points in single runs\. More extensive sharing exposes quality limits\. Recurring selected KV positions across prefill queries and layers further motivate Top\-GG, whose construction and deployment require continued study\. The results identify candidate working sets as a useful unit for inference\-time reuse\.

## References

- Baiet al\.\(2026\)Y\. Bai, Q\. Dong, T\. Jiang, X\. Lv, Z\. Du, A\. Zeng, J\. Tang, and J\. LiIndexCache: accelerating sparse attention via cross\-layer index reuse\.arXiv preprint arXiv:2603\.12201\.External Links:[Link](https://arxiv.org/abs/2603.12201)Cited by:[§1](https://arxiv.org/html/2610.27438#S1.p2.1),[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px4.p1.1)\.
- Baiet al\.\(2025\)Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong,et al\.LongBench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3639–3664\.Cited by:[§5\.1](https://arxiv.org/html/2610.27438#S5.SS1.p1.1)\.
- Brandonet al\.\(2024\)W\. Brandon, M\. Mishra, A\. Nrusimha, R\. Panda, and J\. Ragan\-KelleyReducing transformer key\-value cache size with cross\-layer attention\.Advances in Neural Information Processing Systems37,pp\. 86927–86957\.Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px6.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-V3\.2: pushing the frontier of open large language models\.arXiv preprint arXiv:2512\.02556\.External Links:[Link](https://arxiv.org/abs/2512.02556)Cited by:[§1](https://arxiv.org/html/2610.27438#S1.p1.1),[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2026a\)DeepSeek\-AIDeepSeek\-V4\.1\-Flash: pushing the limits of KV cache compression\.arXiv preprint arXiv:2609\.19969\.External Links:[Link](https://arxiv.org/abs/2609.19969)Cited by:[§1](https://arxiv.org/html/2610.27438#S1.p2.1),[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px4.p1.1)\.
- DeepSeek\-AI \(2026b\)DeepSeek\-AIDeepSeek\-v4: towards highly efficient million\-token context intelligence\.Technical reportDeepSeek\.Note:Model checkpoints:[https://huggingface\.co/collections/deepseek\-ai/deepseek\-v4](https://huggingface.co/collections/deepseek-ai/deepseek-v4)Cited by:[Appendix C](https://arxiv.org/html/2610.27438#A3.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.27438#S1.p1.1),[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.27438#S3.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. GinsburgRULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,Cited by:[§5\.1](https://arxiv.org/html/2610.27438#S5.SS1.p1.1)\.
- Laiet al\.\(2026\)X\. Lai, W\. Xu, Y\. Yang, Q\. Chen, Y\. Xu, L\. Zeng, X\. Li, H\. Sun, H\. Zhu, V\. Zhang, J\. Hu, J\. Li, R\. Gao, Z\. Li, S\. Zhu, J\. Zhou, and P\. ZhaoMiniMax sparse attention\.arXiv preprint arXiv:2606\.13392\.External Links:[Link](https://arxiv.org/abs/2606.13392)Cited by:[Appendix C](https://arxiv.org/html/2610.27438#A3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.27438#S3.p1.1)\.
- Liuet al\.\(2026\)H\. Liu, Y\. Cheng, L\. Niu, Y\. Su, Y\. Xue, A\. Liu, G\. Yu, and J\. ZhuPIVOT: efficient query\-group indexing for token\-level sparse attention\.arXiv preprint arXiv:2607\.24593\.External Links:[Link](https://arxiv.org/abs/2607.24593)Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px5.p1.1)\.
- Meituan LongCat Team \(2026\)Meituan LongCat TeamLongCat sparse attention: taming the lightning via streaming\-aware hierarchical cross\-layer indexing\.arXiv preprint arXiv:2608\.01662\.External Links:[Link](https://arxiv.org/abs/2608.01662)Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px4.p1.1)\.
- OpenAI \(2025a\)OpenAIGraphWalks\.Note:DatasetExternal Links:[Link](https://huggingface.co/datasets/openai/graphwalks)Cited by:[§5\.1](https://arxiv.org/html/2610.27438#S5.SS1.p1.1)\.
- OpenAI \(2025b\)OpenAIMulti\-round coreference resolution \(mrcr\)\.Note:[https://huggingface\.co/datasets/openai/mrcr](https://huggingface.co/datasets/openai/mrcr)Cited by:[§5\.1](https://arxiv.org/html/2610.27438#S5.SS1.p1.1)\.
- Sunet al\.\(2024\)Y\. Sun, L\. Dong, Y\. Zhu, S\. Huang, W\. Wang, S\. Ma, Q\. Zhang, J\. Wang, and F\. WeiYou only cache once: decoder\-decoder architectures for language models\.Advances in Neural Information Processing Systems37,pp\. 7339–7361\.Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px6.p1.1)\.
- Sunet al\.\(2026\)Y\. Sun, Y\. Zhang, L\. Dong, J\. Wang, and F\. WeiYou only index once: cross\-layer sparse attention with shared routing\.arXiv preprint arXiv:2606\.06467\.External Links:[Link](https://arxiv.org/abs/2606.06467)Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px6.p1.1)\.
- Xuet al\.\(2026\)Y\. Xu, F\. Meng, F\. Jiang, Y\. Wang, R\. Zhou, Z\. Wang, J\. Wu, Z\. Pan, X\. Tang, W\. Pei, T\. Liu, D\. Yin, X\. Sun, and M\. ZhangHISA: efficient hierarchical indexing for fine\-grained sparse attention\.arXiv preprint arXiv:2603\.28458\.External Links:[Link](https://arxiv.org/abs/2603.28458)Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px5.p1.1),[§4\.4](https://arxiv.org/html/2610.27438#S4.SS4.p2.1)\.
- Z\.ai \(2026a\)Z\.aiGLM\-5\.1: model card\.Note:[https://huggingface\.co/zai\-org/GLM\-5\.1](https://huggingface.co/zai-org/GLM-5.1)Released 2026\-04\-07; architecture as described in the GLM\-5 technical report \(arXiv:2602\.15763\)Cited by:[Appendix C](https://arxiv.org/html/2610.27438#A3.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2610.27438#S3.p1.1)\.
- Z\.ai \(2026b\)Z\.aiGLM\-5\.2: built for long\-horizon tasks\.Note:Official technical release postExternal Links:[Link](https://huggingface.co/blog/zai-org/glm-52-blog)Cited by:[§2](https://arxiv.org/html/2610.27438#S2.SS0.SSS0.Px4.p1.1)\.

## Appendix AScoring Model and Detailed Reductions

### A\.1Indexer arithmetic and its scope

For query positionttand visible keyjj, the Lightning Indexer computes

s\[t,j\]=∑u=1Hiw\[t,u\]ReLU\(⟨𝐪\[t,u,:\],𝐤\[j,:\]⟩\)\.s\[t,j\]=\\sum\_\{u=1\}^\{H\_\{i\}\}w\[t,u\]\\,\\operatorname\{ReLU\}\\\!\\left\(\\langle\\mathbf\{q\}\[t,u,:\],\\mathbf\{k\}\[j,:\]\\rangle\\right\)\.\(8\)There is one key vector per indexed position shared across query heads\. ScoringQQqueries againstRRcandidates has matrix dimensions

\[B,Q​Hi,Di\]​\[B,Di,R\]⟶\[B,Q​Hi,R\],\[B,QH\_\{i\},D\_\{i\}\]\\,\[B,D\_\{i\},R\]\\ \\longrightarrow\\ \[B,QH\_\{i\},R\],\(9\)whereBBis the sequence batch size\. Counting multiply–accumulate as two operations givesI⁡\(Q,R\)=2​Hi​Di​Q​RI\(Q,R\)=2H\_\{i\}D\_\{i\}QRper sequence\. DeepSeek\-V4\-Flash usesHi=64H\_\{i\}=64,Di=128D\_\{i\}=128,n=21n=21CSA layers,r=4r=4,K=512K=512, and a local window ofW=128W=128tokens\. Thus one query–candidate dot\-product calculation costs16,38416\{,\}384operations in this model\.

ReLU, weighting, and head reduction addO⁡\(Hi​Q​R\)O\(H\_\{i\}QR\)arithmetic\. Selection, gathering, cache access, and transfers are also excluded from the ratios\. Full prefill has quadratic scoring growth inLL; decode has linear growth in cached length per query\. The rectangular model approximates causal pair counts and boundary effects\. All reduction tables useL=220L=2^\{20\}tokens,N=L/r=262,144N=L/r=262\{,\}144,M=16,384M=16\{,\}384, and henceN/M=16N/M=16\. This makes the nominal 1M point explicit; the ratios are not independent of context length\.

### A\.2Coverage under nested pools

###### Proposition 1\(Nested\-pool coverage\)\.

For fixed producer and consumer score vectors, if𝒫K⊆𝒫M\\mathcal\{P\}\_\{K\}\\subseteq\\mathcal\{P\}\_\{M\}, then\|𝒞K∩𝒫K\|≤\|𝒞K∩𝒫M\|\|\\mathcal\{C\}\_\{K\}\\cap\\mathcal\{P\}\_\{K\}\|\\leq\|\\mathcal\{C\}\_\{K\}\\cap\\mathcal\{P\}\_\{M\}\|\.

###### Proof\.

Intersecting nested sets with the same set preserves inclusion\. ∎

This bounds index coverage only\. It does not compare attention distributions or task accuracy, and it does not apply to independently constructed pools unless the required nesting holds\.

### A\.3Layer and query schedules

Forp=⌈n/\(h\+1\)⌉p=\\lceil n/\(h\+1\)\\rceilandc=n−pc=n\-p, a decode cycle containsq​h\+1qh\+1steps\. Each producer scoresNNonce andMMon the remainingq​hqhsteps; each consumer scoresMMon every step\. The cycle work is proportional top​N\+\[p​q​h\+c⁡\(q​h\+1\)\]​MpN\+\[p\\,qh\+c\(qh\+1\)\]M, yielding Equation[4](https://arxiv.org/html/2610.27438#S4.E4)\.

For uniform groups withh\+1h\+1dividingnn, a pool servesT=\(h\+1\)​\(q​h\+1\)T=\(h\+1\)\(qh\+1\)slots\. The reduction becomes

R⁡\(T\)=T​NN\+\(T−1\)​M,limT→∞R⁡\(T\)=N/M\.R\(T\)=\\frac\{TN\}\{N\+\(T\-1\)M\},\\qquad\\lim\_\{T\\to\\infty\}R\(T\)=N/M\.\(10\)A2×22\\times 2uniform tile gives3\.37×3\.37\\times, while the exact 21\-layer\(h,q​h\)=\(1,1\)\(h,qh\)=\(1,1\)schedule gives3\.25×3\.25\\times\. For 1D sharing ath=1,2,3h=1,2,3, the exact producer counts are11,7,611,7,6and the consumer counts are10,14,1510,14,15\. Table[3](https://arxiv.org/html/2610.27438#A1.T3)includes 1D in itsq​h=0qh=0column\.

Table 3:Modeled indexer scoring\-work reduction for the exact 21\-layer stack\. Theq​h=0qh=0column gives layerwise reuse alone\. Values are reduction factors\.
### A\.4Coarse\-producer accounting

The producer scoresN/PN/Prepresentatives and performs its own fine pass overb​P=MbP=Mcandidates\. Its build\-step cost isI⁡\(Q,N/P\)\+I⁡\(Q,M\)I\(Q,N/P\)\+I\(Q,M\)\. Every layer performs one fine pass on every step, so the cycle cost is proportional top​N/P\+n⁡\(q​h\+1\)​MpN/P\+n\(qh\+1\)M, yielding Equation[5](https://arxiv.org/html/2610.27438#S4.E5)\. The grids in Table[4](https://arxiv.org/html/2610.27438#A1.T4)include configurations with substantial observed quality losses; a large modeled reduction is not an operating\-point recommendation\.

keysNNblocks ofBBscore repsrepsN/BN/Bkeep top\-bbgathercandidatesb​BbBrefineoutputtop\-kkb​B\>kbB\>k: refine; candidates discarded

\(a\) HISA: candidates discarded after refine

keysNNgroups ofPPscore repsrepsN/PN/Pkeep top\-bbgathertop\-MMpoolpublishedrefineown outputtop\-KKb​P=MbP=M: pool = candidates; then refine

\(b\) MLIPS\-CP: candidates published as the*pool*, then refine

Figure 5:Coarse\-to\-fine search and shared candidate production\. Both score representatives and gather the highest\-scoring groups\. HISA refines the gathered candidates for one layer; MLIPS\-CP publishes them as a pool and retains a fine selection for each producer and consumer\. Group sizes differ in practice; the diagram uses equal granularity for illustration\.Table 4:Modeled scoring\-work reductions with a coarse producer,b=M/Pb=M/P\. Values are factors relative to full indexing; representative construction and memory costs are excluded\.#### Representative\-score dilution\.

For a single head and a mean representative𝐤¯=P−1​∑j𝐤j\\bar\{\\mathbf\{k\}\}=P^\{\-1\}\\sum\_\{j\}\\mathbf\{k\}\_\{j\}, convexity gives

ReLU⁡\(⟨𝐪,𝐤¯⟩\)≤P−1​∑jReLU⁡\(⟨𝐪,𝐤j⟩\)≤maxj⁡ReLU⁡\(⟨𝐪,𝐤j⟩\)\.\\begin\{split\}\\operatorname\{ReLU\}\(\\langle\\mathbf\{q\},\\bar\{\\mathbf\{k\}\}\\rangle\)&\\leq P^\{\-1\}\\sum\_\{j\}\\operatorname\{ReLU\}\(\\langle\\mathbf\{q\},\\mathbf\{k\}\_\{j\}\\rangle\)\\\\ &\\leq\\max\_\{j\}\\operatorname\{ReLU\}\(\\langle\\mathbf\{q\},\\mathbf\{k\}\_\{j\}\\rangle\)\.\\end\{split\}\(11\)Cancellation can suppress a representative even when its group contains a high\-scoring key\. This single\-head inequality describes a possible ranking failure; it does not prove that full multi\-head rankings or task accuracy degrade monotonically inPP\. Such a claim requires score\-level evidence, the relevant head\-weight assumptions, and matched serving controls\.

## Appendix BImplementation Details

We implement producer and consumer kernels for prefill and decode in vLLM’s DeepSeek\-V4 sparse\-attention path\. The checkpoint weights, attention operator, scheduler, and KV\-cache path are retained\. Baseline and MLIPS use the same software, with configuration flags selecting the indexer path\.

#### Integration and buffers\.

The implementation overrides the Lightning\-Indexer selection stage and a gather module on vLLM’s DeepSeek\-V4 path\. Producers and consumers have separate prefill and decode kernel sites\. An exact producer obtains its own top\-KKand the wider top\-MMpool from one scoring pass; a consumer gathers and scores its own keys at the pooled positions\. A pool row containsMMint32indices:6464KiB atM=16,384M=16\{,\}384\. Total metadata depends on query\-row and persisted\-pool counts, even though each row’s size is fixed at a givenMM\.

During prefill, producers overwrite a transient candidate buffer before their consumers use it\. During decode, per\-producer buffers retain pools across rebuild cycles\. Rows with at mostMMcompressed keys use stock indexing, corresponding to4​M=65,5364M=65\{,\}536tokens atr=4r=4\. This threshold helps explain why workloads with more short contexts obtain smaller gains\.

#### Coarse representatives\.

Representatives are computed in fp32 from dequantized keys and quantized into the indexer key format\. In prefill, running sums track keys appended to each group\. In decode, completed representatives are addressed through the KV\-page block table, with a running sum for an incomplete group\. OneDi=128D\_\{i\}=128MXFP4 representative uses6464value bytes and four scale bytes\. There is one representative perPPkeys; actual allocated overhead additionally depends on page layout and temporary state\. The cache can be replaced by on\-the\-fly pooling at the cost of rereading the original keys\. Coarse production is used above32,76832\{,\}768compressed entries \(131,072 tokens\); shorter rows use exact production\.

#### Decode graphs and recency\.

The persistent path captures rebuild and reuse variants separately\. A field in the batch descriptor selects the replay outside the CUDA graph, avoiding a Python decision fixed at capture time\. Evaluation disables speculative decoding\. Within\-step speculative grouping and reuse across successive steps are different schedules; their combined acceptance behavior and performance are not measured here\.

For longer reuse windows, one possible policy is to admit newly materialized compressed keys and remove low\-priority pool entries\. Such a policy needs its own selection rule and validation\. It does not address older keys that become relevant after the pool was built\.

#### Selection checks\.

The exact producer’s top\-KKdiffers from stock by at most0\.02%0\.02\\%on the million\-token prompts in the reported check, with mismatches attributed to equal scores\. A probe at\(h,q​h\)=\(1,1\)\(h,qh\)=\(1,1\)performs a full scan on reuse steps and compares it with selection from the stored pool\. Over ten RULER\-1M prompts, mean overlap is0\.9740\.974and median overlap is1\.0001\.000\. This quantifies the probe’s selection fidelity; it is not an end\-task accuracy test or a general guarantee against implementation errors\.

## Appendix CFull Recall Measurements

#### Captures and native units\.

The recall study uses the ten largest prompts from each of LongBench\-v2, RULER\-1M, and MRCR\-v2 for each captured model family\. It records four consecutive query positions every 256 positions\. DeepSeek\-V4\-Flash indexes compressed entries and selectsK=512K=512; MiniMax\-M3 selects 16 blocks of 128 tokens; GLM\-5\.1 selects 2,048 tokens\([DeepSeek\-AI, 2026b](https://arxiv.org/html/2610.27438#bib.bib1);[Lai et al\., 2026](https://arxiv.org/html/2610.27438#bib.bib8);[Z\.ai, 2026a](https://arxiv.org/html/2610.27438#bib.bib17)\)\. The configured broad pools contain 16,384 compressed entries, 128 blocks, and 16,384 tokens, respectively\. These widths describe different token coverage and are not equal\-cost comparisons\. GLM\-5\.1 captures end near 198K tokens; the other two families include up\-to\-1M inputs\.

#### Aggregation and distances\.

The layer\-sharing surfaces average over consumers in a schedule withhhconsumers per producer\. Producers are excluded because their own coverage is trivial\. Forh\>1h\>1, a cell averages multiple exact layer distancesddand a different producer placement from neighboring schedules\. The query\-axis surfaces instead use explicit producer–consumer layer and query offsets\. Their same\-query column therefore need not exactly reproduce a schedule\-average cell\.

Position bands are ranges within prompts\. A long prompt contributes queries to several bands; the below\-200K figure is a subset of the all\-position figure\. The broad\-pool columns use nested prefixes of a producer ranking when available\. MiniMax\-M3 forces a local block into its selection\. Its effective\-containment convention counts the guaranteed local hit; the difference from unadjusted containment is about 0\.010–0\.016 at 128 blocks and at most 0\.035 at narrow widths\.

#### Scope of the evidence\.

These are count\-based index recalls on sampled prefill activations, not attention\-mass retention\. Four\-query bursts support query offsets only up to three\. Within a fixed, identical causal prefix, prefill and decode evaluate the same indexing computation, but generated\-token distributions and multi\-step approximation effects require separate tests\. The decode probe in Appendix[B](https://arxiv.org/html/2610.27438#A2)provides a limited additional check\.

![Refer to caption](https://arxiv.org/html/2610.27438v1/mlips-recallA.png)Figure 6:Layer\-sharing recall below 200K query positions\. Rows are model families, columns are suites, and each panel varies schedulehhand producer\-pool widthMM\. The rightmost column is the native top\-KKselection overlap; wider columns show pool coverage\. Units differ across families as labeled\.![Refer to caption](https://arxiv.org/html/2610.27438v1/mlips-recallB.png)Figure 7:Layer\-sharing recall over all sampled positions in up\-to\-1M captures\. Broad candidate pools retain more consumer choices than final\-selection inheritance on these captures; neither measurement alone establishes downstream accuracy\. Axes follow Figure[6](https://arxiv.org/html/2610.27438#A3.F6)\.![Refer to caption](https://arxiv.org/html/2610.27438v1/mlips-queryA.png)Figure 8:Joint layer/query candidate coverage below 200K query positions\. Each panel varies exact sparse\-layer distance \(labeledhhin the original plot,ddin the manuscript\) and query offsetq​hqhat the configured broad pool\. The zero\-layer\-distance row isolates cross\-query reuse\.![Refer to caption](https://arxiv.org/html/2610.27438v1/mlips-queryB.png)Figure 9:Joint layer/query candidate coverage over all sampled positions in up\-to\-1M captures\. Exact layer distance is labeledhhin this original plot\. The self pair is 1\.00; later queries retain substantial but imperfect coverage\. The captured query offsets end at three\.
#### Schedule variation\.

Larger sharing groups generally lower coverage, but averages can rise locally when producer placements change\. For example, MiniMax\-M3 at its 128\-block pool gives schedule\-average coverage of 0\.81 ath=16h=16and 0\.89 ath=19h=19\. These observations motivate evaluating the actual schedule, rather than treatinghhas a smooth distance parameter\. Original plots display schedules throughh=20h=20; captures extend toh=56h=56on MiniMax\-M3 andh=77h=77on GLM\-5\.1\.

## Appendix DServing Setup and Complete Results

### D\.1Model, software, and measurement protocol

The served checkpoint is the released DeepSeek\-V4\-Flash\. Its routed MoE expert weights are FP4, with E8M0 scales over blocks of3232values, executed through vLLM’s MXFP4 kernels\. Attention projections, the shared expert, and the indexer query projection use FP8 E4M3 weights with128×128128\\times 128blocks\. The MoE router, indexer compressor, and normalization weights are bf16\. Computation runs in bf16, with activations quantized on the fly to FP8 for the FP8 GEMMs and to MXFP8 for the expert GEMMs\. The KV cache is FP8 and the indexer key cache is FP4\.

We use vLLM 0\.22\.1 in the NVIDIA vLLM 26\.06 container\. Each run occupies four GPUs of a single NVIDIA GB300 node, with a1,4001\{,\}400W power limit per GPU and tensor parallelism44\. Expert parallelism and speculative decoding are off; CUDA graphs are on\. The chunked\-prefill budget is8,1928\{,\}192tokens, the maximum number of concurrent sequences is2424, the context limit is1,048,5761\{,\}048\{,\}576tokens, and prefix caching is off\. The client maintains2424requests in flight\. The exact\-producer pool hasM=16,384M=16\{,\}384entries\. Rows below the pooling threshold use the stock path as described in Appendix[B](https://arxiv.org/html/2610.27438#A2)\.

The four suites contain503503LongBench\-v2 items,130130RULER\-1M items \(ten in each of1313task categories\),784784MRCR\-v2 8\-needle items, and1,1501\{,\}150GraphWalks items\. Scores are reported on a00–100100scale, withΔ\\Deltaacc in percentage points\. Speedup is stock workload wall time divided by configuration wall time\. Energy saved is one minus the configuration\-to\-stock GPU\-energy ratio\. TheΔ\\Deltapower column is the relative change in mean total GPU power\. Retaining these separate quantities matters because shorter runs need not have lower mean power\.

Each GPU’s power is sampled every500500ms withnvidia\-smi; energy integrates the four\-GPU total from the end of warm\-up to workload completion\. Boot, JIT compilation, and CUDA\-graph capture are excluded\. The measurement does not include host CPUs, host memory, or network equipment\. A second stock boot gives a limited reproducibility reference: suite scores differed by approximately one to two points or less, wall time by at most1\.3%1\.3\\%, and GPU energy by at most0\.3%0\.3\\%\. Every configuration row remains a single serving run, so these two stock boots do not provide uncertainty estimates for treatment effects\.

The stock configurations disable all MLIPS paths by flag\. The exact sharing grid and max\-representative experiment use the first stock rows shown in Tables[5](https://arxiv.org/html/2610.27438#A4.T5)and[12](https://arxiv.org/html/2610.27438#A4.T12)\. The mean\-representative and group\-size campaign uses the separate stock rows shown in Tables[10](https://arxiv.org/html/2610.27438#A4.T10)and[11](https://arxiv.org/html/2610.27438#A4.T11)\. We preserve these baselines rather than pooling measurements across campaigns\. The coarse\-producer comparisons do not include matched repeats or complete per\-arm launch configurations in the current reporting\. We interpret the group\-size trend as an observed quality tradeoff, with representative dilution as one possible explanation\. Verifying the cause requires matched launch configurations and repeated controls\.

### D\.2Complete results and diagnostic observations

#### Layer sharing\.

Table[5](https://arxiv.org/html/2610.27438#A4.T5)includes both theh=1,2,3h=1,2,3configurations in the main paper and deeper schedulesh=5,6h=5,6\. Every measured setting improves wall time and GPU energy relative to its stock row\. Ath=6h=6, speedups are1\.3101\.310,1\.5721\.572,1\.2331\.233, and1\.432×1\.432\\timeson LongBench\-v2, RULER\-1M, MRCR\-v2, and GraphWalks, respectively\. RULER\-1M loses3\.333\.33and4\.424\.42points ath=5,6h=5,6; the other suites have smaller aggregate score changes in these runs\.

Per\-task RULER summaries suggest particular sensitivity inniah\_multikey\_2andniah\_multikey\_3, which contain many distractor keys\. The three single\-needle tasks score100100for stock and the reported layer\-sharing settings, except for one prompt ath=6h=6\. The multi\-key task with one distractor type, multi\-query retrieval, and multi\-value retrieval remain at100100or within one reported prompt of that value\. Ath≤3h\\leq 3, the two more difficult multi\-key tasks change by at most one reported prompt; ath=5,6h=5,6they lose one to two reported prompts each\. Ath=6h=6, the source also reports one\-prompt losses in a single\-needle task, frequent\-word retrieval, and each QA task\. Other tasks remain within two reported prompts of stock; stock variable tracking and common\-word scores are2222and00\. These task patterns do not establish why a candidate restriction changes an answer\. In particular, per\-task scores should not be converted into fixed0\.770\.77\-point suite increments without accounting for the tasks’ scoring rules\.

GraphWalks BFS scores are34\.634\.6for stock and36\.436\.4,34\.434\.4,37\.537\.5,33\.533\.5, and37\.237\.2forh=1,2,3,5,6h=1,2,3,5,6\. On items longer than256256K tokens, the reported paired score change ath=2h=2is−5\.6\-5\.6points; the corresponding changes ath=1,3h=1,3are−0\.2\-0\.2and\+1\.6\+1\.6points\. The aggregate and length\-specific results therefore need not move together\. Theh=2h=2configuration has one boot, and its cause has not been established\.

Ath=1h=1, energy savings are8\.98\.9,16\.916\.9,6\.66\.6, and13\.9%13\.9\\%, whereas the corresponding fractions of wall time saved are6\.26\.2,15\.015\.0,1\.51\.5, and11\.6%11\.6\\%\. Mean GPU power falls by2\.22\.2–5\.2%5\.2\\%\. That reduction is not universal: ath=5,6h=5,6on RULER\-1M mean power rises by approximately33–4%4\\%, while a shorter run still reduces energy\.

Table 5:Complete layer\-sharing results \(q​h=0qh=0\)\. Scores and changes use the stock row within each suite; energy is GPU\-only and measured after warm\-up\. Every configuration is one serving run\.
#### Persistent decode reuse\.

Tables[6](https://arxiv.org/html/2610.27438#A4.T6),[7](https://arxiv.org/html/2610.27438#A4.T7),[8](https://arxiv.org/html/2610.27438#A4.T8), and[9](https://arxiv.org/html/2610.27438#A4.T9)give the entireq​h=1,2,3qh=1,2,3grid ath=0,1,2,3h=0,1,2,3\. Together with Table[5](https://arxiv.org/html/2610.27438#A4.T5), they retain the full measured layer\-and\-query matrix\. At a fixedh≥1h\\geq 1, the reported speedups acrossq​h=0,1,2,3qh=0,1,2,3differ by at most4\.1%4\.1\\%, and energy savings by at most0\.60\.6percentage points\. The query\-only configurations are no faster than stock, reaching0\.9600\.960–1\.000×1\.000\\times\. This is consistent with limited opportunity for decode\-only reuse in these workloads; it does not establish identical execution cost\.

The largest reported score losses include RULER\-1M at\(h,q​h\)=\(3,2\)\(h,qh\)=\(3,2\)\(−9\.68\-9\.68points\) and\(3,3\)\(3,3\)\(−6\.18\-6\.18\), MRCR\-v2 at\(1,3\)\(1,3\)\(−6\.73\-6\.73\) and\(3,2\)\(3,2\)\(−5\.29\-5\.29\), and GraphWalks at\(3,2\)\(3,2\)\(−3\.01\-3\.01\) and\(3,3\)\(3,3\)\(−4\.12\-4\.12\)\. RULER\-1M is not monotone in query reuse ath=3h=3: the changes are−5\.44\-5\.44,−9\.68\-9\.68, and−6\.18\-6\.18points\. The corresponding task scores forniah\_multikey\_2andniah\_multikey\_3are4040and5050at\(3,2\)\(3,2\), and9090and4040at\(2,2\)\(2,2\)\. The source reports that single\-needle, multi\-query, and multi\-value tasks remain at100100or within one prompt in this grid\. The matrix contains single runs and does not identify the mechanism behind individual task failures\.

The isolated timing experiment in Table[2](https://arxiv.org/html/2610.27438#S5.T2)uses3030synthetic prompts of approximately0\.90\.9M tokens,2424sessions offered, and no prefix caching\. Decode time is estimated as the difference between a40,00040\{,\}000\-token generation and a one\-token generation\. Each arm has one boot and two measured draws after discarding a warm\-up draw\. It measuresh=0h=0only; the reported3\.3%3\.3\\%reduction atq​h=3qh=3is not a measured combined layer\-and\-query speedup\. This workload has no answer\-quality evaluation\.

Table 6:Persistent decode reuse on LongBench\-v2 \(q​h≥1qh\\geq 1\)\. Theh=0h=0rows use query reuse alone;q​h=0qh=0results appear in Table[5](https://arxiv.org/html/2610.27438#A4.T5)\. Ratios use the displayed stock row\.Table 7:Persistent decode reuse on RULER\-1M \(q​h≥1qh\\geq 1\)\. Theh=0h=0rows use query reuse alone;q​h=0qh=0results appear in Table[5](https://arxiv.org/html/2610.27438#A4.T5)\. Ratios use the displayed stock row\.Table 8:Persistent decode reuse on MRCR\-v2 \(q​h≥1qh\\geq 1\)\. Theh=0h=0rows use query reuse alone;q​h=0qh=0results appear in Table[5](https://arxiv.org/html/2610.27438#A4.T5)\. Ratios use the displayed stock row\.Table 9:Persistent decode reuse on GraphWalks \(q​h≥1qh\\geq 1\)\. Theh=0h=0rows use query reuse alone;q​h=0qh=0results appear in Table[5](https://arxiv.org/html/2610.27438#A4.T5)\. Ratios use the displayed stock row\.
#### Coarse producer and group size\.

Table[10](https://arxiv.org/html/2610.27438#A4.T10)reports the mean\-representative producer atP=2P=2,b=8,192b=8\{,\}192, andM=16,384M=16\{,\}384, with and withoutq​h=1qh=1\. Ath=1,q​h=0h=1,qh=0, task\-score changes are−0\.60\-0\.60,\+0\.28\+0\.28,−0\.75\-0\.75, and−0\.01\-0\.01points across the four suites; addingq​h=1qh=1changes them to\+0\.79\+0\.79,\+1\.28\+1\.28,−1\.13\-1\.13, and−0\.13\-0\.13\. Ath=2h=2, RULER\-1M loses5\.715\.71points without query reuse and5\.585\.58with it\. Ath=3h=3, MRCR\-v2 loses3\.303\.30and3\.363\.36points\. The coarse producer is faster than its own stock row on most settings, but theh=1h=1MRCR\-v2 configuration is slower with either query setting\.

Table[11](https://arxiv.org/html/2610.27438#A4.T11)variesP=2,4,8,16P=2,4,8,16ath=1,q​h=0h=1,qh=0while holdingb​P=MbP=Mfixed\. Energy savings rise on all suites\. RULER\-1M score changes are\+0\.28\+0\.28,−4\.08\-4\.08,−9\.01\-9\.01, and−14\.30\-14\.30points, while its speedup rises from1\.2291\.229to1\.352×1\.352\\times\. MRCR\-v2 loses3\.473\.47points atP=16P=16; GraphWalks loses1\.361\.36atP=8P=8and2\.522\.52atP=16P=16\. GraphWalks BFS falls from36\.136\.1for this campaign’s stock row to31\.931\.9atP=16P=16\. LongBench\-v2 shows no monotone score trend\. AtP≥4P\\geq 4, mean GPU power falls by approximately99–16%16\\%, compared with approximately55–10%10\\%atP=2P=2\.

The reported RULER\-1M single\-needle tasks score100100throughout this sweep\.niah\_multikey\_3scores5050,1010, and1010atP=4,8,16P=4,8,16, while the multi\-query and multi\-value scores atP=16P=16are7878and7575\. This pattern motivates testing whether coarse candidate construction loses important groups\. It does not, by itself, separate candidate loss from the earlier harness confound or explain each downstream error\.

Table 10:Coarse producer with mean representatives,P=2P=2andb=8,192b=8\{,\}192\. Ratios and score changes use this campaign’s stock rows\. Every configuration is one serving run\.Table 11:Coarse\-producer group\-size sweep ath=1h=1,q​h=0qh=0, and fixedb​P=M=16,384bP=M=16\{,\}384\. TheP=2P=2rows repeat Table[10](https://arxiv.org/html/2610.27438#A4.T10); all ratios use the displayed stock rows\.
#### Representative operator\.

Table[12](https://arxiv.org/html/2610.27438#A4.T12)replaces a group’s mean by its elementwise maximum atP=2P=2\. Ath=1h=1, max\-representative speedups against the displayed stock rows are1\.0051\.005,1\.2031\.203,0\.9540\.954, and1\.072×1\.072\\times, compared with1\.0271\.027,1\.2291\.229,0\.9790\.979, and1\.087×1\.087\\timesfor the separately measured mean campaign\. The GPU energy savings are10\.410\.4,22\.722\.7,7\.77\.7, and17\.1%17\.1\\%, compared with10\.210\.2,22\.522\.5,7\.77\.7, and16\.5%16\.5\\%\. These are campaign\-specific ratios rather than a repeated paired comparison of the two operators\.

The max operator has RULER\-1M score changes of−1\.59\-1\.59,−9\.23\-9\.23, and−2\.14\-2\.14points ath=1,2,3h=1,2,3, versus\+0\.28\+0\.28,−5\.71\-5\.71, and−0\.57\-0\.57for the mean\. Ath=2h=2,niah\_multikey\_2andniah\_multikey\_3score7070and5050with the mean, compared with7070and1010with the max\. The source reports every other task within one prompt of stock under both operators\. Neither operator eliminates the difficult RULER\-1M configuration\. Repeated comparisons with verified matched controls are needed to rank them confidently\.

Table 12:Coarse producer with elementwise\-max representatives,P=2P=2andb=8,192b=8\{,\}192\. Ratios and score changes use the displayed stock rows, also used in Table[5](https://arxiv.org/html/2610.27438#A4.T5)\.

## Appendix EExploratory Top\-GGConstruction and Trace Study

### E\.1Capture and measurement protocol

The Top\-GGstudy uses LB2 traces captured while running MLIPS\. The capture contains eight sequences and 21 compressed sparse\-attention \(CSA\) layers at physical layers2,4,…,422,4,\\ldots,42\. One CSA\-layer step therefore spans two physical model layers\. Each recorded query supplies unordered producer Top\-KKand Top\-MMindex sets, withK=512K=512and nominalM=16​KM=16\\mathrm\{K\}\. The dump lacks aligned producer\-layer scores, so the constructor below uses set membership\. Array position within a captured Top\-MMset is not treated as an importance ranking\. Pass identifiers are trace provenance and do not affect construction\.

Four consecutive queries are recorded every 256 original query positions\. Groups of 64, 256, and 1,024 recorded rows consequently represent nominal windows of 4,096, 16,384, and 65,536 original positions\. Sequence\-wide sharing uses all eligible recorded rows in a sequence\. These labels describe sampling spans, not fully observed query groups\. We cannot infer how recall changes when all intervening queries participate: additional queries may provide useful evidence but also compete for the same budget\.

All reported pools have budgetG=32,000G=32\{,\}000\. Measurements require context length at leastGGand average recall on capacity\-eligible recorded rows\. The primary aggregate in Figure[4](https://arxiv.org/html/2610.27438#S6.F4)covers sequences 0–6 of the eight captured sequences and subsequent CSA\-layer distances 1–6\. Producer\-layer fit, at distance zero, is excluded from reuse results\. The headline results are restricted to these seven sequences\. Per\-sequence results below use the 11 producer layers with a complete ten\-step consumer horizon\. Each sequence–window–distance cell is a query\-cell\-weighted mean over eligible recorded rows and those producers\.

For a recorded queryqq, writeKqℓK\_\{q\}^\{\\ell\}andMqℓM\_\{q\}^\{\\ell\}for producer memberships\. The pool satisfies Equation[6](https://arxiv.org/html/2610.27438#S6.E6), and evaluation uses Equation[7](https://arxiv.org/html/2610.27438#S6.E7)\. Recall is defined only for nonempty consumer selections\. The pool is frozen before examining any consumer layer; consumer Top\-KKlabels are used only for evaluation\.

### E\.2Coverage\-debt constructor

The constructor assigns more weight to queries whose producer selections remain uncovered\. Define the residual producer membershipAq=Mqℓ∖KqℓA\_\{q\}=M\_\{q\}^\{\\ell\}\\setminus K\_\{q\}^\{\\ell\}and, for a partially filled pool𝒢\\mathcal\{G\}, define

D⁡\(S,𝒢\)=\{1−\|S∩𝒢\|/\|S\|,\|S\|\>0,0,\|S\|=0\.D\(S;\\mathcal\{G\}\)=\\begin\{cases\}1\-\|S\\cap\\mathcal\{G\}\|/\|S\|,&\|S\|\>0,\\\\ 0,&\|S\|=0\.\\end\{cases\}\(12\)LetdqK=D⁡\(Kqℓ,𝒢\)d\_\{q\}^\{K\}=D\(K\_\{q\}^\{\\ell\};\\mathcal\{G\}\)anddqA=D⁡\(Aq,𝒢\)d\_\{q\}^\{A\}=D\(A\_\{q\};\\mathcal\{G\}\)\. An unselected candidatexxreceives primary and secondary scores

P⁡\(x\)\\displaystyle P\(x\)=∑q∈C:\|Kqℓ\|\>0dqK\|Kqℓ\|1\[x∈Kqℓ\],\\displaystyle=\\sum\_\{q\\in C:\\,\|K\_\{q\}^\{\\ell\}\|\>0\}\\frac\{d\_\{q\}^\{K\}\}\{\|K\_\{q\}^\{\\ell\}\|\}\\,\\mathbf\{1\}\[x\\in K\_\{q\}^\{\\ell\}\],\(13\)S⁡\(x\)\\displaystyle S\(x\)=∑q∈C:\|Aq\|\>0dqA\|Aq\|1\[x∈Aq\]\.\\displaystyle=\\sum\_\{q\\in C:\\,\|A\_\{q\}\|\>0\}\\frac\{d\_\{q\}^\{A\}\}\{\|A\_\{q\}\|\}\\,\\mathbf\{1\}\[x\\in A\_\{q\}\]\.\(14\)Zero\-size memberships contribute no vote, avoiding undefined normalization\. Candidates are ranked lexicographically by descending\(P⁡\(x\),S⁡\(x\),τ⁡\(x\)\)\(P\(x\),S\(x\),\\tau\(x\)\), whereτ\\tauis a fixed seeded key\-ID hash used for deterministic tie breaking; key ID resolves any remaining hash tie\. Thus residual Top\-MMevidence never overrides a larger primary score\. After each batch of 1,024 selected keys, debts and scores are recomputed\. This is a batched coverage heuristic, not an optimization guarantee\.

Algorithm 1Coverage\-debt pool construction0:Recorded sets

\{Kqℓ,Mqℓ:q∈C\}\\\{K\_\{q\}^\{\\ell\},M\_\{q\}^\{\\ell\}:q\\in C\\\}, capacity

GG, batch

B=1,024B=1\{,\}024
1:

𝒢←∅\\mathcal\{G\}\\leftarrow\\varnothing
2:

Aq←Mqℓ∖KqℓA\_\{q\}\\leftarrow M\_\{q\}^\{\\ell\}\\setminus K\_\{q\}^\{\\ell\}for every

q∈Cq\\in C
3:

U←⋃q∈C\(Kqℓ∪Mqℓ\)U\\leftarrow\\bigcup\_\{q\\in C\}\(K\_\{q\}^\{\\ell\}\\cup M\_\{q\}^\{\\ell\}\)
4:while

\|𝒢\|<min⁡\(G,\|U\|\)\|\\mathcal\{G\}\|<\\min\(G,\|U\|\)do

5:Compute debts using Equation[12](https://arxiv.org/html/2610.27438#A5.E12)

6:Compute

P⁡\(x\),S⁡\(x\)P\(x\),S\(x\)for

x∈U∖𝒢x\\in U\\setminus\\mathcal\{G\}
7:

b←min⁡\(B,G−\|𝒢\|,\|U∖𝒢\|\)b\\leftarrow\\min\(B,G\-\|\\mathcal\{G\}\|,\|U\\setminus\\mathcal\{G\}\|\)
8:

V←V\\leftarrowfirst

bbcandidates in descending

\(P,S,τ\)\(P,S,\\tau\)order

9:

𝒢←𝒢∪V\\mathcal\{G\}\\leftarrow\\mathcal\{G\}\\cup V
10:endwhile

11:return

𝒢\\mathcal\{G\}

As a query becomes covered, its primary vote shrinks and the budget shifts toward other queries\. This construction supplies a concrete example of how producer memberships can reveal a shared candidate working set\. It uses no consumer\-layer labels and no trained predictor\. That restriction concerns the layer axis: constructing the pool from all queries in a group can still make the candidate set for an earlier query depend on later queries\. Strict prefix\-invariant inference therefore requires an explicit construction schedule and causal masks, and may require a different constructor\. Masking future key positions alone does not remove dependence of pool membership on future query evidence\.

### E\.3Aggregate and per\-sequence evidence

For a fixed budget of 32,000 indices, the mean recall across distances 1–6 is77\.6%77\.6\\%,69\.4%69\.4\\%,52\.9%52\.9\\%, and33\.6%33\.6\\%for nominal windows of 4,096, 16,384, 65,536, and the full sequence, respectively \(Figure[4](https://arxiv.org/html/2610.27438#S6.F4)\)\. The first\-to\-sixth\-distance decrease is82\.4%82\.4\\%to73\.5%73\.5\\%for the smallest window\. At 65,536 positions, the sampling span is sixteen times larger while mean recall remains52\.9%52\.9\\%\. These results reveal reuse across queries and layers, together with a clear capacity boundary as the query scope expands\.

The ordering also holds separately for each primary sequence, 0–6: the mean over ten consumer distances decreases as the nominal window grows\. For the 4,096\-position window, per\-sequence ten\-distance mean recall ranges from65\.3%65\.3\\%to84\.1%84\.1\\%, and recall at distance ten ranges from59\.9%59\.9\\%to80\.6%80\.6\\%\. The aggregate trend is consequently not confined to one input, although the coverage level remains workload dependent\.

![Refer to caption](https://arxiv.org/html/2610.27438v1/mlips-topg-sequences.png)Figure 10:Per\-sequence Top\-GGcoverage atG=32,000G=32\{,\}000\. Rows are LB2 sequences 0–6 and columns are subsequent CSA\-layer distances; panels vary the nominal query window\. Cells average eligible recorded queries over the 11 producers with a complete ten\-layer horizon\. The original plot uses*hop*for exact layer distancedd\.
### E\.4What remains to be established

The trace study measures coverage of recorded selected indices\. It does not evaluate every query, end\-task accuracy, construction overhead, consumer selection cost, or serving performance\. The constructor is an exploratory probe rather than a validated runtime policy\. Comparisons against producer Top\-KKfrequency, fixed Top\-KK/Top\-MMquotas, private producer Top\-MMpools, and a future\-selection oracle would help isolate which parts of the construction explain coverage; the present aggregate alone does not establish their relative merit\.

Shared candidate metadata and coordinated KV staging are potential applications, but a pool ofGGentries may exceed each query’s original Top\-MMbudget and increase consumer scoring\. A shared index set also does not imply that different layers share KV values\. Realizing a residency benefit requires layer\-specific data movement and memory measurements\. Pool refresh, query\-group boundaries, causal scheduling, and any recovery for missed candidates remain deployment choices to evaluate\.

## Appendix FCandidate Working Sets and Offloading

Sharing candidate indices does not reduce the total KV state that an autoregressive model must preserve\. A future pool may contain entries absent from the current one\. Those entries must remain recoverable, for example in host memory, if they are evicted from the GPU\. This section analyzes a possible resident working set, not an implemented or measured memory\-saving mechanism\.

#### A simplified capacity model\.

Suppose each producer keeps allNNcompressed positions resident, and each consumer keeps the indexer keys and attention KV for its currentMMcandidates resident\. Under that policy, the compressed\-entry working set overnnlayers isp​N\+c​MpN\+cM, giving the modeled ratio

Rresident=n​Np​N\+c​M\.R\_\{\\mathrm\{resident\}\}=\\frac\{nN\}\{pN\+cM\}\.\(15\)This equals the layerwise dot\-product reduction because both counts assignNNpositions to a producer andMMto a consumer\. The identity is a property of this simplified policy, not an equivalence between compute savings and measured GPU memory\. Attention itself reads only the selected top\-KK; alternative staging policies could use a different resident set\.

If a producer also keeps onlyMMentries during itsq​hqhreuse steps, the cycle\-average ratio becomes

Ravg=n​N​\(q​h\+1\)p​N\+\[p​q​h\+c⁡\(q​h\+1\)\]​M\.R\_\{\\mathrm\{avg\}\}=\\frac\{nN\(qh\+1\)\}\{pN\+\[p\\,qh\+c\(qh\+1\)\]M\}\.\(16\)Forq​h\>0qh\>0, this is a time\-average, not a reduction in the capacity needed for a rebuild step\. The values match Table[3](https://arxiv.org/html/2610.27438#A1.T3), so we do not repeat that table\. Both models exclude weights, local windows, other attention layers, metadata, transfer buffers, and runtime allocation overhead\.

#### Transfers and deadlines\.

Consumer keys that enter a refreshed pool must be fetched before re\-scoring; the selected KV entries must arrive before attention\. The pool becomes known only when the producer publishes it, leaving a short window before the first consumer\. Persistent pools can lengthen this window on later reuse steps\. Achieving the producer’s cycle\-average working set additionally requires reloading its full search domain before the next rebuild\.

Pool turnover, transfer bandwidth, staging buffers, and scheduling determine whether the policy is practical\. The current top\-KK\-in\-pool recall measurements do not measure pool\-to\-pool turnover, so they cannot quantify this transfer volume\. An offloading evaluation should report actual peak GPU residency, transferred bytes, latency, GPU energy, and task quality together\.