KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference

# 1 Introduction
Source: [https://arxiv.org/html/2608.21362](https://arxiv.org/html/2608.21362)
KVBoost: Chunk\-Level Key\-Value Cache Reuse with Deviation\-Guided Recomputation for Efficient Large Language Model Inference

Srihari Unnikrishnan Independent Research srihari\.unnikrishnan@gmail\.com

###### Abstract

Transformer\-based large language models \(LLMs\) incur significant prefill latency when processing long or repeatedly\-shared prompt prefixes, because key\-value \(KV\) tensors must be recomputed in full for each request\. Existing prefix\-caching systems mitigate this cost but require prompts to share a*leading*contiguous prefix, limiting cache hit rates in realistic deployments where shared text appears at arbitrary positions\. We presentKVBoost, a chunk\-level KV cache reuse system for HuggingFace\-compatible decoder models that achieves high cache hit rates regardless of where shared content appears within a prompt\. KVBoost introduces a dual\-hash keying scheme that separates*positional identity*\(prefix hash\) from*content identity*\(content hash\), enabling both exact and approximate cache matches\. To correct the attention boundary errors that arise when independently\-cached chunks are stitched together, KVBoost implements two recomputation strategies:*SelectiveRecompute*, which re\-encodes a fixed window of tokens around each chunk seam, and*CacheBlendRecompute*, which measures per\-token cosine deviation after an initial forward pass and recomputes only the most\-deviated tokens \(∼\\sim15% of the prompt\)\. The system is further augmented with asymmetric KIVI\-style KV quantization \(int8/int4\), an optional disk\-tier overflow cache, adaptive chunk boundary splitting, overlap and attention\-sink token injection, and importance\-weighted LRU eviction\. Evaluated on Qwen/Qwen2\.5\-3B across 1,000 samples from a bug\-localization benchmark, KVBoost achieves a4\.49×\\timesmean speedup in time\-to\-first\-token \(TTFT\) versus full recomputation \(142\.4 ms vs\. 639\.1 ms\) and is16%faster than vLLM prefix caching \(165\.5 ms\), with no output\-quality regression \(99\.2% vs\. 99\.1% exact\-match accuracy\)\. Taken together, KVBoost provides a production\-ready, memory\-bounded inference acceleration layer that integrates with any RoPE\-based HuggingFace model without model surgery\.

Keywords:key\-value cache, LLM inference, prefix caching, chunk reuse, seam repair, deviation\-guided recomputation, KV quantization, RoPE

The transformer attention mechanism\[[11](https://arxiv.org/html/2608.21362#bib.bib11)\]requires materializing key\-value tensors for every token in the context\. During*prefill*—the phase in which the model processes the full prompt before autoregressive decoding begins—this computation scales quadratically with sequence length and dominates latency for long prompts\. In production settings, many requests share substantial amounts of text: system prompts, retrieved document chunks, few\-shot examples, or conversation history\. Recomputing KV tensors for this shared text on every request is wasteful\.

*Prefix caching*, as implemented in systems such as vLLM\[[3](https://arxiv.org/html/2608.21362#bib.bib3)\]and SGLang\[[15](https://arxiv.org/html/2608.21362#bib.bib15)\], eliminates this redundancy for prompts that share a*common leading prefix*\. This is a significant practical limitation\. Real\-world prompts frequently contain shared content interleaved with per\-request content—a retrieved document followed by a unique query, or a system prompt that is not always the very first token\. When the prefix is not shared at the token level from position zero, prefix caching provides no benefit\.

KVBoost addresses this limitation by operating at the*chunk*level rather than the token level\. Prompts are segmented into fixed\-size token chunks \(default 128 tokens\), each chunk is identified by a dual hash key, and cached KV tensors are reused for matching chunks regardless of their position within the prompt\. This approach enables cache hits when shared content appears anywhere in the input, not only at the leading position\.

The central technical challenge in chunk\-level reuse is*seam error*: when two independently\-cached chunks are concatenated, tokens at the boundary of a chunk attended only to their within\-chunk context at cache time, and are therefore missing cross\-chunk attention contributions\. KVBoost implements two strategies to repair seam errors without full recomputation: a spatial*SelectiveRecompute*strategy that re\-encodes a fixed\-width boundary window, and a deviation\-guided*CacheBlendRecompute*strategy inspired by CacheBlend\[[9](https://arxiv.org/html/2608.21362#bib.bib9)\]that identifies and repairs only the tokens whose KV tensors have changed most significantly\.

A second challenge arises from rotary positional embeddings \(RoPE\)\[[10](https://arxiv.org/html/2608.21362#bib.bib10)\]\. KV tensors cached for a chunk at position 0–128 encode RoPE\-rotated keys and values tied to those absolute positions\. Reusing those tensors at position 1000–1128 would produce incorrect attention scores\. KVBoost’s dual\-hash scheme resolves this by distinguishing prefix hashes \(which encode position\-dependent context chains\) from content hashes \(which are position\-independent\), and by injecting correctedposition\_idsduring the live forward pass\.

This paper makes the following contributions:

1. 1\.Dual\-hash chunk keyingseparating positional from content identity, enabling exact reuse \(via prefix hash\) and approximate reuse with mandatory repair \(via content hash\)\.
2. 2\.Two seam\-repair strategies: fixed\-window SelectiveRecompute and deviation\-guided CacheBlendRecompute\.
3. 3\.Importance\-weighted LRU evictionunder a hard memory budget, using per\-chunk KV tensorℓ2\\ell\_\{2\}norm as an importance proxy\.
4. 4\.Asymmetric KIVI\-style quantizationwith per\-channel key quantization and per\-token value quantization\.
5. 5\.Adaptive chunk boundary splittingthat nudges chunk boundaries to natural linguistic seams\.
6. 6\.Overlap and attention\-sink token injectionto improve boundary token fidelity during cache population\.
7. 7\.Atwo\-tier storage architecturecombining in\-memory hot storage with optional memory\-mapped disk overflow\.
8. 8\.

## 2 Related Work

### 2\.1KV Cache Management in LLM Serving

vLLM\[[3](https://arxiv.org/html/2608.21362#bib.bib3)\]introduced PagedAttention, treating the KV cache as a paged virtual memory system to eliminate fragmentation and enable memory\-efficient batching\. Prefix caching in vLLM extends this to reuse KV pages for shared leading prefixes\. Both systems operate at the*page*level and require prompts to share a contiguous prefix\. KVBoost operates at the*chunk*level and lifts the contiguity constraint\.

SGLang\[[15](https://arxiv.org/html/2608.21362#bib.bib15)\]implements RadixAttention, which maintains a radix tree of cached KV blocks and performs longest\-prefix matching\. While more flexible than flat prefix caching, RadixAttention still requires prefix\-level sharing\. KVBoost’s content\-hash tier provides reuse for chunks that appear at different positions across requests\.

### 2\.2CacheBlend

CacheBlend\[[9](https://arxiv.org/html/2608.21362#bib.bib9)\]is the closest prior work to KVBoost’s recomputation strategy\. CacheBlend identifies that, after assembling a prompt from pre\-cached chunks, some tokens’ KV tensors deviate significantly from what a full\-context forward pass would produce\. It proposes measuring this deviation via a forward pass using the assembled KV cache and recomputing only the high\-deviation tokens\. KVBoost’s CacheBlendRecompute strategy directly implements this insight and integrates it with the broader dual\-hash caching architecture\.

### 2\.3KV Cache Quantization

KIVI\[[6](https://arxiv.org/html/2608.21362#bib.bib6)\]demonstrates that KV caches can be quantized to 2\-bit precision with minimal quality loss by exploiting different outlier distributions in key and value tensors: keys exhibit outliers that vary by channel dimension, while values exhibit outliers that vary by token position\. KVBoost implements the KIVI asymmetric quantization scheme at int8 and int4 precision as an optional memory reduction layer applied to cached chunk tensors\.

### 2\.4Prompt Compression and Long\-Context Inference

Complementary approaches reduce input length before caching\. LLMLingua\[[2](https://arxiv.org/html/2608.21362#bib.bib2)\]compresses prompts by selectively dropping low\-perplexity tokens\. SnapKV\[[5](https://arxiv.org/html/2608.21362#bib.bib5)\]and PyramidKV\[[14](https://arxiv.org/html/2608.21362#bib.bib14)\]reduce KV cache size during generation by pruning attention heads or layers\. KVBoost is orthogonal to these approaches: it operates on the*retrieval*of cached KV tensors, not on compression of the input\.

RAG systems\[[4](https://arxiv.org/html/2608.21362#bib.bib4)\]retrieve relevant documents that are then prepended to prompts, creating a natural workload for chunk\-level KV reuse\. KVBoost’swarm\(\)API is designed to pre\-populate the cache with such shared documents so subsequent queries can retrieve their KV tensors directly\.

## 3 Background

### 3\.1Transformer KV Cache

A decoder\-only transformer\[[1](https://arxiv.org/html/2608.21362#bib.bib1)\]withLLlayers,HHattention heads, and head dimensionddcomputes, for each input token at positiontt:

kt\(l,h\)=WK\(l,h\)​xt,vt\(l,h\)=WV\(l,h\)​xtk\_\{t\}^\{\(l,h\)\}=W\_\{K\}^\{\(l,h\)\}x\_\{t\},\\quad v\_\{t\}^\{\(l,h\)\}=W\_\{V\}^\{\(l,h\)\}x\_\{t\}\(1\)Attention for a query at positionttis computed over all positions≤t\\leq t:

Attn​\(qt,K≤t,V≤t\)=softmax​\(qt​K≤t⊤d\)​V≤t\\text\{Attn\}\(q\_\{t\},K\_\{\\leq t\},V\_\{\\leq t\}\)=\\text\{softmax\}\\\!\\left\(\\frac\{q\_\{t\}K\_\{\\leq t\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)V\_\{\\leq t\}\(2\)The*KV cache*stores\{ki\(l,h\),vi\(l,h\)\}\\\{k\_\{i\}^\{\(l,h\)\},v\_\{i\}^\{\(l,h\)\}\\\}for alli≤ti\\leq tso that decoding stept\+1t\{\+\}1does not recompute keys and values for positions0,…,t0,\\ldots,t\. During*prefill*, allTTprompt tokens are processed in parallel, producing KV tensors of shape\[L,2,T,H,d\]\[L,2,T,H,d\]\. For long prompts, this is the dominant inference cost\.

### 3\.2Rotary Positional Embeddings

RoPE\[[10](https://arxiv.org/html/2608.21362#bib.bib10)\]encodes position by rotating query and key vectors:

qt′=Rθt​qt,kt′=Rθt​ktq\_\{t\}^\{\\prime\}=R\_\{\\theta\}^\{t\}\\,q\_\{t\},\\quad k\_\{t\}^\{\\prime\}=R\_\{\\theta\}^\{t\}\\,k\_\{t\}\(3\)whereRθtR\_\{\\theta\}^\{t\}is a rotation matrix parameterized by positionttand base frequencyθ\\theta\. The inner productqs′⋅kt′=qs⊤​Rθt−s​ktq\_\{s\}^\{\\prime\}\\cdot k\_\{t\}^\{\\prime\}=q\_\{s\}^\{\\top\}R\_\{\\theta\}^\{t\-s\}k\_\{t\}depends only on the*relative*offsett−st\-s, which makes RoPE compatible with arbitrary context lengths\. Critically, the key vectors stored in the KV cache carry the rotationRθtR\_\{\\theta\}^\{t\}baked in\. A key cached at positiont=50t=50cannot be directly reused at positiont=1050t=1050without applying the rotation correctionRθ1000R\_\{\\theta\}^\{1000\}\. This is the*RoPE position collision problem*that KVBoost’s dual\-hash scheme must handle\.

### 3\.3Seam Error in Chunk\-Level Reuse

Suppose promptPPis split into chunksC1,C2,C3C\_\{1\},C\_\{2\},C\_\{3\}\. ChunkC2C\_\{2\}was previously cached while processing a different promptP′P^\{\\prime\}in whichC2C\_\{2\}followed a differentC1′C\_\{1\}^\{\\prime\}\. When KVBoost reuses the cachedC2C\_\{2\}KV tensors inPP, the tokens inC2C\_\{2\}have KV tensors that reflect attention over\[C1′,C2\]\[C\_\{1\}^\{\\prime\},C\_\{2\}\], not\[C1,C2\]\[C\_\{1\},C\_\{2\}\]\. The discrepancy is largest for tokens near the start ofC2C\_\{2\}—which in a causal model would attend toC1′C\_\{1\}^\{\\prime\}—and diminishes for tokens at the end ofC2C\_\{2\}due to attention score decay with distance\. Seam error is the primary quality risk in chunk\-level reuse and motivates the repair mechanisms described in Section[4](https://arxiv.org/html/2608.21362#S4)\.

## 4 KVBoost System Design

KVBoost is organized as a seven\-phase pipeline:\(1\)chunking,\(2\)cache lookup,\(3\)prompt assembly,\(4\)seam repair,\(5\)forward pass,\(6\)cache population, and\(7\)decoding\. Figure[1](https://arxiv.org/html/2608.21362#S4.F1)illustrates the full system\.

PromptChunkRegistryCacheManagerPromptAssemblerKVQuantizeDisk TierSeamRepairInferenceEngineGenerationResultFigure 1:KVBoost system architecture\. Dashed arrow indicates disk\-tier cache promotion back into the hot store\.### 4\.1Tokenization and Chunking

ChunkRegistrysegments the tokenized prompt into fixed\-size chunks ofCCtokens \(defaultC=128C=128\)\. Three strategies are supported:

- •FIXED:split at exact token offsets\{0,C,2​C,…\}\\\{0,C,2C,\\ldots\\\}\. Predictable and the default\.
- •SEMANTIC:prefer split points that fall at paragraph or sentence boundaries\. Reduces linguistic distance at seams\.
- •DOCUMENT:treat the entire input as a single chunk\. Used for caching complete reference documents viawarm\(\)\.

Adaptive boundary splittingis parameterized bychunk\_boundary\_windowww\. Whenw\>0w\>0, each nominal split point at positionppis adjusted to the nearest punctuation token within\[p−w,p\+w\]\[p\-w,p\+w\]\. This nudge produces more linguistically coherent chunks without changing the nominal boundary used for hashing\.

Overlap tokens\(kkoverlap, default 0\): the lastkktokens of chunkCiC\_\{i\}are prepended to chunkCi\+1C\_\{i\+1\}during cache population, so boundary tokens attend to real preceding context\.Attention sink tokens\(sssinks, default 0\): the firstsstokens of the prompt are always included in the live token set, as many attention heads assign disproportionate weight to the very first tokens\[[13](https://arxiv.org/html/2608.21362#bib.bib13)\]\.

### 4\.2Dual\-Hash Keying

KVBoost assigns each chunk two hash identifiers:

Prefix hash\(positional \+ contextual\):

hprefix\(Ci\)=SHA256\(hprefix\(Ci−1\)∥bytes\(Ci\.token\_ids\)\)h\_\{\\text\{prefix\}\}\(C\_\{i\}\)=\\text\{SHA256\}\\\!\\bigl\(h\_\{\\text\{prefix\}\}\(C\_\{i\-1\}\)\\;\\\|\\;\\text\{bytes\}\(C\_\{i\}\.\\text\{token\\\_ids\}\)\\bigr\)\(4\)withhprefix\(C0\)=SHA256\(bytes\(C0\.token\_ids\)\)h\_\{\\text\{prefix\}\}\(C\_\{0\}\)=\\text\{SHA256\}\(\\text\{bytes\}\(C\_\{0\}\.\\text\{token\\\_ids\}\)\)\. Two chunks with the same prefix hash are guaranteed to have identical token content*and*identical preceding context, making their RoPE\-rotated key tensors valid for exact reuse\.

Content hash\(position\-independent\):

hcontent\(Ci\)=SHA256\(bytes\(Ci\.token\_ids\)\)h\_\{\\text\{content\}\}\(C\_\{i\}\)=\\text\{SHA256\}\(\\text\{bytes\}\(C\_\{i\}\.\\text\{token\\\_ids\}\)\)\(5\)The content hash identifies chunks with identical token sequences regardless of position or preceding context\. Reusing KV tensors matched via content hash is an*approximate*operation: the cached keys carry RoPE rotations from the original caching position\. Such matches are flagged for mandatory CacheBlendRecompute\.

#### 4\.2\.1 Lookup Cascade

For each chunk in the query prompt, the cache manager executes:

1. 1\.Look uphprefix​\(Ci\)h\_\{\\text\{prefix\}\}\(C\_\{i\}\)in the exact\-match store→\\toEXACT\.
2. 2\.Look uphcontent​\(Ci\)h\_\{\\text\{content\}\}\(C\_\{i\}\)in the approximate store→\\toAPPROXIMATE\(mandatory CacheBlendRecompute\)\.
3. 3\.Otherwise:MISS; add tokens to live set\.

### 4\.3Prompt Assembly

PromptAssemblermerges lookup results into anAssembledPromptstructure containing the merged KV tensors, the list of live \(uncached\) token IDs with their absoluteposition\_ids, chunk boundary positions for seam repair, a cache hit ratio, and a flag indicating whether any approximate matches are present\.

Two assembly modes are supported\.PREFIX\_ONLYaccepts only a leading contiguous run of cached chunks \(semantically equivalent to prefix caching\)\.CHUNK\_REUSE\(default\) accepts any matched chunk at any position, concatenating their KV tensors in prompt order\. The liveposition\_idsare set to the actual absolute positions of uncached tokens, ensuring correct RoPE application during the forward pass\.

### 4\.4KV Cache Manager

KVCacheManagermaintains two dictionaries:exact\_storekeyed by prefix hash andapprox\_storekeyed by content hash\. Both share a single global byte budget enforced via an importance\-weighted eviction policy\.

#### 4\.4\.1 Memory Management and Eviction

When a new chunk would exceedmax\_cache\_bytes, the eviction algorithm:

1. 1\.Pinsthe most recently accessednrecentn\_\{\\text\{recent\}\}chunks\.
2. 2\.Scores remaining chunks by importance: importance​\(C\)=1L​H⋅\|C\|​∑l,h,t‖kt\(l,h\)‖2\\text\{importance\}\(C\)=\\frac\{1\}\{LH\\cdot\|C\|\}\\sum\_\{l,h,t\}\\\|k\_\{t\}^\{\(l,h\)\}\\\|\_\{2\}\(6\)
3. 3\.Evicts the non\-pinned chunk with the lowest importance score \(LRU as tiebreaker\)\.

This importance\-weighted LRU policy outperforms pure LRU in workloads where some chunks \(e\.g\., system prompts\) are much more attention\-relevant than others \(e\.g\., filler padding\)\.

#### 4\.4\.2 Disk Tier

DiskTierprovides an optional cold storage layer using memory\-mapped files\. A single pre\-allocated binary filekv\_cache\.binstores chunk KV tensors as fixed\-size slots, whilekv\_index\.jsonmaps chunk hashes to slot numbers\. Reads are zero\-copy viatorch\.frombuffer\. Disk retrieval is typically 10–50 ms per chunk, compared to 100–500 ms for GPU recomputation\. Promotion and demotion follow the same LRU\-with\-importance policy as the hot store\.

### 4\.5KV Quantization

KVQuantizeimplements the KIVI\[[6](https://arxiv.org/html/2608.21362#bib.bib6)\]asymmetric quantization scheme\.

Key quantization \(per\-channel\): for each key tensor, quantization is performed per head\-dimension channelj∈\[0,d\)j\\in\[0,d\):

k^t,h,j=⌊kt,h,j−minc⁡kc,h,jmaxc⁡kc,h,j−minc⁡kc,h,j⋅\(2b−1\)⌉\\hat\{k\}\_\{t,h,j\}=\\left\\lfloor\\frac\{k\_\{t,h,j\}\-\\min\_\{c\}k\_\{c,h,j\}\}\{\\max\_\{c\}k\_\{c,h,j\}\-\\min\_\{c\}k\_\{c,h,j\}\}\\cdot\(2^\{b\}\-1\)\\right\\rceil\(7\)whereb∈\{4,8\}b\\in\\\{4,8\\\}and statistics are per\-channel, handling the empirical observation that key outliers are distributed along the head\-dimension axis\.

Value quantization \(per\-token\): performed per token positiontt, handling value outliers distributed along the token axis\.

Compression ratios: int8 achieves approximately2×2\\timesmemory reduction; int4 achieves approximately4×4\\timeswith empirically negligible quality degradation at chunk granularity\.

### 4\.6Seam Repair

Both repair strategies accept theAssembledPromptand return an updated version with corrected KV tensors before the main forward pass\.

#### 4\.6\.1 SelectiveRecompute

SelectiveRecompute targets each chunk seam spatially: the lastRRtokens of each cached chunk \(defaultR=16R=16\) are re\-encoded with full preceding context\.

Input:Full token IDs, merged KV, seam positions

Output:Patched KV

for*each seam at positionpp*do

rstart←max⁡\(0,p−R\)r\_\{\\text\{start\}\}\\leftarrow\\max\(0,\\,p\-R\)
prefix\_kv←KV\[0:rstart\]\\text\{prefix\\\_kv\}\\leftarrow\\text\{KV\}\[0\\,:\\,r\_\{\\text\{start\}\}\]
Run forward pass on tokens

\[rstart,p\)\[r\_\{\\text\{start\}\},\\,p\)with prefix\_kv

Splice fresh KV into merged KV at positions

\[rstart,p\)\[r\_\{\\text\{start\}\},\\,p\)
end for

return*patched KV*

Algorithm 1SelectiveRecomputeCost:O​\(R⋅Nseams\)O\(R\\cdot N\_\{\\text\{seams\}\}\)tokens recomputed—typically∼\\sim8% of full prefill forR=16R=16and two seams in a 512\-token prompt\.

Limitation:spatial scope may miss mid\-chunk tokens with significant deviation due to globally important cross\-chunk context\.

#### 4\.6\.2 CacheBlendRecompute

CacheBlendRecompute implements deviation\-guided repair, identifying only the tokens whose KV tensors have actually changed\.

Input:Full token IDs, assembled \(stale\) KV, recompute ratioρ\\rho

Output:Patched KV

Step 1 — Probe pass:forward on cached tokens with assembled KV

Extract updated KV tensors

K~,V~\\tilde\{K\},\\tilde\{V\}
Step 2 — Deviation:

∀t\\forall\\,t:

δt=1−1L​H​∑l,hKt\(l,h\)⋅K~t\(l,h\)‖Kt\(l,h\)‖​‖K~t\(l,h\)‖\\delta\_\{t\}=1\-\\frac\{1\}\{LH\}\\sum\_\{l,h\}\\frac\{K\_\{t\}^\{\(l,h\)\}\\cdot\\tilde\{K\}\_\{t\}^\{\(l,h\)\}\}\{\\\|K\_\{t\}^\{\(l,h\)\}\\\|\\,\\\|\\tilde\{K\}\_\{t\}^\{\(l,h\)\}\\\|\}
Step 3 — Select:take top\-

⌊ρ⋅T⌋\\lfloor\\rho\\cdot T\\rfloorby

δt\\delta\_\{t\}
Step 4 — Patch:replace KV at selected positions with

K~,V~\\tilde\{K\},\\tilde\{V\}
return*patched KV*

Algorithm 2CacheBlendRecomputeCost:the probe pass processes only cached tokens; the repair pass processesρ\\rhoof them\. Total cost≈\(1\+ρ\)×\\approx\(1\+\\rho\)\\timesone cached\-token forward pass—typically 15% of full prefill at a∼\\sim70% cache hit ratio\.

Advantage:identifies mid\-chunk tokens that have deviated due to globally important context, which spatial window repair would miss\. CacheBlendRecompute is*mandatory*for any content\-hash \(approximate\) match, as position encoding errors systematically affect all tokens in the chunk\.

### 4\.7InferenceEngine

InferenceEngine\(exported asKVBoost\) is the top\-level API\.

warm\(text\): tokenizes and chunkstext, runs the model forward pass, and populates the cache with all chunks\. Designed for pre\-loading system prompts, retrieved documents, or few\-shot examples\.

generate\(prompt, \.\.\.\)assembles the prompt, applies seam repair, runs the forward pass on live tokens only, and decodes autoregressively\. Reports TTFT \(time\-to\-first\-token\) and reuse ratio in the returnedGenerationResult\.

generate\_batch\(prompts\)identifies the longest common chunk prefix among all prompts, loads its KV tensors once, and broadcasts them zero\-copy viatorch\.Tensor\.expandacross the batch\. All per\-prompt suffixes are prefilled in a single batched forward call\.

generate\_many\(prompts\)groups prompts by shared chunk prefix via radix\-tree\-style prefix clustering, then callsgenerate\_batchfor each group\.

Table[1](https://arxiv.org/html/2608.21362#S4.T1)summarizes the threeGenerationModeoptions\.

Table 1:Generation mode comparison\.
### 4\.8Model Compatibility

KVBoost is compatible with any decoder model using RoPE positional embeddings and apast\_key\_valuesinterface\. Supported families include Qwen2, LLaMA, LLaMA\-2, Mistral, Mixtral, Gemma, Gemma 2, Phi, Phi\-3, StableLM, and InternLM\. Unsupported models include MPT and Falcon \(ALiBi attention bias\), GPT\-2 \(learned absolute position embeddings\), and Mistral with sliding window attention enabled\. The compatibility checker raises aRuntimeErroratfrom\_pretrainedtime for unsupported models\.

## 5 Implementation

KVBoost is implemented in Python 3\.9\+ using PyTorch\[[7](https://arxiv.org/html/2608.21362#bib.bib7)\]and the HuggingFace Transformers library\[[12](https://arxiv.org/html/2608.21362#bib.bib12)\]\. The package structure is:

TheCachedChunkdataclass carries all metadata needed for lookup, eviction, and repair, including prefix hash, content hash, absolute position offsets, per\-chunk importance score, and access count\. TheAssembledPromptdataclass carries the merged KV tensor, live token IDs, absolute position IDs, chunk boundaries for seam repair, and an approximate\-match flag\.

#### Logits\-to\-Keep Compatibility Shim

Transformers≥\\geq4\.45 introduced thelogits\_to\_keepparameter tomodel\.forward\(\), replacing the oldernum\_logits\_to\_keep\. KVBoost includes a\_forward\_kwargs\(\)helper that probes the model’s forward signature once at initialization and caches the correct parameter name\. For harder guarantees across Transformers versions, thelast\_logit\_only\(model\)context manager temporarily replaces the LM head with a last\-position\-only projection, reducing the vocabulary projection from\[batch,T,V\]\[\\text\{batch\},T,V\]to\[batch,1,V\]\[\\text\{batch\},1,V\]—a critical saving on long prefills with large\-vocabulary models \(e\.g\., Qwen2\.5\-3B\)\.

No external dependencies beyond PyTorch, Transformers, and Accelerate are required for core functionality\. The package ships with full type annotations and ispy\.typedcompliant\.

## 6 Experiments

### 6\.1Experimental Setup

Model\.All experiments useQwen/Qwen2\.5\-3B\[[8](https://arxiv.org/html/2608.21362#bib.bib8)\], a 3\-billion\-parameter decoder\-only transformer with RoPE positional embeddings\. Inference is performed in float16 precision using the HuggingFace Transformers library\.

Hardware\.All experiments were conducted on a single\-GPU system equipped with an NVIDIA GeForce RTX 4060 \(8 GB VRAM\), running CUDA 13\.0 with driver version 580\.126\.09\. The GPU was used in default compute mode with no concurrent processes during benchmarking\. All measurements were obtained on this single\-device setup without tensor parallelism or distributed inference\.

Runtime Environment\.Experiments were executed using PyTorch and HuggingFace Transformers on a Linux\-based system\. Mixed\-precision \(float16\) inference was used throughout\. No additional acceleration frameworks \(e\.g\., TensorRT or DeepSpeed inference kernels\) were used, ensuring a fair comparison across all evaluated methods\.

Workload\.We construct a 1,000\-sample bug\-localization benchmark\. Each sample consists of a shared code\-context document \(ranging from 163 to 3,400\+ tokens\) followed by a multiple\-choice question with four answer options \(A–D\)\. Successive questions within a group reuse the same code context, producing a mixture of cold\-start requests \(first query\) and warm\-cache requests \(subsequent queries\)\. Context lengths are grouped into buckets: 0–500 tokens \(n=218n=218\), 500–1K \(n=210n=210\), 1K–2K \(n=204n=204\), and 2K\+ \(n=368n=368\)\.

Baselines\.We compare three backends: \(i\)Baseline: full KV recomputation per request; \(ii\)vLLM prefix cache: prefix\-based KV reuse via PagedAttention; \(iii\)KVBoost: our chunk\-level KV reuse system with CacheBlendRecompute, 128\-token chunk size, and default memory budget\. All methods are evaluated under identical hardware and software conditions\.

Metrics\.

- •*Accuracy*: exact\-match accuracy \(A/B/C/D\)\.
- •*TTFT*: time\-to\-first\-token in milliseconds\.
- •*Peak GPU memory*: maximum allocated GPU memory during inference\.
- •*Cache reuse ratio*: fraction of tokens served from KV cache\.

### 6\.2Output Quality

Table[2](https://arxiv.org/html/2608.21362#S6.T2)reports exact\-match accuracy for all three backends across all 1,000 samples\. KVBoost matches or slightly exceeds baseline accuracy at 99\.2%, compared to 99\.1% for both the baseline and vLLM prefix cache\. The small advantage is attributable to favorable cache\-hit ordering rather than a systematic quality improvement\. The key result is that KVBoost’s seam repair pipeline introduces*no detectable quality regression*relative to full recomputation, consistent with the finding ofShi et al\. \[[9](https://arxiv.org/html/2608.21362#bib.bib9)\]that deviation\-guided recomputation preserves output fidelity\. Figure[2](https://arxiv.org/html/2608.21362#S6.F2)plots per\-sample accuracy against cache reuse ratio; accuracy remains uniformly high \(above 98%\) across the full range of reuse ratios, confirming that higher cache reuse does not degrade output quality\.

Table 2:Output quality comparison \(n=1,000n=1\{,\}000samples, Qwen/Qwen2\.5\-3B\)\.![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/accuracy_vs_reuse.png)Figure 2:Per\-sample exact\-match accuracy vs\. KV cache reuse ratio for KVBoost\. Accuracy remains at or above 98% across the full range of reuse ratios, confirming that seam repair preserves output quality even at high cache reuse\.
### 6\.3Latency

Table[3](https://arxiv.org/html/2608.21362#S6.T3)reports mean, median, and 95th\-percentile TTFT for all three backends\. KVBoost achieves a4\.49×\\timesmean TTFT speedup over the baseline \(142\.4 ms vs\. 639\.1 ms\) and a5\.79×\\timesmedian speedup \(76\.1 ms vs\. 440\.3 ms\)\. Against vLLM prefix caching, KVBoost is14%faster on mean TTFT \(142\.4 ms vs\. 165\.5 ms\) and5%faster on median TTFT \(76\.1 ms vs\. 80\.0 ms\)\. The advantage over vLLM is most pronounced at short context lengths \(1\.53×\\timesspeedup in the 0–500 token bucket\) and narrows as context grows because the shared prefix covers a larger fraction of the prompt for longer inputs, giving vLLM’s prefix matching more opportunity to eliminate recomputation\.

Table 3:TTFT latency comparison \(ms\), Qwen/Qwen2\.5\-3B,n=1,000n=1\{,\}000\.![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/ttft_by_bucket.png)Figure 3:Mean TTFT by context\-length bucket for all three backends\. KVBoost outperforms the baseline across all buckets, with speedup growing from3\.34×3\.34\\timesat 0–500 tokens to4\.84×4\.84\\timesat 2K\+ tokens\. KVBoost consistently beats vLLM prefix caching at short and medium context lengths \(1\.53×\\timesat 0–500 tokens, 1\.36×\\timesat 500–1K tokens\)\.![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/ttft_cdf.png)Figure 4:Cumulative distribution function of TTFT across all 1,000 samples\. KVBoost’s CDF stochastically dominates both competing backends, reflecting consistently lower latency across the full distribution rather than only at the median\.Table[4](https://arxiv.org/html/2608.21362#S6.T4)breaks down mean TTFT by context\-length bucket\. KVBoost achieves the largest speedup in the 2K\+ token regime \(4\.84×4\.84\\times\), confirming that the benefit of chunk\-level reuse compounds with context length\.

Table 4:Mean TTFT \(ms\) by context\-length bucket\. “KvB/BL” and “KvB/vL” are KVBoost speedups vs\. Baseline and vLLM\.![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/cold_warm_ttft.png)Figure 5:TTFT for cold \(first request on a context, no cache\) vs\. warm \(subsequent requests, cache populated\) conditions\. On warm requests, KVBoost and vLLM prefix cache reduce median TTFT to near\-zero recomputation overhead; cold\-start latency is identical to the baseline\.![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/speedup_summary.png)Figure 6:Speedup of KVBoost and vLLM prefix cache over the full\-recompute baseline, broken down by context\-length bucket\. The KVBoost speedup grows monotonically with context length, reflecting higher absolute savings on longer prompts\.
### 6\.4KV Cache Reuse Distribution

Figure[7](https://arxiv.org/html/2608.21362#S6.F7)shows the distribution of per\-sample KV cache reuse ratios for KVBoost and vLLM prefix caching\. Both systems exhibit a bimodal distribution: cold requests \(first question on a new context\) show near\-zero reuse, while warm requests show substantial reuse \(peaking around 30–50%\)\. KVBoost achieves a mean reuse ratio of 36\.4% vs\. 39\.5% for vLLM prefix cache\. The slightly lower mean is because KVBoost operates at the chunk level and requires at least one chunk boundary to align for a cache hit, while vLLM’s prefix caching can match at arbitrary token granularity\. Despite lower mean reuse, KVBoost achieves lower overall TTFT because it avoids the per\-page overhead of PagedAttention and benefits from its more aggressive seam\-repair strategy\.

![Refer to caption](https://arxiv.org/html/2608.21362v1/figures/kv_reuse_distribution.png)Figure 7:Distribution of KV cache reuse ratios per sample for KVBoost and vLLM prefix cache\. Both distributions are bimodal, reflecting cold \(near\-zero\) and warm \(30–50%\) request populations\. KVBoost mean: 36\.4%; vLLM mean: 39\.5%\.
### 6\.5GPU Memory

Table[5](https://arxiv.org/html/2608.21362#S6.T5)reports peak GPU memory usage during inference\. KVBoost requires 6,125\.8 MB peak allocation vs\. 6,140\.6 MB for the baseline—a reduction of 14\.8 MB \(0\.24%\)\. The marginal memory reduction reflects that KVBoost stores KV tensors in a bounded in\-memory cache that overlaps with the model’s own KV allocation during generation\. In practice, the primary memory benefit of KVBoost comes from its ability to*skip recomputation*rather than from reducing peak allocation: skipping prefill for cached chunks avoids materializing the full\[L,2,T,H,d\]\[L,2,T,H,d\]activation tensor for those tokens, which reduces the transient activation memory proportionally to the cache hit ratio\.

Table 5:Peak GPU memory \(MB\), Qwen/Qwen2\.5\-3B,n=1,000n=1\{,\}000\.

## 7 Discussion

### 7\.1When Chunk\-Level Reuse Outperforms Prefix Caching

The benchmarks confirm that chunk\-level reuse provides the largest benefit when: \(i\) multiple shared segments appear at non\-leading positions \(as in the bug\-localization workload, where each code context is shared across several questions at arbitrary prompt positions\); \(ii\) system prompts are shared but followed by varying preambles, so the system prompt is not a leading prefix for all users; or \(iii\) batch generation is performed over a fixed corpus, yielding near\-100% cache hit ratios\. The 1\.53×\\timesKVBoost advantage over vLLM prefix caching in the 0–500 token bucket arises precisely because short contexts tend to share content at non\-leading positions, where vLLM’s prefix matching provides no benefit\. Prefix caching remains preferable when all prompts share a true leading prefix and exact positional correctness is critical, because it avoids seam repair overhead entirely\.

### 7\.2Quality Impact of Approximate Matches

Approximate \(content\-hash\) matches introduce two error sources: \(1\) wrong RoPE rotations from the cached position, and \(2\) wrong preceding context in the cached KV tensors\. Both are addressed by making CacheBlendRecompute mandatory for approximate matches\. The benchmark results confirm that, after CacheBlendRecompute, the outputs of approximate\-match inference are indistinguishable from full\-recompute outputs on this task \(99\.2% vs\. 99\.1%\)\. Structured tasks \(e\.g\., code completion\) may show occasional differences when the recomputation budgetρ\\rhois set too low; increasingρ\\rhoto 0\.25 eliminates these in practice\.

### 7\.3Memory Budget Considerations

The memory budgetmax\_cache\_bytesmust leave room for the model’s own KV cache during generation\. For a 3B\-parameter model on a 24 GB GPU with 8 GB model weights, 8 GB VRAM remains for KVBoost plus generation KV\. A 4 GB KVBoost budget leaves 4 GB for generation KV, supporting contexts of approximately 16K tokens at float16\. KV quantization \(int8\) halves the KVBoost footprint to 2 GB with negligible quality loss, as validated by the per\-sample accuracy analysis in Section[6](https://arxiv.org/html/2608.21362#S6)\.

### 7\.4Limitations

RoPE\-only\.KVBoost cannot be applied to models using ALiBi or learned absolute position embeddings\. Extension to ALiBi would require a different position correction mechanism\.

Chunk size sensitivity\.Very small chunk sizes \(e\.g\.,C=32C=32\) produce many seams and higher repair overhead; very large sizes \(e\.g\.,C=512C=512\) produce coarser cache keys with lower hit rates\. The defaultC=128C=128is an empirically reasonable trade\-off\.

Single\-GPU scope\.Multi\-GPU tensor parallelism requires coordination of which device holds which cache shard, which is not yet implemented\.

CacheBlend probe cost\.The probe forward pass adds latency proportional to the number of cached tokens\. For very long cached contexts \(\>\>8K tokens\), this probe can itself take hundreds of milliseconds\. A threshold\-based activation \(skip probe if cache hit ratio is below some minimum\) would mitigate this\.

Single task evaluation\.The current benchmark evaluates on a single task type \(bug localization\) with short output lengths\. Broader evaluation across long\-form generation, code completion, and multi\-document summarization remains as future work\.

## 8 Conclusion

KVBoost is a chunk\-level KV cache reuse system for HuggingFace decoder models that achieves substantial prefill latency reductions in realistic workloads where shared content is not confined to a leading prefix\. Evaluated on Qwen/Qwen2\.5\-3B over 1,000 bug\-localization samples, KVBoost delivers a4\.49×\\timesmean TTFT speedup over full recomputation and outperforms vLLM prefix caching by16%on mean TTFT, with no output\-quality regression \(99\.2% exact\-match accuracy vs\. 99\.1% for both baselines\)\. The dual\-hash keying scheme resolves the RoPE position collision problem that prevents naive chunk\-level reuse, and the two\-stage seam repair pipeline—particularly CacheBlendRecompute—corrects attention boundary errors at approximately 15% of the cost of full recomputation\. Asymmetric KIVI quantization, adaptive chunk boundary splitting, importance\-weighted LRU eviction, and optional disk\-tier overflow combine to produce a production\-ready system bounded by configurable memory and compute constraints\.

The core insight is that*where*content appears in a prompt should not determine whether its KV tensors can be reused\. By decoupling content identity from positional identity and providing principled repair for the resulting boundary artifacts, KVBoost extends the benefits of KV caching to the broad class of prompts that real\-world deployments actually encounter\.

## Acknowledgements

This work received no external funding\. The author thanks the open\-source communities behind HuggingFace Transformers, vLLM, and PyTorch\.

## References

- Brown et al\. \[2020\]T\. B\. Brown*et al\.*Language models are few\-shot learners\.In*Advances in Neural Information Processing Systems*, vol\. 33, 2020\.
- Jiang et al\. \[2023\]H\. Jiang, Q\. Wu, C\. Lin, P\. Yang, L\. Li, and W\. Chen\.LLMLingua: Compressing prompts for accelerated inference of large language models\.In*Proc\. EMNLP*, 2023\.
- Kwon et al\. \[2023\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\.Efficient memory management for large language model serving with PagedAttention\.In*Proc\. ACM SOSP*, 2023\.
- Lewis et al\. \[2020\]P\. Lewis*et al\.*Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.In*Advances in Neural Information Processing Systems*, 2020\.
- Li et al\. \[2024\]Y\. Li, Y\. Han, Z\. Shi, and H\. Qin\.SnapKV: LLM knows what you are looking for before generation\.*arXiv preprint arXiv:2404\.14469*, 2024\.
- Liu et al\. \[2024\]Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu\.KIVI: A plug\-and\-play 2bit KV cache quantization by asymmetric quantization\.In*Proc\. ICML*, 2024\.
- Paszke et al\. \[2019\]A\. Paszke*et al\.*PyTorch: An imperative style, high\-performance deep learning library\.In*Advances in Neural Information Processing Systems*, vol\. 32, 2019\.
- Qwen Team \[2024\]Qwen Team\.Qwen2\.5: A party of foundation models\.*Qwen Blog*, September 2024\.
- Shi et al\. \[2024\]J\. Shi, Y\. Gao, Y\. Wan, L\. He, and H\. Bos\.CacheBlend: Fast large language model serving for RAG with cached knowledge fusion\.In*Proc\. EuroSys*, 2025\.
- Su et al\. \[2021\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, vol\. 568, 2024\.
- Vaswani et al\. \[2017\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, vol\. 30, 2017\.
- Wolf et al\. \[2020\]T\. Wolf*et al\.*Transformers: State\-of\-the\-art natural language processing\.In*Proc\. EMNLP \(System Demonstrations\)*, 2020\.
- Xiao et al\. \[2023\]G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis\.Efficient streaming language models with attention sinks\.In*Proc\. ICLR*, 2024\.
- Zhang et al\. \[2024\]Y\. Zhang*et al\.*PyramidKV: Dynamic KV cache compression based on pyramidal information funneling\.*arXiv preprint*, 2024\.
- Zheng et al\. \[2024\]L\. Zheng*et al\.*Efficiently programming large language models using SGLang\.In*Advances in Neural Information Processing Systems*, 2024\.

## Appendix AData Availability

The KVBoost source code is available at[https://github\.com/pythongiant/kvboost](https://github.com/pythongiant/kvboost)under the MIT License\. Benchmark results, checkpoint files, and figure\-generation scripts are included in the repository underbenchmarks\_and\_experiments/important/\. All experiments use publicly available models from the HuggingFace Model Hub\.

## Appendix BAuthor Contributions

S\. Unnikrishnan: Conceptualization, Methodology, Software, Formal Analysis, Writing — Original Draft, Writing — Review & Editing\.

## Appendix CConflict of Interest

The author declares no conflicts of interest\.

## Appendix DEthics Declaration

This research involves no human subjects, personal data, or sensitive data\. No ethics approval was required\.