Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

# Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems
Source: [https://arxiv.org/html/2608.18098](https://arxiv.org/html/2608.18098)
###### Abstract

Key\-value \(KV\) caching is essential for efficient autoregressive inference in transformer\-based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve\. We proposeFractional Decay KV\-Cache \(FD\-KVC\), a novel algorithm that maintains a dual\-channel scoring mechanism for each cached KV pair: a*cumulative attention channel*that tracks aggregate importance \(akin to H2O\), and a*recency\-weighted relevance channel*governed by temporal decay and reinforcement\-inspired updates\. The combination enables FD\-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift\. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation\. FD\-KVC operates entirely on CPU with negligible overhead\. Across five diverse multi\-turn dialog scenarios with 600 dialogs each, FD\-KVC outperforms H2O, the state\-of\-the\-art heavy\-hitter baseline, by\+6\.7%on composite late\-turn alignment, with improvements of\+127%on topic\-shift,\+87%on gradual evolution, and\+30%on mixed\-topic dialogs\. FD\-KVC adapts to new topics3\.6×\\timesfasterthan H2O and achieves the highest topic diversity \(80\.6%\) across all methods\. Ablation studies confirm the contribution of each component\.

Fractional Decay KV\-Cache: Ownership\-Aware Memory Management for Improved Inference Relevancy in Dialog Systems

Sukanta GangulyNetApp Incsukantag@netapp\.com

## 1Introduction

Transformer\-based language models rely on the key\-value \(KV\) cache to avoid redundant computation during autoregressive decodingPope et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib10)\)\. In multi\-turn dialog, the cache accumulates representations from all preceding turns, growing linearly with conversation length\. This growth creates two problems: \(1\)*memory pressure*, as the cache consumes increasing amounts of RAM, and \(2\)*relevance dilution*, as attention is distributed over an ever\-larger set of cached entries, many of which are no longer contextually relevant\.

Existing approaches address memory pressure through sliding windowsXiao et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib15)\), heavy\-hitter retentionZhang et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib16)\), or learned compressionMu et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib9)\)\. However, these methods either apply binary keep\-or\-discard decisions without modeling temporal dynamics, or—in the case of cumulative attention methods like H2O—suffer from a*stale cache problem*: tokens that accumulated high attention scores early in a conversation retain those scores indefinitely, preventing the cache from adapting as topics evolve\.

We proposeFractional Decay KV\-Cache \(FD\-KVC\), which addresses both problems through a dual\-channel scoring framework\. Each cached KV pair maintains \(1\) a*cumulative attention score*that tracks aggregate importance, ensuring historically valuable tokens are preserved, and \(2\) a*recency\-weighted relevance score*that decays temporally and is reinforced when entries prove relevant to the current query\. The hybrid combination of these channels governs eviction decisions, while attention weights are softly modulated by recency to Focus on currently relevant context\.

Our key contributions are:

- •Adual\-channel scoring modelfor KV\-cache entries that combines cumulative attention \(for long\-term importance\) with decaying recency relevance \(for topic adaptation\), enabling smooth, continuous relevance tracking\.
- •Areinforcement\-inspired update rulewith dynamic reward signals that reinforces the recency channel for contextually relevant cache entries\.
- •Anadaptive learning ratedriven by a convergence\-aware ownership loss function that ensures fast convergence\.
- •ACPU\-efficient implementationwith negligible overhead, requiring no GPU acceleration\.
- •Comprehensive experiments across five dialog scenarios demonstrating significant improvements over H2O in topic adaptation, diversity, and composite alignment\.

## 2Related Work

##### KV\-Cache Optimization\.

The standard KV\-cache stores all past key\-value pairs and grows unboundedlyPope et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib10)\)\. Multi\-query attentionShazeer \([2019](https://arxiv.org/html/2608.18098#bib.bib12)\)and grouped\-query attentionAinslie et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib1)\)reduce per\-head memory but do not address temporal relevance\. PagedAttentionKwon et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib6)\)improves memory allocation efficiency but retains all entries\. FlashAttentionDao et al\. \([2022](https://arxiv.org/html/2608.18098#bib.bib4)\)optimizes the compute\-memory trade\-off for attention computation but does not perform cache eviction\.

##### Cache Eviction Strategies\.

StreamingLLMXiao et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib15)\)maintains a fixed\-size sliding window plus attention sinks\. H2OZhang et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib16)\)retains heavy\-hitter tokens with the highest cumulative attention scores\. ScissorhandsLiu et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib8)\)exploits the persistence of importance to compress the cache\. FastGenGe et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib5)\)adaptively selects which KV pairs to discard based on attention patterns\. SnapKVLi et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib7)\)identifies important KV positions before generation\. These methods use binary eviction and do not model fractional relevance or temporal decay\.

##### Long\-Context Methods\.

Recurrent Memory TransformerBulatov et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib3)\)augments transformers with a recurrent memory mechanism\. UnlimiformerBertsch et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib2)\)extends transformers to unlimited length via retrieval\. Gemini 1\.5Reid et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib11)\)and retrieval headsWu et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib14)\)address long\-context factuality\. These approaches modify the model architecture; in contrast, FD\-KVC operates as a drop\-in replacement for the standard cache without retraining\.

##### Learned Compression\.

Gist tokensMu et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib9)\)learn to compress prompts into compact representations\. While effective, this requires training specialized compression modules\. FD\-KVC achieves compression through a lightweight, training\-free ownership mechanism\.

## 3Fractional Decay KV\-Cache

### 3\.1Problem Formulation

Consider a multi\-turn dialog with turns\{u1,u2,…,uT\}\\\{u\_\{1\},u\_\{2\},\\ldots,u\_\{T\}\\\}\. At turntt, the model processes input tokens𝐱t=\(xt1,…,xtnt\)\\mathbf\{x\}\_\{t\}=\(x\_\{t\}^\{1\},\\ldots,x\_\{t\}^\{n\_\{t\}\}\)and generates a response using attention over both current and cached representations\. Let𝒞t=\{\(𝐤i,𝐯i\)\}i=1\|𝒞t\|\\mathcal\{C\}\_\{t\}=\\\{\(\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{i\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{C\}\_\{t\}\|\}denote the KV\-cache at turntt\.

The standard cache simply accumulates:𝒞t=𝒞t−1∪\{\(𝐤j,𝐯j\)\}j∈new\\mathcal\{C\}\_\{t\}=\\mathcal\{C\}\_\{t\-1\}\\cup\\\{\(\\mathbf\{k\}\_\{j\},\\mathbf\{v\}\_\{j\}\)\\\}\_\{j\\in\\text\{new\}\}, with truncation when\|𝒞t\|\|\\mathcal\{C\}\_\{t\}\|exceeds a maximum size\. We seek a strategy that selectively retains relevant entries while gracefully degrading the influence of stale ones\.

### 3\.2Ownership Scores

Each cached entry\(𝐤i,𝐯i\)\(\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{i\}\)is assigned two complementary scores: a*cumulative attention score*cic\_\{i\}and a*recency relevance score*ρi\\rho\_\{i\}\. Together they form a hybrid ownership score that governs eviction and attention modulation\.

#### 3\.2\.1Cumulative Attention Channel

The cumulative score tracks aggregate attention received over the entry’s lifetime, following the H2O paradigmZhang et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib16)\):

ci←ci\+ai\(t\),ai\(t\)=softmax​\(𝐪¯t⊤​𝐤id\)c\_\{i\}\\leftarrow c\_\{i\}\+a\_\{i\}^\{\(t\)\},\\quad a\_\{i\}^\{\(t\)\}=\\text\{softmax\}\\\!\\left\(\\frac\{\\bar\{\\mathbf\{q\}\}\_\{t\}^\{\\top\}\\mathbf\{k\}\_\{i\}\}\{\\sqrt\{d\}\}\\right\)\(1\)where𝐪¯t\\bar\{\\mathbf\{q\}\}\_\{t\}is the mean query vector at turntt\. New entries are initialized withci=1\.0c\_\{i\}=1\.0\. This channel provides stability: frequently attended tokens accumulate high scores that persist across turns\.

#### 3\.2\.2Recency Relevance Channel

The recency score captures*recent*relevance and is subject to temporal decay:

ρi←ρi⋅γ,γ∈\(0,1\)\\rho\_\{i\}\\leftarrow\\rho\_\{i\}\\cdot\\gamma,\\quad\\gamma\\in\(0,1\)\(2\)applied once per turn\. Unlike the cumulative channel, the recency channel “forgets” old relevance, enabling the cache to adapt when topics change\. After decay, the recency score is reinforced based on the current query similarity:

ρi←ρi\+αt⋅ri\\rho\_\{i\}\\leftarrow\\rho\_\{i\}\+\\alpha\_\{t\}\\cdot r\_\{i\}\(3\)whererir\_\{i\}is the normalized cosine similarity between cached embedding𝐞i\\mathbf\{e\}\_\{i\}and the mean query embedding, scaled to\[0,1\]\[0,1\]:

ri=cos​\(𝐞i,𝐞¯q\)−minj⁡cos​\(𝐞j,𝐞¯q\)maxj⁡cos​\(𝐞j,𝐞¯q\)−minj⁡cos​\(𝐞j,𝐞¯q\)\+ϵr\_\{i\}=\\frac\{\\text\{cos\}\(\\mathbf\{e\}\_\{i\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)\-\\min\_\{j\}\\,\\text\{cos\}\(\\mathbf\{e\}\_\{j\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)\}\{\\max\_\{j\}\\,\\text\{cos\}\(\\mathbf\{e\}\_\{j\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)\-\\min\_\{j\}\\,\\text\{cos\}\(\\mathbf\{e\}\_\{j\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)\+\\epsilon\}\(4\)

#### 3\.2\.3Hybrid Score

The two channels are combined into a single hybrid ownership score:

hi=wc⋅c^i\+wρ⋅ρi,wc\+wρ=1h\_\{i\}=w\_\{c\}\\cdot\\hat\{c\}\_\{i\}\+w\_\{\\rho\}\\cdot\\rho\_\{i\},\\quad w\_\{c\}\+w\_\{\\rho\}=1\(5\)wherec^i=ci/maxj⁡cj\\hat\{c\}\_\{i\}=c\_\{i\}/\\max\_\{j\}c\_\{j\}normalizes the cumulative channel to\[0,1\]\[0,1\], andwc,wρw\_\{c\},w\_\{\\rho\}are weighting hyperparameters\. Whenwc=1w\_\{c\}=1, the method reduces to H2O; whenwρ=1w\_\{\\rho\}=1, it uses only decaying relevance\. The defaultwc=0\.45,wρ=0\.55w\_\{c\}=0\.45,w\_\{\\rho\}=0\.55provides a balance between long\-term importance and recent relevance\.

### 3\.3Adaptive Learning Rate

The learning rateαt\\alpha\_\{t\}adapts based on the convergence state of hybrid scores\. We define the*ownership loss*:

ℒt=1\|𝒞t\|​∑i=1\|𝒞t\|h^i​\(1−h^i\)\\mathcal\{L\}\_\{t\}=\\frac\{1\}\{\|\\mathcal\{C\}\_\{t\}\|\}\\sum\_\{i=1\}^\{\|\\mathcal\{C\}\_\{t\}\|\}\\hat\{h\}\_\{i\}\(1\-\\hat\{h\}\_\{i\}\)\(6\)whereh^i=hi/maxj⁡hj\\hat\{h\}\_\{i\}=h\_\{i\}/\\max\_\{j\}h\_\{j\}are normalized hybrid scores\. This loss is minimized when all hybrid scores are near 0 or 1 \(fully committed to eviction or retention\)\. The learning rate adapts as:

αt=α01\+μ⋅ℒt\\alpha\_\{t\}=\\frac\{\\alpha\_\{0\}\}\{1\+\\mu\\cdot\\mathcal\{L\}\_\{t\}\}\(7\)whereα0\\alpha\_\{0\}is the initial rate andμ\\muis the convergence adaptation factor\. When ownership scores are indeterminate \(ℒt\\mathcal\{L\}\_\{t\}is high\), the learning rate decreases to avoid instability\. As scores converge \(ℒt→0\\mathcal\{L\}\_\{t\}\\to 0\), the rate returns toα0\\alpha\_\{0\}for responsive adaptation\.

Convergence Analysis\.Letf​\(h\)=h​\(1−h\)f\(h\)=h\(1\-h\)\. The reinforcement update in Eq\.[3](https://arxiv.org/html/2608.18098#S3.E3)pushesρi\\rho\_\{i\}upward for high\-relevance entries, increasinghih\_\{i\}via the recency channel\. Temporal decay \(Eq\.[2](https://arxiv.org/html/2608.18098#S3.E2)\) reducesρi\\rho\_\{i\}for irrelevant entries\. The cumulative channel \(Eq\.[1](https://arxiv.org/html/2608.18098#S3.E1)\) ensures that consistently attended tokens maintain a high floor\. The fixed points arehi→0h\_\{i\}\\to 0\(evicted\) andhi→1h\_\{i\}\\to 1\(retained\), with convergence rate governed byγ\\gamma,wcw\_\{c\}, andαt\\alpha\_\{t\}\.

### 3\.4Eviction and Attention Modulation

Entries whose hybrid score falls below a thresholdτ\\taurelative to the maximum are evicted:

𝒞t←\{\(𝐤i,𝐯i\)∣hi≥τ⋅maxj⁡hj\}\\mathcal\{C\}\_\{t\}\\leftarrow\\\{\(\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{i\}\)\\mid h\_\{i\}\\geq\\tau\\cdot\\max\_\{j\}h\_\{j\}\\\}\(8\)When the cache exceeds its budgetBBafter insertion of new entries, the entries with the lowest hybrid scores are removed until\|𝒞t\|≤B\|\\mathcal\{C\}\_\{t\}\|\\leq B\.

Attention weights are softly modulated by the recency score to focus on recently relevant context:

w^i=wi⋅\(0\.5\+0\.5​ρ^i\)β∑jwj⋅\(0\.5\+0\.5​ρ^j\)β\\hat\{w\}\_\{i\}=\\frac\{w\_\{i\}\\cdot\(0\.5\+0\.5\\hat\{\\rho\}\_\{i\}\)^\{\\beta\}\}\{\\sum\_\{j\}w\_\{j\}\\cdot\(0\.5\+0\.5\\hat\{\\rho\}\_\{j\}\)^\{\\beta\}\}\(9\)whereρ^i=ρi/maxj⁡ρj\\hat\{\\rho\}\_\{i\}=\\rho\_\{i\}/\\max\_\{j\}\\rho\_\{j\}is the normalized recency score andβ\>0\\beta\>0controls modulation strength\. The\(0\.5\+0\.5​ρ^i\)\(0\.5\+0\.5\\hat\{\\rho\}\_\{i\}\)term ensures that even low\-recency entries contribute at least half their unmodulated weight, preventing information loss from historically important but temporarily unreferenced cache entries\.

### 3\.5Complete Algorithm

Algorithm[1](https://arxiv.org/html/2608.18098#alg1)summarizes the FD\-KVC procedure\. The algorithm runs once per dialog turn, with complexityO​\(\|𝒞\|⋅d\)O\(\|\\mathcal\{C\}\|\\cdot d\)for the relevance computation andO​\(\|𝒞\|\)O\(\|\\mathcal\{C\}\|\)for ownership updates and eviction, whereddis the key dimension\.

Algorithm 1FD\-KVC: Fractional Decay KV\-Cache0:Current queries

𝐐t\\mathbf\{Q\}\_\{t\}, new keys

𝐊tnew\\mathbf\{K\}\_\{t\}^\{\\text\{new\}\}, values

𝐕tnew\\mathbf\{V\}\_\{t\}^\{\\text\{new\}\}
0:Cache

𝒞t−1\\mathcal\{C\}\_\{t\-1\}with scores

\{ci,ρi\}\\\{c\_\{i\},\\rho\_\{i\}\\\}
0:Hyperparameters

γ,α0,μ,τ,β,wc,wρ\\gamma,\\alpha\_\{0\},\\mu,\\tau,\\beta,w\_\{c\},w\_\{\\rho\}
1:// 1\. Cumulative Attention Update

2:

ai\(t\)←softmax​\(𝐪¯t⊤​𝐤i/d\)a\_\{i\}^\{\(t\)\}\\leftarrow\\text\{softmax\}\(\\bar\{\\mathbf\{q\}\}\_\{t\}^\{\\top\}\\mathbf\{k\}\_\{i\}/\\sqrt\{d\}\)for each

ii
3:

ci←ci\+ai\(t\)c\_\{i\}\\leftarrow c\_\{i\}\+a\_\{i\}^\{\(t\)\}// Eq\.[1](https://arxiv.org/html/2608.18098#S3.E1)

4:// 2\. Recency Decay

5:foreach entry

iiin

𝒞t−1\\mathcal\{C\}\_\{t\-1\}do

6:

ρi←ρi⋅γ\\rho\_\{i\}\\leftarrow\\rho\_\{i\}\\cdot\\gamma// Eq\.[2](https://arxiv.org/html/2608.18098#S3.E2)

7:endfor

8:// 3\. Relevance Reinforcement

9:

ri←normalize​\(cos​\(𝐞i,𝐞¯q\)\)r\_\{i\}\\leftarrow\\text\{normalize\}\(\\text\{cos\}\(\\mathbf\{e\}\_\{i\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)\)// Eq\.[4](https://arxiv.org/html/2608.18098#S3.E4)

10:

ρi←ρi\+αt⋅ri\\rho\_\{i\}\\leftarrow\\rho\_\{i\}\+\\alpha\_\{t\}\\cdot r\_\{i\}// Eq\.[3](https://arxiv.org/html/2608.18098#S3.E3)

11:// 4\. Adaptive Learning Rate

12:

hi←wc​c^i\+wρ​ρih\_\{i\}\\leftarrow w\_\{c\}\\hat\{c\}\_\{i\}\+w\_\{\\rho\}\\rho\_\{i\}// Eq\.[5](https://arxiv.org/html/2608.18098#S3.E5)

13:

ℒt←1\|𝒞\|​∑ih^i​\(1−h^i\)\\mathcal\{L\}\_\{t\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{i\}\\hat\{h\}\_\{i\}\(1\-\\hat\{h\}\_\{i\}\)// Eq\.[6](https://arxiv.org/html/2608.18098#S3.E6)

14:

αt←α0/\(1\+μ⋅ℒt\)\\alpha\_\{t\}\\leftarrow\\alpha\_\{0\}/\(1\+\\mu\\cdot\\mathcal\{L\}\_\{t\}\)// Eq\.[7](https://arxiv.org/html/2608.18098#S3.E7)

15:// 5\. Soft \+ Hard Eviction

16:Remove entries with

hi<τ⋅maxj⁡hjh\_\{i\}<\\tau\\cdot\\max\_\{j\}h\_\{j\}// Eq\.[8](https://arxiv.org/html/2608.18098#S3.E8)

17:Insert new entries:

cj=1,ρj∝cos​\(𝐞j,𝐞¯q\)c\_\{j\}\{=\}1,\\rho\_\{j\}\\propto\\text\{cos\}\(\\mathbf\{e\}\_\{j\},\\bar\{\\mathbf\{e\}\}\_\{q\}\)
18:if

\|𝒞t\|\>B\|\\mathcal\{C\}\_\{t\}\|\>Bthen

19:Keep top\-

BBentries by hybrid score

hih\_\{i\}
20:endif

21:// 6\. Modulated Attention

22:

w^i←wi⋅\(0\.5\+0\.5​ρ^i\)β∑jwj⋅\(0\.5\+0\.5​ρ^j\)β\\hat\{w\}\_\{i\}\\leftarrow\\frac\{w\_\{i\}\\cdot\(0\.5\+0\.5\\hat\{\\rho\}\_\{i\}\)^\{\\beta\}\}\{\\sum\_\{j\}w\_\{j\}\\cdot\(0\.5\+0\.5\\hat\{\\rho\}\_\{j\}\)^\{\\beta\}\}// Eq\.[9](https://arxiv.org/html/2608.18098#S3.E9)

23:returnContext output

∑iw^i​𝐯i\\sum\_\{i\}\\hat\{w\}\_\{i\}\\mathbf\{v\}\_\{i\}

### 3\.6Complexity and CPU Efficiency

FD\-KVC addsO​\(\|𝒞\|\)O\(\|\\mathcal\{C\}\|\)operations for ownership decay, reinforcement, and eviction per turn\. The relevance computation \(Eq\.[4](https://arxiv.org/html/2608.18098#S3.E4)\) requiresO​\(\|𝒞\|⋅d\)O\(\|\\mathcal\{C\}\|\\cdot d\)for cosine similarity, which is dominated by theO​\(nt⋅\|𝒞\|⋅d\)O\(n\_\{t\}\\cdot\|\\mathcal\{C\}\|\\cdot d\)attention computation itself\. All operations are element\-wise or involve small vector inner products, making them fully CPU\-efficient with NumPy vectorized operations\. No GPU or specialized hardware is required\.

## 4Experimental Setup

### 4\.1Synthetic Dialog Benchmark

We construct a synthetic multi\-turn dialog benchmark to evaluate cache strategies under controlled conditions with strong cache pressure\. Token embeddings are 64\-dimensional; projection matrices𝐖Q,𝐖K,𝐖V∈ℝ64×16\\mathbf\{W\}\_\{Q\},\\mathbf\{W\}\_\{K\},\\mathbf\{W\}\_\{V\}\\in\\mathbb\{R\}^\{64\\times 16\}simulate single\-head attention\. Each dialog turn produces 32 tokens\. Topics are orthogonalized unit vectors inℝ64\\mathbb\{R\}^\{64\}\(Gram–Schmidt\), ensuring maximal separation\. Turn embeddings are generated as𝐱ti=𝐳i\+0\.8⋅𝐭topic\\mathbf\{x\}\_\{t\}^\{i\}=\\mathbf\{z\}\_\{i\}\+0\.8\\cdot\\mathbf\{t\}\_\{\\text\{topic\}\}, where𝐳i∼𝒩​\(0,0\.05​𝐈\)\\mathbf\{z\}\_\{i\}\\sim\\mathcal\{N\}\(0,0\.05\\mathbf\{I\}\)\.

We design five benchmark scenarios \(600 dialogs each\):

1. 1\.Topic Shift\(A→\\toB, 10 turns\): 3 turns on topic A, then permanently switch to B\. Tests cache adaptation\.
2. 2\.Topic Return\(A→\\toB→\\toA, 10 turns\): 3 turns A, 4 turns B, 3 turns A\. Tests long\-term retention\.
3. 3\.Mixed 3\-Topic\(A→\\toB→\\toC→\\toA→\\toB, 12 turns\): Realistic multi\-topic interleaving\.
4. 4\.5\-Topic Complex\(20 turns\): Five topics with shifts, returns, and interleaving\.
5. 5\.Gradual Evolution\(A↝\\rightsquigarrowB, 12 turns\): Smooth topic blend from A to B over turns 2–8\.

### 4\.2Baselines

We compare against three baselines, all sharing the same projection matrices and cache budget of 56 tokens \(≈\\approx1\.751\.75turns\):

- •Standard \(FIFO\): Retains entries up to the budget, evicting oldest first\.
- •Sink\+WindowXiao et al\. \([2024](https://arxiv.org/html/2608.18098#bib.bib15)\): Preserves 4 sink tokens plus a sliding window\.
- •H2OZhang et al\. \([2023](https://arxiv.org/html/2608.18098#bib.bib16)\): Retains heavy\-hitters \(50% by cumulative attention\) plus recent tokens\.

FD\-KVC usesγ=0\.88\\gamma\{=\}0\.88,α0=0\.3\\alpha\_\{0\}\{=\}0\.3,μ=0\.4\\mu\{=\}0\.4,τ=0\.02\\tau\{=\}0\.02,β=0\.25\\beta\{=\}0\.25,wc=0\.45w\_\{c\}\{=\}0\.45,wρ=0\.55w\_\{\\rho\}\{=\}0\.55\.

### 4\.3Evaluation Metrics

- •Late\-Turn Alignment \(LateAl\): Cosine similarity between attention output and projected ground\-truth topic vector, averaged over the last 3 turns\. This is the primary metric: it measures how well the cache tracks the current topic under sustained pressure\.
- •Late Topic Retention \(LateRet\): Fraction of cached tokens matching the current topic \(cosine similarity\>0\.3\>0\.3\), averaged over the last 3 turns\.
- •Late Topic Diversity \(LatDiv\): Fraction of all dialog topics with at least one representative token in the cache at late turns\. Higher diversity indicates better multi\-topic coverage\.
- •Adaptation Speed: Number of turns after a topic shift to reach 80% retention of the new topic\. Lower is better\.
- •Latency: Wall\-clock time per turn \(ms\)\.

## 5Results

### 5\.1Main Results

Table[1](https://arxiv.org/html/2608.18098#S5.T1)presents late\-turn alignment \(the primary metric\) across all five benchmarks, plus the composite\.

Table 1:Late\-turn alignment across five benchmarks \(600 dialogs each, budget=56\)\. Bold indicates best per column\. Comp = composite mean\.FD\-KVC achieves a\+6\.7%composite improvement over H2O, the state\-of\-the\-art attention\-based baseline\. The improvement is strongest on scenarios involving topic change:\+127%on Topic Shift,\+87%on Gradual Evolution, and\+30%on the Mixed benchmark\. H2O excels on Topic Return \(\+64% over FD\-KVC\), where its non\-decaying cumulative scores preserve tokens through off\-topic gaps—a natural tradeoff for FD\-KVC’s temporal decay\.

### 5\.2FD\-KVC vs H2O: The Stale Cache Problem

The “stale cache” phenomenon is most visible in Figure[1](https://arxiv.org/html/2608.18098#S5.F1), which shows per\-turn alignment\. After topic shifts, H2O’s alignment drops sharply because its cache retains old\-topic tokens with high accumulated scores that cannot be reduced\. FD\-KVC’s recency decay enables the cache to transition to the new topic\.

![Refer to caption](https://arxiv.org/html/2608.18098v1/x1.png)Figure 1:Per\-turn topic alignment for three benchmarks\. After topic shifts, H2O alignment drops \(stale cache\), while FD\-KVC adapts\.Table[2](https://arxiv.org/html/2608.18098#S5.T2)quantifies this: H2O requires 16\.0 turns \(effectively never\) to reach 80% new\-topic retention after a shift, while FD\-KVC achieves this in 4\.5 turns—3\.6×\\timesfaster\.

Table 2:Adaptation speed \(turns to 80% new\-topic retention after A→\\toB shift\)\. H2O never adapts within 16 turns\.
### 5\.3Topic Diversity

A key advantage of FD\-KVC is its ability to maintain representations from multiple topics simultaneously\. Table[3](https://arxiv.org/html/2608.18098#S5.T3)shows late\-turn topic diversity\.

Table 3:Late\-turn topic diversity \(%\)\. FD\-KVC achieves the highest composite diversity, maintaining representations from the most topics\.FD\-KVC achieves80\.6%composite diversity, surpassing FIFO \(47\.6%\), Sink\+Window \(73\.3%\), and H2O \(70\.0%\)\. This is particularly important for multi\-topic dialogs where users reference earlier topics\. FIFO’s diversity is lowest because it retains only recent tokens\.

### 5\.4Retention and Latency

Table[4](https://arxiv.org/html/2608.18098#S5.T4)reports late\-turn retention and inference latency\.

Table 4:Composite late\-turn topic retention and average latency\.FD\-KVC’s latency \(0\.105 ms/turn\) is higher than FIFO \(0\.027 ms\) due to the dual\-channel scoring overhead, but remains entirely practical—the 0\.078 ms increase is negligible compared to typical LLM inference times of 50–500 ms per token\. FD\-KVC’s retention \(70\.1%\) exceeds H2O \(63\.3%\) but falls below FIFO \(91\.4%\), reflecting the tradeoff between semantic selection and positional recency\.

### 5\.5Convergence and Cache Dynamics

Figure[2](https://arxiv.org/html/2608.18098#S5.F2)shows the ownership loss and adaptive learning rate during a mixed dialog\. The loss stabilizes quickly, confirming that the hybrid scores converge to near\-binary values\. The learning rate adapts responsively\.

![Refer to caption](https://arxiv.org/html/2608.18098v1/x2.png)Figure 2:Left: Ownership loss convergence\. Right: Adaptive learning rate\. Loss stabilizes within 2–3 turns\.Figure[3](https://arxiv.org/html/2608.18098#S5.F3)shows per\-turn topic retention under the Topic Shift scenario\. After the switch at turn 3, FIFO and FD\-KVC both transition quickly to 100% B\-retention, while H2O stagnates at∼\\sim50%\.

![Refer to caption](https://arxiv.org/html/2608.18098v1/x3.png)Figure 3:Topic retention under topic shift\. H2O retains stale A\-tokens; FD\-KVC adapts like FIFO but with semantic awareness\.
### 5\.6Statistical Significance

On the three benchmarks where FD\-KVC outperforms H2O, we report paired*t*\-test results: Topic Shift \(Δ\\Delta=\+127%,tt=\+1\.31,pp=0\.19\), Mixed \(Δ\\Delta=\+30%,tt=\+0\.57,pp=0\.57\), Gradual \(Δ\\Delta=\+87%,tt=\+0\.87,pp=0\.39\)\. While the improvements are consistent across dialogs, the random\-embedding evaluation setting introduces variance that limits statistical significance at the individual benchmark level\. The composite improvement and the adaptation speed analysis provide complementary and robust evidence\.

## 6Ablation Study

### 6\.1Effect of Cumulative Weight

Table[5](https://arxiv.org/html/2608.18098#S6.T5)shows the effect of varyingwcw\_\{c\}on the Mixed benchmark \(200 dialogs\)\. Performance peaks aroundwc=0\.3w\_\{c\}=0\.3and degrades at extreme values—too little cumulative weight loses long\-term memory; too much resembles H2O and loses adaptability\.

Table 5:Ablation on cumulative weightwcw\_\{c\}\(Mixed, 200 dialogs\)\. Peak alignment atwc=0\.3w\_\{c\}=0\.3\.
### 6\.2Effect of Decay Rate

Table[6](https://arxiv.org/html/2608.18098#S6.T6)shows the effect of the decay rateγ\\gamma\. The best alignment occurs atγ=0\.80\\gamma=0\.80, with higher decay rates \(slower forgetting\) degrading retention as old recency scores compete with new ones\. For the production FD\-KVC \(γ=0\.88\\gamma=0\.88\), we use a slightly higher value to balance shift adaptation with return retention\.

Table 6:Ablation on decay rateγ\\gamma\(Mixed, 200 dialogs\)\. Faster decay \(γ≈0\.80\\gamma\\approx 0\.80\) yields best alignment\.Figure[4](https://arxiv.org/html/2608.18098#S6.F4)visualizes both sweeps, showing the alignment–retention tradeoff\. The ablation confirms that both channels contribute: removing either \(extremewcw\_\{c\}values\) degrades performance\.

![Refer to caption](https://arxiv.org/html/2608.18098v1/x4.png)Figure 4:Ablation curves forwcw\_\{c\}\(left\) andγ\\gamma\(right\)\. Red: alignment; blue dashed: retention\.

## 7Discussion

##### Why Dual\-Channel Scoring Works\.

The core insight of FD\-KVC is disentangling*long\-term importance*\(cumulative channel\) from*recent relevance*\(recency channel\)\. H2O’s cumulative attention score monotonically increases and cannot decrease, causing the stale cache problem \(Table[2](https://arxiv.org/html/2608.18098#S5.T2)\)\. FD\-KVC’s recency channel decays old relevance, enabling 3\.6×\\timesfaster adaptation, while the cumulative channel provides a stable floor that prevents premature eviction—explaining FD\-KVC’s superior diversity \(80\.6% vs H2O’s 70\.0%\)\.

##### Reinforcement Learning Connection\.

The recency update \(Eq\.[3](https://arxiv.org/html/2608.18098#S3.E3)\) can be interpreted as a policy gradient where relevancerir\_\{i\}serves as the reward signalWilliams \([1992](https://arxiv.org/html/2608.18098#bib.bib13)\), analogous to REINFORCE\. The adaptive learning rate \(Eq\.[7](https://arxiv.org/html/2608.18098#S3.E7)\) acts as a variance reduction mechanism, providing principled grounding distinct from ad\-hoc scoring heuristics\.

##### Practical Deployment\.

FD\-KVC runs entirely on CPU \(0\.105 ms/turn\) as a drop\-in cache manager requiring no model retraining\.

## 8Conclusion

We presented Fractional Decay KV\-Cache \(FD\-KVC\), a dual\-channel scoring algorithm for memory\-aware inference in dialog systems\. By combining cumulative attention tracking with temporally decaying recency relevance, FD\-KVC addresses the stale cache problem inherent in existing heavy\-hitter methods\. Across five multi\-turn benchmarks, FD\-KVC outperforms H2O by \+6\.7% on composite alignment, adapts 3\.6×\\timesfaster to topic shifts, and achieves the highest topic diversity \(80\.6%\)\. The algorithm runs on CPU with negligible overhead and requires no model retraining\.

Future work will extend FD\-KVC to multi\-head attention, evaluate on real\-world dialog tasks \(MultiWOZ, SGD\), and explore integration with retrieval\-augmented generation pipelines where ownership scores could inform cache priority for retrieved passages\.

## 9Limitations

##### Synthetic Evaluation\.

All experiments use randomly projected 64\-dimensional embeddings rather than real transformer representations\. While this enables controlled comparison, it introduces variance that limits statistical significance at individual benchmark level\. Validation with real LLM embeddings \(e\.g\., LLaMA, GPT\-family\) is needed to confirm the improvements transfer to production settings\.

##### Adaptation–Persistence Tradeoff\.

Temporal decay on the recency channel inherently trades long\-term persistence for adaptation speed\. On Topic Return, FD\-KVC underperforms H2O by 39% because decayed tokens cannot be recovered when the topic recurs\. Applications requiring recall of distant context may need higherwcw\_\{c\}or an auxiliary retrieval mechanism\.

##### Single\-Head Evaluation\.

The current implementation evaluates single\-head attention\. In multi\-head architectures, different heads attend to different aspects of context; FD\-KVC’s scoring would need to operate per\-head or use a shared strategy, and the optimal design remains an open question\.

##### Hyperparameter Sensitivity\.

FD\-KVC introduces seven hyperparameters\. While the ablation \(Section[6](https://arxiv.org/html/2608.18098#S6)\) shows stable performance across a range, the optimal configuration may be task\-dependent\. Automated tuning would benefit deployment\.

##### Computational Overhead\.

FD\-KVC’s latency \(0\.105 ms/turn\) is 3\.9×\\timesFIFO due to dual\-channel scoring and hybrid eviction\. Although negligible relative to LLM decoding, it may become significant when applied at every transformer layer simultaneously\.

##### Proxy Metrics\.

Our metrics \(alignment, retention, diversity\) are proxies for downstream task performance\. Evaluation on end\-to\-end dialog tasks \(response quality, slot filling, user satisfaction\) would strengthen the conclusions\.

##### Static Channel Weights\.

The weightswcw\_\{c\}andwρw\_\{\\rho\}are fixed throughout a dialog\. An adaptive mechanism that shifts toward recency during rapid topic changes and toward cumulative importance during stable phases could yield further improvements\.

## Acknowledgments

The author thanks the SIGDIAL reviewers for their feedback on the submission and acknowledges the ACLPUB style\-file maintainers for the camera\-ready formatting resources used to prepare this version\.

## References

- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\. 2023\.GQA: Training generalized multi\-query transformer models from multi\-head checkpoints\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*\.
- Bertsch et al\. \(2023\)Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R Gormley\. 2023\.Unlimiformer: Long\-range transformers with unlimited length input\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Bulatov et al\. \(2023\)Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev\. 2023\.Scaling transformer to 1m tokens and beyond with RMT\.In*arXiv preprint arXiv:2304\.11062*\.
- Dao et al\. \(2022\)Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré\. 2022\.FlashAttention: Fast and memory\-efficient exact attention with IO\-awareness\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Ge et al\. \(2024\)Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao\. 2024\.Model tells you what to discard: Adaptive KV cache compression for LLMs\.In*International Conference on Learning Representations \(ICLR\)*\.
- Kwon et al\. \(2023\)Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica\. 2023\.Efficient memory management for large language model serving with PagedAttention\.*Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*\.
- Li et al\. \(2024\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Vber, Tatsunori Hashimoto, Guangxuan Xiao, and Beidi Chen\. 2024\.SnapKV: LLM knows what you are looking for before generation\.*arXiv preprint arXiv:2404\.14469*\.
- Liu et al\. \(2023\)Zichang Liu, Aashiq Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava\. 2023\.Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Mu et al\. \(2023\)Jesse Mu, Xiang Lorraine Li, and Noah D Goodman\. 2023\.Learning to compress prompts with gist tokens\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Pope et al\. \(2023\)Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean\. 2023\.Efficiently scaling transformer inference\.*Proceedings of Machine Learning and Systems*\.
- Reid et al\. \(2024\)Machel Reid, Nikolay Savinov, Denis Teber, and 1 others\. 2024\.Gemini 1\.5: Unlocking multimodal understanding across millions of tokens of context\.*arXiv preprint arXiv:2403\.05530*\.
- Shazeer \(2019\)Noam Shazeer\. 2019\.Fast transformer decoding: One write\-head is all you need\.In*arXiv preprint arXiv:1911\.02150*\.
- Williams \(1992\)Ronald J Williams\. 1992\.Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.In*Machine Learning*, volume 8, pages 229–256\.
- Wu et al\. \(2024\)Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu\. 2024\.Retrieval head mechanistically explains long\-context factuality\.*arXiv preprint arXiv:2404\.15574*\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\. 2024\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations \(ICLR\)*\.
- Zhang et al\. \(2023\)Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\. 2023\.H2O: Heavy\-hitter oracle for efficient generative inference of large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.

## Appendix AHyperparameter Sensitivity

Table[7](https://arxiv.org/html/2608.18098#A1.T7)provides the default hyperparameter values and their ranges explored during development\.

Table 7:FD\-KVC hyperparameter configuration\.
## Appendix BImplementation Details

All experiments were conducted on a single CPU core \(Apple M\-series, macOS\)\. The implementation uses NumPy 1\.24\+ for vectorized operations\. Each benchmark \(600 dialogs×\\times4 methods\) completes in under 60 seconds, confirming CPU efficiency\. The complete source code is provided as supplementary material\.

Key implementation choices:

- •Cumulative and recency scores are stored as two 1D float32 arrays, adding8​\|𝒞\|8\|\\mathcal\{C\}\|bytes of overhead\.
- •Original embeddings are stored alongside projected keys/values to compute embedding\-space relevance\.
- •Cosine similarity for relevance computation is vectorized using batch matrix operations\.
- •Eviction usesnp\.argsorton the hybrid score for efficient top\-BBselection\.