RaReCache：通过基于秩分歧的选择性重计算弥合跨模型 KV 缓存复用的鸿沟

# RaReCache: Bridging the Gap in Cross-Model KV Cache Reuse via Rank disagreement-based Selective Recomputation
Source: [https://arxiv.org/html/2610.11358](https://arxiv.org/html/2610.11358)
Saptarshi MitraSitao HuangSouvik KunduPeter A\. BeerelAffiliation:University of Southern California University of California, Irvine Intel Labs\{sreetama,pabeerel\}@usc\.edu \{saptarm,sitaoh\}@uci\.edu

###### Abstract

Cross\-model KV\-cache reuse remains a key challenge in modern LLM serving\. Coding agents and multi\-model systems increasingly route a shared context across models: a user may switch models mid\-session, or a cascade may escalate a difficult query\. Because KV caches contain model\-specific representations, each switch typically forces the receiving model to prefill the entire context from scratch\. Recent work shows that closed\-form linear maps can translate KV caches between models in the same family, but transfer accuracy degrades as the model\-size gap widens\. In this paper, we establish that these transfer failures are concentrated in a small subset of information\-dense tokens\. To bridge this gap, we introduceRaReCache, a framework that enables a large target model to decode accurately from a cache prefilled by a much smaller source via selective recomputation\. RaReCache identifies these critical positions using a novel*rank disagreement*metric, scoring each token by the energy of its mapped KV in output directions weakly supported by the calibration data\. Across two model families and five benchmarks, on a23×23\\timesparameter gap \(Qwen3\-0\.6B to 14B\) recomputing just 30% of positions retains 95\-99% of the target accuracy, whereas on a8\.8×8\.8\\timesgap \(Llama3\-8B to 70B\), recomputing 40% retains 96\.5% of the target accuracy\. RaReCache largely removes sensitivity to source\-model size, and achieves up to a3\.04×\\mathbf\{3\.04\\times\}prefill speedup\. For online serving, it handles1\.8×1\.8\\timesthe request throughput of target prefill on a single GPU, and at the target’s saturation load, reduces median and 99th\-percentile time\-to\-first\-token \(TTFT\) by5\.0×\\mathbf\{5\.0\\times\}and6\.4×\\mathbf\{6\.4\\times\}respectively, with a 30% recompute budget\. RaReCache establishes an efficient serving paradigm where small models prefill on behalf of massive targets, enabling large models to recompute only critical tokens, drastically reducing prefill latency\.

22footnotetext:Equally contributing authors## 1Introduction

Modern large language model \(LLM\) serving systems increasingly hand off ongoing contexts across models of different capacities\. This paradigm underpins cost\-quality cascades\([Chen et al\., 2023](https://arxiv.org/html/2610.11358#bib.bib10)\), dynamic request routing\([Ong et al\., 2024](https://arxiv.org/html/2610.11358#bib.bib11)\), and coding agents that let users upgrade to stronger models midway through a session\. However, each context switch carries a hidden cost\. A model’s key\-value \(KV\) cache is intrinsically tied to its own architecture and cannot be read by the receiving model, forcing the target model to prefill the entire context from scratch\.

Reusing KV caches is a promising way to avoiding repeated prefill, but existing reuse techniques like prefix caching\([Kwon et al\., 2023](https://arxiv.org/html/2610.11358#bib.bib6);[Zheng et al\., 2024](https://arxiv.org/html/2610.11358#bib.bib5)\)or DroidSpeak\([Liu et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib4)\)are largely confined to identical architectures\. Other cross\-model approaches train neural fusers or latent adapters to boost accuracy rather than save prefill compute\([Fu et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib2);[Dery et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib3)\)\. Closest to our work,[Heo et al\. \(2026\)](https://arxiv.org/html/2610.11358#bib.bib1)introduces a gradient\-free, closed\-form linear map to convert caches between models within a family\. However, their transfer accuracy degrades significantly when the capacity gap between models is large\.

To close the accuracy gap in KV transfer, recent reuse strategies rely on partial target\-side recomputation\. CacheBlend\([Yao et al\., 2025](https://arxiv.org/html/2610.11358#bib.bib7)\)selectively recomputes tokens based on theℓ2\\ell\_\{2\}deviation of their reused KV states in the target’s prefix layers\. However, this requires processing every token through these initial layers, and raw error magnitude often fails to predict downstream accuracy loss\. Alternatively, DroidSpeak\([Liu et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib4)\)proposes selectively recomputing full intermediate layers which fundamentally requires identical model architectures\. Consequently, achieving accurate cross\-model KV transfer across large capacity gaps requires a new, architecture\-agnostic metric for selective recomputation\.

Figure 1:The figure illustrates our RaReCache pipeline where target cache constitutes mapped cache tokens from source and recomputed tokens in target \(left\)\. RaReCache closes accuracy gap to target while yielding2\.6×2\.6\\timeslower prefill FLOPs \(right\)\.To address these limitations, we presentRaReCache\(Rank disagreement\-based SelectiveRecomputation\), a framework that bridges the capacity gap in cross\-model KV cache transfer\. Rather than relying on standard error magnitudes or attention scores, RaReCache selectively recomputes only the most critical, information\-dense tokens in the target model using a novel metric called*rank disagreement*\. This approach avoids complete target\-side prefilling, enabling highly efficient context hand\-offs without sacrificing the reliability of downstream tasks\.

Contributions\.We introduce RaReCache, driven by*rank disagreement*, a closed\-form metric that identifies which tokens require target\-side recomputation without executing any forward passes in the target\. By measuring the energy of a token’s mapped KV cache outside the well\-described calibration subspace, it isolates critical out\-of\-distribution representations\. We demonstrate why common heuristics fail: attention mechanisms select standard formatting tokens the map already reproduces, while raw mapping error conflates harmless variations on template text with crucial novel content\. Rank disagreement outperforms both, closing the accuracy gap \(§[4\.3](https://arxiv.org/html/2610.11358#S4.SS3)\)\.

We show that a small model’s cache, mapped in closed form and selectively repaired using RaReCache, recovers the accuracy lost by plain transfer even across extreme scale differences\. On a23×23\\timesparameter gap \(Qwen3\-0\.6B to 14B\), recomputing just 30% of positions retains 95–99% of the target’s native accuracy on reasoning benchmarks \(GSM8K and ARC\), largely decoupling transfer quality from the source model’s size\. These gains generalize seamlessly across architectures: on the Llama 3 family\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.11358#bib.bib14)\), transferring from the 8B to the 70B model \(8\.8×8\.8\\timesgap\) at a 40% budget retains 96\.5% of the 70B’s native GSM8K performance \(§[4\.2](https://arxiv.org/html/2610.11358#S4.SS2)\)\.

Because the large target model computes only a fraction of positions selected by rank disagreement, prefill latency during model switch is drastically reduced\. We demonstrate that RaReCache is up to3\.04×\\timesfaster than the large model prefilling the full context from scratch \(§[4\.6](https://arxiv.org/html/2610.11358#S4.SS6)\)\. Under live serving conditions with dynamic request batching, this efficiency reduces head\-of\-line blocking, lowering median TTFT by5\.0×\\mathbf\{5\.0\\times\}and 99th\-percentile tail latency \(p99\) by6\.4×\\mathbf\{6\.4\\times\}at a30%30\\%recompute budget\.

## 2Motivational Case Study

### 2\.1The Capacity Bottleneck of Linear Mapping

We investigate whether linearly mapping a prefilled KV cache from a small source model, as proposed in\([Heo et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib1)\), can fully recover a large target model’s native accuracy\. Figure[2](https://arxiv.org/html/2610.11358#S2.F2)demostrates mapping Qwen3\-0\.6B into Qwen3\-14B\([Yang et al\., 2025](https://arxiv.org/html/2610.11358#bib.bib13)\), a23×23\\timesgap, where marker size is proportional to the parameter count of prefill model\. Transfer using linear map yields a GSM8K accuracy of 71\.6% compared to 95\.1% for the 14B model\. While mapping lifts the accuracy for every source, the remaining gap in accuracy between the source and target is still significant, and this gap grows as the source model shrinks\. We also observe that the fraction of the target’s key variance that the linear map explains barely changes with source size while the accuracy drops a lot\.

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/paper_transfer_pairs.png)Figure 2:GSM8K acc\. for mapping source KV to target on Qwen3\. Marker size is proportional to the parameter count of prefill model\.Mapping more source layers to a single target layer yields negligible gains while increasing the parameter count\.

We hypothesize that the mapping extracts what the source representations linearly encode, and thus the shortfall does not necessarily come from a poorly fitted map\. The remaining gap is bounded by the small model’s representational capacity and must be supplied by the target\. We therefore allow the target to*recompute*a fraction of prompt positions while reading the mapped cache from the source model elsewhere\.

### 2\.2Can Attention identify which tokens to recompute?

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/token_strip_windows_log_baselines.png)Figure 3:Token\-by\-token scores for each selector on a single GSM8K prompt\. Columns are selected tokens from four sections of the prompt \(template, question, instruction, and header\)\. The first four rows marks the tokens in the top 20% positions determined by four different selection metrics\. The last three rows are tokenwise scores for three of the four metrics, excluding the random selection metric which has no associated score\.![Refer to caption](https://arxiv.org/html/2610.11358v1/images/selector_gap_gsm8k_300.png)Figure 4:GSM8K accuracy of Qwen3\-0\.6B→\\to14B when 30% of prompt positions are recomputed, chosen by four selectors\. Arrows give the remaining gap to the 14B alone\.We analyze token selection strategies for selective recomputation in the target\. KV\-cache eviction methods rank tokens by the attention they receive, typically computed over a local query window\([Zhang et al\., 2023](https://arxiv.org/html/2610.11358#bib.bib8);[Li et al\., 2024](https://arxiv.org/html/2610.11358#bib.bib9)\)\. We investigate whether attention can be a reliable metric for selective recomputation\. Figure[3](https://arxiv.org/html/2610.11358#S2.F3)follows one GSM8K prompt, mapped from Qwen3\-0\.6B into Qwen3\-14B, token by token\. It shows the map’s relativeℓ2\\ell\_\{2\}error \(∥mapped−true∥2/∥true∥2\\lVert\\mathrm\{mapped\}\-\\mathrm\{true\}\\rVert^\{2\}/\\lVert\\mathrm\{true\}\\rVert^\{2\}\) calculated for keys and values together, averaged over layers, the source and target model attention each token receives from both models \(last 32 query positions for the last quarter of layers\), and random selection\. In Figure[4](https://arxiv.org/html/2610.11358#S2.F4), we plot the GSM8K accuracy of Qwen3\-0\.6B→\\to14B on 300 samples when 30% of prompt positions are recomputed, chosen by each selector\.

We observe that attention is overwhelmingly concentrated on the initial sink tokens and prompt\-terminating chat templates\. The 14B puts 72% of its attention on the<\|im\_start\|\>sink and 24% on the rest of the template and instruction, leaving 4% for the question\. The source attention tracks the target model’s attention closely\. Both attention selectors score below random and match transfer with no recomputation\. Selecting by attention spends its budget on the template, and thus recomputes the tokens the map already reproduces\. Atρ=0\.2\\rho\{=\}0\.2, attention does not pick a single question token\. Theℓ2\\ell\_\{2\}mapping\-error selector, which reads the 14B’s own prefill, also does no better than random, confirming that rawℓ2\\ell\_\{2\}magnitude fails to identify the critical tokens that actually degrade downstream accuracy\. This motivates the need for a new metric that focuses on the actual question content and can accurately identify which tokens to recompute\.

## 3Methodology

We propose RaReCache, a framework that bridges cross\-model capacity gaps by translating source KV caches and selectively recomputing only the most critical target tokens, as illustrated in Figure[5](https://arxiv.org/html/2610.11358#S3.F5)\. In this section, we formalize the base linear mapping \(§[3\.1](https://arxiv.org/html/2610.11358#S3.SS1)\), our selective recomputation framework \(§[3\.2](https://arxiv.org/html/2610.11358#S3.SS2)\), and the target\-free score used to select those positions \(§[3\.2](https://arxiv.org/html/2610.11358#S3.SS2)\)\.

### 3\.1Mapping the cache

Source and target caches differ in depth, width, and number of layers, but within a family they share a tokenizer, so a prompt ofTTtokens producesTTcache entries in both\. Following[Heo et al\. \(2026\)](https://arxiv.org/html/2610.11358#bib.bib1), we fit an independent linear map per target layerℓ\\elland per cache tensorσ∈\{key,value\}\\sigma\\in\\\{\\text\{key\},\\text\{value\}\\\}\. For simplicity, we omitℓ\\ellandσ\\sigmafrom the notations below\.

Inputs and outputs\.Let a target model’s cache entry be a row vector𝐲∈ℝdt\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\_\{t\}\}, wheredt=nkvt​dhtd\_\{t\}=n^\{t\}\_\{\\text\{kv\}\}d^\{t\}\_\{h\}concatenates the target’s KV heads\. To construct the corresponding source input𝐱∈ℝds\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{s\}\}, we concatenate the entries fromKKspecific source layers\. To determine theseKKlayers, we fit a single\-source least\-squares regression from each source layer’s cache entry to the target layer’s, evaluating each by itsR2R^\{2\}score, and select the topKKsource layers with the highest explained variance\([Heo et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib1)\)\. This results in an input dimension ofds=k​nkvs​dhsd\_\{s\}=k\\,n^\{s\}\_\{\\text\{kv\}\}d^\{s\}\_\{h\}\. Keys are de\-rotated before fitting and re\-rotated after mapping, so the map sees content without position and transfers across context lengths\.

Fitting\.StackingNNcalibration tokens yields matricesX∈ℝN×dsX\\in\\mathbb\{R\}^\{N\\times d\_\{s\}\}andY∈ℝN×dtY\\in\\mathbb\{R\}^\{N\\times d\_\{t\}\}\. After centering both by subtracting their respective column means𝐱¯\\bar\{\\mathbf\{x\}\}and𝐲¯\\bar\{\\mathbf\{y\}\}, we obtain the mapping parameters via ridge regression in closed form:

W=\(X⊤​X\+λ​I\)−1​X⊤​Y,𝐛=𝐲¯−𝐱¯​W,𝐲^=𝐱​W\+𝐛,W=\(X^\{\\top\}X\+\\lambda I\)^\{\-1\}X^\{\\top\}Y,\\qquad\\mathbf\{b\}=\\bar\{\\mathbf\{y\}\}\-\\bar\{\\mathbf\{x\}\}W,\\qquad\\hat\{\\mathbf\{y\}\}=\\mathbf\{x\}W\+\\mathbf\{b\},\(1\)whereW∈ℝds×dtW\\in\\mathbb\{R\}^\{d\_\{s\}\\times d\_\{t\}\}is the projection matrix and𝐲^\\hat\{\\mathbf\{y\}\}is the estimated target cache entry\. The bias𝐛\\mathbf\{b\}aligns the source and target means\. Because theKKselected source layers are highly correlated, theL2L\_\{2\}penaltyλ\\lambdais required to regularize the near\-singularX⊤​XX^\{\\top\}X\. This closed\-form fit is highly efficient: a single offline pass over the calibration set accumulatesX⊤​XX^\{\\top\}XandX⊤​YX^\{\\top\}Y, yieldingWWvia one linear solve without gradient descent\.

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/Kv_cache_transfer.png)Figure 5:The RaReCache Pipeline: The source model prefills the prompt, concatenatingKKselected layers into inputxix\_\{i\}\. An offline\-fitted linear map generates both a full\-rank target cache prediction \(y^i\\hat\{y\}\_\{i\}\) and a reduced\-rank projection \(y^i\(r\)\\hat\{y\}\_\{i\}^\{\(r\)\}\)\. Rank disagreement \(sis\_\{i\}\) calculates the squared gap between these predictions\. As illustrated, tokens whose predictions project outside the familiar calibration subspacespan​\(Vr\)\\text\{span\}\(V\_\{r\}\)yield highsis\_\{i\}scores\. To save compute, the target model only runs a forward pass on the top⌈ρ​T⌉\\lceil\\rho T\\rceilpositions,ρ\\rhobeing the recompute ratio, with the highestsis\_\{i\}, merging these recomputed KV tensors with the directly mapped ones to form the final target model cache\.
### 3\.2RaReCache: Selective Recomputation via Rank Disagreement

Letρ∈\[0,1\]\\rho\\in\[0,1\]be a recompute budget\. We select the top⌈ρ​T⌉\\lceil\\rho T\\rceiltokens in the prompt to be recomputed by the target model, while reading the linearly mapped cache everywhere else\. To maximize efficiency, we use a single shared set of marked positions across all layers \(the top 30% positions at two different layers overlap by 0\.87, as shown in the Appendix\)\.

Rank disagreement\.Instead of estimating mapping error, we measure how much a token’s representation relies on directions the calibration data left undetermined\. When the fitted map is applied to the calibration tokens, their predictions𝐲^1,…,𝐲^N\\hat\{\\mathbf\{y\}\}\_\{1\},\\dots,\\hat\{\\mathbf\{y\}\}\_\{N\}form a distribution inℝdt\\mathbb\{R\}^\{d\_\{t\}\}\. Its covarianceΣy^=V​Λ​V⊤\\Sigma\_\{\\hat\{y\}\}=V\\Lambda V^\{\\top\}yields orthonormal eigenvectors𝐯1,…,𝐯dt\\mathbf\{v\}\_\{1\},\\dots,\\mathbf\{v\}\_\{d\_\{t\}\}and eigenvaluesλ1≥⋯≥λdt\\lambda\_\{1\}\\geq\\dots\\geq\\lambda\_\{d\_\{t\}\}\. The leading directions represent the primary axes of variances in the data; along the non\-dominant directions, the calibration set barely varies\. We call the span of the firstrrdirections the*well\-described subspace*, denoted by the columns ofVrV\_\{r\}\.

Recomputation score\.LetWr=W​Vr​Vr⊤W\_\{r\}=WV\_\{r\}V\_\{r\}^\{\\top\}be the map restricted to this subspace, with its bias𝐛r\\mathbf\{b\}\_\{r\}refit as in reduced\-rank regression\([Izenman, 1975](https://arxiv.org/html/2610.11358#bib.bib12)\), yielding prediction𝐲^i\(r\)\\hat\{\\mathbf\{y\}\}^\{\(r\)\}\_\{i\}for tokenii\. Rank disagreement is the squared gap between the full\-rank and reduced\-rank predictions, summed over every mapped layerℓ\\elland tensor typeσ\\sigma:

si=∑ℓ,σ∥𝐲^i−𝐲^\(r\)i∥2,𝐲^\(r\)=𝐱Wr\+𝐛r,s\_\{i\}=\\sum\_\{\\ell,\\,\\sigma\}\\bigl\\lVert\\hat\{\\mathbf\{y\}\}\_\{i\}\-\\hat\{\\mathbf\{y\}\}^\{\(r\)\}\_\{i\}\\bigr\\rVert^\{2\},\\qquad\\hat\{\\mathbf\{y\}\}^\{\(r\)\}=\\mathbf\{x\}W\_\{r\}\+\\mathbf\{b\}\_\{r\},\(2\)Because the two maps agree entirely within the dominant subspace, their gap captures what the token draws from the remaining, unconstrained directions\. We recompute the⌈ρ​T⌉\\lceil\\rho T\\rceilpositions with the largestsis\_\{i\}\.

A token scores near zero when its mapped entry falls safely within the familiar calibration space, regardless of its raw error\. Text that recurs in every calibration prompt, such as chat templates or task instructions, defines these leading directions and scores near zero\. Conversely, a token scores high when it projects into directions the calibration set rarely explored\. Content specific to one item, such as unique entities, quantities, and relations has no counterpart in the calibration data and therefore scores high\. Figure[13](https://arxiv.org/html/2610.11358#A3.F13)\(Appendix[C](https://arxiv.org/html/2610.11358#A3)\) demonstrates that this distinction, rather than the magnitude of the error, dictates which recomputations successfully recover downstream accuracy\.

#### Cost\.

The basisVrV\_\{r\}is computed once per map from the calibration set, using the source model and the fitted map but never the target\. At inference the map already produces𝐳t\\mathbf\{z\}\_\{t\}, and Eq\.[2](https://arxiv.org/html/2610.11358#S3.E2)can be evaluated as∥𝐳t∥2−∥𝐳t​Vr∥2\\lVert\\mathbf\{z\}\_\{t\}\\rVert^\{2\}\-\\lVert\\mathbf\{z\}\_\{t\}V\_\{r\}\\rVert^\{2\}\(derivation in the Appendix[A](https://arxiv.org/html/2610.11358#A1)\), onedt×rd\_\{t\}\\times rprojection per layer and side\. To prevent mapping and selection overheads from dominating the runtime, we reuse de\-rotated source layer blocks across target layers \(see §[B\.1](https://arxiv.org/html/2610.11358#A2.SS1)for implementation details and hardware profiling\)\.

## 4Experiments

We evaluate RaReCache across varying capacity gaps, model architectures, and downstream tasks\. Specifically, our experiments address three primary research questions:

1. Q1How much of the target model’s native accuracy can RaReCache recover, and how robust is this recovery as the capacity gap widens? \(§[4\.2](https://arxiv.org/html/2610.11358#S4.SS2)\)
2. Q2How does rank disagreement compare to standard token selection heuristics, and which tokens actually require target\-side recomputation? \(§[4\.3](https://arxiv.org/html/2610.11358#S4.SS3)\)
3. Q3Does RaReCache generalize to the reverse setting \(large\-to\-small transfer\), and can it enhance the capabilities of smaller, constrained models? \(§[4\.4](https://arxiv.org/html/2610.11358#S4.SS4)\)

### 4\.1Setup

Models\.We evaluate RaReCache across two model families\. In Qwen3, we consider Qwen3\-14B as the target and Qwen3\-0\.6B, 1\.7B and 4B as source models, which are 23×\\times, 8\.2×\\timesand 3\.5×\\timessmaller than the target\. In Llama3 we test two pairs, Llama\-3\.2\-3B\-Instruct→\\rightarrowLlama\-3\.1\-8B\-Instruct and Llama\-3\.1\-8B\-Instruct→\\rightarrowLlama\-3\.3\-70B\-Instruct \(2\.7×\\timesand 8\.8×\\timesmodel size gaps respectively\)\.

Map and recomputation\.We use the linear mapping explained in §[3\.1](https://arxiv.org/html/2610.11358#S3.SS1)withλ=0\.01\\lambda\{=\}0\.01\. Every target layer is mapped withK=8K\{=\}8source layers for Qwen3 and Llama 3B→\\to8B, andK=20K\{=\}20source layers for Llama 8B→\\to70B\. We calibrate the linear mapping matrices using 128k\-350k tokens from the training subset of each task, ensuring calibration is strictly disjoint from evaluation items \(per\-task calibration pools are detailed in[C\.1](https://arxiv.org/html/2610.11358#A3.SS1)in the Appendix\)\. For selective recomputation, we apply rank disagreement with a fixed rank ofr=128r\{=\}128across all model pairs\. We sweep the recompute budgetρ∈\{0,0\.2,0\.3,0\.4,0\.5\}\\rho\\in\\\{0,0\.2,0\.3,0\.4,0\.5\\\}, whereρ=0\\rho\{=\}0represents transfer without any target\-side recomputation, which reproduces the setting in[Heo et al\. \(2026\)](https://arxiv.org/html/2610.11358#bib.bib1)\.

Benchmarks\.We evaluate our approach on five benchmarks, each on its full test set:GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.11358#bib.bib15)\)\(n=1319n\{=\}1319, chain\-of\-thought, up to 1024 new tokens\),MMLU\-Redux\([Gema et al\., 2025](https://arxiv.org/html/2610.11358#bib.bib16)\)\(n=5700n\{=\}5700, 57 subjects, 0\-shot, direct answer\),ARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.11358#bib.bib17)\)\(n=1172n\{=\}1172\) andARC\-Easy\(n=2376n\{=\}2376, 0\-shot, chain\-of\-thought\), andLongBench\-E QA\([Bai et al\., 2024](https://arxiv.org/html/2610.11358#bib.bib18)\), 0\-4k bucket \(n=467n\{=\}467: HotpotQA, 2WikiMQA, MultiFieldQA\-en, Qasper, TriviaQA; mean prompt length 4\.3k tokens\)\. We use greedy decoding, and model generations are evaluated using exact match, with the exception of LongBench\-E, which is evaluated via token F1\.

### 4\.2Q1: How much of the target model’s accuracy can RaReCache recover?

We present accuracy against the recompute fractionρ\\rhofor three Qwen3 sources mapped into Qwen3\-14B in Figure[6](https://arxiv.org/html/2610.11358#S4.F6)\. On four of five benchmarks, mapping from a small source into a large target improves accuracy, suggesting that the mapped cache carries information the small model cannot use on its own: plain transfer \(ρ=0\\rho\{=\}0\) beats the 0\.6B alone by 12\.4\-24\.6 pp \(Figure[6](https://arxiv.org/html/2610.11358#S4.F6)\)\. However, the accuracies asρ=0\\rho=0are limited by the source model size, with 0\.6B model losing 6\.7–31\.2 pp to the 14B, retaining 60\.3% on MMLU\-Redux and 75\.3% on GSM8K\. For LongBench\-E, plain transfer scores 8\.2 F1 below the 0\.6B answering alone\.

Recomputing a minority of positions recovers most of the gap\.Recomputing 30% of positions recovers 94\.8% of the 14B’s accuracy on GSM8K \(71\.6% to 90\.2%\), 96\.9% on ARC\-Challenge and 98\.6% on ARC\-Easy; on both ARC sets, 20% already recovers more than 96% \(96\.2% and 97\.3%\)\. Atρ=0\.5\\rho\{=\}0\.5, transfer matches the 14B on GSM8K \(95\.3% vs\.95\.1%\), and reaches 92\.4% on MMLU\-Redux and 90\.7% on LongBench\-E\. At these points transfer beats the 0\.6B alone by 21\-39 pp on every benchmark\. With a23×23\\timessmaller source, recomputing 30% of positions recovers more than 94% of target accuracy on three of five benchmarks, and recomputing at most 50% recovers more than 90% on all five \(detailed numbers are in Table[5](https://arxiv.org/html/2610.11358#A3.T5)in the Appendix[C](https://arxiv.org/html/2610.11358#A3)\)\.

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/paper_accuracy_by_task_marked_row.png)Figure 6:Accuracy or F1 score against the recompute fractionρ\\rhofor three Qwen3 sources mapped into Qwen3\-14B\. Dashed lines are each source answering alone; the solid grey line is the 14B alone\.ρ=0\\rho\{=\}0reproduces results from[Heo et al\. \(2026\)](https://arxiv.org/html/2610.11358#bib.bib1)in our setting\.Recomputation substitutes for source capacity\.Without recomputation, source size dominates: plain transfer is ordered by source size on every task \(Figure[6](https://arxiv.org/html/2610.11358#S4.F6)\), with a spread across the three sources of 18\.9 pp on GSM8K, 23\.6 on MMLU\-Redux, 9\.7 on ARC\-Challenge, 5\.9 on ARC\-Easy and 23\.6 F1 on LongBench\-E\. Recomputation collapses the ordering\. On GSM8K the three sources lie within 0\.9 pp atρ=0\.4\\rho\{=\}0\.4and 0\.5 pp atρ=0\.5\\rho\{=\}0\.5, on ARC\-Challenge within 1\.6 pp and on ARC\-Easy within 0\.8 pp atρ=0\.4\\rho\{=\}0\.4\. On LongBench\-E the spread falls from 23\.6 to 1\.9 F1 atρ=0\.5\\rho\{=\}0\.5, and on MMLU\-Redux, the spread narrowsmore than𝟓×\\mathbf\{5\}\\times, from 23\.6 to 4\.3 pp\.

Paired comparisons show the substitution directly\. On GSM8K the 0\.6B atρ=0\.3\\rho\{=\}0\.3matches the 4B with no recomputation, as it does on ARC\-Easy and on ARC\-Challenge atρ=0\.4\\rho\{=\}0\.4\. On LongBench\-E the 0\.6B atρ=0\.4\\rho\{=\}0\.4beats the 4B with no recomputation by 4\.6 F1\. On MMLU\-Redux the 0\.6B atρ=0\.5\\rho\{=\}0\.5matches the 4B atρ=0\.2\\rho\{=\}0\.2while beating the 4B answering alone by 2\.5 pp\.

Table 1:GSM8K Evaluation on Llama 3:Accuracy \(%\) against the recompute fractionρ\\rho, retention relative to the target is in parentheses\. The 8B→\\to70B accuracies are reported on the first 300 test items\.Evaluation on Llama3\.We demonstrate results with Llama 3 in Table[1](https://arxiv.org/html/2610.11358#S4.T1)and observe that LLama shows similar behavior as Qwen3\. Mapping the 8B into the 70B, an8\.8×8\.8\\timesgap, plain transfer scores 81%, no better than the 8B answering alone, so the mapped cache on its own gives no reason to invoke the 70B\. Recomputing 40% of positions raises it to 92%, 96\.5% of the 70B, and∼\\sim10 pp above the 8B alone\. The 3B→\\to8B pair are close in model size gap, with the 8B only 3\.6 points above the 3B\. Mapping again fails to beat the source, whereas recomputation matches the 8B \(85\.7% vs\. 85\.8%\) atρ=0\.2\\rho\{=\}0\.2only\.

### 4\.3Q2: How does rank disagreement compare to standard token selection heuristics?

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/selector_bars_gsm8k_300_wrap.png)Figure 7:Selectors at a matched budget\(ρ=0\.3\\rho\{=\}0\.3\), for 300 samples on GSM8K, transfer mapping Qwen3\-0\.6B→\\to14B\. The shaded band marks random selection\.We compare selectors at a fixed 30% budget on 300 GSM8K items, holding the map and the mapped cache fixed so that only the choice of positions changes \(Figure[7](https://arxiv.org/html/2610.11358#S4.F7)\)\. Two of the selectors, the 14B’s attention, and the positions with the largest mapping error, are oracles that read the 14B’s own prefill and therefore cannot be deployed, but are used for the purpose of our analysis\.

Which positions matters more than how many\.Recomputing 30% of positions at random closes 42% of the gap to the 14B \(73\.3 to 82\.3\)\. Rank disagreement, at the same budget, closes 79% of it \(90\.3\), and beats every other selector, including both oracles\. We illustrate the selected tokens using rank disagreement in Figure[13](https://arxiv.org/html/2610.11358#A3.F13)\(Appendix[C](https://arxiv.org/html/2610.11358#A3)\)\.

Rank disagreement separates the two kinds of error that theℓ2\\ell\_\{2\}error magnitude conflates: interpolation error on tokens the calibration set covered, which can be large and costs nothing, and out\-of\-distribution token errors\. Template tokens lie inside the well\-described subspace whether or not their position was sampled, so they score near zero even where theirℓ2\\ell\_\{2\}error is large \(Figure[13](https://arxiv.org/html/2610.11358#A3.F13)\)\. Across the full GSM8K test set, 0\.23% of the positions it recomputes fall in the shared template prefix\. The question’s content words score high, and they receive under 2% of the 14B’s attention, which is why attention\-based selectors are ineffective\.

### 4\.4Does RaReCache generalize to the large\-to\-small transfer?

A cascade may also hand a large model’s context down to a small one\. We map the 14B’s cache into each of the three smaller models on MMLU\-Redux \(n=5700n\{=\}5700, same calibration recipe as the forward direction\), and compare against each small model prefilling for itself \(Table[2](https://arxiv.org/html/2610.11358#S4.T2)\)\.

Table 2:Large\-to\-small transfer, Qwen3\-14B→\\toeach smaller model on MMLU\-Redux\.*Alone*is the small model prefilling its own cache\. Lift is transfer minus alone\.The small model retains its own accuracy, and gains where it has the capacity to use a richer cache\.At8\.2×8\.2\\timesand3\.5×3\.5\\times, decoding from the 14B’s mapped cache beats the small model decoding from its own prefill, by 14\.4 pp for the 1\.7B and 3\.1 pp for the 4B\. The 1\.7B result is the largest lift over standalone accuracy in either direction: the mapped cache carries more than the 1\.7B could have encoded itself, and the 1\.7B’s decoder can exploit it\. At23×23\\timesthe direction fails, losing 7\.8 pp, mirroring the forward\-direction cliff from the other side\.

Recomputation does not help in reverse\.Atρ=0\.2\\rho\{=\}0\.2, accuracy moves by at most 0\.7 pp at any ratio, inside noise at this sample size, where the same budget buys 6–9 pp forward\. Rank disagreement finds positions where the*source’s*cache falls outside the calibration fit; in reverse the source is the 14B, whose cache the map reconstructs well throughout \(RK2R^\{2\}\_\{K\}0\.90–0\.91 for all three targets\), so there is little of that instability left to find\. Whatever bounds the 0\.6B’s reverse accuracy is not visible to this score\.

### 4\.5Comparison with existing reuse strategies

Figure 8:Comparison against existing selective recomputation strategies for Qwen3\-0\.6B→\\toQwen3\-14B for different recompute ratios for 300 GSM8K samples\.Because existing reuse systems are not designed for cross\-architecture size gaps, we adapt their core selection rules for our pipeline and compare them at a matched compute budgetρ\\rho\(the fraction of a full target prefill\)\. DroidSpeak\([Liu et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib4)\)selectively recomputes layers; since hidden states are incompatible across different model sizes, we adapt this as a layer\-prefix recomputation \(evaluating the firstLLlayers\)\. CacheBlend\([Yao et al\., 2025](https://arxiv.org/html/2610.11358#bib.bib7)\)selects tokens based onℓ2\\ell\_\{2\}KV deviation; we evaluate this in its strongest possible form as an oracle that ranks tokens by their trueℓ2\\ell\_\{2\}mapping error against the target’s actual cache\.

As shown in Figure[8](https://arxiv.org/html/2610.11358#S4.F8), rank disagreement significantly outperforms both adapted baselines\. At a 30% budget \(ρ=0\.3\\rho=0\.3\) on GSM8K, our method achieves 90\.3% accuracy, outperforming theℓ2\\ell\_\{2\}error oracle \(80\.7%\) and layer\-wise recomputation \(79\.0%\) by roughly 10 and 11 points respectively\. Layer prefixing scales poorly because mapping errors are concentrated in a minority of specific tokens across all depths, rather than isolated to early layers\. Meanwhile, the true mapping error oracle performs no better than random selection \(82\.3%\), confirming that rawℓ2\\ell\_\{2\}magnitude fails to identify the critical tokens that actually degrade downstream reasoning\.

### 4\.6System Analysis: End\-to\-End Inference Performance

While the preceding sections establish the recomputation budgetρ\\rhofrom an accuracy perspective, we now evaluate the resulting inference\-time speedups and runtime overheads\. We benchmark the RaReCache pipeline for Qwen3\-0\.6B→\\to14B on a NVIDIA A100 GPU \(80GB\) with PyTorch 2\.6 eager mode on GSM8K \(mean prompt lengthT=149T\{=\}149\)\. We measure the latency across three sequential stages: projecting the source KV cache via the linear map \(along with tensor preprocessing and feature gathering\), scoring positions via rank disagreement, and executing the target model’s forward pass over the selected⌈ρ​T⌉\\lceil\\rho T\\rceilpositions\. We compare against the target model prefilling the entire prompt in all experiments as a baseline\. Our latency evaluation assumes the source model has already prefilled the prompt, reflecting common deployment scenarios such as model cascades or multi\-turn agents where the smaller model runs first \(same setup as[Heo et al\. \(2026\)](https://arxiv.org/html/2610.11358#bib.bib1)\) and its KV cache is already available\. Unless otherwise stated, all reported latencies reflect GPU\-busy time: the sum of individual GPU kernel durations\.

Figure 9:\(a\) Prefill speedup across recompute fractionsρ\\rho\(from §[4\.2](https://arxiv.org/html/2610.11358#S4.SS2)shaded\); \(b, c\) Median and p99 TTFT vs request rate under Poisson arrivals \(T=149T\{=\}149,ρ=0\.3\\rho\{=\}0\.3\)\. The shaded region shows the divergence of the target prefill queue while RaReCache operates at lower device utilization\.Impact of Recomputation Budget on Runtime\.In Figure[9](https://arxiv.org/html/2610.11358#S4.F9)a, we compare prefill speedup against recomputation fractionρ\\rho\. Overall latency gain is determined byρ\\rho, meaning lower recomputation results in greater speedup\. But this latency savings do not scale linearly withρ\\rho\. For instance, recomputing atρ=0\.3\\rho\{=\}0\.3requires38%−63%38\\%\-63\\%of the full prefill latency, not30%30\\%\. Two architectural bottlenecks not captured by theoretical FLOP counts explain this gap:\(a\)Loading the target model’s 27\.5GB of weights from high\-bandwidth memory is required, whether recomputing a single position or the entire sequence, which dominates execution time at shorter prompt lengths, and\(b\)Every recomputed query token must still attend allTTprompt keys, not just the recomputed subset\. For a single request ofT=2048T\{=\}2048, we achieve speedups of1\.38×\\mathbf\{1\.38\\times\},1\.95×\\mathbf\{1\.95\\times\}and2\.31×\\mathbf\{2\.31\\times\}atρ=0\.5,0\.3,0\.2\\rho=0\.5,0\.3,0\.2respectively\.ρ∈\[0\.2,0\.3\]\\rho\\in\[0\.2,0\.3\]is optimal, capturing most prefill latency savings while maintaining near\-native target accuracy \(ρ=0\.3\\rho=0\.3recovers 95\-99% of target model accuracy, as observed in Figure[6](https://arxiv.org/html/2610.11358#S4.F6)\)\. These latency savings widen under request batching, because native target prefill saturates compute at a batch of 4 \(6,2566,256tokens/s\) while our smaller compute kernels achieve higher hardware occupancy, throughput nearly doubles to12,37812,378tokens/s, pushing speedups up to3\.04×\\mathbf\{3\.04\\times\}and2\.36×\\mathbf\{2\.36\\times\}at device saturation forρ=0\.2\\rho\{=\}0\.2and0\.30\.3respectively with T=149 \(see §[B\.2](https://arxiv.org/html/2610.11358#A2.SS2)for complete batch sweeps\)\.

Queuing Dynamics and TTFT Under Load\.Figure[9](https://arxiv.org/html/2610.11358#S4.F9)\(b,c\) demonstrates an interactive serving scenario, where user\-perceived TTFT is the sum of queuing delay and prefill execution latency\. Accelerating prefill reduces both terms simultaneously: requests complete faster, maintaining lower compute utilization and preventing queue accumulation\. Evaluating both systems on a live server \(§[B\.3](https://arxiv.org/html/2610.11358#A2.SS3)details benchmarking setup and scheduler configuration\) under identical Poisson arrival traces shows that RaReCache significantly flattens the TTFT curve under load\. In this specific experiment, we measure end\-to\-end wall\-clock time to account for live queuing delays and framework dispatch overhead\. The requst rate \(λ\\lambdareq//s\) increased eightfold, causing the median TTFT of RaReCache to rise from9595ms to173173ms\. Whereas, the baseline target prefill surged from5151ms to870870ms\. Atλ=42\\lambda\{\}=42req//s, the saturation threshold \(μ\\mu\) of the target, RaReCache reduces median TTFT by5\.0×\\mathbf\{5\.0\\times\}and p99 latency by6\.4×\\mathbf\{6\.4\\times\}\.

Tail latency benefits appear earlier, crossing the baseline happens atλ≈20\\lambda\\approx 20requests/s, compared toλ≈30\\lambda\\approx 30req/s for the median\. The target forms larger batches under contention, which inflates head\-of\-line blocking for subsequent arrivals\. The baseline queue diverges between42\.742\.7and77\.077\.0req//s, while RaReCache operates comfortably at lower device utilization\. To serve this range of request rates, the baseline needs at least two GPUs provisioned, whereas we only need one\. Under light traffic \(λ<30\\lambda<30requests/s\), requests process mostly at batch size 1\. There the target prefill is faster because RaReCache incurs eager\-mode framework dispatch overhead across its smaller kernels\.

## 5Conclusion

In this work, we introduced RaReCache, a framework that bridges the severe accuracy degradation typically seen when transferring KV caches across models with massive capacity gaps\. By demonstrating that cross\-model transfer failures are heavily concentrated in a minority of information\-dense tokens, we developed a novel metric based on rank disagreement that isolates out\-of\-distribution representations without requiring target\-side prefill\. RaReCache allows a massive target model to inherit a context prefilled by a much smaller source model, recovering near\-native reasoning accuracy by selectively recomputing as little as 30% of the prompt\. Ultimately, this approach decouples transfer quality from source\-model capacity and delivers significant prefill speedups, unlocking highly efficient agentic LLM serving paradigms\. We currently report efficiency using standard attention kernels and plan to implement custom sparse causal kernels in the future to attain peak theoretical gains\.

### AI use statement

In compliance with ICLR guidelines, we disclose the use of generative AI \(large language models\) across several phases of this work\. For tasks requiring disclosure, AI tools were used to interpret measurement logs, propose and critique hypotheses regarding the effects reported in §[2](https://arxiv.org/html/2610.11358#S2)and §[4](https://arxiv.org/html/2610.11358#S4), and design experiments by suggesting high\-value configurations for the sweeps reported herein\. For tasks where disclosure is recommended, we utilized these tools to assist with literature search, drafting and editing sections of the manuscript, and writing scripts to generate figures\. Importantly, the core implementation of our method, including the rank\-disagreement scoring, and selective recomputation, was written entirely by the authors without AI assistance\. We did not use AI to generate synthetic datasets or develop theoretical models; the derivation in Appendix A was written and verified by the authors\. We have reviewed all AI\-assisted content and take full responsibility for the final text, claims, and artifacts\.

### Reproducibility statement

To ensure reproducibility, we have detailed all experimental setups, model configurations, and evaluation metrics in the manuscript\. Evaluations use open\-weights models from Qwen3 and Llama 3 families and standard, publicly available benchmarks like GSM8K, MMLU\-Redux, ARC\-Challenge, ARC\-Easy, and LongBench\-E QA\. The system\-level inference and queuing benchmarks transparently specify the hardware and software environments\. The complete source code, including the linear mapping implementation, rank disagreement scoring, system optimization and evaluation scripts will be made publicly available upon de\-anonymization and publication\.

## References

- Baiet al\.\(2024\)Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. LiLongBench: a bilingual, multitask benchmark for long context understanding\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§4\.1](https://arxiv.org/html/2610.11358#S4.SS1.p3.1.6)\.
- Chenet al\.\(2023\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.arXiv preprint arXiv:2305\.05176\.Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2610.11358#S4.SS1.p3.1.4)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2610.11358#S4.SS1.p3.1.2)\.
- Deryet al\.\(2026\)L\. M\. Dery, Z\. Yahav, H\. Prior, Q\. Feng, J\. Shen, and A\. SzlamLatent space communication via K\-V cache alignment\.arXiv preprint arXiv:2601\.06123\.Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p2.1)\.
- Fuet al\.\(2026\)T\. Fu, Z\. Min, H\. Zhang, J\. Yan, G\. Dai, W\. Ouyang, and Y\. WangCache\-to\-cache: direct semantic communication between large language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2510\.03215Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p2.1)\.
- Gemaet al\.\(2025\)A\. P\. Gema, J\. O\. J\. Leang, G\. Hong, A\. Devoto, A\. C\. M\. Mancino, R\. Saxena, X\. He, Y\. Zhao, X\. Du, M\. R\. G\. Madani, C\. Barale, R\. McHardy, J\. Harris, J\. Kaddour, E\. van Krieken, and P\. MinerviniAre we done with MMLU?\.InConference of the Nations of the Americas Chapter of the Association for Computational Linguistics \(NAACL\),Note:arXiv:2406\.04127Cited by:[§4\.1](https://arxiv.org/html/2610.11358#S4.SS1.p3.1.3)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p6.1)\.
- Heoet al\.\(2026\)T\. Heo, R\. Shafipour, R\. Zhao, M\. Golub, M\. M\. Kamani, R\. Borkar, M\. T\. Chandran, P\. Zardoshti, and B\. Darvish RouhaniCross\-model KV cache transfer in LLM families: a closed\-form linear mapping for prefill reuse\.arXiv preprint arXiv:2608\.03893\.Cited by:[Appendix D](https://arxiv.org/html/2610.11358#A4.p1.1),[§1](https://arxiv.org/html/2610.11358#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.11358#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.11358#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.11358#S3.SS1.p2.1),[Figure 6](https://arxiv.org/html/2610.11358#S4.F6),[§4\.1](https://arxiv.org/html/2610.11358#S4.SS1.p2.1),[§4\.6](https://arxiv.org/html/2610.11358#S4.SS6.p1.1)\.
- Izenman \(1975\)A\. J\. IzenmanReduced\-rank regression for the multivariate linear model\.Journal of Multivariate Analysis5\(2\),pp\. 248–264\.Cited by:[Appendix A](https://arxiv.org/html/2610.11358#A1.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.11358#S3.SS2.p3.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with PagedAttention\.InACM Symposium on Operating Systems Principles \(SOSP\),Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p2.1)\.
- Liet al\.\(2024\)Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. ChenSnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.2](https://arxiv.org/html/2610.11358#S2.SS2.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, Y\. Huang, J\. Yao, S\. Feng, Z\. Gu, K\. Du, H\. Li, Y\. Cheng, J\. Jiang, S\. Lu, M\. Musuvathi, and E\. ChoukseDroidSpeak: KV cache sharing across fine\-tuned model variants\.InUSENIX Symposium on Networked Systems Design and Implementation \(NSDI\),Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p2.1),[§1](https://arxiv.org/html/2610.11358#S1.p3.1),[§4\.5](https://arxiv.org/html/2610.11358#S4.SS5.p1.1)\.
- Onget al\.\(2024\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to Route LLMs with Preference Data\.External Links:2406\.18665,[Link](https://arxiv.org/abs/2406.18665)Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§2\.1](https://arxiv.org/html/2610.11358#S2.SS1.p1.1)\.
- Yaoet al\.\(2025\)J\. Yao, H\. Li, Y\. Liu, S\. Ray, Y\. Cheng, Q\. Zhang, K\. Du, S\. Lu, and J\. JiangCacheBlend: fast large language model serving for RAG with cached knowledge fusion\.InEuropean Conference on Computer Systems \(EuroSys\),Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p3.1),[§4\.5](https://arxiv.org/html/2610.11358#S4.SS5.p1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. ChenH2\{\}\_\{2\}O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.2](https://arxiv.org/html/2610.11358#S2.SS2.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. ShengSGLang: efficient execution of structured language model programs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.11358#S1.p2.1)\.

## Appendix ARank disagreement derivation

This appendix derives the score∥𝐳t∥2−∥𝐳t​Vr∥2\\lVert\\mathbf\{z\}\_\{t\}\\rVert^\{2\}\-\\lVert\\mathbf\{z\}\_\{t\}V\_\{r\}\\rVert^\{2\}from Eq\.[2](https://arxiv.org/html/2610.11358#S3.E2)and states the behaviour of the score at the two extremes of the probe rank\. We work with one map, that is one target layer and one side; the score sums these independently\.

#### Setup\.

LetW∈ℝds×dtW\\in\\mathbb\{R\}^\{d\_\{s\}\\times d\_\{t\}\}and𝐛\\mathbf\{b\}be the fitted map of Eq\.[1](https://arxiv.org/html/2610.11358#S3.E1), and let𝐱¯\\bar\{\\mathbf\{x\}\}be the calibration mean of the source features\. The predictions on the calibration set have mean

𝝁^=𝐱¯​W\+𝐛,\\hat\{\\bm\{\\mu\}\}=\\bar\{\\mathbf\{x\}\}W\+\\mathbf\{b\},\(3\)since the map is affine, and covarianceΣy^=1N​∑n\(𝐲^n−𝝁^\)⊤​\(𝐲^n−𝝁^\)=V​Λ​V⊤\\Sigma\_\{\\hat\{y\}\}=\\frac\{1\}\{N\}\\sum\_\{n\}\(\\hat\{\\mathbf\{y\}\}\_\{n\}\-\\hat\{\\bm\{\\mu\}\}\)^\{\\top\}\(\\hat\{\\mathbf\{y\}\}\_\{n\}\-\\hat\{\\bm\{\\mu\}\}\)=V\\Lambda V^\{\\top\}, withVVorthonormal andΛ\\Lambdadiagonal with entries in decreasing order\. LetVrV\_\{r\}be the firstrrcolumns ofVVand

Pr=Vr​Vr⊤P\_\{r\}=V\_\{r\}V\_\{r\}^\{\\top\}\(4\)the orthogonal projection onto the well\-described subspace\. Being an orthogonal projection,PrP\_\{r\}satisfiesPr⊤=PrP\_\{r\}^\{\\top\}=P\_\{r\}andPr2=PrP\_\{r\}^\{2\}=P\_\{r\}, andI−PrI\-P\_\{r\}projects onto the orthogonal complement\.

#### The rank\-rrmap\.

Reduced\-rank regression\([Izenman, 1975](https://arxiv.org/html/2610.11358#bib.bib12)\)restricts the map to the leading directions of its own fitted values, and refits the bias so the restricted map still sends the average input to the average output:

Wr=W​Pr,𝐛r=𝝁^−𝐱¯​Wr,𝐲^t\(r\)=𝐱t​Wr\+𝐛r\.W\_\{r\}=WP\_\{r\},\\qquad\\mathbf\{b\}\_\{r\}=\\hat\{\\bm\{\\mu\}\}\-\\bar\{\\mathbf\{x\}\}W\_\{r\},\\qquad\\hat\{\\mathbf\{y\}\}^\{\(r\)\}\_\{t\}=\\mathbf\{x\}\_\{t\}W\_\{r\}\+\\mathbf\{b\}\_\{r\}\.\(5\)This is not the truncated SVD ofWW\. The truncated SVD keeps the directions in whichWWitself is largest; reduced\-rank regression keeps those in which the map’s outputs on real calibration data vary most\. The two coincide only when the source features are isotropic\. Since thekkconcatenated source layers are strongly correlated, they are far apart here, and truncatingWWdirectly would spend rank on directions the data never excites\.

#### Derivation\.

Write𝐳t=\(𝐱t−𝐱¯\)​W\\mathbf\{z\}\_\{t\}=\(\\mathbf\{x\}\_\{t\}\-\\bar\{\\mathbf\{x\}\}\)Wfor the centered prediction\. Substituting Eq\.[5](https://arxiv.org/html/2610.11358#A1.E5)into the difference of the two predictions and then Eq\.[3](https://arxiv.org/html/2610.11358#A1.E3)for𝝁^\\hat\{\\bm\{\\mu\}\},

𝐲^t−𝐲^t\(r\)\\displaystyle\\hat\{\\mathbf\{y\}\}\_\{t\}\-\\hat\{\\mathbf\{y\}\}^\{\(r\)\}\_\{t\}=\(𝐱t​W\+𝐛\)−\(𝐱t​W​Pr\+𝝁^−𝐱¯​W​Pr\)\\displaystyle=\(\\mathbf\{x\}\_\{t\}W\+\\mathbf\{b\}\)\-\\bigl\(\\mathbf\{x\}\_\{t\}WP\_\{r\}\+\\hat\{\\bm\{\\mu\}\}\-\\bar\{\\mathbf\{x\}\}WP\_\{r\}\\bigr\)\(6\)=𝐱t​W\+𝐛−𝐱t​W​Pr−𝐱¯​W−𝐛\+𝐱¯​W​Pr\\displaystyle=\\mathbf\{x\}\_\{t\}W\+\\mathbf\{b\}\-\\mathbf\{x\}\_\{t\}WP\_\{r\}\-\\bar\{\\mathbf\{x\}\}W\-\\mathbf\{b\}\+\\bar\{\\mathbf\{x\}\}WP\_\{r\}\(7\)=\(𝐱t−𝐱¯\)​W​\(I−Pr\)=𝐳t​\(I−Pr\)\.\\displaystyle=\(\\mathbf\{x\}\_\{t\}\-\\bar\{\\mathbf\{x\}\}\)W\(I\-P\_\{r\}\)\\;=\\;\\mathbf\{z\}\_\{t\}\(I\-P\_\{r\}\)\.\(8\)The bias terms cancel, which is why the score does not depend on either model’s mean cache\. BecauseI−PrI\-P\_\{r\}is an orthogonal projection and the𝐯i\\mathbf\{v\}\_\{i\}are orthonormal,

∥𝐳t\(I−Pr\)∥2=∑i\>r⟨𝐳t,𝐯i⟩2=∥𝐳t∥2−∥𝐳tVr∥2,\\bigl\\lVert\\mathbf\{z\}\_\{t\}\(I\-P\_\{r\}\)\\bigr\\rVert^\{2\}=\\sum\_\{i\>r\}\\langle\\mathbf\{z\}\_\{t\},\\mathbf\{v\}\_\{i\}\\rangle^\{2\}=\\lVert\\mathbf\{z\}\_\{t\}\\rVert^\{2\}\-\\lVert\\mathbf\{z\}\_\{t\}V\_\{r\}\\rVert^\{2\},\(9\)which is Eq\.[2](https://arxiv.org/html/2610.11358#S3.E2)together with the form used to compute it\. Summing Eq\.[8](https://arxiv.org/html/2610.11358#A1.E8)over layers and sides gives the score of Eq\.[2](https://arxiv.org/html/2610.11358#S3.E2)\. The right\-hand form of Eq\.[9](https://arxiv.org/html/2610.11358#A1.E9)is the cheap one: the map already computes𝐳t\\mathbf\{z\}\_\{t\}, so the score costs onedt×rd\_\{t\}\\times rprojection per layer and side\.

#### Limits inrr\.

Atr=0r\{=\}0,Pr=0P\_\{r\}=0andst=∑∥𝐳t∥2s\_\{t\}=\\sum\\lVert\\mathbf\{z\}\_\{t\}\\rVert^\{2\}: the score degenerates to the size of the centered prediction, which measures how unusual a token is without reference to how well the map handles it\. Atr=dtr\{=\}d\_\{t\},Pr=IP\_\{r\}=Iandst=0s\_\{t\}=0for every token\. Between the two,sts\_\{t\}measures only the energy that falls outside the directions the calibration set covers\.

## Appendix BSystem Design

### B\.1Mapping Implementation and Runtime Efficiency

For a Qwen3\-0\.6B→\\to14B KV transfer experiment on GSM8K withρ=0\.3\\rho\{=\}0\.3, the linear map contains6\.7×1086\.7\\times 10^\{8\}parameters; roughly5%5\\%of the target model’s1\.3×10101\.3\\times 10^\{10\}parameters\. This suggests that by arithmetic count alone \(excluding partial target prefill\), cache translation should cost only a fraction of a full prefill\. But in practice, executing the map in FP32 severely bottlenecks our mechanism because an NVIDIA A100 routes FP32 matmul through vector units at19\.519\.5TFLOP/s rather than utilizing its Tensor Cores, which deliver312312TFLOP/s in BF16\. Consequently, an unoptimized FP32 map dominates latency, causing cache transfer to run slower than native target prefill \(0\.630\.63\-0\.85×0\.85\\times\)\. We eliminate this overhead with two systems optimizations that preserve the algorithm:

- •Executing the map in BF16 to leverage Tensor Cores, raising relative performance to0\.780\.78\-1\.38×1\.38\\times\.
- •Sharing precomputed source blocks\. The reference implementation redundantly reconstructs the same5050de\-rotated source blocks640640times across target layers and KV heads; caching these blocks once per prompt pushes overall speedup to1\.16\\mathbf\{1\.16\}\-1\.95×\\mathbf\{1\.95\\times\}\(Figure[10](https://arxiv.org/html/2610.11358#A2.F10), Table[3](https://arxiv.org/html/2610.11358#A2.T3)\)\.

AtT=2048T\{=\}2048, these optimizations reduce mapping latency from195195ms to2727ms \(7\.2×7\.2\\times\) and token selection from7070ms to1717ms \(4\.2×4\.2\\times\), leaving target\-side recomputation as the main source of remaining latency\. The theoretical speedup ceiling of RaReCache can be analytically calculated and explained in detail in §[B\.4](https://arxiv.org/html/2610.11358#A2.SS4)\.

Figure 10:What RaReCache costs, relative to the target prefill\.Phase\-by\-phase runtime across implementation variants at increasing context lengths \(T=149T\{=\}149is the mean GSM8K prompt length\), with numbers above the bars showing overall speedups\. While naive FP32 mapping consumes nearly as much time as a full prefill, BF16 execution combined with source block sharing reduces mapping and selection overhead to a small fraction of total latency\.Table 3:Prefill speedup over the target baseline, at recomputation budgetρ=0\.3\\rho\{=\}0\.3\.Rows are cumulative, each adding an optimization to the row above\. The first three rows are single requests \(batch size 1\); the last is the largest batch the device holds, which is384,128,64,32,16384,128,64,32,16and88requests from left to right \(≈\\approx65k tokens in flight\)\.
### B\.2Batching and Peak Throughput

Figures[11](https://arxiv.org/html/2610.11358#A2.F11)a and b illustrate the batch\-scaling behavior from Table[3](https://arxiv.org/html/2610.11358#A2.T3)\. Native target prefill throughput plateaus beyond an order of magnitude of batch size \(6,2566,256tokens/s\)\. This is because its large GEMMs already saturate GPU compute, leaving no room for amortization\. In contrast, RaReCache pipeline consists of smaller kernels that underutilize GPU compute at lower batch sizes\. Increasing the batch size improves Streaming Multiprocessor \(SM\) occupancy, nearly doubling prefill throughput to12,37812,378tokens/s\.

Figure 11:\(a\) Prefill throughput scaling atT=149T\{=\}149, where target only prefill saturates near batch size44while RaReCache continues to scale\. \(b\) Prefill speedup as a function of batch size across three prompt lengths; the dashed horizontal line indicates the theoretical speedup ceiling\.
### B\.3Live Serving Evaluation and Queuing Methodology

#### Scheduler\.

Both configurations are evaluated using a first\-come\-first\-served \(FCFS\) dynamic batching scheduler deployed on a live server\. Requests arrive via a homogeneous Poisson process at rateλ\\lambda\. When the GPU becomes idle, the scheduler dispatches up to3232queued requests \(oldest first\) as a single execution batch, leaving any remaining requests queued for the subsequent iteration\. Each evaluation sweeps500500requests, discarding the initial5050to capture steady\-state queuing behavior\. TTFT is measured in wall\-clock time rather than GPU\-busy time to capture both queuing delays and framework\-level eager dispatch overheads\.

#### Saturation Throughput and Batch Dynamics\.

We measure each system’s empirical saturation thresholdμ\\muby executing three consecutive maximum\-capacity batches \(3232requests each\) back\-to\-back under a permanently saturated queue:

μ=3×32wall\-clock execution time\.\\mu=\\frac\{3\\times 32\}\{\\text\{wall\-clock execution time\}\}\.\(10\)This yields saturation throughputs ofμtarget=42\.7\\mu\_\{\\text\{target\}\}=42\.7andμRaReCache=77\.0\\mu\_\{\\text\{RaReCache \}\}=77\.0requests/s \(1\.80×1\.80\\timescapacity advantage, consistent with §[B\.2](https://arxiv.org/html/2610.11358#A2.SS2)\)\. We report offered load normalized to target capacity,utarget=λ/μtargetu\_\{\\text\{target\}\}=\\lambda/\\mu\_\{\\text\{target\}\}\. Because the baseline queue diverges forλ≥μtarget\\lambda\\geq\\mu\_\{\\text\{target\}\}, the sweep caps atλ=42\\lambda=42requests/s \(utarget=0\.98u\_\{\\text\{target\}\}=0\.98\)\.

Under dynamic queuing, the mean batch sizeb∗b^\{\*\}satisfies the equilibrium conditionb∗≈λ⋅S⁡\(b∗\)b^\{\*\}\\approx\\lambda\\cdot S\(b^\{\*\}\), whereS⁡\(b\)S\(b\)is the batch service latency\. BecauseStarget​\(b\)≫SRaReCache​\(b\)S\_\{\\text\{target\}\}\(b\)\\gg S\_\{\\text\{RaReCache \}\}\(b\), the baseline target experiences severe queue buildup, causing its mean batch size to extend to24\.124\.1requests atλ=42\\lambda=42requests/s \(approaching the3232\-request cap\)\. In contrast, the shorter service time of RaReCache clears requests rapidly, keeping its mean batch size at only5\.95\.9and drastically preventing head\-of\-line blocking \(Table[4](https://arxiv.org/html/2610.11358#A2.T4)\)\.

Table 4:TTFT under Poisson load\(T=149T\{=\}149,ρ=0\.3\\rho\{=\}0\.3, max batch size3232, wall clock\)\. Offered load is normalized to target saturation capacity \(utarget=λ/μtargetu\_\{\\text\{target\}\}=\\lambda/\\mu\_\{\\text\{target\}\}\)\. Speedup ratios represent target÷\\divRaReCache \(\>1×\>1\\timesdenotes RaReCache is faster\)\.

### B\.4Analytical Speedup Ceiling and Attention Scaling

Once mapper overhead is eliminated, target\-side recomputation accounts for7373–79%79\\%of total RaReCache latency \(§[B\.1](https://arxiv.org/html/2610.11358#A2.SS1)\)\. In our implementation, recomputed query tokens attend over the entire contextTTvia an explicit attention mask, evaluating allρ​T2\\rho T^\{2\}query\-key pairs rather than only the causally valid subset\. Comparing total FLOPs between native target prefill and RaReCache pipeline yields the speedup ceiling:

Target FLOPsRaReCache FLOPs=2​P𝒯​T\+12​α​T2\(2​Pmap\+2​Pprobe\+2​ρ​P𝒯\)​T\+ρ​α​T2,α=4​L𝒯​H𝒯​dh,\\frac\{\\text\{Target FLOPs\}\}\{\\text\{RaReCache FLOPs\}\}=\\frac\{2P\_\{\\mathcal\{T\}\}T\+\\frac\{1\}\{2\}\\alpha T^\{2\}\}\{\\left\(2P\_\{\\text\{map\}\}\+2P\_\{\\text\{probe\}\}\+2\\rho P\_\{\\mathcal\{T\}\}\\right\)T\+\\rho\\,\\alpha T^\{2\}\},\\qquad\\alpha=4L\_\{\\mathcal\{T\}\}H\_\{\\mathcal\{T\}\}d\_\{h\},\(11\)whereP𝒯P\_\{\\mathcal\{T\}\},PmapP\_\{\\text\{map\}\}, andPprobeP\_\{\\text\{probe\}\}represent the parameter counts of the target model, linear map, and rank\-disagreement selector, respectively, andL𝒯L\_\{\\mathcal\{T\}\},H𝒯H\_\{\\mathcal\{T\}\}, anddhd\_\{h\}denote target layer depth, head count, and head dimension\.

As context lengthTTincreases, quadratic attention FLOPs dominate linear feed\-forward terms, causing theoretical speedup to asymptotically approach1/\(2​ρ\)1/\(2\\rho\)\(1\.67×1\.67\\timesatρ=0\.3\\rho=0\.3\)\. Consequently, the speedup ceiling declines with prompt length, dropping from2\.74×2\.74\\timesatT=2,048T\{=\}2\{,\}048to2\.28×2\.28\\timesatT=32,768T\{=\}32\{,\}768\.

A dedicated attention kernel that skips causally masked query\-key pairs would reduce the recomputation attention cost to12​ρ​α​T2\\frac\{1\}\{2\}\\rho\\alpha T^\{2\}, raising the asymptotic ceiling to1/ρ1/\\rho\(3\.33×3\.33\\timesatρ=0\.3\\rho=0\.3\)\. Under this optimized execution, speedups would scale positively with context length \(reaching2\.95×2\.95\\timesatT=32,768T\{=\}32\{,\}768\), making long\-sequence serving the most favorable regime\. We report benchmarks using standard, unspecialized attention kernels and leave custom sparse causal implementations to future work\.

## Appendix CExperimental Results

### C\.1Mapping\.

We use the map of §[3\.1](https://arxiv.org/html/2610.11358#S3.SS1)withλ=0\.01\\lambda\{=\}0\.01throughout, choosing theKKsource layers for each target layer byR2R^\{2\}\. Every target layer is mapped for both keys and values:

- •Qwen3\(all three sources\):k=8k\{=\}8, giving8192→10248192\\to 1024dimensions per token over 40 target layers, for 671M map parameters\.
- •Llama 3B→\\to8B:k=8k\{=\}8,8192→10248192\\to 1024over 32 target layers, 537M parameters\.
- •Llama 8B→\\to70B:k=20k\{=\}20,20480→102420480\\to 1024over 80 target layers, 3\.36B parameters\.

Calibration data is from the train subset of the same task, formatted similarly as the evaluation prompts, and disjoint from every evaluation item:

- •GSM8K uses its train split \(128k tokens, stride 4; 320k, stride 2, for Llama 8B→\\to70B\)\.
- •MMLU\-Redux uses MMLU validation \(128k, stride 1\)\. MMLU\-Redux is a subset of items from the MMLU test split, so the two are disjoint\.
- •ARC uses the pooled ARC\-Challenge and ARC\-Easy train splits \(150k, stride 3\)\.
- •LongBench\-E uses a pool drawn from its 4\-8k and 8k\+ buckets and from LongBench’s standard split \(350k tokens at stride 32\)\. We remove every item that shares a question or a passage of 30 or more words with the 0\-4k evaluation bucket\.

Token positions are subsampled at the stated stride to ensure that the calibration budget covers a diverse range of contexts across the entire corpus\.

#### RaReCache Results for Small\-to\-Large Prefill

The detailed accuracy and retention numbers for the Figure[6](https://arxiv.org/html/2610.11358#S4.F6)is presented in the Table[5](https://arxiv.org/html/2610.11358#A3.T5)\.

Table 5:Qwen3\-0\.6B→\\toQwen3\-14B across five benchmarks\.Accuracy \(%, F1 for LongBench\-E\) as a function of the fractionρ\\rhoof prompt positions recomputed in the target\. Retention relative to the 14B is in parentheses\.ρ=0\\rho\{=\}0is plain transfer\.†Not significantly different from the 14B \(p=0\.86p\{=\}0\.86\)\.‡Below the source model alone\. Each recompute budget improves overρ=0\\rho\{=\}0\. A dash means the arm was not run\.
#### Layer agreement for the recompute set\.

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/layer_agreement_per_layer.png)Figure 12:Layerwise agreement scores with the shared recompute set determined byℓ2\\ell\_\{2\}mapping error and rank disagreement\.For each target layer of Qwen3\-14B, the overlap between that layer’s own top\-ρ\\rhopositions and the single set the method applies \(the top\-ρ\\rhoof the score summed over layers\), averaged over 300 GSM8K prompts, Qwen3\-0\.6B→\\to14B,ρ=0\.3\\rho\{=\}0\.3is demonstrated in Figure[12](https://arxiv.org/html/2610.11358#A3.F12)\. Under rank disagreement every layer overlaps the shared set by 0\.86\-0\.97, and by more than theℓ2\\ell\_\{2\}mapping\-error oracle does at every layer but the last\. The gap is widest in layers 0\-25 \(4\.8 points on average\), where the oracle’s per\-layer choices vary most, and narrows to 1\.7 points in layers 30\-38, where both scores agree most \(0\.93\-0\.97\)\.

#### Visualizing selected tokens with rank disagreement

![Refer to caption](https://arxiv.org/html/2610.11358v1/images/token_strip_windows.png)Figure 13:What the map gets wrong in one GSM8K prompt, and which tokens each signal would recompute\.Columns are prompt tokens from four windows, one per section \(position ranges above\)\. Tokens inblueare those rank disagreement recomputes atρ=0\.2\\rho\{=\}0\.2\. The two lanes mark the top 20% of positions by rank disagreement and by mapping error\. The four rows below are relative mapping error∥mapped−true∥2/∥true∥2\\lVert\\mathrm\{mapped\}\-\\mathrm\{true\}\\rVert^\{2\}/\\lVert\\mathrm\{true\}\\rVert^\{2\}\(keys and values together, averaged over layers\), the rank\-disagreement score, and the attention each token receives from the 14B and from the 0\.6B \(last 32 query positions, last quarter of layers\)\. Each row is scaled to its own 95th percentile over the whole prompt; a black tick marks a clipped value, and the<im\_start\>sink’s share of its row is printed\.The Figure[13](https://arxiv.org/html/2610.11358#A3.F13)shows the top 20% tokens selected with rank disagreement vs\. other selector metrics\. Rank disagreement selects most of the tokens from the question or content, whereas a large budget for other metrics is spent in template tokens\. Recomputing template tokens shows negligible accuracy gains, indicating that the mapper effectively maps the template tokens\. Rank disagreement successfully identifies the crucial content tokens helping close the accuracy gap with target prefill\.

## Appendix DLimitations and Future Work

While RaReCache effectively mitigates the accuracy degradation inherent in cross\-model transfer, the underlying closed\-form linear mapping\([Heo et al\., 2026](https://arxiv.org/html/2610.11358#bib.bib1)\)remains sensitive to the distribution of the calibration data\. In our current pipeline, transferring from source to target requires fitting the mapping matrices on a calibration set that closely matches the downstream evaluation distribution \(e\.g\., using the training split of the specific benchmark\)\. Developing generalized, task\-agnostic linear maps or curating a highly diverse, universally representative calibration corpus will enable fully zero\-shot selective recomputation without prior knowledge of the deployment workload\.

RaReCache successfully bridges massive capacity gaps \(e\.g\.,23×23\\timesfrom 0\.6B to 14B parameters\), but it currently assumes that the source and target models share the same tokenizer and vocabulary space, as is standard within model families like Qwen and Llama\. Extending this framework to support KV cache transfer and selective recomputation across completely heterogeneous model families \(with differing tokenizers\) remains an open challenge for future research\.