Brilliant paper by NVIDIA.

they found a way to make KV cache transferable between models.

the target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again.

let’s understand why this is so important today.

LLM APIs are stateless, so every turn sends the entire conversation back to the model. the model reads it again before generating a new token, and all of it is billed as input.

prompt caching lets providers hold the KV cache for a stable prefix and bill cache hits at roughly 10% of the base input rate because that compute was already done.

this 90% reduction is one of the biggest levers in LLM serving.

but the cache only works on the model that produced it. since keys and values depend on that model’s weights, another model cannot directly reuse them.

this becomes a problem with LLM routing.

if traffic moves to another model for cost or capability reasons, the accumulated KV cache becomes invalid.

the entire context must be processed again and billed at full rate.

NVIDIA’s recent paper treats this as a representation problem.

prefill’s main output is the KV cache, so transferring it means converting one model’s cache into the representation another model expects.

they first checked whether this conversion had enough structure to exploit.

for Qwen3 14B → 32B, a linear regression from a single source layer reconstructed 56% of the variance in the target model’s keys.

because models can have different layer counts, there is no natural one-to-one layer mapping.

so, for every target layer, they rank the source layers by predictive power and use the top eight together.

that pushes reconstruction to 79%.

the mapper has three parts:

each target layer and head gets its own linear map, solved in closed form instead of with gradient descent.

cross-layer selection chooses the source layers that best predict each target layer. their ablation shows this contributes the most.

RoPE adds a position-dependent rotation to keys. they remove it, fit the mapping in position-free space, then apply the target model’s rotation again at inference.

across six model pairs from Qwen3, Llama 3.1, and Ministral 3, four retain 73 to 98% of the receiving model’s standalone accuracy.

conversion is also 3 to 25x faster than processing the context again.

prior cross-model KV reuse methods either train a neural adapter for every pair or require architecturally identical models.

this approach is closed-form and training-free, but there are still important limitations.

every tested pair belongs to the same model family, such as Qwen → Qwen or Llama → Llama.

cross-family transfer is still future work.

the tested pairs also share KV head count and per-head dimension. mismatched configurations remain untested.

and the work currently covers dense full-attention models only, not sliding-window or attention-recurrent hybrids.

link to the paper: https://arxiv.org/abs/2608.03893

i wrote a first-principles breakdown of how KV caching works

the article is quoted below.

Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse

Source: https://arxiv.org/abs/2608.03893 View PDFHTML (experimental)

Abstract:Production deployments often swap between different-sized models in a family for cost-quality cascading, mid-conversation switching, and routing, and each swap forces the receiver to repay the prefill from scratch. We propose cross-model KV cache transfer, where the receiver reuses the source’s KV cache, skipping prefill. We find that cross-model KV has substantial linear structure across matched-KV pairs, where source and target share KV head count and per-head dimension. On Qwen3 14B->32B, one source layer explains 56% of variance in the target’s keys and 32% in values, rising to 79% and 65% with multiple source layers. Building on this, we design a closed-form ridge mapper that operates per head and proceeds in three steps. First, for each target layer we select the top-k most predictive source layers and concatenate their KV as input. Second, we strip RoPE from the keys before mapping, so the fit is position-free and reusable across context lengths. Third, we fit ridge regression on a small calibration set of 500 FineWeb-Edu sequences of 1,024 tokens each. Surprisingly, across six pairs in three families, this linear mapper retains 73-98% of the receiver’s standalone-prefill accuracy on four pairs, while two degrade sharply. A nonlinear MLP recovers up to +37 pp HellaSwag retention on the failures. The mapper runs 2.7-25x faster than re-prefill and remains stable across multi-turn handoff, making cross-model KV cache transfer practical.

Submission history

From: Taekyung Heo [view email] **[v1]**Tue, 4 Aug 2026 16:26:47 UTC (1,193 KB)