Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

arXiv cs.CL Papers
kv-cache llm cache-translation heterogeneous-models mixture-of-experts multi-agent

Summary

This paper introduces Mixture-of-Translators (MoT), a framework for translating KV caches across heterogeneous LLMs, enabling cache reuse between different architectures. Experiments show preserved QA performance and long-context quality across Qwen2.5, GPT-2, and OPT models.

arXiv:2607.28979v1 Announce Type: new Abstract: Heterogeneous Large Language Model (LLM) systems increasingly rely on shared contexts, retrieved evidence, and multi-agent dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators(MoT), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules to capture diverse source--target mappings. To further reduce residual translation error, we introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory. We reveal two competing failure modes in cache translation: propagated translation shift from early injection and last-state shift from late injection. MoT addresses them through translator mixtures and target-side correction. Across homogeneous and heterogeneous translations among Qwen2.5, GPT-2, and OPT models, MoT preserves downstream QA performance, including Qwen2.5-7B-scale translation with 51.0% average closed-set QA accuracy and 0.43 average extractive QA F1. In practical case studies, MoT enables quality-preserving memory reuse for multi-agent reasoning and retains 96.3% of direct-context quality in long-context cache-augmented generation, demonstrating scalable KV cache reuse across heterogeneous LLMs.
Original Article
View Cached Full Text

Cached at: 08/03/26, 07:34 AM 

# Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models
Source: [https://arxiv.org/abs/2607.28979](https://arxiv.org/abs/2607.28979)
[View PDF](https://arxiv.org/pdf/2607.28979)

> Abstract:Heterogeneous Large Language Model \(LLM\) systems increasingly rely on shared contexts, retrieved evidence, and multi\-agent dialogue histories, yet their internal key\-value \(KV\) caches remain model\-specific and cannot be reused across architectures\. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi\-model reasoning and long\-context generation\. We propose Mixture\-of\-Translators\(MoT\), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM\. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules to capture diverse source\-\-target mappings\. To further reduce residual translation error, we introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory\. We reveal two competing failure modes in cache translation: propagated translation shift from early injection and last\-state shift from late injection\. MoT addresses them through translator mixtures and target\-side correction\. Across homogeneous and heterogeneous translations among Qwen2\.5, GPT\-2, and OPT models, MoT preserves downstream QA performance, including Qwen2\.5\-7B\-scale translation with 51\.0% average closed\-set QA accuracy and 0\.43 average extractive QA F1\. In practical case studies, MoT enables quality\-preserving memory reuse for multi\-agent reasoning and retains 96\.3% of direct\-context quality in long\-context cache\-augmented generation, demonstrating scalable KV cache reuse across heterogeneous LLMs\.

## Submission history

From: Jin\-Woo Lee \[[view email](https://arxiv.org/show-email/8761da67/2607.28979)\] **\[v1\]**Fri, 31 Jul 2026 03:07:31 UTC \(2,053 KB\)

Similar Articles

Enabling KV Caching of Shared Prefix for Diffusion Language Models

arXiv cs.LG

This paper proposes BiCache, a novel KV caching technique for shared prefixes in diffusion language models, which avoids accuracy collapse by dynamically reusing cached keys and values in shallow layers and achieves 36.3%–98.3% throughput improvement.