Draft-KV: Learning Useful Latent Communication Between Language Models
Summary
Draft-KV is a method for latent communication between language models that sends key-value states instead of decoded text, improving accuracy on benchmarks like MMLU with minimal parameters while keeping models frozen.
View Cached Full Text
Cached at: 09/29/26, 08:15 AM
Paper page - Draft-KV: Learning Useful Latent Communication Between Language Models
Source: https://huggingface.co/papers/2609.34754
Abstract
Latentcommunicationpassesinternalstatesbetweenlanguagemodelsinsteadofdecodedtext,buthigherreceiveraccuracydoesnotshowthatthereceiverusedthemessagecontent.Acrossfivemethod-datasetpairs,replacingeachmessagewithonefromanunrelatedquestionchangesaccuracybyatmost0.60points,evenwhencommunicationadds15.44pointsoverthereceiveralone.Thustheinterfacecansupplythegainwhilemakingthesharerdispensable.Draft-KVinsteadsendsthekey-valuestatesformedwhilethesharerdraftsananswertothecurrentquestion.Linearprojectionsplacethesestatesinasidememoryreadthroughagatedattentionbranch,andprogressivetrainingmovesfrommessagereconstructiontoanswersupervisionunderaguardonharmfrommismatchedmessages.Bothmodelsremainfrozenandtheinterfacetrains1.05Mparameters,348xfewerthanC2C.WithaQwen3-8Bsharer,afrozenQwen2.5-0.5B-Instructreceiverreaches78.04%onMMLU-Redux,versus37.45%aloneand36.40%withreassignedmessages.Atfixedinterfacesize,scalingthesharerfrom0.6Bto8Braisesaccuracyfrom46.11%to78.04%;communicationalsotransferstoheld-outtasksandcanexceedbothmodelswheneachholdsdifferentevidence.
View arXiv pageView PDFGitHub3Add to collection
Get this paper in your agent:
hf papers read 2609\.34754
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.34754 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.34754 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.34754 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Archer: Adaptive Reuse of Cached Hidden States for Efficient Rollback in Diffusion Language Models
Introduces Archer, a training-free KV caching method for diffusion language models that adaptively reuses cached hidden states to reduce recomputation while preserving rollback capabilities, achieving up to 2.95x speedup and improved generation quality.
MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
MM-ShiftKV is a training-free method that improves KV cache selection for multimodal LLMs by approximating decoding-time query behavior during prefilling, reducing memory footprint while preserving performance.
Dual-Cache Latent Space Communication between Heterogeneous Language Models
The paper introduces XKV, a method for efficient latent space communication between heterogeneous language models in multi-agent systems, improving accuracy and speed over existing text and cache-based protocols.
Speculative Correction: Draft-then-Refine Decoding for Diffusion Language Models
This paper proposes speculative correction, a training-free draft-then-refine decoding strategy for diffusion language models, showing quality-latency improvements using LLaDA2.1 models.
Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models
Switch-KD proposes a novel visual-switch knowledge distillation framework for efficiently compressing vision-language models by unifying multimodal knowledge transfer within a shared text-probability space. The method achieves 3.6-point average improvement across 10 multimodal benchmarks when distilling a 0.5B TinyLLaVA student from a 3B teacher model.