Virtual Head Attention

Hugging Face Daily Papers Papers
attention-mechanism kv-cache long-context efficiency llm inference-optimization moe

Summary

论文提出 Virtual Head Attention (VHA)，通过轻量级线性 Q Premix 与 Linear Postmix 操作同时优化 KV cache 大小、训练/prefill FLOPs 与模型质量，在多个规模上超越 GQA 基线，并将 KV cache 相比 GQA-8 减少 4 倍。

The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute. We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank I + AB^top residual structure. Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standard GQA-2 architecture that directly reuses all existing GQA inference optimizations. With only two KV heads, the group size in large models readily reaches 64, fully exploiting Tensor Core parallelism during decoding. Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline at every scale (+0.05\% to +0.81\%), while reducing KV cache by 4times vs.\ GQA-8 at dense scales and 2times vs.\ GQA-4 at the MoE scale, and lowering training and prefill FLOPs by 7.9\% (1.7B, S{=}4096). On an industrial 30B-A3B MoE model trained on 500B tokens at 8K context, VHA outperforms both MLA and GQA-4 by +0.70\,/\,+2.60\,\% on a 12-benchmark base-model evaluation suite; in a separate matched 10B-A1B MoE comparison after long-context mid-training, VHA attains an average RULER score of 84.4 over 8K--128K versus 73.0 for GQA, while using two rather than four KV heads in the full-attention layers.
Original Article
View Cached Full Text

Cached at: 10/16/26, 11:03 AM

Paper page - Virtual Head Attention

Source: https://huggingface.co/papers/2610.17959

Abstract

Virtual Head Attention improves long-context model efficiency by reducing KV cache and compute while preserving quality through lightweight linear query and output transformations.

The design ofattention mechanismsinlong-context large language modelsfaces an ``impossible triangle’’ amongKV cachesize, training and prefill FLOPs, and model quality.GQAachieves strong quality but incurs highKV cacheoverhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute. We proposeVirtual Head Attention(VHA), which simultaneously improves all three vertices through two lightweight linear operations:Q Premixapplies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, andLinear Postmixfuses inter-head features via a low-rank I + AB^top residual structure. Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standardGQA-2 architecture that directly reuses all existingGQAinference optimizations. With only two KV heads, the group size in large models readily reaches 64, fully exploitingTensor Coreparallelism during decoding. Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standardGQAbaseline at every scale (+0.05\% to +0.81\%), while reducingKV cacheby 4times vs.\GQA-8 at dense scales and 2times vs.\GQA-4 at theMoEscale, and lowering training and prefill FLOPs by 7.9\% (1.7B, S{=}4096). On an industrial 30B-A3BMoEmodel trained on 500B tokens at 8K context, VHA outperforms both MLA andGQA-4 by +0.70\,/\,+2.60\,\% on a 12-benchmark base-model evaluation suite; in a separate matched 10B-A1BMoEcomparison after long-context mid-training, VHA attains an average RULER score of 84.4 over 8K--128K versus 73.0 forGQA, while using two rather than four KV heads in the full-attention layers.

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2610\.17959

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.17959 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.17959 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.17959 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

QuantWM: Temporally Consistent 2-Bit KV Cache Quantization for Video World Models

Hugging Face Daily Papers

QuantWM 提出一种免训练的 2-bit KV cache 量化框架，通过量化敏感性感知聚类（QSAC）与主子空间注意力补偿（PSAC）来稳定视频世界模型的注意力 logits，缓解时间闪烁与视觉退化，并在多个基准上实现最高 6.20 倍 KV cache 内存压缩。

HeadGuard: Selective Head Protection for Low-Bit VLM KV-Cache Quantization

arXiv cs.LG

The paper introduces HeadGuard, a composable method that protects a small subset of KV heads in high precision while applying low-bit quantization to the rest of a VLM's image KV cache, substantially recovering accuracy lost by aggressive 2-bit quantization across multiple models and benchmarks.