Kimi Linear: 一种富有表现力且高效的注意力架构

# Kimi Linear：一种富有表现力且高效的注意力架构
来源：https://arxiv.org/abs/2510.26690
作者：Kimi Team (https://arxiv.org/search/cs?searchtype=author&query=Kimi+Team)：Yu Zhang (https://arxiv.org/search/cs?searchtype=author&query=Zhang,+Y), Zongyu Lin (https://arxiv.org/search/cs?searchtype=author&query=Lin,+Z), Xingcheng Yao (https://arxiv.org/search/cs?searchtype=author&query=Yao,+X), Jiaxi Hu (https://arxiv.org/search/cs?searchtype=author&query=Hu,+J), Fanqing Meng (https://arxiv.org/search/cs?searchtype=author&query=Meng,+F), Chengyin Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+C), Xin Men (https://arxiv.org/search/cs?searchtype=author&query=Men,+X), Songlin Yang (https://arxiv.org/search/cs?searchtype=author&query=Yang,+S), Zhiyuan Li (https://arxiv.org/search/cs?searchtype=author&query=Li,+Z), Wentao Li (https://arxiv.org/search/cs?searchtype=author&query=Li,+W), Enzhe Lu (https://arxiv.org/search/cs?searchtype=author&query=Lu,+E), Weizhou Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+W), Yanru Chen (https://arxiv.org/search/cs?searchtype=author&query=Chen,+Y), Weixin Xu (https://arxiv.org/search/cs?searchtype=author&query=Xu,+W), Longhui Yu (https://arxiv.org/search/cs?searchtype=author&query=Yu,+L), Yejie Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+Y), Yu Fan (https://arxiv.org/search/cs?searchtype=author&query=Fan,+Y), Longguang Zhong (https://arxiv.org/search/cs?searchtype=author&query=Zhong,+L), Enming Yuan (https://arxiv.org/search/cs?searchtype=author&query=Yuan,+E), Dehao Zhang (https://arxiv.org/search/cs?searchtype=author&query=Zhang,+D), Yizhi Zhang (https://arxiv.org/search/cs?searchtype=author&query=Zhang,+Y), T. Y. Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+T), Haiming Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+H), Shengjun Fang (https://arxiv.org/search/cs?searchtype=author&query=Fang,+S), Weiran He (https://arxiv.org/search/cs?searchtype=author&query=He,+W), Shaowei Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+S), Yiwei Li (https://arxiv.org/search/cs?searchtype=author&query=Li,+Y), Jianlin Su (https://arxiv.org/search/cs?searchtype=author&query=Su,+J), Jiezhong Qiu (https://arxiv.org/search/cs?searchtype=author&query=Qiu,+J), Bo Pang (https://arxiv.org/search/cs?searchtype=author&query=Pang,+B), Junjie Yan (https://arxiv.org/search/cs?searchtype=author&query=Yan,+J), Zhejun Jiang (https://arxiv.org/search/cs?searchtype=author&query=Jiang,+Z), Weixiao Huang (https://arxiv.org/search/cs?searchtype=author&query=Huang,+W), Bohong Yin (https://arxiv.org/search/cs?searchtype=author&query=Yin,+B), Jiacheng You (https://arxiv.org/search/cs?searchtype=author&query=You,+J), Chu Wei (https://arxiv.org/search/cs?searchtype=author&query=Wei,+C), Zhengtao Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+Z), Chao Hong (https://arxiv.org/search/cs?searchtype=author&query=Hong,+C), Yutian Chen (https://arxiv.org/search/cs?searchtype=author&query=Chen,+Y), Guanduo Chen (https://arxiv.org/search/cs?searchtype=author&query=Chen,+G), Yucheng Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+Y), Huabin Zheng (https://arxiv.org/search/cs?searchtype=author&query=Zheng,+H), Feng Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+F), Yibo Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+Y), Mengnan Dong (https://arxiv.org/search/cs?searchtype=author&query=Dong,+M), Zheng Zhang (https://arxiv.org/search/cs?searchtype=author&query=Zhang,+Z), Siyuan Pan (https://arxiv.org/search/cs?searchtype=author&query=Pan,+S), Wenhao Wu (https://arxiv.org/search/cs?searchtype=author&query=Wu,+W), Yuhao Wu (https://arxiv.org/search/cs?searchtype=author&query=Wu,+Y), Longyu Guan (https://arxiv.org/search/cs?searchtype=author&query=Guan,+L), Jiawen Tao (https://arxiv.org/search/cs?searchtype=author&query=Tao,+J), Guohong Fu (https://arxiv.org/search/cs?searchtype=author&query=Fu,+G), Xinran Xu (https://arxiv.org/search/cs?searchtype=author&query=Xu,+X), Yuzhi Wang (https://arxiv.org/search/cs?searchtype=author&query=Wang,+Y), Guokun Lai (https://arxiv.org/search/cs?searchtype=author&query=Lai,+G), Yuxin Wu (https://arxiv.org/search/cs?searchtype=author&query=Wu,+Y), Xinyu Zhou (https://arxiv.org/search/cs?searchtype=author&query=Zhou,+X), Zhilin Yang (https://arxiv.org/search/cs?searchtype=author&query=Yang,+Z), Yulun Du (https://arxiv.org/search/cs?searchtype=author&query=Du,+Y)

查看 PDF (https://arxiv.org/pdf/2510.26692)

> 摘要：我们提出了 Kimi Linear，一种混合线性注意力架构。在公平对比下，它首次在多种场景——包括短上下文、长上下文和强化学习（RL）扩展——中均超越全注意力。其核心是 Kimi Delta Attention (KDA)，一个富有表现力的线性注意力模块，它扩展了 Gated DeltaNet，引入了更细粒度的门控机制，从而更有效地利用有限状态 RNN 的有限记忆。我们定制的分块算法通过一种专用的对角加低秩（DPLR）转移矩阵变体实现了高硬件效率，与普通 DPLR 公式相比大幅降低了计算量，同时与经典的 delta 规则保持更高的一致性。我们使用 3B 激活参数和 48B 总参数预训练了一个 Kimi Linear 模型，该模型基于 KDA 和多头潜在注意力（MLA）的逐层混合。实验表明，在使用相同训练方案的情况下，Kimi Linear 在所有评估任务上均以较大优势优于全 MLA，同时将 KV 缓存使用量减少高达 75%，并在 1M 上下文中实现高达 6 倍的解码吞吐量。这些结果表明，Kimi Linear 可以作为全注意力架构的即插即用替代品，在包括更长输入和输出长度的任务中提供更优越的性能和效率。为支持进一步研究，我们开源了 KDA 内核和 vLLM 实现，并发布了预训练和指令微调的模型检查点。

## 提交历史

来自：Yulun Du [查看邮件 (https://arxiv.org/show-email/19b5afec/2510.26692)] **[v1](https://arxiv.org/abs/2510.26692v1)** 2025年10月30日星期四 16:59:43 UTC (645 KB) **[v2]** 2025年11月1日星期六 12:05:18 UTC (691 KB)