Tag
提出了 RAQ-MLA，一种针对 Multi-Head Latent Attention 缓存的角色感知量化方法，无需训练即可将 MLA 缓存压缩至约 2.5 比特，在四个 MLA 模型上推理与编码任务几乎无损。
RoLaQ提出了针对多头潜注意力（MLA）模型的2位KV缓存联合旋转与范围校准的离线量化方法，在DeepSeek-V2-Lite上实现3.96倍BF16 KV容量和2.51倍解码吞吐提升。
QuantMLA 提出一种功能对齐的双路径量化框架，对 MLA 的内容路径与 RoPE 路径进行低比特 KV 缓存压缩，首次实现 INT4 联合缓存且精度几乎无损，并通过自定义低比特注意力内核在 128K 上下文下获得 3.59 倍压缩和 5.168 倍吞吐提升。
This paper presents an exploratory ablation study of TALH, a hybrid language model combining MLA and SSM, showing that SSM integration is more critical for validation performance than MLA in the tested setup, with insights on memory usage and timing on consumer hardware.
The author speculates on the possible architecture of Qwen 4, discussing potential designs like embedding-offloaded linear attention or multi-head latent attention hybrids based on reverse engineering Deepseek.
This paper presents the first mechanistic interpretability study of Multi-head Latent Attention (MLA), analyzing how its low-rank bottleneck separates content from positional information and reshapes transformer circuits.