@AI_Whisper_X: Reposting Su Jianlin's review of the K3 architecture. In one sentence, K3 = KDA + MLA + Stable LatentMoE + AttnRes. The whole design isn't about showing off; the core is making trade-offs among model performance, computational efficiency, and training stability. Here's a brief explanation: KD…

X AI KOLs Timeline News
k3 architecture attention moe mla technical-analysis

Summary

Su Jianlin reviews the K3 architecture, focusing on the combination of KDA + MLA + Stable LatentMoE + AttnRes. He explains the design trade-offs, MoE stability improvements, why MLA was kept, and the relationship between DSV4 and MLA.

Reposting Su Jianlin's review of the K3 architecture. In one sentence, K3 = KDA + MLA + Stable LatentMoE + AttnRes. The whole design isn't flashy; the core is making trade-offs among model performance, computational efficiency, and training stability. Here's a brief explanation: KDA, a linear Attention MLA, a full Attention variant with a small KV cache Stable LatentMoE, a MoE approach with stability improvements over LatentMoE AttnRes, replacing fixed equal-weight residual summation with learnable cross-layer attention Several interesting points: ① K3 uses both KDA and MLA. Also, training still uses the Moonlight version of the Muon optimizer, but the attention weights are switched to Per-Head Muon, with each head optimized independently. Su Shen says this won't directly improve performance; it's mainly more reasonable mathematically and structurally: each head is already relatively independent, so they shouldn't be coupled together during optimization. ② Solving the problem that MoE "easily blows up". LatentMoE first reduces dimensionality, then uses more experts, and finally increases dimensionality. With roughly the same training and inference cost, it performs slightly better. However, the consecutive matrix projections also make training more prone to numerical instability. K3 replaces SwiGLU with SiTU-GLU, using softcap to suppress abnormal activations; it also adds an RMS Norm layer before the LatentMoE up-projection. This Norm not only makes training more stable: with similar validation loss, if it is removed, certain benchmarks consistently get worse. (There are tons of technical details here; if interested, see Su Shen's original post.) ③ About MLA DSV4 seems to have redesigned everything; why is K3 still using MLA? Su Shen's answer is that MLA is still hard to beat comprehensively. During training it is in MHA form, and at inference it has a fairly small KV cache. Given fixed training cost and KV cache size, MLA is still nearly optimal. Its problem is that it is not friendly enough to MTP (speculative decoding). MTP's idea is to "trade computation for speed", but MLA already consumes a lot of computation in the decoding phase. Add MTP on top, and the two compete for compute. Switching to other schemes also comes at a cost. For example, switching to 128+128 GQA8, it's hard to beat MLA in quality, and the KV cache is more than three times that of MLA. Switching to 256+256 MFA (essentially MQA), training and prefill costs go up. There is currently no simple attention design that can simultaneously optimize quality, training cost, prefill, KV cache, and decoding compute. Under the hybrid KDA+MLA architecture, some of MLA's problems are alleviated, so K3 ultimately chose MLA. ④ Su Shen thinks DSV4 isn't really "abandoning MLA". DSV4 appears to have redesigned attention, but it still has traces of MLA underneath. It uses MQA with head_dims=512 and K=V, which is exactly MLA's form in the decoding phase. On top of that it adds Sparse + Compress: the former reduces computation, and the latter further compresses the KV cache while also reducing computation. The direction is quite aggressive, but the infrastructure is also more complex. ⑤ The reason K3's MLA can drop RoPE is that KDA already implicitly provides a form of generalized positional information. This only applies to the hybrid KDA+MLA structure. In a model like K2 that is all MLA, removing RoPE directly still noticeably hurts performance. I feel Su Shen's real view is that there are few free upgrades in LLM architecture. K3's design philosophy is not to find a new architecture that crushes all alternatives, but to find a combination that is currently more suitable across quality, training cost, prefill, decoding, KV cache, and stability. Su Shen's original post has a lot more technical content and is well worth reading. Original post: https://kexue.fm/archives/11848
Original Article
View Cached Full Text

Cached at: 08/05/26, 08:21 AM

Sharing Su Jianlin’s review of the K3 architecture.

In one sentence, K3 = KDA + MLA + Stable LatentMoE + AttnRes. The overall design isn’t flashy; the core is striking a trade-off among model quality, computational efficiency, and training stability.

Here’s a quick explanation:

  • KDA: a linear attention variant
  • MLA: a full attention variant with small KV cache
  • Stable LatentMoE: a MoE approach with stability improvements over LatentMoE
  • AttnRes: uses learnable cross-layer attention to replace fixed equal-weight residual summation

A few interesting points:

  1. K3 uses both KDA and MLA. Also, training still uses the Moonlight version of the Muon optimizer, but the attention weights now use Per-Head Muon, optimizing each head independently. Su says this doesn’t directly improve performance; it’s mainly more reasonable mathematically and architecturally: heads are inherently relatively independent and shouldn’t be coupled together during optimization.

  2. Addressing the “easy to blow up” problem with MoE.

LatentMoE first reduces dimensionality, then uses more experts, and finally projects back up, achieving slightly better results at roughly the same training/inference cost. However, the successive matrix projections also make training more prone to numerical instability.

K3 replaces SwiGLU with SiTU-GLU, using softcap to suppress abnormal activations; it also adds an RMS Norm before the LatentMoE up-projection. This norm doesn’t just make training more stable—when validation loss is roughly the same, certain benchmarks consistently degrade without it.

(There are tons of technical details here; read Su’s original post if interested.)

  1. On MLA

It looks like DSV4 has switched to a new design, so why is K3 still using MLA? Su’s answer: currently, MLA is still hard to fully beat. In training it has the MHA form, and during inference it has a small KV cache; given fixed training cost and KV cache budget, MLA remains nearly optimal.

Its problem is that it’s not very friendly to MTP (speculative decoding). MTP’s idea is to “trade compute for speed,” while MLA itself is already compute-heavy during decoding. Adding MTP on top means the two compete for compute.

Switching to other options also has costs. For example, switching to GQA8 with 128+128 gives results that struggle to beat MLA, and the KV cache is more than three times that of MLA; switching to MFA with 256+256 (essentially MQA) raises training and prefill costs.

Currently, no simple attention design can simultaneously cover quality, training cost, prefill, KV cache, and decoding compute. In the KDA+MLA hybrid architecture, some of MLA’s problems are alleviated, so K3 ultimately chose MLA.

  1. Su believes DSV4 hasn’t truly “abandoned MLA” either.

DSV4 appears to have redesigned attention, but underneath there are still traces of MLA. It uses MQA with head_dims=512 and K=V, which is exactly MLA’s inference-time form; combined with Sparse + Compress, the former reduces compute and the latter further compresses the KV cache while also reducing compute.

The direction is aggressive, but the infra is also more complex.

  1. K3’s MLA can drop RoPE because KDA already implicitly provides a generalized form of positional information.

This only applies to the KDA+MLA hybrid structure; for a full-MLA model like K2, dropping RoPE directly still significantly hurts performance.

It seems Su’s real point is: there are few free lunches in large-model architecture. K3’s design philosophy isn’t to find a new architecture that crushes everything, but to find a currently suitable combination among quality, training cost, prefill, decoding, KV cache, and stability.

Su’s original post contains a lot more technical content and is well worth reading.

Original post: https://kexue.fm/archives/11848

Similar Articles

@qingke_ai: https://x.com/qingke_ai/status/2079035914740019638

X AI KOLs Timeline

Using NVIDIA's Nemotron 3 Super and Moonshot AI's Kimi K3 as examples, this article analyzes how the LatentMoE architecture overcomes the efficiency bottleneck of traditional MoE by compressing the Expert computation dimension, and points out that this is a turning point for the next generation of MoE architectures.

Kimi K3 Architecture Overview and Notes

Hacker News Top

Sebastian Raschka provides an architectural overview of the open-weight Kimi K3 model, highlighting its scaling from 48B to 2.8T parameters, new LatentMoE and attention residual components, removal of RoPE in favor of NoPE, and native multimodal support. The model emphasizes inference efficiency and matches frontier performance.

The Inference Engine Guide for K3 Deployment (10 minute read)

TLDR AI

Kimi K3, a 2.8-trillion-parameter multimodal MoE model with 1M context window, now has day-0 vLLM support. This guide details how vLLM serves K3's novel architecture including Kimi Delta Attention, Attention Residuals, and speculative decoding achieving up to 370 tok/s.

@Datou: Microsoft values its reputation, deliberately avoiding synthetic data. They trained a base model using only human data, then split it into three expert models for different domains. They then distilled these three capabilities back into the base model (weight ratio allocation requires experience), followed by a round of reinforcement learning to enable the distilled model to flexibly apply the right capability based on the problem.

X AI KOLs Timeline

Microsoft releases technical details of MAI-Thinking-1 training: uses purely human data to train a base model, then trains three domain expert models, merges capabilities back into the base model via distillation, and then applies reinforcement learning to enable the model to flexibly utilize different capabilities.