@nrehiew_: > LatentMoE > 16 activated experts out of 896 > Kimi Delta Attention and AttnRes > 2.5x more efficient scaling This is …
Summary
Discussion of LatentMoE architecture with extreme sparsity (16/896 experts) and Kimi Delta Attention, claiming 2.5x more efficient scaling, and speculation about Kimi K3 model capabilities.
View Cached Full Text
Cached at: 07/16/26, 06:22 PM
LatentMoE 16 activated experts out of 896 Kimi Delta Attention and AttnRes 2.5x more efficient scaling
This is some insane sparsity ratio and it looks like going LatentMoE allows for more experts and extreme sparsity.
For reference, Inkling is 1T params and 8/256 experts.
This also tracks with the rumors that OpenAI/Anthropic are significantly more sparse
wh (@nrehiew_): If Kimi K3 is indeed Opus level, the main architectural ideas that would be most interesting are
- What degree of sparsity at 2T params
- What linear attention did they use
Similar Articles
@qingke_ai: https://x.com/qingke_ai/status/2079035914740019638
Using NVIDIA's Nemotron 3 Super and Moonshot AI's Kimi K3 as examples, this article analyzes how the LatentMoE architecture overcomes the efficiency bottleneck of traditional MoE by compressing the Expert computation dimension, and points out that this is a turning point for the next generation of MoE architectures.
Kimi K3 Architecture Overview and Notes
Sebastian Raschka provides an architectural overview of the open-weight Kimi K3 model, highlighting its scaling from 48B to 2.8T parameters, new LatentMoE and attention residual components, removal of RoPE in favor of NoPE, and native multimodal support. The model emphasizes inference efficiency and matches frontier performance.
Sparse By Design (5 minute read)
Moonshot's Kimi K3, a 2.8 trillion parameter open weights model with 896 experts (16 active per token), exemplifies the trend of scaling total parameters while holding active compute constant, and uses attention compression to reduce KV cache size, making frontier inference more accessible but with high storage costs.
Kimi-K3 Technical Report [pdf]
MoonshotAI releases Kimi-K3, a 2.8T-parameter open-weight multimodal agentic model with a 1M-token context window, built on new Kimi Delta Attention and Attention Residuals architecture, achieving significant scaling improvements.
The Inference Engine Guide for K3 Deployment (10 minute read)
Kimi K3, a 2.8-trillion-parameter multimodal MoE model with 1M context window, now has day-0 vLLM support. This guide details how vLLM serves K3's novel architecture including Kimi Delta Attention, Attention Residuals, and speculative decoding achieving up to 370 tok/s.