Tag
The author shares insights from training a small model with DeepSeek's latent attention, observing layer-dependent latent usage and a test-time trick that reduces KV cache 4x without loss change.
Sebastian Raschka provides an architectural overview of the open-weight Kimi K3 model, highlighting its scaling from 48B to 2.8T parameters, new LatentMoE and attention residual components, removal of RoPE in favor of NoPE, and native multimodal support. The model emphasizes inference efficiency and matches frontier performance.