mla

Su Jianlin reviews the K3 architecture, focusing on the combination of KDA + MLA + Stable LatentMoE + AttnRes. He explains the design trade-offs, MoE stability improvements, why MLA was kept, and the relationship between DSV4 and MLA.

This paper proposes functional reconstruction for converting MHA/GQA checkpoints into MLA draft models for speculative decoding, directly optimizing attention modules to preserve token acceptance. It reports consistent improvements across 192 configurations involving Llama/Qwen models and multiple conversion methods.

A researcher questions the reproducibility of MLA outperforming GQA under same KV cache, sharing early small-scale ablation results and plans for scaling experiments to decide on architecture for next large-scale run.

22-year-old developer Kye Gomez reversed Anthropic's Claude Mythos black box architecture in just two days and open-sourced the OpenMythos project, using Recurrent-Depth Transformer and other techniques, achieving performance equivalent to a 1.3B model with 770M parameters.

Sebastian Raschka points out the chain of inspiration from LatentMoE back to eigendecomposition through MLA, LoRA, and SVD.