Tag
This blog post derives Kimi Delta Attention step by step from standard softmax attention through linear attention and DeltaNet variants, explaining the state update equations used by recent Qwen and Kimi models.
Kimi released K3, a 2.8T-parameter open model using delta attention to avoid growing KV cache, enabling a 1-million-token context window with linear memory cost.
Introduces Semidirect Fourier Delta Attention (SFDA), a phase-controlled delta-rule layer that extends Kimi Delta Attention with block-rotational Fourier control operators, providing a constructive chunk-WY theorem for efficient chunkwise computation and demonstrating expressivity for cyclic and register memories.
Delta Attention Residuals improve layer-wise routing in transformer models by attending to feature changes (deltas) rather than cumulative hidden states, achieving 1.7–8.2% validation perplexity gains across scales from 220M to 7.6B parameters.