Tag
ALPHABET is a compact linear-time sequence model that compresses temporal history into stable complex pole modes, achieving competitive performance with fewer parameters and faster inference.
An update on Matrix Recurrent Units (MRU), a linear-time attention alternative. The author explores methods to stabilize training, finding that orthogonal matrices underperform while LDU factorization works best, and shows MRU underperforms transformers on larger datasets like TinyStories.
This paper introduces Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces explicit pairwise query-key comparisons with routing through learned Gaussian mixture components, achieving linear-time complexity in sequence length. Experiments show competitive performance on long-context tasks with fixed-K linear memory scaling.
StateKV is an inference-time method that enables linear-time video prefill for long-video vision-language models by carrying cross-frame context in a fixed-capacity recurrent state, maintaining accuracy close to full self-attention without fine-tuning.