Tag
FreeSolo introduces a native sparse attention mechanism combining short-range sliding window and long-range keypoint-based branches, achieving linear-time complexity and improved zero-shot length extrapolation to 256k+ tokens without costly long-context fine-tuning.
OrthoSSM-130M is a new Phantom state space model optimized for linear-time, constant-memory text generation, positioned as a fast and efficient alternative to Mamba.
ALPHABET is a compact linear-time sequence model that compresses temporal history into stable complex pole modes, achieving competitive performance with fewer parameters and faster inference.
An update on Matrix Recurrent Units (MRU), a linear-time attention alternative. The author explores methods to stabilize training, finding that orthogonal matrices underperform while LDU factorization works best, and shows MRU underperforms transformers on larger datasets like TinyStories.
This paper introduces Gaussian Mixture Attention (GMA), a probabilistic attention mechanism that replaces explicit pairwise query-key comparisons with routing through learned Gaussian mixture components, achieving linear-time complexity in sequence length. Experiments show competitive performance on long-context tasks with fixed-K linear memory scaling.
StateKV is an inference-time method that enables linear-time video prefill for long-video vision-language models by carrying cross-frame context in a fixed-capacity recurrent state, maintaining accuracy close to full self-attention without fine-tuning.