linear-attention

Video DeltaNet presents a hybrid attention mechanism combining Softmax and linear attention to enhance efficiency in video generation models, achieving a 14.5x speedup over baseline methods.

This paper decomposes associative recall in fixed-state recurrences, finding convolution and curriculum learning key to performance, and proposes interventions to address interference rather than capacity limitations.

SpectralShift introduces a spectral reparameterization approach to effectively extend the context window of Gated DeltaNet models by reshaping the decay spectrum, improving long-context capabilities through continual pretraining.

Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.

This paper proposes intervention-based metrics to differentiate retrieval and positional heads in RoPE Transformers, leading to a principled hybrid architecture (HwH) that combines full and linear attention for improved language modeling and long-context extrapolation.

This paper explains why fully quantizing the recurrent Gated DeltaNet layers in hybrid LLMs to 4-bit NVFP4 preserves accuracy across long-context benchmarks, detailing mechanisms like outlier localization and robust delta-rule dynamics.

The article discusses Test Time Training as a potential new scaling axis in AI development, analyzing a paper that frames it as a form of linear attention and exploring its implications for model training and continual learning.

Sliding window attention with sinks outperforms post-trained linear attention on long-context tasks without retraining, offering a cheaper and more reliable inference solution for LLMs.

This paper introduces Spatially-Sparse Linear Attention for low-latency object detection with event cameras, achieving state-of-the-art accuracy with 20x less computation than prior asynchronous methods.

The author speculates on the possible architecture of Qwen 4, discussing potential designs like embedding-offloaded linear attention or multi-head latent attention hybrids based on reverse engineering Deepseek.

Introduces a Query-derived Erase Direction (QED) for linear attention models to improve retrieval and about double the usable context length, as demonstrated on the S-NIAH-1 benchmark.

This paper presents the first systematic study of massive activations in hybrid linear-attention LLMs, uncovering pre-attention spikes and inter-spike plateaus governed by cancellation timing, and showing how their morphology recovers at full-attention limits.

This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.

This paper investigates the failure modes of converting Qwen3-0.6B-Base attention layers to KDA linear attention on a single GPU, identifying an 'interface injury' where the model predicts option labels rather than content, and proposes a format-targeted KL stage to repair it.

A tweet highlights that major open-source models like Qwen3.8-Max may use linear attention, referencing Alibaba's announcement of upcoming open-weight releases for Qwen3.8-Max (2.4T parameters) and Qwen3.8-27B.

A guide recommending a reading order of foundational papers and Kimi model reports to understand the architecture of Moonshot AI's Kimi K3, covering linear attention, MoE, and residual connections.

This blog post derives Kimi Delta Attention step by step from standard softmax attention through linear attention and DeltaNet variants, explaining the state update equations used by recent Qwen and Kimi models.

Kimi Linear proposes a new linear attention architecture designed to enhance both expressiveness and efficiency in Transformer models, with contributions from the Kimi Team at Moonshot AI.

NVIDIA Research introduces SANA-Video 2.0, a hybrid video diffusion transformer that generates high-quality 720p video on a single GPU, achieving up to 120× speedup over Wan 2.2-14B via hybrid linear-softmax attention and block attention residuals.

Kimi released K3, a 2.8T-parameter open model using delta attention to avoid growing KV cache, enabling a 1-million-token context window with linear memory cost.