MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Hugging Face Daily Papers Papers
linear-attention momentum-optimization llm-efficiency mamba-alternative training-acceleration

Summary

The paper introduces Momentum DeltaNet (MDN), a linear attention model that uses stepwise momentum and parallel algorithms to improve training efficiency and performance over models like Mamba2.

Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online stochastic gradient descent (SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. While momentum-based optimizers provide a natural remedy, they pose challenges in simultaneously achieving training efficiency and effectiveness. To address this, we develop a chunkwise parallel algorithm for LA with a stepwise momentum rule by geometrically reordering the update coefficients. Further, from a dynamical systems perspective, we analyze the momentum-based recurrence as a second-order system that introduces complex conjugate eigenvalues. This analysis guides the design of stable gating constraints. The resulting model, Momentum DeltaNet (MDN), leverages Triton kernels to achieve comparable training throughput with competitive linear models such as Mamba2 and KDA. Extensive experiments on the 400M and 1.3B parameter models demonstrate consistent performance improvements over strong baselines, including Transformers, Mamba2 and GDN, across diverse downstream evaluation benchmarks. Code: https://github.com/HuuYuLong/MomentumDeltaNet .
Original Article
View Cached Full Text

Cached at: 05/11/26, 07:22 AM

Paper page - MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

Source: https://huggingface.co/papers/2605.05838

Abstract

Linear attention models face challenges with information decay and convergence, which are addressed through a momentum-based approach that improves training efficiency and performance over existing models like Mamba2 and GDN.

Linear Attention(LA) offers a promising paradigm for scalinglarge language models(LLMs) to long sequences by avoiding the quadratic complexity ofself-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form onlinestochastic gradient descent(SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. Whilemomentum-based optimizersprovide a natural remedy, they pose challenges in simultaneously achieving training efficiency and effectiveness. To address this, we develop a chunkwise parallel algorithm for LA with a stepwise momentum rule by geometrically reordering the update coefficients. Further, from adynamical systemsperspective, we analyze the momentum-based recurrence as a second-order system that introduces complex conjugateeigenvalues. This analysis guides the design of stablegating constraints. The resulting model,Momentum DeltaNet(MDN), leveragesTriton kernelsto achieve comparable training throughput with competitive linear models such as Mamba2 and KDA. Extensive experiments on the 400M and 1.3B parameter models demonstrate consistent performance improvements over strong baselines, including Transformers, Mamba2 and GDN, across diverse downstream evaluation benchmarks. Code: https://github.com/HuuYuLong/MomentumDeltaNet .

View arXiv pageView PDFGitHub2Add to collection

Get this paper in your agent:

hf papers read 2605\.05838

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.05838 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.05838 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.05838 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Dynamic Linear Attention

arXiv cs.CL

This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.

Retrofitting Linear Attention into Diffusion Language Models

arXiv cs.LG

This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.

Dynamic Linear Attention

Hugging Face Daily Papers

DLA introduces adaptive state merging and capacity-bounded memory modeling for multi-state linear attention, improving long-context LLM performance.

You Could Have Come Up with Kimi Delta Attention

Hacker News Top

This blog post derives Kimi Delta Attention step by step from standard softmax attention through linear attention and DeltaNet variants, explaining the state update equations used by recent Qwen and Kimi models.