@BlinkDL_AI: Gated DeltaNet-2 is almost exactly RWKV-7's DPLR recurrence, not acknowledging the elephant in the room
Summary
Ali Hatamizadeh announces Gated DeltaNet-2, a new linear attention model that outperforms KDA and Mamba-3 at 1.3B scale; @BlinkDL_AI notes its recurrence is nearly identical to RWKV-7's DPLR.
View Cached Full Text
Cached at: 05/23/26, 12:05 PM
Gated DeltaNet-2 is almost exactly RWKV-7’s DPLR recurrence, not acknowledging the elephant in the room 🙂
Ali Hatamizadeh (@ahatamiz1): Gated DeltaNet-2 is here. 🚀
🔥 New paper: Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Gated DeltaNet-2 outperforms KDA and Mamba-3, the latest and best recurrent architectures, head to head at 1.3B. 🏆
💡 Here’s the idea behind it:
Linear attention
Similar Articles
@BlinkDL_AI: RWKV-7 G1g is here: the world's best pure RNN LLM, and a competitive LLM in general. Try https://huggingface.co/spaces/…
BlinkDL announces RWKV-7 G1g, a pure RNN LLM that claims to be the best in its class and competitive with general LLMs, with high-speed inference on a single RTX 5090.
MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
The paper introduces Momentum DeltaNet (MDN), a linear attention model that uses stepwise momentum and parallel algorithms to improve training efficiency and performance over models like Mamba2.
On Subquadratic Architectures: From Applications to Principles
This paper compares xLSTM, Mamba-2, and Gated DeltaNet on complex sequence modeling tasks and finds xLSTM superior due to its enhanced state tracking and memory dynamics, validated on synthetic length-generalization tasks.
@jiqizhixin: New from NVIDIA! You can edit a model’s compressed memory without scrambling what it already knows! Enter Gated DeltaNe…
NVIDIA introduces Gated DeltaNet-2, a method for editing compressed model memory without catastrophic forgetting, using independent gates for erase and write operations. It outperforms existing models like Mamba-2 and Mamba-3 on language modeling and long-context tasks.
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
Introduces DualKV, a FlashAttention kernel variant that eliminates redundant prompt token computation in RL post-training (GRPO/DAPO), achieving up to 3.82x speedup on 30B MoE models.