On Subquadratic Architectures: From Applications to Principles
Summary
This paper compares xLSTM, Mamba-2, and Gated DeltaNet on complex sequence modeling tasks and finds xLSTM superior due to its enhanced state tracking and memory dynamics, validated on synthetic length-generalization tasks.
View Cached Full Text
Cached at: 06/11/26, 01:36 PM
Paper page - On Subquadratic Architectures: From Applications to Principles
Source: https://huggingface.co/papers/2606.12364
Abstract
xLSTM demonstrates superior performance in sequence modeling tasks compared to Mamba-2 and Gated DeltaNet due to enhanced state tracking and memory dynamics.
Transformersdominate modernsequence modeling, but their quadraticattentionincurs substantial computational cost.Subquadratic architecturesoffer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches:xLSTM,Mamba-2, andGated DeltaNet. We evaluate these models on tasks with complex dependencies: (1)code-model pre-training, (2)distillationof code models from large language models, and (3) pre-training oftime-series foundation models. Across these settings,xLSTMdelivers the strongest overall performance. To explainxLSTM’s advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing onstate trackingandmemory dynamics. Our results show thatxLSTMenables more flexible and stable memory correction via itsgating scheme. We corroborate these findings on controlled syntheticlength-generalization tasks. Overall, our findings indicate thatxLSTM’s gains on complex tasks stem from robuststate trackingand accumulation.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.12364
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.12364 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.12364 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.12364 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Unlocking Feature Learning in Gated Delta Networks at Scale
This paper derives scaling rules for Gated Delta Networks using Maximal Update Parametrization (μP), enabling zero-shot hyperparameter transfer across model widths for efficient sub-quadratic LLM architectures. Experiments confirm stable learning-rate transfer under both AdamW and SGD, whereas standard parametrization fails.
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.
@BlinkDL_AI: Gated DeltaNet-2 is almost exactly RWKV-7's DPLR recurrence, not acknowledging the elephant in the room
Ali Hatamizadeh announces Gated DeltaNet-2, a new linear attention model that outperforms KDA and Mamba-3 at 1.3B scale; @BlinkDL_AI notes its recurrence is nearly identical to RWKV-7's DPLR.
Looped State-Space Language Models with Adaptive Exit-State Selection
This paper explores looped (recurrent) state-space language models using Mamba and hybrid Mamba-Transformer backbones, showing they outperform non-looped baselines on reasoning tasks and remain competitive under iso-parameter and iso-FLOPs pretraining, with adaptive exit-state selection improving intermediate-depth performance.
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention [P]
Sebastian Raschka reviews recent innovations in LLM architectures focused on long-context efficiency, including KV sharing, compressed convolutional attention, and layer-wise attention budgeting from models like Gemma 4, ZAYA1, Laguna XS.2, and DeepSeek V4.