Tag
Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.
This paper introduces MARCH, a network architecture that scales recurrent state-space models beyond fixed-size dimensions by caching cumulative recurrent-state checkpoints as content-addressable state anchors, enabling efficient long-range memory retrieval that outperforms linear attention variants on long-context benchmarks.
Introduces Maglev, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. It uses a prefiller and decoder with a memory consistency loss, improving validation loss and downstream benchmarks over baselines.
Introduces Semidirect Fourier Delta Attention (SFDA), a phase-controlled delta-rule layer that extends Kimi Delta Attention with block-rotational Fourier control operators, providing a constructive chunk-WY theorem for efficient chunkwise computation and demonstrating expressivity for cyclic and register memories.
This paper compares softmax attention with four linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) and introduces cross-layer routing mechanisms. Experiments at 350M parameters show Kimi Delta Attention with Muon achieves lowest validation loss, while pure Gated DeltaNet with AdamW has highest throughput.
This paper identifies memory retention as the bottleneck in recurrent memory agents for long contexts and proposes Multi-Head Recurrent Memory (MHM), a training-free framework that partitions memory into independent heads with a select-then-update strategy. The lightweight instantiation MHM-LRU significantly improves retention and end-to-end accuracy across 100K–1M token ranges, raising retention from below 30% to 73.96% on RULER-HQA at 896K tokens.
This paper presents preliminary findings from Hierarchos, a 232-million parameter recurrent memory-augmented assistant model, exploring its capabilities in memory retention and assistance tasks.