Tag
A commentary exploring how much of today's agent memory stack — retrieval, summarization, context compression — is merely a workaround for stateless model architectures, and how recurrent-memory designs like BDH-CQ could shift that responsibility into the model itself.
A Pebble ML paper presents a small DeltaNet variant with fixed token-specific key biases that learns 32 key–value pairings per sequence in a fixed 32 KiB recurrent matrix state, achieving 99.95% recall accuracy even with long filler contexts, while parameter-matched vector and Transformer baselines fail to learn the task.
This paper characterizes the minimal recurrent behavioral memory required for imitating an expert under partial observability, using information-theoretic measures and experimental validation.
The paper introduces LatentPort, a method for transferring persistent recurrent state from a 4B to a 9B hybrid language model without the receiver re-reading context, demonstrating improved performance with minimal correction and near-native continuation.
MemBodied introduces a fixed-size episodic memory with associative state and episode anchor to enhance vision-language-action models for history-dependent manipulation tasks, achieving significant performance improvements over baselines.
TinyCeNN-LM proposes a quality-gated post-training conversion framework using CeNN-inspired cellular-recurrent layers to replace attention in pretrained language models, balancing efficiency with performance preservation.
Introduces Kalman Delta Networks, which improve language modeling by reformulating linear attention as a linear-Gaussian state-space model with Kalman-filter updates to track memory uncertainty, yielding efficient approximations that outperform existing linear-attention models.
This paper introduces MARCH, a network architecture that scales recurrent state-space models beyond fixed-size dimensions by caching cumulative recurrent-state checkpoints as content-addressable state anchors, enabling efficient long-range memory retrieval that outperforms linear attention variants on long-context benchmarks.
Introduces Maglev, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. It uses a prefiller and decoder with a memory consistency loss, improving validation loss and downstream benchmarks over baselines.
Introduces Semidirect Fourier Delta Attention (SFDA), a phase-controlled delta-rule layer that extends Kimi Delta Attention with block-rotational Fourier control operators, providing a constructive chunk-WY theorem for efficient chunkwise computation and demonstrating expressivity for cyclic and register memories.
This paper compares softmax attention with four linear attention architectures (DeltaNet, Gated DeltaNet, Kimi Delta Attention, Gated DeltaNet-2) and introduces cross-layer routing mechanisms. Experiments at 350M parameters show Kimi Delta Attention with Muon achieves lowest validation loss, while pure Gated DeltaNet with AdamW has highest throughput.
This paper identifies memory retention as the bottleneck in recurrent memory agents for long contexts and proposes Multi-Head Recurrent Memory (MHM), a training-free framework that partitions memory into independent heads with a select-then-update strategy. The lightweight instantiation MHM-LRU significantly improves retention and end-to-end accuracy across 100K–1M token ranges, raising retention from below 30% to 73.96% on RULER-HQA at 896K tokens.
This paper presents preliminary findings from Hierarchos, a 232-million parameter recurrent memory-augmented assistant model, exploring its capabilities in memory retention and assistance tasks.