recurrent-attention

Tag

Cards List
#recurrent-attention

Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting

arXiv cs.LG ↗ · 2d ago Cached

Kernel Ridge Attention (KRA) is a recurrent sequence mixer that uses recursive ridge regression with covariance-guided memory updates and an Ornstein-Uhlenbeck transition for selective forgetting, achieving constant-size inference state and improved language-modeling performance over attention and recurrent baselines.

0 favorites 0 likes
#recurrent-attention

Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity

Hugging Face Daily Papers ↗ · 2026-05-27 Cached

This paper explores how an exponentially decaying memory module from RAT+ can improve query-aware sparse inference methods for long-context language models, demonstrating consistent accuracy gains across various sparse budgets on needle-in-a-haystack tasks.

0 favorites 0 likes
← Back to home

Submit Feedback