Fast Weight Attention for Continual Learning
Summary
This paper analyzes recurrent fast-weight memories and selective state-space models as online learning rules, deriving normalized update families that improve length extrapolation and remain competitive in language modeling.
View Cached Full Text
Cached at: 09/01/26, 12:08 PM
Paper page - Fast Weight Attention for Continual Learning
Source: https://huggingface.co/papers/2608.27763 Authors:
,
,
,
,
,
,
,
,
,
Abstract
Recurrent fast-weight memories and selective state-space models are analyzed as online learning rules under autoregressive semantics, yielding normalized update families with stable renormalization that improve length extrapolation.
Recurrentfast-weight memoriesandselective state-space modelscompress an expanding context into a fixed-sizerecurrent state, making the state transition anonline learning rule. We study this rule underread-after-write autoregressive semantics. For theprefix-predictionobjective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_{t-1}),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derivenormalized first-order updatesforsquared-error regressionandnegative inner-productobjectives. The regression family comprisesFalcon-1(a scalar NLMS update),Falcon-2(its per-column extension), andFalcon-3(a sliding-window mini-batch update);Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent,masked-parallel, andchunk-parallelforms, together with numerically stablepositive-decay renormalization. Representative variants remain competitive in language modeling and improvelength extrapolationon variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.
View arXiv pageView PDFProject pageGitHub15Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.27763 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.27763 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.27763 in a Space README.md to link it from this page.
Collections including this paper6
Similar Articles
Can a Language Model Learn Facts Continually in Its Weights?
This paper investigates whether language models can learn new facts in their weights through continual learning. Using invented facts and sequential writes into Qwen3 models, it finds that training data breadth determines knowledge type and retention: bare-statement facts are quickly forgotten (1% accuracy after 20 writes), while facts learned from diverse restatements retain 46% accuracy. Forgotten facts are not erased but become behaviorally inaccessible due to later writes redirecting questions, and context remains the reliable channel for fact composition and survival.
Δ-Mem: Efficient Online Memory for Large Language Models
Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.
Dynamic Linear Attention
This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.
Continual Learning Mechanisms Compose for Long-Horizon Memorization
This paper shows that combining complementary continual learning mechanisms enhances long-horizon memorization in language models, boosting retention by 28-fold through data, function, and weight anchors with merged LoRA.
Context Memorization for Efficient Long Context Generation
Proposes attention-state memory, a training-free approach that stores precomputed attention states in lightweight memory to improve accuracy and reduce latency for long prefix inference, outperforming traditional methods on benchmarks.