Continual Learning Mechanisms Compose for Long-Horizon Memorization

Hugging Face Daily Papers Papers
continual-learning language-models catastrophic-forgetting long-horizon-memorization low-rank-adaptation merging-techniques

Summary

This paper shows that combining complementary continual learning mechanisms enhances long-horizon memorization in language models, boosting retention by 28-fold through data, function, and weight anchors with merged LoRA.

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introduce long-horizon memorization, a setting in which a model learns 100 query-answer tasks through continual supervised fine-tuning without retaining earlier training examples or receiving task identifiers at inference. Sequential updates cause catastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, and weight anchors specify what prior information each update should preserve, while low-rank allocation rules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-level successive halving to search the combinatorial design space and use a factorial experiment to measure individual and interaction effects. Our best method combines all three anchors with merged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. The data anchor and merged LoRA provide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improves long-horizon memorization beyond what any individual mechanism achieves.
Original Article
View Cached Full Text

Cached at: 09/16/26, 02:45 AM

Paper page - Continual Learning Mechanisms Compose for Long-Horizon Memorization

Source: https://huggingface.co/papers/2609.06986

Abstract

Composing complementary continual learning mechanisms substantially improves long-horizon memorization in language models by combining data, function, and weight anchors with merged low-rank updates.

Language models may need to internalize information that arrives over time and retain it through many subsequent updates. To study this challenge, we introducelong-horizon memorization, a setting in which a model learns 100 query-answer tasks throughcontinual supervised fine-tuningwithout retaining earlier training examples or receiving task identifiers at inference. Sequential updates causecatastrophic forgetting, and no single continual learning mechanism we evaluate maintains strong retention at this horizon. We hypothesize that mechanisms addressing complementary sources of forgetting will be more effective when composed. We organize these compositions along two design dimensions. Data, function, andweight anchors specify what prior information each update should preserve, whilelow-rank allocationrules determine where successive updates are retained. To test this hypothesis systematically, we construct three distinct 100-task memorization datasets. We introduce task-levelsuccessive halvingto search the combinatorial design space and use afactorial experimentto measure individual and interaction effects. Our best method combines all three anchors withmerged LoRA, ranks among the top 3 methods in all datasets, and raises average final retention from 1.2% under naive sequential fine-tuning to 34.9%, a 28-fold improvement. Thedata anchorandmerged LoRAprovide the largest average gains and interact super-additively on all three datasets. Together, these results show that composing complementary mechanisms substantially improveslong-horizon memorizationbeyond what any individual mechanism achieves.

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2609\.06986

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.06986 in a model README.md to link it from this page.

Datasets citing this paper1

#### cozzyde/long-horizon-memorization Preview• Updatedabout 1 hour ago • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.06986 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles