catastrophic-forgetting

HyperRepair is a framework that recovers the general capabilities of fine-tuned language models while retaining domain specialization at inference cost only, using a hypernetwork that generates a Repairing Side Net from activation fingerprints. It is evaluated across two model families, 12 LoRA adaptation datasets, and 15 general benchmarks, achieving 93.4% of base models' general score while keeping 99.5% of fine-tuned domain performance.

This paper studies catastrophic forgetting in the scientific foundation model Walrus-1.3B during two-stage sim-to-real adaptation, finding that finetuning on simulated flow data can increase one-step errors up to 35x, and shows that replaying just 1% of simulated data recovers most lost performance while retaining adaptation to experimental dynamics.

An empirical study finds that continual harness self-evolution of coding agents across multiple software engineering tasks causes severe catastrophic forgetting, with SWE-Bench performance dropping from 94.0% to 32.0% after evolving on FeatureBench, and offers insights for future continual agent-harness evolution.

Google Research introduces a 'sleep framework' for LLMs that alternates active wake phases with offline sleep phases to consolidate short-term context knowledge into stable long-term parameters, combating catastrophic forgetting and enabling retention of up to 10M tokens of long-context knowledge.

提出REDAR：将类增量学习中的正则化强度选择建模为序列决策问题，用轻量级RL智能体根据当前性能与数据统计动态输出每个阶段的知识蒸馏系数，在CIFAR-100和Tiny-ImageNet上降低了遗忘并提升了前向迁移。

论文提出 Skill Schema Transport (SST) 框架，通过将新领域知识重铸为模型已有技能空间中的可复用程序化 schema 来缓解持续领域适应中的灾难性遗忘，在两个 3B 模型上将 Qwen MATH 流的最大遗忘从 42.41 降至 7.10 个点。

论文提出 LSA-PAST，一种面向组合式增量学习（composition-IL）的参数高效方法，通过在冻结 ViT 中注入深度特化的 LoRA 适配器并用无复习的仿射变换将旧原型映射到新适配器空间，实现了精度-遗忘权衡的 state-of-the-art 表现。

该论文提出 SAGA 方法，将任务边界处训练的稀疏自编码器转换为对表示漂移的低秩局部约束，以缓解持续学习中的灾难性遗忘，并在三个 ViT-B/16 基准上取得最佳的 Class-IL 准确率与最低遗忘率。

A New York University, Google Research, and Cornell paper studies whether language models can support their own self-preservation during continual learning by generating replay data and selecting which prior tasks to replay, showing that task-bank-informed self-replay reduces forgetting on held-out tasks in Qwen models.

该论文发现，在零初始化 LoRA 的 B 矩阵限制的参数池中，密度匹配的随机掩码在基于损失的遗忘指标上持续优于梯度归因选择，质疑了梯度归因额外反向传播开销的必要性。

The paper proposes SFT-as-context, a two-pass inference method where the parent model uses the fine-tuned model's response as context, mitigating catastrophic forgetting of general capabilities after supervised fine-tuning while preserving task-specific improvements.

本文提出 HCPN-GCN，通过用图卷积网络替换线性特征提取器并引入锥形原型与多样性正则化，将分层原型网络扩展到持续图学习场景，在六个基准上以约 30 倍更少的原子原型实现更高准确率和接近零的遗忘。

该论文定位了LLM持续预训练/微调中灾难性遗忘的根源——语料库中罕见词元的输出嵌入，并提出仅提升输出投影层Adam ε值的单行优化器改动，在160M至12B参数规模上消除了39.4%至67.9%的遗忘且不损害目标学习。

An empirical study of adapting adult ASR models to child speech across Arabic and English while retaining adult performance, comparing fine-tuning, LoRA, and weight-space merging (LERP/TIES), with findings that bilingual adaptation and merging improve the adaptation-retention trade-off.

The paper proposes CLEAN, an incremental cognitive diagnosis framework that uses architectural isolation—freezing historical diagnostic functions with orthogonal column masking and adding expandable full-rank branches—to achieve zero representation drift and psychometric consistency when the concept space expands.

PaLoRA introduces a paced low-rank adaptation method for LoRA-based continual learning, deriving an optimal adaptive pacing law that scales gradient step magnitude with the effective rank of accumulated knowledge. The method combines adaptive SVD truncation, null-space gradient projection, and rank-aware pacing, achieving consistent gains including 4% accuracy improvements on 50-task ImageNet-A and ImageNet-R benchmarks.

The paper introduces Continually-Recyclable Unit-Gating (CRUG), a parameter-isolation continual learning method for dynamical systems reconstruction that uses L0-regularized differentiable gates to allocate and recycle units across tasks, achieving zero forgetting and a strong reconstruction-capacity trade-off on nonlinear and chaotic systems.

论文提出 Local Support Learning (LSL)，一种在无需访问历史数据的情况下缓解大模型灾难性遗忘的框架，通过 GMM 门控函数使权重适配器只在自身训练分布的输入上激活，从而在最多 7B 参数的 LLM 上同时保留预训练与微调能力。

A new paper proposes TMLN (Trajectory-Modulatory Landscape Navigation), which treats continual learning as an optimal control problem over a curved loss landscape, using a diagonal empirical Fisher Information Matrix and trajectory-based gradient preconditioning to mitigate catastrophic forgetting without additive regularization penalties.

This paper introduces EoupCT, a novel framework that estimates and orthogonalizes unknown pre-training gradients to mitigate catastrophic forgetting in continual fine-tuning of large language models.