forgetting

A UC Berkeley study finds agents that periodically drop their history and restart from their best result outperform parallel agents by up to 15%; the piece also cites Microsoft's Fork-and-Flush and Drew Breunig's 'context inertia' to frame the open question of what state should persist across a context reset.

H2DF introduces a calibrated target-band objective for language-model unlearning that limits gradient-ascent pressure once forget loss reaches a data-dependent target range, matching GradDiff performance on TOFU while using far less retain replay and improving retention over GA and NPO on MUSE-News.

提出 First Passage Attention (FPA)，将注意力遗忘机制从路径求和改为路径极值，包含 Forgetting Transformer 作为特例，并在 110M 语言模型上实现 8 倍训练长度的完美 passkey 检索。

这篇论文提出"relearning savings"评估协议，发现在Qwen2.5-1.5B中，曾学习过的事实绑定在部分遗忘后比从未学习的绑定更容易被重新习得，恢复所需优化步数减少约18%，说明当前行为准确率会低估既有经验的持久影响。

UC Berkeley researchers show that scaling LLM research agents stalls when agents anchor on their own history, and propose periodically discarding session state and restarting from the best candidate — a 'forgetting' approach that yields up to 15% gains over parallel agents and 39% over a single agent at the same budget.

该论文研究了On-Policy Self-Distillation（OPSD）在LLM自我改进过程中导致已有能力退化的问题，发现这种退化与隐藏表示几何结构恶化（簇内紧凑性和簇间分离度下降）相关，并提出表示几何正则化（RGR）方法，在不改变原有优化流程的情况下有效缓解能力退化。

该论文系统比较了六种参数高效微调（PEFT）方法在语言和扩散模型中的表现，发现LoRA家族方法也近似保持预训练几何结构，质疑了显式几何保持的必要性，并指出LoRA在抑制遗忘方面最稳定、DoRA在多数对比中取得更高任务得分。

本文研究流式说话人分离模型（Streaming Sortformer v2）在小数据适配时出现的'看似提升实则遗忘'现象，揭示聚合指标（DER）掩盖的身份一致性退化，并探讨rehearsal在恢复与跨域迁移之间的权衡。

该论文研究语言模型的自我保存能力，通过构建任务库（Task Banks）引导冻结模型生成回放数据，在Qwen系列模型上实验表明生成式回放能减少持续学习中的遗忘，且模型在部分场景下可自主选择回放目标以保护自身能力。

This paper derives a unified token- and layer-wise decomposition of how learning from one token changes another prediction, connecting data attribution, catastrophic forgetting, and plasticity loss as distinct regimes of the same evolving update-behavior interaction. It yields forward-computable approximations enabling data selection, interference controls, and diagnostics for future learnability across models and training regimes.

This paper presents a benchmark and a multi-agent system to improve scientific diagram generation through multi-turn human feedback, addressing issues like quality drift and forgetting across revisions.

The article compares discrete and continuous memory adapters for frozen language models, finding that binary-mask methods like EPMem forget facts as quickly as continuous ones, emphasizing that the write/allocation rule, not discreteness, is crucial for preventing forgetting.

ForgetBench introduces a benchmark to systematically characterize forgetting behavior in large language models under continual knowledge editing, using concept-based and scenario-based QA to measure temporal decay and retention dynamics.

The article argues that for effective agent memory, it is crucial to forget bad examples, as retaining them degrades performance.

Proposes GSUO, a guidance-signal-aware optimization framework for machine unlearning that uses fine-grained signals to guide the forgetting process, avoiding over-unlearning and under-unlearning, and outperforms 14 baselines.

The paper introduces MoFO, a momentum-filtered optimizer that mitigates forgetting in LLM fine-tuning by updating only parameters with large momentum magnitudes, preserving pre-trained knowledge without extra storage.

Stanford University proposes the AutoMem method, which allows models to learn memory management (selective forgetting) instead of expanding parameters. This doubles the performance of a 32-billion-parameter small model and matches top-tier large models, revealing that memory management is more important than model scale.

This paper proposes a causal auditing framework to evaluate forgetting in Limited Memory Language Models by varying the database state during inference, discovering that parametric leakage is negligible and post-deletion correctness primarily arises from retrieval artifacts rather than residual parametric memory.

The paper introduces Fora (Function-space Orthogonal Residual Adaptation), a method to protect existing capabilities during fine-tuning by projecting updates onto function-space directions derived from activations rather than weight-space directions. Experiments on Qwen3-1.7B show it outperforms weight-space projection and standard regularization in preserving translation and math abilities.

This paper introduces ReRULE, an off-policy replay method for reinforcement unlearning in LLMs, improving forgetting and retention efficiency on benchmarks like RWKU and MUSE.