forgetting

This paper presents a benchmark and a multi-agent system to improve scientific diagram generation through multi-turn human feedback, addressing issues like quality drift and forgetting across revisions.

The article compares discrete and continuous memory adapters for frozen language models, finding that binary-mask methods like EPMem forget facts as quickly as continuous ones, emphasizing that the write/allocation rule, not discreteness, is crucial for preventing forgetting.

ForgetBench introduces a benchmark to systematically characterize forgetting behavior in large language models under continual knowledge editing, using concept-based and scenario-based QA to measure temporal decay and retention dynamics.

The article argues that for effective agent memory, it is crucial to forget bad examples, as retaining them degrades performance.

Proposes GSUO, a guidance-signal-aware optimization framework for machine unlearning that uses fine-grained signals to guide the forgetting process, avoiding over-unlearning and under-unlearning, and outperforms 14 baselines.

The paper introduces MoFO, a momentum-filtered optimizer that mitigates forgetting in LLM fine-tuning by updating only parameters with large momentum magnitudes, preserving pre-trained knowledge without extra storage.

Stanford University proposes the AutoMem method, which allows models to learn memory management (selective forgetting) instead of expanding parameters. This doubles the performance of a 32-billion-parameter small model and matches top-tier large models, revealing that memory management is more important than model scale.

This paper proposes a causal auditing framework to evaluate forgetting in Limited Memory Language Models by varying the database state during inference, discovering that parametric leakage is negligible and post-deletion correctness primarily arises from retrieval artifacts rather than residual parametric memory.

The paper introduces Fora (Function-space Orthogonal Residual Adaptation), a method to protect existing capabilities during fine-tuning by projecting updates onto function-space directions derived from activations rather than weight-space directions. Experiments on Qwen3-1.7B show it outperforms weight-space projection and standard regularization in preserving translation and math abilities.

This paper introduces ReRULE, an off-policy replay method for reinforcement unlearning in LLMs, improving forgetting and retention efficiency on benchmarks like RWKU and MUSE.

Proposes a cognitively grounded multi-factor value function for agentic memory in LLM agents, learning interpretable weights to decide what to encode, forget, and retrieve under memory constraints. Improves gold-evidence retention significantly over similarity-only or recency-based baselines.

This blog post introduces the concept of the 'Forgetting Wall' in long-horizon video generation and world models, arguing that the primary bottleneck is memory (KV cache growth) rather than compute, and explores compression as a key direction for future models.

Proposes FoLoRA, a forgetting-aware optimization framework for fine-tuning foundation models that balances task utility and forgetting penalty via generalized Rayleigh-quotient optimization, achieving better preservation of non-target capabilities.

This paper introduces the concept of 'initialization memory' to study how much of the random initialization bias survives training in deep networks, showing that low-learning-rate SGD preserves initialization while Adam-family optimizers erase it, and linking this to forgetting dynamics.

Introduces SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving LLM memory that measures multiple dimensions beyond aggregate metrics, revealing trade-offs between adaptability and stability.

Researchers introduce Memora, a benchmark that evaluates LLMs’ ability to retain, update, and forget long-term user memories over weeks-to-months conversations, revealing frequent reuse of obsolete memories.