Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
Summary
This paper introduces Retention-aware Policy Optimization (RaPO) to mitigate catastrophic forgetting in visual continual learning using reinforcement fine-tuning. RaPO uses trajectory-level reward shaping and cross-task advantage normalization to close the gap between reinforcement and supervised fine-tuning in class- and domain-incremental learning.
View Cached Full Text
Cached at: 05/20/26, 10:37 AM
Paper page - Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
Source: https://huggingface.co/papers/2605.09640
Abstract
Reinforcement Fine-Tuning suffers from catastrophic forgetting in visual continual learning, which is addressed through Retention-aware Policy Optimization that uses trajectory-level reward shaping and cross-task advantage normalization.
Recent studies suggest thatReinforcement Fine-Tuning(RFT) is inherently more resilient tocatastrophic forgettingthanSupervised Fine-Tuning(SFT). However, whether RFT (e.g.,GRPO) can effectively overcome forgetting in challenging visual continual learning settings, such asclass-incremental learning(CIL) anddomain-incremental learning(DIL), remains an open problem. Through a pilot study, we confirm that while RFT consistently outperforms SFT, it still suffers from non-negligible forgetting. We empirically trace this bottleneck toTrajectory-level Drift Agnosticism: among candidate rollouts achieving identical task rewards, the KL divergence from the preceding-task policy varies substantially, which strongly correlates withcatastrophic forgettingacross sequential tasks. Motivated by this insight, we proposeRetention-aware Policy Optimization(RaPO), a simple yet effective RFT method that explicitly mitigates forgetting throughtrajectory-level reward shaping. Specifically, RaPO comprises two core components: (1) Retention Reward that converts trajectory-level distribution drift into a continuous reward signal, preferentially reinforcing knowledge-preserving rollouts within each group; (2)Cross-Task Advantage Normalization(CTAN), which maintains a persistent exponential moving average of reward statistics across task boundaries to stabilize the optimization progress during continual learning. Leveraging the free-form textual generalization ofMLLMs, we comprehensively evaluate RaPO across five visual continual learning settings. Extensive experiments demonstrate that RaPO achieves leading performance, substantially reducingcatastrophic forgettingwhile preserving strong plasticity. To the best of our knowledge, this work represents the first systematic exploration of RFT in visual continual learning, offering insights that we hope will inspire future research.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.09640
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.09640 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.09640 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.09640 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
RAFT is a two-stage framework for domain-specific fine-tuning of LLMs that addresses catastrophic forgetting by refining supervision data and using on-policy distillation with adaptive loss balancing, achieving significant improvements on domain accuracy while recovering general capabilities.
Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards
Proposes Correction-Oriented Policy Optimization (CIPO), an extension to RLVR that converts failed trajectories into correction-oriented supervision, improving reasoning and correction performance in LLMs across math and code benchmarks.
Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?
This paper investigates the mechanistic origins of catastrophic forgetting in LLMs, finding that reinforcement learning preserves internal computational circuits better than supervised fine-tuning, resulting in less forgetting of prior capabilities.
Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation
This paper proposes Trajectory-Augmented Policy Optimization (TAPO), which constructs micro-reflective correction trajectories using the model's own correct and incorrect rollouts to improve reasoning in large language models, outperforming standard self-distillation methods on math benchmarks.
Attribution-Guided Continual Learning for Large Language Models
This paper proposes an attribution-guided continual fine-tuning framework for large language models that estimates task-specific parameter importance in Transformer layers and modulates gradients accordingly, mitigating catastrophic forgetting while maintaining performance on new tasks.