Pyrecall open source tool for detecting catastrophic forgetting during LLM fine-tuning[P]
Summary
Pyrecall is a new open-source tool that detects catastrophic forgetting during LLM fine-tuning by snapshotting skill scores before and after training, flagging regressions, and rolling back LoRA adapters. It runs fully locally with no external APIs.
Similar Articles
A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use
This paper introduces PRISMS, a framework that uses a small set of failure-specific MLP neurons to detect and steer LLM tool-use errors (over-calling, missing calls, invalid arguments) with sparse readouts, improving reliability across multiple model families.
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
ReCoLoRA is a spectrum-aware framework for continual fine-tuning of large language models that uses recursive consolidation of low-rank adapters to prevent catastrophic forgetting, achieving improved performance on continual GLUE tasks across multiple backbones.
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
This paper introduces Retention-aware Policy Optimization (RaPO) to mitigate catastrophic forgetting in visual continual learning using reinforcement fine-tuning. RaPO uses trajectory-level reward shaping and cross-task advantage normalization to close the gap between reinforcement and supervised fine-tuning in class- and domain-incremental learning.
From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
Researchers introduce Memora, a benchmark that evaluates LLMs’ ability to retain, update, and forget long-term user memories over weeks-to-months conversations, revealing frequent reuse of obsolete memories.
Reflective Recovery: A Self-Supervised Method for Reasoning by Learning from Mistakes
The paper introduces Reflective Recovery, a self-supervised method that enhances LLM reasoning by transforming failed trajectories into training data, breaking scaling collapse and enabling emergent self-correction.