ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Summary
ORBIT proposes a method to mitigate catastrophic forgetting in large language models fine-tuned for generative retrieval by tracking parameter distances and using weight averaging, outperforming common continual learning baselines.
View Cached Full Text
Cached at: 05/13/26, 04:13 PM
Paper page - ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Source: https://huggingface.co/papers/2605.12419
Abstract
ORBIT addresses catastrophic forgetting in large language model fine-tuning for generative retrieval by tracking parameter distances and employing weight averaging to maintain model performance.
Despite the rapid advancements inlarge language model(LLM) development,fine-tuningthem for specific tasks often results in thecatastrophic forgettingof their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of theGenerative Retrieval(GenRetrieval) task. During GenRetrievalfine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses aweight averagingstrategy to constrainmodel driftduring GenRetrievalfine-tuningwhen this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both commoncontinual learningbaselines and related regularization methods that also employweight averaging.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.12419
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.12419 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.12419 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.12419 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context
Proposes Computational Reality Monitoring to detect when language models rely on pretraining memory rather than retrieved context, addressing the attribution blind spot in retrieval-augmented generation.
The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
This paper identifies and formalizes 'recorruption' in multimodal RAG, where adding accurate context causes models to abandon correct predictions due to attentional collapse (visual blindness and positional bias). The authors propose BAIR, a parameter-free inference-time framework that restores visual saliency and penalizes textual distractors, improving reliability across medical, fairness, and geospatial benchmarks.
Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
This paper studies catastrophic forgetting in multilingual expert language models during continual pretraining and proposes five parameter alignment strategies (hard layer freezing, soft regularization, post-hoc weight reversion, and model merging) to mitigate forgetting across 32 training languages with minimal cost to language acquisition.
Attribution-Guided Continual Learning for Large Language Models
This paper proposes an attribution-guided continual fine-tuning framework for large language models that estimates task-specific parameter importance in Transformer layers and modulates gradients accordingly, mitigating catastrophic forgetting while maintaining performance on new tasks.
Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models
This paper introduces Micro-Macro Retrieval (M2R), a retrieve-while-generate framework that reduces hallucination in long-form LLM outputs by ensuring key information stays close to generated text. It uses curriculum learning-based reinforcement learning to train retrieval and grounding skills, showing effectiveness especially in lengthy contexts.