ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Summary
ORBIT proposes a method to mitigate catastrophic forgetting in large language models fine-tuned for generative retrieval by tracking parameter distances and using weight averaging, outperforming common continual learning baselines.
View Cached Full Text
Cached at: 05/13/26, 04:13 PM
Paper page - ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging
Source: https://huggingface.co/papers/2605.12419
Abstract
ORBIT addresses catastrophic forgetting in large language model fine-tuning for generative retrieval by tracking parameter distances and employing weight averaging to maintain model performance.
Despite the rapid advancements inlarge language model(LLM) development,fine-tuningthem for specific tasks often results in thecatastrophic forgettingof their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of theGenerative Retrieval(GenRetrieval) task. During GenRetrievalfine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses aweight averagingstrategy to constrainmodel driftduring GenRetrievalfine-tuningwhen this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both commoncontinual learningbaselines and related regularization methods that also employweight averaging.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2605\.12419
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.12419 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.12419 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.12419 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Continual Learning Mechanisms Compose for Long-Horizon Memorization
This paper shows that combining complementary continual learning mechanisms enhances long-horizon memorization in language models, boosting retention by 28-fold through data, function, and weight anchors with merged LoRA.
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
ReCoLoRA is a spectrum-aware framework for continual fine-tuning of large language models that uses recursive consolidation of low-rank adapters to prevent catastrophic forgetting, achieving improved performance on continual GLUE tasks across multiple backbones.
The Attribution Blind Spot: Detecting When Language Models Rely on Memory Rather Than Retrieved Context
Proposes Computational Reality Monitoring to detect when language models rely on pretraining memory rather than retrieved context, addressing the attribution blind spot in retrieval-augmented generation.
The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
This paper identifies and formalizes 'recorruption' in multimodal RAG, where adding accurate context causes models to abandon correct predictions due to attentional collapse (visual blindness and positional bias). The authors propose BAIR, a parameter-free inference-time framework that restores visual saliency and penalizes textual distractors, improving reliability across medical, fairness, and geospatial benchmarks.
Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
This paper studies catastrophic forgetting in multilingual expert language models during continual pretraining and proposes five parameter alignment strategies (hard layer freezing, soft regularization, post-hoc weight reversion, and model merging) to mitigate forgetting across 32 training languages with minimal cost to language acquisition.