weight-averaging

ORBIT proposes a method to mitigate catastrophic forgetting in large language models fine-tuned for generative retrieval by tracking parameter distances and using weight averaging, outperforming common continual learning baselines.