low-rank-adapters

The paper proposes δ-Vision, a method to reduce computational overhead in multimodal language models by using lightweight low-rank adapters to reconstruct visual states efficiently without discarding visual tokens.

ReCoLoRA is a spectrum-aware framework for continual fine-tuning of large language models that uses recursive consolidation of low-rank adapters to prevent catastrophic forgetting, achieving improved performance on continual GLUE tasks across multiple backbones.

This paper introduces a method for analyzing and controlling language model personalities using the OCEAN framework, training low-rank adapters to manipulate traits, and demonstrates additive composition and safety implications.

This paper introduces OP-Mix, a data mixing algorithm that uses low-rank adapters trained on the current model to cheaply simulate candidate data mixtures, enabling efficient and unified data mixing across pretraining, continual midtraining, and continual instruction tuning. OP-Mix consistently finds near-optimal mixtures while using a fraction of the compute of baselines, improving pretraining perplexity by 6.3% and reducing compute by 66-95% in continual learning settings.

The paper proposes Slice, a gradient-surgery-based initialization for LoRA adapters in continual learning that reconciles conflicting gradients from current and past tasks to reduce catastrophic forgetting, achieving better stability-plasticity trade-offs.