low-rank-adaptation

This paper proposes PALM, a low-rank adapter method to adapt financial language models to specific time points without full retraining, reducing look-ahead bias and computational costs in financial backtests.

This paper introduces ℓp-LoRA, a principled method for automatic rank allocation in low-rank adaptation using ℓp regularization, demonstrating competitive performance on NLP tasks.

The paper proposes ChainDoRA, a parameter-efficient fine-tuning method for large language models that combines Tensor-Train factorization with weight-decomposed low-rank adaptation, achieving a 90.62% reduction in trainable parameters and improved accuracy over LoRA and DoRA.

This paper presents a parameter-efficient adaptation framework using LoRA and contrastive learning to enhance vision transformers for automatic target recognition in synthetic aperture sonar imagery.

LARA is a research project and PyTorch library that enables modular, composable behaviors for frozen large language models using low-rank residual adapters, allowing efficient training and inference-time blending of multiple behaviors.

The paper proposes FLoKD, an adaptive knowledge-distillation framework for federated LoRA fine-tuning of LLMs over wireless networks, reducing communication overhead by 50-65% while maintaining competitive performance.

This paper introduces CLAW, a method that uses hypernetworks to generate low-rank adapters for world models, enabling efficient online adaptation in model-based reinforcement learning with limited test-time data.

This paper presents Iso-LoRA, an optimizer that enhances LoRA by promoting even energy distribution across singular directions through spectral descent on tangent-space perturbations, improving effective rank and downstream performance in language model adaptation.

LOCUS is a task-aware low-rank post-training method that reduces output token length in language models while maintaining preference alignment, achieving up to 39.84% reduction on Pythia-2.8B with minimal parameter updates.

This paper shows that combining complementary continual learning mechanisms enhances long-horizon memorization in language models, boosting retention by 28-fold through data, function, and weight anchors with merged LoRA.

TaRA is a training-aware initialization method for Low-Rank Adaptation (LoRA) that improves gradient fidelity, leading to better fine-tuning performance for large language models across various tasks.

The paper introduces Net Utility, a data-free metric for budget-aware LoRA merging that optimizes rank allocation across tasks, achieving +2.1% improvement on vision tasks and +2.2% on language tasks over uniform methods.

This paper introduces the Von-Neumann State-Space Transformer (VN-SST), a new model inspired by von-Neumann architecture that improves sample efficiency in neural decoding by using a low-rank instruction bank for token-specific operations, outperforming standard Transformers on benchmarks.

AQLoRA is a zero-search method that accelerates quantized LoRA fine-tuning by adaptively keeping layers with high NF4 reconstruction error in fp16, achieving up to 11% faster training with minimal accuracy loss.

This paper introduces LoRA-GA2, a fine-tuning algorithm that leverages multi-step gradient information to improve the performance of Low-Rank Adaptation for large language models, achieving better results on benchmarks while preserving efficiency.

CLEAR introduces a continuous latent adapter routing framework for LLM safety alignment, using a hidden-state gate to modulate safety adapters and improve robustness on HarmBench while preserving utility on benign inputs.

This paper investigates singular components in LoRA and proposes SCLoRA, a method that uses spectral clipping to improve task adaptation while reducing catastrophic forgetting of pre-trained knowledge.

SeFoRA is a proposed federated LoRA algorithm that uses sketch aggregation to handle heterogeneous client ranks and alleviate bilinear mismatch. It includes a rank-homogeneous variant with convergence guarantees and shows state-of-the-art performance on RoBERTa-Large fine-tuning.

This paper investigates why LLMs underperform in Arabic medical tasks, showing via mechanistic analysis that knowledge exists internally but fails to surface, then proposes TLoRA, a targeted low-rank adaptation method that outperforms full-network LoRA on medical QA and introduces a new Arabic clinical dialogue benchmark.

This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.