low-rank-adaptation

SeFoRA is a proposed federated LoRA algorithm that uses sketch aggregation to handle heterogeneous client ranks and alleviate bilinear mismatch. It includes a rank-homogeneous variant with convergence guarantees and shows state-of-the-art performance on RoBERTa-Large fine-tuning.

This paper investigates why LLMs underperform in Arabic medical tasks, showing via mechanistic analysis that knowledge exists internally but fails to surface, then proposes TLoRA, a targeted low-rank adaptation method that outperforms full-network LoRA on medical QA and introduces a new Arabic clinical dialogue benchmark.

This paper proposes Unified LoRA (ULoRA), a two-parameter family of preconditioned gradient initializations for low-rank adaptation, showing that existing LoRA initialization methods are points on a continuum. The authors demonstrate that a tuned ULoRA matches or exceeds full fine-tuning on GLUE tasks with RoBERTa-base and is competitive on GSM8K with LLaMA 2-7B, and introduce ULoRA-Auto for zero-search deployment.

MoLGE assigns dedicated expert modules to clusters of similar languages in a mixture-of-experts framework for large-scale multilingual ASR, achieving improvements across 495 languages with minimal parameter increase.

This paper presents the IHLC submission to the LT-EDI 2026 shared task, using LoRA fine-tuning for gender-neutral rewriting (Rank 3) and activation steering for counter-narrative generation (Rank 6), highlighting both promise and limitations.

TriSP introduces a tri-signal importance metric combining weight magnitude, activation norm, and gradient sensitivity for structured pruning of LLMs, achieving lowest perplexity and high throughput improvements on LLaMA-7B.

This paper sharpens the convergence analysis of LoRA, improving deterministic oracle complexity from exponential to O(epsilon^{-4}), and proposes stochastic variants LoRA-NSGDM and LoRA-STORM with improved oracle complexities of O(epsilon^{-8}) and O(epsilon^{-6}) respectively.

MoE2-LoRA introduces a dual-channel Routing-Conditioned Projection and a global LoRA expert pool to enable MoE-style low-rank adaptation for fine-tuning MoE models, achieving state-of-the-art accuracy while retaining general capabilities.

A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.

Proposes MaLoRA and MaRA, two adapter families that introduce selective state-space recurrence for token-level and context-level adaptation in frozen language models, achieving significant gains on multi-hop reasoning benchmarks like MuSiQue and 2WikiMultihopQA.

SOS-LoRA extends LoRA by decomposing the rank budget into static orthogonal low-rank experts with fixed multi-scale scaling, improving fine-tuning on reasoning, NLU, and math benchmarks without adding inference cost.

CARE-LoRA proposes a compressed activation reconstruction framework to reduce memory consumption during LoRA fine-tuning by leveraging low-rank projections. It achieves competitive performance with reduced memory footprint.

This paper introduces SRM-LoRA, a sub-Riemannian-inspired method that uses a sensitivity-based Riemannian metric to reduce LLM hallucination during low-rank adaptation. The metric reshapes backward gradients to suppress high-cost update directions, improving factual reliability on benchmarks like HaluEval-QA.

TabLoRA proposes a parameter-efficient neural ensemble method for large-scale tabular data by sharing a common backbone with predictor-specific low-rank adaptations, achieving competitive performance against GBDTs and deep learning baselines.

Proposes Stacked LoRA, a framework that decouples subject-invariant and subject-specific knowledge for adapting EEG foundation models to motor imagery decoding, achieving improved accuracy across multiple benchmarks.

LoRA (low-rank adaptation) is the most popular parameter-efficient fine-tuning method for LLMs. This video introduces how LoRA and its variants (LoRA+, QLoRA, VeRA, DoRA) work.

This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.

This paper reveals that the scaling factor α in LoRA optimization is more influential than the learning rate, and proposes LoRA-α, a framework that improves performance and simplifies hyperparameter search by restoring α to its principled regime.

This paper introduces DoRA-RBAC, a framework for composing LLM adapters, and tests whether geometry-aware merging improves multi-domain performance. Results show no consistent advantage over standard averaging, suggesting adapter interference is not primarily driven by parameter-space geometry.

This paper introduces Null-Space Constrained Response-Specified Unlearning (NSRU), a low-rank framework that uses orthogonal-projected LoRA updates confined to the null space of retain subspaces to perform controlled LLM unlearning while preserving benign capabilities.