parameter-efficient-fine-tuning

SeFoRA is a proposed federated LoRA algorithm that uses sketch aggregation to handle heterogeneous client ranks and alleviate bilinear mismatch. It includes a rank-homogeneous variant with convergence guarantees and shows state-of-the-art performance on RoBERTa-Large fine-tuning.

YOLO-PEFT is a structure-aware framework that formulates adapter placement as constraint planning for parameter-efficient fine-tuning of YOLO detectors, achieving better mAP than full fine-tuning with reduced memory.

This paper introduces BBBP-GeoPEFT, a geometry-informed parameter-efficient fine-tuning framework for pre-trained molecular GNNs targeting blood-brain barrier permeability prediction, achieving competitive performance while updating only 10.1% of parameters.

This paper proposes MoEGen, a parameter-efficient fine-tuning framework that uses mixture-of-experts to generate instance-adaptive LoRA updates via expert codes and a lightweight hypernetwork, improving performance on commonsense reasoning benchmarks without storing separate adapters per expert.

LARA is a method for efficient adaptation that adds low-rank corrections to a frozen model's residual stream instead of modifying weights, matching LoRA's performance while enabling composable behaviors and inference-time steering.

MoE2-LoRA introduces a dual-channel Routing-Conditioned Projection and a global LoRA expert pool to enable MoE-style low-rank adaptation for fine-tuning MoE models, achieving state-of-the-art accuracy while retaining general capabilities.

CT-Merging proposes a method to merge LoRA adapters by estimating consensus directions from task subspace projectors and assigning task-level RMS coefficient scales, achieving superior performance on the DC-Merge CLIP adapter benchmark.

SOS-LoRA extends LoRA by decomposing the rank budget into static orthogonal low-rank experts with fixed multi-scale scaling, improving fine-tuning on reasoning, NLU, and math benchmarks without adding inference cost.

This paper evaluates nine open-weight small language models (135M to 3B parameters) on a structured benchmark and shows that parameter-efficient fine-tuning significantly improves accuracy, making them viable for local deployment in structured niche workloads.

CARE-LoRA proposes a compressed activation reconstruction framework to reduce memory consumption during LoRA fine-tuning by leveraging low-rank projections. It achieves competitive performance with reduced memory footprint.

SQuaD-SQL uses LLM-guided knowledge distillation to train small language models for Text-to-SQL, achieving 86.9% execution accuracy on WikiSQL while offering faster inference and lower memory usage.

ReCoLoRA is a spectrum-aware framework for continual fine-tuning of large language models that uses recursive consolidation of low-rank adapters to prevent catastrophic forgetting, achieving improved performance on continual GLUE tasks across multiple backbones.

Proposes FRAME, a mixture-of-experts adapter that uses learnable fractional-Fourier orders to interpolate between spatial and spectral domains, improving parameter-efficient fine-tuning performance on LLMs across multiple benchmarks.

Proposes Hard-Routed MoR-LoRA, a two-stage framework that composes frozen reasoning LoRA experts via hard top-1 routing, preserving expert behavior with fewer trainable parameters than soft-routing baselines.

Introduces Hankel Reduced order Model (HRM) adapter, an SSM-based residual module initialized via Balanced Truncation for parameter-efficient fine-tuning, outperforming LoRA on long-context tasks.

LoRA (low-rank adaptation) is the most popular parameter-efficient fine-tuning method for LLMs. This video introduces how LoRA and its variants (LoRA+, QLoRA, VeRA, DoRA) work.

This article delves into the principles of LoRA and its variants (QLoRA, VeRA, DoRA), explaining how low-rank decomposition reduces trainable parameters to enable efficient fine-tuning of large models.

Proposes ARIADNE, a training-free, adapter-agnostic routing framework that selects the optimal PEFT adapter at inference time by measuring input proximity to adapter-specific centroids in embedding space, recovering 97.44% of upper-bound performance on 23 tasks.

This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.

This paper proposes SDBN, a framework combining adversarial training with parameter-efficient fine-tuning to improve robustness of foundation models under noise and limited data, demonstrating substantial improvements in low-resource settings.