Tag
GRASP proposes a method for multi-source transfer learning that sequentially merges source models into a single target model with constant O(1) memory usage, using gradient-based parameter alignment to avoid negative transfer. Experiments show it outperforms ensemble methods while being much more memory-efficient.
This paper reveals that the scaling factor α in LoRA optimization is more influential than the learning rate, and proposes LoRA-α, a framework that improves performance and simplifies hyperparameter search by restoring α to its principled regime.
ART (Art-based Reinforcement Training) enables parameter-efficient fine-tuning of frozen multimodal LLMs by optimizing raw visual input via gradient backpropagation, achieving performance comparable to LoRA while supporting pre-compiled computational graphs for high-throughput engines like vLLM.
This paper investigates sequential fine-tuning of LLaMA-3.1-8B for automated essay scoring using a curriculum aligned with discourse structure, showing improved coherence and performance compared to independent or randomized training.
The Aizpurua team at Multiverse Computing, Spain, proposes expanding pretrained large models with small quantum circuits. Adding just about 6,000 parameters to Llama 3.1 8B reduces perplexity by 1.4%, demonstrating the feasibility of quantum-circuit-assisted large model scaling.
Researchers from Adelaide University introduce LR-LoRA (Learnable Rank LoRA), a parameter-efficient fine-tuning method that dynamically learns the adapter rank for each transformer layer during training rather than using a fixed global rank. LR-LoRA achieves state-of-the-art performance on language understanding and commonsense reasoning benchmarks, outperforming fixed-rank LoRA baselines.
LoomVideo introduces a 5B-parameter unified architecture for video generation and editing that reduces computational overhead using novel conditioning mechanisms and multi-modal alignment, achieving competitive performance and faster inference.
SALSA introduces a lightweight adaptation method for speech-aware LLMs that learns layer-wise steering vectors via supervised objective, achieving significant improvements (up to 46.8% relative) on out-of-domain speech benchmarks, and shows that steering the encoder layers is more effective than modifying the LLM backbone.
CSULoRA is a post-hoc method for correcting trained LoRA adapters to preserve safety alignment while maintaining utility, using closest safe update estimation.
This paper introduces VisAnomReasoner, a parameter-efficient vision-language model fine-tuned on a novel benchmark (VisAnomBench) with natural-language rationales, achieving over 21pp improvement in precision and F1 for time-series anomaly detection and strong cross-benchmark generalization.
LoRDBA replaces LoRA's floating-point low-rank factors with binary sign carriers and channel-wise scales, enabling efficient on-device fine-tuning with significant footprint reduction and minimal latency overhead, matching fp16 quality.
This paper proposes MODIAD, a framework for multimodal online distributed industrial anomaly detection, addressing resource constraints with a Multi-class Intelligent Scheduling problem and a Resource Efficient Class-Wise Low Rank Adaptation (REC-LoRA) strategy. Experiments on MVTec 3D-AD and Eyecandies datasets demonstrate superior performance and efficiency.
FuRA introduces a full-rank parameter-efficient fine-tuning method using spectral preconditioning via block tensor-train decomposition, achieving higher accuracy than full fine-tuning with LoRA-level efficiency. It outperforms LoRA and full FT on LLM and VLM tasks.
P2D is a unified framework that leverages task-sensitive attention heads for both data selection and structural pruning, achieving an 8.3 pp performance gain and 7.0× speedup by updating only 10% of heads on 10% of data.
Explains five parameter-efficient fine-tuning techniques: LoRA, LoRA-FA, VeRA, Delta-LoRA, and LoRA+, detailing how each modifies model weights during adaptation.
HELLoRA introduces activation-aware adapter placement for MoE models, attaching LoRA only to hot experts to reduce parameters and FLOPs while improving performance on reasoning, code, and safety tasks.
This paper proposes a parameter-efficient vocabulary adaptation method for LLM-based text summarization in specialized domains, augmenting pretrained tokenizers with domain-specific tokens and selectively replacing under-trained ones to reduce training time by 35-55% and parameter counts by up to 37%.
Hugging Face's PEFT library enables parameter-efficient fine-tuning of large models on a single GPU, reducing compute and storage costs while maintaining performance.
PEML proposes a parameter-efficient multi-task learning method that co-optimizes continuous prompts and model weights via low-rank adaptation. It achieves up to 6.67% average accuracy improvement on multiple benchmarks.
This paper introduces three parameter-efficient methods for multi-view proficiency estimation on the Ego-Exo4D dataset, shifting from discriminative classification to generative feedback. The proposed models achieve state-of-the-art accuracy with significantly fewer parameters and training epochs than video-transformer baselines.