Tag
An educational post breaking down 5 parameter-efficient fine-tuning (PEFT) techniques for LLMs — LoRA, LoRA-FA, VeRA, Delta-LoRA, and LoRA+ — with visual explanations of how each adapts low-rank matrices to reduce memory and training costs.
该论文发现，在零初始化 LoRA 的 B 矩阵限制的参数池中，密度匹配的随机掩码在基于损失的遗忘指标上持续优于梯度归因选择，质疑了梯度归因额外反向传播开销的必要性。
The Itgan team describes a unified Whisper+LoRA recipe adapted for the three NADI 2026 Arabic ASR subtasks, showing that per-dialect specialists, base-model choice, and training-free weight averaging each dominate for different subtasks rather than any single novel component.
PaLoRA introduces a paced low-rank adaptation method for LoRA-based continual learning, deriving an optimal adaptive pacing law that scales gradient step magnitude with the effective rank of accumulated knowledge. The method combines adaptive SVD truncation, null-space gradient projection, and rank-aware pacing, achieving consistent gains including 4% accuracy improvements on 50-task ImageNet-A and ImageNet-R benchmarks.
This paper introduces an adaptive phase-switching method for communication-efficient federated LoRA fine-tuning, achieving up to 40.5% round-trip savings in communication costs while maintaining model performance on large language models.