Tag
A plug-and-play repo implementing self-distillation with an EMA teacher that scales to large models by combining LoRA adapters with DeepSpeed ZeRO-3, avoiding the memory and latency overhead of holding separate full teacher and student models. Tested on Qwen3.5 and ToolAlpaca, it includes adapter switching, EMA checkpointing, and memory-optimized DeepSpeed configs.
A paper analyzing LoRA's sensitivity to learning-rate choices and proposing Init[A²], a new initialization scheme with variance scaling that suppresses initialization-induced noise, stabilizing asymmetric LoRA+ training and widening the stable learning-rate region for LLM fine-tuning.
该论文提出 MoPE-LoRA，一种基于音素类别（发音方式）而非口音身份组织低秩专家的参数高效适配框架，用于带口音的英语语音识别；在 L2-ARCTIC 上零样本口音评估中相对单 LoRA 提升 12.3%。
An educational post breaking down 5 parameter-efficient fine-tuning (PEFT) techniques for LLMs — LoRA, LoRA-FA, VeRA, Delta-LoRA, and LoRA+ — with visual explanations of how each adapts low-rank matrices to reduce memory and training costs.
ToPA 提出了一种基于分块Toeplitz矩阵的参数高效微调方法，避免了低秩和稀疏结构的限制，在21个NLP和CV数据集、5种模型架构上验证了其表达力与训练稳定性。
该论文提出使用参数高效微调（PEFT/低秩适配器）来适配超大多模态骨干网络进行脑编码，Brain LoRA 在 Algonauts 2025 挑战赛中取得最佳公开成绩，并通过 multi-LoRA kernel 支持 200+ 模型集成。
该论文系统比较了六种参数高效微调（PEFT）方法在语言和扩散模型中的表现，发现LoRA家族方法也近似保持预训练几何结构，质疑了显式几何保持的必要性，并指出LoRA在抑制遗忘方面最稳定、DoRA在多数对比中取得更高任务得分。
This paper proposes a backdoor purification method for LoRA-tuned LLMs that projects LoRA updates onto orthogonal null spaces without needing trigger knowledge, clean references, or retraining, cutting attack success rates from ~100% to under 10% while preserving base model and adapter performance.
Follow-up update on a native Rust + Vulkan Transformer training backend: 14 architectures now pass strict parity-verified full-training gates against a pinned Hugging Face Transformers oracle, and full PEFT workflows (LoRA, saved modules, adapter switching, exact resume) are verified across 32 architecture surfaces.
The paper proposes FrameFT, a parameter-efficient fine-tuning method using sparse coefficients in a Fusion Frame basis, reducing memory footprint while achieving performance on par with or exceeding state-of-the-art PEFT techniques.
The paper evaluates Large Language Models for hate speech detection in Roman Urdu, a low-resource language, demonstrating that Parameter-Efficient Fine-Tuning with LoRA significantly improves classification performance compared to zero-shot inference.
Tencent researchers propose YOLO-PEFT, a structure-aware framework that treats adapter placement as constraint planning, outperforming full fine-tuning on YOLO detectors while using significantly fewer trainable parameters.
The paper compares five parameter-efficient fine-tuning methods on four small language models for on-device personalization, finding LoRA+ best for energy efficiency and QLoRA best for memory-limited deployment.
This paper proposes a conceptual framework for building an electronic explanatory dictionary of the Tajik language using large language models, integrating morphological analysis, lemmatization, semantic clustering, and dictionary entry generation with PEFT strategies.
FedWeave proposes asymmetric aggregation for federated MoE-LoRA to handle task heterogeneity by separating expert aggregation from router optimization, achieving better specialization and performance.
This paper studies PEFT-BD, a speculative decoding method using a LoRA-like adapter as a block-diffusion drafter, and finds that despite nontrivial accepted prefixes, it does not yield speedup because the drafter still requires a full-backbone pass, making it not compute-efficient.
This paper introduces Super and Supra, sparse parameter-efficient fine-tuning methods that reuse pruning-inspired saliency signals (like Wanda scores) to select trainable supports, and combine them with LoRA to achieve high accuracy on arithmetic tasks with reduced memory and compute.
MiCA (Minor Component Adaptation), a new fine-tuning method that initializes adapters in the minor singular subspace for better knowledge uptake and less forgetting, has been merged into the Hugging Face PEFT library. It is available via the PEFT main branch and integrates through the existing LoRA interface with init_lora_weights='mica'.
This paper investigates the effectiveness of top-1 collapse rate as a stability monitor for short-horizon LoRA fine-tuning of discrete diffusion language models, finding it has zero precision, and proposes max gradient norm as a more reliable alternative with higher precision and F1 score on LLaDA-family models.
Proposes ARIADNE, a training-free, adapter-agnostic routing framework that selects the optimal PEFT adapter at inference time by measuring input proximity to adapter-specific centroids in embedding space, recovering 97.44% of upper-bound performance on 23 tasks.