Tag
The paper introduces Layer-wise Error Modeling Search (LEMS) and KFAC-SVD to improve low-rank SVD compression of LLMs, refining global rank allocation and decomposition quality to deliver up to 4.8 p.p. zero-shot accuracy gains and 15% average perplexity improvements across Mistral, Qwen3, and Llama 3 families.
提出 Squelch——一种无路由、无新增可训练参数的 rank-one LoRA 专家混合方法，通过噪声底限软阈值和 Stein 无偏风险估计为每个 token 自适应选择激活秩，并在常识推理、知识、数学、代码和 GLUE 基准上显著优于 LoRA 与最强混合基线。
This study empirically demonstrates that gradient-based LoRA rank allocation, effective in supervised fine-tuning, degrades performance in GRPO-based reinforcement learning due to flatter gradient landscapes and a gradient amplification effect.