Tag
The paper introduces NSFT, a fine-grained parameter-efficient fine-tuning framework for MoE LLMs that refines adaptation from experts to sub-experts, demonstrating improved performance with fewer trainable parameters.
The paper proposes ChainDoRA, a parameter-efficient fine-tuning method for large language models that combines Tensor-Train factorization with weight-decomposed low-rank adaptation, achieving a 90.62% reduction in trainable parameters and improved accuracy over LoRA and DoRA.
This paper proposes a three-stage pipeline for accented conversational ASR that improves entity and disfluency recall, achieving 80–85% entity recall and outperforming baseline systems with fewer parameters.
This paper presents Iso-LoRA, an optimizer that enhances LoRA by promoting even energy distribution across singular directions through spectral descent on tangent-space perturbations, improving effective rank and downstream performance in language model adaptation.
TaRA is a training-aware initialization method for Low-Rank Adaptation (LoRA) that improves gradient fidelity, leading to better fine-tuning performance for large language models across various tasks.
Normalized Low-Rank Adaptation (NoRA) stabilizes LoRA training by normalizing down-projection matrices, accelerating convergence and improving performance without extra parameters or inference cost.
This paper identifies a structural failure mode in token-level credit assignment for LLM reinforcement learning when using LoRA, where intrinsic signals degenerate. It proposes Adapter-Residual Credit Assignment (ARCA), which derives token salience from adapter hidden-state residuals and remains competitive with baselines.
PreFT proposes applying adapters only to prefill tokens, discarding them during decode, which increases throughput for multi-adapter serving with minimal performance loss.