Dropping learning rate fixed my Qlora fine-tune more than anything else i tried
Summary
A user found that reducing the learning rate from 2e-4 to 1e-4 significantly improved QLoRA fine-tuning of Llama 3.1 8B on a small dataset (8k samples), preventing overfitting and leading to better evaluation results.
Similar Articles
The qlora 2e-4 default is wrong under 10k samples and nobody talks about it [D]
The author argues that the commonly recommended learning rate of 2e-4 for QLoRA fine-tuning is too high for datasets under 10k samples, leading to overfitting and poor evaluation, and suggests using a lower learning rate like 1e-4.
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
AQLoRA is a zero-search method that accelerates quantized LoRA fine-tuning by adaptively keeping layers with high NF4 reconstruction error in fp16, achieving up to 11% faster training with minimal accuracy loss.
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
This paper introduces an adaptive anisotropic learning-rate model for LoRA fine-tuning to address within-module heterogeneity, improving performance and rank capacity utilization across benchmarks.
@jbhuang0604: LoRA, low-rank adaptation, is arguably the most popular parameter-efficient fine-tuning method for LLMs. But how does i…
LoRA (low-rank adaptation) is the most popular parameter-efficient fine-tuning method for LLMs. This video introduces how LoRA and its variants (LoRA+, QLoRA, VeRA, DoRA) work.