sparse-fine-tuning

USAF (Ultra Sparse Adaptive Fine-Tuning) is a new method that allows fine-tuning MoE models on consumer GPUs with as little as 12GB VRAM, including on AMD hardware, by training only the most important sparse weights and the router, unlike LoRA/QLoRA which cannot.