Tag
MiCA (Minor Component Adaptation), a new fine-tuning method that initializes adapters in the minor singular subspace for better knowledge uptake and less forgetting, has been merged into the Hugging Face PEFT library. It is available via the PEFT main branch and integrates through the existing LoRA interface with init_lora_weights='mica'.
Explains five parameter-efficient fine-tuning techniques: LoRA, LoRA-FA, VeRA, Delta-LoRA, and LoRA+, detailing how each modifies model weights during adaptation.
This paper presents EMA, a model adaptation system for learning-based systems that reduces training and labeling costs while improving system performance in evolving environments.
Google introduces T5Gemma, a new collection of encoder-decoder models adapted from the Gemma 2 decoder-only architecture, offering improved quality-efficiency trade-offs for tasks like summarization and translation.
OpenAI has released fine-tuning capabilities for GPT-3.5 Turbo, allowing developers to customize models for specific use cases with improved performance, steerability, and output formatting. The update enables fine-tuned GPT-3.5 Turbo to match GPT-4 performance on certain tasks while reducing prompt sizes by up to 90%.