Tag
A new paper from NaceAI proposes a hypernetwork-based method for injecting knowledge into large language models without modifying their core parameters, potentially enabling efficient continual learning. The approach uses generated low-rank adapters to encode new facts while keeping the base model frozen.
The paper investigates scaling laws for hypernetwork-based knowledge injection into LLMs, finding predictive power law scaling and reliable out-of-distribution generalization, establishing hypernetworks as a scalable alternative to LoRA and full fine-tuning.
PorTAL is a novel architecture that decouples task fine-tuning from specific base model weights, enabling portable task adapters that can be transferred to new models with minimal retraining. It achieves ~98% of LoRA's accuracy gain on unseen models using only half the calibration data.
Independent study shows 227M-parameter hypernetwork adds zero gain over well-crafted few-shot prompts for tool-use in 3B Llama, achieving 79.7% of GPT-5 performance at 10× lower latency.