Tag
The paper proposes FrameFT, a parameter-efficient fine-tuning method using sparse coefficients in a Fusion Frame basis, reducing memory footprint while achieving performance on par with or exceeding state-of-the-art PEFT techniques.
LongAct proposes a saliency-guided sparse update strategy for improving long-context reasoning in LLMs by selectively updating weights associated with high-magnitude activations in query and key vectors, achieving ~8% improvement on LongBench v2.