Tag
The paper proposes REP-LIE, a resource-efficient pruning method for Transformer models that uses low-rank importance estimation from LoRA gradients to enable pruning during finetuning, with competitive performance on LLaMA-7B and Mistral-7B.
CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.