low-rank-importance

The paper proposes REP-LIE, a resource-efficient pruning method for Transformer models that uses low-rank importance estimation from LoRA gradients to enable pruning during finetuning, with competitive performance on LLaMA-7B and Mistral-7B.