Tag
The paper introduces Magnitude Profile (MP) scoring, a calibration-free method for pruning attention heads in transformers, achieving better perplexity on models like OPT-6.7B and RoBERTa-large compared to existing methods, with zero forward passes or calibration data.
The paper introduces GeoPair, a training-free framework for transformer compression that optimizes cross-layer factorizations while preserving activation geometries, achieving state-of-the-art results across diverse architectures.