Tag
论文提出 LADDER(Layer-Wise Allocation via Damage-Driven Evaluation and Refinement),一种针对大语言模型的非均匀后训练剪枝方法,通过逐层评估最小化每移除参数的性能损失,生成嵌套掩码轨迹,在 70% 以上高稀疏度下表现显著优于基线。
The paper presents a physics-inspired approach to pruning LLM blocks by modeling block removal as a constrained binary optimization problem mapped to an Ising glass, achieving significant compression gains without benchmarking each configuration.
This paper introduces Sniper, a two-stage structured pruning framework for LLMs that uses binary knapsack optimization to unify depth and width pruning, achieving near-exact compression ratio adherence and improved performance retention across multiple architectures.
This paper analyzes why compression statistics for LLM pruning can be reproducible yet select suboptimal endpoints, introducing information boundaries and observation fibers to model the gap. It proposes group-resolved and model-specific mask selection methods that improve worst-group perplexity across dense LLMs and OLMoE.
PALS adjusts per-layer sparsity ratios for LLM pruning based on the 99th percentile of activation magnitudes, achieving significant perplexity improvements on LLaMA-2-7B compared to uniform sparsity, with negligible added cost.
This paper presents a cascaded multi-granularity pruning framework for deploying LLMs on Industrial IoT edge devices, achieving up to 13.8x compression with minimal accuracy loss on MHA+GELU architectures while exposing a collapse on GQA+SwiGLU designs.
This paper empirically compares pruning vs. training small language models from scratch, finding that pruning provides a strong advantage under limited token budgets but that the advantage diminishes as training scales, especially with coarse pruning.