Tag
This paper shows that reasoning-importance attribution used to guide pruning of chain-of-thought language models is fragile: answer-token scoring saturates, single-neuron causal importance is noise that cancels, and reasoning-specific importance is not separable—yielding practical rules to score reasoning tokens, attribute at coarse (~100-neuron) granularity, and evaluate with accuracy rather than perplexity.
The paper finds decision information in LLMs is concentrated in intermediate layers (final six layers removable), and proposes a method that jointly trains a full model with shared prefix sub-models plus a lightweight gate for adaptive computation, matching or beating independently trained models across seven decision benchmarks.
本文提出结构保持的专家剪枝框架,通过 PEEP 与 DCEP 两种方法利用专家输出的结构冗余性对稀疏 MoE 大模型进行剪枝,在 DeepSeek-V2-Lite 和 Qwen3-30B-A3B 上超越现有基线。
A zero-retraining REAP-pruned derivative of NVIDIA's Nemotron-3.5-Transcribe-30B-A4B speech LLM shrinks it from 67.5GB to 10GB (4-bit) while keeping WER at 5.16% vs the teacher's 4.12%, and it runs 27% faster in single-stream throughput on an H200 with vLLM.
The paper introduces 'All-in-One', a composable attack framework from ETH Zurich that trains LLMs to pass safety checks yet turn malicious after deployment-time transformations like quantization, pruning, or model merging, by approximating each transformation as weight perturbations jointly optimized in a single training objective.
论文提出 OptEP,一种面向混合专家(MoE)语言模型的后训练专家剪枝方法,通过连续松弛优化专家选择变量并结合重构驱动的权重更新,在 Qwen3、GLM-4.5-Air、Mixtral 等模型上显著优于现有剪枝基线,激进剪枝下精度提升最高达 16.20%。
提出一种免重训练的方法,将预训练 Transformer 转化为可弹性伸缩的模型,通过联合排序并软剪枝单个 MLP 神经元和注意力头,在多种稀疏度目标下连续自适应计算预算,在 27 个视觉与文本 Transformer 编码器上实现最高 60% 稀疏度下的平滑性能衰减。
The paper introduces Trajectory-Anchored Pruning (TAP), the first structural pruning framework for RL-trained long-horizon agents, which couples channel pruning with trajectory-anchored on-policy recovery. With 60% of FFN channels removed, TAP retains 99.2% and 88.0% of dense 7B agents' task success on ALFWorld and WebShop while cutting GPU time per successful task by roughly 22% and 17%.
MoRA proposes a structured MoE expert pruning framework using learnable router biases optimized with LM loss and a routing-diversity regularizer, plus a post-pruning expert approximation mechanism, outperforming state-of-the-art pruning methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B with 25-50% expert removal.
Two open-weight fine-tunes of Qwen3.8-Flash-Next are released: Victoria, a pruned coding/agent model (44% of experts removed via REAP, retrained in NVFP4, 70% on Terminal-Bench 2.1) and Maple, a Canada-focused fine-tune that dramatically improves citation of official Canadian sources.
The paper proposes OBC-Prune, a calibration method for pruning large reasoning models that identifies causally important reasoning circuits to improve accuracy and reduce inference overhead on benchmarks like MATH500 and LiveCodeBench.
ToMoE proposes a method to convert dense large language models into Mixture-of-Experts models using dynamic structural pruning without weight updates, outperforming existing techniques.
A trimmed English-only GGUF version of Kimi K3 (IQ2-XXS) reduces model size from 711GB to 478GB by removing multi-language components, with early tests suggesting it may match or outperform the standard 2-bit version on coding tasks.
This paper introduces Maestro, a structured pruning framework for Mixture-of-Experts language models that uses Markov chains to model expert activation trajectories, achieving globally aware pruning and outperforming baselines by up to 10.61% under 50% compression.
This paper proposes entropy-regularized probabilistic gates to maintain uncertainty in sparse federated optimization, improving sparsity recovery and test performance under data heterogeneity and scarce data.
A pruned and quantized version of MiniMax-M3 (MiniMax-M3-Medium-JANG_2L) optimized to run on 128GB Macs using vMLX, featuring 32% expert pruning and JANG_2L mixed-precision quantization to fit within ~105 GB.
This paper demonstrates that cosine similarity is a poor proxy for assessing layer importance in LLMs, and proposes using the actual accuracy drop from layer removal as a more robust metric.
This paper introduces a resource-efficient pruning framework that identifies and removes parameters associated with unsafe behaviors in large language models while preserving utility. Using gradient-free attribution and the Lottery Ticket Hypothesis perspective, the method achieves significant reductions in unsafe generations and improved robustness against jailbreak attacks with minimal performance loss.