Tag
This paper introduces a compound sparsity framework for LLMs that combines static parameter pruning with dynamic token-level computation, showing that mixing both mechanisms outperforms single-dimension compression and delays performance degradation.
A paper discovers that about 50% of expert computations in MoE models are wasted on tokens that don't need expert processing. The proposed ZEDA method teaches the model to skip these computations, saving up to half of expert calculations.