Tag
This paper introduces Sniper, a two-stage structured pruning framework for LLMs that uses binary knapsack optimization to unify depth and width pruning, achieving near-exact compression ratio adherence and improved performance retention across multiple architectures.
This paper proposes using router weight sensitivity under lightweight fine-tuning (e.g., LoRA) to identify and prune experts in Mixture-of-Experts models, enabling significant memory and latency reductions with minimal accuracy loss.
This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.
CubicQuant proposes a parametric non-uniform scalar quantization format for LLM weights, using a monotonic cubic curve to adapt reconstruction levels at 1-8 bit widths while retaining dense integer code streams for GPU efficiency. Experiments show RMSE reductions over uniform and floating-point baselines, with preliminary H200 kernel measurements.
This preprint challenges the common assumption that quantization damage is additive noise, showing instead that it multiplies decision margins and shrinks them with bit-width, leading to silent failures in tool-use and safety decisions. The authors propose a fitted multiplicative model that predicts flip rates well.
ReQuant introduces a backpropagation-free, fixed-grid discrete refinement stage for post-training quantization (PTQ) that iteratively improves initial quantized models while preserving the quantized format, showing consistent gains across various LLMs and bit-widths.
UniMoMo compresses MoE-based recommendation models by merging experts based on functional behavior and routing traffic, preserving quality while speeding up inference.
Discusses running Gemma 4 on a device with only 500MB of memory, likely through quantization or other optimization techniques.
The paper introduces 'Kilobyte models', an extreme compression paradigm where neural network weights are regenerated from a seed and a quantized latent vector, reducing storage to kilobytes by leveraging Mapping Networks and seeded block-wise bases.
SparseKAN is a unified compression method for Kolmogorov–Arnold Networks that prunes basis functions, neurons, and numerical precision under learnable gates, achieving up to 73% parameter reduction and significant latency improvements on software and FPGA hardware.
Proposes Progressive2, a teacher-student progressive co-evolving knowledge distillation method for substantial model compression, addressing large disparity between server and client capabilities. Introduces progressive teacher layer selection and student size reduction with a multi-feature fusion adapter.
Any-OPD presents the first framework for on-policy distillation between arbitrary latent flow-matching generators, enabling distillation from a 12B FLUX model to a 2.5B SD3.5 model by bridging via a frozen vision representation. It improves the student's PickScore from 0.846 to 0.884, rivaling the teacher at a fifth of its size.
This paper proposes FedSLM, a parameter-centric framework for federated fine-tuning of foundation models with heterogeneous compressed clients, using SVD-based decomposition and a weak-to-strong elicitation step to handle resource asymmetry. Experiments show it outperforms existing federated baselines while reducing client GPU memory by ~50%.
An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.
This paper investigates memory requirements for tabular foundation models like TabPFN and shows that model compression (e.g., INT4 quantization) can reduce memory footprint up to 7.6x with minimal accuracy loss, improving practical deployment efficiency.
TriSP introduces a tri-signal importance metric combining weight magnitude, activation norm, and gradient sensitivity for structured pruning of LLMs, achieving lowest perplexity and high throughput improvements on LLaMA-7B.
Proposes a two-stage structured pruning framework for LLMs that jointly optimizes latency and model size using multi-objective depth pruning and parallel Bayesian optimization, achieving favorable trade-offs for edge deployment.
CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.
A retired engineer proposes Blueprint Distillation (BPD) to separate expensive teacher analysis from student compression for large MoE models like Kimi K3, enabling reusable compression blueprints for multiple student sizes.
This paper introduces Neural Atom Prevalence (NAP), a Bayesian framework for structured node-level model selection that achieves high sparsity, accuracy, and uncertainty quantification in neural networks.