model-compression

Discusses running Gemma 4 on a device with only 500MB of memory, likely through quantization or other optimization techniques.

The paper introduces 'Kilobyte models', an extreme compression paradigm where neural network weights are regenerated from a seed and a quantized latent vector, reducing storage to kilobytes by leveraging Mapping Networks and seeded block-wise bases.

SparseKAN is a unified compression method for Kolmogorov–Arnold Networks that prunes basis functions, neurons, and numerical precision under learnable gates, achieving up to 73% parameter reduction and significant latency improvements on software and FPGA hardware.

Proposes Progressive2, a teacher-student progressive co-evolving knowledge distillation method for substantial model compression, addressing large disparity between server and client capabilities. Introduces progressive teacher layer selection and student size reduction with a multi-feature fusion adapter.

Any-OPD presents the first framework for on-policy distillation between arbitrary latent flow-matching generators, enabling distillation from a 12B FLUX model to a 2.5B SD3.5 model by bridging via a frozen vision representation. It improves the student's PickScore from 0.846 to 0.884, rivaling the teacher at a fifth of its size.

This paper proposes FedSLM, a parameter-centric framework for federated fine-tuning of foundation models with heterogeneous compressed clients, using SVD-based decomposition and a weak-to-strong elicitation step to handle resource asymmetry. Experiments show it outperforms existing federated baselines while reducing client GPU memory by ~50%.

An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.

This paper investigates memory requirements for tabular foundation models like TabPFN and shows that model compression (e.g., INT4 quantization) can reduce memory footprint up to 7.6x with minimal accuracy loss, improving practical deployment efficiency.

TriSP introduces a tri-signal importance metric combining weight magnitude, activation norm, and gradient sensitivity for structured pruning of LLMs, achieving lowest perplexity and high throughput improvements on LLaMA-7B.

Proposes a two-stage structured pruning framework for LLMs that jointly optimizes latency and model size using multi-objective depth pruning and parallel Bayesian optimization, achieving favorable trade-offs for edge deployment.

CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.

A retired engineer proposes Blueprint Distillation (BPD) to separate expensive teacher analysis from student compression for large MoE models like Kimi K3, enabling reusable compression blueprints for multiple student sizes.

This paper introduces Neural Atom Prevalence (NAP), a Bayesian framework for structured node-level model selection that achieves high sparsity, accuracy, and uncertainty quantification in neural networks.

This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal when combined for LLM compression, leading to performance degradation, and proposes a novel Diagonal Adhesive Method (DAM) to mitigate this loss.

Session-Adaptive Orthogonal Distillation (SAOD) is a technique that compresses a 744-billion parameter model (1.5TB) to under 100GB, greatly reducing storage and inference costs.

A Twitter thread explains five key LLM quantization techniques (RTN, GPTQ, AWQ, LLM.int8(), QAT) for fitting large models on limited hardware, and references a comprehensive study paper.

Paper from Meta shows that quantized reasoning models often fail because compression makes them second-guess correct answers, but a small penalty on hesitation words like 'wait' or 'but' can cut reasoning length by 12-23% while maintaining or improving accuracy.

This paper introduces a compound sparsity framework for LLMs that combines static parameter pruning with dynamic token-level computation, showing that mixing both mechanisms outperforms single-dimension compression and delays performance degradation.

Details a method to run a 13 million parameter ASR Conformer model directly on a microcontroller, highlighting advances in edge AI deployment.

Discusses methods or research on reducing the size of model parameters, likely focusing on techniques like pruning or quantization to improve efficiency.