Tag
This paper investigates memory requirements for tabular foundation models like TabPFN and shows that model compression (e.g., INT4 quantization) can reduce memory footprint up to 7.6x with minimal accuracy loss, improving practical deployment efficiency.
TriSP introduces a tri-signal importance metric combining weight magnitude, activation norm, and gradient sensitivity for structured pruning of LLMs, achieving lowest perplexity and high throughput improvements on LLaMA-7B.
Proposes a two-stage structured pruning framework for LLMs that jointly optimizes latency and model size using multi-objective depth pruning and parallel Bayesian optimization, achieving favorable trade-offs for edge deployment.
CausalGate introduces a method that uses causal interventions to measure the importance of transformer sub-layers and distills this into static scalar gates for efficient inference without runtime overhead, outperforming existing pruning and routing methods.
A retired engineer proposes Blueprint Distillation (BPD) to separate expensive teacher analysis from student compression for large MoE models like Kimi K3, enabling reusable compression blueprints for multiple student sizes.
This paper introduces Neural Atom Prevalence (NAP), a Bayesian framework for structured node-level model selection that achieves high sparsity, accuracy, and uncertainty quantification in neural networks.
This paper provides the first mathematical proof that low-rank decomposition and quantization are non-orthogonal when combined for LLM compression, leading to performance degradation, and proposes a novel Diagonal Adhesive Method (DAM) to mitigate this loss.
Session-Adaptive Orthogonal Distillation (SAOD) is a technique that compresses a 744-billion parameter model (1.5TB) to under 100GB, greatly reducing storage and inference costs.
A Twitter thread explains five key LLM quantization techniques (RTN, GPTQ, AWQ, LLM.int8(), QAT) for fitting large models on limited hardware, and references a comprehensive study paper.
Paper from Meta shows that quantized reasoning models often fail because compression makes them second-guess correct answers, but a small penalty on hesitation words like 'wait' or 'but' can cut reasoning length by 12-23% while maintaining or improving accuracy.
This paper introduces a compound sparsity framework for LLMs that combines static parameter pruning with dynamic token-level computation, showing that mixing both mechanisms outperforms single-dimension compression and delays performance degradation.
Details a method to run a 13 million parameter ASR Conformer model directly on a microcontroller, highlighting advances in edge AI deployment.
Discusses methods or research on reducing the size of model parameters, likely focusing on techniques like pruning or quantization to improve efficiency.
KDFlow is a novel knowledge distillation framework for large language models that uses a decoupled architecture with SGLang for teacher inference and FSDP2 for student training, achieving 1.44x to 6.36x speedup over existing frameworks.
Introduces qZACH-ViT, a quantization-aware extension of ZACH-ViT with recursive intrinsic explanations, and Recursive Attribution-Stabilized Optimization (RASO) for stable attribution gradients. Achieves high prediction agreement and speedups on MedMNIST datasets after INT8 conversion.
PrismML announces Bonsai 27B, a binary-quantized version of Qwen3.6 27B that runs on a phone using only 1.125 bits per weight, claiming 89.5% intelligence retention. The model is being independently tested by @sudoingX to verify performance.
The author claims that ternary decomposition performs as well as Q4_K_M quantization while using slightly more VRAM, and is fully ternary and PTQ without QAT.
Announcement of Hy3 1-bit quantized model with 89-93 GB memory footprint.
Prism ML is in talks with Apple to deploy model-shrinking technology that would allow powerful AI models to run directly on iPhones, improving privacy and reducing reliance on cloud inference.
EvoLP is a self-evolving latency predictor for neural network models on edge devices, designed to guide model compression while satisfying strict latency constraints. It outperforms prior methods across multiple edge devices and model variants.