Tag
Tencent has compressed the Hy4-preview model from 1.5TB to about 200GB in GGUF format while maintaining approximately 98% of its performance.
This paper introduces a composite metric for quantization in small large language models that balances speed and quality by combining information retention and throughput gains, evaluated on models like Gemma 3 1B.
This paper introduces a PyTorch-based framework for pruning binarized neural networks and proposes a novel global weighting pruning method that achieves superior accuracy and pruning rates, enabling efficient deployment on edge hardware like FPGAs.
NVIDIA demonstrates how quantization-aware distillation (QAD) using NVIDIA Model Optimizer improves the Nemotron 3.5 Lightning model, reducing memory usage and increasing throughput while preserving accuracy for agentic benchmarks.
ZeroDigest has released new YMQ architecture-aware GGUFs for the Qwen3.8-27B model, enabling it to fit below 10GB and run on 12GB GPUs like the RTX 5070.
This paper surveys 200 works on LLM quantization, formalizing the 'Great Inversion' principle that contrasts energy concentration in coding with within-group flattening in quantization, and offers a guide for transform selection based on deployment regimes and formats like MXFP4 and NVFP4.
The paper proposes REP-LIE, a resource-efficient pruning method for Transformer models that uses low-rank importance estimation from LoRA gradients to enable pruning during finetuning, with competitive performance on LLaMA-7B and Mistral-7B.
GreenLeaf Law Embed Tiny is a compact 0.6B parameter embedding model for legal domain retrieval, achieving competitive performance on benchmarks like MLEB and MTEB(Law, v1) with efficient inference for resource-constrained environments.
The paper introduces Calibration-Preserving Pruning (CPP), a method to enhance pruning for maintaining conformal prediction efficiency, showing reduced prediction set sizes and improved accuracy on benchmarks like DBpedia-14 with models such as Qwen2.5-1.5B.
The paper proposes COEC, a training-free compensation framework for structured pruning of large language models that applies orthogonal rotations and calibration to reduce output error and improve accuracy after column removal.
This paper presents a framework for quantizing vision-language models to 2.7 bits per parameter, enabling efficient mobile deployment by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB while preserving performance on visual QA tasks.
StargazerLabs released depth-pruned versions of the Qwen3.8-27B model, named Mini-Me, in bf16, 8bit, and 4bit quantizations on Hugging Face.
The article claims that the Qwen3.8 27B model is equivalent to a compressed version of GPT-5.6 Luna, suggesting significant advancements in model efficiency and performance.
HyGenQ is a post-training quantization framework for hybrid iterative generative models that addresses challenges like excessive outliers and amplified anomalies, enabling 8-bit precision quantization while maintaining generation quality.
This paper presents a systematic sensitivity analysis of Mixture-of-Experts models using magnitude-based expert masking, finding that late layers are more resilient to masking, which provides a practical path for model compression.
This paper introduces Sniper, a two-stage structured pruning framework for LLMs that uses binary knapsack optimization to unify depth and width pruning, achieving near-exact compression ratio adherence and improved performance retention across multiple architectures.
This paper proposes using router weight sensitivity under lightweight fine-tuning (e.g., LoRA) to identify and prune experts in Mixture-of-Experts models, enabling significant memory and latency reductions with minimal accuracy loss.
This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.
CubicQuant proposes a parametric non-uniform scalar quantization format for LLM weights, using a monotonic cubic curve to adapt reconstruction levels at 1-8 bit widths while retaining dense integer code streams for GPU efficiency. Experiments show RMSE reductions over uniform and floating-point baselines, with preliminary H200 kernel measurements.
This preprint challenges the common assumption that quantization damage is additive noise, showing instead that it multiplies decision margins and shrinks them with bit-width, leading to silent failures in tool-use and safety decisions. The authors propose a fitted multiplicative model that predicts flip rates well.