Tag
Introduces Variable Bit-width Quantization (VBQ), a training-time method where each group of 64 weights learns its own bit-width (1,2,4,8) via Gumbel-Softmax relaxation. VBQ discovers a heterogeneous allocation that yields a 'bigger-but-smaller' regime, e.g., a 131M parameter model at 1.82 mean bits beats a 55M FP16 model while using less storage, and a 1.46B model matches a 593M FP16 with ~3.7x less storage.
PuzzleMoE introduces a pairwise dual-mask expert merging algorithm and bit-level packing technique to compress large Mixture-of-Experts models, maintaining performance while reducing storage and accelerating inference.
Proposes Generic TB-Coverage, a coverage-aware expert pruning method for sparse Mixture-of-Experts language models that uses only generic text corpora for calibration and preserves cross-corpus expert coverage, improving accuracy and reducing perplexity degradation.
This paper investigates the effects of domain-specific expert pruning on both utility and factual reliability of Mixture-of-Experts (MoE) models in the biomedical domain. It finds that moderate pruning preserves in-domain utility without immediate reliability loss, but extreme pruning increases hallucination risks, and generalization degrades rapidly in cross-domain settings.
NVIDIA's 4-bit quantized Qwen3.6-27B (NVFP4) is found to be near-lossless compared to the full bf16 model, with behavioral differences being minor and random rather than systematic, making it a practical drop-in replacement.
A developer is rebuilding Gemma 4 31b into a smaller 26b model by removing weak SWA layers, adding attention-based residual networks (from Moonshot), and using topK logit targets for retraining, aiming for better long context and performance.
Introduces Cross-Space Distillation, a method to transfer knowledge from modern high-capacity diffusion models to compact student models with different latent spaces using a lightweight latent interface called Bridge, enabling quality improvements without modifying the student backbone.
A developer implemented a --skip-layers flag for a llama.cpp fork, allowing users to skip entire transformer blocks at load time with minimal performance impact, based on recent papers. This provides another method for fitting large models on limited hardware, complementing quantization and bake-time pruning.
FlexMoE proposes a one-for-all nested intra-expert pruning method for MoE language models, enabling multiple deployable subnetworks from a single training run with minimal performance loss.
Introduces Proxy-KD, a novel method for distilling knowledge from black-box large language models (like GPT-4) into smaller models using a proxy model, surpassing both traditional black-box and white-box KD techniques.
Clark Labs released Clark Air Sana 1.6B, a ternary-quantized version of the Sana 1.6B text-to-image transformer that is 8.6× smaller than FP16 while maintaining near-FP16 quality, enabling efficient deployment.
A calibration-aware Q4_K_M quantization of Qwen3.5 0.8B using SpectralQuant recovers 96.5% of the BF16 performance gap compared to the standard llama.cpp Q4_K_M quant.
A systematic study of compressing recursive reasoning models for edge hardware finds that aggressive quantization destroys global reasoning while preserving local prediction. The paper introduces per-channel calibrated INT4 to recover reasoning ability and provides deployment recipes fitting 8 MB SoC and 4 MB MCU targets.
This paper presents a cascaded multi-granularity pruning framework for deploying LLMs on Industrial IoT edge devices, achieving up to 13.8x compression with minimal accuracy loss on MHA+GELU architectures while exposing a collapse on GQA+SwiGLU designs.
CAT-Q introduces a post-training ternary quantization method for LLMs that uses learnable modulation and softened ternarization, achieving superior performance over BitNet 1.58-bit while using only 512 calibration samples and scaling to 235B parameters.
NVIDIA releases Nemotron-Labs-3-Puzzle-75B-A9B, a compressed version of Nemotron-3-Super with improved inference efficiency and strong benchmark performance.
A user questions why AutoRound, a quantization tool offering superior accuracy retention at low bits and direct GGUF export, is overlooked despite outperforming standard AWQ and RTN, especially on complex models like Qwen3.6 27B.
Proposes a structural pruning framework for MoE models that maximizes channel-score coverage via attribution-based approximation, achieving 50% or 25% pruning with 4-bit quantization and reducing memory footprint by 5.27x on Qwen3-30B-A3B.
This paper introduces MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE multimodal LLMs that addresses biases in expert importance estimation by decomposing selection frequency by modality and filtering redundant vision tokens, achieving minimal performance loss under aggressive quantization.
Introduces Simplex-Constrained Sparse Bagging (SCSB), a post-training framework that optimizes estimator weights over the probability simplex using out-of-bag samples, achieving up to 96% ensemble compression and improved calibration.