mixed-precision

MixQuant proposes an adaptive mixed-precision quantization framework for LLMs that handles variable memory budgets by marginalizing layer distortion over random upstream configurations, outperforming existing methods across multiple models and budgets.

This paper proposes an operator-aware calibration method for absolute tolerances in tensor kernel correctness tests, using error distribution data to set tighter thresholds. The method achieves a 9.3% absolute recall improvement in bug detection with minimal false positives, demonstrated on the gpuemu corpus.

LaCache introduces a training-free acceleration framework for diffusion-based LLMs that uses lossless caching and precision-adaptive inference to eliminate redundant computation during denoising steps, achieving up to 40.2× end-to-end speedup while maintaining task accuracy.

This paper analyzes the structure of quantization loss in mixed-precision neural networks, showing that saturation makes the loss additive per layer and proposes a coverage model that predicts configuration loss with few parameters, validated on large-scale models.

KronQ is a post-training quantization framework that incorporates gradient covariance using a Kronecker-factored Hessian approximation, enabling bidirectional incoherence processing and improved sensitivity metrics for mixed-precision allocation. It achieves low perplexity even at 2-bit weight-only quantization on large models like LLaMA-3-70B.

A community researcher enabled running GLM-5.2 (753B parameters, all 256 experts) in vLLM without pruning via a hybrid quantization (NVFP4, NF3, MXFP8), fitting on 4×96GB GPUs with ~307k KV cache and near-FP8 accuracy.

This paper introduces MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE multimodal LLMs that addresses biases in expert importance estimation by decomposing selection frequency by modality and filtering redundant vision tokens, achieving minimal performance loss under aggressive quantization.

A mixed-precision quantization of Google's Gemma-4-12B-it model using NVFP4 for MLP weights and FP8 for attention layers, achieving 25% smaller footprint and faster throughput while maintaining quality.

dMX is a differentiable mixed-precision quantization framework that learns optimal floating-point bit-width assignments per layer for LLMs, targeting the MXFP family of formats defined by the OCP standard. It uses continuous optimization with temperature-based annealing and a budget-aware regularization term, consistently outperforming KL-divergence heuristics on Llama, Qwen3, and SmolLM2 models.

Researchers from UiT and University of Oslo propose a differentiable NAS framework that jointly optimizes architectural configurations and mixed-precision quantization for LLM compression, achieving up to 1.4× faster inference or 6% higher accuracy across seven reasoning tasks compared to sequential NAS-then-quantization baselines.

BitsMoE introduces a spectral-energy-guided bit allocation framework for quantizing Mixture-of-Experts LLMs, achieving substantial accuracy improvements and speedups under ultra-low-bit quantization.

ThriftAttention proposes a selective mixed-precision attention method that computes a small fraction of query-key blocks in FP16 and the rest in FP4, achieving near-FP16 quality with FP4 efficiency for long-context inference.

Proposes GEMQ, a global expert-level mixed-precision quantization method for MoE LLMs that uses linear programming and router fine-tuning to reduce memory and accelerate inference with minimal accuracy degradation.

CONF-KV is a KV-cache management system that uses model uncertainty to dynamically adjust cache retention, improving memory efficiency for long-context LLM inference while maintaining accuracy within 1.5-2.1 perplexity points.

This paper introduces RateQuant, a method for optimal mixed-precision KV cache quantization that uses rate-distortion theory to address distortion model mismatch. It significantly reduces perplexity compared to existing methods like KIVI and QuaRot with minimal calibration overhead.

Reddit post compares quantized Qwen3.6-27B variants (INT4, NVFP4, BF16-INT4) showing trade-offs between memory size and accuracy for different use-cases.

Bitnet.cpp presents a mixed-precision matrix multiplication library for efficient edge inference of ternary LLMs like BitNet b1.58, achieving up to 6.25x speedup over full-precision baselines. The system is open-sourced on GitHub.