quantization

A trimmed English-only GGUF version of Kimi K3 (IQ2-XXS) reduces model size from 711GB to 478GB by removing multi-language components, with early tests suggesting it may match or outperform the standard 2-bit version on coding tasks.

A discussion question asking about the theoretically optimal quantization bit-width for LLMs under a fixed memory budget, referencing recent 3-bit/2-bit results and scaling-law work from 2025-2026.

A llama.cpp PR proposes switching quantized-KV decode from the VEC to TILE SYCL kernel on Intel Battlemage, reportedly delivering up to ~169% faster decode at 118K context. The PR is open with caveats and limited independent verification.

A detailed report on quantizing LiquidAI's LFM2.5-2.6B model with various GGUF and KV cache quantizations, showing it fits on an 8GB Raspberry Pi with minimal degradation, but warning against Q4_K_M.

A llama.cpp pull request adds an x86 VNNI implementation for Q2_0 × Q8_0 dot products, delivering 3.0–3.6x CPU-only speedups on Bonsai models with bit-for-bit kernel correctness and 99.2% token agreement.

Unsloth releases new GGUF quantizations of Kimi K3, ranging from 466GB to 649GB, enabling efficient deployment of the large model.

This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.

A user reports that quantising the f16 MTP draft model to Q4_K for Gemma 4 31b gives roughly 10% faster decode (65 to 72 TPS) on dual 3090s compared to the default Q4_0, while Q2_K performs worse.

Tests et réglages détaillés pour optimiser DeepSeek-V4-Flash-0731 en GGUF sur une RTX 3090, atteignant ~15 tok/s à 128K de contexte grâce à différentes quantifications et paramètres de chargement.

This paper proposes NOVA-KV, a transform-coding approach to KV cache quantization that uses attention-preserving transforms to allocate bits where queries actually attend, improving long-context retrieval accuracy at low bit rates compared to prior methods.

Introduces Recurrent Residual Quantization (RRQ), a post-training quantization framework that enables multiple effective precisions from a single LLM checkpoint via additive residual refinement, improving flexibility and construction speed.

Scenema Audio, an expressive text-to-speech model with zero-shot voice cloning, is now available as a native ComfyUI custom node, quantized to run on 8GB VRAM. The release adds inline stage direction cues, 12 preset voices, and simplifies the prompt format for ComfyUI.

Ling-3.0-flash MXFP4, a quantized model, has been released and runs locally on a single DGX Spark, achieving ~80 tok/s decoding and 2,500-3,500 tok/s long-input prefilling, enabling private on-device inference for coding, agents, and offline batch jobs.

jabbatheduck released a GGUF quantized version of the REAP expert-pruned DeepSeek-V4-Flash checkpoint, aggressively compressed for memory-constrained inference on consumer GPUs while preserving router and attention precision.

Kijai's MiniMax-H3-experimental is a work-in-progress model featuring w4a8 quantization and int8_convrot VAE support for ComfyUI, released for testing purposes.

Discusses the bottlenecks of using Transformer for large-scale inference and reviews the evolution of inference optimization technologies for large models from 2023 to 2026, including KV cache quantization, speculative decoding, architectural innovation, and hardware-software co-design.

This paper presents NANQ, a noise-floor-aware mixed-precision non-uniform quantization framework for analog compute-in-memory (CIM) systems, which adapts quantization precision to hardware noise characteristics and improves model accuracy under low-bit constraints.

Proposes OptR, an output-aware rotation method for INT2 KV-cache quantization that minimizes post-output attention error, improving QuaRot and OSCAR across models and benchmarks.

ARCHead is a packed LM-head compressor that combines quantized low-rank factors, INT4 residuals, and an activation-metric correction to reduce storage by ~3.7-3.9x while preserving perplexity, complementing existing block quantizers.

A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.