low-bit

A Virginia Commonwealth University paper explores extreme low-state weight quantization (3, 5, 7, 16 states) for Transformer-based electricity-load forecasting, showing that quantization-aware training and accumulated-update thresholding can retain predictive performance while reducing weight updates by ~99.7%.

A research release introducing RCOL, a modified dynamic low-bit quantization technique (IQ1_M, IQ2_M, IQ3_M) for Qwen 3.5 2B that works under strict VRAM budgets and shows significant KL divergence improvements over standard llama.cpp imatrix quants, especially at extremely low bitrates.

论文指出权重量化中降低重建损失并不总能提升模型性能，甚至可能因输入激活分布偏移而劣化，并提出分布鲁棒量化（DRQ）后处理优化框架，通过在约束激活分布集上最小化最坏情况重建损失来改进 AWQ、GPTQ、ParoQuant 等六种 PTQ 方法的下游表现，且不增加推理开销。

ThinQuant proposes a data-efficient gradient-free rotation learning method for weight and activation quantization of LLMs, using convex-hull-based calibration data selection and an ADMM-based Stiefel manifold optimizer that scales to Llama-3.1-405B on a single H200 GPU with improved perplexity and drastically reduced calibration time.

QuantMLA 提出一种功能对齐的双路径量化框架，对 MLA 的内容路径与 RoPE 路径进行低比特 KV 缓存压缩，首次实现 INT4 联合缓存且精度几乎无损，并通过自定义低比特注意力内核在 128K 上下文下获得 3.59 倍压缩和 5.168 倍吞吐提升。

Introducing Ternary Bonsai 2 27B, a highly compressed AI model that retains 98.2% of performance while being 9x smaller in footprint, enabling efficient local deployment for tasks like reasoning, coding, and multimodal processing.

ReRound is a post-training quantization method that uses a conditional diffusion model to guide rounding of near-midpoint weights, improving small LLM accuracy at 3-bit and 4-bit quantization without inference overhead.

Introduces AuroOFT, a method that enhances quantized orthogonal fine-tuning (QOFT) with a zero-start gated low-rank nonlinear residual, improving mathematical reasoning accuracy over QOFT and QLoRA while using fewer trainable parameters on low-bit language models.

This paper identifies that low-bit KV cache quantization degrades LLM accuracy due to structured local misranking of logits, and proposes DGAP, a method that restores the local distribution of top-K candidates, recovering RULER accuracy from 47.8% to 83.2% on Llama-3.1-8B with minimal overhead.

A user tested quantized 1-bit and 2-bit versions of the 27B-parameter Bonsai model on Terminal-Bench 2.0, achieving results within 8GB VRAM.

This paper analyzes the structure of quantization loss in mixed-precision neural networks, showing that saturation makes the loss additive per layer and proposes a coverage model that predicts configuration loss with few parameters, validated on large-scale models.

This paper introduces BitEmbed, an extreme low-bit framework for LLM-based text embeddings that converts pretrained LLM backbones into BitNet-style encoders with ternary weights and quantized activations. It achieves comparable performance to full-precision models while significantly reducing encoding and storage costs.

Proposes LC-QAT, a 2-bit weight-only vector quantization aware training framework for LLMs that uses a learned affine mapping to enable end-to-end training, achieving state-of-the-art results with only 0.1%-10% of training data.

This paper introduces Qift, a fixed no-zero two-bit weight quantization level set designed for Hadamard-rotated LLMs, achieving improved W2A4/KV4 inference by leveraging the near-zero-centered Gaussian-like distribution of rotated weights. Experiments on LLaMA-2-7B and LLaMA-3.1-8B show consistent perplexity gains over standard W2 quantization.

BitsMoE introduces a spectral-energy-guided bit allocation framework for quantizing Mixture-of-Experts LLMs, achieving substantial accuracy improvements and speedups under ultra-low-bit quantization.

This paper presents Tail-Aware HiFloat4, a W4A4 post-training quantization method for the Wan2.2 text-to-video diffusion model, which uses activation-tail-aware percentile calibration to mitigate outlier effects while preserving HiFloat4 arithmetic.

Quant.npu introduces a fully static quantization framework for mobile NPUs, using learnable parameters and rotation matrices to enable efficient low-bit LLM inference without runtime re-computation, achieving up to 15.1% latency reduction.

This paper measures maximum activation magnitudes across 27 checkpoints from 8 open LLM families, finding significant variance across families, architectures, and training stages, with implications for low-bit quantization and deployment.

A corrected 2-bit GGUF model file has been uploaded to Hugging Face after fixing a bug in the imatrix computation, leading to improved logits recall and reduced error.

A highly efficient AI model architecture using ternary weights (-1, 0, 1) that achieves competitive performance while requiring only 1.58 bits per parameter, enabling deployment on extremely constrained devices.