ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

Hugging Face Daily Papers Papers
quantization reasoning-models two-bit-quantization knowledge-distillation quantization-aware-training model-compression inference-efficiency

Summary

ThinkBit 是一个严格标量的两比特量化框架，通过分层比特分配、Gumbel-softmax 搜索码字、知识蒸馏重校准和量化感知训练来保留推理能力，在 Qwen3-1.7B 和 14B 模型上以远小于 bfloat16 的体积取得接近原模型的 MATH-500 成绩。

Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8times cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwise Gumbel-softmax-based scalar quantization algorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a short quantization-aware training (QAT) stage that lets the model mimic the base model's distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.
Original Article
View Cached Full Text

Cached at: 10/28/26, 01:54 AM

Paper page - ThinkBit: Search Codewords, Recalibrate Scales, Select by Reasoning

Source: https://huggingface.co/papers/2610.34886

Abstract

ThinkBit is a scalar two-bit quantization framework that preserves reasoning by assigning layerwise bit tiers, optimizing codewords via Gumbel-softmax, and recalibrating with knowledge distillation followed by quantization-aware training.

Reasoning language models generate thousands of tokens per answer, and generation speed is limited by memory bandwidth, so storing weights in two bits instead of bfloat16 can potentially make inference 8times cheaper. Two-bit methods are usually judged by perplexity and multiple-choice question-answering accuracy, both of which say little about whether a model can still reason through more difficult problems. We present ThinkBit, a strictly scalar two-bit framework built to preserve reasoning, in which every weight is an integer codeword times a bfloat16 scale shared by a group of weights. We first use a sensitivity metric to assign each layer a ternary, 2-bit, or 3-bit tier to allocate bitwidths optimally among layers. This is followed by a layerwiseGumbel-softmax-basedscalar quantizationalgorithm that searches for optimal codewords block by block. Since a layerwise scheme does not capture how quantized layers interact with each other, we recalibrate scales, with the codewords frozen, using a knowledge-distillation-based loss against the teacher model. We then perform a shortquantization-aware training(QAT) stage that lets the model mimic the base model’s distribution by reassigning codewords. Finally, a paired reasoning test on training problems selects the final checkpoint between the recalibration stage and the QAT stage. On Qwen3-1.7B at matched size, our recalibrated checkpoint scores 46.7 on MATH-500 against 30.0 for EfficientQAT trained on the same data, and 73.0 after QAT on reasoning traces; at 0.52 GB it is smaller than every published 2-bit GGUF of the model yet more accurate on MATH-500. We show that this framework scales and up to 14B, a 3.74 GB model scores 94.8 on MATH-500, against 97.5 for the 29.5 GB bfloat16 model.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.34886

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.34886 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.34886 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.34886 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning

Hugging Face Daily Papers

DoubtLess 提出一种免训练的对比解码方法，用一个更小的推理模型引导大模型输出更简洁的思维链，在七个竞赛数学、研究生科学与编程基准上将 Qwen3-32B 的 CoT 缩短 31% 且准确率不变，优于五个推理时基线。

Boole-1.7B: A 1-bit Qwen3-1.7B model that outperforms Bonsai-1.7B

Reddit r/LocalLLaMA

Precision Numerics 发布 Boole-1.7B，一个基于 Qwen3-1.7B 的 1-bit 量化模型，权重仅 247 MB（比 BF16 小 13.9 倍），在 10 项任务评测中达到 BF16 模型 85.9% 的平均分，超越 1-bit 的 Bonsai-1.7B 并接近 1.58-bit 的 Bonsai-1.7B。模型支持浏览器本地运行、vLLM 与 llama.cpp。

WeiboAI/VibeThinker-3B

Hugging Face Models Trending

VibeThinker-3B is a 3B-parameter model that achieves frontier-level reasoning performance on math, coding, and STEM benchmarks by optimizing the Spectrum-to-Signal Principle (SSP) post-training pipeline, reaching performance comparable to much larger models.

[2506.13771] LittleBit: Ultra Low-Bit Quantization via Latent Factorization

Reddit r/LocalLLaMA

LittleBit 提出了一种通过低秩潜在矩阵分解与二值化实现超低比特（低至0.1 BPW）LLM 量化的框架，配合多尺度补偿机制，在 Llama2-7B 上以 0.1 BPW 超越了现有 0.7 BPW 方法，实现约 31 倍内存压缩与最高 11.6 倍推理加速。