Disaggregated Quantization: Specializing LLM Prefill and Decode

Hugging Face Daily Papers Papers
llm quantization prefill decode disaggregated-quantization

Summary

This paper introduces disaggregated quantization, a method that tailors quantization approaches for LLM prefill and decode phases to enhance inference efficiency and accuracy.

Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computation formats, weights and storage placement to both of these phases. On Qwen 3 and Gemma 3, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while matching or exceeding its accuracy at 2-3-bit decode on both decode-heavy and prefill-heavy tasks. With released Qwen3.8-27B GGUF decoders, training an NVFP4 prefiller improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro without modifying the decode checkpoint. To accommodate the additional checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. On the same 27B model, ODP delivers a 1.78x time-to-first-token speedup over the weight-only baseline at 8K prompt length in llama.cpp. We evaluate accuracy under disaggregated serving in vLLM and further validate shared-weight format disaggregation through post-training quantization on models up to 2.8T parameters.
Original Article
View Cached Full Text

Cached at: 09/28/26, 08:06 PM

Paper page - Disaggregated Quantization: Specializing LLM Prefill and Decode

Source: https://huggingface.co/papers/2609.26333

Abstract

Prefillanddecoderewarddifferentapproachestoquantization:low-precisionarithmeticacceleratespromptprocessing,whilecompactweightsreducememorytrafficduringgeneration.Wepropose“disaggregatedquantization“(DQ),whichspecializescomputationformats,weightsandstorageplacementtobothofthesephases.OnQwen3andGemma3,removingactivationquantizationspecificallyondecodeimprovesaccuracyondecode-heavytaskswithoutincreasinginferencecost.Trainingseparatecompute-nativeprefillweightsacceleratespromptprocessingrelativetoweight-onlyinferencewhilematchingorexceedingitsaccuracyat2-3-bitdecodeonbothdecode-heavyandprefill-heavytasks.WithreleasedQwen3.8-27BGGUFdecoders,traininganNVFP4prefillerimproves1-bitaccuracyby32.5pointsonMMLU-Proand35.3onMMMU-Prowithoutmodifyingthedecodecheckpoint.Toaccommodatetheadditionalcheckpointonasingledevice,offloadeddisaggregatedprefill(ODP)streamsitsweightsfromSSD,amortizingloadingoverpromptlength.Onthesame27Bmodel,ODPdeliversa1.78xtime-to-first-tokenspeedupovertheweight-onlybaselineat8Kpromptlengthinllama.cpp.WeevaluateaccuracyunderdisaggregatedservinginvLLMandfurthervalidateshared-weightformatdisaggregationthroughpost-trainingquantizationonmodelsupto2.8Tparameters.

View arXiv pageView PDFGitHub6Add to collection

Get this paper in your agent:

hf papers read 2609\.26333

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### ISTA-DASLab/Qwen3.8-27B-NVFP4-prefiller 24B• Updatedabout 6 hours ago • 301 • 18

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.26333 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.26333 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

arXiv cs.CL

Mix-Quant proposes a phase-aware quantization framework for agentic LLMs, using NVFP4 quantization for the prefilling stage to accelerate computation while preserving BF16 precision for decoding to maintain accuracy. The method achieves up to 3x speedup in prefilling with minimal performance degradation on agentic benchmarks.

Bits Under ZK-LLM: Evaluating Zero-Knowledge-Friendly Quantization for Verifiable Private LLM Inference

arXiv cs.AI

This paper presents the first systematic study of zero-knowledge-friendly quantization for LLMs, formalizing the concept and evaluating nine models (including Qwen2.5-14B and Qwen3-30B-A3B) across weight, activation, and nonlinear lookup table precision. It finds that activation precision and RMSNorm inverse-square-root lookups dominate ZK proving costs and that conventional low-bit quantization heuristics do not translate to proportional proving savings.

HyQuant: Hybrid-Precision Quantization for LLM Attention

Hugging Face Daily Papers

HyQuant improves low-bit LLM attention quantization by preserving critical vertical-line tokens and local windows in high precision while quantizing the rest, maintaining accuracy with low overhead.

LLM Compression with Jointly Optimizing Architectural and Quantization choices

arXiv cs.LG

Researchers from UiT and University of Oslo propose a differentiable NAS framework that jointly optimizes architectural configurations and mixed-precision quantization for LLM compression, achieving up to 1.4× faster inference or 6% higher accuracy across seven reasoning tasks compared to sequential NAS-then-quantization baselines.