KronQ: LLM Quantization via Kronecker-Factored Hessian
Summary
KronQ is a post-training quantization framework that incorporates gradient covariance using a Kronecker-factored Hessian approximation, enabling bidirectional incoherence processing and improved sensitivity metrics for mixed-precision allocation. It achieves low perplexity even at 2-bit weight-only quantization on large models like LLaMA-3-70B.
View Cached Full Text
Cached at: 07/10/26, 06:16 AM
# KronQ: LLM Quantization via Kronecker-Factored Hessian Source: [https://arxiv.org/abs/2607.07964](https://arxiv.org/abs/2607.07964) [View PDF](https://arxiv.org/pdf/2607.07964) > Abstract:Post\-training quantization \(PTQ\) is a widely adopted technique for compressing large language models \(LLMs\) without retraining\. Existing second\-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer\-wise reconstruction objective\. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline\. Under the Kronecker\-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels\. \(1\) KronQ introduces bidirectional incoherence processing, extending the existing input\-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions\. \(2\) KronQ derives a new sensitivity metric for inter\-layer mixed\-precision allocation, driven by the gradient and activation Hessian traces\. Notably, in the case of 2\-bit weight\-only quantization on LLaMA\-3\-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations \(\>2000 perplexity on WikiText\-2\), KronQ achieves 7\.93 perplexity\. ## Submission history From: Donghyun Lee \[[view email](https://arxiv.org/show-email/b359ea7e/2607.07964)\] **\[v1\]**Wed, 8 Jul 2026 22:34:52 UTC \(5,916 KB\)
Similar Articles
REAL-Q: E2E LLM Quantization via Dynamic Gradient Descent
The paper proposes REAL-Q, a novel end-to-end aligned post-training quantization method for large language models that uses dynamic gradient descent to mitigate quantization errors, achieving significant improvements in KL divergence reduction over state-of-the-art methods.
KLQ: Training-free measured rotation quantization. Beats all training-free rotation-based quantization methods on W4A4KV4-bits. Llama 3.2 1B KLQ-quantized beats SpinQuant and gets close to ReSpinQuant without GPTQ/LDLQ rounding.
KLQ is a training-free LLM quantization method that allocates bits per direction based on measured KL divergence, outperforming existing training-free rotation-based methods on W4A4KV4-bit settings for models like Llama 3.2 1B and Qwen 2.5.
Qift: Shift-Friendly No-Zero W2 Post-Training Quantization for Rotated W2A4/KV4 LLM Inference
This paper introduces Qift, a fixed no-zero two-bit weight quantization level set designed for Hadamard-rotated LLMs, achieving improved W2A4/KV4 inference by leveraging the near-zero-centered Gaussian-like distribution of rotated weights. Experiments on LLaMA-2-7B and LLaMA-3.1-8B show consistent perplexity gains over standard W2 quantization.
Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
Introduces Recurrent Residual Quantization (RRQ), a post-training quantization framework that enables multiple effective precisions from a single LLM checkpoint via additive residual refinement, improving flexibility and construction speed.
CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
CAT-Q introduces a post-training ternary quantization method for LLMs that uses learnable modulation and softened ternarization, achieving superior performance over BitNet 1.58-bit while using only 512 calibration samples and scaling to 235B parameters.