GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
Summary
Proposes GEMQ, a global expert-level mixed-precision quantization method for MoE LLMs that uses linear programming and router fine-tuning to reduce memory and accelerate inference with minimal accuracy degradation.
View Cached Full Text
Cached at: 05/25/26, 08:59 AM
# GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs Source: [https://arxiv.org/abs/2605.23078](https://arxiv.org/abs/2605.23078) [View PDF](https://arxiv.org/pdf/2605.23078) > Abstract:Mixture\-of\-Experts Large Language Models \(MoE\-LLMs\) achieve strong performance but incur substantial memory overhead due to massive expert parameters\. Mixed\-precision quantization mitigates this cost by allocating expert\-wise bit\-widths based on their importance, approaching the accuracy\-memory Pareto frontier and enabling extreme low\-bit quantization\. However, existing methods rely on layer\-wise importance estimation and overlook router shifts induced by quantization, resulting in suboptimal allocation and routing\. In this work, we propose Global Expert\-level Mixed\-precision Quantization \(GEMQ\) to overcome these limitations via \(1\) a global linear\-programming formulation that captures model\-wide expert importance based on quantization error analysis, and \(2\) efficient router fine\-tuning to adapt routing to quantized experts\. These components are integrated into a progressive quantization framework that iteratively refines importance estimation and allocation\. Experiments demonstrate that GEMQ significantly reduces memory and accelerates inference with minimal accuracy degradation\. Source code is available at[this https URL](https://github.com/jndeng/GEMQ)\. ## Submission history From: Jianing Deng \[[view email](https://arxiv.org/show-email/49ac9fb1/2605.23078)\] **\[v1\]**Thu, 21 May 2026 22:23:38 UTC \(653 KB\)
Similar Articles
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
This paper introduces MODE, a modality-decomposed expert-level mixed-precision quantization framework for MoE multimodal LLMs that addresses biases in expert importance estimation by decomposing selection frequency by modality and filtering redundant vision tokens, achieving minimal performance loss under aggressive quantization.
BitsMoE: Efficient Spectral Energy-Guided Bit Allocation for MoE LLM Quantization
BitsMoE introduces a spectral-energy-guided bit allocation framework for quantizing Mixture-of-Experts LLMs, achieving substantial accuracy improvements and speedups under ultra-low-bit quantization.
GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
GEM reformulates LLM data curation as a variational problem on the hypersphere, using geometric entropy mixing and a minorize-maximize algorithm to discover balanced semantic clusters, achieving state-of-the-art improvements in data mixing strategies by up to 1.2% average downstream accuracy.
E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring
This paper introduces E-PMQ, an expert-guided post-merge quantization framework that addresses the combined deviations from merging and quantization, achieving significant accuracy improvements on multi-task merged models like CLIP-ViT and FLAN-T5.
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
Mix-Quant proposes a phase-aware quantization framework for agentic LLMs, using NVFP4 quantization for the prefilling stage to accelerate computation while preserving BF16 precision for decoding to maintain accuracy. The method achieves up to 3x speedup in prefilling with minimal performance degradation on agentic benchmarks.