@0xSero: We found a way to run GLM-5.2 with full context in vLLM without pruning. - top 32 experts NVFP4 - rest fp3 - intel auto…

X AI KOLs Following Models
glm-5.2 vllm quantization mixed-precision nf3-kernel hugging-face open-source

Summary

A community researcher enabled running GLM-5.2 (753B parameters, all 256 experts) in vLLM without pruning via a hybrid quantization (NVFP4, NF3, MXFP8), fitting on 4×96GB GPUs with ~307k KV cache and near-FP8 accuracy.

We found a way to run GLM-5.2 with full context in vLLM without pruning. - top 32 experts NVFP4 - rest fp3 - intel autoround calibration - MXP8 attention - bf16 routers and embedded sees On 3 DGX spark / quad RTX pro 6000 Credit to community https://t.co/QF5SQ0Tc6v
Original Article
View Cached Full Text

Cached at: 07/07/26, 10:18 AM

We found a way to run GLM-5.2 with full context in vLLM without pruning.

  • top 32 experts NVFP4
  • rest fp3
  • intel autoround calibration
  • MXP8 attention
  • bf16 routers and embedded sees

On 3 DGX spark / quad RTX pro 6000

Credit to community

https://t.co/QF5SQ0Tc6v

madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid · Hugging Face

Source: https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid#glm-52-mxfp8-nvfp4-nf3-hybridGLM-5.2-MXFP8-NVFP4-NF3-Hybrid

⚠️Usable — read the hazards.Loads ONLY through the custom serving image below (in-house NF3 3-bit kernel — no official vLLM/inference stack runs it). Serves on 4× 96GB sm120 GPUs with a≈307k-token KV cacheat DCP4 (2.44× the prior fp8-tier build) — run ≈118k context with concurrency headroom, or up to ≈240k single-stream. NOT the official model’s 1M, and not officially supported.

The full GLM-5.2 — 753B, all 256 experts per layer — on 4× 96GB GPUs. No pruning.The high-traffic experts ride NVFP4 byte-exact fromlukealonso/GLM-5.2-NVFP4(credit to Luke for the best-in-class low-KLD 4-bit tier); the long tail is compressed withNF3 — our in-house 3-bit expert format with a matching custom sm120 kernel. The non-expert layers (attention / dense / shared) areMXFP8(OCP microscaling FP8, built from the BF16 original) — served natively, not dequantized at load, which is what buys the large KV cache. ≈327 GB total — the only GLM-5.2 that fits 4 cards with a**≈307k-token KV pool**at DCP4.

https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid#accuracyAccuracy

Same protocol as the reference cards (temperature 1.0, top_p 0.95, max reasoning effort). GPQA-Diamond is measured onthis exact checkpoint; the remaining axes are not yet run.

PrecisionGPQA DiamondSciCodeIFBenchAA-LCRτ²-Bench Telecombaseline (FP8)89.5249.8574.9569.3897.9NVFP4 (full)89.3949.0475.8170.1398.25this model(326 GB, 4× 96GB)88.38pendingpendingpendingpendingREAP-594B (a prune, for contrast)86.8747.77*——— Read:on GPQA-Diamond the hybrid scores88.38 (175/198)— within ≈1 point (2 questions) of full-precision FP8/NVFP4, i.e. statistically tied at this sample size — whilebeating the REAP-594B prune by ≈1.5 points*. That’s the whole point of the build:keepingall 256 experts at mixed precision holds parity wherepruningexperts measurably loses ground. Baselines: nvidia/GLM-5.2-NVFP4 card.

https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid#output-drift-vs-fp8-kldOutput drift vs FP8 (KLD)

How far this model’s next-token distribution moves from theofficial FP8’s— asamenessmeasure,nota quality ranking (two equally good models still differ). Lower = closer to FP8. Prefill KL, 1.05M positions, top-8192 support.

modelsizedrift vs FP8 (KL ↓)fits 4× 96GB?official FP8753 GB—(the reference)❌ needs 8+ GPUsLuke NVFP4*(≈ FP8-grade quality)435 GB0.100❌ too bigthis model(rev 3, mxfp8)327 GB0.177✅ 118k–240k ctxthis model(rev 2, fp8 tier)326 GB0.172✅this model(rev 1, RTN-sourced)*326 GB0.198✅

Mind the reference.This table is drift vsFP8. It isnotthe drift-vs-BF16frame used by thertx6kpro wiki(where FP8 itself is ≈0.079 and Luke ≈0.068) — different baseline, don’t cross-compare the two tables.

Plain read: our outputs drift a bit more from FP8’s than the best 4-bit quant’s do, and we’re the only row that boots on four cards. In practice the model ranks the next token identically to FP8 ≈99.98% of the time (JS 0.035); the drift lives in confidence on near-ties. Rev 3 keeps rev 2’sbyte-identical expertsand only swaps the ≈8-bit non-expert tier fp8→mxfp8; its drift is measured at0.177vs rev 2’s 0.172 — the +0.005 is just the mxfp8 tier deviating slightly from FP8’s fp8 format (a vs-FP8 artifact, not a quality change: GPQA is unchanged at 88.38).

https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid#servingServing

Ready-to-run image — NF3 kernel, hybrid loader, native mxfp8 tier, and the high-concurrency indexer fix all baked in (built on the 2026-07-04 eldritch base):madeby561/vllm\-glm52\-nvfp4\-nf3\-hybrid:v2. Thedocker\-compose\.ymlin this repo is the exact serving configuration:

hf download madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid --local-dir ./glm52-mxfp8
MODEL_DIR=./glm52-mxfp8 docker compose up

Needs 4× 96GB sm120 GPUs + ≈64GB RAM. ServesMTP-5 / DCP4with a**≈301k-token KV pool**(native mxfp8 tier). The compose ships\-\-max\-model\-len 240000so you can actually use the long context — one ≈240k request fills the pool; lower it for more concurrency on shorter requests. Boots in ≈3 min (fastsafetensors). Single-stream decode ≈60-90 t/s (content-dependent); ≈250-600 t/s aggregate at 8-48 concurrent.

By madeby561 — built to answer one question: can a 4-card box run the full model with no lobotomy? On GPQA-Diamond, the answer is yes.

Similar Articles