GLM 5.2 FP8 with FP8 KV - Terminal-Bench 2.1 = 79.8 (with one time-out that I didnt re-run)
Summary
Testing GLM 5.2 with FP8 quantization and FP8 KV cache on H200 yields a score of 79.8% on Terminal-Bench 2.1, with one timeout not rerun.
Similar Articles
4-bit GLM-5.2 (753B MoE) on 4× DGX Spark: 70.8% on Terminal-Bench 2.1 vs 81.0% for the full model
Running a 4-bit quantized version of GLM-5.2 (753B MoE) on 4 DGX Spark machines achieves 70.8% on Terminal-Bench 2.1, compared to 81.0% from the full model.
@0xSero: We found a way to run GLM-5.2 with full context in vLLM without pruning. - top 32 experts NVFP4 - rest fp3 - intel auto…
A community researcher enabled running GLM-5.2 (753B parameters, all 256 experts) in vLLM without pruning via a hybrid quantization (NVFP4, NF3, MXFP8), fitting on 4×96GB GPUs with ~307k KV cache and near-FP8 accuracy.
GLM-5.2 on 8xB200: the deployment math nobody spells out - NVFP4 + 2x TP=4 replicas should beat TP=8 by ~2x. Full config guidance inside.
The article provides the optimal deployment configuration for GLM-5.2 on 8xB200 nodes, showing that NVFP4 with two TP=4 replicas achieves roughly 2x throughput over FP8 TP=8, with detailed performance data and caveats.
Benchmarking Self-Hosted Gemma 2 9B vs. Frontier APIs: The FP8 Quantization Prefill Tax and VRAM Realities on an NVIDIA L4 [P]
This benchmark compares an unquantized Gemma 2 9B model with an FP8 quantized variant on an NVIDIA L4 GPU, revealing that FP8 quantization introduces a prefill tax (higher TTFT) but improves decoding latency and VRAM usage, with minimal semantic drift for narrow tasks.
@RedHat_AI: Quantized checkpoints for GLM-5.2 have been created by the Red Hat AI team! https://huggingface.co/RedHatAI/GLM-5.2-NVF…
Red Hat AI team released quantized checkpoints for GLM-5.2 using NVFP4 and FP8 quantization, reducing model size by over 70% while maintaining high accuracy on GPQA. The quantized model, paired with the DSpark speculator, enables efficient deployment with vLLM.