GLM-5.2 - A Story on Compression

Here’s how a community of tinkerers got GLM-5.2 running at near lossless quality on a 15,000$ budget. This article was put together using tens of thousands of analyzed messages in the RTX Pro 6000 Discord server.

Memory is expensive $$

GLM-5.2 is a large mixture-of-experts model. 753B to be precise, in BF16 that is 1.56TB, before KV Cache, and optimisations. Running this would require about 2TB of high banwidth memory.

You’d need a minimum of 16 DGX Sparks, about 70,000$ at the very least.

0xSero@0xSero·Jun 20I want2131388.1K

FP8 & NVFP4 help reduce that a little bit. By quantising each of the weights from 16 bits to 8 bits we can save on 50%~ of the costs, 4 bits bring us down to only 25% of the memory requirements for the weights.

For about a year this was the floor, getting the model down below 4bits didn’t work well if at all, especially not with the best inference engines. I’d recommend reading this to learn more about SGlang, and VLLM especially.

Ahmad@TheAhmadOsman·May 21 ArticleInference Engines for LLMs & Local AI Hardware (2026 Edition)You don’t pick an inference engine first. You pick a hardware strategy, a workload shape, and a serving model. The engine follows. That is the most useful way to think about LLM inference engines….402571.6K997K

REAP Pruning models to get below 4 bits.

Cerebras’s Research team published this lovely paper on pruning experts from MoEs. https://arxiv.org/abs/2510.13999 They found that pruning would let us further cut down on the size of a model while maintaining 4bits for what remains.

This pruning could be targetted, you run “observations”, collect expert activations and saliency scores based on your calibration dataset. You then preserve or prune whatever you’d live.

This allowed us to about 82% compression for the weights while maintaining a stable coding agent experience. REAP is sacrificing something, whether it be world knowledge, or reasoning quality, or even model confidence and for a while that was the lowest you could go.

In comes GLM-5.2

Once ZAI released their latest flagship, everyone who had any hardware immediately started working towards shrinking the beautiful model as much as possible.

Everyone in the discord was scheming and experimenting on reducing the footprint to get this model into 4x RTX Pro 6000 (50,000) and 3x DGX Sparks (15,000)

GGUF trading speed for VRAM

Llama.cpp has long been able to squeeze most models onto consumer hardware, even down to 2 bits, that brings the memory needed for weights to a tiny 12.5% of what BF16 needs.

Unsloth has also been unmatched at retaining model intelligence at these low bit compressions. You should definitely try it out if your hardware permits, it’s easy to get started on LMStudio

Unsloth AI@UnslothAI·Jun 18GLM-5.2 can now be run locally!

The 2-bit model retains ~82% accuracy after we shrunk it from 1.51TB to 238GB (-84% size).

Run on a 256GB Mac or RAM/VRAM setups.

GLM-5.2 is the strongest open model to date.

Guide: https://unsloth.ai/docs/models/glm-5.2… GGUF: https://huggingface.co/unsloth/GLM-5.2-GGUF…Z.aiQuoteZ.ai@Zai_org·Jun 17Introducing GLM-5.2: Frontier Intelligence, Open Weights

• Significant improvements in coding and agentic tasks
• Strong long-horizon capabilities with a 1M context window
• Two levels of reasoning effort: GLM-5.2 (max) pushes the limits, while GLM-5.2 (high) strikes a strong2731.1K7.3K1.8M

GGUFs and Lllama.cpp are Slow.

While it just works, we’re sacrificing a ton of speed, prefill can be significantly worse on Llama.cpp, and concurrency is just not well supported.

These days each harness spawns subagents, and we often have 2-8 agents running in the background at all times, Codex and Claude Code have really revolutionised automating work.

To have that locally you need not only incredible intelligence, but throughput also often matter even more. So Llama.cpp was not an option.

I want to say I have nothing but respect for llama.cpp and all the innovations they’re sharing with the world, it’s a great place to start and the best “just works” engine out there right now.

REAP First

The first of the VLLM compatible compressions were a series of GLM-5.2 reaps I have produced. I calibrated on a mix of coding, agentic, philosophy, religious scripture, and scientific knowledge. Cutting off 34% of the experts.

In total the REAPs amassed 35k downloads in a month, or about 8% of what the original 16 bit model received. Once pruned I quantised to NVFP4, GGUF, and Intel’s W4A16

**The REAP scored a whooping: **

• 70.8% on terminal-bench-2.1
• 91% on aider-polyglot
• 86% on GPQA Diamond

However lost about 30 points on MMLU Pro, and had a tendency to loop on non-preserved topics. For a while this was the adopted standard by the community.

It allowed us to use NVFP4 which preserves much of the intelligence, and provides a modest speed boost for Blackwell cards. Here u can see how it built a minecraft clone.

0xSero@0xSero·Jun 22GLM-5.2-REAP-NVFP4

This experiment I calibrated the model on 30,000+ samples of my agent sessions, tweets, writing, and codebases.

Pretty charming, every dataset produced something different.20936522K

0xSero@0xSero·Jun 18GLM-5.2 ONE SHOT - REAP + NVFP4

This is the best 1 shot minecraft clone I’ve seen built locally.

• Night day system
• Full item system
• Animals
• Infinite environment
• Beautiful blocks
• Character moves
• Smooth controls

–– 1 bug

I can’t click mine/craft261319814K

Next innovation, Mixed bit inference.

We talked a bit about how quantisation allows us to shrink the model, and fit it on consumer hardware. In April 2026 Deepseek released DS4-Flash An FP8/FP4 mixed bit model with excellent performance, of course this isn’t new.

Dynamic GGUFs (like the ones Unsloth releases) have been popular solutions for minimizing size while maximizing retained intelligence, basically you quantise different parts of the model based on how important those bits are.

In comes this beautiful majestic mismash of compression standards. https://huggingface.co/madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid

madeby561 took GLM’s model weights, and compressed 192 of the experts to NF3, left 64 of the highest saliency experts in NVFP4 and quantised the remainder of the model to MXFP8 (gates, attention, etc.)

Immediately we saw significant improvements. This all required custom kernels, and the collaboration of hundreds of people working day and night on Discord.

Weights are only part of the picture.

Once you can fit the model weights in memory, the next challenge is the KV-Cache, we managed to get the weights down from 1.51TB to 300GB, an 80% compression. While maintaining most of the model’s intelligence.

No pruning, which is the most important thing here. This immediately eliminates a whole class of bugs and challenges.

0xSero@0xSero·Jul 8GLM-5.2 on 4x 6000s in ZCode

/goal build a nintendo ds emulator in C++, do not use a ready made emulator. Don’t stop until you can run the Mario 64 DS rom.241632339K

KV Cache, also must be chopped.

By default in VLLM, a copy of the kv cache must fit on each GPU, this speeds up inference since it minimizes moving data over PCIe, this however costs a lot of VRAM, for the adopted Hybrid mix we were getting about 200k max tokens, which was fine, but not ideal.

By instead opting to split our KV-Cache across the GPUs we climbed to about 370,000 tokens of kv-cache in FP8. https://docs.vllm.ai/en/latest/serving/context_parallel_deployment/#decode-context-parallel

There’s still much room to compress, just like we can compress the WEIGHTS into NVFP4 and even NF3 so can we compress the KV Cache: https://github.com/vllm-project/vllm/issues/32220

The community ran various evals and benchmarks and got to 740k token context in 384GB of memory, this allow for higher concurrency, longer sessions, and even slightly faster inference.

ExllamaV3, TR3 and KL Divergence

KL divergence is a tool we can use to measure model quality. Feed the same fixed tokens through the BF16 teacher and the compressed build, then compare the probability distribution each assigns to the next token.

I have used and found EXL3 quants and Exllamav3 to be a reliable inference engine for serving models below 4 bits with maximal quality. https://github.com/turboderp-org/exllamav3

Trellis-coded quantization

Usually you take every weight, round it down on its own, and hope the errors do not add up too badly. Trellis coding looks at a whole chunk of weights together and picks the best compressed path through them. That lets it spread the damage around instead of ruining one important weight or direction.

So with TR3 we can shove the less important experts down to roughly 3 bits, without deleting them like REAP does. They are still there, just stored in a much more clever compressed form.

QTIP is the research method behind this style of compression: Quantization with Trellises and Incoherence Processing.

It first scrambles the weight space a little, so no one sensitive direction gets all the quantization damage. Then it uses trellis coding to compress whole groups of weights together instead of rounding each one individually.

EXL3 is a practical, faster-to-use version of that idea. TR3 uses it to keep the tail experts at around 3 bits rather than removing them.

https://huggingface.co/brandonmusic/GLM-5.2-NVFP4-TR3-Hybrid

GLM-5.2 in 2 bits? moet

vLLM-Moet takes the huge expert matrices, stores them as sign-symmetric 2-bit weights, then uses an FP4 “delta” cache for hot experts. If the router is uncertain,** it can replay that token with the higher-precision expert weights.**

So the normal path is cheap W2; the rescue path is FP4.

It also has the wild part: when even W2 does not fit in VRAM, experts can live in pinned RAM or NVMe, while the GPU holds the hot working set. A miss fetches the needed experts, then replays the token. That is how it claims to run GLM-5.2 on two 96 GB cards.

We benchmarks Deepseek-v4-flash on a single RTX Pro 6000 @ 100 tok/s on GPQA diamond, and scored nearly identical to NVFP4 with only 1 question failing in the moet method.

Moet doesn’t require static quantisation or pruning, it runs the original model weights, just electing to only store a lightweight cache unless need be. This method is much newer, but I recommend you look into it.

You can now own the frontier.

50,000$ is a lot of money, but 12,000-15,000 is more doable and so more and more people will go local. Sparks are running this model at up to 30 tok/s, that’s enough to do real work.

We all know that increasingly we depend more on these tools, I wanted to let you know we are much better than where we started.

It seems not so distant that we will all own our models.