🧠⚡ Running GLM-5.2 at home the FULL 744B, all 256 experts, UNPRUNED across 4× NVIDIA DGX Spark (GB10).

200K context · MTP spec-decode · fp8 sparse-MLA KV · vLLM native multi-node ~30 tok/s single-stream, 60 tok/s @ 6 concurrent.

Full replicable recipe, open-sourced 👇

tonyd2wild/GLM-5.2-QuantTrio-200K-4x-DGX-Spark

Source: https://github.com/tonyd2wild/GLM-5.2-QuantTrio-200K-4x-DGX-Spark

GLM-5.2 (unpruned QuantTrio Int4-Int8Mix) at 200K context on a 4× NVIDIA DGX Spark (GB10) cluster

A complete, follower-replicable recipe for serving GLM-5.2 — the unpruned QuantTrio GLM-5.2-Int4-Int8Mix checkpoint, all 256 experts intact — across 4× NVIDIA DGX Spark (GB10) nodes over a RoCE fabric, with 200,000-token context, MTP speculative decoding (k=4), fp8 sparse-MLA KV cache, and full CUDA graphs. vLLM native multi-node (no Ray), one plain docker run per node.

Measured: 28.8 tok/s single-stream (median), 60.5 tok/s aggregate at 6 concurrent, 200K context, unpruned.

Measured performance (final, 2026-07-05)

(512-token generations, temp 0, low-depth context. c1 is the warm median of 3 runs — 27.3/29.0/28.8; the cold first request after boot reads lower (16–22 tok/s). The c3+ rows exist only because of patches/fix-indexer-mtp-overhang.py — unpatched vLLM crashes at 3 concurrent. Context-depth tables (16K/32K prefill/decode) are the one remaining PENDING item — see Benchmarks.)

1. Credits & lineage

This recipe stands entirely on the shoulders of the people below. If you use it, their work is what you are using.

Contributions from this deployment (things we found during bring-up, offered back):

• The indexer MTP-overhang fix (patches/fix-indexer-mtp-overhang.py): the DSA indexer under-sizes its expanded block-table buffer by one block when max_model_len is an exact multiple of the block size and MTP is enabled — crashes the engine at ≥3 concurrent requests.
• The load-phase page-cache-drop procedure (Gotcha 6): periodic drop_caches on every node during weight load, which unsticks GB10 kernel-reclaim stalls.
• The memory-budget numbers for exactly 200K context on this checkpoint (Gotcha 7 and the config table): gmu 0.91 + --kv-cache-memory-bytes 10950000000.

Forum threads (read both — they are the primary sources):

• https://forums.developer.nvidia.com/t/glm-5-2-on-a-4x-gb10-cluster-22-tok-s-decode-256k-ctx-recipe/374125
• https://forums.developer.nvidia.com/t/followup-mystery-solved-4x-spark-glm-5-2-nfp4-24tp-s-128k-ctx-no-reap/375416

2. Hardware requirements

• 4× NVIDIA DGX Spark (GB10) — 121 GB unified memory each.
• RoCE fabric between the four nodes. We use a CRS812 switch, fabric subnet 192.168.192.0/24, MTU 9000 (jumbo frames) end-to-end — on the switch AND every NIC. A direct-cabled mesh works too if NCCL can see the IB devices.
• ~420 GB free disk per node — 405 GB of weights plus image, caches, and slack.

3. Why this checkpoint

QuantTrio/GLM-5.2-Int4-Int8Mix — unpruned.

• Zero quality compromise. No REAP/expert pruning: all 256 experts intact. You serve the actual model, not a surgically reduced one.
• Real memory headroom. 405 GB of weights → ~95 GiB weights per node on TP=4. On a 121 GB unified-memory GB10 that leaves genuine room for KV cache, CUDA graphs, and the OS. Contrast the 429 GB NVFP4 hybrid: ~107 GB/node — a knife-edge that OOMs the moment page cache or warmup allocations breathe on it.
• MTP drafter is in-checkpoint (layer 78). No separate drafter model to download, align, or version-match — --speculative-config just points at the checkpoint itself.

4. Repo contents

Not vendored: the 10 Triton kernels. Get them from the upstream repo, CosmicRaisins/glm-5.2-gb10 (Apache-2.0), directory kernels/. The 10 files you need:

b12x_sparse_helpers.py
deepseek_v2.py
flashmla_sparse.py
patch_flashmla_ops.py
sm12x_deep_gemm_fallbacks.py
sm12x_mqa.py
sm12x_sparse_mla_attn.py
sparse_attn_indexer.py
sparse_mla_env.py
sparse_mla_kernels.py

5. Step-by-step

Steps (a)–(b) and (h) run on one build machine (any of the Sparks works). Steps (c)–(f) touch every node. Step (g) runs from the head node only. Do step (h) — the indexer patch — as part of the same bake as (b), before you distribute the image in (c).

a. Build the vLLM image (~35–60 min)

Clone eugr/spark-vllm-docker and build against the pinned vLLM commit:

git clone https://github.com/eugr/spark-vllm-docker
cd spark-vllm-docker
./build-and-copy.sh --vllm-ref ab666069935c1f23e8ef56038b4659ac9e8f19f8 \
  -t vllm-node-tf5-glm52-b12x:probe --tf5

b. Bake the mods into the image

Get the 10 kernel files from CosmicRaisins/glm-5.2-gb10 kernels/ into ~/glm-triton on the build machine, then run both mod scripts inside a container and commit the result. The kernels must be mounted at /root/models/models15/glm-triton — that is the path mods/glm52-sm12x-sparse/run.sh expects (KERNELS= at the top of the script). Exactly as we ran it:

docker run -d --name glm52-modding \
  -v ~/glm-triton:/root/models/models15/glm-triton:ro \
  -v $(pwd)/mods/glm52-sm12x-sparse:/mods/glm52-sm12x-sparse:ro \
  -v $(pwd)/mods/glm52-b12x-sparse:/mods/glm52-b12x-sparse:ro \
  vllm-node-tf5-glm52-b12x:probe sleep infinity

docker exec glm52-modding bash /mods/glm52-sm12x-sparse/run.sh
docker exec glm52-modding bash /mods/glm52-b12x-sparse/run.sh

docker commit \
  --change 'ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]' \
  --change 'CMD []' \
  glm52-modding vllm-node-tf5-glm52-b12x:probe-modded
docker rm -f glm52-modding

WARNING — two docker traps that both bit us:

1. docker commit inherits --entrypoint overrides from the patch container. If the container you’re committing was started with an entrypoint override (or with a bare command like sleep infinity), the committed image carries it forward and will not boot vLLM. Always commit with --change 'ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]' --change 'CMD []' (as shown above).
2. When piping stdin scripts into containers, use docker exec -i. Without -i the script silently no-ops — no error, nothing runs, and you commit an unpatched image.

Both scripts print ✓ lines; the sm12x one must end with === glm52-sm12x-sparse complete === and the b12x one must show a successful import b12x.

c. Distribute the image to all nodes

# from the build machine, for each OTHER node:
docker save vllm-node-tf5-glm52-b12x:probe-modded | \
  ssh <user>@<node> docker load

(Over the RoCE fabric this is minutes, not hours. pigz in the middle helps on slower links.)

d. Weights: download once, rsync to all nodes

Download the checkpoint once (405 GB — do this on the node with the best internet):

hf download QuantTrio/GLM-5.2-Int4-Int8Mix \
  --local-dir /var/tmp/models/glm52-int4-int8mix

Then fan out over the RoCE fabric (not your uplink):

# from the node holding the weights, for each other node's fabric IP:
rsync -a --info=progress2 /var/tmp/models/glm52-int4-int8mix/ \
  <user>@192.168.192.X:/var/tmp/models/glm52-int4-int8mix/

Create the hub-layout symlink on every node (the serve path is /cache/huggingface/hub/glm52-int4-int8mix inside the container):

mkdir -p /var/tmp/models/hub
ln -sfn ../glm52-int4-int8mix /var/tmp/models/hub/glm52-int4-int8mix

e. Stage NCCL 2.30.4 on every node

The image’s bundled NCCL is replaced at runtime via LD_PRELOAD. On each node:

pip download nvidia-nccl-cu13==2.30.4 -d /tmp/nccl --no-deps
mkdir -p /var/tmp/models/hub/nccl-2.30.4
cd /tmp/nccl && unzip -o nvidia_nccl_cu13-2.30.4*.whl 'nvidia/nccl/lib/libnccl.so.2'
cp nvidia/nccl/lib/libnccl.so.2 /var/tmp/models/hub/nccl-2.30.4/

f. Kernels on every node

Copy the 10 .py files from CosmicRaisins/glm-5.2-gb10 kernels/ to ~/glm-triton on every node (launch.sh bind-mounts them file-by-file over the vLLM tree, read-only):

git clone https://github.com/CosmicRaisins/glm-5.2-gb10
for node in 192.168.192.1 192.168.192.2 192.168.192.3 192.168.192.4; do
  rsync -a glm-5.2-gb10/kernels/ <user>@$node:~/glm-triton/
done

g. Launch

Edit the EDIT-marked config block in launch.sh (node IPs, SSH user/key, HCA + interface names), then from the head node:

./launch.sh --dry-run   # sanity-check the generated docker commands first
./launch.sh

Plain docker run per node; vLLM native multi-node (--nnodes/--node-rank), no Ray. Workers start headless first, head last. Expect ~12 min weight load + ~10 min cudagraph warmup before curl http://<head>:8210/v1/models answers.

h. Bake the indexer MTP-overhang patch (required for --max-num-seqs >= 3)

Do this during the same bake session as step (b), before committing and distributing the image. patches/fix-indexer-mtp-overhang.py fixes a vLLM bug where the DSA indexer sizes its expanded block-table buffer from max_model_len alone; MTP spec tokens can extend a request one block past it, and at ≥3 concurrent requests the engine crashes with RuntimeError: The expanded size of the tensor (3125) must match the existing size (3126). See the patch’s docstring for the full story.

VALIDATED (2026-07-05): with the patch verified in-image, the full concurrency sweep (c1–c6) completed with zero crashes — including the c3+ levels that reliably crashed an unpatched engine. See Benchmarks.

Bake it exactly the same way as the mods — mount it into the patch container and run it before the docker commit:

docker run -d --name glm52-modding \
  ... \
  -v $(pwd)/patches:/patches:ro \
  vllm-node-tf5-glm52-b12x:probe sleep infinity

# (after the two mod scripts from step b)
docker exec glm52-modding python3 /patches/fix-indexer-mtp-overhang.py

docker commit \
  --change 'ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]' \
  --change 'CMD []' \
  glm52-modding vllm-node-tf5-glm52-b12x:probe-modded
docker rm -f glm52-modding

It prints patched: .../indexer.py on success and is idempotent (safe to re-run).

6. Key serve config, with rationale

7. Gotchas (hard-won)

1. RoCE fabric IP must live on the right interface — and persist in netplan. If the fabric IP is added ad hoc, a link-local address (169.254.x.x) can squat the port after a reboot/link-flap, which shifts the GID table — your NCCL_IB_GID_INDEX now points at the wrong GID and NCCL either fails or silently degrades. Put the fabric IP in netplan and verify with show_gids after any reboot.
2. IB device passthrough is required. Without --device /dev/infiniband + --cap-add IPC_LOCK + --ulimit memlock=-1:-1, NCCL silently falls back to TCP over the socket interface. Everything works; decode is ~12 tok/s instead of 30+. If numbers look halved, check NCCL_DEBUG=INFO output for NET/IB vs NET/Socket.
3. Page-cache pressure on GB10 unified memory. Loading ~95 GiB of weights fills the page cache on a machine where CPU and GPU share the same 121 GB. Before launch, on each node: sync && echo 3 | sudo tee /proc/sys/vm/drop_caches. This plus explicit --kv-cache-memory-bytes is what makes boot deterministic.
4. Jumbo frames on the switch, not just the NICs. MTU 9000 must be set end-to-end; a switch port at 1500 silently fragments and craters bus bandwidth.
5. Don’t trust --gpu-memory-utilization alone. See the config table: pin --kv-cache-memory-bytes explicitly, or identical launches will OOM sometimes depending on what the page cache looked like at profiling time.
6. Page-cache thrash during weight load. On GB10, big loads stall at 100% CPU in kernel reclaim even with 14–18 GB “free”. Run an unconditional sync; echo 3 > /proc/sys/vm/drop_caches every 60s on every node during the load phase (and once right before launch). Symptom: shard progress freezes mid-load; a manual drop unsticks it within seconds.
7. KV budget for exactly 200K. --gpu-memory-utilization 0.90 leaves only 9.78 GiB for KV — 200000 ctx needs 10.19 GiB. Use gmu 0.91 with --kv-cache-memory-bytes 10950000000; boot allocates a 200,064-token pool.

8. Benchmarks

Final concurrency results (2026-07-05) — measured on this cluster with the final serve config (gmu 0.91, KV 10.95 GB, max-num-seqs 6, MTP k=4), on a boot with the indexer patch verified in-image. All runs: 512-token generations, temperature 0, low-depth context. All 6 concurrency levels completed with zero crashes.

Decode by concurrency (final)

Notes:

• c1 is the warm median of 3 runs (27.3 / 29.0 / 28.8). The cold first request after boot reads lower (16–22 tok/s) — quote the warm median, with this caveat.
• The c3–c6 rows exist only because of the indexer patch (patches/fix-indexer-mtp-overhang.py, step h): without it, the engine crashes at 3 concurrent requests. This run validates the patch — 6/6 concurrency levels, zero crashes.

Context depth (16K / 32K)

PENDING — the one remaining open item: prefill/decode tables at 16K and 32K context depth.

Boot telemetry (verified)

• Weights: 98.07 GiB per node.
• KV pool: 200,064 tokens, fp8_ds_mla.
• Steady state: MemAvailable 0.6–0.9 GB + 3.6–4.5 GB swap parked (by design — matches the upstream author’s memory profile for this config class).

9. License

Apache-2.0 — see LICENSE. Required and deliberate: launch.sh derives from CosmicRaisins’ Apache-2.0 launch.sh (copyright notice preserved in the file header), and the mods/ scripts replicate his Apache-2.0 mods. See NOTICE for attribution.