Same two GB10s. GLM-5.3-Flash, two stacks. One streamed call each, temperature 0.2, 384 tokens, prompts around 3.7k tokens.

Entrpi’s kit on Brandon’s K4 plus the lab MXFP8 drafter: https://github.com/Entrpi/glm-5.3-flash-exl3-2x-spark… https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw… https://huggingface.co/local-inference-lab/GLM-5.3-Flash-DFlash2…

TensorFold on the neko-legends EXL3 plus incoai’s DFlash2: https://github.com/jayleaton/glm53-tensorfold-spark… https://huggingface.co/neko-legends/GLM-5.3-Flash-Uncensored-EXL3… https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2…

Prose decode was 27.4 tok/s on Entrpi and 35.8 on TensorFold. Code at reasoning low was 53.6 and 36.3. Code at reasoning max was 62.7 and 44.6. Prefill on the prose prompt was 1,369 tok/s and 1,342. On the code prompt, 1,306 and 1,364. Context cap is 524,288 on Entrpi and 1,048,576 on TensorFold.

Plain prose and the context window went to TensorFold. Code stayed with Entrpi because that drafter kept more tokens per step, about 6.5, and about 7.4 once reasoning was maxed. Different checkpoints, so the gap is the stack and the quant together.

Entrpi/glm-5.3-flash-exl3-2x-spark

Source: https://github.com/Entrpi/glm-5.3-flash-exl3-2x-spark

GLM-5.3-Flash EXL3 on two DGX Sparks

Run GLM-5.3-Flash, a 320-billion-parameter open model, on a pair of NVIDIA DGX Spark desktops as a private, OpenAI-compatible API. One installer and one large download, and you have a model of frontier-class scale answering on your own hardware with 1.3 million tokens of context memory at full quality (1.7 million with the compact-memory option), vision, tool calling, and quality at parity with the official release.

Status: community derivative. Not an official image of any upstream project. Built and maintained by Entrpi; report problems on this repo’s issue tracker before assigning them upstream.

What you get

Every number above is measured, not estimated; the tables further down say how.

Quick start

You need two DGX Sparks connected directly by their QSFP ports, Docker with GPU support on both, and passwordless SSH from the box that will serve the API (the head) to the other one (the worker).

On the head:

git clone https://github.com/Entrpi/glm-5.3-flash-exl3-2x-spark
cd glm-5.3-flash-exl3-2x-spark
cp .env.example .env    # set the worker's LAN address and the two rail IPs/interfaces
./install.sh

The installer checks both machines, pulls the ~25 GiB serving image, downloads the ~176 GiB model and ~1.3 GiB drafter, installs the launch scripts, starts the worker and then the head, warms up, and runs a test request. Every step is safe to re-run. Downloads dominate the first run; after that a restart takes about four minutes to a live API.

./install.sh --nfs             # keep the weights only on the worker (head needs ~30 GiB disk)
./install.sh --skip-download   # weights already in place
./install.sh --help            # all options and exactly what the script touches

Then talk to it:

curl -s http://<head-ip>:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm-5.3-flash",
  "messages": [{"role": "user", "content": "Summarize the attached contract in five bullets."}]
}'

Any OpenAI SDK works with base_url="http://<head-ip>:8000/v1" and the model name glm-5.3-flash.

Requirements

Using the model

Reasoning is off by default so answers arrive as plain content. Turn it on per request and the reasoning comes back in message.reasoning, never mixed into the answer:

{"model": "glm-5.3-flash", "messages": [...], "chat_template_kwargs": {"enable_thinking": true}}

Tool calling follows the OpenAI tools / tool_choice schema and works with reasoning on or off. Images go in as standard image_url content parts. Greedy decoding (temperature: 0) is fully supported and is what the quality numbers below use.

Long documents. A 133k-token prompt is processed in about 90 s, a 500k-token one in about 6.5 minutes. Repeated turns on the same document reuse the work already done: a follow-up question on a 15k-token conversation answers in about 3 s instead of 12.

Capacity. What limits you is the attention-memory pool, 1,287,194 tokens at the default quality, shared by everything in flight; the per-request maximum (MAX_LEN, 524,288 by default) is just a cap on how much of it one request may take. The default configuration admits up to four requests at once and aggregate throughput rises with concurrency (about 47 tokens/s across four chats). For many short concurrent sessions, or for 1M-token requests, see the configurations below.

Choosing a configuration

The defaults are the validated production setup. Everything else is an environment variable set in .env (or on the launch command line) on both boxes. Pick by what you serve:

Knobs used above: MAX_LEN is the longest request accepted; MAX_SEQS how many requests run at once; MNBT how many prompt tokens are processed per step while reading a document (smaller keeps chats smoother, larger reads faster); SPEC=none turns the speculative drafter off; KV_DTYPE picks the attention-memory format; the two MIXED_PREFILL_* knobs control how document reading and live chats share the GPU.

The memory budget for context (KV_CACHE_MEMORY, 14.4 GB) and the GPU memory fraction (GMU, 0.85) were validated for stability across the configurations above on lightly loaded headless Sparks that had under 6 GB of system memory in use before the server started; the head then serves with about 1–3 GB to spare. DGX Spark unified memory does not fail gracefully when oversubscribed, so a box with more resident services, a desktop session, or a different firmware state may need a smaller budget: lower KV_CACHE_MEMORY (each 1 GB is about 90k pool tokens) and check headroom with tools/memlog.sh before raising anything.

Operating it

Measured performance

Single request, greedy decoding, on the default configuration. Each figure is the median of five 400-token runs; the range shows the run-to-run spread (the drafter accepts more or fewer tokens depending on the exact wording the model chooses, so two runs of the same prompt differ by a few percent):

Speed depends on how predictable the text is: the model runs a small drafter that proposes up to seven tokens per step and keeps the ones the full model agrees with, so repetitive or structured output runs two to three times faster than free prose. Temperature and reasoning mode do not change this.

Several requests at once, each with its own 16k-token prompt, all submitted at the same moment (low-predictability synthetic text, so per-request decode sits below the prose figure above). Time to first token here includes waiting for the other prompts to be read; a short prompt on an idle server answers in about 0.4 s. Measured on this release with MAX_SEQS=10 so all ten could be admitted:

The agentic configuration (no drafter, 12–16 streams) reaches 88–103 tokens/s aggregate at 8k-token contexts.

Context capacity by configuration (tokens of attention memory available, and how many maximum-length requests fit at once):

Measured quality

Greedy decoding, local harness with robust answer extraction, on the default configuration:

An independent KL-divergence panel puts the 4-bit weights at parity with the official FP8 release; the math and retrieval gates above are at parity with an unquantized-attention-memory run of the same model on this hardware.

How it compares

MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks serves the same model on the same hardware through a different engine. On their own benchmark protocol (vendored unmodified as scripts/bench_decode_miaai.py):

Both projects arrived at the same three essential fixes independently; the differences are in engine lineage, memory format, and how much context is kept available per request. Full analysis in docs/COMPARISON.md.

How it works

The two Sparks split the model between them (tensor parallel over the 200GbE link). The 288-expert mixture-of-experts weights are served directly from a 4-bit trellis-quantized checkpoint, with only the parts a token actually routes to read from memory. A one-billion-parameter drafter (DFlash2, installed as the 8-bit copy published by local-inference-lab, which accepts exactly as many tokens and runs a few percent faster) proposes blocks of tokens that the full model verifies in one pass. Attention memory is stored as compact 8-bit records so that half a million tokens of context fit, and the whole decode step is captured as CUDA graphs. The engine is a vLLM fork branch, Entrpi/vllm-glm-5.3-flash-spark, baked into the image ghcr.io/entrpi/glm-5.3-flash-exl3-2x-spark.

The engineering record, including what did not work, is docs/FINDINGS.md; the image’s exact provenance (base, commits, build command, patches) is docs/BUILD.md.

Repository layout

install.sh                      one-shot installer (run on the head)
.env.example                    topology and knobs; copied to .env on first run
scripts/
  launch-glm53-vllm-tp2.sh      per-box launcher (0 = head, 1 = worker)
  glm53-warmup.sh               post-boot warm-up
  bench_glm53.py                single-stream / 5-stream / first-token benchmark
  bench_workloads.py            decode speed by workload: median, spread, acceptance
  bench_decode_miaai.py         MiaAI's protocol, vendored for comparability
  sweep_runner.py               concurrency sweep with identical content per arm
  saturation_bench.py           scheduler-pressure benchmark
  longctx_smoke.py              long-prompt processing + drafting probe
  warm_prefix_test.py           repeated-turn reuse test
  accept_ab_matrix.py           temperature × reasoning acceptance A/B
  check_image.sh                image self-containment verification
  run_tests.sh                  unit tests inside the pulled image
tools/
  dflash_equiv.py               speculative-decoding equivalence harness
  memlog.sh                     memory-headroom sampler
docs/
  FINDINGS.md                   engineering record, measurements, negative results
  COMPARISON.md                 cross-stack comparison with the MiaAI recipe
  BUILD.md                      image provenance: commits, build args, patches
  ANNOUNCEMENT.md               community publishing disclosure

Update check

At start, the head’s launcher reads one line (the current release name) from this repository at most once a day and prints an upgrade hint if it is newer than the release you have. Nothing about your machine or usage is sent, the check has a 3 s timeout and never affects the launch. Turn it off with GLM53_NO_UPDATE_CHECK=1 in .env. ./install.sh --check-update does the same on demand and ./install.sh --version shows what is installed.

Reproducing the numbers

python3 scripts/bench_glm53.py 3                                  # single/5-stream/TTFT, 3-run medians
python3 scripts/bench_workloads.py 5 400                          # per-workload table: median, spread, acceptance
python3 scripts/bench_decode_miaai.py --phase structured --structured --runs 5 --max-tokens 400 --skip-coherence
python3 scripts/bench_decode_miaai.py --phase prose --runs 5 --max-tokens 400 --skip-coherence
python3 scripts/sweep_runner.py --arm mine --cells 1,2,4          # concurrency table
python3 scripts/longctx_smoke.py                                  # long-prompt processing
python3 tools/dflash_equiv.py                                     # speculative equivalence
bash scripts/run_tests.sh && bash scripts/check_image.sh          # GPUs must be free

Quality gates (math_500, GPQA-diamond) used a local harness with greedy decoding and robust answer extraction, n = 50–100 per gate; methodology in docs/FINDINGS.md.

Related work

Acknowledgements

• brandonmusic — the 4-bit quant this recipe serves
• IncoAI — the DFlash2 drafter
• local-inference-lab — the 8-bit (MXFP8) drafter copy and the NVFP4 model conversions
• eugr — the spark-vllm-docker build system the image is built with
• Mia’s AI Lab — the parallel recipe, the benchmark protocol, the checkpoint mirror, and several operational ideas carried over from it
• tonyd2wild, jack6464, malaiwah — forum groundwork and the quality panel
• local-inference-lab, tpurtell, turboderp — the runtimes underneath

License

This repository (installer, scripts, docs) is MIT. The vLLM branch inherits Apache-2.0. Model and drafter weights keep their upstream licenses; the DFlash2 drafter is CC BY-NC-ND 4.0 (research/eval), so the installer downloads it from its source repository and this project never redistributes it.