Delayed but here it is. DeepSeek-V4-Flash-0731 in GGUF at 2.88 bits per weight, 95 GiB, fits and runs on one 128 GB box. Config-I hybrid quant: 2-bit experts, WHT-rotated 3-bit attention, routing kept at full precision. Wikitext PPL 15.6.

Model: https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF…

Operating guide, including the backend traps that will bite you: https://github.com/TheTom/offlabel/blob/main/models/deepseek-v4-flash-0731.md…

Thanks Starlink for taking 3 days to upload!

thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF · Hugging Face

Source: https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#deepseek-v4-flash–config-i-ggufDeepSeek-V4-Flash — Config-I (GGUF)

284B-parameter MoE (21B active) in 95 GiB — fits and runs on 128 GB unified-memory boxes (DGX Spark, Mac).Config-I hybrid quantization at2.88 bpw effective.

Config-I quantization ofdeepseek-ai/DeepSeek-V4-Flash-0731— the official 2026-07-31 release, superseding the preview — (43 layers, 256 routed experts top-6, hash routing, MLA attention, mHC hyper-connections, DSA indexer). Sibling release:Config-I MLXfor Apple Silicon.

📖**How to actually run it well:**theofflabel operating guidecovers the backend traps, sampling, the repetition-loop fix, and the DwarfStar speed comparison in one place.

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#%E2%9A%A0%EF%B8%8F-runtime-requirement⚠️ Runtime requirement

This GGUF uses TurboQuant weight types (TQ3\_1S) and requires theTurboQuant llama.cpp fork(deepseek4-merge branch or later). Stock llama.cpp will not load it.

**Metal (Mac) users:**run withTQ\_NO\_ROTATE=1in the environment. The fused rotate-act Metal kernel currently produces incorrect batched-prefill results on this architecture (decode is unaffected); the env var routes through the safe dequant path at a modest prefill-speed cost.

**CUDA users:**use the fork at or past thedeepseek4\-mergebranch tip — earlier fork builds route TQ3_1S through a fused kernel that produces incorrect output on this architecture (fixed by routing through the verified dequant path; costs some decode speed, correctness first).

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#recipeRecipe

Tensor groupTypebpwExpert gate/up (all layers)Q2_02.25Expert down, attention, indexer, compressorTQ3_1S (WHT-rotated)4.0Token embeddingsq8_08.5Output headq6_K6.6Router gates, mHC mixing, APE, norms, sinksf32/bf16—

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#compressionCompression

SizeFP8/FP4 source167 GBBF16 equivalent~570 GB**Config-I hybrid (2.88 bpw)**95 GiB (102 GB), 3 split files ≤45 GB

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#qualityQuality

• Wikitext PPL:15.64 ± 0.38(ctx 512, 60 chunks) ·12.86 ± 0.29(ctx 2048, 15 chunks) — measured on CUDA (GB10)
• Coherent greedy generation verified on CUDA, CPU, and Metal.

Observed behaviors at 2.88 bpw(honesty notes, measured under heavy I/O contention — quiet-condition re-runs pending):

• Use the official sampling settings:temperature 1.0, top_p 0.95(DeepSeek’s card). Greedy decoding (temp 0) induces repetition loops on this model family and is not representative.
• The post-</think\>content stream can degenerate into repetition while the correct answer sits intact in the reasoning block — parsereasoning\_contentwhen serving with reasoning enabled (observed under greedy decoding; official sampling mitigates).
• Repetition on hard symbolic math was observed under greedy decoding only — being re-evaluated at official sampling settings.
• Broader MMLU/NIAH numbers land here after quiet-condition runs complete.

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#exampleExample

TQ_NO_ROTATE=1 ./build/bin/llama-cli -m DeepSeek-V4-Flash-0731-ConfigI-00001-of-00003.gguf \
  -ngl 99 -c 8192 --temp 1.0 --top-p 0.95 -p "your prompt" -st < /dev/null

https://huggingface.co/thetom-ai/DeepSeek-V4-Flash-ConfigI-GGUF#licenseLicense

Inherits the DeepSeek model license from the base model. Quantization bythetom-ai.