18 HOURS LATER: DeepSeek-V4.1-Flash is now quantized to 4.75 bpw EXL3 for a 4× DGX Spark TP4 target. 🔥

The weights are DONE.

Now comes the fun part: making this monster actually run efficiently across four Sparks.

DeepSeek-V4.1-Flash is not a normal “748B model.”

Underneath it is a 552B MoE backbone with ~8B active during prefill and ~16B during decode, plus roughly 196B parameters of sparse Engram conditional memory.

Getting all of that through the EXL3 conversion path was a serious job.

𝗧𝗣𝟰 𝗘𝗫𝗟𝟯: 𝗖𝗢𝗠𝗣𝗟𝗘𝗧𝗘

Target: 4× NVIDIA DGX Spark

Quant: 4.75 bpw EXL3

Goal: quality-first TP4 deployment while keeping this architecture practical on a small local cluster.

The conversion itself took roughly 18 hours of compute from start to finish.

𝗧𝗛𝗘 𝟰𝟮𝟮 𝗚𝗶𝗕 𝗡𝗨𝗠𝗕𝗘𝗥 𝗡𝗘𝗘𝗗𝗦 𝗖𝗢𝗡𝗧𝗘𝗫𝗧

The full EXL3 pack on disk is approximately:

422 GiB total

Broken down into:

~233 GiB EXL3 model body ~189 GiB n-gram / Engram / PLE-style memory

But the plan is NOT to put all 422 GiB into Spark memory.

The large conditional-memory tables can live on NVMe.

So the TP4 target becomes roughly:

NVMe: ~189 GiB conditional-memory table

Across the 4 Sparks: ~233 GiB of EXL3 model weights

Which works out to roughly:

~58 GiB of model weights per Spark

That is a VERY different memory picture from “422 GiB loaded across four GPUs.”

And on DGX Spark we are talking unified memory, not conventional discrete VRAM, so that ~58 GiB weight residency per node still leaves substantial room for runtime state, KV cache, routing buffers and the serving stack.

Exactly how much usable headroom remains is something I’ll measure on the real TP4 boot rather than guess.

That is one of the biggest reasons I’m excited about this architecture.

The total parameter count is enormous, but the serving problem is much more interesting than simply dividing 748B by four.

And this would not have happened nearly as quickly without community support.

Huge thank you to @NeoAIForecast for funding the compute credits that let me get this conversion out for everyone using @runpod. 🙏

That kind of support directly turns into open weights, recipes and tooling the rest of the community can build on.

𝗧𝗛𝗘 𝗪𝗢𝗥𝗞 𝗜𝗦𝗡’𝗧 𝗢𝗩𝗘𝗥

This first recipe should be treated as PRELIMINARY.

The TP4 pack has NOT yet been booted and validated end-to-end on an actual four-Spark cluster.

That distinction matters.

Quantization complete ≠ serving validated.

@chrismfontes and @CountStaculaAI are already working with me on an improved version of the recipe and runtime path.

I’ll follow up with another post once we have real:

→ TP4 boot validation → actual per-Spark memory residency → NVMe Engram behavior → decode + prefill throughput → KV/cache headroom → speculative decoding → long-context scaling → quality receipts

No theoretical “it should fit” victory lap.

I want the actual receipts.

𝗔𝗡𝗗 𝗜 𝗪𝗔𝗡𝗧 𝗠𝗢𝗥𝗘 𝗗𝗘𝗩𝗦 𝗜𝗡 𝗛𝗘𝗥𝗘

If you work on EXL3, ExLlamaV3, vLLM, SGLang, distributed inference, DeepSeek runtimes or DGX Spark:

Please jump in.

The recipe is public and PRs are welcome.

If you see something that can be improved, don’t just tell me. Submit it. 😁

The goal is to make this useful for everyone.

𝗡𝗘𝗫𝗧: 𝗧𝗣𝟮

I’m not stopping at four Sparks either.

A more aggressive EXL3 targeting:

2× DGX Spark TP2

is already on the way.

That one is going to be VERY interesting.

@deepseek_ai

EXL3: https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw…

TP4 recipe: https://github.com/vcruz305/DeepSeek-V4.1-Flash-EXL3-DGX-Spark-recipe…

vcruz305/DSV4.1-Flash-EXL3-4.75bpw · Hugging Face

Source: https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#deepseek-v41-flash–exl3-sage-mixed-k-475-bpwDeepSeek-V4.1-Flash — EXL3 SAGE Mixed-K (4.75 bpw)

LoadableExLlamaV3pack ofdeepseek-ai/DeepSeek-V4.1-Flash.

Quantized byvcruz305with aninternal SAGEallocator: mixed EXL3Kper tensor (not a single uniform K) so bitrate goes where it protects quality, and stays lean where it does not. The method itself is not published. Kernels readKfrom each trellis tensor at load.

**Status: complete.**32 compiledmodel\-\*\-of\-00032shards + index.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#packPack

ItemValueFormatEXL3 (trellis) + native tables where notedAverage bpw4.75(453.07 GB over the 763B card: backbone + Engram + vision)Shardsmodel\-00001\-of\-00032…model\-00032\-of\-00032Indexmodel\.safetensors\.index\.json``quant\_method``exl3Routed expertsEXL3, mixed K (SAGE, quality-first)EngramNative FP8 retained (00031/00032)Protected non-expert tensorsCopied (attn / shared / DSpark / vision / head), not wholesale-requantized This is a compiled 8 GiB-class shard pack. It is not a dump of per-expert work files.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#what-sage-is-and-is-notWhat SAGE is (and is not)

SAGE is my internal sensitivity-aware EXL3 recipe for this model. Itmixes Ksinstead of stamping one K on every expert. Goal is highest quality at a TP4-class weight budget, not a write-up of the search.

Not in this card: the layer list, the scores, the capture/trace, or the allocator code. If a tensor is EXL3,Kis self-describing on the trellis.

This isnotan identity-Hessian / uncalibrated uniform-K dump labeled as SAGE.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#source-model-deepseek-not-this-packSource model (DeepSeek, not this pack)

V4.1-Flash is DeepSeek’s MIT CED MoE: 40 layers (20 encoder + 20 decoder), 552B backbone + ~196B Engram, 384 routed experts + 1 shared, top-6. Architecture, sampling defaults, and any benchmark rows on theupstream cardareDeepSeek’s. They are not scores of this EXL3 pack. I have not claimed them as mine.

Upstream sampling (start here, then tune):temperature=1\.0,top\_p=0\.95.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#loadLoad

ExLlamaV3, point at this repo (or a local snapshot):

from exllamav3 import Config, Model, Tokenizer

cfg = Config.from_directory("vcruz305/DSV4.1-Flash-EXL3-4.75bpw")
model = Model.from_config(cfg)
model.load()
tokenizer = Tokenizer.from_config(cfg)

Needs a recent ExLlamaV3 with DeepSeek V4.1 / CED support. Multi-GPU via whatever your ExLlamaV3 build exposes (tp, device map). This pack is sized as aTP4-classweight ceiling (~422 GiB on disk), not a promise of a specific serve topology.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#filesFiles

• model\-00001\-of\-00032\.safetensors…model\-00030\-of\-00032\.safetensors— EXL3 body + copied protected tensors
• model\-00031\-of\-00032\.safetensors,model\-00032\-of\-00032\.safetensors— Engram
• model\.safetensors\.index\.json,config\.json,tokenizer\.json,tokenizer\_config\.json

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#licenseLicense

MIT, same asDeepSeek-V4.1-Flash. Cite DeepSeek for the base model. This repository is the EXL3 pack only.

https://huggingface.co/vcruz305/DSV4.1-Flash-EXL3-4.75bpw#notesNotes

• Quantization: vcruz305, SAGE mixed-K EXL3.
• Please do not file “missing experts/ work tree” issues. That is not this repo.

DeepSeek (@deepseek_ai): 🚀 Introducing DeepSeek-V4.1-Flash: smarter, faster, more efficient.

🔹 Introducing the smallest model in our new architecture family, with native visual understanding. 🔹 Designed for greater capability, faster inference, higher throughput, and scaling to larger models.

1/6