My first @huggingface upload 🥳

Qwen3.8-27B-DFlash2 for Blackwell

• same final decode speed
• same output quality
• 2.6x less VRAM

https://t.co/0uxKJR04zl

maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal · Hugging Face

Source: https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#qwen38-27b-dflash2-nvfp4-rtncalQwen3.8-27B-DFlash2-NVFP4-RTNcal

CalibratedNVFP4 (W4A4)quantization of theDFlash 2block-diffusion draft model forQwen3.8-27B, packaged in the NVIDIA ModelOpt layout that SGLang loads natively.

Built for a singleRTX 5090 (32 GB)serving Qwen3.8-27B in NVFP4 with SGLang: the draft shrinks from3.53 GB (BF16) to 1.37 GBof VRAM, which turns into**+44 % KV-cache context (90K → 130K tokens)**at the same decode speed and the same acceptance rate as the BF16 draft. Output quality is unchanged by construction: the target model verifies every drafted token.

DraftVRAMKV context*Decode (end-to-end)Accepted / 8BF16 (upstream)3.53 GB90K215 tok/s3.71NVFP4, calibrated (this repo)1.37 GB130K228 tok/s3.60NVFP4, round-to-nearest,uncalibrated1.37 GB130K210 tok/s3.26 * Targetgittensor\-model\-hub/Qwen3\.8\-27B\-NVFP4\-RTX5090, FP8 KV cache,\-\-mem\-fraction\-static 0\.90, HiCache on. Benchmark: 5 mixed prompts (code, math, French prose, JSON, essay) × 600 generated tokens, 3 runs each, temperature 0.7. Run-to-run noise on the acceptance length is about ±0.2, so BF16 and this checkpoint are at parity.

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#long-context-agentic-benchmarkLong-context agentic benchmark

Coding-agent session on the SGLang source tree (system prompt + 6 tool schemas, tool results = real source files), thinking on, streaming; 3 turns per context size (cold analysis, follow-up on the cached prefix, tool call withtools), 2 passes. TTFT is identical (prefill is the target’s work, ~4K tok/s; cached prefix → 0.04–0.2 s in both), so only decode is shown.

ContextDecode BF16 →NVFP4 cal.(turn 1, thinking)Decode BF16 →NVFP4 cal.(turn 2)Accepted/8 BF16 →**NVFP4 cal.**8K166 →200tok/s202 → 1833.12 → 3.0332K168 →196217 →2353.18 → 3.3264K174 →180203 →2293.44 → 3.1985K157 →166178 →2183.18 → 3.21110Kexceeds BF16 context→207— →226— →3.63 Tool calls were emitted correctly in 10/10 tool turns with this draft (6/8 with BF16, the misses being sampling noise at T=0.7).

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#what-is-quantizedWhat is quantized

• NVFP4 (E2M1, group size 16, FP8-E4M3 block scales, FP32 per-tensor global scale)— all 35 linear projections of the 5 draft layers:self\_attn\.\{q,k,v,o\}\_proj,mlp\.\{gate,up,down\}\_proj. Fused shards (q/k/v,gate/up) share theirweight\_scale\_2andinput\_scale, as ModelOpt requires.
• BF16 (unchanged)—fc(target-feature projection),attention\_conv\.\*/mlp\_conv\.\*(dynamic depthwise convolutions),candidate\_selector\.\*(codebooks + hidden projection), all RMSNorm weights. These are plainnn\.Linear/parameters in SGLang’sDFlash2DraftModeland cannot be quantized there.
• Activations are quantized at runtime by SGLang to NVFP4 with the static per-tensorinput\_scalestored here (W4A4).

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#calibration–the-part-that-mattersCalibration — the part that matters

Round-to-nearest NVFP4 without calibration loses ~13 % acceptance (3.26 vs 3.71). The whole gap is recovered by calibrating theactivation scaleson the draft’s real inputs:

• On-policy data: 460 conversations generated by the target model itself through SGLang (thinking enabled on half of them), 6 domains — English multi-turn chat (ultrachat), French (Wikipedia-based tasks + free prompts), code (glaive-code-assistant + bash/SQL/TS/Rust/Go/CUDA prompts), tool calling (glaive-function-calling-v2 with tool schemas), math (GSM8K, MATH-500), structured output (JSON/YAML/Markdown). 281,649 generated tokens.
• Capture: forward pre-hooks on the 20 quantizable linears of the BF16 draft while it served those conversations (prefill and decode, eager mode), ~621K input rows per layer. Per-tensorinput\_scale = amax / \(6 · 448\).
• Weights: round-to-nearest on the NVFP4 grid; global scaleamax / \(6 · 448\), per-group FP8 scales.

Things that were tried and madeno measurable differenceon this draft (all within ±0.2 noise): GPTQ with full Hessians, keepingq/k/vin BF16, SmoothQuant folding (o\_proj,gate\_up,downonly — theqkvfold is not function-preserving in SGLang because context K/V are materialized fromhidden\_norm\(fc\(features\)\)without the per-layerinput\_layernorm).

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#usage-with-sglangUsage with SGLang

Requires SGLang built frommainat or after commitecbadf0b(addsDFlash2DraftModel), CUDA 13, a Blackwell GPU (sm120) for the NVFP4 GEMMs, and the draft quantization flag:

sglang serve \
  --trust-remote-code \
  --model-path gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090 \
  --served-model-name qwen3.8-27b \
  --kv-cache-dtype fp8_e4m3 \
  --mem-fraction-static 0.90 \
  --attention-backend flashinfer \
  --max-running-requests 1 --cuda-graph-max-bs 1 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --mamba-full-memory-ratio 5.61 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path <this repo> \
  --speculative-draft-model-quantization modelopt_fp4 \
  --speculative-num-draft-tokens 8 \
  --chunked-prefill-size 1024 \
  --cuda-graph-bs-prefill 64 128 256 512 1024 \
  --enable-memory-saver \
  --enable-hierarchical-cache --hicache-ratio 2 --hicache-write-policy write_through \
  --host 0.0.0.0 --port 30000

Notes:

• \-\-speculative\-draft\-model\-quantization modelopt\_fp4is required; SGLang does not requantize custom draft models on the fly.
• \-\-speculative\-num\-draft\-tokens 8is the DFlash 2 block size and cannot be raised.
• Theignorelist inconfig\.json(flat ModelOpt format) is what SGLang reads to leavefc& co. in BF16; keep it if you edit the config.
• \-\-cuda\-graph\-bs\-prefill 64 128 256 512 1024andMAX\_JOBS=2keep host-RAM usage sane on a 64 GB box during graph capture / JIT.
• KV cache in FP4 (\-\-kv\-cache\-dtype nvfp4) does not work with speculative decoding on this SGLang build (hybrid GDN model).

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#filesFiles

• model\.safetensors— 1,550,153,248 bytes (sha2562228b9b2…), 186 tensors (35 uint8 packed FP4 weights, 35 FP8 block scales, 70 FP32 scales, 46 BF16).
• hf\_quant\_config\.json,config\.json(quantization\_configwithignore), tokenizer-free (uses the target’s tokenizer),README\.incoai\-original\.md(upstream card).

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#reproduceReproduce

Scripts (SGLang side):dflash\_calib\_hook\.py(activation capture, temporary patch ofsglang/srt/models/dflash\.py),calib/build\_prompts\.py+calib/run\_calib\.py(on-policy calibration run),quantize\-dflash2\-gptq\-nvfp4\.py \-\-rtn(export). Base weights:incoai/Qwen3\.8\-27B\-DFlash2(BF16). Quantized on 2026-08-29.

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#changelogChangelog

• 2026-08-30—config\.json:quantization\_config\.produceris now ModelOpt’s dict form (was a plain string, which crashed vLLM’s draftModelConfigconstruction withAttributeError: 'str' object has no attribute 'get'; SGLang ignored the key). Weights unchanged (samemodel\.safetensors, sha2562228b9b2…). Thanks to @joelafrite for the report and the vLLM numbers.
• 2026-08-29— initial release.

https://huggingface.co/maurienne-ai/Qwen3.8-27B-DFlash2-NVFP4-RTNcal#licenseLicense

Apache-2.0, same as the base DFlash 2 draft and Qwen3.8-27B.