DeepSeek-v4-Flash-Mini 54GB GGUF running at ~20.5 t/s

Reddit r/LocalLLaMA Models
deepseek gguf quantization local-llm benchmark text-generation

Summary

A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.

Took the REAP adaptation of DeepSeek-V4-Flash (0xSero/DeepSeek-V4-Flash-0731-REAP) along with antirez/deepseek-v4-gguf as inspiration, and decided to see how aggressive we could get with standard quant tricks to create a budget-friendly "Mini" build. For the lulz, naturally. Started with the full 95GB bf16 GGUF and crushed it down to an IQ2_XXS variant with mixed quantization (w2Q2K-AProjQ8-OutQ8). Is extreme 2-bit quantization practical for complex reasoning? Debatable. Did it shave off over 40GB of VRAM/RAM footprint and still generate coherently? Absolutely. Science isn't about why, it's about why not. prompt eval time = 372.26 ms / 12 tokens (31.02 ms per token, 32.24 tokens per second) eval time = 81141.35 ms / 1667 tokens (48.68 ms per token, 20.54 tokens per second) total time = 81513.62 ms / 1679 tokens graphs reused = 1804 The File Sizes: -rw-rw-r-- 1 jabbatheduck jabbatheduck 95G Aug 4 16:10 deepseek-v4-flash-bf16.gguf -rw-rw-r-- 1 jabbatheduck jabbatheduck 54G Aug 4 17:41 DeepSeek-V4-Flash-REAP-IQ2XXS-w2Q2K-AProjQ8-OutQ8-chat-v2.gguf -rw-rw-r-- 1 jabbatheduck jabbatheduck 353M Aug 4 16:37 DeepSeek-V4-Flash-REAP-IQ2XXS-w2Q2K-AProjQ8-OutQ8-chat-v2-imatrix-0731.gguf
Original Article

Similar Articles

DeepSeek-V4-Flash-0731 unsloth gguf on A100

Reddit r/LocalLLaMA

DeepSeek-V4-Flash-0731 is shown running as an unsloth GGUF quant on a single 40GB A100, with 17.7 tok/s and 6 experts loaded into VRAM, enabling a full agentic coding loop.