Hey everyone! We’re releasing new DeepSeek-V4.1-1017 Dynamic V3.1 (preview) GGUFs which showcases how Unsloth’s GGUFs retain the most accuracy for its size across 7 different benchmarks. We benchmarked ISTA's GSQ-RCO GGUFs but could not reproduce their reported results, even when using the highest scores. The biggest discrepancy was on AIME25 where they claimed 100% accuracy recovery to baseline, but was only ~91.88% in benchmark tests. Across AIME25, LiveCodeBench, GPQA Diamond, Aider Polyglot, and more, Unsloth Dynamic V3.1 outperforms GSQ-RCO GGUFs on nearly all metrics. Even our old v3 does better on nearly all metrics than ISTA. We ran each benchmark at least 3 times (AIME25 was 16 times) and averaged the results to ensure a fair, reliable comparison. More results coming next week! All benchmark data, methodologies and results are in our blog. Separately, for KLD, there were queries whether it’s a reliable measure of accuracy. The primary goal of quantization is to minimize KLD but also minimize disk space. If you get a KLD of 0, this means your quantized model == 16bit model, as logits are perfectly modelled. Our benchmarks show on KLD and top-1% across multiple domains (coding, agentic coding, chat etc), UDv3.1 does much better than GSQ-RCO across every domain. We show that lower bit quants do think somewhat longer than the unquantized model - we plan to investigate and make quants even better. However, we're still much ahead vs GSQ-RCO on even reasoning length as well, and provide plots for truncated answers in our blog. The model will be able to run on 128GB RAM/unified memory devices or even on 64GB RAM with engram offloading. Blog with all details and more benchmarks: https://unsloth.ai/docs/models/deepseek-v4.1-flash GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF We also introduced lots of new things to Unsloth Desktop including Browser Use, auto compaction, a whole new TTS pipeline (with voice cloning, separation etc), sandboxing, and background customization and more: https://github.com/unslothai/unsloth Hope you all have a Happy Halloween! Also table of numbers below which is also described with more plots and data in our blog: Quant Size GB AIME25 HMMT Feb26 GPQA-D MMLU-Pro IFEval LCB v6 (32K) Avg of 6 Aider pass@1 UD-Q8_K_XL 508.4 97.08 68.94 90.47 88.61 95.10 85.90 87.68 - UD-Q2_K_XL 334.8 97.50 69.89 89.58 88.28 94.92 84.76 87.49 - UD-IQ2_XXS 300.5 96.88 67.61 88.95 87.53 95.19 83.62 86.63 44.38 UD-IQ1_M 250.4 95.00 62.88 88.38 87.53 95.10 84.38 85.55 42.13 UD-IQ1_S 233.0 92.92 57.01 87.63 86.62 94.92 82.10 83.53 36.52 GSQ-RCO IQ2_XS 265.8 95.00 64.39 88.64 86.62 94.27 84.95 85.64 41.57 GSQ-RCO IQ2_XXS 248.8 95.21 57.95 85.80 86.53 93.72 82.86 83.68 36.52 GSQ-RCO IQ1_M 231.3 91.88 53.60 85.10 85.49 91.77 82.86 81.78 29.78
ISTA-DASLab released GSQ-RCO non-uniform GGUF quantizations of DeepSeek-V4.1-Flash-1017 (763.2B MoE) at 1.75-2.25 bits per weight, assigning per-tensor quantization types via gradient-based search so the model can run on a single 12GB GPU with llama.cpp.
ISTA disputes Unsloth's published quantization comparisons for DeepSeek-V4.1-Flash-1017, arguing the performance gap comes from a 32K vs 128K reasoning-token budget rather than any hidden hyperparameter tuning, and invites independent reproduction of their benchmarks.
GGUF quantizations of DeepSeek V4 Flash in 2-bit, 3-bit, and 4-bit precisions, made available on Hugging Face for local inference with tools like llama.cpp and Ollama.