ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF · Hugging Face

Source: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF

**Update, October 27, 2026.**The files now use llama.cpp’s own DeepSeek-V4.1 layout and load in llama.cppb11870or newer, including images. - **Downloaded the earlier files?**You do not need to download the model again. Runconvert_ds41_upstream.pyon your model folder: it rewrites the files locally into exactly the current ones and fetches only 80 KB. SeeUpdating an existing download. - **New download?**The shared Engram file is shard 6 of every build, so link it once per build you use (<ID\>isIQ1\_M,IQ2\_XXSorIQ2\_XS; seellama.cpp): ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-<ID>-00006-of-00006.gguf

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#overviewOverview

This repository provides GGUF quantizations ofDeepSeek-V4.1-Flash-1017at three sizes:IQ1_M (1.75 bits per weight), IQ2_XXS (2.00) and IQ2_XS (2.25). In contrast to uniform quantization, which applies a single quantization type to all weight tensors, each model here assigns a separate quantization type to every tensor. The assignment is obtained by a gradient-based search that allocates precision according to per-tensor sensitivity, subject to a total size budget. The resulting files are standard GGUF.

DeepSeek-V4.1-Flash-1017 is a sparse mixture-of-experts model with763.2B parameters:384 routed experts per layer, of which 6 are active per token, and a 1M-token context window. The routed experts dominate the parameter count, so they also dominate the size budget, and the search spends most of its freedom there, assigning a quantization typeper layer(not per expert, which GGUF cannot express).

WithStrata, every build runs on asingle GPU with 12 GB of VRAM. SeeUsage.

**Method summary.**GSQ provides accurate low-bit scalar quantization of each tensor at a given quantization type; RCO assigns the per-tensor quantization types under a size budget. Together they yield a non-uniform GGUF at the requested size.

MethodDescriptionGSQ(Gumbel-Softmax Quantization,paper,code)Post-training scalar quantization that jointly learns the per-coordinate grid assignments and the per-group scales via a Gumbel-Softmax relaxation. GSQ closes most of the gap between scalar and vector quantization at 2 to 3 bits while remaining deployable in standard scalar formats such as GGUF.RCO(Riemannian Constrained Optimization,paper,code)Assigns one of K quantization types to each of N tensors under a total size budget. The budget constraint is reformulated as a smooth Riemannian manifold in logit space, which permits gradient-based optimization directly on the task loss while enforcing the budget exactly, without constraint-specific hyperparameter tuning. Both methods were developed at theDeep Algorithms and Systems Lab (DASLab), Institute of Science and Technology Austria.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#available-filesAvailable files

Each build is a backbone GGUF in five shards, named after its quantization identifier:DeepSeek\-V4\.1\-Flash\-1017\-GSQ\-RCO\-<ID\>\-0000i\-of\-00006\.gguf, i = 1 to 5. Shard 1 holds the metadata only, shards 2 to 5 the transformer weights,551.9B parameters. The bpw is the average bit-width over these, which is what the search controls.

TheEngram file,DeepSeek\-V4\.1\-Flash\-Engram\-IQ4\_NL\.gguf, holds the model’s two Engram lookup tables (196.6B parameters) at IQ4_NL, 4.50 bpw. It is identical for all builds, so it is stored once and needs to be downloaded only once, also when trying several builds. llama.cpp loads it as shard 6 of a build, so each build needs a link to it named\.\.\.\-<ID\>\-00006\-of\-00006\.gguf(one command, seellama.cpp). The tables are a lookup, not a matmul weight, so they are held at a fixed type in every build and excluded from the search. They are quantized in the evaluated models, not just the shipped ones.

BuildbpwBackboneEngramTotalNotesIQ1\_M1.75120.72 GB110.60 GB231.32 GBSmallestIQ2\_XXS2.00138.24 GB110.60 GB248.84 GBRecommended; closest to the base modelIQ2\_XS2.25155.23 GB110.60 GB265.83 GB All three builds run on a single 12 GB GPU withStrata.

The Engram column is the same shared file in every row. The vision projector,mmproj\-DeepSeek\-V4\.1\-Flash\-1017\-BF16\.gguf(0.97 GB), also serves all builds. GB is 109bytes.

The IQ2_XXS build is the stronger operating point: it matches the base model on all three benchmarks and so reaches**100.0%**of its task average (93.54). The IQ1_M build trades 1.2 points of task average for 17.5 GB less.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#memory-requirementsMemory requirements

Only the backbone holds transformer weights. The Engram tables are read sparsely, one row per token, so they can stay memory-mapped on disk and do not need to occupy VRAM or RAM. With Strata, the backbone does not need to fit in VRAM either: every build runs on a12 GB GPU, with the rest of the model held in system memory. Keep the Engram file on an SSD so the paging stays cheap.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#resultsResults

All models are evaluated against the base modeldeepseek-ai/DeepSeek-V4.1-Flash-1017as released (FP8 dense weights, FP4 routed experts). We report three reasoning and generation benchmarks:AIME25,GPQA-Diamond, andLiveCodeBench v6. Task average is the mean of the three; recovery is the task average relative to the base model.

All numbers in this card were measured atreasoning effort 50.

VariantbpwAIME25↑GPQA-D↑LCB v6↑Task avg↑RecoveryBase (FP8 / FP4)n/a100.0090.9189.7193.54100.0%GSQ-RCO IQ1_M1.75100.0088.3888.5792.3298.7%GSQ-RCO IQ2_XXS2.00100.0090.9189.7193.54100.0%****GSQ-RCO IQ2_XS2.25100.0090.4089.7193.3799.8% IQ2_XXS (2.00 bpw) keeps**100.0%**of the base model’s task average: it matches the base model on AIME25 (100.00), GPQA-Diamond (90.91) and LiveCodeBench v6 (89.71). IQ2_XS (2.25 bpw) matches it on AIME25 and LiveCodeBench v6 and scores 90.40 on GPQA-Diamond, 99.8% overall. The smallest build, IQ1_M (1.75 bpw), also scores 100.00 on AIME25 and retains 98.7% overall.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#usageUsage

**Runtime support.**llama.cpp supports DeepSeek-V4.1, including images, since releaseb11870(October 25, 2026), and the files here use its layout since October 27, 2026. Downloads from before that date can be converted locally, seeUpdating an existing download. The GGUFs are also supported inStrata.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#strata-engineStrata Engine

Strataruns these GGUFs on asingle NVIDIA GPU with 12 GB of VRAM, with a one-click installer and an OpenAI-compatible endpoint on localhost. The builds were optimized together with Strata for local inference.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#llamacppllama.cpp

Requires llama.cppb11870or newer.

# download one build and the shared Engram file (requires: pip install -U "huggingface_hub[cli]")
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "*IQ2_XXS*" "*Engram*" --local-dir .

# the Engram file is shard 6 of every build
ln -s DeepSeek-V4.1-Flash-Engram-IQ4_NL.gguf DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00006-of-00006.gguf

llama-cli -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  -p "Explain mixed-precision quantization." -ngl 99

\-mtakes the first shard; llama.cpp opens shards 2 to 6 from the same directory. To try another build, download only its shards (\-\-include "\*IQ1\_M\*") and add its link. On Windows,mklink /H <link\> DeepSeek\-V4\.1\-Flash\-Engram\-IQ4\_NL\.ggufmakes the link;convert_ds41_upstream.py, run on the folder, also creates missing links.

The Engram table stays memory-mapped on disk and is read one row per token (llama.cpp’s default\-\-lazy\-mode autodoes this for tensors over 4 GiB), so it needs neither VRAM nor RAM. SeeMemory requirements.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#imagesImages

Download the vision projector and pass it with\-\-mmproj:

hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF \
  --include "mmproj-*" --local-dir .

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf \
  --mmproj mmproj-DeepSeek-V4.1-Flash-1017-BF16.gguf -ngl 99

Withllama\-mtmd\-cli, also pass\-\-jinjaso that the chat template embedded in the GGUF is used.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#chat-template-and-reasoning-effortChat template and reasoning effort

The base repository ships no Jinja chat template; its prompt format is defined by DeepSeek’s encoder,encoding/encoding\.py. The GGUFs embed a template adapted fromUnsloth’s DeepSeek-V4.1 release, checked against that encoder for text chat, tool use, and image formatting. Reasoning effort accepts an integer from 1 to 100 (also a numeric string), orlow,high,max(50, 75, 100). The default effort is DeepSeek’shigh(75), and effort applies only in thinking mode. Structured image blocks, including images in tool results, retain their image placeholders. Image inference needs the vision projector, seeImages.

llama-server -m DeepSeek-V4.1-Flash-1017-GSQ-RCO-IQ2_XXS-00001-of-00006.gguf -ngl 99 \
  --reasoning on --reasoning-effort 50

Effort 50 is the setting all results in this card were measured with. Per request, set"reasoning\_effort": "50"in the chat completion body (a string, also for numbers).\-\-reasoning offdisables thinking.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#updating-an-existing-downloadUpdating an existing download

Files downloaded before October 27, 2026 (four shards per build and a separateDeepSeek\-V4\.1\-Flash\-Engram\-IQ4\_NL\.gguf) were written for an earlier llama.cpp branch and do not load in llama.cpp.convert_ds41_upstream.pyturns them into the current files of this repository, byte for byte, without downloading the model again. It needs Python 3.8+ and no extra packages, and fetches only two small files of this repository (chat\_template\.jinjaandexp\_probs\_b\_vl\.f32, 80 KB).

cd /path/to/downloaded/model-folder
hf download ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF convert_ds41_upstream.py --local-dir .
python3 convert_ds41_upstream.py .

It converts every build in the folder and the mmproj, and writes the new backbone shards next to the old ones; every written tensor is read back and compared with its source. The Engram file is converted in place: only its header changes, its 110 GB of tables stay as they are, and the script adds the shard-6 links. The free space needed is the size of the new shards (121 to 155 GB per build). With\-\-delete\-old, the old shards of a build are deleted as soon as its new ones are verified, so space for one build suffices. The new first shards take the chat template fromchat\_template\.jinja, sopatch\_chat\_template\.pyis not needed.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#quantization-procedureQuantization procedure

1. **Per-tensor database.**Each weight tensor is quantized at every candidate GGUF quantization type with GSQ, yielding a searchable database of quantized tensor variants.
2. **RCO search.**The budget-constrained Riemannian search assigns one quantization type per tensor such that the backbone’s average bit-width meets the target.
3. **Assembly.**The selected per-tensor variants are stitched into a standard GGUF file.

The dense tensors (attention, shared experts, embeddings, output head) and the routed-expert matrices draw on separate quantization ladders, because the routed experts sit at a much lower bit-width than the dense path at any given budget: the dense tensors choose among Q5_K, Q6_K and Q8_0, the routed experts among types from IQ1_S to IQ3_S. The search’s teacher is the released checkpoint itself, with the routed experts in their native FP4, and its calibration data was generated by the model.

Reference implementations:GSQatIST-DASLab/GSQandRCOatIST-DASLab/RCO.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#reproducibility-artifactsReproducibility artifacts

Each released build ships the files needed to audit how it was built:

FileContentstensor\-allocation/DeepSeek\-V4\.1\-Flash\-1017\-GSQ\-RCO\-<ID\>\.rco\-allocation\.txtThe quantization type assigned to every tensor of that build (all six shards, 1043 tensors), with a quant-type histogram and the target and measured bit-width. This is the RCO search result, so the allocation can be inspected without opening the model.convert\_ds41\_upstream\.py,chat\_template\.jinja,exp\_probs\_b\_vl\.f32The converter for downloads from before October 27, 2026 (seeUpdating an existing download), the chat template the GGUFs embed, and the vision router bias the converter adds (40 x 384 F32,ffn\.gate\.bias\_vlof the original checkpoint).

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#citationCitation

If you use these models or methods, please cite both papers:

@article{gsq2026,
  title  = {GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling},
  author = {Dadgarnia, Alireza and Tabesh, Soroush and Nikdan, Mahdi and Helcig, Michael and Kurtic, Eldar and Kleinegger, Maximilian and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2604.18556},
  year   = {2026}
}
@article{rco2026,
  title  = {Model Compression with Exact Budget Constraints via Riemannian Manifolds},
  author = {Helcig, Michael and Alistarh, Dan},
  journal= {arXiv preprint arXiv:2605.00649},
  year   = {2026}
}

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#acknowledgementsAcknowledgements

We thankVerdaand Scientific Computing at the Institute of Science and Technology Austria for providing the compute resources used to produce these models.

https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF#licenseLicense

These quantized weights inherit the license of the base model (DeepSeek-V4.1-Flash-1017, MIT). The GSQ-RCO tooling is released by the Deep Algorithms and Systems Lab under its repository license.