Tried running 748B DeepSeek-V4.1-Flash on a €2k PC

Reddit r/LocalLLaMA News
local-llm deepseek quantization llama.cpp consumer-hardware inference

Summary

A user successfully ran the 748B-parameter DeepSeek-V4.1-Flash-1017 model at 7.2 tok/s on a €2k used PC with an RTX 3090, using a 1-bit Unsloth UD-IQ1_S GGUF quantization and a custom llama.cpp build with on-demand Engram table loading.

I bought a used PC with an RTX 3090 24 GB, 128 GB DDR4 and an Intel i9-9900X last year for under €2,000. With the release of DeepSeek-V4.1-Flash-1017 I decided to try running it on my machine. It is of course the smallest quant available (1-bit), but I was still happy to see it working at 7.2 tok/s. The answer has a physics error in it (wondering if someone can spot it), and I don't see much practical use for it. Qwen models actually work pretty well on my setup. Still, it felt really nice to see a 748B parameter model running on such hardware. Here are some technical details: Unsloth UD-IQ1_S GGUF, 233 GB on disk llama.cpp built from the open PR #30817, CUDA 12.8 The 110 GB Engram table isn't loaded up front. It stays on the NVMe and only the rows the model needs are read on demand, which is how it fits. ​ llama-server -m DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf \ -ngl 99 --n-cpu-moe 36 -ot "per_layer_token_embd.weight=CPU" \ -lm mmap -lzm on -t 10 -c 16384 --jinja \ --temp 1.0 --top-p 1.0 --top-k 0 --min-p 0.0 \ --chat-template-kwargs '{"enable_thinking":false}' Now I only need to decide whether to keep the model or free up the disk space :)
Original Article

Similar Articles

DeepSeek-V4-Flash-0731 unsloth gguf on A100

Reddit r/LocalLLaMA

DeepSeek-V4-Flash-0731 is shown running as an unsloth GGUF quant on a single 40GB A100, with 17.7 tok/s and 6 experts loaded into VRAM, enabling a full agentic coding loop.

Deepseek V4 Flash running on RTX 5090 MoE

Reddit r/LocalLLaMA

User shares optimization benchmarks for DeepSeek-V4-Flash (Q2_K) running on an RTX 5090 using a fork of llama.cpp, achieving 21.3 tokens/s generation and 1 million context size.

DeepSeek-v4-Flash-Mini 54GB GGUF running at ~20.5 t/s

Reddit r/LocalLLaMA

A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.