The article reports on running DeepSeek-V4-Flash-0731 on two Radeon AI PRO R9700 GPUs with the affinity inference engine, achieving 40-50 tok/s decode speed via a prebuilt quantization and stability fixes.
I've been running DeepSeek-V4-Flash-0731 on two Radeon AI PRO R9700s (32 GB each, 192 GB system RAM) using affinity(https://codeberg.org/StillDeadcode/affinity), an inference engine written by **Yoshi Exeler (StillDeadcode)** specifically for DeepSeek-V4-Flash on one or two RDNA4 cards. All the credit for the engine goes to them. It keeps the hot experts on the GPUs, streams the rest from host RAM, and runs a DeepSeek-native speculative draft. If you have R9700s, go look at it. What I added: - A prebuilt 0731 quant, so you can skip the 149 GB download and the ~2.5 h quantize: [neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw](https://huggingface.co/neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw). It was made with affinity's own quantizer: routed experts at 2.875 bpw, the teamblobfish imatrix, dense weights in FP8, plus 0731's bundled draft. It loads only in affinity, not llama.cpp or vLLM. - A fork with a few stability fixes ([github.com/neuromaniacMD/affinity](https://github.com/neuromaniacMD/affinity)). The main one: on 2×32 GB cards the server would start producing fluent garbage after the first long prompt and never recover, caused by hugepage collapse under a GPU mapping with too little free VRAM. The fixes are also submitted upstream as PRs. There's a Docker build on TheRock ROCm 7.14, so the host needs no ROCm install. Numbers on my box (2× R9700, 262K context fully allocated at load): - Prefill: ~1,260–1,360 tok/s at 4K–64K context - Decode: ~40–50 tok/s on long structured answers, ~31–35 on free-form prose and reasoning. It depends on how often the draft's guesses are accepted. - Needle-in-a-haystack 3/3 at 4K/16K/64K. A multi-turn agentic task (Excel → DuckDB via shell tools, 15 turns) passed every check, with no malformed tool calls. Caveats: - RDNA4 (gfx1201) only, and two cards max. - Use the model card's sampling (temp 1.0). At temp 0.6 this quant fell into reasoning loops for me. - You need ~64 GB+ of RAM for the experts that don't fit on the cards. - There's also an experimental DeepSeek-V4.1-Flash branch. It's correct now, but only ~14 tok/s on two cards. Happy to answer questions. Engine questions really belong with the upstream project.
This repository provides configuration, patches, and tuning to run the DeepSeek V4 Flash 304B checkpoint on a single AMD MI300X in production, achieving 168 tok/s decode without quantization. It includes correctness overlays for vLLM ROCm, AITER tuning tables, and a hybrid KV cache strategy.
The article details a customized quantized version of DeepSeek-V4-Flash with MTP self-speculation enabled, achieving significant speedups on dual RTX PRO 6000 Max-Q GPUs using a patched vLLM setup.
Technical post detailing how to run DeepSeek V4 Flash on two Nvidia 4090d GPUs using custom Triton kernels and vLLM, achieving ~105 tokens/second with 262k context.
DeepSeek V4 Flash (98GB) now runs up to 7 tokens per second on a single RTX 4060 Ti with CPU offloading, a 3x speed improvement over the previous week's 2 t/s.
A developer successfully runs DeepSeek-V4-Flash (284B total, 13B active) locally on four RTX 2080 Ti GPUs with a $2,500 budget, achieving 255 prefill tokens/s using custom Turing CUDA kernels, W8A8 quantization, and heterogeneous inference. The implementation is open-sourced.