Someone made DSV4.1 run faster than official API on A100(s)
Summary
Someone has optimized DeepSeek V4.1 to run faster than the official API on NVIDIA A100 GPUs, despite A100 not supporting FP4 precision, and shared the implementation on GitHub.
Similar Articles
@YRSM_Simon: Amazing!
shi3z shares optimization experience for running DeepSeek v4.1 Flash on an A100 GPU without FP4 support, boosting inference speed from 33 tok/s to 673 tok/s, surpassing the official API speed.
DeepSeek-V4-Pro-0813-NVFP4 (7 minute read)
NVIDIA releases a quantized version of DeepSeek's V4-Pro-0813 model on Hugging Face, optimized with NVFP4 for efficient deployment in agentic AI and reasoning applications.
Running DeepSeek-V4 locally with 4x legacy RTX 2080 Ti ($2k budget setup). Custom Turing kernels, W8A8 quantization, and 255 prefill tok/s!
A developer successfully runs DeepSeek-V4-Flash (284B total, 13B active) locally on four RTX 2080 Ti GPUs with a $2,500 budget, achieving 255 prefill tokens/s using custom Turing CUDA kernels, W8A8 quantization, and heterogeneous inference. The implementation is open-sourced.
I have (even faster) DeepSeek V4 Pro at home
A user reports successfully running the DeepSeek V4 Pro model locally using ktransformers and sharing detailed benchmark results across various context depths, demonstrating improved inference speeds.
DeepSeek-V4-Flash-0731 unsloth gguf on A100
DeepSeek-V4-Flash-0731 is shown running as an unsloth GGUF quant on a single 40GB A100, with 17.7 tok/s and 6 experts loaded into VRAM, enabling a full agentic coding loop.