What GPUs will give me GOOD speeds and on DSV4 Flash and similar models, and not have to run a mega quantized version? Budget around $15k-ish.
Summary
A user in a tech forum is seeking advice on GPUs for running AI models like DSV4 Flash locally with good performance, comparing AMD and NVIDIA options within a $15k budget.
Similar Articles
4xR9700, 2xMi210 or 4x4080S 32G
The user is comparing GPU options like R9700, Mi210, and 4080S to achieve 128GB VRAM for running multiple AI models in parallel, considering factors like cost, performance, and compatibility.
3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed
The author conducted experiments to run DeepSeek-V4-Flash-0731 with 4-bit quantizations on a 128GB RAM system, using optimizations like memory mlocking and prompt processing strategies to achieve acceptable inference speeds.
@ciruai: Testing DeepSeek v4 Flash on the AMD Ryzen AI Max+ 395 Strix Halo with 128GB RAM. Getting ~15 TPS over a decently long …
Testing DeepSeek v4 Flash on the AMD Ryzen AI Max+ 395 with 128GB RAM achieves ~15 TPS for a 284B MoE model (13B active) locally, costing $3,000 versus $25,000+ for a datacenter setup, highlighting the feasibility of running large models on consumer hardware.
Running DeepSeek-V4 locally with 4x legacy RTX 2080 Ti ($2k budget setup). Custom Turing kernels, W8A8 quantization, and 255 prefill tok/s!
A developer successfully runs DeepSeek-V4-Flash (284B total, 13B active) locally on four RTX 2080 Ti GPUs with a $2,500 budget, achieving 255 prefill tokens/s using custom Turing CUDA kernels, W8A8 quantization, and heterogeneous inference. The implementation is open-sourced.
DeepSeek v4 Flash on 4090 + DDR5, my experience
A user shares their experience running the DeepSeek v4 Flash model with a 24GB GPU and DDR5 RAM, including performance numbers and tips for optimization.