Overspill is an experimental disk tier for FreeToken that allows running large MoE models like DeepSeek-V4-Flash on limited consumer hardware by using memory-mapping and caching optimizations, achieving approximately 3 tokens per second on a setup with a 12 GB GPU and 64 GB RAM.
I've been experimenting with ways to run MoE models that don't fit comfortably in RAM, and I ended up making Overspill, a disk tier for FreeToken. The basic idea came from looking at how Colibri handles experts across disk/RAM/VRAM so I took inspiration from the general approach. Repo: https://github.com/IvanAdriazola/overspill Apache-2.0 · experimental My hardware RTX 3060 12 GB Ryzen 9 7900 64 GB DDR5-6000 (WSL2 capped at 48 GB) NVMe, accessed through WSL2 Windows 11 + WSL2 Ubuntu 24.04 I tested DeepSeek-V4-Flash REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b), which is ~85 GB with FP4 experts. Results Same model, same FP4 experts, cold start, greedy decoding, all on the same PC: Overspill llama.cpp Colibri Decode 2.8–3.4 tok/s 0.4–0.5 tok/s 1.1–1.2 tok/s TTFT, 6.4k prompt 102 s 373 s ~26 min TTFT, first short prompt after cold start 43 s 44 s 25 s TTFT, next short prompt 10 s 40 s 18 s These are just my measurements on this particular machine, so I wouldn't read too much into the comparisons yet, also i don't consider myself an expert, there was some heavy vibecoding invoved. The non-expert weights also aren't identical between the engines (the experts are identical in all three, but llama.cpp's GGUF stores the ~8 GB of non-expert weights (attention etc.) in Q8_0, while FreeToken and Colibri use DeepSeek's original FP8, so the runs aren't bit-identical). What I changed The main things I experimented with were: Memory-mapping experts that don't fit in RAM, letting the OS page cache act as another tier. Using madvise(WILLNEED) so Linux reads each layer's routed experts in parallel with large reads, instead of pulling them in page fault by page fault. Keeping the embedding/output layers in RAM so I could free some VRAM. Using larger prompt chunks to reduce how often the experts have to be streamed. Running short prompts on the CPU instead of moving the full expert set through the GPU. The biggest improvement I saw was expert loading from disk, which went roughly 4× faster in my tests. I also changed FreeToken's checkpoint converter, which was running out of memory on models larger than RAM. The fix worked for me, but I'd like the FreeToken devs to confirm that it's the right approach. Sanity checks On Qwen3.6-35B-A3B, which fits in RAM, I get byte-identical output to stock FreeToken on the prompts I tested. I also managed to run the DeepSeek model through a coding test and some multi-turn tool-calling tasks, although I haven't done anything resembling a comprehensive evaluation yet. One thing I tried that didn't work well was prefetching the next layer's experts from RAM → GPU. It was functional, but ended up 9–29% slower on my 3060. My current guess is that the transfers are competing with GPU computation, but I could be misunderstanding what's actually happening. This is very much an experimental proof of concept right now: one machine, one large model, WSL2, and one request at a time. In Overspill's disk path the expert math runs on the CPU (FreeToken's CPU executor, which used the AVX-512 path on my Zen 4 Ryzen 9 7900), and the experts stream from disk through RAM. So these numbers depend heavily on the CPU, RAM speed (DDR5-6000 here) and storage, not just the GPU. A CPU without AVX-512 (many Intel consumer chips) falls back to slower code paths, and fewer cores, slower RAM, a slower SSD or less RAM for the page cache will all likely lower decode speed. Please don't read my ~3 tok/s as a general figure; treat it as what one fairly strong CPU + DDR5 + NVMe setup gets, and I'd really like to see how it scales on other machines. If anyone with native Linux, faster storage, more RAM, or different hardware wants to try it, I'd be very interested in the results. And if I've misunderstood something about FreeToken, Colibri, mmap/page caching, or the performance measurements, please tell me. Most of this is thanks to the existing work from the FreeToken team and the ideas Colibri came up with. I mainly put a relatively small experimental layer on top of FreeToken to see whether this approach could work with disk-backed experts. Final warning: the LLM space moves ridiculously fast, so there's a good chance this is already old news by the time I post it. Sorry in advance if someone already got this. 😅
DeepSeek V4 Flash (98GB) now runs up to 7 tokens per second on a single RTX 4060 Ti with CPU offloading, a 3x speed improvement over the previous week's 2 t/s.
A community build crushes DeepSeek-V4-Flash down to a 54GB IQ2_XXS GGUF variant with aggressive 2-bit quantization, achieving ~20.5 tokens/s on local hardware while drastically reducing memory footprint.
User shares optimization benchmarks for DeepSeek-V4-Flash (Q2_K) running on an RTX 5090 using a fork of llama.cpp, achieving 21.3 tokens/s generation and 1 million context size.
A developer successfully runs DeepSeek-V4-Flash (284B total, 13B active) locally on four RTX 2080 Ti GPUs with a $2,500 budget, achieving 255 prefill tokens/s using custom Turing CUDA kernels, W8A8 quantization, and heterogeneous inference. The implementation is open-sourced.
Testing DeepSeek v4 Flash on the AMD Ryzen AI Max+ 395 with 128GB RAM achieves ~15 TPS for a 284B MoE model (13B active) locally, costing $3,000 versus $25,000+ for a datacenter setup, highlighting the feasibility of running large models on consumer hardware.