@mitsuhiko: And the ds4 SSD caches are great. This is continuing a session after the server was shut down which was already 63k tok…
Summary
A user reports positive performance from ds4 SSD caches when resuming a long-running LLM inference session with 63k tokens, noting acceptable startup times.
View Cached Full Text
Cached at: 05/12/26, 12:46 AM
And the ds4 SSD caches are great. This is continuing a session after the server was shut down which was already 63k tokens in! Yes, takes a bit to come up, but not horribly bad! https://t.co/aVkS6uAdvY
Similar Articles
@antirez: DS4 running on DGX Spark (GB10 / CUDA), private branch for now. 12 tokens/sec, the memory bandwidth is limited in this …
Antirez reports benchmarking DS4 inference on the DGX Spark (GB10), noting 12 tokens/sec generation speed and high prefill performance, with plans to merge the codebase once mature.
@ttasanen: Just fired up DS4 by @antirez on my Mac Studio M3 Ultra 256GB and man, it’s seriously impressive. A clean, purpose-buil…
DS4 is a specialized inference engine by antirez designed to run DeepSeek V4 Flash locally on high-end Mac hardware, featuring optimized KV cache handling and 1M context support.
@techNmak: Your LLM inference is burning 50% of its compute on work it has already done. If you're running RAG or Multi-Turn Chat,…
LMCache is an open-source library that makes KV cache persistent and shareable across requests, eliminating recomputation in RAG and multi-turn chat workloads, achieving up to 15x throughput gain and 3-10x reduction in time-to-first-token.
@Tono_Ken3: Oh man, I did it! It went off—DeepSeek-V4-Flash-FP8 8 parallel aggregate 400TPS!! Local LLM revolution yesssssss lol
Achieved 400 tokens per second with DeepSeek-V4-Flash-FP8 using 8 parallel aggregates on local hardware, marking a significant milestone for local LLM inference.
@Michaelzsguo: Found this great tool that may be handy for your local LLM inference optimization: https://kvcache.ai/tools/kv-cache-ca…
A tweet shares the KV Cache Size Calculator from KVCache.ai, a tool for estimating KV cache memory usage for local LLM inference, highlighting that 1M tokens for DeepSeek V4 Pro uses only 5GB of RAM.