From the creator of Redis; run LLM locally with ds4

# DwarfStar 4 (ds4): Local DeepSeek V4.1, Qwen and GLM Source: [https://dwarfstar.sh/](https://dwarfstar.sh/) DS4· LOCAL FRONTIER INFERENCE ## Run frontier open weights locally with ds4\. DwarfStar 4 is a narrow C inference engine for high\-memory Mac, CUDA and ROCm machines\. It supports DeepSeek V4 and V4\.1 Flash, GLM 5\.x and Qwen3\.8 Flash Next, with text and vision models, local APIs, a CLI and a native agent in one stack\. SUPPORTED: DEEPSEEK V4 / V4\.1 \+ GLM 5\.x \+ QWEN3\.8·MIT LICENSE·C / METAL / CUDA / ROCM·[QWEN ON 64GB](https://dwarfstar.sh/docs/qwen-3-8-flash-next/) PRINCIPLE· LOCAL MODEL STACK PHASE 1· THE GIANT ### A 284\-billion\-parameter star DeepSeek V4 Flash is a large mixture\-of\-experts model\. The usual path is remote serving; ds4 starts from the opposite constraint\. PHASE 2· THE COLLAPSE ### Compressed, not lobotomized Asymmetric quantization targets the routed experts while preserving critical paths\. The model becomes practical on high\-memory machines\. PHASE 3· THE DWARF STAR ### Dense, resident, yours The local engine exposes a CLI, HTTP APIs and a native agent, all sharing the same model state and cache\. [How the collapse works →](https://dwarfstar.sh/docs/architecture/) SCROLL ▾ DESIGN CHOICES ## Local frontier inference, narrow on purpose Not a generic GGUF runner\. ds4 follows a small, opportunistic set of model families and validates each supported layout end to end\. CORE 01 ### Asymmetric 2\-bit quantization Compress the routed experts, keep critical shared paths precise\. That is how the supported routed\-MoE builds fit their target machines\. CORE 02 ### KV cache as a disk citizen Save long prefixes to SSD and resume by prompt hash\. Restarts do not have to mean full re\-prefill\. CORE 03 ### One engine, three interfaces Use`\./ds4`for chat,`\./ds4\-server`for local APIs and`\./ds4\-agent`for persistent coding sessions\. - SSD STREAMING - TENSOR PARALLELISM - SESSION BATCHING - DSPARK \+ MTP - VISION INPUT ARCHITECTURE ## How the ds4 stack fits together Project GGUFs, a self\-contained engine and agent\-facing interfaces, checked against official model outputs\. MODELDeepSeek V4 / V4\.1GLM 5\.x · Qwen3\.8supported GGUF layouts onlyasymmetric 2\-bit \+ imatrixloadENGINEds4 enginewritten in Cmetal · cuda · rocmparallel · batch · speculateKV cacheRAM ⇄ SSD · survives restartsserve\./ds4interactive CLI\./ds4\-serverOpenAI \+ Anthropic API\./ds4\-agentnative coding agentpersonal → distributed memory classes RUNTIME MAP· SIMPLIFIED\. SEE[ARCHITECTURE NOTES](https://dwarfstar.sh/docs/architecture/)FOR THE FULL DRAWING\. RUN IT ## Run ds4 in three steps Download the project GGUF, build for your backend, then start the CLI or server\. Generic GGUF files are not the target\. STEP 1· FETCH THE WEIGHTS ds4 · zsh$git clone https://github\.com/antirez/ds4 $cd ds4 && \./download\_model\.sh ds4f\-q2 STEP 2· BUILD FOR YOUR BACKEND ds4 · zsh$make $make cuda\-spark STEP 3· TALK TO IT ds4 · zsh$\./ds4 $\./ds4\-server \-\-ctx 100000 FIT CHECK ## ds4 hardware fit: local, streamed and distributed Pick your platform and memory: get a conservative starting path and understand which execution modes apply\. PLATFORMMEMORY ✓ Runs well V4 Flash Q2 is the baseline\. At 128 GB, GLM 5\.3 Q2 and Qwen Q4 also fit; V4\.1 Q2 streams from SSD\. ``` ./download_model.sh ds4f-q2 && make ``` REF· M5 MAX 128GB · 32K CTX: 34\.4 T/S GEN · 557 T/S PREFILL Estimates from the[ds4 benchmark table](https://dwarfstar.sh/benchmarks/)\. Full guide in[Hardware](https://dwarfstar.sh/hardware/)and[Installation](https://dwarfstar.sh/docs/installation/)\. BENCHMARKS ## ds4 benchmarks: prefill and generation Reference rows from upstream\. Read prefill and generation separately, especially for long\-context agent workloads\. MachineContextPrefill t/sGeneration t/sM5 Max, 128 GBq2 · 2,048 tok790\.239\.4M5 Max, 128 GBq2 · 65,536 tok398\.527\.6DGX Spark, 128 GBq2 · 2,048 tok825\.818\.1DGX Spark, 128 GBq2 · 65,536 tok823\.013\.8[All benchmarks→](https://dwarfstar.sh/benchmarks/) API & AGENTS ## Use ds4 from Codex, Claude Code and OpenCode ds4\-server speaks OpenAI and Anthropic\-style APIs, so local coding agents can connect to your own machine with a base URL\. ## Own your local AI inference\. Start with the quickstart, check the hardware matrix, then connect your editor, agent or API client to the local server\.