@shenxiankk: Holy cow! These geeks are squeezing godlike performance out of a CPU! No dedicated GPU needed — they're running a 284B large model purely on 64GB of RAM! An insane open-source bombshell just dropped in the LLM community: synchopate/llama-modchip. Back in the day, running 284 billion parameters…
Summary
The open-source project synchopate/llama-modchip released v2.5.5, enabling a 284B large model (such as DeepSeek-V4) to run on a pure CPU with just 64GB of RAM — no dedicated GPU required — via custom quantization and SIMD optimizations, along with a Web UI and Claude Code backend support.
View Cached Full Text
Cached at: 11/16/26, 10:18 AM
Damn! These geeks squeezed god-tier performance out of a CPU! Not a single discrete GPU — with 64GB of RAM alone, they got a 284B large model running! A mind-blowing open-source beast just dropped in the LLM scene: synchopate/llama-modchip.
Back in the day, running a 284-billion-parameter DeepSeek-V4 would have taken several 8×H100 servers. This project flipped the whole table: no discrete GPU required — just an ordinary home CPU and 64GB of system RAM, and it stuffed the 284B behemoth right in, running it blazing fast. Mingzi cross-checked the official benchmark data to verify its terrifying specs:
- Zero GPUs, all RAM: No need to buy expensive graphics cards. The 284B DeepSeek model is compressed to just 56.9GB of usage, swallowed whole by a normal 64GB desktop — with no virtual memory swapping triggered the whole way through.
- Pure CPU speed is genuinely sufficient: Thanks to vector instruction set optimization, the 284B model’s generation speed holds steady at 9–14 tok/s, spiking up to 25 tok/s with speculative sampling, while cosine similarity stays at 99.15%.
- A 30B model takes off: Swap in a 30B-class MoE model — only 13.3GB in size — and even a 16GB ultrabook can handle it, with pure CPU hitting a blistering 111 tokens per second.
- Batteries included: Ships with a dependency-free dark Web UI and terminal interface, and even natively supports being plugged straight into Claude Code as a local backend.
But for those wanting to hop on board, keep these two limitations in mind first:
- You need a genuine 64GB of physical memory; anything less will trigger constant Swap paging and freeze the system.
- It’s based on a custom patched build of llama.cpp — the official standard Ollama can’t recognize this special quantization format, so you have to launch it with the bundled script.
GPU vendors are probably breaking a sweat seeing this project — taming a 284B model to this degree on CPU alone. How much RAM does your machine have right now? Dare to drag down this 284B colossus and give it a spin? 👉 Open-source repo: https://github.com/synchopate/llama-modchip…
synchopate/llama-modchip
Source: https://github.com/synchopate/llama-modchip
llama-modchip (v2.5.5)
100% Donor-Free ConeCollapse (MaxQ-v4) & 3.00–3.75 bpw SIMD Inference for 284B & 30B MoE Models
Continuous ref5632.kld & WikiText-2 PPL · Logit KLD · 100-Pt Gate · Linux, macOS Metal, CUDA, Vulkan & Windows 11
Release License Quality Gate VQ3%20%C2%B7%20PPL%2FKLD-2ea043?style=flat-square) Technical Paper PDF Architecture Drafter-d29922?style=flat-square) Submit Benchmark (https://github.com/synchopate/llama-modchip/issues/new?template=cpu_benchmark_submission.yml)
Quickstart · Signal Fidelity & Benchmarks · How 3D VQ & Fractal-64 Work · Quantization Specs · Supported Models · Leaderboard · v2.4 Paper Snapshot (paper/) · FAQ
100/100 (30B) · 70/80 Func. (284B) Quality Gate + WikiText-2 / wiki-alt PPL 99.15% / 97.78% CosSim vs BF16 (Fractal / VQ3) 57.99 / 56.93 GiB DeepSeek-V4 284B (MaxQ-v4 / Fractal) 13.30 GiB Nemotron-30B VQ3 GGUF 24.2–111.6 tok/s 30B MoE CPU spec decode 10.4 ms/dev/shm Claude 40-tool restore 0GPUs required Hardware Class System RAM Target Checkpoint Throughput (0 GPUs) Architecture & Memory Notes Developer laptop / Mini-PC 16–32 GB Nemotron-3.5-30B-A3B-VQ3 (13.30 GiB, Aligned Default; opt-in nemotron-ablit) 24.2–26.1 Gate / 40.9–60.4 M=7 tok/s 14.6 GiB VmRSS (-c 32768) / 16.04 GiB (-c 262144); WikiText-2 PPL 7.4400 (6.9582 BF16); 100/100 Gate 64 GB Desktop / Server 64 GB DeepSeek-V4-Flash 284B MaxQ-v4 (57.99 GiB, 100% Donor-Free Default) / MaxQ-Fractal (56.93 GiB) / VQ3 (56.40 GiB) 36.1–41.1 cold / 44.4 warm pp · 8.66–9.06 N=1 / 8.8–25.3 spec tg 57.54–59.13 GiB anonymous-heap VmRSS in ~62.5 GiB usable RAM (-c 32768 -ub 512 desktop default; -c 262144 headless 64GB w/ zram or 96GB+) Hybrid Desktop (CPU + 16 GB GPU) 32–64 GB Nemotron-30B / Qwen3-30B (3.75–4.25 bpw) 80.80 tok/s Hot attention/SSM trunk on GPU (-ngl 99), cold routed MoE experts in CPU RAM 8-Channel Workstation (Threadripper) 128 GB+ DeepSeek-284B / Nemotron-30B 111.62 tok/s (30B) 8-CCD AVX-512BW _1x8 register tiling saturates 8-channel DDR5 bandwidth
Overview
Mixture-of-Experts (MoE) inference on CPU is bounded primarily by DRAM bandwidth per generated token. Standard 4.50-bpw scalar quantization (Q4_0) leaves significant memory-bandwidth headroom unused, while conventional 2–3 bit scalar K-quants (Q2_K, Q3_K) require multi-shift bit unpacking and fall back to Q4_0 whenever expert matrix dimensions are not multiples of 256 (such as 1856 × 2688 in Nemotron-3.5-30B-A3B). llama-modchip v2.5.5 ships 100% donor-free ConeCollapse + FP64 Woodbury DualGram (MaxQ-v4, < 58.00 GiB, zero custom tensor types — native Metal, CUDA, Vulkan, ARM64 NEON, & x86 SIMD) alongside two block-64 quantization formats engineered for SIMD register lookup—Fractal-64 (GGML_TYPE_Q2_0 = 42, 30 bytes / 64 weights = 3.75 bpw) and LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 24 bytes / 64 weights = 3.00 bpw)—paired with cache-aware KV management (HARP-Cache), /dev/shm Slot-0 prefix snapshots, and RouteCrack + Stockfish-PVS v2 speculative decoding:
-
Domain-Aware
LUT-VQ3-64 + Fractal-64Hybrid Quantization: On uncompressedBF16MoE weights (blk.1.ffn_up_exps.weight),Fractal-64(3.75 bpw) reaches17.75 dBSQNR (99.15%Cosine Similarity,+1.45 dBoverQ3_K), whileLUT-VQ3-64(3.00 bpw) reaches13.57 dBSQNR (97.78%Cosine Similarity,+3.42 dBoverQ2_K) via 128 3D Lloyd-Max centroids decoded in registers with AVX-512 VBMIVPERMI2B. Keeping boundary layers and residual output projections (ffn_down_exps, MLA attention) inFractal-64while encoding interior expansion projections inLUT-VQ3-64cutsup_expsfootprint by33.3%vs.Q4_0with a+0.4818(+6.92%)WikiText-2Perplexity delta vs. TrueBF16(7.4400vs.6.9582on12,288wiki.test.rawtokens;+0.0790/+1.07%vs.Fractal-647.3610),0.1495 / 0.0944 nats/tokLogit KLD vs. TrueBF16(78.1%Step-0 /85.2%Matched Top-1,98.0%Top-5), and100.0 / 100.0onbenchmarks/eval_quality_and_tools.py. -
DeepSeek-V4-Flash-0731 284B(13.17BActive) Inside64 GBRAM: Five 3-shard0731checkpoints execute resident inside64 GBDDR5 RAM, led by the 100% donor-free ConeCollapse safety-aligned defaultDeepSeek-V4-Flash-0731-MaxQ-v4(https://huggingface.co/Synchopate/DeepSeek-V4-Flash-0731-MaxQ-v4-64GB-CPU-GGUF) (57.99 GiB/62,270,037,248B across1,362standardggmltensors —0custom tensor types, built directly from0731 @ 7872f01bBF16/FP8/MXFP4via Deterministic Routed-Stride2048 -> 1792ConeCollapse on34layers + FP64 Woodbury DualGram +34exactF32blk.*.ffn_down_exps.bias+446/4462D tensors weighted by clean UTF-824,576-tokenimatrix_v4_clean_24k.dat+129routed-MoE tensors weighted by@antirez’s1.5M-tokenrouted-moe-ds4-1p5m.dat[@ f71f23d552d6,chat-v2]:AtomicChatwiki-alt.txtctx=563251-chunkref5632.kld[143,565scored]Mean PPL(Q) = 6.8592 ± 0.0482,Mean KLD = 0.64851 ± 0.00354 nats,Median KLD = 0.18382,Same top p = 73.283%,RMS Δp = 27.562%,Max KLD = 20.5424vsMean PPL(base) = 4.3310[-6.01%PPL,-12.36%Mean KLD,-32.02%Median KLD &+2.886pp Top-1 vsMaxQ-Fractal7.2977/0.74000; on the 49 unseen chunks3..51(137,935tok):PPL(Q) = 6.8471,Mean KLD = 0.62574(-12.84%vsMaxQ-Fractal0.71791;-3.54%vsMaxQ-v30.64872, paired p = 6.58 \times 10^{-5}),Same top p = 73.595%(+1.152pp vsv3, p = 1.26 \times 10^{-13}),RMS Δp = 26.726%(-0.793pp, p = 8.84 \times 10^{-10})]), alongside single-passDeepSeek-V4-Flash-0731-MaxQ-Fractal(https://huggingface.co/Synchopate/DeepSeek-V4-Flash-0731-MaxQ-Fractal-64GB-CPU-GGUF) (56.93 GiB,--preset dsv4-maxq-fractal,51cPPL = 7.2977 ± 0.0504,Mean KLD = 0.74000,WikiText-2PPL = 11.6139 ± 0.5845,70.0 / 80Functional Gate [30/30Tools,20/30Code,20/20Needle;8.66 tok/spureN=1/8.80 tok/sServed Gate]), its opt-in uncensored twinDeepSeek-V4-Flash-0731-Abliterated-Fractal(https://huggingface.co/Synchopate/DeepSeek-V4-Flash-0731-Abliterated-Fractal-64GB-CPU-GGUF) (56.93 GiB,--preset dsv4-ablit,WikiText-2PPL = 11.3944 ± 0.5650,70.0 / 80Functional Gate [8.75 tok/sServed Gate],0.0959step-0 /0.0512 nats/tokprefix-matched KLD vsMaxQ-Fractal), compactDeepSeek-V4-Flash-0731-VQ3(https://huggingface.co/Synchopate/DeepSeek-V4-Flash-0731-VQ3-64GB-CPU-GGUF) (56.40 GiB,Unsloth UD-IQ1_S @ fbbb5b93gate/updonor,wiki-alt.txt12cPPL = 7.6579 ± 0.1077,WikiText-2PPL = 12.2345 ± 0.5976,64.0 / 80Functional Gate,9.05 tok/sN=1), andDeepSeek-V4-Flash-0731-Fractal(https://huggingface.co/Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF) (56.57 GiB,WikiText-2PPL = 11.5288 ± 0.5563,64.0 / 80Functional Gate,9.06 tok/sN=1). -
Nemotron-3.5-30B-A3B(3.0BActive) in16–24 GiBRAM: Safety-alignedNVIDIA-Nemotron-3.5-30B-A3B-VQ3-64(https://huggingface.co/Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64-CPU-GGUF) (13.30 GiB,WikiText-2PPL = 7.4400 ± 0.2626vs. TrueBF166.9582 ± 0.2415) andFractal-64(https://huggingface.co/Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Fractal-64-CPU-GGUF) (14.36 GiB,PPL = 7.3610 ± 0.2602,0.1254 / 0.0929 natsKLD vs.BF16) save-4.44to-5.67 GiBRAM vs.Q4_0at24.18–25.02 tok/sServed Gate decode (+23.7%to+28.0%vs.Q4_019.54 tok/s, with up to48.39 avg / 60.40 peak tok/sonQ4_0in-contextM=7refactor bursts), alongside an opt-in uncensoredAbliterated-VQ3-64(https://huggingface.co/Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-Abliterated-VQ3-64-CPU-GGUF) (13.30 GiB,PPL = 7.7721 ± 0.2835,0.0308 / 0.0312 natsKLD &87.5%Step-0 /92.4%Matched Top-1 vs.VQ3-64,--preset nemotron-ablit). -
Sub-20ms
/dev/shmSlot-0 Restore + Cross-Platform Native Support: Restores the 40-tool Claude Code prefix (2,776tokens) from/dev/shmin10.4 ms(0.16sTTFT) on Linux, runs on Apple Silicon Metal (57.65 tok/sM1 Max,78.60 tok/sM3 Max,27.0 tok/sM5 Max284B), and compiles natively on Windows 11 (MinGW-w64 UCRT64, verified up to1Mcontext).
Operating Trade-Offs & Engine Constraints:
- Tensor Types & Engine Requirements:
DeepSeek-V4-Flash-0731-MaxQ-v4(57.99 GiB) uses zero custom tensor types (100%standardggmltypes:F32,BF16,I32,Q8_0,Q6_K,Q5_K,Q4_K,MXFP4,IQ2_XXS,IQ1_M,IQ1_S— native across CPU AVX2/AVX-512/NEON, Metal, CUDA MMQ, and Vulkan), requiring only thev2.5.5deepseek4.cpploader support for per-layern_ff_exp_i(1792/2048) andblk.*.ffn_down_exps.bias. By contrast,Fractal-64(GGML_TYPE_Q2_0 = 42,30 B / 64 W) repurposes upstreamGGML_TYPE_Q2_0 = 42(18 B / 32 W) in-place andLUT-VQ3-64addsGGML_TYPE_VQ3_0 = 43; vanillallama.cpp,Ollama,vLLM, andLM Studiocannot loadFractal-64,VQ3-64, ordeepseek4GGUFs without thellama-modchipruntime.- 64 GB Anonymous-Heap Margin (
284B):--load-mode none --repackallocates57.54–59.13 GiBin non-evictable anonymous heap (VmRSS). On64 GBLinux hosts (~62.5 GiBusable RAM), use-c 32768 -ub 512and enablezramor close heavy desktop browsers; reserve-c 262144(256Kcontext) for headless64 GBnodes with swap/zramor96 GB+systems.- Dual-Channel DDR5 Throughput Envelope (
284B): On 2-channel DDR5-5600 (~75 GB/ssustained), rawN=1unprunedTop-6decode is8.66–9.06 tok/s(8.80–9.55 tok/sin Served Gate decode) and cold prefill is36.1–44.0 tok/s(30.7 tok/sat3,038tokens); higher sustained rates require multi-channel servers (EPYC/Threadripper) or AppleM5 Max(286.5 GB/sunified memory).
Signal Fidelity and End-to-End Benchmarks
1. Direct Signal Fidelity on Uncompressed BF16 MoE Weights
Measured via quantize_row_*_ref \to dequantize_row_* on the uncompressed 59.0 GiB BF16 master checkpoint (blk.1.ffn_up_exps.weight, N = 4,194,304 weights, kurtosis \kappa = 3.2–4.8):
| Quantization Format | Block Layout & Size | Effective bpw | Direct BF16 SQNR | Cosine Sim vs. BF16 | Relative L_2 Error | SIMD Decode Path |
|---|---|---|---|---|---|---|
BF16 (Reference) | 128 B / 64 weights | 16.000 bpw | \infty\text{ dB} | 100.00% | 0.00% | Native F32/BF16 Dot |
Baseline-Q4_0 | 36 B / 64 weights | 4.500 bpw | 21.09 dB | 99.61% | 8.82% | Scalar Uniform AVX2/AVX-512 |
Fractal-64 (GGML_TYPE_Q2_0 = 42) | 30 B / 64 weights | 3.750 bpw | 17.75 dB (+1.45 dB vs Q3_K) | 99.15% | 12.96% | Single-Cycle Dyadic VPSHUFB / Metal |
Standard Q3_K | 27.5 B / 64 weights | 3.438 bpw | 16.30 dB | 98.84% | 15.32% | Multi-Shift Q3_K Unpack |
LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43) | 24 B / 64 weights | 3.000 bpw | 13.57 dB (+3.42 dB vs Q2_K) | 97.78% | 20.97% | Single-Cycle 3D VQ VPERMI2B |
Standard Q2_K | 21 B / 64 weights | 2.625 bpw | 10.15 dB | 95.37% | 31.08% | Multi-Shift Q2_K Unpack |
2. End-to-End Quality & Speed Ablations (benchmarks/eval_quality_and_tools.py + llama-bench)
Measured on 16-Core AMD Zen 5 (--cpu-range 0-15 --cpu-strict 1, Transparent HugePages enabled; full methodology and shipped-hybrid vs. full-VQ3 ablation breakdown in benchmarks/QUALITY_GATE_VQ3_64.md).
A. NVIDIA-Nemotron-3.5-30B-A3B (WikiText-2 PPL, 32-Probe / Multi-Step Logit KLD vs. True BF16, & 100-Point Gate)
| Metric / Suite | Reference BF16 | Standard Q4_0 | Fractal-64 (v2.4.0) | LUT-VQ3-64 (v2.5.4 Aligned) | Abliterated-VQ3-64 (Opt-In) |
|---|---|---|---|---|---|
| GGUF File Size on Disk | 58.84 GiB | 17.60 GiB (15.95 MXFP4) | 14.36 GiB | 13.30 GiB (13.08 GiB --all-up-exps1) | 13.30 GiB |
Resident RAM (VmRSS, 256K Context) | 59.40 GiB | 21.71 GiB | 17.27 GiB | 16.04 GiB1 (-5.67 GiB vs Q4_0) | (~16.04 GiB via VQ31) |
WikiText-2 (wiki.test.raw, 12,288 tok) PPL | 6.9582 ± 0.2415 | (7.2927 ± 0.2538 MXFP4) | 7.3610 ± 0.2602 (+5.79%) | 7.4400 ± 0.2626 (+6.92% vs BF16, +1.07% vs Fractal) | 7.7721 ± 0.2835 (+4.46% vs VQ3) |
Logit KLD vs. True BF16 (32p Step-0 / Matched) | 0.0000 nats (100% Ref) | (0.1623 / 0.1204 MXFP4, 78.1% / 88.4% Top-1) | 0.1254 / 0.0929 nats (81.2% / 89.8% Top-1) | 0.1495 / 0.0944 nats (78.1% / 85.2% Top-1) (0.0522 vs Fractal) | 0.1431 / 0.1109 nats (75.0% / 84.4% Top-1) (0.0308 / 0.0312 vs VQ3) |
Served Gate Decode (Stockfish-PVS E2E, Suite 4) | 7.82 tok/s (N=1 Ref) | 19.54 tok/s (66.7% acc) | 25.02 tok/s (59.1% acc; 26.08 MXFP4) | 24.18 tok/s1 (70.2% acc, +23.7% vs Q4_0) | (~24.2 tok/s via VQ31) |
In-Context Refactor Burst (M=7 / Hybrid) | — | 40.91–60.40 tok/s (48.39 avg M=7) | 71.3–78.6 tok/s (M3/M4 Max Metal) | — (Gate E2E above1) | — |
1–3. Tools (30), Code (30), & Needle (20) | 80.0 / 80.0 | 80.0 / 80.0 | 80.0 / 80.0 | 80.0 / 80.01 | (80.0 via VQ31; 8/8 Audit) |
Total Quality Gate Score (100 pts) | 100.0 / 100.0 | 100.0 / 100.0 | 100.0 / 100.0 | 100.0 / 100.01 | (100.0 via VQ31) |
(1 WikiText-2 PPL [7.4400] and BF16/same-family Logit KLD [0.1495 / 0.0944, 0.0522] on LUT-VQ3-64 and Abliterated-VQ3-64 [7.7721, 0.0308] are measured directly on the shipped 13.30 GiB GGUFs; eval_nemotron_vq3_64.json [24.18 tok/s, 16.04 GiB VmRSS, 100/100 Gate] was recorded on the 13.08 GiB 23-layer --all-up-exps VQ3 ablation and serves as the architecture speed/VmRSS proxy for Abliterated-VQ3-64.)
B. DeepSeek-V4-Flash-0731 284B (284.33B Total / 13.17B Active, Unpruned Top-6 Routing)
| Metric / Suite | Reference BF16 (150.8 GiB MXFP4) | Fractal-64 (56.57 GiB) | LUT-VQ3-64 (56.40 GiB) | MaxQ-Fractal (56.93 GiB, v2.5.4) | MaxQ-v4 (57.99 GiB, v2.5.5 Default) | Abliterated-Fractal (56.93 GiB, Opt-In) |
|---|
Similar Articles
@AISuperDomain: Stop buying multi-GPU workstations to run large models! Open-source inference engine FreeToken integrates CPU, GPU, and memory: 8GB VRAM slim laptops run 35B MoE, home single-GPU gaming laptops handle 290B+! Completely solves the VRAM capacity issue, open-source and free: #AI #LLM …
Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.
@NFTCPS: 4GB VRAM running 70B large model? It actually works! AirLLM did a clever trick — layered inference, not loading the whole model into VRAM at once, but layer by layer, compute and discard, squeezing the giant into a small GPU. The best part: 100% open source, freebie warning https://github.com/0xSo…
AirLLM is a fully open-source tool that uses layered inference (loading and releasing VRAM layer by layer) to enable 70B large language models to run on GPUs with only 4GB VRAM, without quantization, distillation, or pruning. It already supports running Llama3.1 405B on 8GB VRAM.
@NFTCPS: Running large models locally – looking at the hundreds of GB of weights and VRAM requirements, most people are discouraged immediately, and I was too. Colibri changes the game by treating VRAM, RAM, and disk as a unified hierarchy, streaming weights from disk on demand. Written in pure C with zero dependencies, it already has over 25,000 stars. Here are a few points: …
Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.
@lxfater: With 64GB RAM, the 100-billion-parameter model is already deployed. We've got the hammer — now where's the nail?
A 12GB GPU with 64GB of system RAM is enough to run a 100-billion-parameter large model: the open-source project Strata can run quantized versions of Qwen3.8-Flash-Next — about 94 tokens/s with the Q2_0 quantization and about 53 tokens/s with IQ3_S on an RTX 5070. The author remarks that the hammer is ready; the question is where the nail (i.e., the application scenario) is.
@NFTCPS: Attention to those running large models locally! Someone has transformed llama.cpp into a performance beast — BeeLlama.cpp. With the same VRAM, inference speed triples and context capacity expands 7.5x. This isn't a slide deck; it's real benchmark data. It stuffs three top-tier optimizations into one codebase: DFlash speculative decoding…
BeeLlama.cpp is a fork of llama.cpp that integrates DFlash speculative decoding, TurboQuant/TCQ KV-cache compression, and adaptive draft control, achieving up to 3x faster inference and 7.5x context expansion on the same hardware.