prometheusAIR/Kimi-K3-REAP55-GGUF · Hugging Face

Source: https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#kimi-k3-reap55-iq1_m-ggufKimi-K3-REAP55-IQ1_M-GGUF

Kimi-K3 (2.8T-A104B) with 55% of its experts removed, then quantised to ~2.15 bpw — 319 GiB.

REAP expert pruning (896 →400experts per layer, all 92 MoE layers) followed byiq1\_mgate/up +iq2\_xxsdown with a self-built imatrix. Dense weights atq8\_0.

So far, the smallest Kimi-K3 GGUF that has been shown to work. Highly experimental build mainly for comparison purposes. Built specifically to avoid theiq1\_s/iq2\_s/iq3\_stensor types, which have broken CUDA matmuls onsm_120(RTX PRO 6000 Blackwell, RTX 50-series) and silently produce garbage.

size319 GiB(8 shards)experts400 / 896 per layer (55.4% pruned), top-16 unchangedsaliency retained77.6% of router-weighted expert saliencyexpert quantiq1\_mgate/up,iq2\_xxsdowndense / embed / outputq8\_0overall~2.15 bpwneedle test15/15 exactthrough 64K, at 15% / 50% / 85% depthdecode0.48 tok/s warm, 0.32 tok/s cold*(see hardware note)*prefill82–90 tok/s at depth

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#requires-llamacpp-pr-26185Requires llama.cpp PR #26185

Kimi-K3 support isnot in any llama.cpp release. You needPR #26185(kimi\-k3arch). Nothing here works without it.

Known chat parser issue

llama.cpp’s auto-parser derives delimiters by diffing template renders and picks the shortest distinguishing strings. For XTML that gives it <|sep|> and <|close|> — both of which appear in every channel. So reasoning terminates at the first bare <|close|> and the remaining scaffolding falls into content. The fix is a dedicated handler matching full channel tags, dispatched on <|open|>think<|sep|> + <|open|>response<|sep|>, following the same pattern llama.cpp already uses for GPT-OSS’s channel format. It also wires up tool calls — currently untested and almost certainly broken for the same ambiguity reason.

Parser fix for llama.cpp

You must build the small patch included here (kimi-k3-chat-parser.patch). See Building llama.cpp below. If/when the architecture is merged upstream, this requirement goes away.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#running-itRunning it

llama-server \
  --model Kimi-K3-REAP55-IQ1_M-00001-of-00008.gguf \
  -ngl 99 -ot "ffn_.*_exps=CPU" --no-repack \
  -c 32768 -ctk f16 -ctv f16 \
  -b 8192 -ub 8192 \
  --threads 8 --fit off --jinja

Four of those flags are not obvious and were each measured, not guessed:

• **\-\-no\-repackis mandatory.**Without it the CPU backend claims the expert tensors for its repack buffer and tries to allocate the entire pool in RAM; the load aborts withfailed to allocate CPU\_REPACK buffer of size \.\.\..
• \-ngl 99, never a partial value.llama.cpp offloads thelastN layers (i\_gpu\_start = n\_layer \+ 1 \- ngl), so any partial\-nglputslayer 0on CPU — and layer 0 is K3’s single dense layer, carrying a fused Gated Delta Net op. The result is a hard abort inggml\-cuda\.cuafterprompt processing appears to succeed.GGML\_CUDA\_DISABLE\_GRAPHS=1does not help. This works becauseq8\_0dense is only ~55 GiB.
• \-ub 8192is ~5x faster than the defaulton a disk-streamed setup. Every ubatch draws essentially all experts, so each one costs a full sweep of the expert pool — you want asfewsweeps as possible. Measured on a 5099-token prompt:\-ub 512= 1383 s,\-ub 8192= 274 s. This is the opposite of the usual VRAM-resident intuition.
• **\-\-threads 8, not more.**Prefill here is memory-bandwidth-bound, not core-bound. On a 16-core/32-thread part, 8 threads beat 12 by 11.7% and 30 threads were slower than 12. Tune to whereyouraggregate memory bandwidth peaks.

Keep mmapon(do not pass\-\-no\-mmap) — the expert pool streams from disk by design.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#hardware-note–the-speeds-above-are-storage-bound-not-model-boundHardware note — the speeds above are storage-bound, not model-bound

Measured on: RTX PRO 6000 Blackwell (96 GB), Ryzen 9 9950X3D (16c/32t), 125 GiB DDR5, 2× Samsung 9100 PRO Gen5 NVMe in RAID0.

Dense weights sit on the GPU; the 262 GiB expert pool streams from NVMe through the page cache. With 125 GiB of RAM only ~42% of the pool stays cached, sodecode speed is set by how fast your storage and page cache can supply expert weights, not by the GPU. Consequences:

• **Warm ≠ cold.**Sustained work keeps the pool warm and reaches ~0.48 tok/s; a first request after load is ~0.32. Both are honest, they measure different things.
• **More RAM helps more than a faster GPU.**If the pool fits in page cache, decode becomes RAM-bandwidth-bound instead and should be several times faster.
• Decode isflat with context— 0.43–0.51 tok/s from 4K to 64K, no roll-off, because the cost is streaming experts rather than attention.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#what-was-pruned-and-what-that-costs-youWhat was pruned, and what that costs you

REAP (REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression, Cerebras) scores each expert bygate x \|\|expert\_output\|\|over a calibration set and keeps the highest. Saliency was measured over264,141 tokenswith a purpose-built tool that hooks llama.cpp’s graph directly, so no PyTorch pipeline or original safetensors were needed.

The calibration corpus was deliberately narrow— roughly 33% code, 28% technical prose, 20% tool-calling/agentic traces, 20% long-context — andmultilingual content was dropped entirely.

**Whatever a calibration corpus under-represents gets pruned away silently.**Expect this build to be weaker than the full model on multilingual work, and on domains far from code, technical English, reasoning and tool use. If you need breadth, use an unpruned quant.

Every sample was rendered through K3’s real chat template with populated thinking channels, so the experts were scored on the chat+reasoning distribution the model actually runs in rather than on raw text.

Prune depth in context: Cerebras validated ~30–50% pruning with near-parity; at 50% on Qwen3-480B-Coder they retained 97.6% of coding ability.This build is at 55%, just past that envelope, which is why it is domain-targeted rather than general-purpose.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#validationValidation

• Census: 2573 tensors, 276 expert tensors at 400 slots, expert params exactly276 x 11,010,048 x 400, zero sm_120-broken types.
• Imatrix coverage: all 400 kept experts in all 92 layers received activations (minimum 225, mean 10,813) — no expert was quantised blind.
• Needle-in-a-haystack: 15/15 exact retrieval at 4K/8K/16K/32K/64K x depth 0.15/0.5/0.85.
• Reasoning spot-check: correct arithmetic with self-verification, and it terminates naturally rather than running to the token cap.

Not tested: multilingual, tool-calling end-to-end, context beyond 64K (1M would need 28 GiB of KV alone), and any standard benchmark suite. Perplexity is not quoted because the only corpus measured was our own calibration set, which flatters a model pruned toward it.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#known-issuesKnown issues

• XTML markers leak intocontent(<\|open\|\>think<\|sep\|\>,<\|close\|\>response<\|sep\|\>). This is an upstream parser gap, reproducible on the unpruned model — not a defect of this build.
• K3 is always-thinking and there is no reasoning-budget control yet, so allow a generous completion budget or answers get truncated inside the thinking block.Apply included patch for temporary fix

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#provenanceProvenance

Converted from the nativeMXFP4-QATrelease. The routed experts ship as MXFP4, so conversion to GGUF is a lossless byte repack — this build derives from a full-precision master, not from a requantised intermediate. Pruning was done in GGUF space by slicing the expert dimension, which is outermost and therefore contiguous per expert, so surviving experts are byte-exact copies.

Licensed under theKimi-K3 license, inherited from the base model.

https://huggingface.co/prometheusAIR/Kimi-K3-REAP55-GGUF#building-llamacppBuilding llama.cpp

The patch is generated against upstream mastercf11c4c05(2026-07-27), where it is verified to apply cleanly and compile with zero errors. A nearby master should work with minor fuzz.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout cf11c4c05
git apply /path/to/kimi-k3-chat-parser.patch

cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120   # 120 = Blackwell; set to your GPU
cmake --build build --target llama-server -j