I got Kimi-k3 running.....
Summary
User successfully runs the Kimi-k3 model using llama.cpp on high-end hardware, achieving low tokens per second (0.41 prompt eval, 0.23 generation).
Similar Articles
Kimi K3 full model running on 16x GB10 cluster at 20+tps
Kimi K3 full model runs on a 16x GB10 cluster at 20+ tokens per second average, with plans to publish the vllm image and instructions.
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
WASTE is a new open-source C inference engine that streams expert weights from disk to run the 2.78-trillion-parameter Kimi K3 model on a consumer laptop with just 29 GB of RAM, achieving 0.49–0.54 tokens/s.
First Kimi K3 results on home lab ~ 4t/s
User shares first home-lab results running Kimi K3 on 768GB DDR5 and 2x5090 using a llama.cpp fork and Q2_K quant, reporting prefill speeds of 50-70 tps and decoding tps that increases over time.
@UnTalNixon_exe: Forget about GPUs and million-dollar clusters. They just made the world's largest open model (Kimi K3 – 2.78 trillion p…
A new tool called kimi-k3-in-c runs the 2.78T-parameter Kimi K3 open model on a single CPU with as little as 8.24 GB RAM, streaming experts from disk and achieving deterministic output at 10-32 seconds per token.
Kimi K3 in and on C (c99 and cpu)
A project claims to run Kimi K3, a 2.78T parameter MoE model, on CPU with 8GB RAM by streaming expert weights from NVMe SSD and using MXFP4 compression, trading speed for memory efficiency.