@bkdgiffug: A model with 2.78 trillion parameters, runnable on 8GB of memory? Sounds absurd, but this project is indeed tinkering w…
Summary
A project demonstrates a 2.78-trillion-parameter model that can run on just 8GB of memory using a pure C99 inference engine with no BLAS, frameworks, or GPU reliance, trading speed for extreme memory efficiency (about 26.5 seconds/token at 8GB, 5.6 seconds/token with 128GB+).
View Cached Full Text
Cached at: 10/06/26, 01:51 PM
A model with 2.78 trillion parameters, runnable on 8GB of memory?
Sounds absurd, but this project is indeed tinkering with exactly that.
It uses pure C99 to write the inference engine, without relying on BLAS, frameworks, or GPUs. The weights are about 1.56TB, with real-world peak memory usage around 8.24GB, and the engine itself is only 176KB.
The trade-off is straightforward: the smaller the memory, the slower the speed.
① 8GB memory: about 26.5 seconds/token ② 128GB+: about 5.6 seconds/token ③ Results from both configurations are consistent; the main difference is just waiting time It’s more like an extreme inference experiment—worth checking out the source code if you’re curious about how massive models can run on everyday hardware. 🔗
Similar Articles
How i managed to run a 193B Parameter model using only 24gb of Ram
Describes Iris Ai, a system that routes queries across 8 specialized LLMs on consumer hardware, achieving large-model performance with low memory by keeping only one model active at a time and dynamic model swapping.
@savipww: a 744B parameter model just booted on a laptop with 25 gigs of ram and i read the repo twice before i believed it repo …
A 744B parameter mixture-of-experts model boots on a laptop with 25GB RAM by storing expert weights on SSD and only loading the active ~40B parameters per token, enabling local inference despite the model's size.
@UnTalNixon_exe: Forget about GPUs and million-dollar clusters. They just made the world's largest open model (Kimi K3 – 2.78 trillion p…
A new tool called kimi-k3-in-c runs the 2.78T-parameter Kimi K3 open model on a single CPU with as little as 8.24 GB RAM, streaming experts from disk and achieving deterministic output at 10-32 seconds per token.
We built our own inference engine so a 32B model could keep 262K context inside a 16 GB GPU budget. Here's an uncut run.
Blockway 团队发布了 Volundr 32B 模型及自研 Agens Engine 推理引擎（MIT 开源），通过让 72 层中的 54 层使用固定大小状态替代增长的 KV cache，使 262K 长上下文能在 16 GB GPU 预算内运行，并演示了 110K token 冷读约 69 秒、后续追问 0.5 秒回答的实测。
Computer build using Intel Optane Persistent Memory - Can run 1 trillion parameter model at over 4 tokens/sec
A community member details a custom PC build using discontinued Intel Optane Persistent Memory to successfully run the 1-trillion parameter Kimi K2.5 model locally at roughly 4 tokens per second via llama.cpp.