@NFTCPS: Running large models locally – looking at the hundreds of GB of weights and VRAM requirements, most people are discouraged immediately, and I was too. Colibri changes the game by treating VRAM, RAM, and disk as a unified hierarchy, streaming weights from disk on demand. Written in pure C with zero dependencies, it already has over 25,000 stars. Here are a few points: …
Summary
Introducing colibri, an open-source inference engine written in pure C that unifies VRAM, RAM, and disk as a hierarchical structure to stream large model weights, supporting various cutting-edge MoE models to run locally on consumer hardware, lowering the barrier to using large models.
View Cached Full Text
Cached at: 08/23/26, 03:38 PM
Website · Discord · English · Simplified Chinese · Traditional Chinese · Italiano
Web Dashboard (./coli web): A 744B model running at 4 tokens per second, TTFT 1.6s, zero disk usage—full expert residency on 6× RTX 5090 GPUs, featuring live token metrics, per-turn time breakdown, a VRAM/RAM/disk tier bar, and a live mini-brain visualization in the corner.
Brain Page: All 19,456 experts visualized as a living cortex—color indicates storage tier, brightness reflects routing heat, and every expert invoked during a turn flashes white. Hovering reveals the expert’s measured topic affinity.
Atlas Page: The measured expert atlas displayed as a 3-D galaxy—13,260 characterized experts, 1,041 replicated specialists clustered by topic (poetry, law, Chinese, SQL…). Position represents measured routing affinity, not a learned embedding. Drag to spin.
Similar Articles
@NFTCPS: 4GB VRAM running 70B large model? It actually works! AirLLM did a clever trick — layered inference, not loading the whole model into VRAM at once, but layer by layer, compute and discard, squeezing the giant into a small GPU. The best part: 100% open source, freebie warning https://github.com/0xSo…
AirLLM is a fully open-source tool that uses layered inference (loading and releasing VRAM layer by layer) to enable 70B large language models to run on GPUs with only 4GB VRAM, without quantization, distillation, or pruning. It already supports running Llama3.1 405B on 8GB VRAM.
@yibie: Recommend this project—a single person wrote an inference engine in pure C, making the 744-billion-parameter GLM-5.2 run on a consumer machine with 25GB RAM. No GPU, no BLAS, no Python runtime—about 1300 lines of C. The core insight is simple: MoE…
Colibri is an inference engine written in pure C, approximately 1300 lines of code, zero dependencies. It can run the 744-billion-parameter GLM-5.2 MoE model on a consumer machine with 25GB RAM, achieved by streaming loaded routing experts and efficient caching, no GPU or Python runtime needed.
@Michaelzsguo: https://x.com/Michaelzsguo/status/2053217839729791221
This article is a guide for local large model deployment, covering hardware selection, memory calculations, Runtime tool comparisons, and model quantization options, helping users from getting started to optimizing their local inference experience.
@AISuperDomain: Stop buying multi-GPU workstations to run large models! Open-source inference engine FreeToken integrates CPU, GPU, and memory: 8GB VRAM slim laptops run 35B MoE, home single-GPU gaming laptops handle 290B+! Completely solves the VRAM capacity issue, open-source and free: #AI #LLM …
Open-source inference engine FreeToken integrates CPU, GPU, and memory, enabling consumer hardware like 8GB VRAM laptops to run 35B MoE models, and home single-GPU gaming laptops to run 290B+ models, completely solving the VRAM limitation.
Wow, this project lets your 4060 run large models??? Researchers from UC Berkeley, MIT, and UT Austin have just open-sourced FreeToken: an inference framework specifically designed for running ultra-large MoE models on personal computers. According to the official paper, several sets of data have been produced...
FreeToken is an open-source inference framework developed by researchers from UC Berkeley, MIT, and UT Austin, specifically designed for running ultra-large MoE models on personal computers, achieving efficient inference by dynamically allocating GPU, CPU, and memory resources.