Recommend this project: One person built an inference engine in pure C that runs a 744-billion-parameter GLM-5.2 on a consumer machine with 25 GB of RAM. No GPU, no BLAS, no Python runtime — just about 1,300 lines of C. The core insight is simple: the MoE model only activates ~40 billion parameters per token; the dense part stays resident in memory, and the routed experts are streamed from disk.

Colibri: a 25 GB machine running a 744B model
Colibri (Italian for “hummingbird”) is a pure-C inference engine — one .c file and a few small headers, zero dependencies, running entirely on CPU.

Core insight
GLM-5.2 is a 744B-parameter Mixture-of-Experts model. But it only activates ~40B parameters per token: the dense part is ~17B (attention, shared experts, embeddings), at int4 it’s ~9.9 GB and stays RAM-resident. The 21,504 routed experts are spread across 75 layers, each ~19 MB at int4, totaling ~370 GB. They live on NVMe, streamed on demand, with a per‑layer LRU cache and an optional pinned hot‑expert area.

A hummingbird weighs just a few grams, hovers in the air, and visits a thousand flowers a day. This engine uses hummingbird rations to keep a 744B giant alive: 25 GB RAM, 12 CPU cores, and a lot of disk patience.

Measured data (WSL2, 12 cores, 25 GB RAM, NVMe):

• Model on disk (int4 container): ~370 GB
• Resident RAM (dense part, int4): 9.9 GB
• Load time: ~30 s
• Peak RSS: ~20 GB (auto‑capped)
• Cold decode: ~11 GB disk read per token → 0.05–0.1 tok/s
• MTP speculative decoding (int8 head): 2.2–2.8 tok/forward

On other machines (community data):

• Apple M5 Max (128 GB, internal SSD, 14.2 GB/s disk): 1.06 tok/s
• PCIe5 NVMe, 64 GB RAM plus 40 GB pinned experts: estimated 2–4 tok/s
• 128–256 GB RAM, 12 cores: estimated 2–4 tok/s (matmul‑bound)

Technical highlights

MLA attention + compressed KV-cache
576 floats per token instead of 32,768 (57× smaller). MLA weight absorption (DeepSeek’s trick) means no per‑token k/v reconstruction during decode.

MTP speculative decoding
GLM-5.2’s own multi‑token‑prediction head (layer 78) drafts tokens; the main model verifies in one batched forward. At int8 the acceptance rate is 39–59%. Lossless — stays lossless under sampling via rejection sampling. 2.2–2.8 tokens/forward.

Asynchronous expert pre‑read
While a set of experts is being multiplied, the kernel is already reading the next set.

Self‑learning cache
The engine records which experts your usage actually routes to (.coli_usage, updated every turn). At startup, it automatically pins the hottest experts into spare RAM. The more you use Colibri, the faster it gets — not a metaphor.

Integer arithmetic kernels
AVX2 maddubs optimizations give int8 matmuls 1.4–2.5× speedup.

RAM safety
The expert cache auto‑sizes from MemAvailable at startup, estimating peak usage (working set, KV, MTP row, reconstruction buffers) honestly, so the kernel’s OOM‑killer never fires. On a 24 GB machine it automatically limits to 2 slots per layer.

Installation
A single shell script does everything — checks GCC/OpenMP, builds, self‑tests:

cd c && ./setup.sh
./coli convert --model /nvme/glm52_i4
COLI_MODEL=/nvme/glm52_i4 ./coli chat

The converter is one command: shard‑by‑shard download of GLM-5.2-FP8 (the full 756 GB never needs to exist on disk at once), conversion to int4 container, resumable if interrupted. Pre‑built int4 models are available on HuggingFace.

Quality benchmarks pending
The author has never measured how much accuracy the int4 quantization loses — on the dev machine’s ~1 GB/s disk a full run takes most of a day. Full‑precision GLM-5.2 scores ~85–95% on MMLU/HellaSwag/ARC. If the int4 container is within a few percent of those numbers, the quantization is validated; if not, it’s time to invest in mixed/group‑scale quantization.

Summary
This is a project done by one person on a 12‑core, 25 GB RAM laptop. It’s not a proof‑of‑concept — it chats.
If you have a faster machine, you’ll get interactive inference speeds. If you have data, please file an issue.
This is the future of open‑source AI: not bigger GPU clusters, but one person making a huge model run on a small machine.

Project: https://github.com/JustVugg/colibri…
HuggingFace: https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4…
License: Apache 2.0
#opensource #localAI #GLM5.2 #MoE

JustVugg/colibri

Source: https://github.com/JustVugg/colibri

Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.

$ ./coli chat 🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU ✓ pronto in 32s · residente 9.9 GB › ciao! ◆ Ciao! 😊 Come posso aiutarti oggi?

The idea

A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:

• the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
• the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.

The engine is a single C file (c/glm.c, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.

What’s implemented

• Faithful GLM-5.2 (glm_moe_dsa) forward — validated token-exact against a transformers oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
• MLA attention (q/kv-LoRA, interleaved partial RoPE) with compressed KV-cache: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
• DeepSeek-V3-style sigmoid router (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
• Native MTP speculative decoding — GLM-5.2’s own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. The head must be int8 (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it’s 39–59% acceptance, 2.2–2.8 tokens/forward (community-measured, #8 (https://github.com/JustVugg/colibri/issues/8)). Lossless — and stays lossless under sampling via rejection sampling. Honest caveat from the same measurement: on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net time loss until the cache/pin warms up — the adaptive guard and DRAFT=0 are there for that.
• True sampling — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
• Integer-dot kernels (Q8_0-style int8 activations, AVX2 maddubs): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
• MLA weight absorption (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs kv_b, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere.
• Async expert readahead: while one block of experts is being multiplied, the kernel is already reading the next (WILLNEED).
• Quantization kernels: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container.
• DSA sparse attention: in progress — the lightning-indexer weights (a ~108 GB extraction from the FP8 repo, --indexer converter mode) are downloading; the indexer forward lands next. Until then attention is dense and exact for contexts ≤ 2048 tokens.
• Batch-union MoE: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it.
• Byte-level BPE tokenizer in C (GPT-2-style with Unicode-property regex, 320k merges).
• RAM safety: the expert cache is auto-sized from MemAvailable at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires.
• Offline FP8→int4 converter (c/convert_fp8_to_int4.py): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine’s container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable.

Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)

This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.

SSD note

Cold starts are heavy on random reads (~11 GB/token), but reads don’t meaningfully wear an SSD — colibrì’s streaming is read-only. The real concerns under heavy use are (1) swap traffic if the system runs out of RAM (writes do wear the drive — keep a sane --ram budget; colibrì’s auto-budget is designed to stay clear of swap) and (2) sustained thermals: hours at full read duty cycle will heat cheaper drives. Monitor drive temperature and health.

Download the model

A pre-converted GLM-5.2 int4 model for colibrì is available on Hugging Face: https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4

If the MTP files there are still the int4 head (see #8 (https://github.com/JustVugg/colibri/issues/8) — sizes 1765523544/2686077736/536747200 = int4, unusable), grab the int8 MTP heads from the community clone by matey-0: https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp

Download the repository and point COLI_MODEL to its directory:

COLI_MODEL=/path/to/GLM-5.2-colibri-int4 ./coli chat

This skips the FP8 → int4 conversion step entirely. Thanks DatPat for your help!

Quick start

cd c
./setup.sh   # checks gcc/OpenMP, builds, self-tests
# ONE command does everything model-side: downloads GLM-5.2-FP8 shard by shard
# (never needs the full 756 GB at once), converts to the int4 container, then
# converts the MTP head for speculative decoding. Resumable at any point.
# Conversion (only) needs python with: pip install torch safetensors huggingface_hub numpy
./coli convert --model /nvme/glm52_i4   # ~400 GB free on a real ext4/NVMe path
# chat — RAM budget, expert cache and MTP are all detected automatically:
COLI_MODEL=/nvme/glm52_i4 ./coli chat

The engine at runtime is pure C — python is only used by the one-time converter.

Experimental resident CUDA backend

colibrì includes an opt-in CUDA backend for model-resident tensors. Streaming experts deliberately remain on the original CPU path for now: copying an expert from NVMe to the GPU on every use would only replace the disk bottleneck with a PCIe bottleneck. Resident quantized tensors are uploaded lazily once and reused.

cd c
make cuda-test CUDA=1   # q8/q4/q2/f32 kernel correctness
make CUDA=1   # optional dense-path experiment (hot experts are configured below)
COLI_CUDA=1 COLI_GPU=0 CUDA_DENSE=1 SNAP=/nvme/glm52_i4 ./glm 64 4 4

Requirements: Linux, an NVIDIA driver, and a CUDA Toolkit under /usr/local/cuda (override with CUDA_HOME=/path/to/cuda). CUDA_ARCH=native builds for the GPU in the current machine; set an explicit architecture when cross-compiling. Requesting CUDA with a CPU-only binary, an invalid device, or an unavailable runtime fails at startup instead of silently falling back. The normal make build and runtime behavior are unchanged.

CUDA defaults to an expert-only accelerator: resident dense/attention tensors stay on CPU because fixture measurements show that moving them does not help while expert I/O is the bottleneck. CUDA_DENSE=1 keeps the earlier all-resident experimental path.

A measured PIN profile can promote its hottest experts into the persistent VRAM tier while keeping the rest in RAM:

STATS=stats.txt SNAP=/nvme/glm52_i4 ./glm 64 4 4   # collect routing frequencies first
COLI_CUDA=1 COLI_GPU=0 CUDA_EXPERT_GB=16 \
  PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4
# multi-GPU expert tier, 96 GB total budget across six devices
COLI_CUDA=1 COLI_GPUS=0,1,2,3,4,5 CUDA_EXPERT_GB=96 \
  PIN=stats.txt PIN_GB=160 SNAP=/nvme/glm52_i4 ./glm 64 4 4

Selected experts are uploaded during startup, so capacity failures occur before inference and the log reports their exact tensor footprint. The budget is clamped against free VRAM after reserving the projected dense resident set and 2 GB of runtime headroom per selected device. With COLI_GPUS, CUDA_EXPERT_GB is a total budget across the device set; experts are assigned whole to the least-loaded device that can hold them. A NUMA-local RAM backing store is not implemented yet.

Current limitations: devices use independent contexts and synchronous host-staged activation copies—there is no P2P/NCCL dependency yet. The kernels are correctness-first custom kernels rather than cuBLAS/Tensor Core kernels. This draft intentionally makes no end-to-end speedup claim before the full model is benchmarked.

For a reproducible backend A/B without the full checkpoint, generate the deterministic 313M-parameter glm_moe_dsa fixture and run fixed-token replay:

cd c
python make_glm_bench_model.py --output /nvme/colibri-bench-medium --device cuda
python benchmark_cuda_fixture.py --model /nvme/colibri-bench-medium --gpu 0

The fixture has random weights and is not a language model. It exists only to preserve the real MLA/MoE/streaming shapes and compare CPU streaming, dense-only CUDA, CPU hot-store, and CUDA hot-expert execution with identical replay tokens.

Useful knobs (env or flags): --temp T token sampling temperature (default 0.7 + nucleus 0.90 — tuned for int4; 0 = greedy), --topp 0.7 adaptive expert top-p (30–40% less disk), --ngen N max tokens per answer (:piu in chat continues a truncated one), AUTOPIN=0 disable the learning cache’s auto-pin, THINK=1 enable GLM-5.2’s reasoning block, DRAFT=n MTP draft depth, TF=1 teacher-forcing validation, PILOT=1 router-lookahead disk prefetch (experimental — see below), CAP_RAISE=0 don’t auto-grow the expert cache.

The expert cache auto-sizes to your RAM (since 2026-07-10): the engine now raises the LRU cap to fill your --ram budget instead of only lowering it. Before this fix a 128 GB machine ran with the same 8-experts/layer cache as a 16 GB one (issue #12) — if you benchmarked colibrì before this date, rerun: your numbers were capped.

Router-lookahead prefetch (PILOT=1, experimental): GLM-5.2’s expert routing is measurably predictable ahead of time — applying layer L+1’s router to layer L’s post-attention state recalls 71.6% of the true top-8 (vs 41.3% for “same experts as last token”). PILOT=1 uses this to issue next-layer expert readahead from a dedicated I/O thread while the current layer computes. On our dev box the disk is already ~80% saturated, so it measures neutral; on machines where compute and disk are balanced (like the Ryzen AI 9 in issue #12: 43% disk / 46% matmul) it should overlap real work — measurements welcome.

The learning cache: the engine records which experts your usage actually routes to (.coli_usage next to the model, updated every turn) and at startup automatically pins the hottest ones in spare RAM. colibrì literally gets faster the more you use it.

Got a better machine? Try it — here’s what to expect

colibrì was built on deliberately humble hardware (12 cores, 25 GB RAM, NVMe behind a WSL2 VHDX that caps random reads at ~1 GB/s). Every one of those constraints is a knob your machine can turn up.

The engine needs: Linux (or WSL2), gcc with OpenMP, AVX2, ≥16 GB RAM, and the ~370 GB int4 model on a local NVMe (ext4 — never a network/9p mount).

How to test it, in order:

cd c && ./setup.sh   # build + architecture self-test (expects 32/32)

# 1) measure YOUR disk the way the engine uses it (parallel 19 MB random reads):
gcc -O2 -fopenmp iobench.c -o iobench
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 0   # buffered, 8 threads
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1   # O_DIRECT

# 2) chat; watch the per-turn stats line (tok/s, expert hit-rate, RSS):
COLI_MODEL=/path/to/glm52_i4 ./coli chat

# 3) record expert usage, then pin the hottest experts in your spare RAM:
STATS=stats.txt ./coli chat
PIN=stats.txt PIN_GB=20 ./coli chat   # scale PIN_GB to your free RAM

# 4) quality benchmarks (MMLU/HellaSwag/ARC):
./coli bench

Back-of-envelope predictions (decode is disk-bound: a cold token costs ~11.4 GB of expert reads; MTP speculation roughly halves the effective cost once the cache is warm; RAM turns cold reads into free cache hits):

These are estimates, not measurements — if you run colibrì on serious hardware, please open an issue with your numbers: real datapoints from better machines are exactly what this project needs next.

Community benchmarks (measured)

Real numbers from real machines, stock build (setup.sh, gcc 13), greedy decoding, --ngen 32, MTP active:

| machine | disk (