Built a from-scratch BitNet inference engine in pure C — 1.8× faster than bitnet.cpp on Xeon (36 tok/s), zero dependencies [BitNet & Bonsai CPU testers wanted]
Project Zero is a from-scratch C99 LLM inference engine that runs BitNet and Qwen Bonsai-27B on CPU with zero dependencies, achieving 1.8× speedup over bitnet.cpp on Xeon. The project seeks community benchmarks for both models.
Hey r/LocalLLM, Built Project Zero — a from-scratch CPU-only LLM inference engine in pure C99. It beats bitnet.cpp by 1.8× on the same hardware. We also fully support Qwen Bonsai-27B on CPU, and we are looking for the community's help to get x86 CPU benchmark data on the board for both models. What it is Single binary, zero external dependencies — no Python, no CUDA, no ONNX, no PyTorch. GCC + make + CPU. Supports: - Microsoft BitNet b1.58-2B-4T — ternary weights ({−1, 0, +1}), 1.18 GB binary, full REPL + agentic loop - Qwen Bonsai-27B — reads GGUF directly (e.g. Ternary-Bonsai-27B-Q2_0.gguf), memory-safe mmap architecture meaning it never OOMs even on constrained RAM setups. BitNet performance — the good part Hardware Project Zero bitnet.cpp Speedup Intel Xeon (Emerald Rapids, 4C) 36.25 tok/s 19.33 tok/s 1.87× i5-11300H (Tiger Lake, dual DDR4) ~16.1 tok/s ~13.0 tok/s 1.23× We're sitting at ~95% of the theoretical DRAM bandwidth ceiling on the Xeon. There's essentially nothing left to squeeze out of BitNet on that box. How the speedup happens: BitNet weights are ternary packed 4/byte. Instead of unpacking → float → FMA, we use a 3-instruction VBMI kernel (vpermi2b + vpternlogd + vpaddb) feeding directly into INT8 VNNI accumulation (vpdpbusds). The thread pool is C11 atomics spin-then-sleep to eliminate futex syscalls. The Community Challenge: BitNet & Bonsai Benchmarks We've only benchmarked BitNet on 2 machines so far. We need to see if the fallback ternary kernels still provide a speedup on older CPU architectures, and map out the memory bandwidth ceiling on server hardware. Furthermore, PrismML is actively looking for community benchmark numbers for Bonsai-27B. Right now, every single entry on their leaderboard is GPU-based (CUDA/Metal/MLX). Zero CPU-only x86 entries exist. We want to change that. Because Project Zero uses a zero-copy mmap architecture, you can run Bonsai-27B on severely constrained hardware without crashing. If you have an older AVX2 chip, or a high-core Xeon/EPYC, we want to know what token rates you get for either model. How to test & benchmark Clone and build: bash git clone https://github.com/shifulegend/project-zero.git cd project-zero make demo Run BitNet or Bonsai-27B: ```bash For BitNet (b1.58-2B-4T): ./adaptive_ai_engine --model models/bitnet-b1.58-2B-4T.bin --tokenizer models/bitnet-b1.58-2B-4T_tokenizer_proper.bin --threads 4 For Qwen Bonsai-27B (GGUF): ./adaptive_ai_engine --model models/Ternary-Bonsai-27B-Q2_0.gguf --threads 4 ``` Where to post results: You can post your results right here in this thread, or drop them in Discussion #3 on the repo. Repo: https://github.com/shifulegend/project-zero Happy to answer questions about the ternary kernel design, the AVX-512 VNNI dispatch, the DRAM bottleneck, or why we focused on Bonsai-27B!
Bitnet.cpp presents a mixed-precision matrix multiplication library for efficient edge inference of ternary LLMs like BitNet b1.58, achieving up to 6.25x speedup over full-precision baselines. The system is open-sourced on GitHub.
A user released a BitNet trainer that Microsoft never published, along with custom kernels for training and inference, while also highlighting Microsoft's bitnet.cpp inference framework for fast 1-bit LLM inference on CPUs and GPUs.
A highly efficient AI model architecture using ternary weights (-1, 0, 1) that achieves competitive performance while requiring only 1.58 bits per parameter, enabling deployment on extremely constrained devices.
This paper introduces BitEmbed, an extreme low-bit framework for LLM-based text embeddings that converts pretrained LLM backbones into BitNet-style encoders with ternary weights and quantized activations. It achieves comparable performance to full-precision models while significantly reducing encoding and storage costs.