bitnet

A developer trained a tiny Mamba-based autoregressive language model and built an inference engine to run it on the 6502 processor inside a BBC Micro, demonstrating modern machine learning on 1975 hardware with only 25KB of user memory.

The author optimized a matmul kernel for BitNet's ternary models on CPU, achieving 29x speedup in isolation, but found that the model is memory-bound, resulting in only 6-10% end-to-end gain. The inference engine is available as open-source.

Project Zero is a from-scratch C99 LLM inference engine that runs BitNet and Qwen Bonsai-27B on CPU with zero dependencies, achieving 1.8× speedup over bitnet.cpp on Xeon. The project seeks community benchmarks for both models.

A user released a BitNet trainer that Microsoft never published, along with custom kernels for training and inference, while also highlighting Microsoft's bitnet.cpp inference framework for fast 1-bit LLM inference on CPUs and GPUs.

Microsoft has released a new embedding model called bitnet-embedding-0.6b on HuggingFace, but the model card is missing.

A blog post describing an efficient method to pack ternary numbers into 8-bit bytes using SIMD-friendly unpacking, achieving 1.6 bits per trit, with applications in LLM weight quantization like BitNet b1.58.

This paper introduces BitEmbed, an extreme low-bit framework for LLM-based text embeddings that converts pretrained LLM backbones into BitNet-style encoders with ternary weights and quantized activations. It achieves comparable performance to full-precision models while significantly reducing encoding and storage costs.

Microsoft open-sourced bitnet.cpp, a 1-bit LLM inference framework that enables running 100B parameter models on local CPUs without GPUs, achieving 6.17x faster inference and 82.2% less energy consumption.

The article questions why ternary language models like BitNet have not scaled beyond 2B parameters, given their initial promise, and discusses the apparent lack of progress from open-weight AI labs.

A discussion on the lack of a community project for training LLMs from scratch on consumer hardware (8GB VRAM) using modern techniques like BitNet and Muon, proposing a collaborative effort to build one.

New BitCPM4-CANN models (1B, 3B, 8B) from OpenBMB released on Hugging Face; awaiting llamacpp support for testing.

Bitnet.cpp presents a mixed-precision matrix multiplication library for efficient edge inference of ternary LLMs like BitNet b1.58, achieving up to 6.25x speedup over full-precision baselines. The system is open-sourced on GitHub.