@HeLi8866: 卧槽!微软开源的这个项目,是准备改写AI行业的游戏规则!?! 项目叫 bitnet.cpp,核心是用1.58-bit三值量化技术,把大模型庞大的浮点运算直接变成了CPU最擅长的整数加减法。 带来的结果就是: · 在x86 CPU上,推理速…
摘要
微软开源的 bitnet.cpp 项目采用 1.58-bit 三值量化技术,将大模型浮点运算转为 CPU 整数运算,在 x86 和 ARM CPU 上实现最高 6.17 倍推理加速和 82% 能耗降低,使个人设备可离线运行千亿参数模型。
查看缓存全文
缓存时间: 2026/11/20 07:57
卧槽!微软开源的这个项目,是准备改写AI行业的游戏规则!?!😱
项目叫 bitnet.cpp,核心是用1.58-bit三值量化技术,把大模型庞大的浮点运算直接变成了CPU最擅长的整数加减法。
带来的结果就是👇:
· 在x86 CPU上,推理速度最高提升6.17倍,能耗最高降低82.2%
· 在ARM CPU(比如苹果M系列)上,速度提升1.37到5.07倍,能耗降低55%到70%
· 单个CPU就能跑1000亿参数的模型,生成速度达到每秒5到7个token,基本就是人类正常的阅读速度。
这意味着什么?你的MacBook、你的ThinkPad,不再只是一个连云端API的终端,它本身就可以变成一个完全离线、数据不出本地的AI工作站。隐私和成本,一次性解决。
当年英伟达靠GPU算力筑起的护城河,微软现在正试图用“算法瘦身”的方式从底层绕过去。
虽然它没法完全取代数据中心的大规模GPU集群,但对于个人开发者、隐私敏感场景和边缘设备来说,这是一个全新的可能性。
GitHub 仓库: https://github.com/microsoft/BitNet… https://x.com/byteonai/status/2122717876054310952/video/1…
microsoft/BitNet
Source: https://github.com/microsoft/BitNet
📰 News
07/23/2026: 📣 We released VibeASR.cpp — a real-time multilingual ASR inference engine on CPU using BitNet I2_S quantization, achieving RTF < 1 with very few threads on x86 (AVX2) and ARM (NEON) platforms. [Code] [Models] [Report]
07/20/2026: 📣 We released BitNet-embedding-0.6B and BitNet-embedding-270M on Hugging Face — the first 1-bit embedding models that deliver competitive embedding quality with significantly faster inference on CPUs.
- 1.42x to 2.28x speedup over F16 on BitNet-embedding-0.6B prefill (8 threads)
- 1.32x to 1.74x speedup over F16 on BitNet-embedding-270M prefill (8 threads)
- Supports I2_S conversion with optimized kernels on x86 CPUs
- Lossless inference with 2 bits per weight
07/16/2026: 📣 Released BitNet Embeddings 0.6B/270M: I2_S Conversion and Inference Optimization — detailed guide for converting and running BitNet embedding models with optimized I2_S kernels.
01/15/2026: 📣 Released BitNet CPU Inference Optimization — parallel kernel implementations with configurable tiling and embedding quantization support, achieving 1.15x to 2.1x additional speedup over the original implementation.
05/20/2025: 📣 Released BitNet Official GPU inference kernel — extending 1-bit inference beyond CPUs.
04/14/2025: 📣 Released BitNet Official 2B Parameter Model on Hugging Face — the first official BitNet b1.58 model trained with 4T tokens.
02/18/2025: 📑 Bitnet.cpp: Efficient Edge Inference for Ternary LLMs — system-level paper on bitnet.cpp’s architecture and design.
11/08/2024: 📑 BitNet a4.8: 4-bit Activations for 1-bit LLMs — enabling 4-bit activations for further efficiency gains.
10/21/2024: 📑 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs — the technical report behind bitnet.cpp.
10/17/2024: 📣 bitnet.cpp 1.0 released.
03/21/2024: 📑 The-Era-of-1-bit-LLMs: Training Tips, Code, FAQ
02/27/2024: 📑 The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits — the foundational paper introducing BitNet b1.58.
10/17/2023: 📑 BitNet: Scaling 1-bit Transformers for Large Language Models — the original BitNet paper.
Overview
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels that support fast and lossless inference of 1.58-bit models on CPU and GPU (NPU support coming next).
Try it out via this online demo, or build and run it on your own CPU or GPU.
bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the technical report for more details.
Model Releases
1. BitNet-b1.58-2B-4T - 1-bit Large Language Model
BitNet-b1.58-2B-4T is the first official BitNet b1.58 model with 2.4B parameters, trained on 4 trillion tokens. It is a ternary (1.58-bit) language model that delivers competitive performance with full-precision models of similar size while enabling significantly faster and more energy-efficient inference.
- Fast CPU Inference: Achieves up to 6.17x speedup on x86 CPUs and 5.07x on ARM CPUs compared to full-precision models.
- Energy Efficient: Reduces energy consumption by up to 82.2% on x86 and 70.0% on ARM.
- GPU Support: Official GPU inference kernel available for accelerated deployment.
- Chat-Ready: Supports conversational mode for interactive use.
🤗 Hugging Face | 🔗 Online Demo | 📄 Technical Report
2. BitNet-embedding-0.6B - 1-bit Embedding Model
BitNet-embedding-0.6B is a 0.6B-parameter 1-bit embedding model that achieves competitive embedding quality with significantly faster CPU inference. It is the first model to demonstrate that ternary weights can deliver strong performance on embedding tasks.
- 1.42x to 2.28x speedup over F16 on prefill (8 threads, x86)
- Lossless Quality: Competitive embedding quality with 2 bits per weight
- I2_S Kernel: Supports optimized I2_S conversion on x86 CPUs
3. BitNet-embedding-270M - Lightweight 1-bit Embedding Model
BitNet-embedding-270M is a compact 270M-parameter 1-bit embedding model designed for resource-constrained environments, offering fast inference with minimal memory footprint.
- 1.32x to 1.74x speedup over F16 on prefill (8 threads, x86)
- Lossless Quality: Competitive embedding quality with 2 bits per weight
- Lightweight: Only 270M parameters for edge deployment scenarios
Supported Models
| Model | Parameters | CPU | Kernel | ||
|---|---|---|---|---|---|
| I2_S | TL1 | TL2 | |||
| Official Models | |||||
| BitNet-b1.58-2B-4T | 2.4B | x86 | ✅ | ❌ | ✅ |
| ARM | ✅ | ✅ | ❌ | ||
| BitNet-embedding-0.6B | 0.6B | x86 | ✅ | ❌ | ❌ |
| ARM | ❌ | ❌ | ❌ | ||
| BitNet-embedding-270M | 270M | x86 | ✅ | ❌ | ❌ |
| ARM | ❌ | ❌ | ❌ | ||
| Community Models | |||||
| bitnet_b1_58-large | 0.7B | x86 | ✅ | ❌ | ✅ |
| ARM | ✅ | ✅ | ❌ | ||
| bitnet_b1_58-3B | 3.3B | x86 | ❌ | ❌ | ✅ |
| ARM | ❌ | ✅ | ❌ | ||
| Llama3-8B-1.58-100B-tokens | 8.0B | x86 | ✅ | ❌ | ✅ |
| ARM | ✅ | ✅ | ❌ | ||
| Falcon3 Family | 1B-10B | x86 | ✅ | ❌ | ✅ |
| ARM | ✅ | ✅ | ❌ | ||
| Falcon-E Family | 1B-3B | x86 | ✅ | ❌ | ✅ |
| ARM | ✅ | ✅ | ❌ | ||
❗️We use existing 1-bit LLMs available on Hugging Face to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.
Installation
Requirements
- python>=3.10
- cmake>=3.22
- clang>=18
-
For Windows users, install Visual Studio 2022. In the installer, toggle on at least the following options(this also automatically installs the required additional tools like CMake):
- Desktop-development with C++
- C++-CMake Tools for Windows
- Git for Windows
- C++-Clang Compiler for Windows
- MS-Build Support for LLVM-Toolset (clang)
-
For Debian/Ubuntu users, you can download with Automatic installation script
bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
-
- conda (highly recommend)
Build from source
If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands. Please refer to the FAQs below if you see any issues.
- Clone the repo
git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet
- Install the dependencies
# (Recommended) Create a new conda environment
conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp
pip install -r requirements.txt
- Build the project
# Manually download the model and run with local path
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s
usage: setup_env.py [-h] [--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}] [--model-dir MODEL_DIR] [--log-dir LOG_DIR] [--quant-type {i2_s,tl1}] [--quant-embd]
[--use-pretuned]
Setup the environment for running inference
optional arguments:
-h, --help show this help message and exit
--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}, -hr {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}
Model used for inference
--model-dir MODEL_DIR, -md MODEL_DIR
Directory to save/load the model
--log-dir LOG_DIR, -ld LOG_DIR
Directory to save the logging info
--quant-type {i2_s,tl1}, -q {i2_s,tl1}
Quantization type
--quant-embd Quantize the embeddings to f16
--use-pretuned, -p Use the pretuned kernel parameters
Usage
Basic usage
# Run inference with the quantized model
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
usage: run_inference.py [-h] [-m MODEL] [-n N_PREDICT] -p PROMPT [-t THREADS] [-c CTX_SIZE] [-temp TEMPERATURE] [-cnv]
Run inference
optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Path to model file
-n N_PREDICT, --n-predict N_PREDICT
Number of tokens to predict when generating text
-p PROMPT, --prompt PROMPT
Prompt to generate text from
-t THREADS, --threads THREADS
Number of threads to use
-c CTX_SIZE, --ctx-size CTX_SIZE
Size of the prompt context
-temp TEMPERATURE, --temperature TEMPERATURE
Temperature, a hyperparameter that controls the randomness of the generated text
-cnv, --conversation Whether to enable chat mode or not (for instruct models.)
(When this option is turned on, the prompt specified by -p will be used as the system prompt.)
Demo
A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:
https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
Benchmark
We provide scripts to run the inference benchmark providing a model.
usage: e2e_benchmark.py -m MODEL [-n N_TOKEN] [-p N_PROMPT] [-t THREADS]
Setup the environment for running the inference
required arguments:
-m MODEL, --model MODEL
Path to the model file.
optional arguments:
-h, --help
Show this help message and exit.
-n N_TOKEN, --n-token N_TOKEN
Number of generated tokens.
-p N_PROMPT, --n-prompt N_PROMPT
Prompt to generate text from.
-t THREADS, --threads THREADS
Number of threads to use.
Here’s a brief explanation of each argument:
-m,--model: The path to the model file. This is a required argument that must be provided when running the script.-n,--n-token: The number of tokens to generate during the inference. It is an optional argument with a default value of 128.-p,--n-prompt: The number of prompt tokens to use for generating text. This is an optional argument with a default value of 512.-t,--threads: The number of threads to use for running the inference. It is an optional argument with a default value of 2.-h,--help: Show the help message and exit. Use this argument to display usage information.
For example:
python utils/e2e_benchmark.py -m /path/to/model -n 200 -p 256 -t 4
This command would run the inference benchmark using the model located at /path/to/model, generating 200 tokens from a 256 token prompt, utilizing 4 threads.
For the model layout that do not supported by any public model, we provide scripts to generate a dummy model with the given model layout, and run the benchmark on your machine:
python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile models/dummy-bitnet-125m.tl1.gguf --outtype tl1 --model-size 125M
# Run benchmark with the generated model, use -m to specify the model path, -p to specify the prompt processed, -n to specify the number of token to generate
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128
Convert from .safetensors Checkpoints
# Prepare the .safetensors model file
huggingface-cli download microsoft/bitnet-b1.58-2B-4T-bf16 --local-dir ./models/bitnet-b1.58-2B-4T-bf16
# Convert to gguf model
python ./utils/convert-helper-bitnet.py ./models/bitnet-b1.58-2B-4T-bf16
Acknowledgements
This project is based on the llama.cpp framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp’s kernels are built on top of the Lookup Table methodologies pioneered in T-MAC. For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.
FAQ (Frequently Asked Questions)📌
Q1: The build dies with errors building llama.cpp due to issues with std::chrono in log.cpp?
A: This is an issue introduced in recent version of llama.cpp. Please refer to this commit in the discussion to fix this issue.
Q2: How to build with clang in conda environment on windows?
A: Before building the project, verify your clang installation and access to Visual Studio tools by running:
clang -v
This command checks that you are using the correct version of clang and that the Visual Studio tools are available. If you see an error message such as:
'clang' is not recognized as an internal or external command, operable program or batch file.
It indicates that your command line window is not properly initialized for Visual Studio tools.
• If you are using Command Prompt, run:
"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64
• If you are using Windows PowerShell, run the following commands:
Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll" Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"
These steps will initialize your environment and allow you to use the correct Visual Studio tools.
相似文章
microsoft/BitNet
微软官方 BitNet 仓库(bitnet.cpp)的持续更新合集:提供 1-bit/1.58-bit LLM 的高效 CPU/GPU 推理框架,近期新增 BitNet-embedding 0.6B/270M 嵌入模型、I2_S 量化优化内核以及实时多语种 ASR 推理引擎 VibeASR.cpp。
@XAMTO_AI: 终端AI编程助手这个赛道现在卷得是真离谱,oh-my-pi 算是这周杀出来的一匹黑马,目前Star数已经13.7万多了 它直接在终端里跑,最核心的卖点就一个——改代码改得极准。 这个项目是从Pi分支出来的,在原版基础上加了一大堆硬货。 最…
介绍了终端AI编程助手oh-my-pi,它从Pi分支出来,凭借Hashline编辑系统实现高精度代码修改,减少61% token消耗,内置32个工具,支持40多种语言和40多个LLM提供商,可直接从主流工具导入配置,迁移成本低。
@shenxiankk: 我靠!这群极客简直把 CPU 榨出了神仙性能! 一块独立显卡都不用,光靠 64G 内存就硬生生跑起 284B 大模型! 大模型圈子刚蹦出一个逆天级开源狠货:synchopate/llama-modchip。 搁以前,想跑 2840 亿参数…
开源项目 synchopate/llama-modchip 发布 v2.5.5,通过定制量化与 SIMD 优化,让纯 CPU 加 64GB 内存即可运行 284B 大模型(如 DeepSeek-V4),无需任何独立显卡,并提供 Web UI 与 Claude Code 后端支持。
@MaxForAI: 卧槽,这个项目可以让你的4060跑大模型???? 来自 UC Berkeley、 MIT、UT Austin 的研究者,刚刚开源了 FreeToken:专门解决超大 MoE 模型在个人电脑上的推理框架。 根据官方的论文,目前已经跑出了几组…
FreeToken是一个开源推理框架,由UC Berkeley、MIT和UT Austin的研究者开发,专门用于在个人电脑上运行超大MoE模型,通过动态分配GPU、CPU和内存资源实现高效推理。
我开发了 BitRun:一个独立的 1-bit LLM 运行器(在 Apple Silicon 上达到 100–127 tokens/s,在 PC CPU 上达到 40–45 tokens/s,且无需任何编译)
BitRun 是一个可在普通笔记本 CPU 上零编译运行 1-bit 三值 LLM 的独立工具,在 Apple M4 Pro 上可达 98–127 t/s、Intel i5 上 40–45 t/s,并修复了微软 bitnet.cpp 代码库中的多个底层 C++ 内核 bug 并提交了上游 PR。