如果你的显卡支持 Blackwell，那么请务必阅读本文！https://github.com/turbo-tan/llama.cpp-tq3… 已更新至 turbo4/turbo3 TQ3_4S——这是一种受 Google 的 TurboQuant 以及 TheTom 的 @no_stp_on_snek TurboQuant KV-cache 工作启发而设计的权重量化方案。它不再对所有张量进行均匀量化，而是将误差分配到影响最小的区域，同时保持敏感张量为高精度（Q6）。结果：体积缩小约 40%，在接近 Q3 大小的前提下实现了 Q4 级别质量——在我的测试中超越了 Q4_K_M。一直以来，问题都出在 prefill 阶段。旋转带来的开销使提示处理变慢，这一直困扰着我。后来我拿到了一台 ASUS GX10（NVIDIA DGX Spark / GB10 的姊妹款），亲眼见识了 Blackwell 的 FP4 算力有多强。于是灵光一现：TQ3_4S 天然适配 FP4。因此，我在运行时将 TQ3_4S 的块重新映射为 FP4 友好的 tile——Blackwell 直接执行其原生块缩放 FP4 Tensor Core MMA，而不是使用更重量的反量化内核。早期的 prompt/prefill 结果： • GB10 27B MTP：+221% • GB10 9B dense：+188% • RTX 5060 Ti：+37–80% 注意事项：MoE 模型几乎没有收益，decode 阶段没有变化。它让模型读取更快。首 Token 生成时间以及长提示/代码速度飞快。对于编程、长上下文和智能体（agent）工作，你将看到巨大差异。

turbo-tan/llama.cpp-tq3

来源：https://github.com/turbo-tan/llama.cpp-tq3

llama.cpp

llama 许可证：MIT (https://opensource.org/licenses/MIT) 发布页 (https://github.com/ggml-org/llama.cpp/releases) 服务端 (https://github.com/ggml-org/llama.cpp/actions/workflows/server.yml) Docker (https://github.com/ggml-org/llama.cpp/actions/workflows/docker.yml) Winget (https://github.com/ggml-org/llama.cpp/actions/workflows/winget.yml) 宣言 (https://github.com/ggml-org/llama.cpp/discussions/205) / ggml (https://github.com/ggml-org/ggml) / ops (https://github.com/ggml-org/llama.cpp/blob/master/docs/ops.md)

用 C/C++ 实现的 LLM 推理

TurboQuant TQ3_4S Blackwell FP4

本分支包含 TurboQuant TQ3_4S CUDA 路径。在 Blackwell GPU 上，prompt prefill 路径可以在运行时将 TQ3_4S 块映射为 FP4 tile，并使用 Blackwell 的 Tensor Core。GGUF 文件在磁盘上仍保持 TQ3_4S 格式；FP4 形式仅作为运行时计算路径使用。

构建标志：

# RTX 5060 Ti / sm_120
cmake -S . -B build-sm120 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=120

# GB10 / DGX Spark / sm_121
cmake -S . -B build-sm121 \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121

运行时标志：

# 快速 prompt 路径
GGML_CUDA_TQ3_4S_FP4=1 GGML_CUDA_TQ3_4S_FP4_CACHE=1

# 低 VRAM 模式
GGML_CUDA_TQ3_4S_FP4_CACHE=0 GGML_CUDA_TQ3_4S_FP4_TRANSIENT=1

# 禁用 FP4 路径
GGML_CUDA_TQ3_4S_FP4=0

推荐的 27B MTP 服务端设置：

llama-server \
  -m /path/to/Qwen3.6-27B-MTP-TQ3_4S.gguf \
  -ngl 99 -fa on -ctk q8_0 -ctv tq3_0 \
  --spec-type draft-mtp \
  --spec-draft-n-min 1 \
  --spec-draft-n-max 2 \
  --spec-draft-p-min 0.0

prompt 速度图表：docs/turboquant/tq3_4s_fp4_blackwell.html。

近期 API 变更

• libllama API 变更日志 (https://github.com/ggml-org/llama.cpp/issues/9289)
• llama-server REST API 变更日志 (https://github.com/ggml-org/llama.cpp/issues/9291)

热门话题

• Hugging Face 缓存迁移：使用 -hf 下载的模型现在存储在标准的 Hugging Face 缓存目录中，从而可以与其他 HF 工具共享。
• 指南：使用 llama.cpp 的新 WebUI (https://github.com/ggml-org/llama.cpp/discussions/16938)
• 指南：在 llama.cpp 中运行 gpt-oss (https://github.com/ggml-org/llama.cpp/discussions/15396)
• [反馈] 改善 llama.cpp 的打包以更好地支持下游用户 🤗
• 已添加对原生 MXFP4 格式的 gpt-oss 模型的支持 | PR (https://github.com/ggml-org/llama.cpp/pull/15091) | 与 NVIDIA 的合作 (https://blogs.nvidia.com/blog/rtx-ai-garage-openai-oss) | 评论 (https://github.com/ggml-org/llama.cpp/discussions/15095)
• 多模态支持已登陆 llama-server：#12898 (https://github.com/ggml-org/llama.cpp/pull/12898) | 文档
• 用于 FIM 补全的 VS Code 扩展：https://github.com/ggml-org/llama.vscode
• 用于 FIM 补全的 Vim/Neovim 插件：https://github.com/ggml-org/llama.vim
• Hugging Face Inference Endpoints 现已原生支持 GGUF！https://github.com/ggml-org/llama.cpp/discussions/9669
• Hugging Face GGUF 编辑器：讨论 (https://github.com/ggml-org/llama.cpp/discussions/9268) | 工具 (https://huggingface.co/spaces/CISCai/gguf-editor)
• 浏览器中现已支持 WebGPU，请在此处查看介绍博客/演示 (https://reeselevine.github.io/llamas-on-the-web/)。

快速入门

上手 llama.cpp 非常简单。以下是在你的机器上安装它的几种方法：

• 使用 brew、nix、winget 或 conda-forge 安装 llama.cpp
• 使用 Docker 运行——请参阅我们的 Docker 文档
• 从发布页 (https://github.com/ggml-org/llama.cpp/releases) 下载预构建的二进制文件
• 通过克隆此仓库从源码构建——请查阅 构建指南

安装完成后，你需要一个模型才能使用。请前往获取和量化模型部分了解更多信息。

示例命令：

# 使用本地模型文件
llama-cli -m my_model.gguf

# 或直接从 Hugging Face 下载并运行模型
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF

# 启动 OpenAI 兼容的 API 服务
llama-server -hf ggml-org/gemma-3-1b-it-GGUF

描述

llama.cpp 的主要目标是以最小的设置和最先进的性能在广泛的硬件上（本地和云端）实现 LLM 推理。

• 纯 C/C++ 实现，无任何依赖
• Apple silicon 是一等公民——通过 ARM NEON、Accelerate 和 Metal 框架优化
• 支持 AVX、AVX2、AVX512 和 AMX 的 x86 架构
• 支持 RVV、ZVFH、ZFH、ZICBOP 和 ZIHINTPAUSE 的 RISC-V 架构
• 1.5-bit、2-bit、3-bit、4-bit、5-bit、6-bit 和 8-bit 整数量化，用于更快的推理和减少内存使用
• 针对 NVIDIA GPU 运行 LLM 的自定义 CUDA 内核（通过 HIP 支持 AMD GPU，通过 MUSA 支持摩尔线程 GPU）
• Vulkan 和 SYCL 后端支持
• CPU+GPU 混合推理，可部分加速超过总 VRAM 容量的模型

llama.cpp 项目是开发 ggml (https://github.com/ggml-org/ggml) 库新功能的主要试验场。

模型

通常，以下基础模型的微调版本也同样支持。添加新模型支持的说明：HOWTO-add-model.md

纯文本

• LLaMA 🦙
• LLaMA 2 🦙🦙
• LLaMA 3 🦙🦙🦙
• Mistral 7B (https://huggingface.co/mistralai/Mistral-7B-v0.1)
• Mixtral MoE (https://huggingface.co/models?search=mistral-ai/Mixtral)
• DBRX (https://huggingface.co/databricks/dbrx-instruct)
• Jamba (https://huggingface.co/ai21labs)
• Falcon (https://huggingface.co/models?search=tiiuae/falcon)
• Chinese LLaMA / Alpaca (https://github.com/ymcui/Chinese-LLaMA-Alpaca) 和 Chinese LLaMA-2 / Alpaca-2 (https://github.com/ymcui/Chinese-LLaMA-Alpaca-2)
• Vigogne (French) (https://github.com/bofenghuang/vigogne)
• BERT (https://github.com/ggml-org/llama.cpp/pull/5423)
• Koala (https://bair.berkeley.edu/blog/2023/04/03/koala/)
• Baichuan 1 & 2 (https://huggingface.co/models?search=baichuan-inc/Baichuan) + derivations (https://huggingface.co/hiyouga/baichuan-7b-sft)
• Aquila 1 & 2 (https://huggingface.co/models?search=BAAI/Aquila)
• Starcoder models (https://github.com/ggml-org/llama.cpp/pull/3187)
• Refact (https://huggingface.co/smallcloudai/Refact-1_6B-fim)
• MPT (https://github.com/ggml-org/llama.cpp/pull/3417)
• Bloom (https://github.com/ggml-org/llama.cpp/pull/3553)
• Yi models (https://huggingface.co/models?search=01-ai/Yi)
• StableLM models (https://huggingface.co/stabilityai)
• Deepseek models (https://huggingface.co/models?search=deepseek-ai/deepseek)
• Qwen models (https://huggingface.co/models?search=Qwen/Qwen)
• PLaMo-13B (https://github.com/ggml-org/llama.cpp/pull/3557)
• Phi models (https://huggingface.co/models?search=microsoft/phi)
• PhiMoE (https://github.com/ggml-org/llama.cpp/pull/11003)
• GPT-2 (https://huggingface.co/gpt2)
• Orion 14B (https://github.com/ggml-org/llama.cpp/pull/5118)
• InternLM2 (https://huggingface.co/models?search=internlm2)
• CodeShell (https://github.com/WisdomShell/codeshell)
• Gemma (https://ai.google.dev/gemma)
• Mamba (https://github.com/state-spaces/mamba)
• Grok-1 (https://huggingface.co/keyfan/grok-1-hf)
• Xverse (https://huggingface.co/models?search=xverse)
• Command-R models (https://huggingface.co/models?search=CohereForAI/c4ai-command-r)
• SEA-LION (https://huggingface.co/models?search=sea-lion)
• GritLM-7B (https://huggingface.co/GritLM/GritLM-7B) + GritLM-8x7B (https://huggingface.co/GritLM/GritLM-8x7B)
• OLMo (https://allenai.org/olmo)
• OLMo 2 (https://allenai.org/olmo)
• OLMoE (https://huggingface.co/allenai/OLMoE-1B-7B-0924)
• Granite models (https://huggingface.co/collections/ibm-granite/granite-code-models-6624c5cec322e4c148c8b330)
• GPT-NeoX (https://github.com/EleutherAI/gpt-neox) + Pythia (https://github.com/EleutherAI/pythia)
• Snowflake-Arctic MoE (https://huggingface.co/collections/Snowflake/arctic-66290090abe542894a5ac520)
• Smaug (https://huggingface.co/models?search=Smaug)
• Poro 34B (https://huggingface.co/LumiOpen/Poro-34B)
• Bitnet b1.58 models (https://huggingface.co/1bitLLM)
• Flan T5 (https://huggingface.co/models?search=flan-t5)
• Open Elm models (https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca)
• ChatGLM3-6b (https://huggingface.co/THUDM/chatglm3-6b) + ChatGLM4-9b (https://huggingface.co/THUDM/glm-4-9b) + GLMEdge-1.5b (https://huggingface.co/THUDM/glm-edge-1.5b-chat) + GLMEdge-4b (https://huggingface.co/THUDM/glm-edge-4b-chat)
• GLM-4-0414 (https://huggingface.co/collections/THUDM/glm-4-0414-67f3cbcb34dd9d252707cb2e)
• SmolLM (https://huggingface.co/collections/HuggingFaceTB/smollm-6695016cad7167254ce15966)
• EXAONE-3.0-7.8B-Instruct (https://huggingface.co/LGAI-EXAONE/EXAONE-3.0-7.8B-Instruct)
• FalconMamba Models (https://huggingface.co/collections/tiiuae/falconmamba-7b-66b9a580324dd1598b0f6d4a)
• Jais (https://huggingface.co/inceptionai/jais-13b-chat)
• Bielik-11B-v2.3 (https://huggingface.co/collections/speakleash/bielik-11b-v23-66ee813238d9b526a072408a)
• RWKV-7 (https://huggingface.co/collections/shoumenchougou/rwkv7-gxx-gguf)
• RWKV-6 (https://github.com/BlinkDL/RWKV-LM)
• QRWKV-6 (https://huggingface.co/recursal/QRWKV6-32B-Instruct-Preview-v0.1)
• GigaChat-20B-A3B (https://huggingface.co/ai-sage/GigaChat-20B-A3B-instruct)
• Trillion-7B-preview (https://huggingface.co/trillionlabs/Trillion-7B-preview)
• Ling models (https://huggingface.co/collections/inclusionAI/ling-67c51c85b34a7ea0aba94c32)
• Liquid LFM2 models (https://huggingface.co/collections/LiquidAI/lfm2)
• Liquid LFM2.5 models (https://huggingface.co/collections/LiquidAI/lfm25)
• Liquid Nanos (https://huggingface.co/collections/LiquidAI/liquid-nanos)
• Hunyuan models (https://huggingface.co/collections/tencent/hunyuan-dense-model-6890632cda26b19119c9c5e7)
• BailingMoeV2 (Ring/Ling 2.0) models (https://huggingface.co/collections/inclusionAI/ling-v2-68bf1dd2fc34c306c1fa6f86)
• Mellum models (https://huggingface.co/JetBrains/models?search=mellum)

多模态

• LLaVA 1.5 models (https://huggingface.co/collections/liuhaotian/llava-15-653aac15d994e992e2677a7e), LLaVA 1.6 models (https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2)
• BakLLaVA (https://huggingface.co/models?search=SkunkworksAI/Bakllava)
• Obsidian (https://huggingface.co/NousResearch/Obsidian-3B-V0.5)
• ShareGPT4V (https://huggingface.co/models?search=Lin-Chen/ShareGPT4V)
• MobileVLM 1.7B/3B models (https://huggingface.co/models?search=mobileVLM)
• Yi-VL (https://huggingface.co/models?search=Yi-VL)
• Mini CPM (https://huggingface.co/models?search=MiniCPM)
• Moondream (https://huggingface.co/vikhyatk/moondream2)
• Bunny (https://github.com/BAAI-DCAI/Bunny)
• GLM-EDGE (https://huggingface.co/models?search=glm-edge)
• Qwen2-VL (https://huggingface.co/collections/Qwen/qwen2-vl-66cee7455501d7126940800d)
• LFM2-VL (https://huggingface.co/collections/LiquidAI/lfm2-vl-68963bbc84a610f7638d5ffa)

绑定

• Python: ddh0/easy-llama (https://github.com/ddh0/easy-llama)
• Python: abetlen/llama-cpp-python (https://github.com/abetlen/llama-cpp-python)
• Go: go-skynet/go-llama.cpp (https://github.com/go-skynet/go-llama.cpp)
• Node.js: withcatai/node-llama-cpp (https://github.com/withcatai/node-llama-cpp)
• JS/TS (llama.cpp server client): lgrammel/modelfusion (https://modelfusion.dev/integration/model-provider/llamacpp)
• JS/TS (Programmable Prompt Engine CLI): offline-ai/cli (https://github.com/offline-ai/cli)
• JavaScript/Wasm (works in browser): tangledgroup/llama-cpp-wasm (https://github.com/tangledgroup/llama-cpp-wasm)
• Typescript/Wasm (nicer API, available on npm): ngxson/wllama (https://github.com/ngxson/wllama)
• Ruby: yoshoku/llama_cpp.rb (https://github.com/yoshoku/llama_cpp.rb)
• Ruby: docusealco/rllama (https://github.com/docusealco/rllama)
• Rust (more features): edgenai/llama_cpp-rs (https://github.com/edgenai/llama_cpp-rs)
• Rust (nicer API): mdrokz/rust-llama.cpp (https://github.com/mdrokz/rust-llama.cpp)
• Rust (more direct bindings): utilityai/llama-cpp-rs (https://github.com/utilityai/llama-cpp-rs)
• Rust (automated build from crates.io): ShelbyJenkins/llm_client (https://github.com/ShelbyJenkins/llm_client)
• C#/.NET: SciSharp/LLamaSharp (https://github.com/SciSharp/LLamaSharp)
• C#/VB.NET (more features - community license): LM-Kit.NET (https://docs.lm-kit.com/lm-kit-net/index.html)
• Scala 3: donderom/llm4s (https://github.com/donderom/llm4s)
• Clojure: phronmophobic/llama.clj (https://github.com/phronmophobic/llama.clj)
• React Native: mybigday/llama.rn (https://github.com/mybigday/llama.rn)
• Java: kherud/java-llama.cpp (https://github.com/kherud/java-llama.cpp)
• Java: QuasarByte/llama-cpp-jna (https://github.com/QuasarByte/llama-cpp-jna)
• Zig: deins/llama.cpp.zig (https://github.com/Deins/llama.cpp.zig)
• Flutter/Dart: netdur/llama_cpp_dart (https://github.com/netdur/llama_cpp_dart)
• Flutter: xuegao-tzx/Fllama (https://github.com/xuegao-tzx/Fllama)
• PHP (API bindings and features built on top of llama.cpp): distantmagic/resonance (https://github.com/distantmagic/resonance) (more info) (https://github.com/ggml-org/llama.cpp/pull/6326)
• Guile Scheme: guile_llama_cpp (https://savannah.nongnu.org/projects/guile-llama-cpp)
• Swift srgtuszy/llama-cpp-swift (https://github.com/srgtuszy/llama-cpp-swift)
• Swift ShenghaiWang/SwiftLlama (https://github.com/ShenghaiWang/SwiftLlama)
• Delphi Embarcadero/llama-cpp-delphi (https://github.com/Embarcadero/llama-cpp-delphi)
• Go (no CGo needed): hybridgroup/yzma (https://github.com/hybridgroup/yzma)
• Android: llama.android

界面

（要在此处列出项目，应明确声明其依赖 llama.cpp）

• AI Sublime Text plugin (https://github.com/yaroslavyaroslav/OpenAI-sublime-text) (MIT)
• BonzAI App (https://apps.apple.com/us/app/bonzai-your-local-ai-agent/id6752847988) (proprietary)
• cztomsik/ava (https://github.com/cztomsik/ava) (MIT)
• Dot (https://github.com/alexpinel/Dot) (GPL)
• eva (https://github.com/ylsdamxssjxxdd/eva) (MIT)
• iohub/collama (https://github.com/iohub/coLLaMA) (Apache-2.0)
• janhq/jan (https://github.com/janhq/jan) (AGPL)
• johnbean393/Sidekick (https://github.com/johnbean393/Sidekick) (MIT)
• KanTV (https://github.com/zhouwg/kantv?tab=readme-ov-file) (A