Attention to those running large models locally!

Someone has transformed llama.cpp into a performance beast — BeeLlama.cpp. With the same VRAM, inference speed hits 3x, and context capacity expands by 7.5x. This isn’t just a slide deck; it’s real benchmark data.

It packs three top-tier optimization schemes into a single codebase: DFlash speculative decoding TurboQuant/TCQ KV cache compression Adaptive draft depth control

A blessing for budget users — no need to swap GPUs to fly.

https://github.com/Anbeeld/beellama.cpp…

Anbeeld/beellama.cpp

Source: https://github.com/Anbeeld/beellama.cpp

Anbeeld’s BeeLlama.cpp

BeeLlama.cpp logo

BeeLlama.cpp (or just Bee) is a performance-focused llama.cpp fork for squeezing more speed and context out of local GGUF inference. It keeps the familiar llama.cpp tools and server flow, then adds DFlash speculative decoding, adaptive draft control, TurboQuant/TCQ KV-cache compression, and reasoning-loop protection, with full multimodal support.

Not quite a pegasus, but close enough.

Here’s a plug-and-play Qwen 3.6 27B setup with a config to run it in Q5 + 200k of practically lossless KV cache + vision on a single RTX 3090 or 4090.

Support my work! (https://anbeeld.com/support)

Fork Features

• DFlash speculative decoding: --spec-type dflash drives a DFlash draft GGUF alongside the target model. The target captures hidden states into a per-layer 4096-slot ring buffer, the drafter cross-attends to the most recent --spec-dflash-cross-ctx hidden-state tokens and proposes drafts for target verification.
• TurboQuant / TCQ KV-cache compression: Five cache types (turbo2, turbo3, turbo4, turbo2_tcq, turbo3_tcq) spanning from 4x to 7.5x compression, with higher-bit options being practically lossless in many cases. Set independently with --cache-type-k and --cache-type-v.
• Adaptive draft-max control: The server adjusts the active draft horizon at runtime instead of using a fixed --spec-draft-n-max. The default profit controller compares speculative throughput against a no-spec baseline; the fringe alternative maps acceptance-rate bands to draft depth.
• Full multimodal support: When --mmproj is active, the server keeps flat DFlash available for text generation. The model can be fully offloaded to CPU with no problems to reduce VRAM pressure.
• Reasoning-loop protection: The server detects repeated hidden reasoning output and intervenes. Default mode is force-close with --reasoning-loop-window and --reasoning-loop-max-period tuning available.
• Sampled DFlash verification: --spec-draft-temp enables rejection-sampling drafter behavior. Activates when both draft and target temperature exceed zero. Draft log probabilities must be available for rejection sampling to produce correct output.
• DDTree branch verification: optional --spec-branch-budget adds branch nodes beyond the main draft path with GPU parent_ids, tree masks, and recurrent tree kernels. Disabled automatically when the target model spans more than one GPU. This one is very much work in progress!
• Request-level speculative overrides: Draft-max and branch budget can be overridden per-request through JSON fields without restarting the server.
• CopySpec model-free speculation: --spec-type copyspec provides rolling-hash suffix matching over previous tokens without a draft model.

For the full feature and public-repo comparison, read docs/beellama-features.md. For the complete argument reference, read docs/beellama-args.md.

TurboQuant (WHT-based scalar quantization) originates from TheTom/llama-cpp-turboquant (https://github.com/TheTom/llama-cpp-turboquant). TCQ (Trellis-Coded Quantization) and basic DFlash implementation originate from spiritbuun/buun-llama-cpp (https://github.com/spiritbuun/buun-llama-cpp) (paper: Closing the Gap: Trellis-Coded Quantization for KV Cache at 2-3 Bits (https://huggingface.co/datasets/spiritbuun/turboquant-tcq-kv-cache)).

DFlash Speedup

Here’s your typical “write in Python” best-case ceiling benchmark with Qwen 3.6 27B (https://huggingface.co/unsloth/Qwen3.6-27B-GGUF) using Q4_K_M drafter (https://huggingface.co/spiritbuun/Qwen3.6-27B-DFlash-GGUF) on a single RTX 3090 24GB. Like any other speculative prediction, DFlash is strongest on structured, repetitive generation: code, tests, boilerplate, JSON-like formats, and other low-entropy continuations.

Acceptance: accepted to proposed draft tokens / accepted draft tokens to final generated tokens.

This is not a claim about all workloads. DFlash can go much faster on highly predictable code generation than on normal chat. Open-ended prose is much less predictable, so gains are smaller.

On the bright side, adaptive draft-max will track how much DFlash is helping on the current task and will adjust its intensity accordingly, or even turn it off if you would dip below the baseline otherwise.

TurboQuant / TCQ cache

TurboQuant is not truly lossless at any point, but on the higher end it might very well be practically lossless for most tasks. Especially when one’s practicality is heavily influenced by VRAM constraints and how to get the most out of it.

Installation

Quickstart: DFlash on a Single GPU

For a step-by-step walkthrough with Qwen 3.6 on a 24 GB NVIDIA card (RTX 3090/4090, etc.), see docs/quickstart-qwen36-dflash.md. It covers model download, prebuilt binaries, and a tuned launch command.

Prebuilt (Windows)

Download the release archive for your CUDA version (12.4 or 13.1) from the releases page (https://github.com/Anbeeld/beellama.cpp/releases). Extract it. The server binary is llama-server.exe. Don’t forget to download a separate archive with CUDA libraries and place it in the same folder!

Building from source with -DGGML_NATIVE=ON may result in a tiny bit better performance, so it might still be a good idea to do that if/when you decide to use this fork long-term.

CUDA Build

`bash

Linux (GCC + CUDA)

cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
-DCMAKE_BUILD_TYPE=Release cmake –build build -j

Windows (MSVC + CUDA)

cmake -B build -DGGML_CUDA=ON -DGGML_NATIVE=ON ^ -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON ^ -DCMAKE_BUILD_TYPE=Release cmake –build build –config Release –parallel

macOS (Metal)

cmake -B build -DGGML_METAL=ON -DCMAKE_BUILD_TYPE=Release cmake –build build -j `

GGML_CUDA_FA_ALL_QUANTS=ON is required for TurboQuant and TCQ cache types. Add -DCMAKE_CUDA_ARCHITECTURES=86 for RTX 3090, or -DCMAKE_CUDA_ARCHITECTURES=89 for RTX 4090, if cross-compiling or building in CI without a GPU.

Other Backends

Bee inherits llama.cpp backend support, including Metal, HIP, Vulkan, SYCL, BLAS, CANN, MUSA, OpenVINO, OpenCL, and RPC. Use the upstream-style build docs in docs/build.md and backend-specific pages under docs/backend.

Common Commands

Local CLI

sh llama-cli -m model.gguf llama-cli -m model.gguf -cnv --chat-template chatml llama-cli -m model.gguf -n 256 --grammar-file grammars/json.gbnf -p "Request: schedule a call at 8pm; Command:"

OpenAI-Compatible Server

sh llama-server -m model.gguf --port 8080 llama-server -m model.gguf -c 16384 -np 4 llama-server -m model.gguf -md draft.gguf

DFlash And TurboQuant Together

sh llama-server -m target.gguf --spec-type dflash \ --spec-draft-model drafter.gguf \ --spec-draft-ngl all \ --flash-attn on --cache-type-k turbo4 --cache-type-v turbo3_tcq

Documentation

• BeeLlama features and public repo diff
• BeeLlama args reference
• Build docs
• Server docs
• Docker docs
• Performance troubleshooting

Contributing

Keep PRs small and scoped. Run the narrowest relevant tests or benchmarks before opening a PR, and include the exact commands. For fork-specific speculative decoding, DFlash, TurboQuant, or reasoning-loop changes, update the corresponding docs when behavior or args change.

Read CONTRIBUTING.md for inherited llama.cpp contribution conventions and this fork’s AI usage policy.

Dependencies

• yhirose/cpp-httplib (https://github.com/yhirose/cpp-httplib) - single-header HTTP server used by llama-server - MIT
• stb-image (https://github.com/nothings/stb) - single-header image decoder used by multimodal code - public domain
• nlohmann/json (https://github.com/nlohmann/json) - single-header JSON library - MIT
• miniaudio.h (https://github.com/mackron/miniaudio) - single-header audio decoder - public domain
• subprocess.h (https://github.com/sheredom/subprocess.h) - process launching helper - public domain
• Snowflake ArcticInference (https://github.com/snowflakedb/ArcticInference) - suffix tree and int32 map used in speculative decoding (common/suffix-tree.*, common/int32-map.h) - Apache-2.0
• Intel OpenVINO (https://github.com/openvinotoolkit/openvino) - frontend header used in OpenVINO backend (ggml/src/ggml-openvino/openvino/frontend.h) - Apache-2.0
• Intel SYCL/oneAPI - SYCL backend (ggml/src/ggml-sycl/) - Apache-2.0 WITH LLVM-exception

See the licenses/ directory for full license texts.

Support my work! (https://anbeeld.com/support)

Bird Geek | Blue Bird Society 🕊️ (@NFTCPS): Burning tokens so hard it hurts? This open-source project saves you 90%.

If you’ve used Crayfish or Hermes, you know token costs are a bottomless pit. I recently found an open-source project called OpenSquilla that specifically solves this with two core tricks:

1️⃣ Intelligent routing: Simple queries go to cheap models, complex tasks call strong models — all automatic, no manual work needed.

2️⃣