prism-ml/Ternary-Bonsai-2-27B-gguf · Hugging Face

Source: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf

Prism ML Website|Whitepaper|Demo & Examples|Discord

Full 27B-class reasoning in ternary transformer weights, for llama.cpp (CUDA, Metal, CPU)

~9.3xsmaller than FP16 (ideal) |98.2%of FP16 intelligence retained |~47 tok/son an Apple M5 Max laptop

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#highlightsHighlights

• ~5.9 GBlanguage model (down from ~54 GB FP16) — full 27B-class reasoning on a standard laptop or a single GPU
• 98.2% of FP16 intelligence retained: 84.78 average across 14 thinking-mode benchmarks — far above the conventional IQ2_XXS build (72.59) at less than two-thirds of its footprint, and within 0.4 points of UD-Q4_K_XL at three times the footprint
• Retains thinking, reasoning, and agentic behaviordeep in the sub-4-bit regime, where conventional low-bit representations collapse: math within half a point of full precision (96.57), coding level with the baseline (89.42), agentic tool calling at 74.92
• End-to-end ternary language weightsacross embeddings, attention projections, MLP projections, and LM head, at atrue1.72 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships as a separate Q8_0 mmproj pack
• 262K-token contexton-device, kept practical by the Qwen3.8-27B hybrid-attention backbone (~75% linear attention)
• Two GGUF packingswith custom ternary hybrid-attention kernels for llama.cpp (CUDA, Metal) —PTQ1_0packs trits densely (1.75 bits/weight, 5.95 GB),PQ2_0stores each trit in a 2-bit slot (2.13 bits/weight, 7.21 GB); packed weights are consumed directly, never expanded back to FP16
• MLX companion: also available asTernary-Bonsai-2-27B-mlx-2bitfor native Apple Silicon inference

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#resourcesResources

• Whitepaper— full methodology, benchmarks, and measurement notes
• Demo & examples—the source of truth for running these models: tested setup for every backend, pinned binaries, serving, benchmarking and integration, kept current as the runtimes move
• Low-bit kernels:llama.cpp fork(CUDA + Metal) ·MLX fork(Apple Silicon) ·mlx-swift fork(iOS/macOS)
• Discord— join the community for support, discussion, and updates

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#model-overviewModel Overview

ItemSpecificationBase modelDerived from Qwen3.8-27B, a 27B hybrid-attention causal language model (architecture unchanged)Parameters27.36B total — 24.35B language backbone (64 blocks) + 2.54B embedding/LM head + 0.46B vision tower (27 blocks)ArchitectureHybrid attention (~75% linear / ~25% full attention), SwiGLU MLP, RoPE, RMSNormContext length262K tokens (inherited from the base model; kept practical on-device by the predominantly linear-attention backbone)Weight formatTernary g128: {−1, 0, +1} weights with FP16 group-wise scaling, packed asPTQ1_0(dense trits) orPQ2_0(2-bit slots)Weight basisBlockwise Hadamard rotation (block 1024, fixed ±1 signs) folded into the stored weights; the matching transform is applied to activations at runtimeLow-bit coverageEmbeddings, attention projections, MLP projections, LM headVision toweroptional ~0.63 GB mmproj pack (Q8_0), loaded only for image inputDeployed size5.95 GB(PTQ1_0) or7.21 GB(PQ2_0); 5.8 GB ideal at 1.72 bits/weight — see belowBackendsllama.cpp (CUDA, Metal, CPU)LicenseApache 2.0

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#weight-representation-ternary-g128Weight Representation: Ternary g128

Each weight takes a value from {−1, 0, +1}, with one shared FP16 scale factor for every group of 128 weights. A ternary value carries log₂3 ≈ 1.585 bits of information, so the effective storage cost of the format is**~1.71 bits/weight**(ternary code + 16-bit scale amortized over 128 weights); counting the small set of tensors held above the ternary representation brings the model as a whole to1.72 bits/weight— an idealized ~9.3x reduction vs FP16.

The weights are stored in arotated basis: each matrix is transformed blockwise by an orthogonal Hadamard rotation before the ternary assignment, and the runtime applies the matching transform to activations. The rotation is folded into the stored weights offline, so it costs no extra bits and no extra weight traffic; the packed model declares its rotation as metadata, so a runtime either applies the matching transform or refuses to load the file.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#memory-requirementMemory Requirement

FormatTrue bits/weightSizeReductionFP16 (baseline)16.0~54 GB1.0xTernary g128 (ideal)1.725.8 GB~9.3xGGUF PTQ1_0(dense trits)1.755.95 GB~9.0x****GGUF PQ2_0(2-bit slots)2.137.21 GB~7.5x Practical deployment needs packing formats that efficient kernels can consume, and this repo ships two:PTQ1_0packs trits densely and lands essentially on the information-theoretic target, whilePQ2_0stores each trit in a 2-bit slot, trading footprint for cheaper unpacking. Neither is uniformly faster — see the throughput table below for where each wins. These sizes describe the language model alone, the only component that must stay resident for text inference; 26.2M parameters (**0.0976%**of the language model — the recurrent state path of the linear-attention layers, plus the normalization weights) remain in higher precision and are counted in the 1.72 figure.

Unlike conventional low-bit builds — whose advertised labels understate their true average bit-width (a widely-used “2-bit” build of Qwen3.8-27B is really 2.8 bits/weight at 9.4 GB) — the Bonsai representation carries a bit-width that matches its name.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#shipped-componentsShipped Components

The vision tower ships alongside the language model as an optional component (on-disk sizes):

ComponentPackSizeResidencyLanguage modelternary g128 (PTQ1_0)5.95 GBresidentLanguage modelternary g128 (PQ2_0)7.21 GBresidentVision towermmproj (Q8_0)0.63 GBoptional — multimodal input onlyVision towermmproj BF16 (reference)0.93 GBoptional The Q8_0 file carries the vision tower in an 8-bit container. It is usually offloaded, loaded only when an image actually arrives, so text-only serving never pays for it.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#best-practicesBest Practices

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#generation-parametersGeneration Parameters

We recommend using the following sets of sampling parameters for generation:

• Thinking Mode:temperature=1\.0,top\_p=0\.95,top\_k=20,min\_p=0\.0,presence\_penalty=0\.0,repetition\_penalty=1\.0 - Instruct (or non-thinking) mode:temperature=0\.7,top\_p=0\.80,top\_k=20,min\_p=0\.0,presence\_penalty=1\.5,repetition\_penalty=1\.0

These match the base model’s owngeneration\_config\.jsonand are the values carried in the GGUF metadata (general\.sampling\.\*), so a client that reads model defaults will use them without being told. They are also the settings used for the reported benchmark results (thinking mode).

The model usesxhighreasoning effort by default; usemediumfor shorter responses and a balance of speed and accuracy.lowreasoning effort is not supported and when selected the model will behave close toxhigh.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#system-promptSystem Prompt

You can use a simple system prompt such as:

You are a helpful assistant

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#choosing-a-packingChoosing a Packing

PQ2_0is the faster decode on H100, A100, and the Blackwell cards, is faster at prompt processing everywhere, and is the pack measured on Apple Silicon.PTQ1_0is the faster decode on the Ada-generation cards and the L4, and is the pick wherever memory is tightest. See the throughput table below.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#quickstartQuickstart

**PrismML-Eng/Bonsai-demois the source of truth for running these models.**It carries the tested setup for every backend, pins a known-good binary release, and is kept current as the kernels move. Where anything here disagrees with it, it is right.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#these-files-need-our-llamacpp-buildThese files need our llama.cpp build

The ternary hybrid-attention kernels live in thePrismML-Eng/llama.cppfork.**Stock llama.cpp will not run these files.**It rejectsPQ2\_0andPTQ1\_0as unknown types, and it loadsQ2\_0without any warning and produces garbage, because it has no Hadamard activation runtime. Use a binary from the fork.

# prebuilt, pick the archive for your platform
# https://github.com/PrismML-Eng/llama.cpp/releases/latest
tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1

# or build it
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j    # drop -DGGML_CUDA=ON on macOS, Metal is default

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .

./bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -c 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 \
    -p "Explain quantum computing in simple terms." -n 256

The binary is\./bin/llama\-clifrom an extracted release archive, or\./build/bin/llama\-cliif you built the fork yourself.\-ngl 99offloads every layer,0is CPU-only;\-csets the context, up to 262144.

This is a reasoning model and it thinks by default. For the server, tool calling, reasoning budgets, image input with themmprojfile, and speculative decoding, followBonsai-demo, which ships run scripts that pick the right flags for your hardware.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#cross-platform-throughputCross-Platform Throughput

tg128is token-generation throughput over 128 generated tokens (the memory-bandwidth-bound, interactive phase);pp512is prompt-processing throughput over 512 input tokens (the compute-bound phase). Both in tokens/s, measured withllama\-benchon these GGUF packs (custom low-bit kernels), at batch size 1 and depth 0 with no vision tower. NVIDIA energy is board power including HBM/GDDR.

PlatformPQ2_0 TG128PQ2_0 PP512PQ2_0 J/tokPTQ1_0 TG128PTQ1_0 PP512PTQ1_0 J/tokRTX 5090 (32 GB)129.938931.95120.518052.15RTX PRO 6000 Blackwell124.840202.49117.919722.77H100 SXM (80 GB)113.928302.6986.912373.18RTX 6000 Ada (48 GB)82.824312.5190.416572.49RTX 4090 (24 GB)81.231242.9991.116452.58L40S (48 GB)74.428683.2481.815432.82A100 SXM (80 GB)73.913283.4354.77064.28L4 (24 GB, 72 W)29.87772.4232.14672.25Laptop (Apple M5 Pro, Metal)28.1387———— On the laptop the FP16 baseline (~54 GB) does not fit at all — the meaningful statement is not a speedup ratio but that a 27B model runs interactively on an everyday laptop. The measured decode streams ~204 GB/s of weights on the M5 Pro, confirming the memory-bandwidth-dominated profile that the low-bit representation is built to exploit. The M5 Pro figure is measured on a quiet machine; this laptop swings ~4% with background load.

The two packings are a genuine trade rather than a strict ordering. PTQ1_0 moves 17% less weight data per step, but unpacking dense trits costs arithmetic, so it wins on the Ada-generation parts and the L4 — where memory is the binding constraint — and loses on H100, A100, and the Blackwell cards, where batch-1 decode is limited by instruction throughput and launch overhead instead. Prompt processing, being compute-bound, favors PQ2_0 everywhere.

The Apple row carries no per-token energy figure because the two platforms’ instrumentation does not enclose the same components:nvidia\-smiincludes the card’s HBM/GDDR, while Apple’spowermetricsreports CPU, GPU, and ANE with no DRAM rail. What the measurement does support is absolute draw: the M5 Pro decodes at27.5 Won the GPU rail and 34.1 W across CPU and GPU, against 300–455 W of board power for the NVIDIA cards above.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#additional-apple-platformsAdditional Apple Platforms

Measured on the earlier pre-rotation build and reported pending re-measurement on the current stack (llama.cpp Metal backend):

PlatformFootprintTG128 (tok/s)PP512 (tok/s)Laptop (Apple M5 Max, Metal)7.2 GB47.0765Laptop (Apple M5 Pro, Metal)7.2 GB28.7393Laptop (Apple M4 Pro, Metal)7.2 GB18.0125 On the wider M5 Max the model reaches ~47 tok/s; on the M4 Pro, prefill (~125 tok/s) rather than decode is the practical limit for very long prompts.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#benchmarksBenchmarks

Evaluated with EvalScope + vLLM on NVIDIA H100 under identical infrastructure, decoding, and scoring, inthinking mode— where the model’s full reasoning is exercised and the sub-4-bit collapse of conventional methods is most visible. 14 benchmarks across six skill categories. Bit-widths are true averages; “vs FP16” is relative to the Qwen3.8-27B FP16 reference.

VariantTrue bpwFootprintThinking avgvs FP16Qwen3.8-27B FP1616.054 GB86.32100%Qwen3.8-27B UD-Q4_K_XL (“4-bit”)5.217.6 GB85.1898.7%Qwen3.8-27B IQ2_XXS (“2-bit”)2.89.4 GB72.5984.1%Bonsai 2 27B1.725.9 GB84.7898.2% At 5.9 GB, Bonsai 2 27B outscores the sub-4-bit conventional build by more than twelve points at less than two-thirds of its size, and comes within 0.4 points of UD-Q4_K_XL at a third of its footprint.

The aggregate gap also understateshowthe conventional builds fail: their degradation is selective, concentrated on the benchmarks that demand sustained chains of reasoning. IQ2_XXS falls to 57.5 on AIME26 and 56.4 on LiveCodeBench while still scoring 88.93 on MMLU-Redux — which is why casual testing misses the collapse. Bonsai 2 holds exactly these benchmarks, scoring 95.83 and 90.07. The previous Bonsai 27B report showed the same pattern on a second model family, Gemma-4-31B, so the collapse is a property of the methods rather than of one base model.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#by-skill-categoryBy Skill Category

CategoryBenchmarksFP16Bonsai 2 27BKnowledge & reasoningMMLU-Redux, MuSR85.5579.86MathGSM8K, MATH-500, AIME25, AIME2697.0696.57CodingHumanEval+, MBPP+, LiveCodeBench89.0789.42Instruction followingIFEval, IFBench81.2582.66Agentic / tool callingBFCL v376.7474.92VisionMMMU-Pro, OCR Bench v271.3666.19Overall (14)86.3284.78 The reasoning backbone comes through intact: math falls only from 97.06 to 96.57, coding is level with the baseline, and instruction following is slightly ahead of it. The remaining gap is concentrated in the most demanding categories — knowledge and reasoning, and vision.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#full-per-benchmark-resultsFull Per-Benchmark Results

Expand full per-benchmark results (thinking mode)BenchmarkFP16UD-Q4_K_XLIQ2_XXSBonsai 2 27BMMLU-Redux91.4693.3588.9389.09MuSR79.6373.0166.9970.63GSM8K97.1996.6689.9096.66MATH-50099.8099.4084.6098.80AIME2596.6792.9166.6795.00AIME2694.5893.0057.5095.83HumanEval+93.2995.7391.4695.12MBPP+83.8683.8678.8983.07LiveCodeBench90.0587.9656.4090.07IFEval91.5088.8384.0391.31IFBench (prompt-loose)71.0065.6553.7674.00BFCL v376.7475.0570.2874.92MMMU-Pro81.7381.7365.1975.49OCR Bench v260.9965.4561.7056.88Average (14)86.3285.1872.5984.78

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#intelligence-densityIntelligence Density

Intelligence density captures the ratio of a model’s capability to its deployed size:

D = -log2(1 - score/100) / size_GB

VariantSize (GB)Benchmark avgIntelligence Density (1/GB)Bonsai 2 27B5.8084.78****0.469Ternary Bonsai 27B (previous release)5.7580.980.416Qwen3.8-27B IQ2_XXS9.472.590.199Qwen3.8-27B UD-Q4_K_XL17.685.180.157Qwen3.8-27B FP165486.320.053 Bonsai 2 27B delivers over2.3xthe density of the densest conventional build (IQ2_XXS at 0.199) and nearly9xFP16 — no conventional build of Qwen3.8-27B exceeds 0.2. Each stored gigabyte is translated into far more usable intelligence. Against the previous Bonsai 27B release, density rises from 0.416 to 0.469, a 12.5% gain; that row is recomputed on these same 14 benchmarks for a like-for-like comparison.

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#use-casesUse Cases

• Laptop-local 27B agents: full 27B reasoning and tool use on a standard laptop at ~28 tok/s, with the 262K context available for long-document analysis, full-repository code work, and other tasks that depend on holding a large working set in context
• Privacy-sensitive and offline settings: on-device execution keeps prompts and data on the device by construction, and works with intermittent or no connectivity
• Single-GPU and commodity-GPU serving: 27B-class quality from a single consumer or entry-level datacenter GPU — ~130 tok/s on an RTX 5090, ~30 tok/s on a 72 W L4 — with headroom for larger batches, longer contexts, or co-resident models
• Quality-first low-bit deployment: 98.2% of the full-precision model’s benchmark average at roughly a ninth of its size

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#limitationsLimitations

• The quality–footprint trade-off: the ternary model retains 98.2% of the full-precision average, and the gap is modest and predictable — the reasoning core (math, coding) stays within a few points of baseline, with the difference concentrated in the most demanding categories
• Native low-bit kernels: the dense PTQ1_0 packing (1.75 bits/weight, 5.95 GB) now exists, but unpacking trits costs arithmetic — it is faster on Ada-class and smaller accelerators and slower on Ampere, Hopper, and Blackwell, where batch-1 decode is not bandwidth-starved; returning the footprint advantage as latency on every target is an active engineering target

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#citationCitation

If you use Bonsai 2 27B, please cite:

@techreport{bonsai2_27b,
    title   = {Bonsai 2 27B: A 27B Ternary Reasoning Model},
    author  = {Prism ML},
    year    = {2026},
    month   = {September},
    url     = {https://prismml.com}
}

https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#contactContact

For questions, feedback, or collaboration inquiries:[email protected]