prism-ml/Bonsai-27B-gguf · Hugging Face

Source: https://huggingface.co/prism-ml/Bonsai-27B-gguf

Prism ML Website|Whitepaper|Demo & Examples|Discord

Full 27B-class reasoning in binary transformer weights, for llama.cpp (CUDA, Metal, CPU)

~14.2xsmaller than FP16 |~90%of FP16 intelligence retained |~44 tok/son an Apple M5 Pro laptop

https://huggingface.co/prism-ml/Bonsai-27B-gguf#highlightsHighlights

• ~3.9 GBdeployed footprint (down from ~54 GB FP16) — a 27B model on everyday laptops and single GPUs
• Retains thinking, reasoning, and agentic behaviordeep in the sub-4-bit regime, where conventional low-bit representations collapse — 76.11 average across 15 thinking-mode benchmarks (89.5% of FP16), including math at 91.66 and coding at 81.88
• End-to-end binary language weightsacross embeddings, attention projections, MLP projections, and LM head, at atrue1.125 bits per weight — no high-precision escape hatches behind a low-bit label; the vision tower ships in compact 4-bit HQQ
• 262K-token contexton-device, kept practical by the Qwen3.6-27B hybrid-attention backbone (~75% linear attention) and 4-bit KV-cache quantization
• GGUF Q1_0_g128format with custom 1-bit hybrid-attention kernels for llama.cpp (CUDA, Metal) — packed weights are consumed directly, never expanded back to FP16
• Ships with a DSpark speculative-decoding drafter layertrained against the Bonsai 27B target — a lossless1.37xdecode speedup on the CUDA serving path
• MLX companion: also available asBonsai-27B-mlx-1bitfor native Apple Silicon inference, including iPhone (~11 tok/s on iPhone 17 Pro Max via MLX Swift)
• Ternary companion: the quality-oriented operating point (~7.2 GB, 95% of FP16) is also published in GGUF asTernary-Bonsai-27B-gguf

https://huggingface.co/prism-ml/Bonsai-27B-gguf#resourcesResources

• Whitepaper— full methodology, benchmarks, and measurement notes
• Demo & examples— serving, benchmarking, and integrating Bonsai
• Low-bit kernels:llama.cpp fork(CUDA + Metal) ·MLX fork(Apple Silicon) ·mlx-swift fork(iOS/macOS)
• Discord— join the community for support, discussion, and updates

https://huggingface.co/prism-ml/Bonsai-27B-gguf#model-overviewModel Overview

ItemSpecificationBase modelDerived from Qwen3.6-27B, a 27B hybrid-attention causal language model (architecture unchanged)Parameters~27.3B binary language weights (~24.8B backbone across 64 blocks + ~2.5B embedding/LM head) + ~0.46B vision tower (27 blocks)ArchitectureHybrid attention (~75% linear / ~25% full attention), SwiGLU MLP, RoPE, RMSNormContext length262K tokens (full-context capable on-device, enabled by the predominantly linear-attention backbone)KV cacheNear-lossless 4-bit KV quantization; the hybrid backbone grows a full-attention cache on only 16 of 64 layers (~4.3 GB at the full 262K window)Weight formatGGUF Q1_0_g128: {−1, +1} weights with FP16 group-wise scalingLow-bit coverageEmbeddings, attention projections, MLP projections, LM headVision towerHQQ 4-bit; optional ~0.63 GB mmproj pack (Q8_0 container), loaded only for image inputDeployed size**~3.9 GB**(~14.2x smaller than FP16)AccelerationDSpark speculative-decoding drafter layer providedBackendsllama.cpp (CUDA, Metal, CPU)LicenseApache 2.0

https://huggingface.co/prism-ml/Bonsai-27B-gguf#weight-representation-q1_0_g128Weight Representation: Q1_0_g128

Each weight is a single sign bit:0maps to−scale,1maps to\+scale. Every group of 128 weights shares one FP16 scale factor.

Effective bits per weight:1.125(1 sign bit + 16-bit scale amortized over 128 weights) — an idealized ~14.2x reduction vs FP16. This is the most aggressive operating point in the Bonsai 27B family: it minimizes both stored footprint and the weight traffic incurred at every decoding step. The GGUF Q1_0_g128 pack is the model’s native layout — ideal and deployed sizes match.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#memory-requirementMemory Requirement

FormatTrue bits/weightSizeReductionFP16 (baseline)16.0~54 GB1.0xGGUF Q1_0_g1281.125~3.9 GB****~14.2x The deployed figure describes the language model alone — the only component that must stay resident for text inference; a negligible tail of normalization and scale parameters remains in higher precision.

Unlike conventional low-bit builds — whose advertised labels understate their true average bit-width (a widely-used “2-bit” build of Qwen3.6-27B is really 2.8 bits/weight at 9.4 GB) — the Bonsai representation carries a bit-width that matches its name.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#shipped-componentsShipped Components

Two optional components ship alongside the language model (on-disk sizes):

ComponentPackSizeResidencyLanguage model1-bit g128 (Q1_0)~3.9 GBresidentDSpark drafterQ4_1 (default)1.79 GBoptional — speculative decodingDSpark drafterbf16 (reference)7.29 GBoptionalVision towermmproj HQQ 4-bit (Q8_0 container)0.63 GBoptional — multimodal input onlyVision towermmproj BF16 (reference)0.93 GBoptional The vision tower is usually offloaded: it sits outside the accelerator’s resident budget and is loaded only when an image actually arrives, so text-only serving never pays for it.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#peak-memory-at-contextPeak Memory at Context

What a device must actually accommodate ispeakmemory — weights plus KV cache plus activations and runtime buffers (~1.3 GB across backends). Measured, language model only, no KV-cache compression (sizes in decimal GB; the Q4_K_XL row is derived from its weight footprint plus the same measured cache-and-overhead build-up, all other rows directly measured):

BuildWeights4K ctx10K ctx100K ctx1-bit Bonsai (llama.cpp Q1_0)3.795.25.611.6Qwen3.6-27B “4-bit” (Q4_K_XL)17.619.219.625.627B 16-bit (GGUF bf16)51.2552.653.359.3 The 1-bit build holds a100K-token context at 11.6 GB without any KV-cache compression— a budget that fits mainstream laptops outright; the conventional Q4_K_XL build needs ~25.6 GB before the first long document is loaded. These peaks are the conservative case, with the cache left at FP16. Enabling the 4-bit KV cache shrinks the context-dependent term ~4x: the 100K peak drops to ~6.8 GB, and the full 262K window fits in ~9.4 GB peak.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#best-practicesBest Practices

https://huggingface.co/prism-ml/Bonsai-27B-gguf#generation-parametersGeneration Parameters

ParameterSuggestedTemperature0.7Top-p0.95Top-k20 These are the settings used for all reported benchmark results (thinking mode).

https://huggingface.co/prism-ml/Bonsai-27B-gguf#system-promptSystem Prompt

You can use a simple system prompt such as:

You are a helpful assistant

https://huggingface.co/prism-ml/Bonsai-27B-gguf#quickstartQuickstart

https://huggingface.co/prism-ml/Bonsai-27B-gguf#llamacpp-cudallama.cpp (CUDA)

# Clone the PrismML fork of llama.cpp (includes the Q1_0_g128 hybrid-attention kernels)
git clone https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp

# Build with CUDA support
cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# Download the 1-bit GGUF weights
hf download prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf --local-dir .

# Run inference
./build/bin/llama-cli \
    -m Bonsai-27B-Q1_0.gguf \
    -p "Explain quantum computing in simple terms." \
    -n 256 \
    --temp 0.7 --top-p 0.95 --top-k 20 \
    -ngl 99

https://huggingface.co/prism-ml/Bonsai-27B-gguf#llamacpp-metal–macosllama.cpp (Metal / macOS)

# Build with Metal support (default on macOS)
cmake -B build && cmake --build build -j

# Run inference
./build/bin/llama-cli \
    -m Bonsai-27B-Q1_0.gguf \
    -p "Explain quantum computing in simple terms." \
    -n 256 \
    --temp 0.7 --top-p 0.95 --top-k 20 \
    -ngl 99

https://huggingface.co/prism-ml/Bonsai-27B-gguf#llamacpp-serverllama.cpp Server

./build/bin/llama-server \
    -m Bonsai-27B-Q1_0.gguf \
    --host 0.0.0.0 --port 8080 -ngl 99

Open the web UI athttp://127.0.0.1:8080, or see ourllama.cpp forkfor more examples.

**Deploying to a phone?**iPhone deployment uses the MLX Swift runtime — seeBonsai-27B-mlx-1bit(~11 tok/s on iPhone 17 Pro Max).

https://huggingface.co/prism-ml/Bonsai-27B-gguf#cross-platform-throughputCross-Platform Throughput

tg128is token-generation throughput over 128 generated tokens (the memory-bandwidth-bound, interactive phase);pp512is prompt-processing throughput over 512 input tokens (the compute-bound phase). Both in tokens/s, measured withllama\-benchon this GGUF pack (custom low-bit kernels).

PlatformFootprintTG128 (tok/s)PP512 (tok/s)Laptop (Apple M5 Max, Metal)3.9 GB66.4874Laptop (Apple M5 Pro, Metal)3.9 GB44.2421Laptop (Apple M4 Pro, Metal)3.9 GB26.0133Single GPU (H100, CUDA)3.9 GB104.82755 On the edge platforms the FP16 baseline (~54 GB) and even conventional “4-bit” builds (17.6 GB) do not fit at all — the meaningful statement is not a speedup ratio but that a 27B model runs on the device in the first place. The H100 row is the exception that proves the rule: at batch size 1 a datacenter GPU is limited by kernel-launch and synchronization latency rather than weight bandwidth, so the binary and ternary variants converge there (104.8 vs 98 tok/s) despite their ~1.9x difference in bytes per step.

Decode energy on the M5 Pro measures0.275 mWh/token(with the DSpark drafter enabled) — an order of magnitude more energy-efficient per token than datacenter GPUs (0.63–1.32 mWh/token across the GPU classes). Local inference is not just private and low-latency but cheap in energy.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#speculative-decoding-dsparkSpeculative Decoding: DSpark

1-bit Bonsai 27B ships with aDSparkdrafter layer trained against the low-bit target — a semi-autoregressive drafter with confidence-scheduled verification. Speculative decoding is lossless: verification preserves the target distribution exactly, so accepted tokens are indistinguishable from ordinary generation.

The drafter is a compactsix-layer block-parallel transformerconditioned on hidden states tapped from five evenly spaced layers of the target; its drafter-unique weights add roughly0.5 GB at serving precision(embeddings and output head are shared with the resident target). It follows the DSpark recipe with a diffusion-flavored block-denoising objective, survival-probability-weighted distillation, per-source-normalized hidden-state taps, and a draft block size chosen from a measured verify-cost model of the serving stack. The drafter ships 4-bit quantized — the ~1.79 GB Q4_1 pack is the default; it drafts faster than the bf16 reference at essentially unchanged draft quality, and because verification preserves the target distribution exactly, drafter precision affects only speed, never output quality.

On the CUDA serving path the drafter is a measured net win — an accepted length of τ ≈ 3.6 at draft depth k = 4 turns into a1.37xend-to-end decode speedup on H100 (104.8 → 143.8 tok/s). On Apple Silicon the batch-1 verification pass does not yet amortize, so the drafter layer is not enabled by default on-device.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#benchmarksBenchmarks

Evaluated with EvalScope + vLLM on NVIDIA H100 under identical infrastructure, decoding, and scoring, inthinking mode— where the model’s full reasoning is exercised and the sub-4-bit collapse of conventional methods is most visible. 15 benchmarks across six skill categories. For cross-family context the table also includes Gemma-4-31B, a model of the same capability tier, with its conventional low-bit builds — the collapse below 4 bits is a property of the methods, not of one base model. Bit-widths are true averages; “vs FP16” is relative to the Qwen3.6-27B FP16 reference.

VariantTrue bpwFootprintThinking avgvs FP16Qwen3.6-27B FP1616.054 GB85.07100%Qwen3.6-27B Q4_K_XL (“4-bit”)5.217.6 GB84.9999.9%Qwen3.6-27B IQ2_XXS (“2-bit”)2.89.4 GB72.7385.5%Gemma-4-31B FP1616.061.5 GB84.5899.4%Gemma-4-31B QAT (“4-bit”)6.023.3 GB83.4198.0%Gemma-4-31B Q2_K_XL (“2-bit”)3.011.8 GB73.3186.2%Ternary Bonsai 27B1.715.9 GB80.4994.6%1-bit Bonsai 27B1.1253.9 GB76.1189.5% The aggregate gap also understateshowthe conventional builds fail: their degradation is selective, concentrated on the benchmarks that demand sustained chains of reasoning. IQ2_XXS falls to 57.5 on AIME26 and 56.4 on LiveCodeBench while still scoring 88.93 on MMLU-Redux — which is why casual testing misses the collapse. 1-bit Bonsai holds exactly these benchmarks, keeping AIME above 87 at a third of IQ2_XXS’s footprint.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#by-skill-categoryBy Skill Category

CategoryBenchmarksFP161-bit 27BKnowledge & reasoningMMLU-Redux, MuSR83.1573.39MathGSM8K, MATH-500, AIME25, AIME2695.3391.66CodingHumanEval+, MBPP+, LiveCodeBench88.7481.88Instruction followingIFEval, IFBench78.4765.74Agentic / tool callingBFCL v3, τ²-Bench80.0066.03VisionMMMU-Pro, OCR Bench v272.6159.57Overall (15)85.0776.11 The reasoning backbone comes through intact: math stays at 91.66 — within four points of full precision — and coding at 81.88, the behaviors that conventional sub-4-bit representations lose first. The 1-bit model trades part of the ternary model’s margin on the most demanding categories for the smallest footprint in the family.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#full-per-benchmark-resultsFull Per-Benchmark Results

Expand full per-benchmark results (thinking mode)BenchmarkFP161-bit 27BMMLU-Redux93.4282.75MuSR72.8864.02GSM8K95.3092.80MATH-50099.4098.00AIME2593.2988.75AIME2693.3387.08HumanEval+95.1289.63MBPP+83.3379.60LiveCodeBench87.7776.40IFEval88.9179.11IFBench (prompt-loose)68.0352.36BFCL v377.1070.72τ²-Bench82.9061.34MMMU-Pro79.9460.48OCR Bench v265.2858.65Average (15)85.0776.11

https://huggingface.co/prism-ml/Bonsai-27B-gguf#intelligence-densityIntelligence Density

Intelligence density captures the ratio of a model’s capability to its deployed size:

D = -log2(1 - score/100) / size_GB

VariantSize (GB)Benchmark avgIntelligence Density (1/GB)1-bit Bonsai 27B****3.976.110.530Ternary Bonsai 27B5.980.490.400Qwen3.6-27B IQ2_XXS9.472.730.199Gemma-4-31B Q2_K_XL11.873.310.162Qwen3.6-27B Q4_K_XL17.684.990.155Gemma-4-31B QAT23.383.410.111Qwen3.6-27B FP165485.070.051Gemma-4-31B FP1661.584.580.044 1-bit Bonsai 27B delivers roughly2.7xthe density of the densest conventional build (IQ2_XXS at 0.199) and over10xFP16 — no conventional build of Qwen3.6-27B or Gemma-4-31B exceeds 0.2. Each stored gigabyte is translated into far more usable intelligence.

https://huggingface.co/prism-ml/Bonsai-27B-gguf#use-casesUse Cases

• Laptop-local 27B agents: full 27B reasoning and tool use on any standard laptop at ~26–66 tok/s (M4 Pro through M5 Max), with the 262K context available for long-document analysis and full-repository code work
• Privacy-sensitive and offline settings: on-device execution keeps prompts and data on the device by construction, and works with intermittent or no connectivity
• Single-GPU and commodity-GPU serving: 27B-class quality from a single consumer or entry-level datacenter GPU, with headroom for larger batches, longer contexts, or co-resident models — combined with the KV-cache quantization, high-throughput serving and long-context document analysis become practical on a single 24 GB GPU
• Phone deployment via MLX: the same weights ship asBonsai-27B-mlx-1bit— the first 27B-class model to run on a phone

https://huggingface.co/prism-ml/Bonsai-27B-gguf#limitationsLimitations

• The quality–footprint trade-off: the binary model retains 89.5% of the full-precision average, and the gap is modest and predictable — the reasoning core (math, coding) stays within a few points of baseline, with the difference concentrated in the most demanding categories; if quality is the priority, consider the ternary GGUF build (94.6%)
• Agentic coding(long-horizon, multi-file, run-test-and-repair workflows) is not yet a strong target of this release; a Bonsai 27B variant tuned for agentic coding is next on the roadmap
• KV compression headroom: this release standardizes on a 4-bit KV cache; Bonsai’s tolerance to KV-cache error grows with context length, and early results show the key cache can be pushed toward the sub-2-bit regime — a path to still longer contexts within a fixed device-memory budget

https://huggingface.co/prism-ml/Bonsai-27B-gguf#citationCitation

If you use 1-bit Bonsai 27B, please cite:

@techreport{bonsai27b,
    title   = {Bonsai 27B: Full 27B-Class Reasoning in Binary and Ternary
               Transformer Weights --- on Laptops and Phones},
    author  = {Prism ML},
    year    = {2026},
    month   = {July},
    url     = {https://prismml.com}
}

https://huggingface.co/prism-ml/Bonsai-27B-gguf#contactContact

For questions, feedback, or collaboration inquiries:[email protected]