Bonsai-27B & Ternary-Bonsai-27B - Updates (on PRs)

Below Upstream Status sections are from https://github.com/PrismML-Eng/Bonsai-demo Upstream Status for Binary Q1_0 is supported out of the box in upstream llama.cpp across many backends: CPU (generic, NEON, and optimized x86), Metal, CUDA, and Vulkan. Runtime Status llama.cpp (CPU, Metal, CUDA, Vulkan) ✅ Merged upstream, works out of the box MLX (1-bit) ⏳ Pending upstream: mlx#3161; until it merges, use PrismML-Eng/mlx (branch prism, built automatically by setup.sh) Upstream Status for Ternary Ternary support is in the middle of migrating into mainline llama.cpp: backends are landing one by one, so today it is a mix of mainline and our fork. The practical consequence first: we currently ship three ternary GGUF variants, and each one needs to run in the right place. File Format Runs on *-Q2_0.gguf Group size 128. The format this demo uses, compatible with our fork. Once the llama.cpp migration completes, these files will be deprecated and replaced by the PQ2_0 ggufs This demo / the fork binaries. Will not load on mainline (same type id, different block size) *-Q2_0_g64.gguf Group size 64 (2.25 bpw). The official llama.cpp format; these will be renamed to plain Q2_0, replacing the current ones Mainline llama.cpp (CPU and Metal so far) *-PQ2_0.gguf Not supported yet. Planned as the fork format going forward: the same format as the current group-128 Q2_0, just under its own ggml type id so it can coexist with the upstream Q2_0 Nothing yet (fork support planned) Backend-by-backend migration status: Backend Status Where CPU (ARM NEON + generic scalar) ✅ Merged in mainline llama.cpp ggml-org/llama.cpp#24448 Metal ✅ Merged in mainline llama.cpp ggml-org/llama.cpp#25419 Vulkan 🔄 In progress upstream (separate PR, not ours) ggml-org/llama.cpp#25430 CUDA 🔄 In review upstream ggml-org/llama.cpp#25707 x86 (AVX-512-VNNI) ⏳ Pending TBD Above Vulkan PR got approved✅ & waiting to be merged by soon/EOD. Other Open PRs(related to Bonsai) on llama.cpp to be merged: cuda: extract Q1_0 elements via __byte_perm #25628 ( t/s +5-10% & pp +1-2.5% ) ggml-cpu: ARM Repack kernels for Q1_0 #23492 (Check t/s benchmarks inside) ggml-cpu: Optimized Arm NEON cpu q1_0 dot (with plain/DP/I8MM) #23358 (Check t/s benchmarks inside) Hope to see more PRs on optimizations of both Bonsai & Ternary-Bonsai models. Yesterday I saw some folks(from other Bonsai threads) mentioned that these models are not doing stuff like Agentic coding well. I think it's too much expectation from 1-bit version models. But Model creators mentioned notes on their model cards. Sharing below. Bonsai-27B Limitations The quality–footprint trade-off: the binary model retains 89.5% of the full-precision average, and the gap is modest and predictable — the reasoning core (math, coding) stays within a few points of baseline, with the difference concentrated in the most demanding categories; if quality is the priority, consider the ternary GGUF build (94.6%) Agentic coding (long-horizon, multi-file, run-test-and-repair workflows) is not yet a strong target of this release; a Bonsai 27B variant tuned for agentic coding is next on the roadmap KV compression headroom: this release standardizes on a 4-bit KV cache; Bonsai's tolerance to KV-cache error grows with context length, and early results show the key cache can be pushed toward the sub-2-bit regime — a path to still longer contexts within a fixed device-memory budget Ternary-Bonsai-27B Limitations The quality–footprint trade-off: the ternary model retains 94.6% of the full-precision average, and the gap is modest and predictable — the reasoning core (math, coding) stays within a few points of baseline, with the difference concentrated in the most demanding categories Does not fit a phone: at ~7.2 GB the ternary build exceeds the ~6 GB per-app iOS memory budget; use the 1-bit companion via MLX Swift for phone deployment Served in 2-bit slots today: the deployed footprint (~7.2 GB) sits above the representation's ~5.9 GB native target; native ternary kernels are an active engineering target and would return the remaining bandwidth and footprint advantage directly as latency and energy improvements Agentic coding (long-horizon, multi-file, run-test-and-repair workflows) is not yet a strong target of this release; a Bonsai 27B variant tuned for agentic coding is next on the roadmap KV compression headroom: this release standardizes on a 4-bit KV cache; early results show the key cache can be pushed toward the sub-2-bit regime — a path to still longer contexts within a fixed device-memory budget Hope we get best extreme optimized models(without above limitations) from them next time onwards. I just llama-benched Bonsai-27B without any params. Got pp - 600 t/s & tg - 30 t/s on my Laptop GPU 4060 (8GB VRAM). I'm not expecting any miracles from this model, glad to have this to get it running with my tiny VRAM. Gonna use this one for chatting & writing. Waiting for follow-up optimizations by Bonsai team & llama.cpp folks to get best performance. What t/s are you getting? Anyone tried this for coding, writing & other stuff? Also anyone got it on phone? I can't load this on my 8GB RAM phone, would be too slow. Please share your feedback on performance of this model.