ik_llama.cpp now fully supports PrismML's Ternary Bonsai 2 with native CUDA offload, lossless weight repacking, and fast CPU prompt ingestion via -rtr 1, eliminating prior sign tensor bottlenecks. The post provides a setup guide covering PQ2_0 conversion and recommended command-line flags.
Hey folks, Quick heads up for anyone playing around with PrismML's Ternary Bonsai 2! Until very recently, running it outside of Prism's own fork was a bit of a pain. Upstream llama.cpp uses a block size of QK=256 (TQ2_0), while Prism uses QK=128 (PQ2_0), and trying to offload layers to CUDA on other forks would bottleneck or break due to Hadamard sign tensor handling. With the latest commits landed in ik_llama.cpp, we finally have complete, native support across both CUDA and CPU. Here is how to set it up cleanly. Convert upstream weights losslessly (PR #2694) If you already have a native PQ2_0 GGUF downloaded from Prism, you can skip this. If you downloaded upstream TQ2_0 or TQ1_0 files, ik_llama.cpp now ships with a script that losslessly repacks them to PQ2_0 in just a few seconds (exact trits preserved, no requantization needed): python3 scripts/repack_ternary_to_prism.py Bonsai-2-27B-TQ2_0.gguf Bonsai-2-27B-PQ2_0.gguf Native CUDA Full Offload (PR #2691 & PR #2747) Native CUDA kernels for PQ2_0 are now merged. You can offload everything to your Nvidia GPU without token corruption or falling back to heavy k-quants: ./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -ngl 99 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20 CPU Inference with fast prompt ingestion (-rtr 1) If you're testing on CPU, ik_llama.cpp uses an 8-row interleaved format (PQ2_0_R8) backed by iqk_mul_mat kernels. Remember to pass -rtr 1 (runtime repack) to drastically accelerate prompt processing: ./build/bin/llama-cli -m Bonsai-2-27B-PQ2_0.gguf -rtr 1 -t 8 -p "Explain the theory of relativity in simple terms." --temp 0.7 --top-p 0.95 --top-k 20 TL;DR: No more sign tensor bottlenecks on GPU. PR #2747 brings native CUDA kernels, PR #2694 gives us instant lossless repack from upstream weights, and CPU prompt ingestion with -rtr 1 is flying. Big shoutout to Iwan Kawrakow and the contributors for getting ternary models working natively!
Release of Ternary-Bonsai-2-27B-gguf, a 27B-class language model using ternary weights for extreme compression (5.9 GB) while retaining 98.2% of FP16 intelligence, optimized for efficient inference on laptops and single GPUs.
Updates on Bonsai-27B and Ternary-Bonsai-27B models, detailing upstream merge status in llama.cpp across CPU, Metal, CUDA, Vulkan backends, and discussing model limitations and roadmap.
Ternary-Bonsai-2-27B is a 27B parameter AI model compressed into 2-bit ternary format, running on llama.cpp with CUDA and Metal support, facilitating lightweight on-device AI deployment.
Prism ML released a ternary weight 27B-class AI model optimized for on-device use on Apple laptops, retaining 98.2% of full-precision intelligence with an 8.60 GB footprint and ~47 tok/s performance.