ternary

This article tracks updates on various low-bit AI models, including Bonsai, BitCPM, and others, with performance improvements and compatibility updates for backends like llama.cpp.

Maple-Preview is a ternary 20B MoE model that runs at 120 tokens per second on an iPhone, showcasing efficient on-device inference.

User expresses disappointment with the ternary Bonsai model from Prisml, questioning its quality despite hype, and wonders if it's just typical model overhype.

Prism-ML announces that their ternary Bonsai models can now be fine-tuned, with example code and a recommendation to use a high learning rate.

A new ternary quantized version of Qwen3.6 27B, called Bonsai 27B, allows running the model on 12GB GPUs with 10x less memory and 95% of original performance, making it accessible for local deployment.

User tests ternary quantized Qwen3.6 27B on an RTX 3090, achieving 60 tk/s with two slots and 100k KV cache using 21GB VRAM, with good quality and stable tool calls.

Ternary Bonsai 27B, a large language model, is demonstrated running locally on an NVIDIA RTX 5090 GPU, requiring under 6GB of memory and enabling end-to-end agentic workflows on consumer hardware.

Prism ML releases Ternary-Bonsai-27B-mlx-2bit, a ternary-quantized 27B-parameter language model that achieves ~95% of FP16 performance while fitting in ~7.2 GB, enabling full reasoning on laptops.

BitCPM is a new open-source model from ModelBest, Tsinghua, and OpenBMB that uses ternary weights (-1,0,1) to run full-sized AI models on phones.

PrismML releases Bonsai 27B, a vision-language model with agentic tool calling and long context, along with 1-bit and ternary variants. The demo repository allows running these models locally on various hardware.