@AgentSparko: I tested the PrismML Bonsai 27B on the DGX Spark but I think I messed something up with llama.cpp build because the spe…

I tested the PrismML Bonsai 27B on the DGX Spark but I think I messed something up with llama.cpp build because the speed test results are nothing like I would of expected.

Ternary-Bonsai 27B + DSpark + Q4_0 KV cache results

1. Short prompt

• Context: 73 tokens
• Prefill: 29.3 tok/s
• Decode (80 tokens): 20.61 tok/s
• Draft acceptance: 60/76 (79%)

2. 6K context + 1K decode

• Context: 7,153 tokens
• Prefill: 434.9 tok/s
• Decode (1024 tokens): 10.98 tok/s
• Draft acceptance: 560/1846 (30.3%)

These are the numbers with Q4 KV cache enabled (-ctk q4_0 -ctv q4_0).

BTW: This is the first time I see the GPU at 40-50% use during inference, usually is at 96% and GPU power draw 40W-85W (just Qwen 35B and 27B hang around 40W)

I usually use AEON-7 Ultimate vLLM docker container with Qwen 27B NVFP4 and DFlash draft and I get 40-50 tok/s single stream decode speed and I would expected the Ternary quant with DSpark to be much faster.

Please check out the attached screenshots and if you tested it on a DGX Spark and used other flags for building llama.cpp let me know.

I personally do not like llama.cpp and use just vLLM so last time I used it was in March and I don’t recall which flags worked best for compiling it.

PrismML (@PrismML): Today, we’re announcing Bonsai 27B: the first 27B-class model to run on a phone.

Bonsai 27B is the new multimodal flagship of the Bonsai family. Based on Qwen3.6 27B, it brings a new capability tier to local AI: multi-step reasoning, structured tool use, long-context workflows,