I turned Qwen3.8-27B Q2_64 + llama.cpp into a fully TypeSafe AI-compatible Jev-like system. OpenAI API still intact! World’s first Vision-enabled Jev-like model! <10 GB VRAM, 170 ms on an RTX 3090 and ~140 tok/s in chat. 76% vs. 88% Jev-1.13 Acc. on a diverse 22,000-request typed-decision benchmark

Yesterday I released Bonsai-Llama-Jev - a Jev-like Typed Decision inference system for locally operated, sovereign AI. In my new typed-decision-bench, covering more than 22,000 decisions across 175 use cases, it is currently the strongest open system I’ve tested, reaching around 76% Soft Accuracy vs. ~88% for Jev-1.13, leaving Von-1.1, Laya and even Qwen3.5-4B (best OpenJev variant) behind. The full benchmark results are available here. At the same time, Bonsai-Llama-Jev is, in my view, Pareto-optimal in terms of VRAM usage and latency: <10 GB VRAM, including KV cache and vision projector ~171 ms p50 latency vs. 716 ms p50 for Jev roughly 4× faster than Jev - while running locally Among the open systems I tested, it also achieves the lowest Calibration Error (ECE-15) - only 13% vs 8.4% Jev! This came from my new post-hoc model calibration method I discovered using my VAE Auto Research Harness. It discovered this math trick and then I immediately wanted to understand it and apply it. I propose the `calibration.json` standard for post-hoc calibration of open models that are used in Jev-like inference setups. I wrote up everything... the calibration method, experimental results, and why it works in my calibration paper. A few comparison numbers against the open models competition: +29.8 percentage points vs. Laya (it's fast but only performs well on a non-diverse benchmark, and calibration is really bad) +27.9 percentage points vs. Von-1.1 (2% better than Laya in the diverse benchmark, also very badly calibrated) +2.3 percentage points vs. OpenJev / SemIf-Qwen3.5-4B (It's a good model! Also well-calibrated! But not useful to run in parallel to a 27B model on ONE local GPU!) Now Bonsai-Llama-Jev can do everything, and well! It's only 11.6 percentage points behind Jev-1.13 in Acc.! To the best of my current knowledge, this is also the first open Jev-like setup with multimodal Typed Decisions: you can make structured decisions not only from text/JSON, but directly from images + text/JSON. For this, I extended the System One API standard while preserving full compatibility (see my repo, and the deep links; I tried not to re-invent the wheel. "state" field becomes a JSON here and it adapts the llama.cpp/OpenAI standards for base64 and url-based image transmission). The part I find most interesting - and probably the most relevant one for many people here - is the unified architecture of this llama.cpp fork: Locally, we usually cannot just run many different AI models in parallel, right? So I took a fork of PrismLM’s llama.cpp fork, adapted it as the inference engine, and implemented TypeSafe AI’s OpenAPI-specified System One API directly inside it. Concretely, I added just the endpoint and the inference math behind that: POST /v1/systemone alongside the existing OpenAI-compatible and llama.cpp native endpoints. The result is now E2E-tested with all official TypeSafe Python and JavaScript SDKs and it just works! The typed-decisions-bench is btw. the best test - it runs 22k requests against the endpoint with massive parallelism and it never broke down! Latency was measured by me including HTTP network latency. So instead of operating several specialized models locally, you can now run one unified and relatively small but capable model! On an RTX 5090 it still delivers around 143 tok/s for chat completion, while simultaneously acting as a strong Typed Decision engine with Vision support. A bit on the history: To get there, I tried several different backbone approaches, starting last weekend on Friday. First, I built an mmBERT-based model with my own fine-tuning, which reached around 55% Soft Accuracy - a bit better than Laya, worse then Von-1.1. Wasn't amazing t.b.h. Then I tried Spark-X2.5 - that's also included in my benchmark. That was already stronger than Laya and Von-1.1, but still weaker than Qwen3.5-4B (OpenJev etc.). Eventually, I tried PrismML's Bonsai-2-27B as a LM backbone and was blown away. MUCH better calibration. Slightly better accuracy. Extremely fast for a 27B dense model! And super small with 9250 MB VRAM usage! It is effectively Qwen3.8-27B in PrismML’s Q2_64 quantization. PrismML used some black magic QAT with a proprietary dataset to retain roughly 98% of the performance of the unquantized Qwen3.8-27B dense model. It still runs with 100+ tok/s peak on my RTX 3090. So I think this works really well for all of our local deployments! I also published two papers covering what I learned (yes, AI helped me with formulas and proving, but I did the hard work manually! No hallucination BS here even though these papers are super long. Please proof me wrong if I did mistakes or overclaim things. I seek intellectual honesty! And I did a lot of this to learn and share my learnings with everyone): How to turn existing inference engines into Typed Decision engines The benchmark, post-hoc calibration method, and why it works So you all can now reproduce _and refute_ my work! One of the next things I'm currently working on is fine-tuning of these models (I have a huuge un-released dataset already because of a sudden idea for a new deterministic method of transfer dataset generation (I might post on this in a few days after the HF release). Only 700k of the examples improved my tiny mmBERT so much that I believe this is almost the coolest trick since sliced bread ;)... btw. just one bit on that. RLCD is overrated guys! Use CE as the primary loss with an eye on Brier+NLL; it's cheaper and more effective! Also, MiCA is helping, but only if the model is already saturated. On calibration, we can improve my post-hoc calibration A LOT still. We just need to make the benchmarks bigger and not only look after Qtype-stratified but more nuaunced bins! This will improve all of our open models in calibration, when the inference engine applied targeted temperature scaling as to the generated `calibration.json`'s data! Inference engines should implement this - the simple variant I've implemented already helped by a few %(!) And then there are a million more ideas that I have... This will be fascinating times! We currently see the beginning of another AI revolution! I cannot even sleep anymore.. So please guys, try my inference engine! Tell me how well it works for you. I only tested on Linux/CUDA so far. And I didn't have time to optimize for ROCm and MLX yet. Please help me make this work flawlessly for everyone! Also for people using Windows! Please help me improve the benchmark too! Most benchmarks released for typed decision models are suited for a very non-diverse small capability set, have very little samples (statistically irrelevant!), or test by applying established benchmarks in reply - which basically means testing for HUGE amounts of technical test on metrics that most people will rarely find useful in real-world use-cases. But my benchmark focuses on real use-cases from the real-world. It's synthetic but I did deep research the capability set using GPT-6 Astra in Pro mode and checked it with my experience (20+ years in the field). I believe this isn't a stupid paperclip benchmark; but also here.. please proof me wrong and help me improve this! We need more real use-cases. This is also important for the post-hoc calibration! (To make the Jev-like model not being so over or underconfident - but for real-world use-cases!) And last but not least.. please, can someone make such a cool DOOM video using this? :D I promise I will share it everywhere! Links Bonsai-Llama-Jev: kyr0/Bonsai-Llama-Jev typed-decision-bench: kyr0/typed-decision-bench Benchmark results: https://kyr0.github.io/typed-decision-bench/ Typed Decision Engine paper: https://kyr0.github.io/Bonsai-Llama-Jev/ Calibration paper: https://kyr0.github.io/typed-decision-bench/paper/ Setup is basically: git clone https://github.com/kyr0/Bonsai-Llama-Jev cd Bonsai-Llama-Jev make setup and it runs. Disclaimer: AI helped me formatting this post (I'm not a native English speaker), but nothing here is hallucinated. This was crazy hard work for 5 days almost day and night ongoing.