Turning GLM-5.3-Flash into a Jev-like decision model

# Turn GLM-5.3-Flash into a Jev-like System One model Source: [https://www.privatemode.ai/blog/system-one-from-glm-flash](https://www.privatemode.ai/blog/system-one-from-glm-flash)   Sep 24, 2026 Typed decisions with a probability for every option, in a single forward pass: matching Jev's accuracy and speed with an LLM\.   Marko Rosenmüller, PhD Technical Lead AI **TL;DR:**In this post, we show how an off\-the\-shelf LLM can make typed decisions in a single forward pass\. This approach makes it possible to turn an LLM into a Jev\-like decision model\. We evaluate the approach using GLM\-5\.3\-Flash running on[Privatemode](https://www.privatemode.ai/)\. Using a benchmark constructed from public data sets, we show that this setup delivers results that are on par with[TypeSafe's Jev](https://docs.typesafe.ai/models)in terms of decision accuracy/correctness and speed\. As a bonus, the setup with GLM\-5\.3\-Flash enables typed decisions on images, which is not possible with Jev\. [How it works](https://www.privatemode.ai/blog/system-one-from-glm-flash#turning-an-llm-into-a-decision-model)[Playground](https://www.privatemode.ai/blog/system-one-from-glm-flash#try-it)[Benchmark](https://www.privatemode.ai/blog/system-one-from-glm-flash#benchmark-results)## Why typed decisions Much of what software asks an LLM is a decision\. "Which team should handle this ticket?", or "Does this contract clause belong in the liability section?"\. In such cases, software typically requires that the LLM's response follows a certain format like JSON and that it comes from a pre\-defined set like "yes" and "no"\. Given the right instructions, LLMs can typically already fulfill this reliably\. However, in the basic approach, speed and costs become an issue: For each decision, the LLM needs to write a whole JSON object, and a reasoning model may think for hundreds of tokens before that\. Further, you also don't learn the confidence of the model \(unless you explicitly ask it\)\. All these aspects can matter a lot in practice and have so far prevented people from employing LLMs for decision making in high\-volume/high\-throughput scenarios\. Specialized decision models \(or "System One" models\) like Jev and[Laya](https://huggingface.co/convaiinnovations/laya)are designed to address this\. You pass in a piece of state and a set of named options, and you get back the chosen option together with a confidence value \(i\.e\., probability\) for each one\. ## Turning an LLM into a decision model Initially, we asked ourselves if an LLM could be turned into a decision model with Jev\-like properties\. The short answer is: "yes"\. In the following, we show how it works\. To understand our approach, it's important to understand how LLMs work: An LLM never writes text directly\. Given a prompt, an LLM outputs a probability distribution over its entire vocabulary of tokens\. In text generation, in the simplest case, the token with the highest probability is selected as the*next token*\. The selected token then is appended to the prompt and the whole process repeats\. As described above, this is costly and slow if you just want to set a few fields in a JSON object\. Our core insight is that it's unnecessary to have the LLM predict the whole JSON object, as we already know its shape\. We're only interested in the LLM's*typed judgement*for a given input\. We realized that it's possible to craft prompts so that we get the typed judgement in a single run of the LLM — with no fine\-tuning, on the model exactly as it ships\. This is the difference to Jev and Laya, which are models trained for the purpose\. The basic steps are as follows: 1. **Number the options\.**The state, the question, and the output options go into the prompt as JSON, with an index on every option\. The instruction asks the model to answer with`choice\_index:`followed by an index\. 2. **Prefill the answer\.**The prompt ends with`choice\_index:`\. Consequently, the first token the model produces will be an index into the pre\-defined options\. 3. **Evaluate the output\.**Rather than reading the token the model emits, we read the probabilities it assigned to all option indexes at that single position\. Normalized over the options, these give a probability for each answer, and we simply pick the most probable one\. End the prompt inside the answer user \{"state": "I was charged twice for my order\.", "question": "Which team?", "options": \[ \{"index":0, "name": "payments"\}\{"index":1, "name": "complaints"\}\{"index":2, "name": "technical"\} \]\} assistant choice\_index: Read one row of logits 0 1 2…whole vocabularymax\_tokens: 1temperature: 0allowed\_token\_ids Keep the options, renormalize - 0payments62\.1% - 1complaints37\.7% - 2technical0\.2% Answerpayments confidence0\.39, where 1 means certain and 0 means the options are equally likely The prompt numbers the options and it ends with the assistant’s answer already begun as`choice\_index:`, so the next token is the index\. A mask on the vocabulary allows only the option indexes, the API returns their log probabilities, and normalizing them over the options gives a probability for each answer\. The logit values in the middle panel are illustrative\.We implemented the above steps for GLM\-5\.3\-Flash running on vLLM \(in Privatemode\)\. We use the`/chat/completions`endpoint with`continue\_final\_message`and`add\_generation\_prompt: false`, because these let the model continue the prefilled assistant turn from step 2 instead of starting a new one\. They also let us pass images next to the text, which is what makes typed decisions on images possible\. For vLLM and GLM\-5\.3\-Flash, we found the following details to matter: 1. vLLM's`allowed\_token\_ids`can be used to limit the LLM's output vocabulary only to allowed options\. It drops every other token to`\-inf`\. We set it, but it is a guardrail rather than a requirement\. 2. `top\_logprobs`is not enough for step 3\. It reports the distribution before the restriction is applied, so formatting tokens such as a leading space take up the top slots, and some options drop off the list and appear to have a probability of zero\. vLLM's`logprob\_token\_ids`solves this: it returns the log probability of exactly the token ids you ask for\. 3. The token ids of the indexes depend on the model's tokenizer\. Digits aren't always single tokens\. GLM\-5\.3\-Flash, for example, has a single token for`12`\. Rather than shipping a model\-specific tokenizer, the library gets the token ids from the server, which keeps it simple to use with any model: sending a prompt to`/completions`with`echo`returns its exact tokenization by the model that is actually serving\. You can find our implementation in the below repository\. [edgelesssys/privatemode\-decisionsThe Python library: token oracle, prompt, masking and renormalization, against any vLLM\-backed endpoint\.](https://github.com/edgelesssys/privatemode-decisions)## Try it The playground below runs GLM\-5\.3\-Flash queried with the above setup on Privatemode, directly from your browser\. Pick one of the examples, among them a scanned invoice and a question that depends on your local time, or write your own questions and add images\. Each answer comes back as a distribution over its options, typically within a few hundred milliseconds\. The distribution is often very useful, e\. g\., to decide whether to include a human\-in\-the\-loop\. The model solves most classic trick questions, but not all of them\. ## Benchmark results We evaluated our approach using a custom benchmark, which is available in the below repository\. [edgelesssys/privatemode\-decisions\-benchmarkThe benchmark: methodology, frozen dataset specs, harness and aggregation\. Every number in this post can be recomputed from it\.](https://github.com/edgelesssys/privatemode-decisions-benchmark)We compared three systems on 29 public, labeled datasets: GLM\-5\.3\-Flash hosted on Privatemode and queried with the technique above, TypeSafe's Jev, and Convai's Laya\. The datasets have between 2 and 151 options and cover intent routing, sentiment, topic classification, moderation, entailment, question answering, legal text, and scanned documents\. Both English and German text is included in the corpus\. All three systems receive the same state, the same option names in the same order, and the same instruction\. We ran each dataset twice\. Even at temperature 0, our GLM\-5\.3\-Flash and Jev changed up to 3\.5% of their answers between identical runs: temperature 0 removes the randomness from sampling, but batching and floating\-point arithmetic still keep a forward pass on a busy server from being bit\-reproducible\. We therefore treat smaller differences as noise\. We ran Jev and Laya with their default settings and did not tune our prompt on these datasets\. ### Accuracy Compared across the 28 text datasets, GLM\-5\.3\-Flash and Jev are on par\. Each is more accurate on 10 datasets; on the remaining 8, the two are within one percentage point of each other\. The median gap is 0\.7 percentage points in Jev's favor, which is not statistically significant \(p = 0\.64\)\. Laya, a model with 421 million parameters that we ran locally, scores lower than both on most datasets\. Its median gap is 13 to 15 percentage points, which is statistically significant \(p < 0\.001\)\. The number of options has a larger effect on accuracy than the choice between Jev and GLM\-5\.3\-Flash\. Which one is more accurate, dataset by dataset? 10GLM\-5\.3\-Flashmore accurate8about the same10Jev more accurate The typical gap is0\.7percentage points, slightly in Jev’s favor\. Two equally accurate systems would show a gap at least this large in about6out of 10 comparisons, so it is well within chance\. Accuracy by number of options GLM\-5\.3\-FlashJevLaya with reasoningembedding similarity 20%40%60%80%100%26datasets3–68datasets7–208datasets21–804datasets81\+1dataset Hover or tap a mark for its numbers\.Top: each square is one of the28datasets both hosted systems answer, colored by which of the two was more accurate on it; within one percentage point counts as the same, the variation between two identical runs\. The chance estimate is a two\-sided Wilcoxon signed\-rank test over the per\-dataset differences \(p =0\.64\)\. Bottom: mean accuracy by number of options, averaged only over datasets all three systems answered, so each point covers the same questions; the bands along the bottom are not evenly sized\. The dashed and dotted lines are controls on the same datasets, both zero\-shot like the rest: GLM\-5\.3\-Flash allowed to reason before it answers, and plain embedding similarity with no decision model\.Across datasets, the number of options changes along with everything else about the task\. Three datasets, however, label the same questions twice, once coarsely and once finely, so the task stays the same and only the number of options changes\. On TREC, going from 6 to 42 options, Jev drops from 92\.1% to 85\.6%, GLM\-5\.3\-Flash from 91\.2% to 79\.6%, and Laya from 88\.4% to 51\.2%\. On MASSIVE, going from 18 scenarios to 59 intents raises the scores of Jev and GLM\-5\.3\-Flash in both languages, while Laya's score drops\. The number of options alone doesn't determine how hard a task is\. GLM\-5\.3\-Flash on PrivatemodeJevLaya TREC questions 40%60%80%100%85\.6%79\.6%51\.2%6options42options MASSIVE, English 40%60%80%100%83\.0%82\.4%44\.6%18options59options MASSIVE, German 40%60%80%100%79\.2%77\.9%22\.0%18options59options Hover or tap a mark for its numbers\.The same questions labeled twice, once coarsely and once finely, so the only variable that changes is the number of options\. TREC splits 6 question types into 42; MASSIVE splits 18 scenarios into 59 intents, in English and in German\. Numbers on the right are accuracy at the finer granularity\.### Latency and cost We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model\. As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time\. From Germany, Privatemode answered in 180 ms and Jev in 264 ms\. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode\. On cost, Jev is cheaper\. One million decisions cost about EUR 62 with GLM\-5\.3\-Flash and about EUR 16 with Jev, at each service's list prices\. Time per decision From Germany GLM\-5\.3\-Flash on Privatemode180ms\(173–263\) From the US GLM\-5\.3\-Flash on Privatemode299ms\(271–402\) 0 ms450ms Cost per million decisions GLM\-5\.3\-Flash on Privatemode€62 Hover or tap a mark for its numbers\.Time per decision is the model call as a user sees it, network included, measured one request at a time from Germany and from the US at the same time, on four datasets\. The dot is the median; the band runs from fast requests \(10th percentile\) to slow ones \(95th percentile\)\. Cost is what one million decisions cost at each service’s list prices, the median over the28datasets both systems answer, so the scanned documents Jev cannot read are not in it\.Most of the difference comes from the price per input token, and some from how each system packages a question\. Jev adds roughly 270 tokens of fixed overhead and about 10 tokens per option\. The GLM\-5\.3\-Flash prompt adds about 55 tokens of fixed overhead and about 20 per option\. Below about 21 options, it sends fewer tokens than Jev; above that, it sends more\. Extra input tokens GLM\-5\.3\-Flash sends per question, compared with Jev Below zero GLM\-5\.3\-Flash sends fewer tokens, above zero more\. \-2000\+200\+400\+600\+800020406080100options per question↑ GLM\-5\.3\-Flash sends more↓ GLM\-5\.3\-Flash sends fewerbreak\-even at ~21optionssst2,2options:127vs317tokens \(\-190\)ledgar,100options:2,040vs1,207tokens \(\+833\) Hover or tap a mark for its numbers\.One point per dataset: how many input tokens GLM\-5\.3\-Flash on Privatemode sends per question, minus how many Jev sends for the same question\. The question text is identical for both, so it cancels; what is left is the packaging\. Jev adds a large fixed block and little per option, GLM\-5\.3\-Flash a small fixed block and more per option\. The dashed line is the trend across all datasets: GLM\-5\.3\-Flash starts about219tokens below Jev and adds about11more per option\. The outlier at 13 options is scotus, whose long court opinions the two tokenizers split differently\.Laya runs locally, so there is no comparable latency or price per decision for it\. ## Multimodal decisions The state doesn't have to be text\. GLM\-5\.3\-Flash is a vision\-capable model, so a question can come with images, such as a scanned invoice, a photo of a damaged parcel, or a screenshot\. The image goes into the same prompt, and the answer is still a single token with a probability for every option\. The playground's*Scanned document*example shows this; you can also paste or drop in your own image\. According to its documentation, Jev works on text, and Laya is a text encoder, so neither takes images as input\. On RVL\-CDIP, a set of 1,600 scanned business documents in 16 classes, GLM\-5\.3\-Flash reaches an accuracy of 70\.2% and is the only one of the three that can answer\. A document costs more than a sentence: the image adds about 1,350 input tokens, so a million document decisions cost about EUR 270\. ## Capabilities With many options, each system hits a limit\. Laya's option names share a budget of 192 tokens, which is enough for the 77 intents of banking77 but not for the 151 intents of CLINC150\. Privatemode's deployment of GLM\-5\.3\-Flash reports at most 128 entries in`logprob\_token\_ids`, while the mask in`allowed\_token\_ids`takes every option\. So the library sends a question with 151 options twice, identically, and reads the probabilities of 128 options from the first response and of the other 23 from the second\. Both requests run the same forward pass, so the merged result is the distribution a single request would return, up to the run\-to\-run noise above\. On CLINC150, GLM\-5\.3\-Flash reaches 87\.5% this way, against 78\.4% for Jev\. The second request and the long option list cost time: a decision takes 719 ms, against 249 ms for Jev\. Past that, the ceiling is the 191 option indexes that GLM\-5\.3\-Flash spells as a single token\. Some tasks only one or two of the systems can handle at all\. GLM\-5\.3\-Flash on PrivatemodeJevLayaScanned documents RVL\-CDIP, 1,600 business documents, 16 types 70\.2%text onlytext onlyChoice of model Same code, another model any model on the APIfixedfixed Numbers are accuracy on the named dataset\.## Further findings **Some of the remaining errors are in the labels\.**When most systems agree on an answer and the dataset's label disagrees, the label is often one of two defensible answers\. banking77 has many such pairs, for example`get\_physical\_card`and`order\_physical\_card`, or`declined\_transfer`and`failed\_transfer`\. About 17% of banking77's examples fall into this category, so the highest score any system could reach there is about 85% rather than 100%\. **Reasoning helps, at a price\.**As a control, we let the same GLM\-5\.3\-Flash reason before it answers, on all 29 datasets\. It is more accurate in every band of option counts, from 89\.9% against 85\.5% with two options to 82\.0% against 79\.2% between 21 and 80\. It also writes hundreds of tokens per decision instead of one and costs about EUR 350 per million decisions, against EUR 62\. At the other end, plain embedding similarity, which picks the option closest to the text with no decision model at all, reaches between 45\.9% and 72\.8% depending on the band\. **Renaming the options affects the systems differently\.**We re\-ran every dataset with each option replaced by a synonym and nothing else changed\. On boolq, where`true`and`false`became`correct`and`wrong`, GLM\-5\.3\-Flash lost 20 points, while the other two lost less than three\. Because this renaming also changes the meaning of the question, it doesn't isolate memorization\. The results for every dataset are in the benchmark repository\. ## Build it yourself Our library is written in Python and works with any vLLM\-backed endpoint; it relies on vLLM's extensions to the OpenAI API, such as`allowed\_token\_ids`and`logprob\_token\_ids`\. Its README explains how to set it up with Privatemode, and an AGENTS\.md file tells coding agents what an implementation has to get right\. The benchmark repository contains the methodology, the dataset specifications, the test harness, the aggregation, and every raw run as a download, so every number in this post can be reproduced without running anything again\. If you know a setting that serves any of the three systems better, find a mistake, or want to add a dataset or another system, we welcome pull requests\. With Privatemode, these decisions are protected by confidential computing: your data stays encrypted in memory even during processing, and the client verifies the deployment's attestation report before sending anything\. You can read more on our[security page](https://www.privatemode.ai/security-and-encryption)\. ## Build it on Privatemode Create an account, then give this post to Claude Code, Codex, or the coding tool of your choice\. Together with the GitHub repository, it has everything needed to build typed decisions into your own code, protected by confidential computing\. ## Explore other articles ## Confidential containers in Kubernetes: what a real operator exclusion requires A technical operator exclusion in Kubernetes is possible today, but only one architecture actually delivers it\. The short version, plus where to read the full paper from the 21st German IT Security Congress\. ## Building AI services for professionals bound by confidentiality obligations: a guide with Privatemode AI How to use LLMs productively without the provider accessing the data – confidential computing and the confidentiality agreement under § 203 StGB that technology can't replace\.