Banger paper from IBM on decision gates for agents.

They add a runtime gate to tool-using agents and keep the models and tools the same.

JevAct is a check that sits between the agent and its tools. An LLM first turns the domain policy into code checks and yes/no questions. Before a selected tool call runs, the code checks the facts in the tool records, and a Jev-like decision model answers the questions about the conversation. If the call fails a check, it goes back to the agent with feedback so the agent can revise it.

On τ²-bench Telecom, Opus 4.6’s success across all three runs rose from 39.5% to 80.7%.

On Telecom, all eight agents improved by 16.9 points on average in three-run success. All nine improved on BFCL. Airline results were smaller and mixed.

They show that the gains come from the harness.

https://arxiv.org/abs/2610.38620

Chat with Paper: https://academy.dair.ai/papers/think-before-you-act-decision-gate-for-llm-agents-2610.38620…

Think Before You Act:Decision Gate for LLM Agents

Source: https://arxiv.org/html/2610.38620 Yuval David11footnotemark:1Affiliation:IBM SILOffer AkrabiAffiliation:IBM SILErez BilgoryAffiliation:IBM SILIdo LevyAffiliation:IBM SILSami MarreedAffiliation:IBM SILSergey ZeltynAffiliation:IBM SILSegev Shlomov††thanks:Corresponding author: [email protected]:IBM SIL

Abstract

LLM agents increasingly perform consequential actions through external tools, yet their proposed actions are often executed without independent verification of policy compliance or task completion. We introduceJevAct, a runtime verification framework that intercepts selected agent actions before execution and enables feedback-driven correction. JevAct compiles domain policies into executable state checks and structured questions evaluated by a prefill-only decision model, which produces probability-based judgments without relying on LLM generation. By combining deterministic verification of tool-state facts with probabilistic assessment of conversational context, the framework identifies potentially invalid actions and premature task termination, providing targeted feedback for bounded revision. JevAct operates independently of the underlying agent model and requires no modifications to its parameters or tools. We evaluate JevAct across eleven LLM agents onτ2\tau^{2}-bench Airline and Telecom tasks and BFCL. Results show improvements in agent reliability across multiple models and benchmarks, with particularly consistent gains on Telecom and BFCL, while performance on Airline is more mixed. Overall, these findings suggest that combining structured decision models with executable verification can improve the reliability of tool-using LLM agents.

ReActJevActτ2\tau^{2}-Airline0%20%40%60%80%100%GPT-4.136.036.0 46.046.0 Opus4.672.072.0 76.076.0 gpt-oss-120b42.042.0 50.050.0 Qwen3.8-27B74.074.0 76.076.0 τ2\tau^{2}-TelecomGPT-4.111.411.4 17.517.5 Opus4.639.539.5 80.780.7 gpt-oss-120b28.928.9 51.851.8 Qwen3.8-27B57.957.9 81.681.6 BFCLGPT-4.138.238.2 47.847.8 Opus4.655.955.9 59.159.1 gpt-oss-120b39.939.9 47.647.6 Qwen3.8-27B48.148.1 49.949.9

Figure 1:pass^3 results from Table2.## 1Introduction

Language-model agents increasingly use tools to change records, make bookings, and guide users through multi-step procedures. A mistaken answer can sometimes be corrected in a later turn; an unauthorized write or a premature hand-off may be harder to undo. Yet many agent loops execute a proposed tool call as soon as the model produces it. This leaves a gap between generating an action and checking that it is appropriate to carry out.

ReAct agents interleave reasoning with tool use, while reflection methods such as Self-Refine and Reflexion add critique and feedback to improve later proposals or trials(Yao et al., 2022;Madaan et al., 2023;Shinn et al., 2023). These methods can improve agent behavior, but they do not by themselves verify a particular action against a domain policy immediately before execution. Runtime constraints and learned tool-call checks address this separate problem(Wang et al., 2026;Chen et al., 2025;Mou et al., 2026;Kang et al., 2026a).

A separate generative LLM could review each proposal and explain its concerns, as in LLM-as-a-judge methods(Zheng et al., 2023;Liu et al., 2023). But free-form explanations must still be interpreted before a gate can act on them, and the cost of making this decision with a frontier model is high. Decision models instead return choices and probabilities for fixed questions, making threshold-based checks straightforward. Their probabilities are not guaranteed to be calibrated or correct, so their value has to be tested in the agent loop rather than assumed(TypeSafe AI, 2026;Deußer et al., 2026).

We present JevAct, a harness that checks selected proposals before execution. Offline, a compiler turns a domain policy into questions, rules, executable checks over tool state, and where needed, a hand-off checklist. At runtime, code checks facts available in tool records and Kev (an open sourced Jev-like(TypeSafe AI, 2026)decision model) answers questions that depend on the conversation. A passing proposal proceeds; a flagged proposal is returned to the agent with targeted feedback and a bounded chance to revise. The agent still proposes actions; Kev does not choose them. This separates JevAct from decision-centric runtimes that delegate action selection to a decision model(ZJUNLP, 2026).

Our central question in this is work is whether a decision model can serve as a probability-thresholded verifier alongside executable tool-state checks. We compare JevAct with a ReAct harness using the same agent LLM in both arms, across Airline and Telecom tasks inτ2\tau^{2}-bench and multi-turn function-calling tasks in BFCL(Barres et al., 2025;Patil et al., 2025). On Telecom, all eight agents improve both at-least-once success and success across all three runs. On BFCL, all nine improve in three-run success, but at-least-once results are mixed. Airline effects are smaller and vary by agent (see Figure1). Because JevAct adds compiled checks and revision opportunities, these comparisons evaluate the full harness, not the judge in isolation.

Our contributions in this work are threefold:

• •We introduce JevAct, which combines executable checks over tool state with thresholded decision-model judgments of conversational context before selected actions are executed.
• •We lay the foundation for a workflow that compiles domain policies into questions, decision rules, executable state checks, and hand-off checklists for a decision-model gate.
• •We evaluate the harness against same-model ReAct baselines across three benchmark settings, with component and judge ablations alongside task-success, repeatability, and estimated judge-cost results.

2Related Work

2.1LLM Agents and Test-Time Feedback

ReAct interleaves model reasoning with environment actions(Yao et al., 2022). Self-Refine and Reflexion add test-time critique or feedback to improve later proposals and trials without updating model weights(Madaan et al., 2023;Shinn et al., 2023). These methods motivate giving an agent a chance to revise, but do not by themselves check whether a consequential tool call satisfies a domain policy before execution.

2.2Runtime Policy Enforcement

Several systems place explicit controls around agent execution. GuardAgent uses a separate agent to generate and execute verification code from user-defined guard requirements(Xiang et al., 2024). Zwerdling et al. compile company policies into executable guards that check compliance before tool actions(Zwerdling et al., 2025). AgentSpec represents runtime constraints as triggers, predicates, and enforcement actions; ShieldAgent derives verifiable rules from policy documents; and CUGA applies governance interventions across an execution graph(Wang et al., 2026;Chen et al., 2025;Shlomov et al., 2026). ToolSafe uses a learned guardrail to assess proposed tool calls and return feedback, while PolicyGuard and PolicyGuide use model-based verifiers for policy adherence and workflow guidance(Mou et al., 2026;Kang et al., 2026a;Kang et al., 2026b). Related approaches include verifier-grounded runtime mediation, which uses executable procedural checks to select interventions(Yabusaki, 2026), and ToolFence, which combines deterministic authorization with judge-based capability grants(Li et al., 2026a). CaMeL instead enforces security through separation of trusted control flow and untrusted data(Debenedetti et al., 2026). JevLoop assigns action and control decisions to a decision model; JevAct leaves proposals to the ReAct LLM and places Kev, an open-source Jev-type decision model, in a separate verification step before selected actions commit(ZJUNLP, 2026). We focus on probability-thresholded verification using a fixed-choice decision model, combined with executable checks of tool-state facts and feedback-driven revision.

2.3LLM Judges and Decision Models

LLM-as-a-judge methods use generative models to compare responses, as in MT-Bench, or apply explicit rubrics, as in G-Eval; such judgments can also be sensitive to response order(Zheng et al., 2023;Liu et al., 2023;Wang et al., 2024). Jev introduced decision models that return typed choices, scores, and probabilities rather than generated text(TypeSafe AI, 2026). Initial evaluations study its closed-label decisions and rubric-based scoring(Deußer et al., 2026). Two recent preprints evaluate Jev as a judge: Li et al. report competitive results on preference and evidence-grounded factuality but weaker performance when derivation is required, while Rao and Callison-Burch report mixed rubric results and overlapping errors with LLM judges(Li et al., 2026b;Rao and Callison-Burch, 2026). Those studies judge completed outputs. JevAct instead uses Kev, an open-source Jev-type model, online before selected actions execute, combining its fixed-choice judgments with programmatic state checks and feedback for revision(Palmer, 2026).

3Method

In this work we introduce JevAct, which augments the harness of a ReAct agent(Yao et al., 2022)by inserting a selective checking stage between message proposal and execution. The checks can defer execution and request a revised proposal from the LLM with temporary feedback, subject to a fixed revision budget. The agent’s LLM, base prompt, and tools remain unchanged.

3.1Overview

Setting.

At each steptt, an LLM reads the conversation history and proposes a message, which may contain user-facing text, tool calls, or both. The harness normally executes the proposal and adds it and any tool results to the history. In policy-governed tasks, JevAct checks selected proposals before execution because some mistakes are hard to undo. Awrite callchanges the environment, for example by booking, cancelling, or changing a setting. Ahand-offends the agent’s attempt, either by transferring the user to a human or by telling the user that nothing more can be done. The same runtime design can also apply generic checks when no written policy is available.

Components.

Our method JevAct has four main components, as illustrated in Figure2. (1)Decision loop: Similar to a ReAct agent, every step is composed of four stages:propose,trigger,check, andcommit or revise. The trigger selects a check for a write call or hand-off; other proposals proceed as in ReAct. A passing proposal is committed, while a flagged one is returned to the agent with feedback so it can revise. (2)Compiler: Offline, an LLM turns each domain’s policy into yes/no questions, rules that combine their answers, code that computes facts from tool results, and (where needed) a hand-off checklist. (3)Judge: A decision model answers the questions with probabilities, allowing a check to act only on confident answers. (4)Checks: Thewrite gatetests whether a proposed write call is allowed by the policy. Thehand-off checklooks for required troubleshooting steps that remain undone. For write calls, code computes what can be established from tool records; the judge answers questions that depend on the conversation. The hand-off check uses the judge to assess which problem applies and which checklist steps are settled. Although the components rely on each other, each is modular and can be independently replaced or improved in future work.

Historyhth_{t}LLMπ\piproposesmmwrite callor hand-off?Checkcode: list items, compute factsjudge (Kev): yes/no per itemrule: pass or flagCommitexecutemmyespassno: as in ReActflag: discardmm, return feedbackff(shown once)1234Checksquestions, rules, code,hand-off checklistCompiler(LLM)PolicyΠ\Pi, tool schemas,unlabeled logsofflineonline, at every step of the agent

Figure 2:JevAct. Offline (top), an LLM compiles the policy into checks. Online (bottom), the agent’s proposal goes through four stages:(1)propose,(2)trigger,(3)check when required, and(4)commit or revise. Proposals without a write call or hand-off are executed as in ReAct. A flagged proposal is discarded, and the LLM proposes again with temporary feedback.

3.2Decision loop

At steptt, the agent goes through four stages:

1. 1.Propose.The LLM proposes a messagem∼π(⋅∣ht)m\sim\pi(\cdot\mid h_{t})given the historyhth_{t}.
2. 2.Trigger.Ifmmcontains a write call, the write gate applies. Ifmmhands off, the hand-off check applies. Any other message is committed directly.
3. 3.Check.The check returnspassorflag. A flag comes with a feedback textffthat names the problem.
4. 4.Commit or revise.A passed message is executed and added to the history, as in ReAct. A flagged message is discarded: it is never executed and never stored. The LLM then proposes again with the feedback appended for this one call,m′∼π(⋅∣ht,f)m^{\prime}\sim\pi(\cdot\mid h_{t},f).

Each check has a revision budget:R=2R=2for the write gate andR=1R=1for the hand-off check. While budget remains, a flagged proposal is discarded and the agent gets another chance to respond to the feedback. Once the budget is exhausted, the next proposal is committed without another check. Thus, a check cannot block the agent indefinitely, and it adds at mostRRextra LLM calls. Feedback is temporary and is not added to the conversation history, so critiques of earlier steps do not accumulate in the context.

3.3Compiler

The compiler runs offline for each domain. It uses the policy and tool schemas to build the checks; code generation and repair also use example tool outputs and calls from development logs. For each write tool, it produces:

• •yes/no questions about policy requirements, the source of call arguments, user intent, and any required prior steps or confirmations;
• •a Boolean rule that combines the answers and specifies when the call is allowed; and
• •a Python function that computes policy-relevant facts available in tool records (such as dates, amounts, and counts) and returns any violations those records establish.

For hand-offs, it also produces the problem types covered by the policy and the steps the agent should try for each problem before giving up. Each step has asettledquestion: was it tried, verified, or ruled out? One instruction in the compile prompt matters most:a condition that was never discussed is not ruled out. Without it, “enable roaming” compiles to “… or the user is not traveling”, which looks settled even though nobody asked. With it, the checklist gets its own step: ask whether the user is traveling.

For write gates, the compiler gets one repair pass using logged calls without outcome labels. Next, questions that the judge answers with a confident violation on more than 10% of a tool’s development calls are dropped; this filters questions that are hard to answer from the available evidence. The hand-off checklist has a separate filter: steps flagged on more than 5% of hand-offs in solved training episodes are dropped. The prompts and examples are in SectionA.1.

3.4Judge

For the judge in this work, we use Kev(Palmer, 2026), an open-source Jev-like decision model. Given a prompt and questions with fixed answer options, it returns a probability for each option in one forward pass, without generating text. Questions about the same prompt share that pass. We use Kev-27B, a fine-tune of Qwen3.8-27B, without further training for our setup. Its probabilities let the checks act only on confident answers, and the cost of a check does not grow with the number of questions. Because Kev is separate from the agent LLM, the agent does not judge its own proposals. The checks divide the work by evidence source: code calculates facts available in tool records, while Kev answers focused questions that require interpreting the conversation, such as whether the user requested an action or whether a step was tried. This asks the judge to assess specific candidate items rather than diagnose an entire problem from scratch. Other models that answer the same questions can take Kev’s place. We chose Kev as open source model.

3.5Checks

Write gate.

The write gate runs before each write call, in three steps:

1. 1.Code:the compiled function reads the proposed call’s arguments and earlier tool results, then returns any violations established by those records.
2. 2.Judge:Kev answers the questions for that tool using the policy, conversation, tool results (with dates annotated and list values ranked), and proposed call. An answer counts asnoonly when Kev is confident, meaningP⁡(yes)≤0.05P(\text{yes})\leq 0.05; otherwise it counts asyes.
3. 3.Rule:the call is flagged if the code finds a violation or the Boolean rule is not satisfied by the question answers.

Feedback lists the established violations and failed questions, then asks the agent to correct the call, ask the user for missing information, or explain why the request cannot be fulfilled. If the agent believes the call is already correct, it may submit it again. After a second flag, the feedback directs the agent to respond to the user rather than call the tool again.

Hand-off check.

The hand-off check runs before a transfer call and before a message that gives up in words. A keyword filter first identifies possible give-up messages, which the judge then confirms. From the conversation, Kev identifies the user’s problem, whether the user says it is fixed, and whether each checklist step is settled. A step isopenwhen the probability that the user’s problem requires it is at least0.50.5and the probability that it is settled is at most0.50.5. If the problem is not fixed and at least one required step is open, the check flags the hand-off and returns the open steps as feedback. The hand-off check allows one revision; after that budget is used, the next proposal is committed without another check.

Function calling without a policy.

BFCL multi-turn(Patil et al., 2025)has no written policy, and its user turns are fixed, so we use two generic checks. Theend-of-turn checkruns when the agent ends its turn while a tool error stating a requirement (for example, “needs to” or “must”) remains unresolved. Kev selects the tool that meets the requirement from the tool list, or selects none. If the probability of a tool being the right one is at least0.60.6, feedback directs the agent to call it rather than ask the user. Therequested-action gateruns before each state-changing call and asks whether the user requested it or a tool error requires it. If Kev assigns at most0.050.05probability to “yes,” the gate flags the call as unrequested and directs the agent to do only what was asked or required.

4Experimental Setup

4.1Benchmarks

τ2\tau^{2}-bench(Barres et al., 2025): an agent serves a simulated user under a domain policy and calls tools that read and write a database. We use two domains.Airline(50 tasks) tests policy-compliant write calls (book, cancel, change).Telecom(the 114-task base split) tests troubleshooting, in which the agent also guides the user through actions on their own device; here the main failure is giving up too early. The user simulator is GPT-4.1, as inτ2\tau^{2}-bench. Scoring isτ2\tau^{2}-bench’s own: the final database state and the required communication.

BFCL multi-turn(Patil et al., 2025): 800 entries, 200 in each of four categories (base, missing function, missing parameter, long context). The user turns are fixed, so there is no user simulator. We follow BFCL’s function-calling loop (no system prompt, up to 20 steps per user turn) and score with its own multi-turn checker: an entry is solved only if every turn’s end state and response match the ground truth. BFCL has no written policy, so JevAct uses its two generic checks here, the end-of-turn check and the requested-action gate (Section3.5).

4.2Agent LLMs

We evaluate eleven agent LLMs. GPT-4.1, Claude Opus 4.6, gpt-oss-20b, gpt-oss-120b, Qwen3.8-27B, and DeepSeek-V4-Pro are tested on all three benchmark settings. Claude Haiku 4.5 and Claude Opus 5.5 are tested on Telecom only; Llama 3.3 70B, Mistral Small 3.2 24B, and Gemma 4 31B are tested on BFCL only (Table2). Within each comparison, the ReAct and JevAct arms use the same agent model. Agent calls use temperature 0 and a maximum of 4,096 output tokens; the GPT-4.1 user simulator onτ2\tau^{2}-bench also runs at temperature 0.

4.3Judge and Compiler

Unless stated otherwise, every JevAct result uses the same configuration. The judge is Kev-27B, used as is, for every agent LLM and benchmark. Claude Opus 4.6 compiles the checks offline for policy-governed tasks (Section3.3). The write gate counts an answer asnoonly whenP⁡(yes)≤0.05P(\text{yes})\leq 0.05and allowsR=2R=2revisions. The hand-off check is active only on Telecom, withR=1R=1. On BFCL, the end-of-turn check acts atP≥0.6P\geq 0.6and the requested-action gate atP≤0.05P\leq 0.05. For the main results and judge-size ablation, thresholds and dropped items were set with Kev-27B. In the judge-choice ablation, each alternative LLM judge uses its own label-free dropped-question list. Table5varies judge size with the checks and thresholds fixed; Table4also changes the dropped-question list for alternative LLM judges.

4.4Metrics

Main results and judge-comparison runs use three runs per task. The single-trial Airline component ablation is reported as pass^1 (Table3). Withccofnnruns of a task solved, we report, averaged over tasks(Chen et al., 2021;Yao et al., 2024),

pass@k=1−(n−ck)(nk),pass^k=(ck)(nk).\text{pass@k}=1-\frac{\binom{n-c}{k}}{\binom{n}{k}},\qquad\text{pass\textasciicircum{}k}=\frac{\binom{c}{k}}{\binom{n}{k}}.Withn=k=3n=k=3, pass@3 is the share of tasks solved in at least one run, and pass^3 is the share solved in all three runs. pass^1 is the mean success rate across individual runs. Cost is reported as tokens per episode for the agent LLM and judge, along with the number of proposals sent back (Appendix able9).

4.5Checks Per Benchmark

Table1summarizes which checks are active on each benchmark. The hand-off check is disabled onτ2\tau^{2}-bench Airline because, in offline tests, it flagged 26% of hand-offs in otherwise solved episodes.

Table 1:Checks per benchmark, with the number of compiled items (dropped items in parentheses).

5Results

Table2compares JevAct with ReAct using the same agent LLM in each pair. The results vary by benchmark: the strongest and most consistent gains appear on Telecom, while Airline and BFCL show a more mixed pattern.

Onτ2\tau^{2}-Telecom, all eight agents improve on both metrics. Averaged across agents, pass@3 rises by 16.0 percentage points and pass^3 by 16.9 points. The largest change is for Opus 4.6, whose success across all three runs rises from 39.5% to 80.7%. GPT-4.1’s pass@3 rises from 45.6% to 83.3%. Even the strongest ReAct baseline, Opus 5.5, improves in pass^3 from 74.6% to 82.5%. The Telecom hand-off check is designed to catch early abandonment, so this pattern is consistent with that check addressing a frequent failure mode. The comparison evaluates the full harness, however, and does not isolate the hand-off check’s effect.

Onτ2\tau^{2}-Airline, gains are smaller and mixed. Pass@3 improves for four of six agents, is unchanged for one, and falls for one. Success across all three runs improves for four agents and falls for two. GPT-4.1’s pass@3 rises from 66.0% to 82.0%, while Opus 4.6 rises from 84.0% to 92.0%. With 50 tasks, a single task changes a score by 2 percentage points, so small differences should be read cautiously. The write gate also intervenes infrequently for some agents: it sends back 0.08 drafts per run for Opus 4.6 and 0.03 for Qwen3.8-27B (Table9).

On BFCL, pass^3 increases for all nine agents, by 0.1 to 9.6 points. Pass@3 improves for four agents and declines for five, by 0.2 to 3.6 points. Thus, higher success across all three runs does not mean that the gate expands task coverage for every agent; the pattern is instead consistent with more repeatable performance on some tasks.

Table 2:Main results (%). pass@3: tasks solved in at least one of three runs. pass^3: tasks solved in all three runs. Both arms use the same LLM and differ only in the harness; the higher value of each pair is inbold.### 5.1Ablations

We examine three aspects of JevAct: the contribution of its verification components, the trade-off between decision models and generative LLM judges, and the effect of judge size.

Write-gate components.

Table3examines the contribution of different write-gate components using a single-trial Airline evaluation with GPT-4.1 as the agent. The full gate improves success from 50.0% to 72.0%, while executable checks alone achieve 68.0%. In contrast, Kev-27B questions alone reach 57.1%, suggesting that programmatic verification accounts for much of the improvement in this setting. Replacing Kev-27B with the smaller Kev-9B reduces success to 58.0%, indicating that judge choice can substantially affect performance. Reducing the revision budget from two to one has a relatively small effect (68.0%). Importantly, flagging every write call once without checking its validity gives no improvement over ReAct (50.0%), with three runs per task and a gpt-oss-120b user simulator, an unconditional re-check of every write leaves GPT-4.1’s pass^3 unchanged (26.0% to 28.0%) while the full gate raises it to 40.0% (paired 95% CI of the gain over ReAct:+2.0+2.0to+26.0+26.0points); for gpt-oss-120b as the agent, the same re-check, or random flags at the gate’s rate, raise pass^3 as much as the full gate (32.0% to 42.0%), so whether the judge’s verdicts matter beyond the revision opportunity depends on the agent. The flag rates further show that the full gate intervenes on fewer than half of write calls, while achieving the highest success rate among the tested configurations. As these results are based on a single trial, differences between configurations should be interpreted cautiously.

Table 3:Single-trial write-gate ablation on Airline with GPT-4.1 as the agent. The experiment uses the same configuration as Table2(code checks, Kev-27B, andR=2R=2).

Judge choice and cost.

Table4compares Kev with open-weight and API-based generative LLM judges. All judged configurations improve pass^1 over their corresponding ReAct baselines by 9.4–18.7 percentage points, suggesting that the verification framework is not dependent on a particular judge architecture. Kev-27B achieves competitive performance: with GPT-4.1 as the agent, it reaches 70.0% success, compared with 67.3–72.0% for generative judges; with Haiku 4.5, it reaches 68.0%, compared with 64.7–66.0%. The main advantage of Kev is efficiency. Its estimated judge cost ranges from $0.0003 to $0.0017 per task, compared with 0\.0020–0.0040 for open-weight LLM judges and 0\.0127–0.0231 for API judges. Thus, prefill-only decision models can provide competitive runtime verification at substantially lower estimated judge cost, without requiring autoregressive generation. These estimates exclude the agent’s own inference cost; SectionA.6provides further details.

Judge size.

Table5evaluates whether increasing the decision model’s size improves verification. Surprisingly, larger judges do not consistently yield better downstream task performance. On Airline, Kev-27B performs best for GPT-4.1 (65.3% pass^1, compared with 59.3% for Kev-9B and 58.0% for Kev-0.8B), but the remaining agents show no consistent preference for larger models. On Telecom, smaller judges frequently match or outperform Kev-27B, indicating that increased model capacity does not necessarily translate into more successful agent interactions. Moreover, in a replay of 609 identical hand-off states, Kev-9B agrees with Kev-27B on 97.0% of decisions, while Kev-0.8B agrees on 90.6% (SectionA.2). This suggests that substantially smaller decision models can often reproduce the decisions of a larger judge.

One factor affecting judge size is input length. On Airline, 41% of write-check inputs exceed 8,192 tokens, beyond the smaller judges’ validated context length. Offline analysis shows that Kev-9B misses more invalid calls on long inputs, while threshold recalibration reduces the gap on shorter inputs. Kev-0.8B remains weaker even on short inputs (SectionA.3, andA.4). Together, these findings suggest that effective judge selection depends not only on model capacity, but also on context length, decision thresholds, and the type of verification required. Since downstream success also depends on the agent’s response to feedback, these results should not be interpreted as direct rankings of judge accuracy.

Overall findings.

The ablations highlight three observations. First, executable state checks provide a substantial part of the measured benefit on Airline, and selective verification beats indiscriminate revision for GPT-4.1 but not for gpt-oss-120b, for which a re-check alone gives the same gain. Second, prefill-only decision models offer a competitive and comparatively inexpensive alternative to generative LLM judges, reducing cost with small drop in performance. Third, larger judges are not uniformly better: smaller models can be effective when their context capacity and decision thresholds are appropriate for the verification task. These results motivate future work of a modular verification approach in which deterministic checks handle structured facts and lightweight decision models assess conversational context.

Table 4:The same JevAct write gate onτ2\tau^{2}-Airline with different judges answering its compiled questions. Each LLM judge uses its own label-free list of dropped questions. Judge cost per task is estimated from logged requests and tokens; see SectionA.6. ReAct is run separately for each block; only the first block’s baseline is shown (the GPT-4.1 baseline for the Kev-size block is 52.7%). Best judge per block and column inbold.Table 5:Judge size ablation. JevAct uses Kev-0.8B, Kev-9B or Kev-27B; compiled checks, thresholds and dropped checklist steps (all set with Kev-27B) are fixed across arms. On Airline the judge answers the write gate’s questions; on Telecom it also answers the hand-off checklist. Best value of each row and metric is inbold.

6Conclusion

We introduced JevAct, a runtime gate that checks selected agent proposals before execution, combines executable checks over tool state with probability-based judgments of conversational context, and returns feedback for bounded revision. Compared with same-model ReAct baselines, the full harness shows its clearest gains on Telecom: all eight agents improve both at-least-once success and success across all three runs. On BFCL, all nine improve in three-run success, while at-least-once results are mixed. Airline results are smaller and mixed across agents. These findings support further study of decision-model gates, but do not establish uniform gains or isolate the judge’s contribution. Two caveats apply: with a gpt-oss-120b user simulator the Telecom gain does not appear (pass^3 changes by−1.8-1.8points, 95% CI−7.9-7.9to+4.4+4.4), and on 260 labelled development write calls 42 of the 101 incorrect calls break no compiled question, because the questions encode the policy rather than the user’s request.

Future work.

The next step is to make the gate work across agent architectures rather than tying it to one ReAct loop. A common interface to proposed actions, tool state, and execution outcomes could support different action formats and orchestration frameworks. We also aim to automate more of the setup: deriving candidate checks from policies and tool schemas, generating and validating executable checks, and selecting thresholds and revision budgets on held-out traces. Such automation should account for both missed violations and unnecessary interventions, while retaining human escalation for uncertain or high-impact decisions. Broader evaluations should include more domains and frameworks, and report end-to-end latency, cost, and safety outcomes alongside task success and repeatability.

References

• Anthropic (2026)Anthropic.Pricing.https://platform.claude.com/docs/en/about-claude/pricing, 2026.Claude Haiku 4.5: $1 / $5 per million input / output tokens. Accessed 2026-10-14.
• Barres et al. (2025)Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ2\tau^{2}-bench: Evaluating conversational agents in a dual-control environment.arXiv preprint arXiv:2506.07982, 2025.
• Chen et al. (2021)Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021.
• Chen et al. (2025)Zhaorun Chen, Mintong Kang, and Bo Li.Shieldagent: Shielding agents via verifiable safety policy reasoning.arXiv preprint arXiv:2503.22738, 2025.
• Debenedetti et al. (2026)Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis, and Florian Tramèr.Defeating prompt injections by design.In2026 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 587–618. IEEE, 2026.
• Deußer et al. (2026)Tobias Deußer, Lorenz Sparrenberg, and Rafet Sifa.Evaluating and benchmarking the system one model jev.arXiv preprint arXiv:2609.37647, 2026.
• Kang et al. (2026a)Seongjae Kang, Taehyung Yu, and Sung Ju Hwang.Policyguard: A dialogue-grounded sub-agent verifier for policy adherence in llm agents.arXiv preprint arXiv:2606.29225, 2026a.
• Kang et al. (2026b)Seongjae Kang, Taehyung Yu, and Sung Ju Hwang.Policyguide: From guarding one action to guiding the whole workflow for policy-compliant llm agents.arXiv preprint arXiv:2608.19861, 2026b.
• Li et al. (2026a)Yanjie Li, Xiangyu He, Xuelong Dai, and Bin Xiao.Toolfence: Fine-grained authorization for secure tool-using llm agents.arXiv preprint arXiv:2609.37196, 2026a.
• Li et al. (2026b)Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman.Jev-as-a-judge: Accept when confident, escalate when unsure.arXiv preprint arXiv:2609.26550, 2026b.
• Liu et al. (2023)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.G-eval: Nlg evaluation using gpt-4 with better human alignment.InProceedings of the 2023 conference on empirical methods in natural language processing, pages 2511–2522, 2023.
• Madaan et al. (2023)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al.Self-refine: Iterative refinement with self-feedback.Advances in neural information processing systems, 36:46534–46594, 2023.
• Mou et al. (2026)Yutao Mou, Zhangchi Xue, Lijun Li, Peiyang Liu, Shikun Zhang, Wei Ye, and Jing Shao.Toolsafe: Enhancing tool invocation safety of llm-based agents via proactive step-level guardrail and feedback.InFindings of the Association for Computational Linguistics: ACL 2026, pages 37125–37153, 2026.
• OpenAI (2026)OpenAI.GPT-4.1 model pricing.https://developers.openai.com/api/docs/models/gpt-4.1, 2026.GPT-4.1: $2 / $8 per million input / output tokens, standard tier. Accessed 2026-10-14.
• Palmer (2026)Jared Palmer.KEV: Prefill-only decision models.https://github.com/jaredpalmer/kev, 2026.GitHub repository.
• Patil et al. (2025)Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez.The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models.InForty-second International Conference on Machine Learning, 2025.
• Rao and Callison-Burch (2026)Delip Rao and Chris Callison-Burch.Jev vs. llms as rubric judges: Cheaper, faster, and wrong in the same places.arXiv preprint arXiv:2609.29769, 2026.
• Shinn et al. (2023)Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.Reflexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023.
• Shlomov et al. (2026)Segev Shlomov, Iftach Shoham, Alon Oved, Ido Levy, Sami Marreed, Harold Ship, Offer Akrabi, Sergey Zeltyn, Avi Yaeli, and Nir Mashkif.Governance by construction for generalist agents.InProceedings of the ACM Conference on AI and Agentic Systems, pages 1280–1287, 2026.
• TypeSafe AI (2026)TypeSafe AI.Introducing system one models and jev.https://typesafe.ai/blog/introducing-system-one-models-and-jev, 2026.Released September 15, 2026.
• Wang et al. (2026)Haoyu Wang, Christopher M Poskitt, and Jun Sun.Agentspec: Customizable runtime enforcement for safe and reliable llm agents.InProceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, pages 2938–2950, 2026.
• Wang et al. (2024)Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al.Large language models are not fair evaluators.InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 9440–9450, 2024.
• Xiang et al. (2024)Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, et al.Guardagent: Safeguard llm agents by a guard agent via knowledge-enabled reasoning.arXiv preprint arXiv:2406.09187, 2024.
• Yabusaki (2026)Hitoshi Yabusaki.Verifier-grounded runtime mediation for llm agent procedural compliance.In2026 IEEE International Conference on Software Services Engineering (SSE), pages 34–44. IEEE, 2026.
• Yao et al. (2022)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022.
• Yao et al. (2024)Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ\tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024.
• Zheng et al. (2023)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023.
• ZJUNLP (2026)ZJUNLP.JevLoop: Acta: A decision-centric runtime for agents.https://github.com/zjunlp/JevLoop, 2026.GitHub repository, accessed October 14, 2026.
• Zwerdling et al. (2025)Naama Zwerdling, David Boaz, Ella Rabinovich, Guy Uziel, David Amid, and Ateret Anaby Tavor.Towards enforcing company policy adherence in agentic workflows.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 595–606, 2025.

Appendix AAppendix

A.1Offline compilation of the questions and checks

This section expands Section3.3: how the compiler turns a policy into questions, rules, code, and a hand-off checklist. It includes the available prompt templates and representative outputs.

Inputs and outputs.

Claude Opus 4.6 makes a separate prompted call for each artifact-generation step; these are single-turn calls, though the client may retry transport failures. Steps 1, 3, 4, and the checklist call in step 6 allow up to 16,384 output tokens; the rule call allows 8,192. Telecom checklist generation also makes a separate prerequisite-extraction call with a 2,048-token limit. Step 5 is a local filter over Kev’s answers, not a Claude call. Inputs include the domain policy as the agent receives it, tool schemas, and, for code generation, one logged call and result per read tool to show the JSON shapes.Table6lists the steps. Steps 1–5 build the write gate; step 6 builds the hand-off checklist.

Table 6:The offline compilation workflow. “Logged” calls and hand-offs come from development runs. Telecom step 6 also includes a separate prerequisite-extraction call.In the prompts below, text in angle brackets (<…>) is filled by code; everything else is sent verbatim, except that line breaks inside a paragraph are removed here for typesetting.

Step 1: questions per write tool.

The prompt describes the reader the questions are for: a small model that sees the conversation, cannot compute, and answers yes/no with a probability. It asks for questions that each test one fact and together cover the policy’s conditions on the call, the provenance of every argument, whether the user wants the action, and the required steps before it. Two output fields matter at run time.bad_ifgives the answer that means a violation, so the gate knows each question’s polarity. Arguments are written as placeholders,{reservation_id}, which code fills with the proposed call’s values before Kev reads the question. The prompt also asks the compiler not to make the reviewer compute anything; arithmetic is left to step 3.

You are compiling a customer-service agent’s policy into atomic checks that an independent reviewer will answer before each database-changing tool call runs.The reviewer is a small reading model. It sees the whole conversation: the user’s messages, the agent’s messages, every tool call and its JSON result, and the policy. Dates in the conversation are annotated with their offset from the current time, and lists of options in tool results come with pre-computed rankings (lowest to highest price, etc.). It cannot do arithmetic itself and cannot see anything outside the conversation. It answers yes/no questions with a calibrated probability.For EACH write tool below, write 12 to 20 atomic yes/no questions such that, together, they cover every policy rule and every way the call could be wrong for this user:- conditions the policy puts on the action (eligibility, required preconditions, what must be checked first);- whether each argument value is right: grounded in a tool result, what the user actually asked for or explicitly chose (not just agreed to after the agent proposed it), and consistent with the policy’s formulas;- whether the action is wanted at all (the user asked for it; not something the user declined or did not request);- required steps before acting (records retrieved, explicit confirmation of the details).Each question must test ONE fact, be answerable from the conversation, and refer to the proposed call’s arguments as {arg_name} (e.g. {reservation_id}). Prefer facts read from tool results over the user’s claims. Do not ask the reviewer to compute anything.Return only JSON: {“<tool name>”: [{“id”: “<short_snake_case>”, “question”: “...”, “bad_if”: “yes” or “no”, “rule”: “<the policy rule it checks, short>”}, ...], ...}POLICY:<the domain policy, verbatim>WRITE TOOLS (OpenAI schemas):<the write tools’ OpenAI function schemas, as JSON>For airline this produced 86 questions over six write tools (12 to 21 per tool); for telecom, 75. Part of the compiled list forcancel_reservation:

{“id”: “reservation_id_correct”, “bad_if”: “no”,“question”: “Does {reservation_id} match a reservation ID associated with the user as confirmed by tool results?”}{“id”: “no_flown_segments”, “bad_if”: “no”,“question”: “Based on tool results, have zero flight segments in {reservation_id} already been flown (status ’flying’ or past departure)?”}{“id”: “cancellation_eligible_24hr”, “bad_if”: “no”,“question”: “Was the booking for {reservation_id} made within the last 24 hours according to tool results?”}{“id”: “at_least_one_cancel_condition_met”, “bad_if”: “no”,“question”: “Is at least one cancellation eligibility condition met: booked within 24hrs, airline cancelled, business cabin, or insurance-covered reason?”}{“id”: “user_requested_cancellation”, “bad_if”: “no”,“question”: “Did the user actually request to cancel this reservation (not just inquire about it)?”}{“id”: “user_confirmed_cancellation”, “bad_if”: “no”,“question”: “Did the agent list the cancellation details and did the user explicitly confirm (say yes) before this tool call?”}

Step 2: the decision rule.

A second call reads the policy and the step-1 questions (ids and text only) and writes, per tool, one Python boolean expression over the question ids that is true when the call is allowed. This is where alternatives are combined: the policy allows a cancellation if any one of four conditions holds, which a list of independent questions cannot express.

Below is a customer-service policy and, for each database-changing tool, a list of yes/no questions an independent reviewer answers about a proposed call (each with an id).For each tool, write the policy’s decision as ONE Python boolean expression over the question ids, True when the call is allowed and correct. Each id evaluates to True if the reviewer answered yes. Use `and`, `or`, `not` and parentheses. Combine alternative eligibility conditions with `or` (e.g. any one of several cancellation conditions is enough). Leave out questions that are not needed for the decision or that ask about things the policy does not require. Return only JSON: {“<tool name>”: “<expression>”, ...}POLICY:<the domain policy, verbatim>QUESTIONS:<the tool’s questions from the first compile, as JSON {tool: [{id, question}, ...]}>The compiled rule forcancel_reservationkeeps the summary eligibility question and leaves out the four single-condition ones:

user_id_obtained and reservation_id_correct and reservation_retrieved and no_flown_segments andcancellation_reason_obtained and at_least_one_cancel_condition_met and user_requested_cancellation anduser_confirmed_cancellationAt run time each id is true unless Kev answers a confidentno(P⁡(yes)≤0.05P(\text{yes})\leq 0.05), and the rule is evaluated with Python’sevaland no builtins. If it is false, the feedback lists the questions that failed.

Step 3: code for what the records decide.

Questions that need arithmetic or date comparisons (free baggage allowance, hours since booking, certificate amounts) are what Kev answers worst. A third call writes one Python function per write tool that reads the proposed arguments and every earlier tool result, computes the facts the policy depends on, and returns the violations the records make certain. The prompt restricts the code to the standard library; it is executed with a whitelist of builtins and imports, and an exception counts as no violation.

You are writing helper code for an independent reviewer that checks a customer-service agent’s database-changing tool calls against the policy below. The reviewer is a reading model: it cannot do arithmetic, compare dates, look up tables or compute minima. Your code does that for it.For EACH write tool listed, write a Python functiondef derive_<tool name>(args: dict, results: list) -> dict:- `args`: the proposed call’s arguments.- `results`: every earlier tool call in the conversation, oldest first, as dicts {“tool”: <tool name>, “args”: <its arguments>, “result”: <its parsed JSON result, or the raw string>}. Example results of each read tool are given below, so you know the JSON shapes.- Return a dict of COMPUTED FACTS the policy’s rules for this tool depend on, as short readable strings or numbers, e.g. allowances and totals the policy defines, the value an argument should have by the policy’s formula next to the value the call uses, time elapsed since a timestamp vs the policy’s current time, balances vs amounts, counts, which earlier records the call’s ids refer to and their relevant fields. Name each fact clearly (e.g. “expected_nonfree_baggages”, “hours_since_booking”).- Also return the key “violations”: a list of short strings, one per policy rule this call clearly breaks according to the RECORDS (tool results) and your computations, e.g. “nonfree_baggages should be 0 (gold, economy, 1 passenger: 3 free; total 2), call has 1”, or “reservation created 312 h before now; not within 24 h, not business, no insurance, no airline cancellation: not eligible for cancellation”. Only include a violation when the records make it certain; if a needed record is missing, or the rule depends on what the user said or wants, do not report a violation (a reviewer handles those). An empty list means no violation found.- Use only the standard library modules json, math, datetime (already imported; no other imports). Be defensive: if something is missing, skip that fact. Never raise.Return only the Python code, no explanations.POLICY:<the domain policy, verbatim>WRITE TOOLS:<the write tools’ OpenAI function schemas, as JSON>EXAMPLE RESULTS OF READ TOOLS (from logs, truncated):<one logged call and result per read tool, as JSON {tool: {args, result}}; each result cut to 1,500 characters>An excerpt of the compiled airline code forupdate_reservation_baggages(helper functions omitted):

free_per_pax = _free_bags_per_passenger(membership, cabin)if free_per_pax is not None:total_free = free_per_pax * num_passengersexpected_nonfree = max(0, new_total - total_free)if new_nonfree != expected_nonfree:violations.append(f“nonfree_baggages should be {expected_nonfree} “f”({membership}, {cabin}, {num_passengers} passenger(s): “f”{total_free} free; new total {new_total}), call has {new_nonfree}“)

Step 4: one repair round.

The step-3 code is run on the write calls in the logs of train-split tasks. Up to 16 calls on which it reported a violation, at most one per tool and first violation, are shown back to the compiler with the records the code read. No gold action or outcome is shown; the compiler judges each reported violation against the policy and the records and fixes the bugs it finds (for example, reading an error string such as“Error: User not found“as a user record, or using a reservation that a later tool result had superseded). On held-out airline tasks this round cut the code’s false alarms from 11.3% to 1.4% of correct write calls.

Below is Python code you wrote earlier that computes, for a customer-service agent’s proposed database-changing tool call, the facts the policy depends on and a list of “violations” that the records make certain.It was run on calls from real conversations. Here are calls where it reported violations, each with the call, the violations it reported, and the earlier tool calls and results it read (oldest first, truncated). For each one, check against the policy and the records whether the reported violation is real. Where it is not (a bug: e.g. it read an error message as a record, used a stale record that a later tool result superseded, misread a field, applied the wrong rule), fix the code. Keep every check that is right. Return the complete corrected code only.POLICY:<the domain policy, verbatim>CODE:```python<the compiled module from the previous step>```FLAGGED CALLS:<up to 16 logged calls (train-split tasks) on which the code reported a violation, at most one per tool and first violation; each given as the call, the violations it reported, and the last 8 earlier tool calls with their results (each result cut to 700 characters)>

Step 5: label-free question filter.

Kev-27B answers every compiled question on every write call in the development logs, with the same state it sees online (the policy, the conversation, every tool result with dates annotated and lists ranked, and the proposed call). A questionfireson a call when its answer is a confident violation. Any question that fires on more than 10% of its tool’s logged calls is dropped: real violations are rare, so a question that fires that often is asking something Kev cannot read from the conversation or the policy does not require. A dropped question is never reported and counts asyesin the rule. The thresholds (P≤0.05P\leq 0.05, 10%) were set once on airline and reused unchanged. This drops 17 of 86 Airline questions and 5 of 75 Telecom questions (175 logged Telecom write calls).

Step 6: the hand-off checklist.

The hand-off prompt asks for the problem types the policy troubleshoots and, for each, the steps the agent must try before handing off. Each step carries asettledquestion, the problems that require it (prerequisites included) and the agent tools that perform it. The prompt below describes the checklist-generation call.

You are compiling a customer-service agent’s policy into a checklist for an independent reviewer.The reviewer is called at the moment the agent wants to END its attempt to solve the user’s problem: hand the user off to a human agent, or tell the user nothing more can be done. The policy asks the agent to try every relevant resolution step first. The reviewer reads the conversation as the agent saw it (the user’s messages, the agent’s messages, the agent’s own tool calls and their results) and flags a REQUIRED step that is still OPEN.1. “problems”: the kinds of problem the policy has troubleshooting for, as {“<id>”: “<one-line description>”}.2. “steps”: every resolution / troubleshooting step the agent must go through (itself with a tool, or by guiding the user on their device) before handing off. Include steps that start by establishing a fact (e.g. asking the user something) when the policy’s later steps depend on it: the agent cannot skip them because the user did not volunteer the fact. For each step:- “id”: short snake_case id;- “step”: what the agent must do, one sentence, naming the tool or device action (and the question to ask, if any);- “problems”: the problem ids it is required for, INCLUDING problems for which the policy makes it a prerequisite (e.g. a step for problem A is also required for problem B if B’s troubleshooting says to fix A first);- “settled”: a yes/no question, YES when the conversation shows the step is settled: it was tried, OR the setting or condition it checks was verified to be fine, OR the agent established that the step does not apply (e.g. the user answered a question that rules it out). NO when the conversation never addressed it. A condition that was never discussed is NOT ruled out: if nobody asked (e.g. whether the user is traveling), the step is open. Do not put an “or <condition> does not hold” escape in the question unless it says “the conversation established that”;- “agent_tools”: the agent tools (names from the list below) that perform or verify the step, [] if none.Cover the policy’s own steps, in its order; skip identity verification and pure information requests. Reply with one JSON object: {“problems”: {...}, “steps”: [...]}.AGENT TOOLS: <the names of the agent’s tools, comma-separated>POLICY:<the domain policy, verbatim>

Online, Kev answers in one request which problem the user has (one of the compiled types orother), whether the user confirmed the problem is fixed, and every step’s settled question. A step is open if the probability of the problems that require it is at least 0.5 andP⁡(settled)≤0.5P(\text{settled})\leq 0.5.

To select checklist steps for dropping, Kev-27B answered the checklist questions at the first hand-off in each episode across six Telecom development runs (GPT-4.1 and Opus 4.6). We used the 74 train-split tasks (217 hand-offs, 36 in solved episodes): any step flagged on more than 5% of solved hand-offs was dropped, since a correct hand-off should pass. This removed seven steps: airplane mode, APN settings, SIM status, line suspension, overdue bills, contract end date, and resuming the line after payment. Both thresholds remained at 0.5. The 40 test-split tasks did not determine these seven drops. They were held out only for the filter; the checklist itself was revised after a results review (Step 6).

BFCL.

Nothing is compiled for BFCL multi-turn: it has no written policy, and its two checks ask the same fixed questions on every task (Section3.5).

A.2Judge size on identical hand-off states (offline)

We replayed the 609 telecom hand-off states from the Kev-27B runs through each judge, holding the checklist, thresholds, and dropped steps fixed. This isolates judge differences from changes in the agent, user simulator, or episode. A catch is a flag naming a required fix that remains undone; a false alarm names a fix the task does not need.

Table 7:Offline replay of the same 609 telecom hand-off states for each Kev size.Kev-9B closely matches Kev-27B in this replay. Kev-0.8B flags less, catching fewer required fixes but also raising fewer false alarms. Neither smaller judge shows an overall decision-quality advantage that would explain its higher online Telecom scores in Table5.

A.3Judge input length

Kev-0.8B and Kev-9B are validated up to 8,192 tokens; Kev-27B is validated up to 65,536(Palmer, 2026). In the Kev-27B runs, 41% of Airline write-check inputs exceed 8,192 tokens, compared with 4% of Telecom hand-off inputs. A write check includes the policy and tool results as well as the conversation, while a hand-off check reads the conversation and checklist.

A.4Judge size on labeled write calls (offline)

To separate ability from calibration, we replayed the same 260 labeled Airline write calls (101 bad, 159 good) through each judge using the deployed questions and dropped-question list. Calls were split at 8,192 input tokens (180 short, 80 long). The table reports ranking quality without a threshold, catches and false alarms at the deployed threshold, and results at each judge’s best threshold subject to Kev-27B’s false-alarm count. These best thresholds use the labels and are therefore an upper bound for the smaller judges.

Table 8:Offline comparison on the same 260 labeled Airline write calls.On long inputs, Kev-9B catches 2 of 32 bad calls at either threshold, versus 11 for Kev-27B; its AUROC is 0.61 versus 0.85. On short inputs, recalibrating Kev-9B from 0.05 to 0.10 raises catches from 8 to 16 (Kev-27B catches 21), without adding a false alarm. Kev-0.8B ranks worse even on short inputs (AUROC 0.63 versus 0.75). These results suggest that the size gap reflects both input length and calibration, while the smallest judge also has lower ranking quality.

A.5Token usage

Table9reports mean tokens per run for the main model and Kev in the runs of Table2. Onτ2\tau^{2}-bench, usage comes from run logs; agent drafts sent back for revision are not recorded, so their count is reported separately. On BFCL, Kev requests were rebuilt from logged conversations and counted with the model tokenizer. Requests that could not be rebuilt are omitted and counted in the table.

Table 9:Mean tokens per run, in thousands, for the runs in Table2. Agent usage is logged; Kev input is logged onτ2\tau^{2}-bench and rebuilt on BFCL. Drafts sent back and checks not rebuilt are counts per run.ReActJevActModelModelKevDraftsNotBenchmarkModelinoutinoutinsent backrebuiltτ2\tau^{2}-benchairline50 tasks×\times3GPT-4.158.40.960.10.99.00.390Opus 4.664.61.564.11.56.70.080gpt-oss-20b50.43.056.83.45.20.190gpt-oss-120b44.12.543.82.66.50.360Qwen3.8-27B79.88.280.38.56.40.030DeepSeek-V4-Pro64.01.863.21.89.50.110τ2\tau^{2}-benchtelecom114 tasks×\times3GPT-4.1130.61.2167.31.416.81.440Haiku 4.5138.82.1184.92.721.01.910Opus 4.6171.71.7201.41.812.70.610Opus 5.5178.12.2183.02.29.60.100gpt-oss-20b157.39.1167.99.422.11.300gpt-oss-120b117.74.2139.05.124.11.380Qwen3.8-27B143.68.0143.68.012.70.230DeepSeek-V4-Pro199.61.8207.31.815.80.630BFCLmulti-turn800 entries×\times3GPT-4.144.10.546.50.53.20.020.02Opus 4.674.01.474.91.44.40.110.19gpt-oss-20b49.72.249.92.23.50.030.04gpt-oss-120b49.51.650.61.73.70.040.08Qwen3.8-27B88.83.387.83.35.50.110.19DeepSeek-V4-Pro81.91.481.91.45.90.130.22Llama 3.3 70B111.30.6116.50.75.20.050.09Mistral Small 3.2 24B51.70.754.50.73.00.020.03Gemma 4 31B62.40.562.30.53.60.050.09

A.6Judge cost

The judge costs in Table4are estimates per episode, based on logged requests and tokens. API costs use public standard-tier prices as of October 14, 2026 (GPT-4.1: 2/8 and Haiku 4.5: 1/5 per million input/output tokens)(OpenAI, 2026;Anthropic, 2026); these are not invoiced costs. Missing API output-token counts are imputed from offline mean outputs.

For self-hosted judges, GPU time per request was measured offline on an A100-80GB (bf16) with four concurrent requests: 0.53 seconds for Kev-0.8B, 1.22 for Kev-9B, 2.72 for Kev-27B, 4.02 for Qwen3.8-27B, and 6.10 for Gemma-4-31B. Costs use these measurements with an assumed $2 per GPU-hour and full utilization. Estimates exclude the agent, user simulator, and one-time compilation. GPU cost comparisons depend on serving stack and utilization; open LLMs ran without a thinking phase.