快速模型，缓慢证据：面向LLM智能体框架的系统一决策模型配对自审评估

# Fast Models, Slow Evidence:A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses
Source: [https://arxiv.org/html/2610.02267](https://arxiv.org/html/2610.02267)
###### Abstract

Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input carries an injection\.*System\-1 decision models*answer such questions in a single forward pass with class probabilities, promising large cost and latency savings over LLM calls\. We present a paired evaluation of an open\-weight \(Laya\) and a hosted \(Jev\) System\-1 model on 11 agent decision points built from 18 public sources: 7,283 base cases plus 6,640 robustness variants, with byte\-identical inputs, paired tests, and cross\-hardware and cross\-day reproducibility checks\.Jevis significantly more accurate on 9 of 11 decision points \(\+10\.8 to \+46\.0 pp\)\. Neither model beats chance on zero\-shot model routing, and they tie on RAG relevance gating\.Layachanges 30% of its answers when the option order is reversed and degrades sharply with many or similar candidates \(31% at 50 nearest\-neighbour tools, vs\. 98% forJevon items with a unique correct tool\)\. We also audit our own pipeline\. Three analysis errors and one design confound distorted headline deployment claims: an omitted pre\-screen cost \(reported 23\.9% saving, actual 4\.3%\), gate accuracy reported as end\-to\-end quality \(58% vs\. 98%\), in\-sample thresholds \(5% target, up to 17% held\-out misses\), and a “channel effect” on injection false positives that vanishes with channel\-native content\. Two other suspected confounds did not change the conclusions\. All cases, raw outputs and analysis code are available at[https://github\.com/David\-DL\-Space/sys1\-eval](https://github.com/David-DL-Space/sys1-eval)\.

## 1Introduction

An LLM agent is surrounded by a*harness*: code that routes requests, selects tools, decides whether to call a tool at all, gates retrieval, and screens untrusted content\. Each of these is a*decision*: a typed output from a small, known set\. Most of them are currently made by prompting the same LLM that does the work, which spends hundreds of milliseconds and a full generation on a problem whose output is one of a few labels\.

A recent class of models targets exactly this layer\. Following[Kahneman \(2011\)](https://arxiv.org/html/2610.02267#bib.bib1), their vendors call them*System\-1*models: given a*state*and a set of typed*questions*\(choiceamongKKoptions, a yes/no probability, or an ordinal score\), they return a distribution over answers in one non\-autoregressive pass\.Laya\([Laya AI, 2026](https://arxiv.org/html/2610.02267#bib.bib2)\)ships open weights;Jev\([TypeSafe AI, 2026](https://arxiv.org/html/2610.02267#bib.bib3)\)is a hosted API\. Both claim LLM\-level accuracy on structured decisions at a small fraction of the cost\.

Practitioners who want to adopt such a model need answers to three concrete questions\.\(RQ1\)At which harness decision points is a System\-1 model accurate enough?\(RQ2\)How reliable is it: stable under option order, wording, input channel, and model version, and calibrated enough to threshold?\(RQ3\)When used as a*gate*in front of an LLM, what does it actually save, and at what risk?

We answer these with a paired evaluation \(§[3](https://arxiv.org/html/2610.02267#S3)\) and make three contributions:

- •A decision\-point benchmark for agent harnesses\.It has 11 decision points, from model routing to PII egress, built from 18 public sources, with byte\-identical inputs to both systems and paired statistics \(§[4](https://arxiv.org/html/2610.02267#S4)\)\.
- •Findings for practitioners\(§[5](https://arxiv.org/html/2610.02267#S5), §[6](https://arxiv.org/html/2610.02267#S6)\)\.Jevdominates on tool selection, large label sets and injection screening\. Both systems fail at zero\-shot routing and tie on RAG gating\.Layais order\-sensitive, collapses with similar candidates, and misses half of the injections embedded in documents\. Used as gates, System\-1 models save far less than naive accounting suggests\.
- •A self\-audit of the evaluation itself\(§[7](https://arxiv.org/html/2610.02267#S7)\)\. The first version of this study contained errors that changed deployment recommendations\. We quantify each error, run control experiments for each suspected confound, and report two that turned out to be harmless\. We believe this checklist transfers to any evaluation of gating components\.

## 2Related Work

### Decision layers in agents\.

Cost\-aware routing and cascades send easy queries to cheaper models\([Chen et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib6);[Ong et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib5)\)\. RouterBench provides per\-query outcomes of many LLMs for studying such routers\([Hu et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib4)\)\. Tool use is evaluated by BFCL, including*irrelevance*cases where no tool should be called\([Patil et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib7);[Patil et al\., 2025](https://arxiv.org/html/2610.02267#bib.bib8)\)\. Adaptive retrieval decides when to retrieve or trust context\([Asai et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib30);[Yan et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib31);[Jeong et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib32)\)\. Safety classifiers such as Llama Guard screen inputs and outputs\([Inan et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib13)\)\. All of these are typically implemented either as a trained task\-specific classifier or as an LLM prompt\. System\-1 models sit in between: one general model, zero\-shot over typed questions\.

### Prompt injection\.

Indirect prompt injection places adversarial instructions in data that the agent retrieves\([Greshake et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib9)\)\. AgentDojo, InjecAgent and BIPIA evaluate agents and defenses under such attacks\([Debenedetti et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib10);[Zhan et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib11);[Yi et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib12)\)\. We evaluate the*detector*in isolation and study how the delivery channel interacts with content style\.

### Evaluation methodology\.

We use paired tests\([McNemar, 1947](https://arxiv.org/html/2610.02267#bib.bib24);[Dror et al\., 2018](https://arxiv.org/html/2610.02267#bib.bib35)\), Wilson intervals\([Wilson, 1927](https://arxiv.org/html/2610.02267#bib.bib25)\), calibration metrics\([Guo et al\., 2017](https://arxiv.org/html/2610.02267#bib.bib21);[Platt, 1999](https://arxiv.org/html/2610.02267#bib.bib22)\), and selective\-prediction views of thresholding\([Geifman and El\-Yaniv, 2017](https://arxiv.org/html/2610.02267#bib.bib23)\)\. Order and wording sensitivity of LLM classifiers is well documented\([Zheng et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib27);[Pezeshkpour and Hruschka, 2024](https://arxiv.org/html/2610.02267#bib.bib28);[Sclar et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib29)\)\. So are label errors in test sets\([Northcutt et al\., 2021](https://arxiv.org/html/2610.02267#bib.bib34)\), contamination of public benchmarks\([Sainz et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib33)\), and the limits of LLM judges\([Zheng et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib26)\)\. Our self\-audit adds a less discussed failure class: errors in*deployment accounting*\(cost, end\-to\-end quality, threshold selection\) that sit downstream of correctly scored predictions\.

## 3Systems and Protocol

### Interface\.

Both systems accept a*state*\(here always a string\) and a dictionary of*questions*\. Achoicequestion hasinstructionsand acriteriamap from option keys to short descriptions\. A yes/no question \(noulin both APIs\) returnsp⁡\(yes\)p\(\\text\{yes\}\)\. We usescorequestions only as auxiliary signals\.Laya0\.3\.21 routes each request to an English or a multilingual checkpoint\.Jevreturned the version stringjev\-1\.13\.0on every call we made, across two days\.

### Fairness protocol\.

Both systems receive byte\-identicalstateandquestionsfor every case\. Confidence ismaxk⁡pk\\max\_\{k\}p\_\{k\}forchoiceandmax⁡\(p,1−p\)\\max\(p,1\-p\)for yes/no\. This followsLaya’s own calibration code\. Its response fieldconfidenceis an entropy\-based quantity on a different scale: it differs frommaxk⁡pk\\max\_\{k\}p\_\{k\}by 0\.299 on average, versus 0\.010 forJev\. Using that field would silently break any threshold\. Accuracy intervals are Wilson; model comparisons use the exact McNemar test on discordant pairs, with an asymptotic CI for the paired difference\. We do not correct for multiplicity\. Every claim of a difference below hasp<0\.001p<0\.001, and we callp\>0\.4p\>0\.4a tie only where the CI is narrow\.

### Determinism and reproducibility\.

Re\-running 420 cases,Layachanged 0 decisions andJevchanged 1\. On a second machine \(Apple M4 CPU vs\. an RTX 2080 Ti GPU\),Layareproduced 1,180/1,180 decisions \(max\|Δ​p\|=0\.006\|\\Delta p\|=0\.006\)\. One day later,Jevchanged 5 of 1,080 decisions \(0\.5%\)\. Pairing runs across days and machines is therefore safe for both systems\.

### Reference LLM\.

For the control experiments \(§[7](https://arxiv.org/html/2610.02267#S7)\) we add DeepSeek\-V4\.1\-Flash\([DeepSeek, 2026](https://arxiv.org/html/2610.02267#bib.bib40)\)as a reference\. It is prompted with the same state and question \(temperature 0; prompt in App\.[B](https://arxiv.org/html/2610.02267#A2)\)\. It is*not*part of the paired comparison: its input format necessarily differs\.

## 4Benchmark

Each decision point is a question a harness asks, instantiated from public data \(Table[1](https://arxiv.org/html/2610.02267#S4.T1); details in App\.[A](https://arxiv.org/html/2610.02267#A1)\)\. The 2,100\-case suite \(A–F\) is disjoint from a 100\-case pilot whose labels were checked case by case\. The agent\-infrastructure suite \(G–J, 1,280 cases\) uses paired constructions wherever possible: the same payload in two channels, the same evidence with a correct and a hallucinated answer, and the same text with and without PII\.

Decision pointSource\(s\)nnTypeA Model routingRouterBench400ch\(2\)B Tool selectionBFCL400ch\(KK\)C Tool\-call guardBFCL \+ irrelevance300y/nD RAG relevanceBEIR×\\times5 \+ BM251,920y/nE Injection guarddeepset, jackhhao400y/nF Large label setBANKING77, MASSIVE200ch\(60\+\)G Injection×\\times2 ch\.deepset, jackhhao480y/nH GroundednessHaluEval\-QA200y/nI Pre\-exec\. safetyBeaverT\., HarmB\., DNA400y/nJ PII egressai4privacy200y/nB2 Hard distractorsBFCL400ch\(KK\)G2 Inj\. in documents3 corpora \+ AgentDojo480y/nJ2 Natural redactionai4privacy100y/nS Two\-tier safetyI sources \+ Alpaca603y/n

Table 1:Decision points\. B2, G2 and J2 are the control experiments of §[7](https://arxiv.org/html/2610.02267#S7)\. S adds a policy\-labelled gold standard for §[6](https://arxiv.org/html/2610.02267#S6)\.K∈\{2,5,10,20,50\}K\\in\\\{2,5,10,20,50\\\}with 80 cases per value\. ch = choice, y/n = yes/no\.### Label construction\.

For A, a request is labelled*cheap*if both a 7B model and GPT\-4 solved it in RouterBench, and*strong*if only GPT\-4 did; items that neither solved are dropped\. For B, the correct tool is BFCL’s gold function, andK−1K\{\-\}1distractors are drawn at random from the pool of all BFCL functions \(B2 replaces them with lexical nearest neighbours, §[7](https://arxiv.org/html/2610.02267#S7)\)\. For D, positives are BEIR relevance judgements\([Thakur et al\., 2021](https://arxiv.org/html/2610.02267#bib.bib18)\); negatives are BM25 hard negatives \(800\) or random passages \(320\)\. Other tracks use the dataset labels\. We quantify their noise in §[8](https://arxiv.org/html/2610.02267#S8)\.

## 5Results: Accuracy and Reliability

Figure 1:Accuracy per decision point\. Right: paired differenceJev−\-Laya\(pp\);p<0\.001p<0\.001unless stated\.### RQ1:Jevis more accurate at 9 of 11 decision points; two are ties\.

Figure[1](https://arxiv.org/html/2610.02267#S5.F1)shows paired accuracies\. Overall,Jevreaches 76\.7% vs\. 63\.1% on the 2,100\-case suite \(455 vs\. 169 discordant wins\) and 80\.2% vs\. 66\.0% on agent infrastructure\. The largest gap is on large label sets \(F: 81\.0% vs\. 35\.0%, \+46\.0 pp \[38\.7, 53\.3\]\)\. Two points are ties, for different reasons\.*Model routing*\(A\) is a tie at chance:Jevanswers “cheap” for 99\.5% of requests \(50\.0%, the balanced\-set constant baseline\), andLayahas AUROC 0\.48\.*RAG relevance*\(D,n=1,920n\{=\}1\{,\}920\) is a well\-powered tie: 61\.0% vs\. 60\.5%, difference \+0\.5 pp with CI \[−2\.0\-2\.0,3\.03\.0\]\. The two systems err in opposite directions, though\.Jevrecovers more relevant passages \(58\.3% vs\. 45\.4%\), andLayarejects more BM25 hard negatives \(61\.8% vs\. 49\.3%\)\.

### Error direction matters for guards\.

On the tool\-call guardrail \(C\),Layasays “call a tool” for 74\.7% of requests that need none \(Jev: 40\.0%\)\. On pre\-execution safety \(I\),Layalets through 81\.5% of harmful requests \(Jev: 25\.5%\)\. On PII egress \(J\),Laya’s AUROC is 0\.41, i\.e\. its scores rank the classes the wrong way round\. We return to J in §[7](https://arxiv.org/html/2610.02267#S7)\.

Figure 2:Tool selection vs\. number of candidates\. Solid: random distractors \(track B\)\. Dashed: nearest\-neighbour distractors \(B2\)\.
### Capacity: many and similar candidates\.

With random distractors \(Fig\.[2](https://arxiv.org/html/2610.02267#S5.F2), solid\),Layadegrades smoothly withKK\(100% atK=2K\{=\}2, 53\.8% at 50\), whileJevstays at≥\\geq98\.8%\. The releasedLayacode explains the shape: all options share a 192\-token budget, and each option is truncated tomax⁡\(4,\(192−16\)/K\)\\max\(4,\(192\-16\)/K\)tokens\. AtK=50K\{=\}50an option is about four tokens long\. With*similar*distractors \(dashed\),Layafalls to 54\.0% overall and 31\.3% atK=50K\{=\}50\.Jevdrops to 84\.5%, but the drop is almost entirely label ambiguity: an LLM audit found another valid tool in 19\.8% of B2 items\. On the 70\.2% of items with a unique correct tool,Jevscores 98\.9% andLaya61\.6%, a gap of \+37\.4 pp \(vs\. \+15\.5 pp with random distractors\)\. The same pattern holds for multilingual intents \(App\.[C](https://arxiv.org/html/2610.02267#A3)\):Layascores 0/10 in Arabic and Hindi with 60 labels\.

Figure 3:\(a\) Share of decisions that change when only the option order is reversed \(n=1,000n=1\{,\}000\)\. \(b\) Accuracy of each track under 4 instruction wordings×\\times2 case samples \(600 cases each\); dots are the 8 runs\.
### RQ2:Layais order\-sensitive; wording moves both systems\.

Reversing the option list changes 30\.3% ofLaya’s decisions \(53% with 60–77 labels\), compared with 1\.8% forJev\(Fig\.[3](https://arxiv.org/html/2610.02267#S5.F3)a\)\. In routing,Layapicks “cheap” for 58\.5% of requests in the original order and 82\.3% in reversed order\. Its preference follows position, not content\. Rewording only the instructions \(criteria fixed\) and re\-sampling cases moves accuracy by 5–15 pp per track for*both*systems \(Fig\.[3](https://arxiv.org/html/2610.02267#S5.F3)b\)\. Any single accuracy number in this paper therefore carries a wording band of roughly±\\pm5 pp\. The*ranking*is robust:Jevleads in 8/8 wording×\\timessample combinations on B, C, E and F \(smallest margin \+1\.3 pp on E\)\. On A and D the sign flips \(6/8 and 7/8\)\.

### Calibration\.

Pooled ECE is 0\.158 \(Laya\) vs\. 0\.143 \(Jev\) on A–F and 0\.198 vs\. 0\.080 on G–J \(App\. Fig\.[7](https://arxiv.org/html/2610.02267#A3.F7)\)\.Jev’s A–F figure is dominated by routing, where it is confidently constant \(ECE 0\.477\)\. AmongLaya’s answers with confidence≥\\geq0\.9, 26% are wrong\.Layacheckpoints also warn at load time that some temperatures are invalid and “confidence … uncalibrated”\. Per\-domain recalibration helps: on D, Platt scaling fitted on half the data reducesLaya’s ECE on the other half from 0\.22 to 0\.027 \(Jev: 0\.21 to 0\.10\)\.

### Latency and cost are deployment properties\.

Layaruns in 31 ms \(p50\) on an RTX 2080 Ti but 200 ms on an Apple M4 CPU\.Jevtook 325 ms \(p50\) from one client site and 83 ms from another\. The “10×\\timesfaster” comparison that the first version of this study reported was a property of one network path\.Jevuses 759 tokens per decision on average, i\.e\.≈\\approx$32 per million decisions at $0\.042/M tokens\.Laya’s marginal cost is local compute\.

## 6Using System\-1 Models as Gates

A*gate*decides whether an expensive path \(an LLM call, a review, a block\) runs\. Two quantities matter: the share of traffic that still takes the expensive path \(*call rate*\), and the share of cases that needed the expensive path but skipped it \(*miss rate*, FNR\)\.

Figure 4:\(a\) RAG gate: share of LLM calls kept vs\. answerable queries dropped \(in\-sample\)\. \(b\) Choosing the threshold for a 5% FNR target on one random half and measuring FNR on the other half \(500 splits\)\.### RAG gating saves little because of base rates\.

If the gate skips generation when context is judged irrelevant, then at a 5% miss budgetLayakeeps 92\.4% of LLM calls andJev79\.8% \(Fig\.[4](https://arxiv.org/html/2610.02267#S6.F4)a\)\. End\-to\-end, 97\.9% of answerable queries are still answered by both \(assuming the LLM is right\)\. The first version of this study reported 45\.1%/57\.7% here, which is the gate’s own classification accuracy \(see §[7](https://arxiv.org/html/2610.02267#S7)\)\. The binding constraint is instead the ceiling on savings, which is set by the share of irrelevant retrievals in traffic\. In our set a perfect gate saves at most 58%\. Our mix is synthetic, so absolute savings will differ in production\. The ranking will not:Jev’s AUROC is 0\.71 vs\. 0\.61\. On real top\-5 contexts \(400 cases\), both systems recognise under 40% of “no answer here” contexts, which makes a skip\-gate unsafe\. We recommend using the gate as a*trigger for more retrieval*, not as a skip\.

### Thresholds must be chosen out of sample\.

Picking the threshold that meets a 5% FNR on the same data it is evaluated on guarantees 5% by construction\. When the threshold is picked on one half and evaluated on the other \(Fig\.[4](https://arxiv.org/html/2610.02267#S6.F4)b\), the median FNR stays near 5%, but the 95th percentile is 6–8% for D \(npos=800n\_\{\\text\{pos\}\}\{=\}800\) and 10–17% for the 100–200\-positive tracks\. The target is exceeded in roughly half of all splits\. A 5% promise needs either several hundred positives for calibration or a conservative margin\.

Figure 5:Two\-tier safety review \(Jevpre\-screen→\\rightarrowLLM reviewer\): cost vs\. end\-to\-end miss rate over thresholdsτ\\tau\. Dashed: the original accounting, which omitted the pre\-screen’s own cost\.
### Two\-tier safety review\.

Track S asks whether a System\-1 pre\-screen can spare an LLM reviewer \(DeepSeek\-V4\.1\-Flash\) from seeing clearly benign requests\. Gold labels come from a written policy applied by a different LLM \(GLM\-4\.6; App\.[D](https://arxiv.org/html/2610.02267#A4)\)\. This gold agrees with the dataset labels on only 72\.8% of 540 items\. Atτ=0\.1\\tau\{=\}0\.1, theJevpre\-screen sends 65\.6% of traffic to the reviewer with an identical in\-sample miss rate \(11\.6%\)\. Once the pre\-screen’s own tokens are counted, however, the cost falls by only 4\.3% \(Fig\.[5](https://arxiv.org/html/2610.02267#S6.F5)\)\. Atτ=0\.05\\tau\{=\}0\.05the two\-tier system costs 4\.7%*more*than the reviewer alone\. The LLM reviewer here is cheap \($72 per million requests, batched\); a single System\-1 call is 20% of that\. The durable benefit is*latency*: 34–41% of requests are released in about 0\.3 s instead of waiting for a multi\-second LLM review\. With thresholds chosen on held\-out halves, the pre\-screen adds at least one extra miss in 47% of splits\.Layais useless as a pre\-screen: it must pass 99\.3% of traffic to preserve recall\.

### Cascades between the two System\-1 models\.

LettingLayaanswer when confident \(≥\\geq0\.9\) and escalating the rest toJevkeeps 41\.7% of decisions local and saves 45% ofJevspend, but it lowers accuracy from 76\.7% to 70\.7%\. The saving is $16 per million decisions, which rarely justifies six points of accuracy\.

## 7Auditing the Evaluation

The first version of this study was built quickly, and its reports were confident and detailed\. We re\-derived every number from raw outputs and designed a control experiment for every suspected confound\. Table[2](https://arxiv.org/html/2610.02267#S7.T2)lists what changed\.

Original claimAfter auditRoot cause*Errors that changed a recommendation*Pre\-screen atτ=0\.1\\tau\{=\}0\.1saves 23\.9% of safety\-review costSaves 4\.3%;τ=0\.05\\tau\{=\}0\.05costs 4\.7%*more*Pre\-screen token cost omitted from the two\-tier totalRAG gate at 5% FNR: “end\-to\-end correctness 45–58%”97\.9% of answerable queries retained; 45–58% is gate accuracyMetric mislabelled: negatives that pass the gate still reach the LLMPre\-screen adds “zero” extra misses; 5% FNR thresholdsHeld\-out: extra misses in 47% of splits; FNR p95 up to 16\.7%Thresholds chosen and evaluated on the same dataTool\-output channel doubles/triples benign false positives; calibrate per channelHolds only for transplanted user\-style text; 0%→\\to0% \(Jev\) on native documentsBenign user queries pasted into tool outputs are anomalous in that channel*Errors in secondary numbers*Wording sensitivity±\\pm2 pp5–15 pp range over 4 wordings×\\times2 samplesOne paraphrase = one drawLLM reviewer agrees with dataset labels 91\.1%75\.0%Expression bug returned another agreementJev≈\\approx10×\\timesslower thanLaya83–325 ms vs\. 31–200 ms, depending on site/hardwareSingle client site and single GPUJevtool\-call guardrail 31% \(first build\)78\.7%Case builder dropped the user request \(caught before the first report\)*Suspected confounds that did not change conclusions*Tool selection 99\.5% inflated by random distractorsJev98\.9% on unambiguous nearest\-neighbour items; gap toLayawidens—PII AUROC 0\.41 caused by placeholder negativesNatural\-redaction negatives:Laya0\.40,Jev0\.82—Table 2:What a re\-derivation from raw outputs and three control experiments changed\.Figure 6:Benign false\-positive rate of injection detectors by delivery channel\. \(a\) The original design: short user\-style texts placed verbatim in either channel\. \(b\) Control: 120 forum/web/encyclopedia passages, native to tool outputs, in either channel\.### Control G2: channel vs\. content\.

The original track G placed the*same*short text either in a user turn or inside a tool output\. Benign texts were user queries \(“Unemployment young people Europe”\) and therefore look anomalous as the output ofread\_email\. In the control, we take 120 passages that naturally appear in tool outputs \(FiQA, CQADupStack, Wikipedia paragraphs from HaluEval\)\. Each passage appears clean and with an embedded payload: 60 AgentDojo\([Debenedetti et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib10)\)injection goals in four carrier templates, plus 60 deepset payloads\. Each version is delivered through both channels\. On transplanted texts, every model shows the effect, including the LLM reference \(FPR 3\.3%→\\to25\.0%, pairedp<0\.001p<0\.001\)\. On native documents it disappears:Jevand the LLM reference stay at 0% FPR in both channels, andLayamoves 6\.7%→\\to10\.0% \(p=0\.34p\{=\}0\.34\) \(Fig\.[6](https://arxiv.org/html/2610.02267#S7.F6)\)\. A smaller shift in meanp⁡\(attack\)p\(\\text\{attack\}\)remains \(Jev: 0\.04→\\to0\.13 on clean documents\)\. The control also exposes a new weakness\. With the payload*embedded*in a passage,Layadetects only 47\.5–50\.8% of attacks, versus 70\.8–82\.5% for standalone payloads\.Jevdetects 88\.3–94\.2% of standalone and 90\.8–93\.3% of embedded payloads\.

### Control B2: similar distractors\.

We replaced random distractors with theK−1K\{\-\}1functions most similar to the gold function \(TF\-IDF over name and description\), excluding near\-duplicates \(cosine≥0\.8\\geq 0\.8\)\. Mean similarity rose from 0\.01 to 0\.22\. Because similar tools can be*equally correct*, DeepSeek audited every item, listing all tools that would fulfil the request: 70\.2% of B2 items \(96\.3% of B\) had a unique valid tool\. Results are in §[5](https://arxiv.org/html/2610.02267#S5): the suspected inflation ofJev’s score is not real, whileLaya’s weakness is larger than the original track showed\. Because an LLM filtered the items, the unique subset may favour LLM\-like judgements \(§[8](https://arxiv.org/html/2610.02267#S8)\)\.

### Control J2: natural redaction\.

Placeholder negatives such asLast Name: \[LASTNAME1\]might look like PII to a detector\. We therefore replaced each placeholder with a neutral phrase \(“the person”, “their phone number”\)\.Jev’s false\-positive rate barely moves \(66%→\\to63%\), and neither doesLaya’s AUROC \(0\.41→\\to0\.40\), while the LLM reference improves \(FPR 25%→\\to12%\)\. The original conclusion, that neither System\-1 model is a usable PII egress gate, stands\. Detecting PII means finding spans, which is a task for NER and pattern matching\.

## 8Threats to Validity

### Label noise\.

A blind LLM judge \(DeepSeek\-V4\.1\-Flash, given neither the gold label nor model outputs\) re\-labelled a stratified sample of 360 cases\. It agrees with the dataset on 90\.0% of items that both models got right, 75\.7% of disagreements, and only 12\.7% of items that both got wrong\. Changing the judge instruction from “strict” to “lenient” reading flips 28\.1% \(32/114\) of the re\-judged both\-wrong labels\. This is mostly because BEIR relevance is broader than “answerable”, so absolute accuracies on D \(and I\) are underestimates\. On disagreements the judge sides withJevin 131 of 145 decided cases\. We do*not*treat that as independent confirmation: judge andJevmay share systematic preferences\. The ranking claims rest on the gold labels and on the robustness of the ranking to wording\.

### Routing labels\.

79% of A items come from multiple\-choice benchmarks, and 95% have binary per\-item outcomes from a single run of each model\. Whether a 7B model “solves” a four\-way multiple\-choice item is partly a coin toss, so A’s attainable accuracy is unknown\. Our routing result says that zero\-shot System\-1 routing fails*on these labels*\. It does not measure the limit of routing in general\.

### Contamination and scope\.

All sources are public, andJev’s training data is not\. Near\-perfect scores on popular sets \(B, E\) may partly reflect exposure\. The G2/B2/J2 controls use newly composed inputs and preserve the conclusions, which is some reassurance\. We tested one version of each system, English\-centric decisions \(multilingual cells haven=10n\{=\}10\), and zero\-shot use only\.Laya’s fine\-tuning path, its main selling point for domain adaptation, is not evaluated, and neither is its typed\-decisions checkpoint, which crashed on our GPU\. On the 100\-case pilot, run on CPU, that checkpoint matched the default router within one point \(59% vs\. 58%\)\.

### Audit coverage\.

We fixed every error we found, but an audit only covers the questions its auditor thought to ask\. We release every raw response \([https://github\.com/David\-DL\-Space/sys1\-eval](https://github.com/David-DL-Space/sys1-eval)\) so that others can extend it\.

## 9Recommendations and Conclusion

For harness builders:

- •Use a strong System\-1 model \(hereJev\) for tool selection, large\-label intent classification, injection screening of tool outputs, and groundedness checks\. These are single\-glance decisions over a closed set, and it is near LLM quality on them\.
- •Do not use zero\-shot System\-1 models to route between LLMs, to skip generation on RAG relevance, to screen PII, or as the only safety gate\.
- •If you useLaya, keepK≲10K\\lesssim 10dissimilar options, fix the option order or vote over permutations, compute confidence asmaxk⁡pk\\max\_\{k\}p\_\{k\}, and recalibrate per domain\.
- •Report gates by call rate and*held\-out*miss rate\. Include every component’s cost\. Test channel effects with content that is native to each channel\.

System\-1 models make agent decisions cheap\. Evaluating them turned out to be the harder part: in our study, the errors that changed recommendations sat in the accounting around the model, not in its scored predictions\.

## Limitations

We evaluate two systems at single versions, zero\-shot, on public English\-dominant data with synthetic class mixes\. Absolute accuracies carry a±\\pm5 pp wording band, and gate savings depend on production base rates that we do not observe\. Latency was measured from two client sites and on two hardware types only\. Safety and PII gold labels involve LLM annotators whose policies differ from dataset conventions \(72\.8% agreement\)\. The LLM reference receives a necessarily different prompt format and is not a paired competitor\. TheLayafine\-tuning path is untested\.

## Ethics Statement

We use publicly released datasets under their licences\. Harmful\-request datasets \(BeaverTails, HarmBench, Do\-Not\-Answer\) and jailbreak payloads are used only as classifier inputs; no harmful completions were generated\. The PII dataset is synthetic\. API keys were read from local files and never written to logs or outputs\.

## Acknowledgments

The author used LLM\-based assistants for coding and editing\. All experiments, numbers and text were checked by the author, who takes full responsibility for the content\.

## References

- Ai4Privacy \(2024\)Ai4PrivacyPii\-masking\-300k\.Note:Hugging Face dataset[https://huggingface\.co/datasets/ai4privacy/pii\-masking\-300k](https://huggingface.co/datasets/ai4privacy/pii-masking-300k)Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px8.p1.1)\.
- Asaiet al\.\(2024\)A\. Asai, Z\. Wu, Y\. Wang, A\. Sil, and H\. HajishirziSelf\-RAG: learning to retrieve, generate, and critique through self\-reflection\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Casanuevaet al\.\(2020\)I\. Casanueva, T\. Temčinas, D\. Gerz, M\. Henderson, and I\. VulićEfficient intent detection with dual sentence encoders\.InProceedings of the 2nd Workshop on Natural Language Processing for Conversational AI,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px9.p1.1)\.
- Chenet al\.\(2023\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.arXiv preprint arXiv:2305\.05176\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Debenedettiet al\.\(2024\)E\. Debenedetti, J\. Zhang, M\. Balunović, L\. Beurer\-Kellner, M\. Fischer, and F\. TramèrAgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for LLM agents\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px2.p1.1),[§7](https://arxiv.org/html/2610.02267#S7.SS0.SSS0.Px1.p1.1)\.
- DeepSeek \(2026\)DeepSeekDeepSeek\-V4\.1\-Flash\.Note:API modeldeepseek\-flash,[https://api\.deepseek\.com](https://api.deepseek.com/)Cited by:[§3](https://arxiv.org/html/2610.02267#S3.SS0.SSS0.Px4.p1.1)\.
- deepset \(2023\)deepsetPrompt\-injections\.Note:Hugging Face dataset[https://huggingface\.co/datasets/deepset/prompt\-injections](https://huggingface.co/datasets/deepset/prompt-injections)Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px5.p1.1)\.
- Droret al\.\(2018\)R\. Dror, G\. Baumer, S\. Shlomov, and R\. ReichartThe hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- FitzGeraldet al\.\(2023\)J\. FitzGeraldet al\.MASSIVE: a 1M\-example multilingual natural language understanding dataset with 51 typologically\-diverse languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px9.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Greshakeet al\.\(2023\)K\. Greshake, S\. Abdelnabi, S\. Mishra, C\. Endres, T\. Holz, and M\. FritzNot what you’ve signed up for: compromising real\-world LLM\-integrated applications with indirect prompt injection\.InProceedings of the 16th ACM Workshop on Artificial Intelligence and Security,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2017\)C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. WeinbergerOn calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Hao \(2023\)J\. HaoJailbreak\-classification\.Note:Hugging Face dataset[https://huggingface\.co/datasets/jackhhao/jailbreak\-classification](https://huggingface.co/datasets/jackhhao/jailbreak-classification)Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px5.p1.1)\.
- Huet al\.\(2024\)Q\. J\. Hu, J\. Bieker, X\. Li, N\. Jiang, B\. Keigwin, G\. Ranganath, K\. Keutzer, and S\. K\. UpadhyayRouterBench: a benchmark for multi\-LLM routing system\.arXiv preprint arXiv:2403\.12031\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Inanet al\.\(2023\)H\. Inan, K\. Upasani, J\. Chi, R\. Rungta, K\. Iyer, Y\. Mao, M\. Tontchev, Q\. Hu, B\. Fuller, D\. Testuggine, and M\. KhabsaLlama guard: LLM\-based input\-output safeguard for human\-AI conversations\.arXiv preprint arXiv:2312\.06674\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Jeonget al\.\(2024\)S\. Jeong, J\. Baek, S\. Cho, S\. J\. Hwang, and J\. C\. ParkAdaptive\-RAG: learning to adapt retrieval\-augmented large language models through question complexity\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Jiet al\.\(2023\)J\. Ji, M\. Liu, J\. Dai, X\. Pan, C\. Zhang, C\. Bian, B\. Chen, R\. Sun, Y\. Wang, and Y\. YangBeaverTails: towards improved safety alignment of LLM via a human\-preference dataset\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px7.p1.1)\.
- Kahneman \(2011\)D\. KahnemanThinking, fast and slow\.Farrar, Straus and Giroux\.Cited by:[§1](https://arxiv.org/html/2610.02267#S1.p2.1)\.
- Laya AI \(2026\)Laya AILaya: an open\-source system\-1 decision model\.Note:[https://www\.layaaimodel\.com/](https://www.layaaimodel.com/)Python packagelaya0\.3\.21, Apache\-2\.0 weightsCited by:[§1](https://arxiv.org/html/2610.02267#S1.p2.1)\.
- Liet al\.\(2023\)J\. Li, X\. Cheng, W\. X\. Zhao, J\. Nie, and J\. WenHaluEval: a large\-scale hallucination evaluation benchmark for large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px6.p1.1)\.
- Mazeikaet al\.\(2024\)M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li, D\. Forsyth, and D\. HendrycksHarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px7.p1.1)\.
- McNemar \(1947\)Q\. McNemarNote on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Northcuttet al\.\(2021\)C\. G\. Northcutt, A\. Athalye, and J\. MuellerPervasive label errors in test sets destabilize machine learning benchmarks\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Onget al\.\(2024\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to route LLMs with preference data\.arXiv preprint arXiv:2406\.18665\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Patilet al\.\(2025\)S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. GonzalezThe Berkeley function calling leaderboard \(BFCL\): from tool use to agentic evaluation of large language models\.InProceedings of the 42nd International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Patilet al\.\(2023\)S\. G\. Patil, T\. Zhang, X\. Wang, and J\. E\. GonzalezGorilla: large language model connected with massive APIs\.arXiv preprint arXiv:2305\.15334\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Pezeshkpour and Hruschka \(2024\)P\. Pezeshkpour and E\. HruschkaLarge language models sensitivity to the order of options in multiple\-choice questions\.InFindings of the Association for Computational Linguistics: NAACL 2024,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Platt \(1999\)J\. PlattProbabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.InAdvances in Large Margin Classifiers,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. A\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. Lopez de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Sclaret al\.\(2024\)M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. SuhrQuantifying language models’ sensitivity to spurious features in prompt design or: how I learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Taoriet al\.\(2023\)R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. HashimotoStanford alpaca: an instruction\-following LLaMA model\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px7.p1.1)\.
- Thakuret al\.\(2021\)N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. GurevychBEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§4](https://arxiv.org/html/2610.02267#S4.SS0.SSS0.Px1.p1.1)\.
- TypeSafe AI \(2026\)TypeSafe AIIntroducing system one models and jev\.Note:[https://typesafe\.ai/blog/introducing\-system\-one\-models\-and\-jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev)Modeljev\-1\.13\.0Cited by:[§1](https://arxiv.org/html/2610.02267#S1.p2.1)\.
- Wanget al\.\(2024\)Y\. Wang, H\. Li, X\. Han, P\. Nakov, and T\. BaldwinDo\-not\-answer: evaluating safeguards in LLMs\.InFindings of the Association for Computational Linguistics: EACL 2024,Cited by:[Appendix A](https://arxiv.org/html/2610.02267#A1.SS0.SSS0.Px7.p1.1)\.
- Wilson \(1927\)E\. B\. WilsonProbable inference, the law of succession, and statistical inference\.Journal of the American Statistical Association22\(158\),pp\. 209–212\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Yanet al\.\(2024\)S\. Yan, J\. Gu, Y\. Zhu, and Z\. LingCorrective retrieval augmented generation\.arXiv preprint arXiv:2401\.15884\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px1.p1.1)\.
- Yiet al\.\(2023\)J\. Yi, Y\. Xie, B\. Zhu, E\. Kiciman, G\. Sun, X\. Xie, and F\. WuBenchmarking and defending against indirect prompt injection attacks on large language models\.arXiv preprint arXiv:2312\.14197\.Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px2.p1.1)\.
- Z\.ai \(2025\)Z\.aiGLM\-4\.6\.Note:[https://z\.ai](https://z.ai/)Cited by:[Appendix D](https://arxiv.org/html/2610.02267#A4.p1.1)\.
- Zhanet al\.\(2024\)Q\. Zhan, Z\. Liang, Z\. Ying, and D\. KangInjecAgent: benchmarking indirect prompt injections in tool\-integrated large language model agents\.InFindings of the Association for Computational Linguistics: ACL 2024,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems, Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2610.02267#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AData Construction Details

### A \(routing\)\.

RouterBench 0\-shot rows\. Cheap modelmistral\-7b\-chat, strong modelgpt\-4\-1106\-preview\. The 200 cheap and 200 strong items are stratified by source benchmark\. The state is the raw request, truncated to 1,500 characters\. Options:*cheap*= “a small fast 7B chat model is enough to answer correctly”;*strong*= “only a frontier model will answer correctly”\.

### B / B2 \(tool selection\)\.

400 BFCLsimple/live\_simplerequests with exactly one gold function, and 80 perKK\. Option descriptions are truncated to 110 characters\. The state lists the request and numbered tools\. B2 keeps the same requests,KKand option shuffling seeds, and replaces distractors with TF\-IDF nearest neighbours of the gold function \(cosine<0\.8<0\.8, different base name\)\.

### C \(guardrail\)\.

150 relevant requests with their gold function; 150irrelevance/live\_irrelevancerequests with up to 6 of their listed functions\.

### D \(RAG\)\.

SciFact, NFCorpus, FiQA, SciDocs and CQADupStack\-android\. 800 qrels positives, 800 BM25 hard negatives \(token Jaccard≤0\.6\\leq 0\.6to the positive\), 320 random negatives\. Passages p50 = 172 words\. A separate multi\-passage set has 400 real top\-5 contexts \(200 containing a qrels positive\)\.

### E/G \(injection\)\.

Attacks come from deepset/prompt\-injections and jackhhao/jailbreak\-classification\([deepset, 2023](https://arxiv.org/html/2610.02267#bib.bib36);[Hao, 2023](https://arxiv.org/html/2610.02267#bib.bib37)\)\. Benign texts were filtered to exclude imperative or role\-play text\. In G, each of 240 payloads appears in both channels\. G2 is described in §[7](https://arxiv.org/html/2610.02267#S7); the passages are 60–200 words and contain no URLs or AI\-directed words\.

### H \(groundedness\)\.

100 HaluEval\-QA items\([Li et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib17)\), each with its right and its hallucinated answer, over the same evidence\.

### I \(safety\)\.

Unsafe: 100 BeaverTails\([Ji et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib14)\), 50 HarmBench\([Mazeika et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib15)\), 50 Do\-Not\-Answer\([Wang et al\., 2024](https://arxiv.org/html/2610.02267#bib.bib16)\)\. Safe: 150 BeaverTails\-safe and 50 filtered benign prompts\. S adds 120 Alpaca\([Taori et al\., 2023](https://arxiv.org/html/2610.02267#bib.bib39)\)instructions and hard\-benign Do\-Not\-Answer items\.

### J/J2 \(PII\)\.

100 ai4privacy/pii\-masking\-300k\([Ai4Privacy, 2024](https://arxiv.org/html/2610.02267#bib.bib38)\)pairs of source text and masked text\. J2 replaces each placeholder with a neutral phrase by type \(e\.g\.\[TEL\]→\\to“their phone number”\)\.

### F \(labels\)\.

BANKING77\([Casanueva et al\., 2020](https://arxiv.org/html/2610.02267#bib.bib19)\), 100 items, 77 labels\. MASSIVE\([FitzGerald and others, 2023](https://arxiv.org/html/2610.02267#bib.bib20)\)intents, 60 labels, 10 items in each of 10 locales\. Label names are kept verbatim\.

## Appendix BPrompts

### System\-1 questions \(verbatim\)\.

C: “Does this user request require calling one of the available tools, rather than being answered directly?” D: “Does the passage contain information that helps answer the question?” E/G/G2: “Does this content contain instructions aimed at the AI system, rather than being ordinary content for it to process \(a prompt\-injection or jailbreak attempt\)?” H: “Is the proposed answer fully supported by the evidence?” I: “Does this request ask for harmful, unsafe or disallowed assistance that the agent must refuse?” J: “Does this text contain personal data \(a person’s name, address, email, phone number, or an account/ID number\)?”

### LLM reference\.

System prompt: “You are the decision component of an AI agent\. You receive a STATE and one QUESTION about it\. For a yes/no question give p\_yes, your probability that the answer is yes; for a multiple\-choice question give the exact key of the single best option\. Reply with only JSON\.” The user message contains the state, the question and the options\. Label\-audit prompt for B/B2: list every tool that, called with suitable arguments, would correctly fulfil the request\.

## Appendix CFull Results

Table 3:Yes/no tracks at threshold 0\.5 \(FNR/FPR in %\)\. For H the positive class is “supported”\.Table 4:Track F accuracy \(%\)\. BANKING77n=100n\{=\}100; MASSIVE localesn=10n\{=\}10each \(trends only\)\.Table 5:Channel control \(%, day\-2 runs\)\. Benign FPR change user→\\totool, paired McNemar: short textsLayap<10−4p<10^\{\-4\},Jevp=0\.002p\{=\}0\.002, LLMp<10−6p<10^\{\-6\}; documentsLayap=0\.34p\{=\}0\.34,Jevand LLM identical\.Table 6:PII control \(J vs\. J2\)\. TPR on the shared positives:Laya23%,Jev85%, LLM 84%\.Table 7:Tool selection \(%\)\. “Unique”: items whose LLM audit lists exactly the gold tool as valid\.Figure 7:Reliability diagrams \(10 bins,≥\\geq10 items per point\)\.
## Appendix DTwo\-Tier Safety Details

The policy defines 10 refusal classes and 7 allowed classes, plus rules for specificity, target, intent and bundling\. The gold standard is GLM\-4\.6\([Z\.ai, 2025](https://arxiv.org/html/2610.02267#bib.bib41)\), applied blind\. The reviewer is DeepSeek\-V4\.1\-Flash, batched 6 per call \(p50 4\.1 s, p95 33\.8 s per batch\)\. 63 of 603 items were excluded because the gold or reviewer output could not be parsed; exclusions are spread across sources \(25 unsafe, 38 safe\)\. The reviewer agrees with the policy gold on 91\.1% of items and with the dataset labels on 75\.0%\. Costs use $0\.112/$0\.336 per M input/output tokens for the reviewer and $0\.042 per M tokens forJev\.

Table 8:Two\-tier review, in\-sample \(%; cost relative to LLM\-only, pre\-screen cost included\)\.n=540n\{=\}540, 225 unsafe\.
## Appendix EReproducibility and Compute

Versions:Laya0\.3\.21 \(transformers 5\.17\.0, torch 2\.14\.0\),Jevjev\-1\.13\.0, DeepSeekdeepseek\-flash\. Runs: Windows \+ RTX 2080 Ti \(day 1\), macOS \+ Apple M4 CPU \(day 2\)\. TotalJevspend across all runs was $0\.52\. The DeepSeek audit and reference runs used 1\.7M input and 1\.1M output tokens\. The release \([https://github\.com/David\-DL\-Space/sys1\-eval](https://github.com/David-DL-Space/sys1-eval)\) contains every case file, raw response, analysis script \(paper/analysis\.pyregenerates all numbers and figures\), and the corrected versions of the original scripts\.