SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents

# SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents
Source: [https://arxiv.org/html/2609.00434](https://arxiv.org/html/2609.00434)
###### Abstract

Evaluating task\-oriented dialogue agents requires judging not merely whether a reply reads well but whether each turn advances the underlying workflow state correctly—a distinction conventional holistic LLM judges can miss because they evaluate the available context as a single unit and require one or more full\-model calls per turn\. We proposeSAGE\(*State\-Grounded Abstention\-Aware Evaluation*\), which compiles a workflow specification and per\-turn state diff into atomic, schema\-grounded criteria and routes each through a cascade of symbolic and encoder/NLI verifiers that*abstain*rather than guess, aggregating criterion verdicts into a turn\-level decision with an evidence trace\. Its recommended operating point,SAGE\-Core, decides8181–91%91\\%of criteria with only the compiler, symbolic rules, and on\-device encoders—at zero paid LLM cost—whileSAGE\-LLMadds an optional focused\-LLM fallback for open\-class criteria\. Across four slices spanning MultiWOZ, Schema\-Guided Dialogue, and ABCD, no evaluated LLM\-as\-a\-judge baseline—including a state\-aware GPT\-4\.1 judge and cheaper GPT\-4\.1\-mini variants—significantly exceeds SAGE\-Core on any slice, even though the GPT\-4\.1 G\-Eval judge costs $4\.7–8\.0 per1,0001\{,\}000turns to SAGE\-Core’s $0\. A two\-annotator human audit \(n=200n\{=\}200,κ=0\.94\\kappa\{=\}0\.94\) confirms strong label fidelity on the transcript\-visible failure classes—where, excluding the weak\-salience IUV class, SAGE\-Core is statistically tied with the strongest LLM judge—and honestly scopes ignored\-user\-value as a state\-consistency signal with weak broad\-human salience\. We analyze construct\-validity limits from injected failures and partial symbolic circularity\.

Microsoft

\{rayankhoury, shihyaolin, pmishra\}@microsoft\.com

## Introduction

![Refer to caption](https://arxiv.org/html/2609.00434v1/figures/fig1_motivation.png)Figure 1:A fluent, on\-topic agent turn that is nonetheless workflow\-wrong: the user asks to book in*Oxford*, but the agent confirms*Cambridge*and never recordsbook\_people, so the state diffΔt\\Delta\_\{t\}violates the workflow three ways\. A holistic transcript\-only LLM judge may rate the turn*acceptable*;SAGEinstead compiles the workflow spec𝒲\\mathcal\{W\}and per\-turn state pair\(st−1,st\)\(s\_\{t\-1\},s\_\{t\}\)into atomic criteria \(UV, IUV, PC\), routes each to the cheapest deciding stage \(S1 symbolic, S2 encoder/NLI, optional S3 focused LLM\), and returns a*failure*verdict with a per\-criterion evidence trace at zero paid API cost\. Figure[2](https://arxiv.org/html/2609.00434#Sx3.F2)shows the full pipeline\.Task\-oriented dialogue \(TOD\) agents increasingly front structured business workflows – booking, support, IT\-ticketing – whose correctness depends less on what the agent says than on how it changes an underlying*dialogue state*\. Following the dialogue\-state\-tracking tradition\([Budzianowski et al\. 2018](https://arxiv.org/html/2609.00434#bib.bib4);[Rastogi et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib7)\), write the slot\-value record before and after turnttasst−1s\_\{t\-1\}andsts\_\{t\}, and the per\-turn*state diff*asΔt=diff⁡\(st−1,st\)\\Delta\_\{t\}=\\operatorname\{diff\}\(s\_\{t\-1\},s\_\{t\}\)\(the set of changed\-slot triples; formalised in the Method\); the workflow advances correctly only ifΔt\\Delta\_\{t\}obeys four conditions – the values the user supplied are written to state, the next field the agent requests is schema\-legal, confirmation occurs only once its preconditions hold, and every written value is supported by the dialogue history\. Figure[1](https://arxiv.org/html/2609.00434#Sx1.F1)shows the stakes: on a turn that is fluent and on\-topic but writes an unsupported value, confirms before its preconditions hold, and drops a value the user just supplied, a holistic LLM\-as\-judge labels the turn acceptable, whereas a state\-grounded evaluator readsΔt\\Delta\_\{t\}and catches the failure with a deterministic, evidence\-bearing verdict\.

The dominant paradigm for multi\-turn LLM\-agent evaluation is nonetheless*holistic LLM\-as\-a\-judge*: a strong model reads the transcript and returns a single acceptability score, via direct rating\([Zheng et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib3)\)or rating\-token probability\-weighted scoring \(G\-Eval\)\([Liu et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib2)\)\. For stateful TOD this paradigm is structurally mismatched: it*conflates fluency with workflow correctness*\(a well\-phrased turn that silently fails to record a value the user just supplied is scored acceptable\); it is*blind to the specification*\(premature confirmation requires evaluating the outcome predicate𝒪⁡\(st−1\)\\mathcal\{O\}\(s\_\{t\-1\}\), inaccessible in principle to a transcript\-only judge that never seesst−1s\_\{t\-1\}\); and it is*monolithic*\(every turn costs a full LLM call, even when a deterministic check onΔt\\Delta\_\{t\}would settle it\)\.

SAGE\(*State\-Grounded Abstention\-Aware Evaluation*\) replaces holistic judging with a*compiler*that consumes a workflow specification𝒲=\(𝒮,ℱ,≺,𝒪\)\\mathcal\{W\}=\(\\mathcal\{S\},\\mathcal\{F\},\\prec,\\mathcal\{O\}\)– a schema𝒮\\mathcal\{S\}, required fieldsℱ\\mathcal\{F\}with a legal\-ordering relation≺\\prec, and an outcome predicate𝒪\\mathcal\{O\}– together with the historyh<th\_\{<t\}, the turn\(ut,at\)\(u\_\{t\},a\_\{t\}\), and the state pair\(st−1,st\)\(s\_\{t\-1\},s\_\{t\}\), and emits atomic per\-criterion questions \(“doesata\_\{t\}confirm while𝒪⁡\(st−1\)\\mathcal\{O\}\(s\_\{t\-1\}\)is false?”; “is the value written totrain\.destinationinΔt\\Delta\_\{t\}entailed byh<t∪\{ut\}h\_\{<t\}\\cup\\\{u\_\{t\}\\\}?”\)\. Each criterion is routed by a cost\-aware policy through a three\-stage*cascade*\([Chen et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib1)\):S1, deterministic rules overΔt\\Delta\_\{t\}and the schema;S2, a pretrained encoder/NLI judge whose confidence gate abstains when the entailment marginm=p⁡\(ent\)−p⁡\(contra\)m=p\(\\text\{ent\}\)\-p\(\\text\{contra\}\)satisfies\|m\|<τ\|m\|<\\tau; andS3, a focused single\-criterion LLM invoked only on S1∪\\cupS2 abstentions \(also the open\-class fallback for criteria outside the symbolic/NLI specialisation\)\.

Every verifier returnstrue,false, orabstain, and abstentions propagate up the cascade rather than being coerced into decisions\. Because criteria are verified independently, SAGE produces evidence\-bearing verdicts, pays LLM cost only on unresolved criteria, and abstains rather than guessing\.

We inject workflow\-grounded failures \(IUV, WNF, PC, UV\) at controlled rates into MultiWOZ 2\.4\([Ye et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib6)\)\(which refines the validation and test annotations of MultiWOZ 2\.1\([Eric et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib5)\)while retaining its training annotations\) and cross\-evaluate on the Schema\-Guided Dialogue corpus\([Rastogi et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib7)\)and the Action\-Based Conversations Dataset \(ABCD\)\([Chen et al\. 2021](https://arxiv.org/html/2609.00434#bib.bib27)\)to test that the compiler and cascade generalise across schemas and beyond slot\-grounded state\. We compare against G\-Eval\([Liu et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib2)\)\(holistic LLM\-as\-judge\), a FrugalGPT\([Chen et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib1)\)cascade, and a single\-call LLM\-as\-judge\([Zheng et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib3)\), each in*transcript\-only*and*state\-aware*configurations, so that SAGE’s advantage can be decomposed into the*information*advantage of state grounding versus the*architectural*advantage of criterion routing, NLI verification, and abstention \(adding state context in fact*lowers*the GPT\-4\.1 baselines’F1F\_\{1\}by up to0\.090\.09; see Discussion and Appendix[D](https://arxiv.org/html/2609.00434#A4)\)\. Our contributions are: \(i\) a compiler from workflow specs and per\-turn state diffs to atomic, schema\-grounded criteria; \(ii\) a three\-stage heterogeneous verifier cascade with abstention as a first\-class output and confidence\-gated S2 escalation; \(iii\) an abstention\-aware aggregator producing per\-turn verdicts with evidence traces and structured repair signals; and \(iv\) an empirical study – with paired\-bootstrap significance tests – isolating state\-grounded decomposition and cost\-aware cascading on the precision/recall/cost trade\-off across three benchmarks\.

## Related Work

#### LLM\-as\-a\-judge and cost\-aware cascades\.

Direct LLM rating dominates open\-ended evaluation—MT\-Bench\([Zheng et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib3)\)and G\-Eval\([Liu et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib2)\)\(CoT\([Wei et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib17)\)with rating\-token probability\-weighted scoring\) correlate with human preference; Prometheus\([Kim et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib18)\)and PandaLM\([Wang et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib19)\)train dedicated evaluator models, while AlpacaEval\([Dubois et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib20)\)studies LLM\-based pairwise auto\-evaluation and its biases\. These approaches score responses holistically rather than compiling workflow\-state changes into independently verifiable criteria, and in our experiments even strong judges*given the state diff*over\-predict acceptability on workflow\-grounded failures \(see Results\)\. Cost\-aware routing is likewise query\-level: FrugalGPT\([Chen et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib1)\)routes whole queries through ascending\-cost models, and AutoMix\([Aggarwal et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib22)\), RouteLLM\([Ong et al\. 2025](https://arxiv.org/html/2609.00434#bib.bib21)\), and LLM\-Blender\([Jiang et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib23)\)add self\-verification, preference routing, and ensembling\. SAGE instead routes single*criteria*compiled from workflow state through*heterogeneous*stages \(symbolic, encoder/NLI, focused LLM\), exploiting that many workflow violations are symbolically decidable from the state diff\.

#### NLI grounding, TOD evaluation, and abstention\.

Claim\-level factuality and consistency evaluation is established in fact\-checking, summarisation, and retrieval\-augmented generation\([Laban et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib9);[Manakul et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib25);[Min et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib24);[Es et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib26)\); SAGE adopts an NLI\-based formulation for atomic state\-support claims, with DeBERTa\-v3\([He et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib12)\)keeping inference cheap\. SAGE’s S2 answers atomic claims of the form “valuevvwritten to slotssis supported byh<th\_\{<t\},” with a margin gate that abstains into S3\. Classic TOD metrics \(JGA, slotF1F\_\{1\}, inform/success\)\([Budzianowski et al\. 2018](https://arxiv.org/html/2609.00434#bib.bib4);[Rastogi et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib7)\)require a gold trajectory and break under multiple legitimate completions, which LLM judges avoid but at the cost of workflow grounding; SAGE recovers grounding from schema, workflow spec, and state diff, and uses SGD\-X\([Lee et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib8)\)paraphrases as a robustness probe\. Selective classification\([Geifman and El\-Yaniv 2017](https://arxiv.org/html/2609.00434#bib.bib16)\)motivates abstention as a first\-class output: every SAGE stage may abstain, yielding interpretable coverage/accuracy decompositions\.

## The SAGE Framework

![Refer to caption](https://arxiv.org/html/2609.00434v1/figures/fig2_pipeline.png)Figure 2:The SAGE pipeline with measured routing \(Appendix[G](https://arxiv.org/html/2609.00434#A7)\)\. The compiler emits atomic schema\-grounded criteria tagged by failure class \(UV/IUV/PC/WNF\) and routes each to the cheapest deciding stage;*solid*arrows are decisions,*dashed*arrows are abstentions escalating rightward\.SAGE\-Core\(dashed boundary: compiler\+\{\+\}S1\+\{\+\}S2\) decides8181–91%91\\%of compiled criteria at zero paid LLM cost and is the recommended operating point; the optional S3 tier \(SAGE\-LLM\) handles the designated open\-class residual\. Criteria on which all stages abstain yield anundecidedturn verdict \(selective evaluation, Appendix[H](https://arxiv.org/html/2609.00434#A8)\)\.Figure[2](https://arxiv.org/html/2609.00434#Sx3.F2)shows the pipeline end\-to\-end; we formalise each block below\.

### Problem Setting

Let a workflow specification𝒲=\(𝒮,ℱ,≺,𝒪\)\\mathcal\{W\}=\(\\mathcal\{S\},\\mathcal\{F\},\\prec,\\mathcal\{O\}\)consist of a schema𝒮\\mathcal\{S\}defining slots and their types, a set of required fieldsℱ⊆𝒮\\mathcal\{F\}\\subseteq\\mathcal\{S\}, an ordering relation≺\\precoverℱ\\mathcal\{F\}that constrains the legal next field, and an outcome predicate𝒪\\mathcal\{O\}that determines when the workflow may be confirmed\. A dialogue is a sequence of turns\(ut,at,st\)t=1T\(u\_\{t\},a\_\{t\},s\_\{t\}\)\_\{t=1\}^\{T\}whereutu\_\{t\}is the user utterance,ata\_\{t\}is the agent response, andsts\_\{t\}is the workflow state after the agent acts\. Throughout, we refer tost−1s\_\{t\-1\}as the*state\-before*andsts\_\{t\}as the*state\-after*the turn; both are dialogue states\([Budzianowski et al\. 2018](https://arxiv.org/html/2609.00434#bib.bib4)\)constrained by𝒲\\mathcal\{W\}\. Define the state diffΔt=\{\(s,st−1​\[s\],st​\[s\]\):st−1​\[s\]≠st​\[s\]\}\\Delta\_\{t\}=\\\{\(s,\\,s\_\{t\-1\}\[s\],\\,s\_\{t\}\[s\]\):s\_\{t\-1\}\[s\]\\neq s\_\{t\}\[s\]\\\}, the set of slots whose value changed over the turn\. The evaluator’s task at turnttis to decide whether\(ut,at,Δt\)\(u\_\{t\},a\_\{t\},\\Delta\_\{t\}\)is*acceptable*with respect to𝒲\\mathcal\{W\}and the historyh<th\_\{<t\}\. We define a fixed taxonomy𝒞\\mathcal\{C\}of four workflow failures:ignored user value \(IUV\)– the user supplies a valuevvfor slotssinutu\_\{t\}\(or recenth<th\_\{<t\}\) butΔt\\Delta\_\{t\}fails to writevvtoss, sossremains unfilled or stale insts\_\{t\};wrong next field \(WNF\)–ata\_\{t\}requests a slotf∉Next⁡\(𝒲,st−1\)f\\notin\\mathrm\{Next\}\(\\mathcal\{W\},s\_\{t\-1\}\);premature confirmation \(PC\)–ata\_\{t\}confirms the workflow while𝒪⁡\(st−1\)=false\\mathcal\{O\}\(s\_\{t\-1\}\)=\\mathrm\{false\}; andunsupported value \(UV\)–Δt\\Delta\_\{t\}writes a value to a slot that is not entailed byh<t∪\{ut\}h\_\{<t\}\\cup\\\{u\_\{t\}\\\}\.

A turn is*acceptable with respect to our workflow\-failure taxonomy*𝒞\\mathcal\{C\}iff no criterion compiled from𝒞\\mathcal\{C\}fires\. This is a taxonomy\-relative notion: a turn may still be dysfluent, unsafe, or unhelpful in ways outside𝒞\\mathcal\{C\}; SAGE targets workflow\- and state\-grounded failures, not holistic response quality\.

### Compiler

The compiler maps each turn to a set of atomic, separately evaluated criteria\. Given\(𝒲,h<t,ut,at,st−1,st\)\(\\mathcal\{W\},h\_\{<t\},u\_\{t\},a\_\{t\},s\_\{t\-1\},s\_\{t\}\)it emits a listKt=\{kt\(i\)\}iK\_\{t\}=\\\{k\_\{t\}^\{\(i\)\}\\\}\_\{i\}where eachkt\(i\)=\(type,args,evidence\)k\_\{t\}^\{\(i\)\}=\(\\textsc\{type\},\\textsc\{args\},\\textsc\{evidence\}\)binds a failure class to concrete schema\-grounded arguments and a minimal evidence span\. For IUV and WNF the compiler reads the schema and state directly \(for IUV, it cross\-references the user’s last\-turn extraction againstΔt\\Delta\_\{t\}\); for PC it reads the outcome predicate; for UV it iterates overΔt\\Delta\_\{t\}and emits one criterion per written value, with the supporting span search restricted toh<t∪\{ut\}h\_\{<t\}\\cup\\\{u\_\{t\}\\\}\. The compiler is deterministic and stateless across turns\. Its output is a per\-turn evaluation*plan*that downstream stages consume independently of the dialogue surface\.

### Verifier Cascade

Each criterionkkis routed to the cheapest applicable stage, with abstention propagating upward:

#### S1: Deterministic rules\.

IUV, WNF, and PC are symbolic checks against the schema and the state diff\. S1 either returns a definitive verdict or, if the schema does not admit a check \(e\.g\., the workflow ordering is partial and the requested slot is in the indeterminate set\), abstains\. S1 has zero LLM cost and microsecond latency\.

#### S2: Encoder/NLI grounding with confidence gating\.

UV criteria, and a subset of IUV/PC criteria that depend on natural\-language understanding \(paraphrased confirmations, indirect slot fills\), are routed to S2\. We use a pretrained encoder to score entailment between the candidate claim and a windowed context\. A confidence gate computes a marginm=p⁡\(ent\)−p⁡\(contra\)m=p\(\\text\{ent\}\)\-p\(\\text\{contra\}\)and abstains when\|m\|\|m\|falls below a thresholdτ\\tau, calibrated on a held\-out development split\. A redundancy check is state\-gated: it only fires when the state diff actually introduces a value, avoiding spurious abstentions on no\-op turns\.

#### S3: Focused single\-criterion LLM \(optional\)\.

Criteria that S1 and S2 abstain on*and*that fall in a designated open\-class set \(e\.g\. hallucinated\-value and history\-contradiction checks that need semantic interpretation\) are sent to a focused LLM prompt containing only the criterion text, the minimal evidence span, and an instruction to answer \{true,false,abstain\}\. Because the prompt is criterion\-local rather than transcript\-global, S3 calls are short, cacheable, and parallelisable across criteria\. S3 is optional: it can be disabled for a fully zero\-LLM deployment – the recommended operating point on the compiled taxonomies we study – and any residual abstention is handled by the aggregator below\.

#### Two named configurations\.

We call the zero\-paid\-LLM configuration \(compiler \+ S1 \+ S2, S3 disabled\)SAGE\-Coreand the full cascadeSAGE\-LLM\. SAGE\-Core is the recommended operating point: it issues zero paid API calls and, as the ablation shows \(Appendix[F](https://arxiv.org/html/2609.00434#A6)\), matches or exceeds SAGE\-LLM on every slice – a designation fixed before the paired significance tests below, which confirm rather than motivate it\. S3 remains available as an extensibility mechanism for open\-class criteria outside the symbolic and encoder/NLI stages, but our empirical claims do not depend on it\.

### Aggregation

A turn\-level verdict reduces the per\-criterion outputs:*predict failure if any criterion istrue; predict acceptable only if every compiled criterion is decided andfalse; otherwise—no criteriontruebut at least oneabstain—mark the turnundecided*\. All headline results use*full\-coverage*scoring \(undecided→\\toacceptable\); the same per\-criterion margins also support*selective*evaluation – declining the lowest\-confidence turns improvesF1F\_\{1\}along a risk–coverage frontier on the encoder\-heavy slices \(Appendix[H](https://arxiv.org/html/2609.00434#A8), Table[10](https://arxiv.org/html/2609.00434#A8.T10)\)\.

### Repair Signals

Because each verdict is tied to a typed criterion, SAGE emits a structured repair signal alongside it \(e\.g\. IUV→\\to“record valuevvat span\[ℓ,r\]\[\\ell,r\]into slotss”; PC→\\to“required slots\{fi\}\\\{f\_\{i\}\\\}still empty, do not confirm”\)\. Repair signals are a side output for downstream agent self\-correction, not consumed by the evaluation metric; Appendix[E](https://arxiv.org/html/2609.00434#A5)gives per\-class examples\.

## Experiments

### Benchmark Construction

We build four slices from three corpora, running the*same*evaluator code on all four \(only the compiledWorkflowSpecdiffers\)\. From MultiWOZ 2\.4\([Ye et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib6)\)\(which refines the validation and test annotations of MultiWOZ 2\.1\([Eric et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib5)\)while retaining its training annotations\) we sample250250validation turns stratified across five domains \(taxi, hotel, restaurant, train, attraction;5050each\) and either retain the original acceptable turn or inject one failure from𝒞\\mathcal\{C\}, mutating the agent response and state diff*together*so the failure is genuinely state\-grounded \(MW\-mixed\)\. TheUV\-richsplit instead concentrates theunsupported\_valueclass \(121121injected,129129clean\) to stress S2’s entailment path\. For cross\-dataset generalisation we buildSGDfrom the Schema\-Guided Dialogue test split\([Rastogi et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib7)\)\(280280turns spanning3737\(service, intent\) workflows, capped at88turns/intent and1212files/service, same four classes at a50%50\\%rate\) andABCD\-UVfrom the Action\-Based Conversations Dataset\([Chen et al\. 2021](https://arxiv.org/html/2609.00434#bib.bib27)\)\(223223action\-grounded turns, UV\-only\)\. No SAGE code changes across corpora; per\-class injection templates are in Appendix[A](https://arxiv.org/html/2609.00434#A1)\.

### Baselines

We compare against three LLM\-as\-judge families, each*adapted*to turn\-level workflow acceptability and evaluated in*transcript\-only*and*state\-aware*forms; state\-aware prompts receive the same schema and pre/post state block SAGE consumes, and both forms share model, prompt template, sampling protocol, and threshold, differing only in the per\-turn context block\. OurG\-Eval\-style judge\(adapted from[Liu et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib2)\) usesgpt\-4\.1with a task\-specific CoT/form\-filling prompt andn=6n\{=\}6samples over a11–55acceptability scale, thresholded on a development split; the six\-sample count is our implementation choice \(the original approximates unavailable token probabilities withn=20n\{=\}20\)\. OurFrugalGPT\-style cascade\(adapted from[Chen et al\. 2024](https://arxiv.org/html/2609.00434#bib.bib1)\) routesgpt\-4\.1\-mini/gpt\-4\.1: the cheap stage decides each turn and only low\-confidence turns escalate \(its confidence model is state\-aware, so the transcript\-only row largely reflectsgpt\-4\.1\-minialone\)\. OurMT\-Bench\-style direct judge\(adapted from[Zheng et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib3)\) is a singlegpt\-4\.1call \(temperature00\) on the multi\-class slices returning a failure/clean verdict plus an optional class tag; we report the stronger of its two modes per slice \(state\-aware on MW\-mixed, transcript\-only on ABCD\), re\-selected within each bootstrap resample \(Metrics\)\. Prompt and threshold details are in Appendix[D](https://arxiv.org/html/2609.00434#A4)\.

#### SAGE variants\.

We report both named configurations –SAGE\-Core\(S1\+S2, the recommended zero\-paid\-LLM operating point\) andSAGE\-LLM\(SAGE\-Core \+ optional S3\) – together with a per\-tier leave\-one\-out ablation \(no\_S1,no\_S2,no\_S3, the last being exactly SAGE\-Core; Appendix[F](https://arxiv.org/html/2609.00434#A6)\), each disabling one verifier tier while holding workflow, deployment, and per\-turn data fixed\. The compiler cannot be cleanly ablated within the cascade – S1 and S2 consume its per\-criterion plan as input – so the no\-compiler reference points are the two G\-Eval rows in Table[4](https://arxiv.org/html/2609.00434#A2.T4)\.

### Metrics

For each evaluator we report precision, recall,F1F\_\{1\}, total dollar cost \(computed from per\-call token usage and public OpenAI API list prices, retrieved July 2026; runs were conducted May–July 2026\), and per\-turn latency \(median, reported as p50\)\. Point estimates in Tables[1](https://arxiv.org/html/2609.00434#Sx5.T1)–[8](https://arxiv.org/html/2609.00434#A6.T8)are accompanied by95%95\\%bootstrap CIs\([Efron and Tibshirani 1993](https://arxiv.org/html/2609.00434#bib.bib29)\)over10,00010\{,\}000turn\-level resamples \(seed00, percentile method\) reported in the captions\. To compare SAGE against the strongest baseline on each slice we use a*paired*bootstrap: on every resample we recompute both evaluators’F1F\_\{1\}on the*same*resampled turn indices and record the differenceΔ​F1\\Delta F\_\{1\}, yielding a95%95\\%CI on the paired delta and a two\-sidedpp\-value \(twice the smaller bootstrap tail probability thatΔ​F1\\Delta F\_\{1\}takes the opposite sign\)\. Where a slice’s comparator is the strongest of several evaluated GPT\-4\.1 baselines, we re\-select that maximum\-F1F\_\{1\}baseline*inside every resample*, so the CI andpp\-value account for baseline\-selection uncertainty\. SAGE rows are computed on the*identical per\-turn slices*used for every baseline \(verified row\-for\-row: same turn IDs and gold labels\), so all paired tests are aligned\.

### Implementation

The compiler and S1 are pure Python over the MultiWOZ 2\.4 and SGD schemas\. S2 uses two public HuggingFace encoders:cross\-encoder/nli\-deberta\-v3\-base\(revision6c749ce;≈184\{\\approx\}184M params\), a DeBERTa\-v3\([He et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib12)\)cross\-encoder fine\-tuned on SNLI and MultiNLI\([Bowman et al\. 2015](https://arxiv.org/html/2609.00434#bib.bib10);[Williams et al\. 2018](https://arxiv.org/html/2609.00434#bib.bib11)\), for entailment on UV/PC criteria, andsentence\-transformers/all\-MiniLM\-L6\-v2\(revision1110a24;≈23\{\\approx\}23M params\), a MiniLM\([Wang et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib14)\)/Sentence\-BERT\([Reimers and Gurevych 2019](https://arxiv.org/html/2609.00434#bib.bib13)\)encoder, for paraphrase similarity on redundancy criteria\.111Immutable Hugging Face revision pages:https://huggingface\.co/cross\-encoder/nli\-deberta\-v3\-base/tree/6c749ce;https://huggingface\.co/sentence\-transformers/all\-MiniLM\-L6\-v2/tree/1110a24; andhttps://huggingface\.co/sentence\-transformers/all\-mpnet\-base\-v2/tree/e8c3b32\.The entailment threshold isτ=0\.20\\tau\{=\}0\.20on the marginm=p⁡\(ent\)−p⁡\(contra\)m=p\(\\text\{ent\}\)\-p\(\\text\{contra\}\)\(\|m\|<τ\|m\|<\\tauabstains\), with an extraction\-confidence floor of0\.50\.5; both were tuned on a4040\-turn development split disjoint from evaluation and applied unchanged to SGD\. S3 and the FrugalGPT cheap stage use Azure OpenAIgpt\-4\.1/gpt\-4\.1\-mini\(snapshot2025\-04\-14, temperature00\); costs derive from response\-metadata token counts at OpenAI API list prices \($2/$8\\$2/\\$8and$0\.40/$1\.60\\$0\.40/\\$1\.60per Mtok in/out\)\.222OpenAI API list prices,https://openai\.com/api/pricing/, retrieved 21 July 2026; runs were conducted May–July 2026\.Threshold ranges swept, run counts, and the computing infrastructure are in Appendix[J](https://arxiv.org/html/2609.00434#A10)\.

## Results

We evaluate on four slices spanning three datasets, running the*same*evaluator code on all four \(only the compiledWorkflowSpecdiffers\):MW\-mixed\(MultiWOZ 2\.4, all four failure classes,n=250n\{=\}250,126126injected\);UV\-rich\(MultiWOZ 2\.4, single\-classunsupported\_valuestress test,n=250n\{=\}250,121121injected\);SGD\(cross\-dataset Schema\-Guided Dialogue,n=280n\{=\}280,151151injected across3737workflows\); andABCD\-UV\(Action\-Based Conversations\([Chen et al\. 2021](https://arxiv.org/html/2609.00434#bib.bib27)\), UV\-only,n=223n\{=\}223,103103injected across3030action\-grounded workflows\)\. Per\-method precision/recall/cost/latency for every slice are in Appendix[B](https://arxiv.org/html/2609.00434#A2)\.

Table 1:Headline result across all four slices:SAGE\-Core\(compiler\+S1\+S2, zero paid LLM calls\) vs\. the strongest*evaluated*GPT\-4\.1 baseline and the strongest*reduced\-cost*GPT\-4\.1\-mini judge, byF1F\_\{1\}\. Paired\-bootstrapΔ​F1\\Delta F\_\{1\}\(SAGE\-Core−\-baseline\) andpp—*vs\. GPT\-4\.1:*MW\-mixed\+\.069\{\+\}\.069\(p=\.022p\{=\}\.022\), UV\-rich−\.003\{\-\}\.003\(p=\.89p\{=\}\.89\), SGD\+\.075\{\+\}\.075\(p=\.011p\{=\}\.011\), ABCD\+\.056\{\+\}\.056\(p=\.044p\{=\}\.044\);*vs\. mini:*MW\-mixed\+\.107\{\+\}\.107\(p=\.001p\{=\}\.001\), UV\-rich\+\.019\{\+\}\.019\(p=\.46p\{=\}\.46\), SGD\+\.005\{\+\}\.005\(p=\.83p\{=\}\.83\), ABCD\+\.152\{\+\}\.152\(p<\.001p\{<\}\.001\)\. SGD survives Holm–Bonferroni correction \(over the four pre\-specified GPT\-4\.1 slice comparisons\); MW\-mixed and ABCD wins are nominal \(do not survive it\)\. Per\-method breakdowns in Appendix[B](https://arxiv.org/html/2609.00434#A2)\(Tables[3](https://arxiv.org/html/2609.00434#A2.T3),[4](https://arxiv.org/html/2609.00434#A2.T4),[5](https://arxiv.org/html/2609.00434#A2.T5)\); reduced\-cost judges in Appendix[D](https://arxiv.org/html/2609.00434#A4)\.No evaluated baseline significantly exceeds SAGE\-Core on any slice, at zero paid API cost\(local symbolic/encoder compute is unmonetized; Table[1](https://arxiv.org/html/2609.00434#Sx5.T1)\)\. It wins on SGD \(\+\.075\{\+\}\.075,p=\.011p\{=\}\.011; the one comparison surviving Holm–Bonferroni correction\([Holm 1979](https://arxiv.org/html/2609.00434#bib.bib28)\)over the four pre\-specified GPT\-4\.1 slice comparisons\) and nominally on MW\-mixed \(\+\.069\{\+\}\.069,p=\.022p\{=\}\.022\) and ABCD \(\+\.056\{\+\}\.056,p=\.044p\{=\}\.044\), and ties the far costlier G\-Eval on UV\-rich \(−\.003\{\-\}\.003,p=\.89p\{=\}\.89\)\. These pairedpp\-values re\-select the strongest baseline within each resample; on SGD \(four evaluated GPT\-4\.1 baselines within\.013\.013F1F\_\{1\}\) this raisesppto\.011\.011—still surviving Holm—while the other slices, each dominated by a single baseline, are unaffected\. UV\-rich is the only slice where a holistic judge holds the top*point*F1F\_\{1\}\(\.923\.923\) – expected when a single failure class makes the11–55acceptability score well\-anchored – but the bootstrap cannot separate the two: G\-Eval reaches it at $4\.70/1k and12\.012\.0s median latency, versus SAGE\-Core at $0 paid API cost and0\.030\.03s warm local latency \(≈14×\\approx 14\\timesthe cost and≈44×\\approx 44\\timesthe latency relative to SAGE\-LLM\)\. SAGE\-Core’s tie is threshold\-robust: a77\-setting sweep of the S2 NLI floors keeps itsF1F\_\{1\}within a\.007\.007band\.

No cheaper\-backbone judge significantly exceeds SAGE\-Core either\.A GPT\-4\.1\-mini judge \(≈5×\\approx 5\\timescheaper at matched token usage; $0\.9–1\.6/1k vs\. $4\.7–8\.0/1k\) overtakes, on SGD, the pre\-specified FrugalGPT baseline \(\.859\>\.789\.859\{\>\}\.789\); even so it only*ties*SAGE\-Core \(SGD\+\.005\{\+\}\.005,p=\.83p\{=\}\.83; UV\-rich\+\.019\{\+\}\.019,p=\.46p\{=\}\.46\) and loses on the multi\-class slices \(MW\-mixed\+\.107\{\+\}\.107,p=\.001p\{=\}\.001; ABCD\+\.152\{\+\}\.152,p<\.001p\{<\}\.001\)\. So across both the pre\-specified suite and the cheaper\-backbone mini judges on all four slices,*no evaluated LLM\-as\-a\-judge baseline significantly exceeds SAGE\-Core*\(Appendix[D](https://arxiv.org/html/2609.00434#A4)\)\.

The focused\-LLM tier is not load\-bearing, and information access does not explain the gap\.Enabling S3 \(SAGE\-LLM\) never improvesF1F\_\{1\}and mildly lowers it on UV\-rich and ABCD, so we recommend SAGE\-Core \(per\-tier ablation, Appendix[F](https://arxiv.org/html/2609.00434#A6)\)\. Giving the GPT\-4\.1 judges the same schema/state block SAGE consumes*lowers*theirF1F\_\{1\}by up to0\.090\.09in our prompt template \(the effect reverses for the mini backbone on SGD; Appendix[D](https://arxiv.org/html/2609.00434#A4)\), so the advantage is not information access alone\. Per class, SAGE\-Core catches essentially all UV \(59/6159/61MW\-mixed,98/9898/98SGD,95%95\\%UV\-rich\) and saturates PC/WNF recall, but under\-fires IUV \(16/3216/32MW\-mixed,18/4118/41SGD\) – the class we revisit in the human audit\. We caveat ABCD on construct grounds: most of its criteria are decided by S1/S2 predicates that partially mirror the injection operators \(Discussion\), so it evidences recovery of designed\-for failures rather than in\-the\-wild detection\.

Additional analyses\.Appendices report per\-method precision/recall/cost/latency tables \(Appendix[B](https://arxiv.org/html/2609.00434#A2)\), per\-tier ablations showing S2 carries the precision floor and SAGE\-Core \(S1\+S2\) matches or beats the full cascade on every slice \(Appendix[F](https://arxiv.org/html/2609.00434#A6)\), a cascade cost anatomy \(8181–91%91\\%of criteria decided at zero API cost; Appendix[G](https://arxiv.org/html/2609.00434#A7)\), calibrated selective evaluation \(Appendix[H](https://arxiv.org/html/2609.00434#A8)\), and SGD\-X paraphrase robustness \(verdictσ=0\.002\\sigma\{=\}0\.002vs\.0\.0050\.005–0\.0150\.015for LLM judges; Appendix[I](https://arxiv.org/html/2609.00434#A9)\)\.

## Human Validation Audit

Our benchmarks use injected failures, so an evaluator might be recovering synthetic perturbations rather than human\-meaningful workflow errors\. We ran a blind audit on a stratifiedn=200n\{=\}200sample:100100injected failures spanning all four classes \(MW\-mixed \+ SGD\),5050gold\-clean turns, and5050ABCD \+ UV\-rich examples\. Two annotators independently labeled each turnacceptable/failuregiven the dialogue history, the*state pair*\(st−1,st\)\(s\_\{t\-1\},s\_\{t\}\), and the agent response, blind to both the injected label and every model verdict; the3%3\\%of turns on which they disagreed are dropped from the consensus set\. Reliability was high \(raw agreement0\.970\.97, Cohen’sκ=0\.939\\kappa\{=\}0\.939\([Cohen 1960](https://arxiv.org/html/2609.00434#bib.bib30)\)\); protocol and sheets are released with the code\.

#### Label fidelity\.

On the consensus set \(n=194n\{=\}194;66of the200200sampled turns dropped for annotator disagreement\), humans agreed with the injected gold label on158/194=0\.814158/194\{=\}0\.814of turns \(Cohen’sκ=0\.639\\kappa\{=\}0\.639\), with*perfect*agreement on the ABCD \+ UV\-rich subset\. Agreement is high for every transcript\-visible class—PC1\.001\.00, UV0\.950\.95, and WNF0\.730\.73\(Table[2](https://arxiv.org/html/2609.00434#Sx6.T2)\)—supporting construct validity: these perturbations are perceived as genuine failures, not label noise\. WNF is the softest \(0\.730\.73\), cohering with MultiWOZ’s partial slot ordering, which also makes the symbolic verifier fire conservatively; two independent signals thus agree WNF is a softer constraint than the others\.

#### The IUV class\.

The one sharp disagreement is IUV \(ignored user value\): humans endorsed the injected label on only1/301/30turns, and2929of the3535injected\-turn human–injection disagreements are IUV \(the3636th consensus\-set disagreement is a gold\-clean turn a human flagged as a natural failure; see below\)\. This is not label noise—the injection is verifiable by construction \(a user\-supplied value is deleted fromsts\_\{t\}\)—nor a transcript\-only artifact: annotators*saw*the state pair, but our protocol does not disambiguate whether they failed to perform the slot\-by\-slot reconciliation ofutu\_\{t\}againststs\_\{t\}that would surface the omission, or performed it and did not consider the dropped value a failure\. Detection is correspondingly method\-dependent and trades against precision, and no evaluator dominates \(per\-evaluator counts in Appendix[K](https://arxiv.org/html/2609.00434#A11)\); SAGE\-Core sits at the high\-precision end\. We thus treat IUV as a class whose*labels*are verifiable but whose*broad\-human construct\-alignment is weak*, and do not claim SAGE\-Core is uniquely able to detect it\.

#### Agreement with human judgment\.

Treating the human consensus as ground truth, SAGE\-Core reachesF1=0\.825F\_\{1\}\{=\}0\.825over all audited turns versus0\.9040\.904for the strongest evaluated LLM judge \(transcript\-only G\-Eval\); a paired bootstrap over alln=194n\{=\}194audited turns confirms this as a significant deficit \(Δ​F1=−0\.079\\Delta F\_\{1\}\{=\}\{\-\}0\.079,95%95\\%CI\[−0\.142,−0\.019\]\[\{\-\}0\.142,\{\-\}0\.019\],p=0\.01p\{=\}0\.01\)\. The gap is concentrated almost entirely in IUV: on the transcript\-visible subset \(n=164n\{=\}164\) SAGE\-Core reaches0\.9090\.909versus0\.9200\.920, a difference a paired bootstrap cannot distinguish from zero \(Δ​F1=−0\.011\\Delta F\_\{1\}\{=\}\{\-\}0\.011,p=0\.68p\{=\}0\.68\)\. The honest reading is*not*that SAGE\-Core is a better proxy for human judgment; rather, on the transcript\-visible failure classes excluding IUV it is statistically tied with the strongest LLM judge, at zero paid API cost, while its overall deficit reflects the state\-only IUV class on which human labels are themselves weak\. Crucially, the headline conclusions are robust to dropping this contested class: re\-running the pre\-specified paired bootstraps with all injected\-IUV turns removed leaves every verdict unchanged and*widens*two margins \(MW\-mixedΔ​F1=\+0\.152\\Delta F\_\{1\}\{=\}\{\+\}0\.152, SGD\+0\.149\{\+\}0\.149, bothp<0\.001p\{<\}0\.001\)\. Of the5050clean turns, annotators flagged one naturally\-occurring failure \(SAGE\-Core missed it\); withn=1n\{=\}1we draw no conclusion beyond noting that a larger natural\-failure study remains the clearest path to fuller external validity\.

Table 2:Human validation audit \(n=200n\{=\}200turns, two annotators blind to the injected label and all model verdicts; annotators saw the transcript, the state pair, and the agent response\)\. Top: reliability and label fidelity\. Middle: per\-class human agreement with the injected label on MW\-mixed \+ SGD\. Bottom: agreement with the human consensus \(F1F\_\{1\}\), overall and on the transcript\-visible subset \(excluding the state\-only IUV class\)\.

## Discussion

#### Why state grounding is necessary but not sufficient\.

Contrary to expectation, state\-aware G\-Eval is*weaker*than transcript\-only on MultiWOZ \(F1F\_\{1\}0\.8340\.834vs\.0\.9230\.923, Table[4](https://arxiv.org/html/2609.00434#A2.T4)\) and tied on SGD, and the state\-aware LLM\-as\-judge over\-flags ABCD’s terse text \(Table[1](https://arxiv.org/html/2609.00434#Sx5.T1)\): extra context appears to anchor the holistic 1–5 score on the consistency ofsts\_\{t\}rather than the support ofata\_\{t\}\. This is prompt\-sensitive, not a universal law – with agpt\-4\.1\-minibackbone the effect*reverses*on SGD \(state context*helps*,\+0\.070\{\+\}0\.070; Appendix[D](https://arxiv.org/html/2609.00434#A4)\)\. The surviving, defensible claim is narrower but robust: even the best\-informed, best\-backboned holistic judge does not significantly exceed SAGE\-Core \(Table[6](https://arxiv.org/html/2609.00434#A4.T6)\)\. The SAGE–baseline gap is therefore consistent with – but not proof of – an architectural advantage from per\-criterion decomposition, cascade routing, and abstention\-aware aggregation, i\.e\.*changing the unit of judgement*rather than seeing more context\.

#### Cheaper\-backbone baselines\.

Our G\-Eval\-style judge’sn=6n\{=\}6sampling configuration\([Liu et al\. 2023](https://arxiv.org/html/2609.00434#bib.bib2)\)is sequential, driving its≈14\\approx 14–44×44\\timeslatency multiples\. To rule out that SAGE’s advantage is a model\-class artifact of the expensive GPT\-4\.1 backbone, we re\-ran G\-Eval on the≈5×\\approx 5\\timescheaper \(at matched token usage\)gpt\-4\.1\-miniin both prompt modes, at $0\.9–1\.6/1k vs\. $4\.7–8\.0/1k for GPT\-4\.1 \(Appendix[D](https://arxiv.org/html/2609.00434#A4)\)\. The cheap judge overtakes our pre\-specified FrugalGPT baseline on SGD but only*ties*SAGE\-Core there and stays below it on MultiWOZ; the single\-call LLM\-as\-judge in Table[1](https://arxiv.org/html/2609.00434#Sx5.T1)is a further reduced\-cost point SAGE matches or leads\. No evaluated judge, at any tested budget, significantly exceeds SAGE\-Core, which pays nothing\.

#### Symbolic\-injection circularity, and what is non\-circular\.

For three classes \(WNF, PC, IUV\) the injection operator and S1 share a predicate \(e\.g\. PC is injected when state writesconfirmedwith a required field unfilled, exactly the S1 firing condition\); where SAGE wins symbolically at near\-zero cost \(notably S1\+S2 on ABCD\) it is largely inverting its own injector on the symbolically\-decidable subset, and we read those numbers as the compiler*recovering failures of the form it is designed to detect*\. UV is closer to non\-circular: detection runs an off\-the\-shelf DeBERTa\-v3 NLI step \(cross\-encoder/nli\-deberta\-v3\-base\) that does not share the symbolic predicate, though the injected value is still chosen to be NLI\-detectable, so this remains constructive rather than fully in\-the\-wild\. SAGE’s UV recall on MultiWOZ \(SAGE\-Core0\.970\.97, SAGE\-LLM0\.980\.98\) and ABCD \(0\.9510\.951/0\.9610\.961\) is the strongest external evidence of detection power\.

#### Limitations\.

SAGE requires a workflow specification and is unsuited to open\-ended chat\. MultiWOZ’s partial required\-field ordering causes S1 to fire conservatively \(23%23\\%FP rate\); the dominant SGD failure is value\-normalisation mismatch \(IUV recall0\.440\.44\); and on ABCD,5/1035/103true failures slip under the recommended SAGE\-Core \(4/1034/103under SAGE\-LLM\) when the bot asserts a value the post\-turn state does not record\. Most fundamentally, all benchmarks use*injected*failures\. Our two\-annotator human audit \(*Human Validation Audit*, Table[2](https://arxiv.org/html/2609.00434#Sx6.T2);n=200n\{=\}200,κ=0\.94\\kappa\{=\}0\.94\) probes the resulting construct\-validity risk: it confirms strong fidelity on the transcript\-visible classes, but flags WNF as a softer constraint \(agreement0\.730\.73\) and IUV as a formal state\-consistency signal with weak broad\-human salience \(1/301/30\) rather than a fully human\-validated class; all headline conclusions are unchanged when IUV is excluded\. A naturally\-occurring failure set disjoint from S1’s predicates remains the highest\-leverage addition and is left to future work\. Headline numbers use oracle \(service, intent\) routing; learned routers move SAGEF1F\_\{1\}by<0\.003\{<\}0\.003on SGD \(Appendix[C](https://arxiv.org/html/2609.00434#A3)\)\.

## Conclusion

SAGE compiles workflow specifications and per\-turn state diffs into atomic criteria, verifies them through a symbolic/encoder cascade with abstention, and returns evidence\-bearing turn verdicts\. Its recommendedSAGE\-Coreconfiguration makes zero paid LLM calls and is not significantly exceeded by any evaluated LLM judge across MultiWOZ, SGD, and ABCD slices;SAGE\-LLM’s optional paid tier is not load\-bearing\. A two\-annotator human audit \(n=200n\{=\}200,κ=0\.94\\kappa\{=\}0\.94\) supports strong fidelity for PC and UV \(including the ABCD/UV\-rich subset\) and moderate fidelity for WNF, while identifying IUV as a weaker construct\-validity case\. These results argue for criterion\-level, state\-grounded evaluation over holistic transcript scoring for workflow\-driven dialogue agents\.

## References

- Aggarwalet al\.\(2024\)P\. Aggarwal, A\. Madaan, A\. Anand, S\. P\. Potharaju, S\. Mishra, P\. Zhou, A\. Gupta, D\. Rajagopal, K\. Kappaganthu, Y\. Yang, S\. Upadhyay, M\. Faruqui, and MausamAutoMix: automatically mixing language models\.InAdvances in Neural Information Processing Systems,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Bowmanet al\.\(2015\)S\. R\. Bowman, G\. Angeli, C\. Potts, and C\. D\. ManningA large annotated corpus for learning natural language inference\.InProceedings of EMNLP,Cited by:[Implementation](https://arxiv.org/html/2609.00434#Sx4.SSx4.p1.1)\.
- Budzianowskiet al\.\(2018\)P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. GašićMultiWOZ – a large\-scale multi\-domain Wizard\-of\-Oz dataset for task\-oriented dialogue modelling\.InProceedings of EMNLP,Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p1.1),[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1),[Problem Setting](https://arxiv.org/html/2609.00434#Sx3.SSx1.p1.1)\.
- Chenet al\.\(2021\)D\. Chen, H\. Chen, Y\. Yang, A\. Lin, and Z\. YuAction\-based conversations dataset: a corpus for building more in\-depth task\-oriented dialogue systems\.InProceedings of NAACL\-HLT,Cited by:[Table 3](https://arxiv.org/html/2609.00434#A2.T3),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[Benchmark Construction](https://arxiv.org/html/2609.00434#Sx4.SSx1.p1.1),[Results](https://arxiv.org/html/2609.00434#Sx5.p1.1)\.
- Chenet al\.\(2024\)L\. Chen, M\. Zaharia, and J\. ZouFrugalGPT: how to use large language models while reducing cost and improving performance\.Transactions on Machine Learning Research\.Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p3.1),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1),[Baselines](https://arxiv.org/html/2609.00434#Sx4.SSx2.p1.1)\.
- Cohen \(1960\)J\. CohenA coefficient of agreement for nominal scales\.Educational and Psychological Measurement20\(1\),pp\. 37–46\.Cited by:[Human Validation Audit](https://arxiv.org/html/2609.00434#Sx6.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Efron and Tibshirani \(1993\)B\. Efron and R\. J\. TibshiraniAn introduction to the bootstrap\.Chapman & Hall/CRC\.Cited by:[Metrics](https://arxiv.org/html/2609.00434#Sx4.SSx3.p1.1)\.
- Ericet al\.\(2020\)M\. Eric, R\. Goel, S\. Paul, A\. Sethi, S\. Agarwal, S\. Gao, A\. Kumar, A\. Goyal, P\. Ku, and D\. Hakkani\-TurMultiWOZ 2\.1: a consolidated multi\-domain dialogue dataset with state corrections and state tracking baselines\.InProceedings of the 12th Language Resources and Evaluation Conference \(LREC\),Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[Benchmark Construction](https://arxiv.org/html/2609.00434#Sx4.SSx1.p1.1)\.
- Eset al\.\(2024\)S\. Es, J\. James, L\. Espinosa\-Anke, and S\. SchockaertRAGAs: automated evaluation of retrieval augmented generation\.InProceedings of EACL: System Demonstrations,Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Geifman and El\-Yaniv \(2017\)Y\. Geifman and R\. El\-YanivSelective classification for deep neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Heet al\.\(2023\)P\. He, J\. Gao, and W\. ChenDeBERTaV3: improving DeBERTa using ELECTRA\-style pre\-training with gradient\-disentangled embedding sharing\.InInternational Conference on Learning Representations,Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1),[Implementation](https://arxiv.org/html/2609.00434#Sx4.SSx4.p1.1)\.
- Holm \(1979\)S\. HolmA simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[Results](https://arxiv.org/html/2609.00434#Sx5.p2.1)\.
- Jianget al\.\(2023\)D\. Jiang, X\. Ren, and B\. Y\. LinLLM\-Blender: ensembling large language models with pairwise ranking and generative fusion\.InProceedings of ACL,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Kimet al\.\(2024\)S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. SeoPrometheus: inducing fine\-grained evaluation capability in language models\.InInternational Conference on Learning Representations,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Labanet al\.\(2022\)P\. Laban, T\. Schnabel, P\. N\. Bennett, and M\. A\. HearstSummaC: re\-visiting NLI\-based models for inconsistency detection in summarization\.Transactions of the Association for Computational Linguistics\.Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Leeet al\.\(2022\)H\. Lee, R\. Gupta, A\. Rastogi, Y\. Cao, B\. Zhang, and Y\. WuSGD\-X: a benchmark for robust generalization in schema\-guided dialogue systems\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[Appendix I](https://arxiv.org/html/2609.00434#A9.p1.1),[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2023\)Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p2.1),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1),[Baselines](https://arxiv.org/html/2609.00434#Sx4.SSx2.p1.1),[Cheaper\-backbone baselines\.](https://arxiv.org/html/2609.00434#Sx7.SSx4.SSS0.Px2.p1.1)\.
- Manakulet al\.\(2023\)P\. Manakul, A\. Liusie, and M\. J\. F\. GalesSelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of EMNLP,Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. W\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of EMNLP,Cited by:[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1)\.
- Onget al\.\(2025\)I\. Ong, A\. Almahairi, V\. Wu, W\. Chiang, T\. Wu, J\. E\. Gonzalez, M\. W\. Kadous, and I\. StoicaRouteLLM: learning to route LLMs from preference data\.InInternational Conference on Learning Representations,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Rastogiet al\.\(2020\)A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. KhaitanTowards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[Table 5](https://arxiv.org/html/2609.00434#A2.T5),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p1.1),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[NLI grounding, TOD evaluation, and abstention\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px2.p1.1),[Benchmark Construction](https://arxiv.org/html/2609.00434#Sx4.SSx1.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-BERT: sentence embeddings using siamese BERT\-networks\.InProceedings of EMNLP\-IJCNLP,Cited by:[Implementation](https://arxiv.org/html/2609.00434#Sx4.SSx4.p1.1)\.
- Songet al\.\(2020\)K\. Song, X\. Tan, T\. Qin, J\. Lu, and T\. LiuMPNet: masked and permuted pre\-training for language understanding\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix C](https://arxiv.org/html/2609.00434#A3.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2020\)W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. ZhouMiniLM: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.InAdvances in Neural Information Processing Systems,Cited by:[Implementation](https://arxiv.org/html/2609.00434#Sx4.SSx4.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, Z\. Yu, W\. Yao, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. ZhangPandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.InInternational Conference on Learning Representations,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,Cited by:[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1)\.
- Williamset al\.\(2018\)A\. Williams, N\. Nangia, and S\. R\. BowmanA broad\-coverage challenge corpus for sentence understanding through inference\.InProceedings of NAACL,Cited by:[Implementation](https://arxiv.org/html/2609.00434#Sx4.SSx4.p1.1)\.
- Yeet al\.\(2022\)F\. Ye, J\. Manotumruksa, and E\. YilmazMultiWOZ 2\.4: a multi\-domain task\-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation\.InProceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue \(SIGDIAL\),Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[Benchmark Construction](https://arxiv.org/html/2609.00434#Sx4.SSx1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2609.00434#Sx1.p2.1),[Introduction](https://arxiv.org/html/2609.00434#Sx1.p5.1),[LLM\-as\-a\-judge and cost\-aware cascades\.](https://arxiv.org/html/2609.00434#Sx2.SS0.SSS0.Px1.p1.1),[Baselines](https://arxiv.org/html/2609.00434#Sx4.SSx2.p1.1)\.

## Appendix AFailure Injection Templates

For each sampled turn we either retain the original \(acceptable\) turn or programmatically inject one failure from𝒞\\mathcal\{C\}by mutating the agent response and the state diff*together*, so the failure is genuinely state\-grounded rather than only surface\-grounded:

- •IUV:rewrite the post\-turn statests\_\{t\}so that a valuevvthe user supplied inutu\_\{t\}is dropped \(the slot remains unfilled or keeps a stale value\), forcing a downstream re\-ask or stalled workflow\.
- •WNF:replaceata\_\{t\}with a request for a slot from another required\-field bucket not legal underNext⁡\(𝒲,st−1\)\\mathrm\{Next\}\(\\mathcal\{W\},s\_\{t\-1\}\)\.
- •PC:replaceata\_\{t\}with a confirmation surface \(“Booking confirmed\.”\) while leavingst−1s\_\{t\-1\}with at least one required slot empty\.
- •UV:inject a value intoΔt\\Delta\_\{t\}that is plausible for the slot type but does not appear inh<t∪\{ut\}h\_\{<t\}\\cup\\\{u\_\{t\}\\\}\.

The same operator implementations are reused across datasets where applicable; MultiWOZ and SGD use all four classes, while ABCD\-UV uses only the UV operator\. Only the WorkflowSpec source differs\.

## Appendix BFull Per\-Method Result Tables

Tables[3](https://arxiv.org/html/2609.00434#A2.T3),[4](https://arxiv.org/html/2609.00434#A2.T4), and[5](https://arxiv.org/html/2609.00434#A2.T5)give the per\-method precision, recall,F1F\_\{1\}, dollar cost, and latency underlying the headline summary \(Table[1](https://arxiv.org/html/2609.00434#Sx5.T1)\); Figure[3](https://arxiv.org/html/2609.00434#A2.F3)plots the UV\-rich cost\-quality frontier\.

Table 3:Headline cost\-quality on two slices\.MW\-mixed: MultiWOZ 2\.4, all four failure classes mixed \(n=250n\{=\}250,126126injected\)\.ABCD\-UV: Action\-Based Conversations Dataset\([Chen et al\. 2021](https://arxiv.org/html/2609.00434#bib.bib27)\), UV\-only injection \(n=223n\{=\}223,103103injected,3030workflows\)\.SAGE\-Core=compiler\+S1\+S2 \(zero paid LLM, recommended\);SAGE\-LLM=SAGE\-Core\+optional S3\. LLM\-judge is the stronger\-mode single\-callgpt\-4\.1verdict per slice \(state\-aware on MW\-mixed; transcript\-only on ABCD\)\. Dollar cost is measured tokens at OpenAI API list prices; SAGE\-Core makes no API call, so its cost is $0\.00 \(its symbolic/on\-device compute is not monetized\)\. Best per column inbold\. Paired bootstrap of SAGE\-Core vs\. the strongest per\-slice baseline: MW\-mixedΔ​F1=\+\.069\\Delta F\_\{1\}\{=\}\{\+\}\.069\(p=\.022p\{=\}\.022\), ABCD\-UV\+\.056\{\+\}\.056\(p=\.044p\{=\}\.044\)\. SAGE\-CoreF1F\_\{1\}95%95\\%bootstrap CI – MW\-mixed\[\.818,\.909\]\[\.818,\.909\], ABCD\-UV\[\.889,\.962\]\[\.889,\.962\]; SAGE\-LLM – MW\-mixed\[\.781,\.879\]\[\.781,\.879\], ABCD\-UV\[\.841,\.929\]\[\.841,\.929\]; LLM\-judge\[\.745,\.845\]\[\.745,\.845\]/\[\.823,\.916\]\[\.823,\.916\], G\-Eval\[\.711,\.835\]\[\.711,\.835\]/\[\.726,\.852\]\[\.726,\.852\], FrugalGPT\[\.683,\.791\]\[\.683,\.791\]/\[\.596,\.718\]\[\.596,\.718\]\.Table 4:Single\-class UV stress test on MultiWOZ 2\.4 \(n=250n\{=\}250,121121injectedunsupported\_valuefailures\)\.SAGE\-Core=compiler\+S1\+S2 \(zero paid LLM\);SAGE\-LLM=SAGE\-Core\+optional S3\. Cost from measured tokens and OpenAI API list prices; latency wall\-clock per\-turn \(SAGE\-Core issues no API call, so its p50 is warm in\-process S1\+S2 compute, not directly comparable to the API\-based rows’ harness wall\-clock\)\.tr\.=transcript\-only,st\.=state\-aware\. Best per column inbold\. Paired bootstrap SAGE\-Core vs\. G\-Eval\-tr \(the strongest baseline here\):Δ​F1=−\.003\\Delta F\_\{1\}\{=\}\{\-\}\.003,95%95\\%CI\[−\.052,\+\.046\]\[\{\-\}\.052,\{\+\}\.046\],p=\.89p\{=\}\.89\(tie\)\.F1F\_\{1\}95%95\\%bootstrap CI: SAGE\-Core\[\.883,\.952\]\[\.883,\.952\], SAGE\-LLM\[\.828,\.913\]\[\.828,\.913\], FrugalGPT\-tr\[\.707,\.813\]\[\.707,\.813\], FrugalGPT\-st\[\.645,\.756\]\[\.645,\.756\], G\-Eval\-tr\[\.885,\.955\]\[\.885,\.955\], G\-Eval\-st\[\.776,\.885\]\[\.776,\.885\]\.Table 5:Cross\-dataset on Schema\-Guided Dialogue\([Rastogi et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib7)\)\(n=280n\{=\}280,151151injected failures,3737\(service, intent\) workflows\)\. Same evaluator code as Table[4](https://arxiv.org/html/2609.00434#A2.T4); only the WorkflowSpec source differs\.SAGE\-Core=zero\-paid\-LLM S1\+S2;SAGE\-LLM=\+optional S3 \(latency/cost as Table[4](https://arxiv.org/html/2609.00434#A2.T4)\)\.tr\.=transcript\-only,st\.=state\-aware\. Paired bootstrap SAGE\-Core vs\. the strongest evaluated GPT\-4\.1 baseline \(FrugalGPT\-tr on the full slice\), re\-selecting the maximum within each resample:Δ​F1=\+\.075\\Delta F\_\{1\}\{=\}\{\+\}\.075,95%95\\%CI\[\+\.015,\+\.102\]\[\{\+\}\.015,\{\+\}\.102\],p=\.011p\{=\}\.011\.F1F\_\{1\}95%95\\%bootstrap CI: SAGE\-Core\[\.820,\.904\]\[\.820,\.904\], SAGE\-LLM\[\.804,\.891\]\[\.804,\.891\], FrugalGPT\-tr\[\.738,\.834\]\[\.738,\.834\], FrugalGPT\-st\[\.726,\.821\]\[\.726,\.821\], G\-Eval\-tr\[\.725,\.837\]\[\.725,\.837\], G\-Eval\-st\[\.724,\.837\]\[\.724,\.837\]\.![Refer to caption](https://arxiv.org/html/2609.00434v1/figures/fig_cost_quality.png)Figure 3:Cost\-quality Pareto on MultiWOZ 2\.4 UV\-rich, all six Table[4](https://arxiv.org/html/2609.00434#A2.T4)configurations \(log\-cost axis; SAGE\-Core’s $0\.00 is clamped to the left edge\)\.SAGE\-Coresits at the Pareto corner – essentially tied onF1F\_\{1\}with the top method at zero paid API cost\. Transcript\-only G\-Eval is the top\-F1F\_\{1\}point but at $4\.70/1k and12\.012\.0s median latency \(vs\. SAGE\-Core’s $0 paid API cost and0\.030\.03s warm local latency;≈14×\\approx 14\\timesthe cost and≈44×\\approx 44\\timesthe latency relative to SAGE\-LLM\); SAGE\-LLM \(adding S3\) is dominated by SAGE\-Core here\. State\-aware variants of both LLM judges fall below their transcript\-only counterparts\.![Refer to caption](https://arxiv.org/html/2609.00434v1/figures/fig_pr_f1.png)Figure 4:Precision, recall, andF1F\_\{1\}on MultiWOZ 2\.4 UV\-rich \(n=250n\{=\}250\); exact values in Table[4](https://arxiv.org/html/2609.00434#A2.T4)\. SAGE\-Core \(zero paid LLM\) attains the second\-highestF1F\_\{1\}, statistically tied with transcript\-only G\-Eval, which holds the topF1F\_\{1\}on this single\-class slice but at $4\.70/1k and12\.012\.0s median latency \(vs\. SAGE\-Core’s $0 and0\.030\.03s;≈14×\\approx 14\\timescost and≈44×\\approx 44\\timeslatency relative to SAGE\-LLM\); enabling S3 \(SAGE\-LLM\) does not help here\.
## Appendix CRouter Robustness Details

This appendix expands the “Oracle routing” paragraph in the Discussion\. We measured router robustness on the SGD cross\-dataset slice \(n=296n\{=\}296, headline injectors\)\. Two zero\-shot routers were evaluated end\-to\-end against the gold \(service, intent\) oracle used in the headline tables:

#### MPNet bi\-encoder\.

A pretrained MPNet\([Song et al\. 2020](https://arxiv.org/html/2609.00434#bib.bib15)\)bi\-encoder \(sentence\-transformers/all\-mpnet\-base\-v2, revisione8c3b32\) encodes the schema’s service\+intent natural\-language descriptions and the dialogue prefix; cosine similarity selects one of the3737\(service, intent\) candidates\. Top\-1 accuracy on the SGD slice is0\.4760\.476; top\-8 accuracy is0\.8920\.892\.

#### LLM\-on\-shortlist\.

gpt\-4\.1\-miniis prompted with the MPNet top\-88candidates \(rendered as schema headers\) and the dialogue prefix, and constrained to pick one\. Top\-1 accuracy is0\.8040\.804\. This pattern is a cheap distillation: the bi\-encoder narrows the search space; the LLM disambiguates the residual\.

#### End\-to\-end results\.

We report the recommendedSAGE\-Core\(S1\+S2,no\_S3\) configuration; it issues no S3 calls and is therefore deterministic on fixed turns \(no regeneration band\)\.F1F\_\{1\}on thisn=296n\{=\}296SGD slice:

- •Oracle routing:F1=0\.861F\_\{1\}=0\.861
- •LLM\-on\-shortlist router:F1=0\.861F\_\{1\}=0\.861
- •MPNet router:F1=0\.861F\_\{1\}=0\.861

All three coincide to within<0\.001<0\.001\(P=0\.887P\{=\}0\.887,R=0\.836R\{=\}0\.836throughout\): the router choice does not move SAGE\-Core’sF1F\_\{1\}on this slice\. On the155155MPNet\-misrouted turns specifically SAGE\-Core still reachesF1=0\.862F\_\{1\}=0\.862; on the5858LLM\-router\-misrouted turns it reaches0\.8360\.836\. For reference, the full\-cascadeSAGE\-LLMexhibits the same invariance at nonzero API cost \(0\.854/0\.852/0\.8550\.854/0\.852/0\.855for oracle/LLM/MPNet\)\. Thesen=296n\{=\}296numbers are not directly comparable to then=280n\{=\}280headline SGDF1F\_\{1\}\(SAGE\-Core0\.8640\.864, Table[5](https://arxiv.org/html/2609.00434#A2.T5)\); the slice differs\.

#### Why misrouting barely hurts\.

SGD intent confusions are overwhelmingly intra\-service: services likeRestaurants\_1andRestaurants\_2share most slot names;Travel\_1confusions are between sub\-intents \(FindAttractionsvs\.GetAttractionInfo\) with overlapping slot spaces\. A misrouted scenario therefore shares the slot space of the correct one, so the UV grounding check \(values must be entailed by history\) and most S1 transition rules still fire on the right signals\. SAGE’s quality is governed by the per\-criterion verifier cascade rather than by workflow selection: in our tested SGD setting, even the MPNet router at0\.4760\.476top\-1 accuracy produced no measurable SAGE\-CoreF1F\_\{1\}degradation, because most routing errors stayed within services with overlapping slot spaces\.

#### Implication for the head\-to\-head\.

Because oracle routing introduces<0\.003<0\.003F1F\_\{1\}bias on the SAGE side, and the transcript\-only baselines do not consume schema at all \(state\-aware variants consume the same schema/state context SAGE sees but do not route per\-criterion\), the SAGE/G\-Eval/FrugalGPT comparison in Tables[4](https://arxiv.org/html/2609.00434#A2.T4)–[5](https://arxiv.org/html/2609.00434#A2.T5)is not confounded by the oracle\. Removing the oracle from SAGE and adding a router to the baselines would not change the relative ranking\.

## Appendix DCheaper\-Backbone Judge Comparison

A natural objection to the headline cost story is that SAGE’s advantage might be a*model\-class*artifact: the holistic baselines use GPT\-4\.1, and a cheaper backbone would close the cost gap without changing the ranking\. To test this we re\-ran G\-Eval on the≈5×\\approx 5\\timescheaper \(at matched token usage\)gpt\-4\.1\-minibackbone, in both state\-aware and transcript\-only prompt variants, on the identical MultiWOZ UV\-rich and SGD headline slices \(Table[6](https://arxiv.org/html/2609.00434#A4.T6); SAGE rows use the headline\-slice per\-turn verdicts, not the regenerated ablation slice, so all rows in the table are on one generation\)\. Swapping the backbone cuts the holistic judge’s cost to $0\.9–1\.6/1k, versus $4\.7–8\.0/1k for the GPT\-4\.1 judge \(depending on prompt mode and slice;≈5×\\approx 5\\timescheaper at matched token usage\), and it materially changes the baseline ranking on SGD: the state\-aware mini judge reaches0\.8590\.859, overtaking our pre\-specified strongest SGD baseline \(FrugalGPT\-tr,0\.7890\.789\)\. It is therefore the strongest SGD judge in our evaluation\. Even so, a paired bootstrap \(SAGE\-Core vs\. this judge, same turns\) shows only a statistical*tie*, not a loss: SGDΔ​F1=\+\.005\\Delta F\_\{1\}\{=\}\{\+\}\.005\(0\.8640\.864vs\.0\.8590\.859,95%95\\%CI\[−\.045,\+\.054\]\[\{\-\}\.045,\{\+\}\.054\],p=\.83p\{=\}\.83\); on MultiWOZ the best cheap judge \(0\.9010\.901, transcript\-only\) stays below SAGE\-Core \(0\.9200\.920;Δ​F1=\+\.019\\Delta F\_\{1\}\{=\}\{\+\}\.019,\[−\.033,\+\.071\]\[\{\-\}\.033,\{\+\}\.071\],p=\.46p\{=\}\.46\)\. These two reduced\-cost paired tests are uncorrected, but both are ties, so multiplicity does not affect the conclusion\. No reduced\-cost judge*exceeds*SAGE\-Core on either slice, while SAGE\-Core issues*zero*paid API calls\.

To keep the “no judge on a cheaper backbone exceeds SAGE\-Core” claim load\-bearing on*all four*headline slices rather than only the two single\-class stress slices in Table[6](https://arxiv.org/html/2609.00434#A4.T6), we also ran both mini variants on the two multi\-class slices \(MW\-mixed and ABCD\-UV\)\. There the gap widens rather than narrows: the best reduced\-cost mini judge reaches only0\.7590\.759on MW\-mixed \(transcript\-only\) and0\.7770\.777on ABCD \(state\-aware\), well below SAGE\-Core’s0\.8660\.866and0\.9290\.929\. This holds even on MW\-mixed, the slice where state context*helped*the GPT\-4\.1 judge \(\+0\.063\+0\.063\); the state\-aware mini judge there scores just0\.6980\.698\. The paired bootstrap confirms SAGE\-Core wins significantly on both \(MW\-mixedΔ​F1=\+\.107\\Delta F\_\{1\}\{=\}\{\+\}\.107,p=\.001p\{=\}\.001; ABCDΔ​F1=\+\.152\\Delta F\_\{1\}\{=\}\{\+\}\.152,p<\.001p\{<\}\.001\)\. Summarising across all four slices: no reduced\-cost mini judge exceeds SAGE\-Core anywhere – it ties the strongest mini judge on the two single\-class stress slices \(UV\-rich, SGD\) and significantly beats it on the two multi\-class slices, at $0\.00\.

The cheaper backbone also flips the state\-context effect: the mini judge is*helped*by state context on SGD \(→0\.8590\.789\\\!\\to\\\!0\.859,\+0\.070\{\+\}0\.070\), the reverse of the GPT\-4\.1 judges, which are*hurt*by it \(up to−0\.09\-0\.09\)\. This scopes our “state context lowersF1F\_\{1\}” finding to the GPT\-4\.1 prompt implementation rather than to holistic judges in general, and correspondingly narrows the “architectural, not informational” argument \(Discussion\): the defensible claim is that even the best\-informed, best\-backboned holistic judge does not significantly exceed SAGE\-Core\. Because SAGE\-Core \(S1\+S2\) also matches or beats the full SAGE\-LLM cascade, theF1F\_\{1\}gap is*not merely a backbone artifact*and is consistent with the architectural benefit of state\-grounded criterion decomposition rather than the choice of LLM backbone\.

Table 6:Reduced\-cost LLM\-judge comparison\.Replacing G\-Eval’s GPT\-4\.1 backbone with the≈5×\\approx 5\\timescheaper \(at matched token usage\) GPT\-4\.1\-mini \(in both state\-awarest\.and transcript\-onlytr\.prompt variants\) cuts the holistic judge’s cost to $0\.9–1\.6/1k, versus $4\.7–8\.0/1k for GPT\-4\.1, yet no reduced\-cost judge exceeds SAGE\-Core’sF1F\_\{1\}on either slice: the best cheap judge reaches0\.9010\.901on MultiWOZ \(vs\. SAGE\-Core0\.9200\.920\) and0\.8590\.859on SGD \(vs\.0\.8640\.864\), while SAGE\-Core issues*zero*paid API calls\. On SGD the state\-aware mini judge \(0\.8590\.859\) overtakes the pre\-specified FrugalGPT baseline but only*ties*SAGE\-Core \(paired bootstrapp=\.83p\{=\}\.83\); no reduced\-cost judge*exceeds*SAGE\-Core on either slice, so the gap is*not merely a backbone artifact*and is consistent with SAGE’s state\-grounded architecture\. p50 \(s\) is the per\-turn headline\-harness wall time \(consistent with Tables[4](https://arxiv.org/html/2609.00434#A2.T4)–[5](https://arxiv.org/html/2609.00434#A2.T5)\); SAGE rows are warm in\-process compute and are not directly comparable to the API rows’ harness wall\-clock\.
## Appendix EStructured Repair Signals

Every SAGE verdict carries an evidence trace, and for a detected failure the aggregator emits a*repair signal*: the concrete state edit or clarifying action that would resolve the violated criterion\. These are a side output of the per\-criterion decomposition – we do not evaluate a repair\-execution loop – but they illustrate what the evidence trace makes actionable\. Table[7](https://arxiv.org/html/2609.00434#A5.T7)shows representative examples drawn from the MultiWOZ and SGD traces, one per failure class\.

Table 7:Representative structured repair signals emitted alongside SAGE evidence traces \(illustrative side output; not part of the quantitative evaluation\)\. Class codes: IUV = ignored user value, PC = premature confirmation, UV = unsupported value\.
## Appendix FPer\-Tier Ablation

Table[8](https://arxiv.org/html/2609.00434#A6.T8)reports per\-tier leave\-one\-out ablations\. S1 is the only tier compiled for WNF, PC, and IUV; on SGD, disabling S1 drops recall→0\.6820\.848\\\!\\to\\\!0\.682, with WNF/IUV collapsing to zero\. On MultiWOZ, by contrast, removing S1*raises*F1F\_\{1\}\(→0\.9110\.870\\\!\\to\\\!0\.911\): the partial slot ordering in MultiWOZ makes some S1 next\-field checks fire conservatively on legal turns, so dropping S1 removes those false positives; on SGD, whose schema ordering is cleaner, S1 remains necessary for WNF/IUV recall\. S2 carries the precision floor: removing it drops precision→0\.660\.87\\\!\\to\\\!0\.66on SGD and→0\.610\.79\\\!\\to\\\!0\.61on MultiWOZ\. On both injected slices theSAGE\-Core\(S1\+S2,no\_S3\) configuration is stronger thanSAGE\-LLM\(full\) – this is the deployed operating point; a pinned\-config rerun of ABCD confirms the same pattern \(SAGE\-LLMF1=0\.888→F\_\{1\}\{=\}0\.888\\toSAGE\-Core0\.9290\.929, the latter at zero LLM cost\)\. S3 is retained as an extensibility hook for open\-class failures not expressible in the compiled criteria; on the injected slices in this paper it does not net positive, and we do not claim it as a load\-bearing contribution\.

Table 8:Per\-tier leave\-one\-out ablation on both benchmark slices\.F1F\_\{1\}over the full set of original turns; orphaned criteria abstain and any gold failure they would catch is a miss\. Latency is per\-turn wall time on the current dev box; absolute values are higher than Table[4](https://arxiv.org/html/2609.00434#A2.T4)because the headline run used a faster machine\. This ablation was run on an independently regenerated slice \(same sampled turns, pipeline re\-run\), so absoluteF1F\_\{1\}can differ from Tables[1](https://arxiv.org/html/2609.00434#Sx5.T1)–[5](https://arxiv.org/html/2609.00434#A2.T5)by up to0\.0060\.006\(the run\-to\-run regeneration band, e\.g\. SGDfull0\.8560\.856here vs\.0\.8500\.850in Table[5](https://arxiv.org/html/2609.00434#A2.T5)\); this band comes entirely from S3/API stochasticity in thefullconfig, whereas SAGE\-Core \(no\_S3\) is bitwise deterministic on fixed turns and its P/R/F1F\_\{1\}are therefore identical across generations\. The ablation should be read for within\-table tier contributions rather than absolute cross\-table comparison\. Best per slice in bold\.full=SAGE\-LLM\(S1\+S2\+S3, the full\-cascade config\);no\_S3=SAGE\-Core\(S1\+S2\), the recommended zero\-paid\-LLM operating point for compiled taxonomies, which matches or exceeds SAGE\-LLM on both slices at $0\.00\.F1F\_\{1\}95%95\\%bootstrap CIs \(MW//SGD\): full\[\.82,\.91\]/\[\.81,\.90\]\[\.82,\.91\]/\[\.81,\.90\], no\_S1\[\.87,\.94\]/\[\.74,\.85\]\[\.87,\.94\]/\[\.74,\.85\], no\_S2\[\.68,\.79\]/\[\.67,\.78\]\[\.68,\.79\]/\[\.67,\.78\], no\_S3\[\.88,\.95\]/\[\.82,\.90\]\[\.88,\.95\]/\[\.82,\.90\]\.
## Appendix GCascade Routing and Cost Anatomy

Table[9](https://arxiv.org/html/2609.00434#A7.T9)reports how SAGE\-LLM’s cascade resolves criteria, making the “abstention\-aware” mechanism concrete \(under SAGE\-Core the S3\-routed criteria abstain instead of being decided by a paid LLM call; Table[10](https://arxiv.org/html/2609.00434#A8.T10)reports the resulting undecided rates\)\. Across all four slices the majority of criteria are decided at zero API cost:5656–65%65\\%by S1’s symbolic rules and a further2525–31%31\\%by S2’s on\-device encoder, leaving99–19%19\\%that escalate to the focused\-LLM stage S3\. Between24%24\\%and53%53\\%of*turns*invoke S3 at all, giving0\.240\.24–0\.530\.53paid LLM calls per turn for SAGE\-LLM versus the one\-or\-more full\-transcript calls every holistic judge pays; SAGE\-Core makes zero\. On the multi\-criteria MultiWOZ/SGD slices this escalation profile makes SAGE\-LLM roughly1212–22×22\\timescheaper than the GPT\-4\.1 G\-Eval judge \(depending on prompt mode and slice\), and SAGE\-Core removes even that residual cost\. On the UV\-only ABCD slice a larger share of criteria are open\-class and reach S3, so SAGE\-LLM is only marginally cheaper than the single\-call judge \($1\.37\\$1\.37vs\.$1\.52\\$1\.52/1k\); SAGE\-Core decides ABCD at no API cost and*higher*F1F\_\{1\}\(0\.9290\.929vs\.0\.8880\.888\), which is why we recommend it\.

Table 9:Cascade routing per slice: share of compiled criteria resolved at each stage \(S1 symbolic / S2 encoder\-NLI / S3 focused\-LLM\), fraction of turns with a criterion resolved at S3, and mean paid LLM calls per turn\. S1 and S2 incur zero API cost; only S3 calls are billed\.
## Appendix HSelective Evaluation

The confidence gate that routes low\-margin S2 decisions upward is also a turn\-level abstention signal: assigning each turn the minimum decision margin across its decided criteria \(deterministic S1 decisions score maximal margin\) yields a ranking whose lowest\-confidence tail can be declined\. Table[10](https://arxiv.org/html/2609.00434#A8.T10)reports these numbers on theSAGE\-Core \(S1\+S2,no\_S3\) operating point– the recommended deployed configuration, which is also the SAGE\-Core headline rows of Tables[1](https://arxiv.org/html/2609.00434#Sx5.T1)–[5](https://arxiv.org/html/2609.00434#A2.T5)\(*not*the SAGE\-LLM rows\)\. TheseF1​@​100%F\_\{1\}@100\\%values are computed on the identical per\-turn slices used for every baseline \(same turn IDs and gold labels, verified row\-for\-row\); because the S1\+S2 pipeline is deterministic given the compiled plan, they coincide with the deterministicno\_S3leave\-one\-out ablation \(Table[8](https://arxiv.org/html/2609.00434#A6.T8)\) to within that ablation’s stated regeneration band, and for ABCD the reproduction anchor is the pinned\-config S1\+S2 rerun \(0\.9290\.929\)\. We report the full\-coverageF1F\_\{1\}\(undecided→\\toacceptable\), the all\-abstain*undecided*rate, and selectiveF1F\_\{1\}when the least\-confident10%10\\%/20%20\\%of turns are declined\. Declining the least\-confident decile raisesF1F\_\{1\}on the three multi\-margin slices \(MW\-mixed\+0\.023\{\+\}0\.023, SGD\+0\.009\{\+\}0\.009, UV\-rich\+0\.008\{\+\}0\.008\): the signal is*calibrated*– the turns SAGE is least sure about are disproportionately the ones it gets wrong – so an operator can trade coverage for reliability along a risk–coverage frontier rather than being forced into a single operating point\. ABCD behaves differently and instructively: its10\.8%10\.8\\%undecided turns are*all*gold\-acceptable \(so full\-coverage scoring is lossless there andF1​@​90%F\_\{1\}@90\\%is unchanged at0\.9290\.929\), while its residualF1F\_\{1\}errors are*confident*symbolic S1 decisions rather than low\-margin S2 ones – so declining the low\-confidence tail cannot reorder them andF1F\_\{1\}is flat\-to\-slightly\-negative \(−0\.009\{\-\}0\.009at80%80\\%coverage\)\. Selective evaluation thus buys reliability precisely when the residual errors live in the low\-margin region \(the encoder\-heavy slices\), and correctly abstains\-without\-improving when they are confident symbolic decisions \(ABCD\)\. This realises the abstention\-aware aggregator as a tunable selective evaluator, not merely a cost\-routing device\.

Table 10:Selective evaluation of theSAGE\-Core \(S1\+S2,no\_S3\) operating point–*not*the SAGE\-LLM headline config \(Tables[1](https://arxiv.org/html/2609.00434#Sx5.T1)–[5](https://arxiv.org/html/2609.00434#A2.T5)\)\.Undec\.=fraction of turns where every compiled criterion abstains \(mapped to acceptable under full coverage\)\.F1​@​cF\_\{1\}@c=F1F\_\{1\}on the most\-confidentccof turns, ranked by minimum per\-criterion decision margin\.F1​@​100%F\_\{1\}@100\\%is computed on the identical per\-turn slices as all baselines \(verified row\-for\-row\); given the deterministic S1\+S2 pipeline it coincides with theno\_S3rows of Table[8](https://arxiv.org/html/2609.00434#A6.T8)\(MW/SGD\) within that ablation’s regeneration band, and matches the pinned S1\+S2 ABCD rerun\. Declining the least\-confident decile improvesF1F\_\{1\}on the three encoder\-heavy slices; ABCD’s undecided turns are all acceptable and its residual errors are confident symbolic decisions, so its curve is flat\-to\-slightly\-negative\.
## Appendix ISchema\-Paraphrase Robustness \(SGD\-X\)

To test whether SAGE\-Core’sF1F\_\{1\}rests on schema\-grounded reasoning rather than canonical\-name overlap, we replay the recommendedSAGE\-Core\(S1\+S2,no\_S3\) configuration on SGD\-X\([Lee et al\. 2022](https://arxiv.org/html/2609.00434#bib.bib8)\)\(55paraphrased schema variants, samen=280n\{=\}280slice\)\. MeanF1F\_\{1\}\(σ\\sigma, invariant\-verdict share\): SAGE\-Core0\.6280\.628\(0\.0020\.002,99\.3%99\.3\\%\), FrugalGPT0\.5870\.587\(0\.0050\.005,74\.3%74\.3\\%\), G\-Eval0\.4980\.498\(0\.0150\.015,93\.2%93\.2\\%\)\. All three drop relative to canonical SGD, but G\-Eval’s drop is largest \(→0\.4980\.784\\\!\\to\\\!0\.498\); SAGE\-Core has≈37×\\approx 37\\timesfewer cross\-variant disagreements than FrugalGPT and≈10×\\approx 10\\timesfewer than G\-Eval\. SAGE\-Core’s own absolute drop \(→0\.6280\.864\\\!\\to\\\!0\.628, relative to its canonical SGDF1F\_\{1\}in Table[5](https://arxiv.org/html/2609.00434#A2.T5)\) comes almost entirely from S2: paraphrased slot names weaken the NLI encoder’s entailment margins on UV/IUV criteria, lowering recall, while S1’s symbolic checks – keyed to slot*identity*rather than surface name – stay stable\. The claim is thus low*variance*across paraphrases \(σ=0\.002\\sigma\{=\}0\.002\), not paraphrase\-immune accuracy\.

## Appendix JReproducibility Details

#### Hyperparameter values tried and selection criterion\.

SAGE has two tunable numbers, both in S2\. The entailment margin thresholdτ\\tauonm=p⁡\(ent\)−p⁡\(contra\)m=p\(\\text\{ent\}\)\-p\(\\text\{contra\}\)was selected asτ=0\.20\\tau\{=\}0\.20by maximisingF1F\_\{1\}on the4040\-turn development split disjoint from every evaluation slice, and then applied unchanged to MultiWOZ, SGD, and ABCD\. As a post\-hoc robustness check we additionally sweptτ\\tauover seven values –\{0\.05,0\.10,0\.15,0\.20,0\.25,0\.30,0\.40\}\\\{0\.05,0\.10,0\.15,0\.20,0\.25,0\.30,0\.40\\\}– on the full UV\-rich slice \(n=250n\{=\}250\), the slice that most exercises the entailment path\.F1F\_\{1\}varied only within\[0\.881,0\.888\]\[0\.881,0\.888\]\(a range of0\.0070\.007, with precision0\.8100\.810–0\.8220\.822and recall constant at0\.9650\.965\), and cost varied within$0\.33\\$0\.33–$0\.37\\$0\.37per11k turns as the abstention rate shifted the S3 escalation volume\. The reported operating point is therefore not a tuned optimum: any threshold in the swept range reproduces the headline conclusion\. The extraction\-confidence floor was fixed at0\.50\.5a priori and not swept\. Baseline thresholds \(the G\-Eval\-style judge’s cut on the11–55acceptability scale and the FrugalGPT\-style cascade’s escalation confidence\) were likewise fixed on the development split and held constant across slices\.

#### Number of runs per reported result\.

Every evaluator is executed exactly once per slice\. This is the correct protocol here because all components are deterministic under the settings above: the symbolic compiler and S1 are pure functions of the schema and state diff, the S2 encoders are deterministic at pinned revisions, and every LLM call \(S3 and all three baseline families\) runs at temperature00against a pinned model snapshot\. Reported uncertainty therefore comes from resampling turns –10,00010\{,\}000bootstrap replicates at seed00– rather than from repeated runs\. The single exception is the G\-Eval\-style judge, which drawsn=6n\{=\}6samples per turn*within*its single run by construction; those samples are aggregated into one score before scoring, so it too contributes one verdict per turn\.

#### Computing infrastructure\.

All symbolic and encoder computation ran on a single laptop\-class CPU: an Intel Core Ultra 7 268V \(88cores,88threads\) with31\.731\.7GB RAM under Windows 11 Enterprise \(build10\.0\.2610010\.0\.26100\)\. No GPU was used; the S2 encoders and the MPNet router run on CPU under a CPU\-only PyTorch build, which is why SAGE\-Core’s warm local p50 latency of0\.030\.03s is achievable without accelerator hardware and why we report its compute as unmonetized\. The software stack is Python3\.11\.93\.11\.9, PyTorch2\.12\.02\.12\.0\(CPU build\), Transformers5\.8\.15\.8\.1, and Sentence\-Transformers5\.5\.15\.5\.1\. S3 and all LLM baselines call Azure\-hostedgpt\-4\.1/gpt\-4\.1\-mini\(snapshot2025\-04\-14\) over the network, so their latencies include API round\-trip time and are not CPU\-bound\.

## Appendix KPer\-Evaluator IUV Detection

On the3030audited IUV turns, detection trades sharply against precision: transcript\-only LLM judges catch≤2/30\\leq 2/30, a state\-aware LLM judge7/307/30, a reduced\-cost state\-aware GPT\-4\.1\-mini judge13/3013/30\(at a9%9\\%clean\-turn flag rate\), SAGE\-Core17/3017/30at an8%8\\%clean\-turn flag rate, and a trigger\-happy baseline27/3027/30only by flagging70%70\\%of clean turns\. No evaluator dominates; SAGE\-Core sits at the high\-precision end of this frontier, which is why we report it as the recommended operating point rather than claiming it uniquely detects IUV\.