TRACER: Early Failure Detection for Task-Oriented Dialogue

# TRACER: Early Failure Detection for Task-Oriented Dialogue
Source: [https://arxiv.org/html/2607.03974](https://arxiv.org/html/2607.03974)
Anthony Rios The University of Texas at San Antonio \{erfan\.nourbakhsh, anthony\.rios\}@utsa\.edu

###### Abstract

Task\-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has already gone wrong\. We present TRACER, a method for early failure detection in task\-oriented dialogue\. TRACER predicts from a partial dialogue whether the full conversation will eventually fail by combining simple trajectory signals from belief\-state changes with text representations of the evolving dialogue state\. We evaluate the method in both oracle and generated belief\-state settings, and test how well it works when only 25%, 50%, 75%, or 100% of the dialogue is visible\. Across these settings, TRACER detects useful failure signals well before the end of the conversation and outperforms heuristic, classical, and single\-stream baselines\. These results suggest that early failure detection can provide a practical warning signal for dialogue systems before the interaction fully breaks down\. Source code can be found here:[https://github\.com/erfan\-nourbakhsh/TRACER](https://github.com/erfan-nourbakhsh/TRACER)\.

![[Uncaptioned image]](https://arxiv.org/html/2607.03974v1/figures/logo.png)TRACER: Early Failure Detection for Task\-Oriented Dialogue

Erfan Nourbakhsh, Rocky Slavin, Ke Yang, and Anthony RiosThe University of Texas at San Antonio\{erfan\.nourbakhsh, anthony\.rios\}@utsa\.edu

![Refer to caption](https://arxiv.org/html/2607.03974v1/x1.png)Figure 1:Task overview\. The example shows a conversation breaks down\.## 1Introduction

Task\-oriented dialogue \(TOD\) systems are designed to help users complete concrete goals, booking travel, finding restaurants, resolving customer\-service queries, and they power an expanding range of commercial applications\(Young et al\.,[2013](https://arxiv.org/html/2607.03974#bib.bib26); Wen et al\.,[2017](https://arxiv.org/html/2607.03974#bib.bib23); Qin et al\.,[2023](https://arxiv.org/html/2607.03974#bib.bib18)\)\. As these systems are deployed in ever\-more\-demanding settings, their ability to complete user goals reliably has become a practical engineering requirement that goes beyond benchmark scores\(Walker et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib22); Peng et al\.,[2021](https://arxiv.org/html/2607.03974#bib.bib17)\)\. Large\-scale benchmarks such as MultiWOZ\(Budzianowski et al\.,[2018](https://arxiv.org/html/2607.03974#bib.bib2)\), the Schema\-Guided Dialogue \(SGD\) dataset\(Rastogi et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib20)\), and ABCD\(Chen et al\.,[2021](https://arxiv.org/html/2607.03974#bib.bib5)\)have fueled rapid progress in dialogue state tracking \(DST\)\(Henderson et al\.,[2014](https://arxiv.org/html/2607.03974#bib.bib7); Wu et al\.,[2019](https://arxiv.org/html/2607.03974#bib.bib25),[2020](https://arxiv.org/html/2607.03974#bib.bib24)\), end\-to\-end neural response generation\(Hosseini\-Asl et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib9); Lei et al\.,[2018](https://arxiv.org/html/2607.03974#bib.bib11)\), and, more recently, LLM\-based dialogue agents\(Hudeček and Dusek,[2023](https://arxiv.org/html/2607.03974#bib.bib10)\)\. Yet despite this progress, the dominant evaluation paradigm measures*outcome*rather than*trajectory*: whether the final state is correct, whether the final response is appropriate, or whether the overall task succeeded\. For practical deployment this is often too late\.

As shown in Figure[1](https://arxiv.org/html/2607.03974#S0.F1), many failures in TOD do not appear as a single abrupt mistake; they emerge*progressively*through slot\-value instability, unresolved contradictions, missing constraints, and loss of task focus\. By the time the final outcome is measured, the dialogue may already have drifted too far for a lightweight intervention to help\.

Several lines of research address related but distinct problems\. PARADISE\(Walker et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib22)\)established that task success rates underspecify dialogue quality, while breakdown detection\(Higashinaka et al\.,[2016](https://arxiv.org/html/2607.03974#bib.bib8)\)and TD\-EVAL\(Acikgoz et al\.,[2025](https://arxiv.org/html/2607.03974#bib.bib1)\)target individual turn quality or post\-hoc analysis of completed dialogues\. In open\-domain conversation,Chang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2607.03974#bib.bib4)\)showed that sequential models over growing contexts can forecast derailment before breakdown, establishing partial\-context prediction as principled and achievable\. Clarification and repair\(Carletta,[1992](https://arxiv.org/html/2607.03974#bib.bib3); Maier et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib14); Ngo et al\.,[2024](https://arxiv.org/html/2607.03974#bib.bib16)\), deep RL\(Li et al\.,[2016](https://arxiv.org/html/2607.03974#bib.bib12)\), task pretraining\(Wu et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib24)\), and unified generative models\(Hosseini\-Asl et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib9)\)have each improved dialogue quality or policy, but none defines intervention policies targeting trajectory\-level failure\.

Despite these advances, a critical gap remains:*there is no principled framework for forecasting impending failure from partial task\-oriented dialogue context and converting those forecasts into triggered recovery decisions*\. Breakdown detection\(Higashinaka et al\.,[2016](https://arxiv.org/html/2607.03974#bib.bib8)\)predicts whether a target system utterance, given its preceding dialogue context, causes a dialogue breakdown\. It does not forecast eventual task failure from partial dialogue prefixes or define an intervention policy over failure scores; post\-hoc evaluators\(Walker et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib22); Acikgoz et al\.,[2025](https://arxiv.org/html/2607.03974#bib.bib1)\)analyze completed dialogues; derailment forecasting\(Chang and Danescu\-Niculescu\-Mizil,[2019](https://arxiv.org/html/2607.03974#bib.bib4)\)relies on social signals rather than belief\-state evolution \(e\.g\., if a user is acting as troll\); and LLM\-based agents\(Hudeček and Dusek,[2023](https://arxiv.org/html/2607.03974#bib.bib10)\)do not model failure trajectories or define thresholded intervention policies\.

We present a dual\-stream framework that addresses this gap by combining trajectory\-level signals of belief\-state evolution with serialized belief\-state text to forecast impending dialogue failure from partial context\. A development\-tuned threshold converts failure scores into intervention decisions, enabling a principled study of early\-warning timing and recovery quality\.

This paper makes three contributions: \(1\)We formulate task\-oriented dialogue failure prediction as a*partial\-dialogue forecasting*problem\.We introduceTRACER, a dual\-stream model that predicts whether a task\-oriented dialogue is likely to fail before the conversation ends\. It combines trajectory features from dialogue\-state changes with serialized belief\-state text\. \(2\) We evaluate TRACER on multiple task\-oriented dialogue benchmarks\. The results show that useful failure signals appear from as little as 25% of the dialogue, and that TRACER outperforms heuristic, classical, and single\-stream baselines on early failure prediction\.

![Refer to caption](https://arxiv.org/html/2607.03974v1/x2.png)Figure 2:Overview of TRACER\. Partial dialogues are encoded by a belief\-state text stream and a temporal trajectory stream\. The text stream uses RoBERTa to encode serialized belief states, while the trajectory stream models turn\-level dialogue\-state dynamics\. Their representations are fused to predict failure risk and decide whether to trigger recovery\.
## 2Related Work

#### TOD systems, DST, and pretrained encoders\.

MultiWOZ\(Budzianowski et al\.,[2018](https://arxiv.org/html/2607.03974#bib.bib2)\), SGD\(Rastogi et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib20)\), and ABCD\(Chen et al\.,[2021](https://arxiv.org/html/2607.03974#bib.bib5)\)established multi\-domain benchmarks for state tracking, response generation, and task completion\. DST progress has been driven by cross\-domain generalization\(Wu et al\.,[2019](https://arxiv.org/html/2607.03974#bib.bib25)\), task\-specific pretraining\(Wu et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib24)\), and sequence\-to\-sequence formulations\(Hosseini\-Asl et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib9)\), with end\-to\-end surveys\(Qin et al\.,[2023](https://arxiv.org/html/2607.03974#bib.bib18)\)documenting the shift toward LLM\-based agents\(Peng et al\.,[2021](https://arxiv.org/html/2607.03974#bib.bib17); Hudeček and Dusek,[2023](https://arxiv.org/html/2607.03974#bib.bib10)\)\. BERT\(Devlin et al\.,[2019](https://arxiv.org/html/2607.03974#bib.bib6)\)and RoBERTa\(Liu et al\.,[2019](https://arxiv.org/html/2607.03974#bib.bib13)\)underpin strong dialogue encoders; TOD\-BERT\(Wu et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib24)\)extends this with dialogue\-specific objectives\. TRACER treats belief states as*input*, using RoBERTa to encode their evolving content while a trajectory stream captures temporal dynamics not accessible from text alone\.

#### Failure detection, forecasting, and repair\.

PARADISE\(Walker et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib22)\)showed that dialogue quality cannot be reduced to a single end\-of\-dialogue label; the Dialogue Breakdown Detection Challenge\(Higashinaka et al\.,[2016](https://arxiv.org/html/2607.03974#bib.bib8)\)operationalized turn\-level breakdown detection; and TD\-EVAL\(Acikgoz et al\.,[2025](https://arxiv.org/html/2607.03974#bib.bib1)\)demonstrated that multi\-granularity analysis exposes errors aggregate metrics miss\.Chang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2607.03974#bib.bib4)\)showed that sequential models over growing contexts can forecast conversational derailment before breakdown, establishing partial\-context prediction as a principled framing\. TRACER adapts this to TOD by grounding forecasting signals in belief\-state evolution rather than social cues and couples prediction with a thresholded intervention policy\. On the recovery side, clarification and repair are classical coherence mechanisms\(Carletta,[1992](https://arxiv.org/html/2607.03974#bib.bib3); Maier et al\.,[1997](https://arxiv.org/html/2607.03974#bib.bib14)\);McRoy and Hirst \([1995](https://arxiv.org/html/2607.03974#bib.bib15)\)showed that abductive inference over mismatched speech acts can help identify/repair problems in task\-oriented settings, an early computational treatment of failure; recent work analyzing precursor cues\(Ngo et al\.,[2024](https://arxiv.org/html/2607.03974#bib.bib16)\); we build on this by asking whether an automated forecaster can supply a reliable trigger, comparing a template\-based approach against a fine\-tuned T5 generator\(Raffel et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib19)\)\.

## 3Methodology

Figure[1](https://arxiv.org/html/2607.03974#S0.F1)shows the basic idea behind TRACER\. In many task\-oriented dialogues, failure does not come from a single bad turn\. Instead, the dialogue gradually moves off track through state instability, contradictions, missing constraints, or domain confusion\. Our goal is to detect this rising failure risk early, before the final breakdown becomes obvious, so that the system can decide whether to intervene\. We therefore cast the problem as*partial\-dialogue forecasting*: given only the dialogue observed so far, the model predicts whether the full dialogue will eventually fail\.

We formulate task\-oriented dialogue failure prediction as a*partial\-dialogue\-based forecasting*problem\. Let a dialogue beD=\(u1,s1,…,uT,sT\)D=\(u\_\{1\},s\_\{1\},\\ldots,u\_\{T\},s\_\{T\}\), whereutu\_\{t\}andsts\_\{t\}denote the user and system utterances at turntt, and lety∈\{0,1\}y\\in\\\{0,1\\\}denote the final dialogue outcome, withy=1y=1indicating eventual failure andy=0y=0indicating success\. Instead of evaluating only on the full dialogue, we construct a supervised example for every visible partial dialogueD≤tD\_\{\\leq t\}, where the model observes turns1:t1\{:\}tand predicts whether the*full*dialogue will eventually fail\. This means that every dialogue contributes multiple training and test instances, one per partial dialogue instance, while all instances from the same dialogue share the same terminal label\. Formally, let𝒟\\mathcal\{D\}be the full dialogue collection and letTiT\_\{i\}denote the length of dialogueii\. The partial\-dialogue dataset is:𝒟partial=\{\(D≤t,yi\):i∈𝒟,t∈\{1,…,Ti\}\}\\mathcal\{D\}\_\{\\text\{partial\}\}=\\bigl\\\{\(D\_\{\\leq t\},\\;y\_\{i\}\):i\\in\\mathcal\{D\},\\;t\\in\\\{1,\\ldots,T\_\{i\}\\\}\\bigr\\\}whereyi=1y\_\{i\}=1if dialogueiifails andyi=0y\_\{i\}=0if it succeeds\. A single dialogue of lengthTiT\_\{i\}contributesTiT\_\{i\}supervised examples, all sharing labelyiy\_\{i\}\. The TRACER model learns a scoring functionfθ:D≤t↦p^i,t∈\[0,1\]f\_\{\\theta\}:D\_\{\\leq t\}\\mapsto\\hat\{p\}\_\{i,t\}\\in\[0,1\], and a thresholdτ\\tauconverts the continuous score into an intervention decision: trigger recovery whenp^i,t\>τ\\hat\{p\}\_\{i,t\}\>\\tau\.

This setting is different from standard turn classification in two ways\. First, the label is about the future\. The goal is not to decide whether the current turn is problematic, but whether the dialogue is moving toward failure\. Second, the prediction is meant to support intervention\. To be useful in practice, the model’s score must be turned into a decision about when to act\. Therefore, our method includes both a failure predictor and a threshold\-based intervention policy\. Figure[2](https://arxiv.org/html/2607.03974#S1.F2)shows the overall pipeline\. A partial dialogue is encoded using a trajectory stream and a belief\-text stream\. These representations are combined to predict a failure score, which triggers a recovery action when it exceeds a threshold tuned on the development set\.

### 3\.1Partial Dialogue Representation

Each partial dialogue is represented using two complementary views of the dialogue state, corresponding to the two input streams shown in Figure[2](https://arxiv.org/html/2607.03974#S1.F2)\.

#### Trajectory features\.

As shown in the Temporal Trajectory Dynamics stream of Figure[2](https://arxiv.org/html/2607.03974#S1.F2), for every turn we compute a 5\-dimensional trajectory vector that summarizes how the dialogue state is evolving up to that point: \(1\) oscillation score, the number of slots whose values change repeatedly over the dialogue history; \(2\) coverage rate, the fraction of required slots filled for the active domain; \(3\) conflict count, the cumulative number of detected contradictions between the user input and the current dialogue state; \(4\) fill velocity, the short\-horizon rate at which new slot information is being filled; and \(5\) domain shifts, the number of distinct domains visited so far\. Aslotis a something the system uses to remember one detail the user cares about, such as the restaurant’s food type, price range, or location\. For example, in “find a cheap Italian restaurant downtown,” the slots areprice = cheap,food = Italian, andarea = downtown\. These are things mentioned in the conversation\. These features are computed incrementally for every partial dialogue, producing a sequence of turn\-level vectors\. This sequential view is important because many task failures emerge through unstable trajectories rather than through any single isolated error\.

We define each feature formally\. Letbt:𝒮→𝒱∪\{∅\}b\_\{t\}:\\mathcal\{S\}\\to\\mathcal\{V\}\\cup\\\{\\emptyset\\\}be the cumulative belief state at turntt, where𝒮\\mathcal\{S\}is the set of domain\-slot pairs and𝒱\\mathcal\{V\}is the value vocabulary\. Letdtd\_\{t\}denote the active domain,\|bt\|\|b\_\{t\}\|the number of filled slots, andΔt=\{s∈𝒮:bt​\(s\)≠bt−1​\(s\)\}\\Delta\_\{t\}=\\\{s\\in\\mathcal\{S\}:b\_\{t\}\(s\)\\neq b\_\{t\-1\}\(s\)\\\}the set of slots that changed at turntt\.Concretely,𝒮\\mathcal\{S\}contains pairs such ashotel\-areaorrestaurant\-food, and𝒱\\mathcal\{V\}the corresponding values such asnorthoritalian\. The codomain𝒱∪\{∅\}\\mathcal\{V\}\\cup\\\{\\emptyset\\\}allows slots that have not yet been mentioned to map to∅\\emptyset\(the unfilled state\)\. For example, after three turns a belief state might readb3​\(hotel\-area\)=northb\_\{3\}\(\\texttt\{hotel\-area\}\)=\\texttt\{north\},b3​\(hotel\-pricerange\)=∅b\_\{3\}\(\\texttt\{hotel\-pricerange\}\)=\\emptyset\(not yet specified\),b3​\(restaurant\-food\)=italianb\_\{3\}\(\\texttt\{restaurant\-food\}\)=\\texttt\{italian\}, and the change setΔ3\\Delta\_\{3\}would contain onlyrestaurant\-foodif that was the only slot updated at turn 3\.

#### Oscillation Score\.

LetVstV\_\{s\}^\{t\}be the ordered sequence of distinct consecutive values assigned to slotssthrough turntt\. A slot oscillates when its value has been revisedat least twice, that is, when three or more distinct consecutive values have been assigned to it \(\|Vst\|\>2\|V\_\{s\}^\{t\}\|\>2\):

OscScoret=\|\{s∈𝒮:\|Vst\|\>2\}\|\\text\{OscScore\}\_\{t\}=\\bigl\|\\\{s\\in\\mathcal\{S\}:\|V\_\{s\}^\{t\}\|\>2\\\}\\bigr\|Example\.If a user first requests acheaphotel, then revises toexpensive, and then reverts tocheap, the slothotel\-pricerangeaccumulates three distinct consecutive values, so\|Vst\|=3\>2\|V\_\{s\}^\{t\}\|=3\>2and it contributes 1 toOscScoret\\text\{OscScore\}\_\{t\}\. A slot that is set once and never changed would have\|Vst\|=1\|V\_\{s\}^\{t\}\|=1and would not count\.

#### Coverage Rate\.

Let𝒮d∗\\mathcal\{S\}^\{\*\}\_\{d\}be the set of required informable slots for domaindd\. Coverage measures the fraction of required slots that are currently filled:

CoverageRatet=\|\{s∈𝒮dt∗:bt​\(s\)≠∅\}\|\|𝒮dt∗\|\\text\{CoverageRate\}\_\{t\}=\\frac\{\|\\\{s\\in\\mathcal\{S\}^\{\*\}\_\{d\_\{t\}\}:b\_\{t\}\(s\)\\neq\\emptyset\\\}\|\}\{\|\\mathcal\{S\}^\{\*\}\_\{d\_\{t\}\}\|\}Example\.Suppose the hotel domain requires three informable slots:area,pricerange, andtype\. If at turnttthe user has specifiedarea = northandtype = guesthousebut has not yet mentioned a price range, thenCoverageRatet=2/3≈0\.67\\text\{CoverageRate\}\_\{t\}=2/3\\approx 0\.67\. A dialogue that reaches its final turn without ever providing the price range would end with coverage below 1\.0, signalling an incomplete task specification\.

#### Conflict Count\.

A conflict at turnτ\\tauis detected when a slot value is overwritten and the user utterance is semantically distant from the prior value\. Letϕ​\(⋅\)\\phi\(\\cdot\)denote sentence embeddings andvsτ−1v\_\{s\}^\{\\tau\-1\}the text"The \{slot\} is \{old\_value\}":

cτ=𝟏\[∃s∈Δτ∩dom\(bτ−1\):ϕ​\(uτ\)⊤​ϕ​\(vsτ−1\)‖ϕ​\(uτ\)‖⋅‖ϕ​\(vsτ−1\)‖<δ\]c\_\{\\tau\}=\\mathbf\{1\}\\\!\\Bigl\[\\exists\\,s\\in\\Delta\_\{\\tau\}\\cap\\operatorname\{dom\}\(b\_\{\\tau\-1\}\):\\dfrac\{\\phi\(u\_\{\\tau\}\)^\{\\top\}\\phi\(v\_\{s\}^\{\\tau\-1\}\)\}\{\\\|\\phi\(u\_\{\\tau\}\)\\\|\\cdot\\\|\\phi\(v\_\{s\}^\{\\tau\-1\}\)\\\|\}<\\delta\\Bigr\]

ConflictCountt=∑τ=1tcτ\\text\{ConflictCount\}\_\{t\}=\\sum\_\{\\tau=1\}^\{t\}c\_\{\\tau\}whereδ=0\.3\\delta=0\.3is the cosine\-similarity threshold below which a change is flagged as conflicting\.Example\.Suppose the current belief state recordshotel\-area = northand at turnτ\\tauthe user says “actually, I would prefer the south side\.” The slot value changes tosouth, but the user utterance is semantically distant from the probe phrase “The area is north” \(cosine similarity well below 0\.3\), socτ=1c\_\{\\tau\}=1and the conflict count increments\. By contrast, if the user says “yes, north works” and the value stays unchanged, no slot appears inΔτ\\Delta\_\{\\tau\}andcτ=0c\_\{\\tau\}=0\.

#### Fill Velocity\.

A rolling two\-turn window measures the rate of new slot filling:

FillVelocityt=\{\|bt\|t\+1if​t<2\|bt\|−\|bt−2\|2otherwise\\text\{FillVelocity\}\_\{t\}=\\begin\{cases\}\\dfrac\{\|b\_\{t\}\|\}\{t\+1\}&\\text\{if \}t<2\\\\\[6\.0pt\] \\dfrac\{\|b\_\{t\}\|\-\|b\_\{t\-2\}\|\}\{2\}&\\text\{otherwise\}\\end\{cases\}Example\.If four slots are filled at turnt=5t\{=\}5and only two were filled at turnt=3t\{=\}3, thenFillVelocity5=\(4−2\)/2=1\.0\\text\{FillVelocity\}\_\{5\}=\(4\-2\)/2=1\.0: the conversation is gathering new information at a steady pace\. If at turnt=7t\{=\}7the count is still four \(no new slots added since turn 5\), thenFillVelocity7=0\\text\{FillVelocity\}\_\{7\}=0, signalling that the information exchange has stalled\.

#### Domain Shifts\.

The count of distinct domains activated up to turntt:DomainShiftst=\|\{dτ:τ≤t,dτ≠∅\}\|\\text\{DomainShifts\}\_\{t\}=\\bigl\|\\\{d\_\{\\tau\}:\\tau\\leq t,\\;d\_\{\\tau\}\\neq\\emptyset\\\}\\bigr\|

Example\.If a dialogue discusses a hotel in turns 1–3, switches to a restaurant query in turn 4, and adds a train booking in turn 6, thenDomainShifts6=3\\text\{DomainShifts\}\_\{6\}=3\. A single\-domain conversation that only ever toucheshotelwould haveDomainShiftst=1\\text\{DomainShifts\}\_\{t\}=1throughout its entire length\.

The complete trajectory feature vector at turnttis defined as𝐟t=\(OscScoret,CoverageRatet,\\mathbf\{f\}\_\{t\}=\(\\text\{OscScore\}\_\{t\},\\ \\text\{CoverageRate\}\_\{t\},ConflictCountt,FillVelocityt,DomainShiftst\)\\text\{ConflictCount\}\_\{t\},\\ \\text\{FillVelocity\}\_\{t\},\\ \\text\{DomainShifts\}\_\{t\}\)and lies inℝ5\\mathbb\{R\}^\{5\}\.

#### Belief\-state text\.

In parallel, as shown in the Belief\-State Text stream of Figure[2](https://arxiv.org/html/2607.03974#S1.F2), we serialize the belief state at each turn into text and concatenate the resulting descriptions across the visible partial dialogue\. Concretely, each turn contributes a segment of the form\[TURNtt\] slot is value, \.\.\., and the partial dialogue text is the concatenation of all such segments up to turntt\. This gives the model direct access to the semantic content of the evolving belief state, while keeping the input format agnostic to the exact ontology size\.

### 3\.2TRACER Architecture

TRACER is a dual\-stream predictor that combines a trajectory encoder with a belief\-text encoder\.

#### Stream A: trajectory encoder\.

As shown in Figure[2](https://arxiv.org/html/2607.03974#S1.F2), the trajectory stream projects each 5\-dimensional turn vector into a latent representation, adds sinusoidal positional encodings to preserve turn order, and passes the sequence through a temporal Transformer encoder\(Vaswani et al\.,[2017](https://arxiv.org/html/2607.03974#bib.bib21)\)\. Masked mean pooling over visible turns yields a fixed\-dimensional trajectory representationhAh\_\{A\}\. In our experiments, this stream uses a 3\-layer Transformer encoder with model dimension 64 and 4 attention heads\. Formally, each feature vector𝐟t∈ℝ5\\mathbf\{f\}\_\{t\}\\in\\mathbb\{R\}^\{5\}is projected and position\-encoded:

𝐞tA=𝐖A​𝐟t\+𝐩t,𝐖A∈ℝdA×5,dA=64\\mathbf\{e\}\_\{t\}^\{A\}=\\mathbf\{W\}\_\{A\}\\mathbf\{f\}\_\{t\}\+\\mathbf\{p\}\_\{t\},\\quad\\mathbf\{W\}\_\{A\}\\in\\mathbb\{R\}^\{d\_\{A\}\\times 5\},\\;d\_\{A\}=64

where𝐩t\\mathbf\{p\}\_\{t\}is the sinusoidal positional encoding at positiontt\. The projected sequence\(𝐞1A,…,𝐞tA\)\(\\mathbf\{e\}\_\{1\}^\{A\},\\ldots,\\mathbf\{e\}\_\{t\}^\{A\}\)is processed by a 3\-layer multi\-head self\-attention Transformer withH=4H=4heads, and the outputs at valid positions are mean\-pooled:

hA=1t​∑τ=1t\[TransformerEncoderA​\(𝐞1:tA\)\]τ∈ℝ64h\_\{A\}=\\frac\{1\}\{t\}\\sum\_\{\\tau=1\}^\{t\}\\bigl\[\\mathrm\{TransformerEncoder\}\_\{A\}\(\\mathbf\{e\}\_\{1:t\}^\{A\}\)\\bigr\]\_\{\\tau\}\\in\\mathbb\{R\}^\{64\}

#### Stream B: belief\-text encoder\.

As shown in the Belief\-State Text stream of Figure[2](https://arxiv.org/html/2607.03974#S1.F2), the text stream encodes the serialized belief\-state text with a pretrained RoBERTa encoder\(Liu et al\.,[2019](https://arxiv.org/html/2607.03974#bib.bib13)\)\. We use the contextualized first\-token representation and project it into a 256\-dimensional belief\-state embeddinghBh\_\{B\}\. To stabilize training, the lower encoder layers are partially frozen, while the upper layers and projection head remain trainable\. Formally, letxtx\_\{t\}be the serialized belief\-state text for the partial dialogue up to turntt:

hB=tanh⁡\(𝐖B⋅RoBERTa​\(xt\)\[CLS\]\)∈ℝ256,𝐖B∈ℝ256×768h\_\{B\}=\\tanh\\\!\\bigl\(\\mathbf\{W\}\_\{B\}\\cdot\\mathrm\{RoBERTa\}\(x\_\{t\}\)\_\{\[\\mathrm\{CLS\}\]\}\\bigr\)\\in\\mathbb\{R\}^\{256\},\\quad\\mathbf\{W\}\_\{B\}\\in\\mathbb\{R\}^\{256\\times 768\}

whereRoBERTa​\(⋅\)\[CLS\]\\mathrm\{RoBERTa\}\(\\cdot\)\_\{\[\\mathrm\{CLS\}\]\}denotes the contextualized representation of the first token\.

#### Fusion and prediction\.

As shown in the Fusion MLP of Figure[2](https://arxiv.org/html/2607.03974#S1.F2), the final TRACER representation is the concatenation\[hA;hB\]∈ℝ320\[h\_\{A\};\\,h\_\{B\}\]\\in\\mathbb\{R\}^\{320\}, which is passed through a small multilayer perceptron with ReLU activation and dropout to produce a scalar logit\. After a sigmoid transformation, this yieldsp^​\(y=1∣D≤t\)\\hat\{p\}\(y=1\\mid D\_\{\\leq t\}\), the probability that the dialogue will eventually fail:

p^​\(y=1∣D≤t\)=σ​\(𝐖3​ReLU​\(𝐖2​\[hA;hB\]\+𝐛2\)\+b3\)\\hat\{p\}\(y=1\\mid D\_\{\\leq t\}\)=\\sigma\\\!\\Bigl\(\\mathbf\{W\}\_\{3\}\\,\\mathrm\{ReLU\}\\bigl\(\\mathbf\{W\}\_\{2\}\\,\[h\_\{A\};\\,h\_\{B\}\]\+\\mathbf\{b\}\_\{2\}\\bigr\)\+b\_\{3\}\\Bigr\)

where𝐖2∈ℝ256×320\\mathbf\{W\}\_\{2\}\\in\\mathbb\{R\}^\{256\\times 320\},𝐖3∈ℝ1×256\\mathbf\{W\}\_\{3\}\\in\\mathbb\{R\}^\{1\\times 256\}, andσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function\.

### 3\.3Training Objective

TRACER is trained with a binary classification objective over partial dialogues, combined with an early\-prediction reward that upweights correct failure predictions made at earlier turns\. Intuitively, a correct forecast early in a dialogue is more valuable than one made near the end, when intervention opportunities are limited\. This objective reflects the practical goal of the system: not merely to classify dialogues accurately, but to surface risk early enough to be actionable\.

Formally, let𝒫train=\{\(i,t\):i∈𝒟train,t∈\{1,…,Ti\}\}\\mathcal\{P\}\_\{\\text\{train\}\}=\\\{\(i,t\):i\\in\\mathcal\{D\}\_\{\\text\{train\}\},\\;t\\in\\\{1,\\ldots,T\_\{i\}\\\}\\\}be the set of all partial\-dialogue training instances\. The combined loss is:

ℒBCE=−1\|𝒫train\|​∑\(i,t\)∈𝒫train\[yi​log⁡p^i,t\+\(1−yi\)​log⁡\(1−p^i,t\)\]\\mathcal\{L\}\_\{\\text\{BCE\}\}=\-\\frac\{1\}\{\|\\mathcal\{P\}\_\{\\text\{train\}\}\|\}\\sum\_\{\(i,t\)\\in\\mathcal\{P\}\_\{\\text\{train\}\}\}\\Bigl\[y\_\{i\}\\log\\hat\{p\}\_\{i,t\}\+\(1\-y\_\{i\}\)\\log\(1\-\\hat\{p\}\_\{i,t\}\)\\Bigr\]

The early\-prediction reward term additionally encourages the model to fire correct failure warnings as early as possible, weighted by how much dialogue remains:

ℒearly=−λ\|𝒫train\|​∑\(i,t\)∈𝒫trainw​\(t,Ti\)⋅𝟏​\[p^i,t\>12,yi=1\]\\mathcal\{L\}\_\{\\text\{early\}\}=\-\\frac\{\\lambda\}\{\|\\mathcal\{P\}\_\{\\text\{train\}\}\|\}\\sum\_\{\(i,t\)\\in\\mathcal\{P\}\_\{\\text\{train\}\}\}w\(t,T\_\{i\}\)\\cdot\\mathbf\{1\}\\\!\\bigl\[\\hat\{p\}\_\{i,t\}\>\\tfrac\{1\}\{2\},\\;y\_\{i\}=1\\bigr\]

The two terms combine into the total training loss:ℒ=ℒBCE\+ℒearly\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{BCE\}\}\+\\mathcal\{L\}\_\{\\text\{early\}\}wherew​\(t,Ti\)=\(Ti−t\)/Ti∈\[0,1\]w\(t,T\_\{i\}\)=\(T\_\{i\}\-t\)/T\_\{i\}\\in\[0,1\]is a temporal weight assigning higher reward to correct failure predictions made at earlier turns, andλ=0\.3\\lambda=0\.3\. The reward term is active only for true\-positive partial\-dialogue predictions \(failed dialogues where the model correctly assignsp^i,t\>0\.5\\hat\{p\}\_\{i,t\}\>0\.5\)\.Crucially,w​\(t,Ti\)w\(t,T\_\{i\}\)uses the known full\-dialogue lengthTiT\_\{i\}, which is available during*training*because the supervision is derived from completed dialogues\. At inference time,ℒearly\\mathcal\{L\}\_\{\\text\{early\}\}is not computed; the model simply outputsp^i,t\\hat\{p\}\_\{i,t\}from the observed partial dialogue without requiring knowledge of the future dialogue length\.To address class imbalance from partial\-dialogue expansion, the binary cross\-entropy applies positive\-class upweighting with weightα\+=\|𝒫train−\|/\|𝒫train\+\|\\alpha\_\{\+\}=\|\\mathcal\{P\}^\{\-\}\_\{\\text\{train\}\}\|/\|\\mathcal\{P\}^\{\+\}\_\{\\text\{train\}\}\|\. Full experimental details are provided in Appendix[A](https://arxiv.org/html/2607.03974#A1)\.

CategoryModelBelief StateAUC\-ROC↑\\uparrowF1↑\\uparrowEDS↑\\uparrowDet\. Turn↓\\downarrowHeuristic TriggerNo interventionoracle0\.5000\.0000\.000∞\\inftyFixed\-schedule \(every 3\)oracle0\.4990\.2360\.6902\.00Slot\-confidence \(Osc\>\>0\)oracle0\.4950\.0250\.0097\.50Feature\-threshold ensembleoracle0\.5350\.2860\.8640\.25ClassicalFeature ModelPartial dialogue logreg \(turn\-level\)oracle0\.5740\.3440\.8490\.35Partial dialogue logreg \(engineered\)oracle0\.6000\.3510\.7281\.03Zero\-ShotPromptingLlama\-3\.1\-8Boracle0\.5190\.2680\.4742\.59Mistral\-7Boracle0\.5250\.2780\.5582\.08Qwen2\.5\-7Boracle0\.5310\.3210\.7721\.32Zero\-Shot CoTPromptingLlama\-3\.1\-8Boracle0\.5160\.2900\.5662\.49Mistral\-7Boracle0\.5260\.2750\.5681\.95Qwen2\.5\-7Boracle0\.5190\.3180\.7561\.34Few\-ShotPromptingLlama\-3\.1\-8Boracle0\.4920\.3370\.9930\.06Mistral\-7Boracle0\.5240\.3120\.6481\.89Qwen2\.5\-7Boracle0\.5180\.2990\.6681\.67Our ApproachTRACERgenerated0\.6170\.3700\.7511\.22TRACERoracle0\.741†0\.463†0\.7271\.36Table 1:Main failure\-forecasting results on MultiWOZ\. TRACER performs best overall, though some heuristic baselines obtain higher EDS through aggressive triggering\.†Significantly better than both the best classical baseline \(logreg engineered\) and best LLM baseline \(Qwen2\.5\-7B zero\-shot\):p<0\.0001p<0\.0001by a paired bootstrap test\.
### 3\.4Triggered Recovery Policy

The predictor becomes a control layer by thresholding the partial\-dialogue\-level failure score\. For a partial dialogue at turntt, ifp​\(y=1∣D≤t\)\>τp\(y=1\\mid D\_\{\\leq t\}\)\>\\tau, the system triggers a recovery action\. We selectτ\\tauon the development set under a false\-positive\-rate constraint and then freeze that operating point for test\-time evaluation\. This distinction matters because unconstrained low thresholds can maximize early\-detection metrics trivially while producing an impractically large number of unnecessary interventions\. We study two recovery methods under this shared trigger policy; their design and implementation are described in Appendix[I](https://arxiv.org/html/2607.03974#A9)\.

## 4Results

Evaluation Settings\.We evaluate TRACER in both an*oracle*setting, where gold belief states are provided, and a*generated*setting, where belief states are produced automatically to better reflect deployment conditions\. This comparison lets us separate the forecasting capacity of the model under clean structured inputs from its robustness when upstream state representations are noisy\. Details of the belief\-state generation prompting procedure are provided in Appendix[J\.2](https://arxiv.org/html/2607.03974#A10.SS2)\. We report four metrics\. Two standard metrics:AUC\-ROC,False Positive Rate \(FPR\), andF1\(with a threshold set via the development\-set\)\.Early Detection Score \(EDS\)measures how early the model first correctly triggers on a failing dialogue:EDSi=\(Ti−ti∗\)/Ti\\text\{EDS\}\_\{i\}=\(T\_\{i\}\-t\_\{i\}^\{\*\}\)/T\_\{i\}, whereti∗t\_\{i\}^\{\*\}is the first trigger turn; EDS=1=1for a turn\-1 detection and EDS=0=0if the model never triggers\.Det\. Turn\(t¯∗\\bar\{t\}^\{\*\}\) is the mean turn at which the first correct trigger fires, averaged over detected failing dialogues; lower is earlier\. Formal definitions are in Appendix[H](https://arxiv.org/html/2607.03974#A8)\.

Baselines\.We compare against three groups\.*Heuristic triggers*include a no\-intervention baseline, a fixed\-schedule trigger \(fires every 3 turns\), a slot\-confidence trigger \(fires on any oscillation\), and a feature\-threshold ensemble \(fires when any hand\-designed threshold over the five trajectory features is exceeded\)\.*Classical feature models*are two logistic\-regression classifiers trained on trajectory features, one using the turn\-level feature vector and one using an engineered partial\-dialogue summary\.*Prompting baselines*evaluate Llama\-3\.1\-8B, Mistral\-7B\-Instruct, and Qwen2\.5\-7B under zero\-shot, zero\-shot chain\-of\-thought, and few\-shot paradigms\. Full baseline details are in Appendix[A](https://arxiv.org/html/2607.03974#A1)\.

Main Forecasting Performance\.Table[1](https://arxiv.org/html/2607.03974#S3.T1)shows the main partial\-dialogue\-based forecasting results on the MultiWOZ test split\. TRACER achieves the strongest discrimination performance overall, with an AUC\-ROC of 0\.741 and an F1 score of 0\.463\. This is a clear improvement over the strongest learned single\-stream baseline, the text\-only model, which achieves 0\.692 AUC and 0\.432 F1, and a larger gain over classical engineered partial\-dialogue baselines and hand\-designed trigger heuristics\. These results support the paper’s central modeling claim: combining trajectory dynamics with belief\-state text is more effective for forecasting eventual failure than relying on either source alone\.

At the same time, Table[1](https://arxiv.org/html/2607.03974#S3.T1)also shows why forecasting quality cannot be reduced to a single early\-trigger metric\. Some heuristic baselines achieve high EDS by firing extremely aggressively\. For example, the feature\-threshold ensemble attains an EDS of 0\.864 despite much weaker ranking and classification quality \(0\.535 AUC, 0\.286 F1\), and the fixed\-schedule baseline reaches 0\.690 EDS while remaining close to chance in AUC\. TRACER’s advantage lies in producing a more reliable failure score, not merely in triggering as early as possible, a distinction examined in Section[2](https://arxiv.org/html/2607.03974#S4.T2)\.Several methods in Table[1](https://arxiv.org/html/2607.03974#S3.T1)report mean detection turns below 3, which may seem surprising for MultiWOZ dialogues that can exceed 10 turns\. The key is that Det\. Turn is the mean*first*trigger turn, averaged only over detected failing dialogues\. Aggressive baselines \(e\.g\., the feature\-threshold ensemble with Det\. Turn=0\.25=0\.25\) fire almost immediately on nearly every dialogue, which yields a near\-zero mean detection turn but also produces extremely high false\-positive rates\. TRACER’s Det\. Turn of 1\.36 reflects a threshold calibrated under the FPR≤0\.15\\leq 0\.15constraint, so it triggers early but selectively\.

Relationship between Det\. Turn in Table[1](https://arxiv.org/html/2607.03974#S3.T1)and Fixed\-Context Forecasting\.These are two distinct analyses\. Table[1](https://arxiv.org/html/2607.03974#S3.T1)reports the*threshold\-triggered*behavior: the model sees turns one by one and fires once the failure score crossesτ\\tau, so Det\. Turn is the earliest crossing point\. Fixed\-Context Forecasting \(Section below\) instead*locks*the visible context to a fixed fraction of the dialogue \(25%, 50%, 75%, or 100%\) and evaluates classification and ranking quality at that snapshot; no threshold trigger is involved\. The two analyses are complementary: one measures*when*the system would intervene, while the other measures*how discriminative*partial information is at different stages\.

Oracle vs\. generated belief states\.TRACER with oracle belief states achieves higher AUC\-ROC \(0\.741 vs\. 0\.617\) and F1 \(0\.463 vs\. 0\.370\) than with generated states, as expected from cleaner supervision\. However, the generated setting sometimes shows marginally higher EDS \(0\.751 vs\. 0\.727\)\. This counter\-intuitive pattern arises because the development\-set threshold is re\-calibrated for each belief\-state condition, and the noisier generated belief states shift the model’s score distribution in ways that can move the frozen threshold to a slightly more aggressive trigger point\. It does not indicate that generated states provide better information; AUC\-ROC, which is threshold\-independent, is the appropriate summary of discrimination quality\.

Ablation Results\.Table[2](https://arxiv.org/html/2607.03974#S4.T2)isolates the contribution of the two input streams under both oracle and generated belief\-state inputs\. With oracle belief states, the text\-only model is stronger than the features\-only model, indicating that the serialized belief\-state text already contains substantial predictive signal about eventual failure\. With generated belief states, both single\-stream ablations degrade, but the full fused model remains strongest, showing that trajectory dynamics and belief\-state text remain complementary even when the belief state is noisier\.

ModelBelief State TypeAUC\-ROC↑\\uparrowF1↑\\uparrowStream A only\(features\)generated0\.5540\.337oracle0\.6230\.363Stream B only\(text\)generated0\.5830\.332oracle0\.6920\.432TRACERgenerated0\.6170\.370oracle0\.7410\.463Table 2:Ablation on TRACER streams under oracle and generated belief\-state inputs\. The fused model outperforms both single\-stream ablations in both settings\.Threshold Tradeoff Analysis\.The raw forecasting scores become operational only after they are converted into intervention decisions through a threshold\. We select this threshold on thedevelopment set\(distinct from the test set used for final results\) under an explicit false\-positive\-rate constraint, so that the operating point reflects a principled choice rather than post\-hoc tuning on test data\. Figure[3](https://arxiv.org/html/2607.03974#S4.F3)plots the threshold sweep on the development set\. The main pattern is clear: lower thresholds detect more failures earlier, but they also create many false positives\. At the extreme, the unconstrained best\-EDS point occurs atτ=0\.1\\tau=0\.1, where the detection rate reaches 1\.0, but the false positive rate also reaches 1\.0, making the operating point unusable in practice\.

We therefore freeze a threshold selected under an explicit false\-positive\-rate constraint\. The final operating point,τ=0\.7788\\tau=0\.7788, yields a test false positive rate of 0\.141, a detection rate of 0\.540, mean detection turn 3\.70, and F1 of 0\.374\. This is substantially more conservative than the threshold that maximizes development\-set F1 \(τ≈0\.544\\tau\\approx 0\.544\), which increases recall and F1 but also drives the false positive rate to roughly 0\.47 on the development set\. The threshold study therefore supports a key claim of the paper: EDS alone is not sufficient for choosing an intervention policy, and practical dialogue forecasting requires an explicit tradeoff between early warning and unnecessary interventions\.More precisely, the intervention threshold encodes a cost\-benefit decision: a lowerτ\\tauwarns earlier but interrupts more successful dialogues \(higher FPR\), while a higherτ\\tauis more precise but misses more failures or catches them late\. In settings where an unnecessary intervention is low\-cost, such as a soft confirmation prompt, a more aggressive threshold may be preferable\. In settings where interrupting a user mid\-task carries higher friction, a conservative threshold like the one we freeze \(τ=0\.7788\\tau=0\.7788\) is more appropriate\. We report the full threshold sweep in Appendix[D](https://arxiv.org/html/2607.03974#A4)\.

![Refer to caption](https://arxiv.org/html/2607.03974v1/x3.png)Figure 3:Threshold tradeoff on the development set\. The figure should show how lower thresholds improve early detection but sharply increase false positives\.Fixed\-Context Forecasting\.Figure[4](https://arxiv.org/html/2607.03974#S4.F4)evaluates TRACER under fixed visible\-context conditions, where only the first 25%, 50%, 75%, or 100% of each dialogue is revealed\. This analysis makes the early\-forecasting claim more concrete than aggregate EDS alone\. Even with only the first quarter of the dialogue visible, TRACER already achieves an AUC\-ROC of \.666, indicating that a meaningful failure signal is available well before task completion\. Performance improves steadily as more context is revealed: AUC rises to \.722 at 50%, \.811 at 75%, and 0\.819 at full context, while F1 increases from \.209 to \.317, \.461, and \.524, respectively\.

The most notable pattern is that by 75% of the dialogue, TRACER is already close to its full\-context discrimination quality\. This suggests that impending failure is often visible before the final turns, which is exactly the regime where proactive intervention is plausible\. Detection rate also increases monotonically from 0\.141 at 25% to 0\.540 at100% \(i\.e\., the complete dialogue\), where the frozen threshold is applied to the full test set, while EDS rises from 0\.103 to 0\.299\.F1 at 25% is modest \(0\.209\), which reflects the expected difficulty of predicting eventual task outcome from very limited context: at this stage most slot values have not yet been mentioned and the trajectory features carry little signal\. This is consistent with the monotonic improvement in AUC\-ROC across context lengths, which shows that the model’s ranking quality improves steadily rather than steeply only near the end\.Taken together, these results provide direct evidence that the forecasting problem is not purely retrospective: partial dialogues contain enough signal to support useful early\-warning behavior\.

Cross\-Domain Transfer\.Table[4](https://arxiv.org/html/2607.03974#A4.T4)in Appendix[D](https://arxiv.org/html/2607.03974#A4)reports zero\-shot transfer from a MultiWOZ\-trained TRACER model to SGD and ABCD\. The results are mixed\. On ABCD, TRACER transfers reasonably well, reaching 0\.771 AUC\-ROC, 0\.772 F1, and 0\.819 EDS, which suggests that the forecasting signal learned from trajectory dynamics and belief\-state text can generalize to at least some task\-oriented domains beyond the source benchmark\. In contrast, zero\-shot transfer to SGD is weak: AUC falls to 0\.494 and F1 to 0\.069, even though EDS remains moderately high\. These results indicate that cross\-domain generalization is non\-uniform: TRACER transfers well to ABCD but fails on SGD, positioning it as a strong in\-domain forecaster with partial transfer ability rather than a universally portable predictor\.

![Refer to caption](https://arxiv.org/html/2607.03974v1/x4.png)Figure 4:Fixed\-context forecasting: TRACER recovers strong predictive signal from partial context, with substantial gains at 50% and 75% of the dialogue\.Qualitative Error Analysis\.Error analysis reveals two characteristic patterns\. False positives often involve dialogues that are ultimately successful but pass through genuine instability, domain confusion, repeated corrections, or cross\-domain shifts, that TRACER correctly flags as risky before the dialogue self\-corrects\. Not every false positive is semantically uninformative: the system detects real risk in dialogues that happen to recover\. False negatives show the opposite: some failed dialogues remain locally coherent at the feature level until the final turns, making failure genuinely difficult to anticipate from partial context\.

Human\-Evaluated Failure Taxonomy Analysis\.To interpret TRACER failures, we define a 4\-category taxonomy:Information Incompleteness\(task constraints never stably assembled\),Contradiction / Misalignment\(system\-user misalignment on facts or constraints\),State Instability\(belief states oscillating without convergence\), andTask Drift / Domain Confusion\(interactions shifting away from the active task or mixing incompatible domains\)\. These labels are*not*ground\-truth annotations, but serve as a human\-evaluated qualitative framework over sampled failed dialogues\. Full definitions are in Appendix[G](https://arxiv.org/html/2607.03974#A7)\.

To validate this taxonomy, we annotate a balanced sample of 60 failed dialogues \(15 per category\); Table[3](https://arxiv.org/html/2607.03974#S4.T3)reports heuristic and human\-evaluated on\-diagonal counts\. Alignment is strongest forContradiction / MisalignmentandTask Drift / Domain Confusion\(15/15 each\), whileInformation IncompletenessandState Instabilityshow minor leakage \(2 off\-diagonal each\), as missing information and repeated slot revisions frequently co\-occur\.

LabelHeur\.On\-Diag\.Information Incompleteness1513Contradiction / Misalignment1515State Instability1513Task Drift / Domain Confusion1515Table 3:Human\-evaluated qualitative taxonomy on a balanced 60\-dialogue sample\.Heur\.: heuristic count;On\-Diag\.: human\-evaluated on\-diagonal count\.
## 5Conclusion

We presented TRACER, a framework for early failure detection in task\-oriented dialogue\. Instead of waiting until the end of the conversation, TRACER predicts from partial dialogue context whether the interaction is likely to fail\. It combines trajectory dynamics with belief\-state text, which lets it capture both how the dialogue state changes over time and what the system currently believes the user wants\. In our experiments, this approach detects useful failure signals well before dialogue completion and outperforms heuristic, classical, and single\-stream baselines\. These results suggest that early failure detection can serve as a practical control layer for task\-oriented dialogue systems, complementing standard post\-hoc evaluation methods\. In future work, we plan to test TRACER in interactive settings with live recovery, improve transfer across datasets, and further validate the failure categories with human annotation\.

## Acknowledgments

This material is based upon work supported by the National Science Foundation \(NSF\) under Grant No\. 2145357\.

## Limitations

TRACER shows that it is possible to predict dialogue failure from partial conversations, and that performance improves when the model uses both trajectory features and belief\-state text\. Still, the method has several limitations\. First, the recovery analysis is offline, so the results do not show that TRACER improves task success in live conversations with real users\. For example, a recovery action might look helpful in an offline evaluation but still confuse a real user or interrupt a conversation at the wrong time\. This limitation is important because a dialogue system is only useful in practice if it helps real users complete their tasks\.

Second, the failure taxonomy is only used to help interpret errors, not as gold\-standard supervision, because public task\-oriented dialogue datasets do not include these labels\. For example, a dialogue may be labeled as showing task drift or state instability by our taxonomy, but that label is an interpretation rather than an official dataset annotation\. This is still important because the taxonomy helps explain what kinds of failures the model is detecting, even if it is not a formal benchmark target\.

Third, cross\-domain transfer is inconsistent: the model transfers well to ABCD but poorly to SGD\. For example, a model trained on MultiWOZ may still detect failure patterns in ABCD, but fail to generalize to SGD because the dialogue structure, domains, or failure types differ\. This matters because a method is much more useful if it works across datasets rather than only on the domain it was trained on\.

Fourth, some trajectory features are intentionally coarse\. The DomainShifts feature counts the number of*distinct*domains activated so far, rather than modeling actual sequential domain transitions\. A dialogue that visits hotel, restaurant, and hotel again registers the same DomainShifts value as one that visits three entirely new domains in sequence\. This simplification makes the feature easy to compute incrementally and correlates with multi\-domain failure risk, but it does not capture the directionality or recency of domain switches\. A proper transition\-count or recency\-weighted domain\-shift feature could capture more subtle task\-drift patterns and is a direction for future work\.

Finally, EDS by itself is not enough for evaluation, since a system can get a high EDS simply by triggering too often and creating too many false positives\. For example, a model could warn very early in almost every conversation, which would improve EDS but would also lead to many unnecessary interventions in successful dialogues\. This is important because a good intervention system must balance early warning with not interrupting users when nothing is wrong\.

Future work should test the model in interactive settings, improve transfer across datasets, and validate the failure categories more carefully with human annotation\.

## References

- Acikgoz et al\. \(2025\)Emre Can Acikgoz, Carl Guo, Suvodip Dey, Akul Datta, Takyoung Kim, Gokhan Tur, and Dilek Hakkani\-Tur\. 2025\.[TD\-EVAL: Revisiting task\-oriented dialogue evaluation by combining turn\-level precision with dialogue\-level comparisons](https://aclanthology.org/2025.sigdial-1.7/)\.In*Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 113–132, Avignon, France\. Association for Computational Linguistics\.
- Budzianowski et al\. \(2018\)Paweł Budzianowski, Tsung\-Hsien Wen, Bo\-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić\. 2018\.[MultiWOZ \- a large\-scale multi\-domain Wizard\-of\-Oz dataset for task\-oriented dialogue modelling](https://doi.org/10.18653/v1/D18-1547)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 5016–5026, Brussels, Belgium\. Association for Computational Linguistics\.
- Carletta \(1992\)Jean Carletta\. 1992\.[Planning to fail, not failing to plan: Risk\-taking and recovery in task\-oriented dialogue](https://aclanthology.org/C92-3134/)\.In*COLING 1992 Volume 3: The 14th International Conference on Computational Linguistics*\.
- Chang and Danescu\-Niculescu\-Mizil \(2019\)Jonathan P\. Chang and Cristian Danescu\-Niculescu\-Mizil\. 2019\.[Trouble on the horizon: Forecasting the derailment of online conversations as they develop](https://doi.org/10.18653/v1/D19-1481)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 4743–4754, Hong Kong, China\. Association for Computational Linguistics\.
- Chen et al\. \(2021\)Derek Chen, Howard Chen, Yi Yang, Alexander Lin, and Zhou Yu\. 2021\.[Action\-based conversations dataset: A corpus for building more in\-depth task\-oriented dialogue systems](https://doi.org/10.18653/v1/2021.naacl-main.239)\.pages 3002–3017\.
- Devlin et al\. \(2019\)Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\. 2019\.[BERT: Pre\-training of deep bidirectional transformers for language understanding](https://doi.org/10.18653/v1/N19-1423)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 4171–4186, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Henderson et al\. \(2014\)Matthew Henderson, Blaise Thomson, and Jason D\. Williams\. 2014\.[The second dialog state tracking challenge](https://doi.org/10.3115/v1/W14-4337)\.In*Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue \(SIGDIAL\)*, pages 263–272, Philadelphia, PA, U\.S\.A\. Association for Computational Linguistics\.
- Higashinaka et al\. \(2016\)Ryuichiro Higashinaka, Kotaro Funakoshi, Yuka Kobayashi, and Michimasa Inaba\. 2016\.[The dialogue breakdown detection challenge: Task description, datasets, and evaluation metrics](https://aclanthology.org/L16-1502/)\.In*Proceedings of the Tenth International Conference on Language Resources and Evaluation \(LREC’16\)*, pages 3146–3150, Portorož, Slovenia\. European Language Resources Association \(ELRA\)\.
- Hosseini\-Asl et al\. \(2020\)Ehsan Hosseini\-Asl, Bryan McCann, Chien\-Sheng Wu, Semih Yavuz, and Richard Socher\. 2020\.[A simple language model for task\-oriented dialogue](https://proceedings.neurips.cc/paper_files/paper/2020/file/e946209592563be0f01c844ab2170f0c-Paper.pdf)\.In*Advances in Neural Information Processing Systems*, volume 33, pages 20179–20191\. Curran Associates, Inc\.
- Hudeček and Dusek \(2023\)Vojtěch Hudeček and Ondrej Dusek\. 2023\.[Are large language models all you need for task\-oriented dialogue?](https://doi.org/10.18653/v1/2023.sigdial-1.21)In*Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 216–228, Prague, Czechia\. Association for Computational Linguistics\.
- Lei et al\. \(2018\)Wenqiang Lei, Xisen Jin, Min\-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin\. 2018\.[Sequicity: Simplifying task\-oriented dialogue systems with single sequence\-to\-sequence architectures](https://doi.org/10.18653/v1/P18-1133)\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 1437–1447, Melbourne, Australia\. Association for Computational Linguistics\.
- Li et al\. \(2016\)Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao\. 2016\.[Deep reinforcement learning for dialogue generation](https://doi.org/10.18653/v1/D16-1127)\.In*Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*, pages 1192–1202, Austin, Texas\. Association for Computational Linguistics\.
- Liu et al\. \(2019\)Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov\. 2019\.[Roberta: A robustly optimized bert pretraining approach](https://arxiv.org/abs/1907.11692)\.*arXiv preprint arXiv:1907\.11692*\.
- Maier et al\. \(1997\)Elisabeth Maier, Norbert Reithinger, and Jan Alexandersson\. 1997\.[Clarification dialogues as measure to increase robustness in a spoken dialogue system](https://aclanthology.org/W97-0606/)\.In*Interactive Spoken Dialog Systems: Bringing Speech and NLP Together in Real Applications*\.
- McRoy and Hirst \(1995\)Susan W\. McRoy and Graeme Hirst\. 1995\.[The repair of speech act misunderstandings by abductive inference](https://aclanthology.org/J95-4001/)\.*Computational Linguistics*, 21\(4\):435–478\.
- Ngo et al\. \(2024\)Anh Ngo, Dirk Heylen, Nicolas Rollet, Catherine Pelachaud, and Chloé Clavel\. 2024\.[Exploration of human repair initiation in task\-oriented dialogue: A linguistic feature\-based approach](https://doi.org/10.18653/v1/2024.sigdial-1.51)\.In*Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue*, pages 603–609, Kyoto, Japan\. Association for Computational Linguistics\.
- Peng et al\. \(2021\)Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao\. 2021\.[Soloist: Building task bots at scale with transfer learning and machine teaching](https://doi.org/10.1162/tacl_a_00399)\.*Transactions of the Association for Computational Linguistics*, 9:807–824\.
- Qin et al\. \(2023\)Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li\. 2023\.[End\-to\-end task\-oriented dialogue: A survey of tasks, methods, and future directions](https://doi.org/10.18653/v1/2023.emnlp-main.363)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 5925–5941, Singapore\. Association for Computational Linguistics\.
- Raffel et al\. \(2020\)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J\. Liu\. 2020\.[Exploring the limits of transfer learning with a unified text\-to\-text transformer](http://jmlr.org/papers/v21/20-074.html)\.*Journal of Machine Learning Research*, 21\(140\):1–67\.
- Rastogi et al\. \(2020\)Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan\. 2020\.[Towards scalable multi\-domain conversational agents: The schema\-guided dialogue dataset](https://doi.org/10.1609/aaai.v34i05.6394)\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 34\(05\):8689–8696\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin\. 2017\.[Attention is all you need](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\.In*Advances in Neural Information Processing Systems*, volume 30\. Curran Associates, Inc\.
- Walker et al\. \(1997\)Marilyn A\. Walker, Diane J\. Litman, Candace A\. Kamm, and Alicia Abella\. 1997\.[PARADISE: A framework for evaluating spoken dialogue agents](https://doi.org/10.3115/976909.979652)\.In*35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics*, pages 271–280, Madrid, Spain\. Association for Computational Linguistics\.
- Wen et al\. \(2017\)Tsung\-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M\. Rojas\-Barahona, Pei\-Hao Su, Stefan Ultes, and Steve Young\. 2017\.[A network\-based end\-to\-end trainable task\-oriented dialogue system](https://aclanthology.org/E17-1042/)\.In*Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers*, pages 438–449, Valencia, Spain\. Association for Computational Linguistics\.
- Wu et al\. \(2020\)Chien\-Sheng Wu, Steven C\.H\. Hoi, Richard Socher, and Caiming Xiong\. 2020\.[TOD\-BERT: Pre\-trained natural language understanding for task\-oriented dialogue](https://doi.org/10.18653/v1/2020.emnlp-main.66)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 917–929, Online\. Association for Computational Linguistics\.
- Wu et al\. \(2019\)Chien\-Sheng Wu, Andrea Madotto, Ehsan Hosseini\-Asl, Caiming Xiong, Richard Socher, and Pascale Fung\. 2019\.[Transferable multi\-domain state generator for task\-oriented dialogue systems](https://doi.org/10.18653/v1/P19-1078)\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 808–819, Florence, Italy\. Association for Computational Linguistics\.
- Young et al\. \(2013\)Steve Young, Milica Gašić, Blaise Thomson, and Jason D\. Williams\. 2013\.[Pomdp\-based statistical spoken dialog systems: A review](https://doi.org/10.1109/JPROC.2012.2225812)\.*Proceedings of the IEEE*, 101\(5\):1160–1179\.

## Appendix AExperimental Setup

### A\.1Datasets

We train and evaluate TRACER primarily on MultiWOZ 2\.4\(Budzianowski et al\.,[2018](https://arxiv.org/html/2607.03974#bib.bib2)\), which provides multi\-domain dialogues with explicit belief states and rich opportunities for gradual failure through missing information, slot inconsistency, or domain drift\. Partial\-dialogue supervision is derived from terminal dialogue outcomes, with trajectory features computed turn by turn\. To test generalization, we evaluate the MultiWOZ\-trained model zero\-shot onSGD\(Rastogi et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib20)\)andABCD\(Chen et al\.,[2021](https://arxiv.org/html/2607.03974#bib.bib5)\); these datasets are used for transfer evaluation only, not joint training\.

### A\.2Baselines

We compare TRACER against three groups of forecasting baselines, plus recovery baselines described in Appendix[E](https://arxiv.org/html/2607.03974#A5): \(1\)Heuristic and classical baselines, comprising four non\-trainable heuristics,No intervention\(always predicts success\),Fixed\-schedule confirmation\(triggers every three turns\),Slot\-confidence\(fires when oscillation is detected\), and afeature\-threshold ensemble\(predicts failure whenever any hand\-designed threshold over the five trajectory features is exceeded\), as well as two logistic\-regression classifiers trained on trajectory features, one using only the current turn\-level feature vector and one using a summarized partial\-dialogue representation; \(2\)Prompting baselines, in which three instruction\-tuned LLMs, Llama\-3\.1\-8B, Mistral\-7B\-Instruct, and Qwen2\.5\-7B, are evaluated under zero\-shot, zero\-shot chain\-of\-thought \(CoT\), and few\-shot paradigms, each receiving the serialized partial dialogue and belief state and outputting aSUCCESS/FAILUREprediction with a confidence score mapped to a continuous failure probability, with few\-shot prompts including three labeled examples \(one per dominant failure type\) and all prompting baselines using oracle belief states \(full prompt templates in Appendix[J](https://arxiv.org/html/2607.03974#A10)\); and \(3\)Neural ablations, consisting of two single\-stream variants used in the ablation study \(Section[4](https://arxiv.org/html/2607.03974#S4)\), afeatures\-onlypredictor using only the trajectory encoder and atext\-onlypredictor using only the belief\-text encoder, which isolate the contribution of each information source before fusion\.

### A\.3Metrics

For forecasting we report AUC\-ROC \(ranking quality\), F1 at thresholdτ\\tau, Early Detection Score \(EDS\) \(normalized earliness of the first correct trigger, ranging from 1\.0 at turn 1 to 0\.0 if never fired\), and mean detection turnt¯∗\\bar\{t\}^\{\*\}\. For threshold analysis we sweepτ∈\[0\.1,0\.9\]\\tau\\in\[0\.1,0\.9\]and report detection rate and false positive rate \(FPR\); the final threshold is selected on the development set to maximize EDS subject toFPR≤0\.15\\text\{FPR\}\\leq 0\.15\. Formal definitions are in Appendix[H](https://arxiv.org/html/2607.03974#A8)\.

### A\.4Implementation Details

TRACER uses a 3\-layer Transformer \(d=64d\{=\}64, 4 heads\) over 5\-dimensional trajectory features and a RoBERTa\-base belief\-text encoder with lower layers partially frozen, fused by a two\-layer MLP \(hidden size 256\)\. Training uses AdamW \(lr=10−4\\text\{lr\}\{=\}10^\{\-4\}, reduced encoder rate\), batch size 32, and early stopping on development AUC\. The trigger threshold is selected on the development split by sweepingτ∈\[0\.1,0\.9\]\\tau\\in\[0\.1,0\.9\]to maximize EDS subject toFPR≤0\.15\\text\{FPR\}\\leq 0\.15, yieldingτ=0\.7788\\tau=0\.7788, which is then frozen for all test\-time evaluation\. Cross\-domain evaluation is zero\-shot: trained on MultiWOZ, tested on SGD and ABCD without adaptation\. Full training details are in Appendix[C](https://arxiv.org/html/2607.03974#A3)\.

## Appendix BDatasets and Preprocessing

All three datasets are converted into a common dialogue representation\. Each dialogue is stored as a sequence of turns containing the user utterance, the aligned system response, a cumulative belief state \(a mapping from domain\-slot pairs to current values\), a turn\-level change set of newly modified slots, the active domain, and a dialogue\-level success label\. One cached split is produced per partition \(train, development, test\)\. Because the forecasting task is partial\-dialogue\-based, the cache stores one dialogue object per conversation; the expansion into one supervised example per visible prefix is performed at data\-loading time, ensuring consistent views across all models and analyses\.

### B\.1MultiWOZ 2\.4

#### Source and structure\.

MultiWOZ 2\.4 is a large\-scale, human\-to\-human, multi\-domain corpus covering seven service domains: hotel, restaurant, attraction, train, taxi, hospital, and police\. Every dialogue begins with a user goal specifying information requests or booking tasks, possibly spanning several domains\. Raw dialogue logs are merged with processed annotation files that provide curated belief\-state annotations and per\-turn domain labels\.

#### Turn representation\.

Each turn records: the cumulative belief state \(a flat mapping of all confirmed slot values so far\), the turn\-level change set \(slots added or revised at this turn\), and the active domain from the dialogue’s declared domain list\.

#### Success labeling\.

For booking\-capable domains \(hotel, restaurant, train\), a dialogue is successful only if at least one booking was confirmed\. For information\-only domains \(attraction, taxi, hospital, police\), success requires the system’s final state to satisfy the user’s original goal constraints\. Value matching normalizes surface forms, handles pipe\-separated alternatives, and allowsdontcareto match any value\. Dialogues with empty goals are successful by default\.

#### Data splits\.

We follow the standard train, development, and test partitions from the official MultiWOZ 2\.4 release\. The test partition is held out for all final reported results\.

### B\.2Schema\-Guided Dialogue \(SGD\)

#### Source and structure\.

SGD is a large, crowd\-sourced, multi\-domain corpus in which each conversation involves one or more named services described by a schema\. Conversations alternate strictly between user and system turns\. System turns carry explicit action annotations \(e\.g\., confirming a value, requesting a slot, notifying transaction success or failure\), making it possible to infer dialogue outcome directly from the action stream\.

#### Turn representation\.

For each user turn, structured frame annotations yield the active intent and cumulative slot\-value pairs across all active services\. Slot names are prefixed with their service identifier to prevent ambiguity\. Consecutive user–system pairs are merged into unified turn objects; the belief state comes from the user turn’s frame, and the change set is computed by comparing consecutive cumulative states\.

#### Success labeling\.

Success is transaction\-based: each dialogue is scanned for system turns carrying a success or failure notification\. The final such notification determines the label\. Dialogues with no transactional actions \(purely informational\) are treated as successful by default\.

#### Data splits\.

SGD provides standard train, development, and test partitions alongside service schema definitions\. The MultiWOZ\-trained TRACER model is evaluated on the SGD test partition zero\-shot, with no fine\-tuning or schema adaptation\.

### B\.3ABCD \(Action\-Based Conversations Dataset\)

#### Source and structure\.

ABCD is a customer\-service corpus structured around the completion of operational actions rather than slot filling\. Each dialogue has a scenario encoding the customer’s context and a knowledge\-base\-defined sequence of required agent actions\. Unlike MultiWOZ and SGD, ABCD provides no formal belief\-state annotations; task progress is tracked through the discrete actions the agent takes\.

#### Turn representation\.

Conversations involve three participant types: customer, agent, and an action track recording operational steps\. Customer–agent utterances are paired into dialogue turns \(de\-lexicalized versions are preferred when available\)\. Action turns are matched against a catalogue of known operational phrasings and recorded as discrete task steps\.

#### Pseudo belief\-state construction\.

We construct a proxy belief state by tracking when scenario values \(customer name, email, order identifier, etc\.\) appear in utterances\. Each mentioned value accumulates into a growing pseudo\-state\. This representation is noisier than the curated belief states in MultiWOZ and SGD, which is one reason cross\-domain transfer results involving ABCD should be interpreted with caution\.

#### Success labeling\.

A dialogue is successful if the agent completes at least 80% of the knowledge\-base\-required actions in order\. This threshold accommodates abbreviated conversations while still requiring substantial task progress\.

#### Data splits\.

Train, development, and test partitions are stored in a single file with per\-dialogue split annotations\. The MultiWOZ\-trained model is evaluated on the ABCD test partition zero\-shot\.

### B\.4Feature Computation

Trajectory features are computed from the cached unified representations and stored separately per split\. For MultiWOZ and SGD, the feature extractor computes the 5\-dimensional trajectory vector \(oscillation score, coverage rate, conflict count, fill velocity, domain shifts\) at every turn\. Coverage is measured against required domain slots; conflict count uses a sentence\-embedding\-based detector that consults the current turn delta and the previous belief state, grounding detection in actual slot changes rather than raw semantic similarity\.

ABCD requires adapted proxy features because it lacks formal belief states and ontology\-aligned slot tracking\. Domain shifts are effectively collapsed \(ABCD dialogues are single\-flow\), and the remaining features are interpreted through the pseudo\-state and action history\. The ABCD feature space is therefore comparable but not identical to that of MultiWOZ and SGD\.

### B\.5Partial Dialogue Dataset Construction

For a dialogue of lengthTT, one supervised example is created for every visible partial dialogue ending at turnt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}\. Each example contains: \(i\) the trajectory\-feature sequence from turn 1 throughtt, \(ii\) the serialized belief\-state text for the same prefix, \(iii\) the final dialogue label, and \(iv\) the turn position and total length\. This construction is shared across the full TRACER model and all ablations, ensuring all forecasting comparisons operate on the same supervision and visible context\. One practical consequence is that each dialogue contributes multiple labeled examples, creating class imbalance at the partial\-dialogue level; another is that the same dialogue can appear easy at some prefix lengths and hard at others, which is what enables the threshold tradeoff and fixed\-context analyses\.

## Appendix CAdditional Training and Implementation Details

### C\.1Optimization Setup

TRACER is trained with AdamW \(base learning rate10−410^\{\-4\}; belief\-text encoder at 10% of base rate\)\. Training uses linear warmup followed by cosine decay, with a maximum of 30 epochs, batch size 32, gradient clipping at 1\.0, and early stopping \(patience 5\) on development\-set AUC\. The full model combines binary cross\-entropy with the early\-prediction reward \(λ=0\.3\\lambda=0\.3\)\. Positive\-class upweighting compensates for class imbalance from partial\-dialogue expansion\.

### C\.2Checkpoint Selection and Threshold Freezing

Model checkpoints are selected on the development set by development\-set AUC; the same procedure applies to the features\-only and text\-only ablations\. All final reported results use these development\-selected checkpoints, reused across the threshold sweep, fixed\-context evaluation, and cross\-domain transfer study without retraining\. The trigger threshold is then selected by sweepingτ∈\[0\.1,0\.9\]\\tau\\in\[0\.1,0\.9\]on the development set and choosing the point that maximizes EDS subject toFPR≤0\.15\\text\{FPR\}\\leq 0\.15, yieldingτ=0\.7788\\tau=0\.7788, which is frozen for all test\-time analyses\. This two\-stage procedure ensures that differences across experiments reflect experimental conditions rather than model configuration\.

### C\.3T5 Recovery Training

The T5\-small recovery generator is fine\-tuned separately from the failure predictor on MultiWOZ\-derived confirmation and clarification utterances \(5 epochs, batch size 16, learning rate5×10−55\\times 10^\{\-5\}\)\. Training inputs concatenate trajectory features, the current belief state, and recent dialogue context; targets are short recovery utterances\.

## Appendix DAdditional Forecasting Results

### D\.1Cross\-Domain Transfer Results

DatasetAUC\-ROCF1EDSDet\. TurnSGD0\.4940\.0690\.5983\.29ABCD0\.7710\.7720\.8191\.15Table 4:Zero\-shot cross\-domain transfer from a MultiWOZ\-trained TRACER model\. Transfer is mixed: the method remains strong on ABCD but degrades sharply on SGD\.
### D\.2Threshold Operating Points

Table[5](https://arxiv.org/html/2607.03974#A4.T5)reports all representative operating points from the threshold sweep with the full set of associated quantities\. The numbers show how precision, recall, detection rate, false positive rate, and mean detection turn move together across different operating policies\. Notably, the unconstrained best\-EDS point \(dev,τ=0\.1\\tau\{=\}0\.1\) achieves EDS of 1\.0 only by triggering on essentially every dialogue immediately, making it an unusable intervention policy\. The frozen threshold \(τ=0\.779\\tau\{=\}0\.779\) selected underFPR≤0\.15\\text\{FPR\}\\leq 0\.15provides a practical balance between early warning and unnecessary interventions\.

SplitSelectionτ\\tauPrec\.RecallF1EDSDet\. RateFPRdevFrozen threshold0\.7790\.4890\.2330\.3150\.2430\.4850\.148testFrozen threshold0\.7790\.5520\.2830\.3740\.2990\.5400\.141devBest EDS0\.1000\.1991\.0000\.3311\.0001\.0001\.000devBest F10\.5440\.3610\.5450\.4340\.5980\.8010\.466devLowest\-FPR nontrivial0\.9000\.6050\.1060\.1810\.1180\.2350\.041Table 5:Expanded threshold operating points from the MultiWOZ threshold sweep\. The unconstrained best\-EDS point is included for completeness but is not a practical intervention policy\.
### D\.3Fixed\-Context Numeric Results

Table[6](https://arxiv.org/html/2607.03974#A4.T6)gives the full numeric fixed\-context evaluation used to support Figure[4](https://arxiv.org/html/2607.03974#S4.F4)\. The pattern is monotonic across all major metrics: as more of the dialogue becomes visible, discrimination quality, detection rate, and EDS improve steadily\. This table is useful as an exact reference because the main text emphasizes the trend, while the appendix preserves the specific values associated with each visible\-context condition\.

Visible ContextAUC\-ROCPrecisionRecallF1AccuracyEDSDet\. RateMean Detect Turn25%0\.6660\.5100\.1310\.2090\.8030\.1030\.1410\.6150%0\.7220\.5000\.2320\.3170\.8020\.1590\.2581\.6375%0\.8110\.5660\.3890\.4610\.8200\.2400\.4443\.08100%0\.8190\.5500\.5000\.5240\.8200\.2990\.5403\.70Table 6:Numeric fixed\-context forecasting results on MultiWOZ\. The model recovers substantial predictive signal from partial dialogue context well before full dialogue completion\.

## Appendix ETriggered Recovery Analysis

### E\.1Recovery Methods

We compare two recovery methods under the same frozen trigger policy: atemplate\-based moduleand aT5\-based generator\. Both receive identical inputs at trigger time and differ only in how they produce a recovery utterance\. Full design and implementation details are in Appendix[I](https://arxiv.org/html/2607.03974#A9)\.

### E\.2Overall Recovery Comparison

We evaluate whether TRACER’s trigger policy provides a useful intervention point for lightweight recovery actions\. We compare the two available recovery methods under the same frozen threshold and the same triggered dialogue set\. The intervention\-rate and relative\-overhead columns quantify the cost of the shared trigger policy, not method\-specific generation cost; they are intentionally identical across recovery methods\. The central result is unambiguous: the template\-based recovery strategy is substantially stronger than the learned T5 generator on our offline recovery proxy\. The gap is large on groundedness \(0\.669 vs\. 0\.374\), problem targeting \(0\.376 vs\. 0\.011\), and clarification behavior\.

This result suggests that the main challenge in the current recovery setting is not only language generation quality, but*task targeting*\. The T5 generator often produces short, fluent continuations, but these utterances frequently fail to request the missing information or explicitly clarify the instability that caused the trigger\. By contrast, the template\-based method directly leverages the detected feature pattern and current belief state, making it much more likely to produce a grounded clarification or missing\-slot request\. Full recovery comparison results are given in Table[7](https://arxiv.org/html/2607.03974#A5.T7)\.

### E\.3Recovery by Trigger Quality

Table[8](https://arxiv.org/html/2607.03974#A5.T8)stratifies the triggered cases intoEarly correct,Late correct, andFalse positivetriggers\. Early correct triggers test whether the system can intervene with meaningful lead time on dialogues that truly fail\. Late correct triggers test whether a warning that arrives closer to the end of the dialogue still yields a useful intervention\. False positives quantify the quality and cost of recovery utterances on dialogues that would have succeeded without intervention\.

Template recovery remains substantially stronger than T5 across all three trigger\-quality buckets\. The especially large differences in*problem targeting*are notable: T5 stays near zero across all buckets, indicating that its outputs often fail to address the concrete source of instability that caused the trigger\.

The bucketed analysis also shows that recovery quality depends jointly on*timing*,*dialogue context*, and*recovery method*\. Taken together, these results support the conclusion that forecasting and recovery should be studied jointly, and that the current recovery setup does not yet establish a causal improvement in live end\-task success\.

### E\.4Expanded Overall Recovery Comparison

Table[7](https://arxiv.org/html/2607.03974#A5.T7)expands the overall recovery comparison by including trigger timing, utterance length, and generation\-failure statistics\. These additional values reinforce the interpretation in Section 6\. The template\-based method is consistently stronger on groundedness, problem targeting, and missing\-slot requests while operating under the same intervention rate and trigger threshold\. By contrast, the learned T5 generator produces shorter outputs on average and remains much weaker at explicitly targeting the instability that triggered intervention\.

RecoveryNumGroundedTargetingMiss\-SlotClarif\.Mean Trigger TurnMean Trigger FractionUtterance Len\.Template1000\.6690\.3760\.4341\.0003\.570\.47923\.44T51000\.3740\.0110\.0110\.5703\.570\.47914\.50

Table 7:Expanded overall recovery comparison under the frozen trigger policy\. Both available recovery methods are evaluated on the same triggered dialogue set, so the differences reflect recovery quality rather than intervention frequency\.
### E\.5Expanded Recovery by Trigger Quality

Table[8](https://arxiv.org/html/2607.03974#A5.T8)gives the fuller trigger\-quality breakdown\. Two patterns are worth highlighting\. First, the template\-based method remains stronger than T5 in every bucket, including false positives, where an unnecessary intervention can still be more or less well grounded\. Second, the template outputs are longest in the late\-correct bucket, which is consistent with the fact that later triggers often occur after the belief state has accumulated more information and therefore support more detailed confirmation\-style recovery utterances\.

RecoveryTrigger QualityNumGroundedTargetingMiss\-SlotClarif\.Utterance Len\.TemplateEarly correct190\.5260\.3070\.6841\.00018\.63TemplateLate correct170\.7060\.3910\.2791\.00028\.18TemplateFalse positive510\.6610\.4090\.4251\.00023\.29T5Early correct190\.5880\.0150\.0070\.52611\.74T5Late correct170\.1240\.0320\.0340\.70615\.47T5False positive510\.3620\.0060\.0070\.54914\.14Table 8:Expanded recovery\-by\-trigger\-quality results\. The template method remains substantially better than T5 across all buckets, especially on problem targeting and missing\-slot behavior\.
### E\.6Representative Recovery Patterns

The saved recovery outputs also make the behavioral difference between the two recovery methods more concrete\. Strong template examples tend to be explicit missing\-slot requests or grounded confirmation prompts that restate the currently tracked belief state and ask for the unresolved information needed to continue\. Representative cases include hotel and attraction dialogues where the template recovers by naming the current domain, restating the known slots, and directly requesting the remaining fields needed to proceed\. Weaker template examples are much rarer and usually occur in generic taxi clarifications such as “are we still looking for a taxi,” which are grammatical but weakly grounded in the actual source of instability\.

The T5 generator fails in a different way\. Its strongest outputs are typically fluent but underspecified, often confirming some piece of state without clearly requesting the missing information\. Its weakest outputs are generic repetitions or malformed domain\-shifted responses that do not address the triggering problem at all\. This is consistent with the very low problem\-targeting scores reported throughout the paper and helps explain why fluent generation alone is not enough for useful recovery under the shared trigger policy\.

## Appendix FQualitative Examples

### F\.1Forecasting Examples

Table[9](https://arxiv.org/html/2607.03974#A6.T9)gives four representative examples\. The false\-positive examples are especially informative because they show that some so\-called mistakes are semantically understandable under an early\-warning framing\. One representative false\-positive case begins as a restaurant request, then shifts to an attraction interpretation, and later returns to restaurant constraints\. TRACER raises the failure score early because the dialogue exhibits exactly the kind of domain confusion and state inconsistency that the model is designed to monitor, even though the final dataset label is success\.

The false\-negative cases show the opposite failure mode\. In one representative case, the dialogue moves from hotel booking to attraction search to taxi planning, but the local turn\-to\-turn behavior remains relatively smooth\. The score never crosses the trigger threshold, illustrating a failure that is hard to anticipate from partial context when the dialogue looks locally coherent until the final stages\.

Example TypeOutcomeKey SignalObservationFalse positivesuccessDomain confusionThe interaction begins as a restaurant search, shifts into attraction lookup, and later returns to restaurant constraints\. TRACER fires early due to genuine cross\-domain instability, even though the final outcome is success\.False negativefailLate\-revealed failureThe dialogue moves from hotel booking to attraction search to taxi planning, but local turn\-to\-turn behavior stays relatively smooth\. The score never crosses threshold, illustrating a failure that is hard to forecast from partial context\.Good recovery \(template\)failMissing\-slot targetingThe template recovery restates the known hotel name and directly requests the unresolved required slots, yielding a grounded, targeted, and actionable intervention\.Bad recovery \(T5\)failWeak recovery targetingThe T5 generator outputs a bare taxi confirmation without clarifying the unstable arrival\-time field or requesting missing information, failing to address the instability that caused the trigger\.Table 9:Representative qualitative examples illustrating the main interpretation patterns: semantically understandable false positives, hard\-to\-anticipate false negatives, and the contrast between targeted template recovery and weak learned T5 recovery\.
### F\.2Recovery Examples

The recovery case studies further clarify what distinguishes a useful intervention from an unhelpful one\. Strong template examples produce hotel or restaurant clarifications that name the currently tracked belief\-state slots and explicitly request the remaining constraints needed to proceed\. Even when the trigger is a false positive, the template recovery remains grounded and asks for the right missing information\.

By contrast, the weak T5 examples are not merely shorter; they are misaligned with the control objective\. A representative T5 failure outputs a bare taxi confirmation without addressing the unstable timing field\. In another case, the generator degenerates into outputting raw feature strings \(e\.g\., oscillation and coverage values\) rather than a natural clarification move\. These patterns explain why T5 underperforms so sharply on problem targeting despite producing fluent text\.

### F\.3Taxonomy\-Oriented Examples

The qualitative taxonomy analysis also benefits from concrete patterns observed across sampled dialogues\. RepresentativeContradiction / Misalignmentcases contain a stable topic but conflicting commitments or responses across turns\. RepresentativeTask Drift / Domain Confusioncases shift away from the active task or mix incompatible domains in a way that derails progress\. The noisier boundary remains betweenInformation IncompletenessandState Instability: dialogues in these categories often co\-exhibit missing slots and repeated revisions, which partly explains why those two categories retain the small amount of disagreement reported in Table[3](https://arxiv.org/html/2607.03974#S4.T3)\.

## Appendix GAnnotation Details

### G\.1Taxonomy Definitions

Table[10](https://arxiv.org/html/2607.03974#A7.T10)provides the formal definitions of the four failure categories used in the qualitative annotation study\.

CategoryDefinitionInformation IncompletenessThe task fails because required constraints, booking attributes, or other task\-critical details are never stably assembled\. The dialogue ends without the user’s needs fully specified\.Contradiction / MisalignmentThe system and user are misaligned on facts or constraints, leading to explicit corrections, unresolved contradictions, or repeated repair sequences\.State InstabilitySlot values or the dialogue state flip repeatedly without converging to a stable representation\. The belief state oscillates without reaching a coherent final state\.Task Drift / Domain ConfusionThe interaction shifts away from the active task or mixes incompatible domains, causing the system to respond to the wrong goal or lose track of the user’s current objective\.Table 10:Definitions of the four failure categories in the TRACER qualitative taxonomy\. These categories are used as an interpretive analysis framework and are not treated as ground\-truth supervision\.
### G\.2Sampling and Annotation Setup

The annotation study reported in the paper is based on a balanced sample of 60 failed MultiWOZ dialogues\. We first assign each dialogue a heuristic primary label using the saved failure\-analysis pipeline, then sample 15 dialogues from each heuristic category\. This produces a balanced analysis set with 15 examples each for Information Incompleteness, Contradiction / Misalignment, State Instability, and Task Drift / Domain Confusion\.

The sampled dialogues are then passed through a human annotation step\. The purpose of this stage is not to create ground\-truth labels for model training\. Instead, it provides a structured second\-pass interpretation of the sampled failures so that we can assess whether the taxonomy tracks recognizable failure patterns\. The resulting annotations are therefore used only as qualitative evidence in support of the analysis sections of the paper\.

### G\.3Observed Label Distribution

Within the 60\-dialogue sample, the heuristic labeling is exactly balanced by construction, with 15 dialogues per class\. The human\-annotated primary\-label distribution is slightly different: 18 dialogues are assigned to Contradiction / Misalignment, 16 to Task Drift / Domain Confusion, 13 to Information Incompleteness, and 13 to State Instability\. This small redistribution is consistent with the main\-paper result that contradiction and drift are comparatively easy to recognize, while incompleteness and instability remain somewhat harder to separate cleanly\.

### G\.4Heuristic–Human Alignment Pattern

The main alignment pattern can be summarized as follows\. All 15 dialogues heuristically labeled as Contradiction / Misalignment remain on\-diagonal under the human annotation, and the same is true for all 15 Task Drift / Domain Confusion cases\. Information Incompleteness keeps 13 of 15 cases on\-diagonal, with one case moving to Contradiction / Misalignment and one to Task Drift / Domain Confusion\. State Instability also keeps 13 of 15 cases on\-diagonal, with the remaining two cases moving to Contradiction / Misalignment\.

This pattern is important for interpretation\. It suggests that the taxonomy is quite stable for contradiction\-heavy and task\-drift\-heavy failures, but still retains a soft boundary between incompleteness and instability\. That is a sensible qualitative outcome: missing information and unstable state updates often co\-occur in practice, especially in longer task\-oriented dialogues where slot filling and slot revision interact over time\.

### G\.5Scope and Limitations of the Annotation Study

This appendix should be read with the same caveat stated in the main paper: these annotations are not gold labels\. The qualitative study is intended to help interpret what kinds of failures TRACER tends to detect, not to define a new benchmark for supervised taxonomy prediction\. The balanced sampling design is useful for analysis because it ensures coverage of all four categories, but it should not be mistaken for the true label distribution of failures in the underlying dataset\. Likewise, the human annotation adds a structured second opinion, but it does not replace further adjudication or interactive user studies\.

## Appendix HEvaluation Metric Definitions

This appendix provides formal mathematical definitions for all evaluation metrics used in the paper\. Letp^i,t∈\[0,1\]\\hat\{p\}\_\{i,t\}\\in\[0,1\]denote the predicted failure probability for dialogueiiat turntt,yi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}the terminal label \(yi=1y\_\{i\}=1for failure\),ℱ=\{i:yi=1\}\\mathcal\{F\}=\\\{i:y\_\{i\}=1\\\}the set of failed dialogues,𝒮=\{i:yi=0\}\\mathcal\{S\}=\\\{i:y\_\{i\}=0\\\}the set of successful dialogues, andTiT\_\{i\}the total length of dialogueii\. A thresholdτ\\tauconverts continuous scores into binary predictionsy^i,t=𝟏​\[p^i,t≥τ\]\\hat\{y\}\_\{i,t\}=\\mathbf\{1\}\[\\hat\{p\}\_\{i,t\}\\geq\\tau\]\.

### H\.1Forecasting Metrics

#### AUC\-ROC\.

The area under the receiver operating characteristic curve equals the probability that the model ranks a random failing instance above a random succeeding one across all partial\-dialogue pairs:

AUC\-ROC=ℙ\(p^i,t\>p^j,t′∣yi=1,yj=0\)\\text\{AUC\-ROC\}=\\mathbb\{P\}\\bigl\(\\hat\{p\}\_\{i,t\}\>\\hat\{p\}\_\{j,t^\{\\prime\}\}\\mid y\_\{i\}=1,\\;y\_\{j\}=0\\bigr\)

#### Early Detection Score \(EDS\)\.

Letti∗=min⁡\{t:p^i,t\>τ\}t\_\{i\}^\{\*\}=\\min\\\{t:\\hat\{p\}\_\{i,t\}\>\\tau\\\}be the first turn at which the trigger fires for dialogueii\(ti∗=∞t\_\{i\}^\{\*\}=\\inftyif it never fires\)\. The per\-dialogue EDS is the normalized detection earliness:

EDSi=Ti−ti∗Ti\\text\{EDS\}\_\{i\}=\\frac\{T\_\{i\}\-t\_\{i\}^\{\*\}\}\{T\_\{i\}\}EDSi=1\\text\{EDS\}\_\{i\}=1when detection occurs at the first turn;EDSi=0\\text\{EDS\}\_\{i\}=0when no trigger fires\. The corpus\-level EDS averages over all failed dialogues:

EDS=1\|ℱ\|​∑i∈ℱEDSi\\text\{EDS\}=\\frac\{1\}\{\|\\mathcal\{F\}\|\}\\sum\_\{i\\in\\mathcal\{F\}\}\\text\{EDS\}\_\{i\}

#### Mean Detection Turn\.

t¯∗=1\|\{i∈ℱ:ti∗<∞\}\|​∑i∈ℱti∗<∞ti∗\\bar\{t\}^\{\*\}=\\frac\{1\}\{\|\\\{i\\in\\mathcal\{F\}:t\_\{i\}^\{\*\}<\\infty\\\}\|\}\\sum\_\{\\begin\{subarray\}\{c\}i\\in\\mathcal\{F\}\\\\ t\_\{i\}^\{\*\}<\\infty\\end\{subarray\}\}t\_\{i\}^\{\*\}

### H\.2Threshold Tradeoff Metrics

#### Detection Rate and False Positive Rate\.

DetRate=\|\{i∈ℱ:ti∗<∞\}\|\|ℱ\|\\text\{DetRate\}=\\frac\{\|\\\{i\\in\\mathcal\{F\}:t\_\{i\}^\{\*\}<\\infty\\\}\|\}\{\|\\mathcal\{F\}\|\}
FPR=\|\{i∈𝒮:∃t,p^i,t\>τ\}\|\|𝒮\|\\text\{FPR\}=\\frac\{\|\\\{i\\in\\mathcal\{S\}:\\exists\\,t,\\;\\hat\{p\}\_\{i,t\}\>\\tau\\\}\|\}\{\|\\mathcal\{S\}\|\}A dialogue in𝒮\\mathcal\{S\}contributes to the FPR numerator if the trigger fires at*any*turn, not just at one specific turn\. This definition penalizes policies that fire frequently on successful dialogues regardless of when they fire\.

### H\.3Recovery Quality Metrics

For each triggered dialogue, TRACER fires at the first turnt∗t^\{\*\}wherep^i,t∗\>τ\\hat\{p\}\_\{i,t^\{\*\}\}\>\\tau\. A recovery utterancerris then generated and scored against the dialogue state at turnt∗t^\{\*\}\.

#### Grounded \(GG\)\.

Measures whetherrris grounded in the current belief statebt∗b\_\{t^\{\*\}\}\. Up to 6 non\-empty belief\-state slot\-value pairs are considered\. For each slotsswith valuevv, a match occurs if either the slot name \(in any of its surface variants\) or the slot value appears in the normalized text ofrr:

G​\(r\)=\|\{\(s,v\)∈bt∗\(6\):slot\_match​\(s,r\)∨value\_match​\(v,r\)\}\|\|bt∗\(6\)\|G\(r\)=\\tfrac\{\\bigl\|\\\{\(s,v\)\\in b\_\{t^\{\*\}\}^\{\(6\)\}:\\text\{slot\\\_match\}\(s,r\)\\vee\\text\{value\\\_match\}\(v,r\)\\\}\\bigr\|\}\{\\bigl\|\\,b\_\{t^\{\*\}\}^\{\(6\)\}\\,\\bigr\|\}wherebt∗\(6\)b\_\{t^\{\*\}\}^\{\(6\)\}denotes the first 6 non\-empty slots ofbt∗b\_\{t^\{\*\}\}sorted lexicographically\.

#### Targeting \(PP\)\.

Measures whetherrraddresses the*problem slots*: slots that are missing from the current belief state \(required but unfilled\) or that become unstable after the trigger turn \(their value changes in at least one future turn\)\. Letℳ\\mathcal\{M\}be the set of missing required slots and𝒰\\mathcal\{U\}the set of future\-unstable slots\. The combined problem set isΠ=ℳ∪\{s​’s local name:s∈𝒰\}\\Pi=\\mathcal\{M\}\\cup\\\{s\\text\{'s local name\}:s\\in\\mathcal\{U\}\\\}:

P​\(r\)=\|\{s∈Π:slot\_match​\(s,r\)\}\|\|Π\|P\(r\)=\\frac\{\|\\\{s\\in\\Pi:\\text\{slot\\\_match\}\(s,r\)\\\}\|\}\{\|\\Pi\|\}IfΠ=∅\\Pi=\\emptyset, thenP​\(r\)=0P\(r\)=0\.

#### Missing\-Slot Request \(MM\)\.

Measures whetherrrexplicitly requests the*missing required*slots only \(not unstable slots\):

M​\(r\)=\|\{s∈ℳ:slot\_match​\(s,r\)\}\|\|ℳ\|M\(r\)=\\frac\{\|\\\{s\\in\\mathcal\{M\}:\\text\{slot\\\_match\}\(s,r\)\\\}\|\}\{\|\\mathcal\{M\}\|\}Ifℳ=∅\\mathcal\{M\}=\\emptyset, thenM​\(r\)=0M\(r\)=0\.

#### Clarification Intent \(CC\)\.

A binary indicator that checks whetherrrcontains a question mark or any of the following keywords:confirm, clarify, correct, right, make sure, just to, could you provide, could you confirm, is that, would you like, i still need:

C\(r\)=𝟏\[“?”∈r∨∃k∈𝒦,k∈lower\(r\)\]C\(r\)=\\mathbf\{1\}\\bigl\[\\text\{\`\`?''\}\\in r\\;\\vee\\;\\exists\\,k\\in\\mathcal\{K\},\\;k\\in\\text\{lower\}\(r\)\\bigr\]where𝒦\\mathcal\{K\}is the keyword set above\.

#### Intervention Rate\.

The fraction of test dialogues for which the trigger fires at least once:

Interv\. Rate=\|\{i∈𝒟test:ti∗<∞\}\|\|𝒟test\|\\text\{Interv\. Rate\}=\\frac\{\|\\\{i\\in\\mathcal\{D\}\_\{\\text\{test\}\}:t\_\{i\}^\{\*\}<\\infty\\\}\|\}\{\|\\mathcal\{D\}\_\{\\text\{test\}\}\|\}

#### Relative Overhead\.

Under the assumption that each recovery intervention adds one extra dialogue turn, the relative overhead is the intervention rate normalized by the mean dialogue lengthT¯\\bar\{T\}:

Rel\. Overhead=Interv\. RateT¯\\text\{Rel\. Overhead\}=\\frac\{\\text\{Interv\. Rate\}\}\{\\bar\{T\}\}
T¯=1\|𝒟test\|​∑i∈𝒟testTi\\bar\{T\}=\\frac\{1\}\{\|\\mathcal\{D\}\_\{\\text\{test\}\}\|\}\\sum\_\{i\\in\\mathcal\{D\}\_\{\\text\{test\}\}\}T\_\{i\}Both Template and T5 recovery share the same Interv\. Rate and Rel\. Overhead because they operate under the same TRACER trigger policy; only the generated utterance differs\.

## Appendix IRecovery Method Details

### I\.1Triggered Recovery Policy

The predictor becomes a control layer by thresholding the partial\-dialogue\-level failure score\. For a partial dialogue at turntt, ifp​\(y=1∣D≤t\)\>τp\(y=1\\mid D\_\{\\leq t\}\)\>\\tau, the system triggers a recovery action\. We selectτ\\tauon the development set under a false\-positive\-rate constraint and then freeze that operating point for test\-time evaluation\. This distinction matters because unconstrained low thresholds can maximize early\-detection metrics trivially while producing impractically many unnecessary interventions\.

We study two recovery methods under this shared trigger policy\. The first is atemplate\-based recovery module, which maps dominant feature patterns such as oscillation, coverage gaps, contradiction, or domain confusion to short clarification or confirmation prompts\. The second is aT5\-based recovery generator, implemented with a fine\-tuned T5 model\(Raffel et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib19)\)\. Its input concatenates the current trajectory features, the current belief state, and the most recent dialogue context, and it generates a short recovery utterance intended to confirm the user goal or request missing information\.

This appendix describes the two recovery methods in detail\. Both methods receive the same input at trigger time: the trajectory feature vector𝐟t∗\\mathbf\{f\}\_\{t^\{\*\}\}, the current belief statebt∗b\_\{t^\{\*\}\}, the active domain, and a short window of recent dialogue turns\. They differ only in how they map this input to a recovery utterance\.

### I\.2Template\-Based Recovery

The template\-based recovery module first classifies the dominant failure type from the trigger\-turn feature vector, then selects a template from a small fixed set and fills it with the current belief\-state content\.

#### Failure type detection\.

The dominant failure type is determined by thresholding the trajectory features in priority order:

1. 1\.Oscillation\(OscScoret∗≥2\\text\{OscScore\}\_\{t^\{\*\}\}\\geq 2\): the belief state is oscillating; use a confirmation template\.
2. 2\.Coverage gap\(CoverageRatet∗<0\.4\\text\{CoverageRate\}\_\{t^\{\*\}\}<0\.4\): required slots are missing; use a missing\-slot request template\.
3. 3\.Conflict\(ConflictCountt∗≥2\\text\{ConflictCount\}\_\{t^\{\*\}\}\\geq 2\): repeated contradictions detected; use a clarification template\.
4. 4\.Domain confusion\(DomainShiftst∗≥3\\text\{DomainShifts\}\_\{t^\{\*\}\}\\geq 3\): excessive domain switching; use a domain\-focus template\.
5. 5\.General\(none of the above\): use a general confirmation template\.

#### Template structure\.

Each failure type maps to 2–3 surface\-form templates\. For example, the coverage\-gap type maps to utterances of the form:

> “To complete your\{domain\}request, I still need:\{missing\_slots\}\. Could you provide those?”

and the oscillation type maps to utterances of the form:

> “I notice some changes in your preferences\. Just to confirm, you’d like a\{domain\}with\{slot\_text\}?”

One template per type is selected uniformly at random and filled with the domain name, the formatted belief\-state slots, and the list of missing required slots\. The resulting utterance is always a short, directly actionable clarification or confirmation move\.

### I\.3T5\-Based Recovery

The learned recovery model is a fine\-tuned T5\-small sequence\-to\-sequence generator\(Raffel et al\.,[2020](https://arxiv.org/html/2607.03974#bib.bib19)\)\. It is trained separately from the failure predictor on MultiWOZ\-derived confirmation and clarification utterances from the training split\.

#### Input format\.

Each training and inference input concatenates three information sources in a fixed order:

1. 1\.A string rendering of the five trajectory features at the trigger turn, e\.g\.osc=1\.0 cov=0\.33 conflict=1 velocity=0\.5 domains=2\.
2. 2\.A compact serialization of the current belief state, e\.g\.hotel\-area: west, hotel\-pricerange: cheap\.
3. 3\.The most recent 3 dialogue turns formatted as alternatingUSER:andSYSTEM:lines\.

#### Training\.

The T5\-small generator is fine\-tuned for 5 epochs with batch size 16 and learning rate5×10−55\\times 10^\{\-5\}\. Training targets are system utterances from the training split that function as confirmations or recovery moves \(e\.g\. utterances following a belief\-state correction or preceding a booking action\)\. Training uses a linear warmup schedule over 10% of total steps\.

#### Inference\.

At trigger time, the input is assembled from the live trajectory features, belief state, and recent context\. Generation uses greedy decoding with a maximum of 128 new tokens\. The T5 checkpoint is loaded once and reused for all triggered dialogues in the evaluation set\.

As shown in the main paper, the T5 generator substantially underperforms the template method on problem targeting despite producing fluent text\. This outcome highlights that the challenge in recovery is not language generation quality but correct identification and addressing of the instability source, a property that the template method handles directly through its feature\-driven routing logic\.

## Appendix JLLM Prompt Templates

This appendix provides the prompt templates used in two distinct LLM inference pipelines: \(1\) zero\-shot failure prediction, where an instruction\-tuned model classifies each partial dialogue as likely to succeed or fail; and \(2\) generated belief\-state tracking, where a model predicts cumulative belief states turn by turn\.

### J\.1Zero\-Shot Failure Prediction Prompts

These prompts are used to evaluate Llama\-3\.1\-8B, Mistral\-7B\-Instruct, and Qwen2\.5\-7B as zero\-shot failure forecasters \(theZero\-Shot Promptingrows in Table[1](https://arxiv.org/html/2607.03974#S3.T1)\)\. Each model receives the same system and user prompts\.

System PromptYou are evaluating a task\-oriented dialogue prefix\. Your job is to forecast whether the full dialogue will eventually end in SUCCESS or FAILURE\. Use only the visible prefix\. Respond with valid JSON only\.

#### User prompt template\.

The following template is instantiated for every partial dialogue\(i,t\)\(i,t\)\.\{dialogue\_block\}is the turn\-by\-turn prefix withTurnkkUSER:/TurnkkSYSTEM:prefixes;\{belief\_text\}is the oracle cumulative belief state serialized as"domain\-slot: value"pairs\.

User PromptGiven the task\-oriented dialogue prefix below, predict whether the full dialogue will eventually succeed or fail\.Rules:•Predict FAILURE if the dialogue appears likely to end without satisfying the user’s full goal\.•Predict SUCCESS if the visible evidence suggests the task is likely to be completed\.•Use the visible prefix only\. Do not assume future repair unless strongly supported\.•Return valid JSON only\.Visible prefix:\{dialogue\_block\}Current cumulative belief state:\{belief\_text\}Return JSON with exactly this schema:\{“prediction”: “FAILURE” or “SUCCESS”,“confidence”:<<integer from 0 to 100\>\>,“rationale”: “<<one short sentence\>\>”\}

The confidence score is mapped to a failure probability viap^=confidence/100\\hat\{p\}=\\texttt\{confidence\}/100whenpredictionis"FAILURE"andp^=1−confidence/100\\hat\{p\}=1\-\\texttt\{confidence\}/100otherwise, yielding a continuous score for AUC computation\.

### J\.2Generated Belief\-State Prompts

These prompts are used to generate predicted belief states with Qwen3\-30B\-A3B\. The predicted belief states replace oracle states in theTRACER \(generated\)row of Table[1](https://arxiv.org/html/2607.03974#S3.T1)\. The pipeline processes all dialogues turn by turn in lockstep, feeding each model output as theprevious\_belief\_statefor the next turn\.

#### System prompt \(constrained\_careful\)\.

This is the primary system prompt used for belief\-state generation\. It enforces the MultiWOZ domain–slot ontology and provides detailed tracking rules\.

System PromptYou are a precise dialogue state tracker for MultiWOZ\-style task\-oriented dialogue\. Predict the cumulative belief state after the latest USER utterance\.Rules:•Use only the dialogue text shown in the prompt\.•Output only valid JSON\. Do not include markdown, explanations, or comments\.•The JSON must have exactly these top\-level keys: “domain” and “belief\_state”\.•“domain” must be the active domain/service after the latest USER utterance\.•“belief\_state” must be a JSON object mapping slot keys to string values\.•Slot keys must use the format “domain\-slot”, for example “restaurant\-area”\.•Use lowercase domain and slot names\.•Start from the previous belief\_state given in the prompt\.•Return the full cumulative belief\_state, not only the new changes\.•Keep earlier slot values across domains unless the user explicitly corrects, replaces, or cancels them\.•If the user corrects a domain, move the value to the corrected domain and remove the wrong\-domain version\.•Track user constraints such as area, food, price range, type, name, day, time, people, departure, destination, leave time, and arrival time\.•Track booking information when the user gives booking day, time, stay length, or number of people\.•If the system offers a specific entity and the user accepts it or asks follow\-up details about it, include that entity as the corresponding “domain\-name”\.•If the user asks for address, postcode, phone, entrance fee, reference number, rating, or car type, treat these as requested information, not belief\-state slots\.•If the user says “same area”, copy the relevant earlier area value to the new domain\.•Do not hallucinate values from database results unless the dialogue text makes the user accept or refer to that entity\.•Omit unknown, not mentioned, none, empty, or unsupported slots\.•Values should be short normalized strings, not full sentences\.

#### User prompt template \(constrainedstyle\)\.

The following template is instantiated at each turntt\.\{schema\}lists alloweddomain: slot, slot, \.\.\.lines;\{prefix\}is the dialogue context rendered asTurnkkUSER:/TurnkkSYSTEM:lines; and\{previous\_json\}is the JSON\-serialized belief state from the previous turn\.

User PromptAllowed domains and slots:\{schema\}Dialogue prefix:\{prefix\}Previous belief\_state before the latest USER utterance:\{previous\_json\}Return the belief state after the latest USER utterance\.Required JSON format:\{"domain": "restaurant", "belief\_state": \{"restaurant\-area": "north"\}\}

Slot alias normalization is applied to model outputs before the belief state is stored: for example,centre/center→\\toarea,price→\\topricerange,depart→\\toleaveat\. Values are lowercased and the special form"center"is normalized to"centre"\. If the model output cannot be parsed as valid JSON, a regex fallback extracts the innermost JSON object from the raw output string\. The turn deltaΔt\\Delta\_\{t\}is then computed by comparing the predicted belief state against the previous turn’s predicted state\.