Wait! There's a Way Out: A Decision Mechanism for Forecasting Conversational Derailment

# A Decision Mechanism for Forecasting Conversational Derailment
Source: [https://arxiv.org/html/2605.29243](https://arxiv.org/html/2605.29243)
Laerdon Kim, Vivian Nguyen, Cristian Danescu\-Niculescu\-Mizil Cornell University \{lyk25, vn72\}@cornell\.educristian@cs\.cornell\.edu

###### Abstract

Forecasting conversational derailment is the task of predicting, as the conversation unfolds, whether it will eventually derail into personal attacks\. Since forecasting models operate in an online fashion, they must decide whether to "trigger" an alert after each utterance—for example, to notify participants or a moderator that the conversation is at risk of derailing\. Existing approaches make this decision solely based on the estimated likelihood of derailment given the preceding utterances, implicitly assuming that the conversation’s future trajectory is fixed\. As a result, they ignore the possibility of future recovery and incur an unnecessarily high rate of false positives\.

In this work we propose a method for decoupling the decision to trigger from the derailment likelihood estimation\. Our approach is inspired by the first human baseline on this task, which shows that humans achieve dramatically lower false positive rates by selectively deferring their decision to trigger when they anticipate that tension is likely to subside\. We operationalize this insight with a deferral mechanism that uses forward\-looking simulations to assess whether a tense moment admits plausible paths to recovery\. Incorporating this mechanism into a state\-of\-the\-art forecasting model substantially reduces false positives without sacrificing forecasting accuracy\. More broadly, this work highlights the value of treating decision making as a first\-class component of forecasting systems\.

Wait\! There’s a Way Out: A Decision Mechanism for Forecasting Conversational Derailment

Laerdon Kim, Vivian Nguyen, Cristian Danescu\-Niculescu\-MizilCornell University\{lyk25, vn72\}@cornell\.educristian@cs\.cornell\.edu

## 1Introduction

> "The best way to predict the future is to invent it\." – Alan Kay

![Refer to caption](https://arxiv.org/html/2605.29243v1/delay-intro-figure-CR.jpg)Figure 1:Current systems for forecasting conversational derailment conflate tension estimation with the decision to trigger an alert\. As illustrated in this example, this can lead to false positives\. We devise a mechanism that defers triggering when it anticipates a possible de\-escalation \(via forward\-looking simulations\)\. Background colors indicate the probability of derailment calculated based on the context up to that utterance \(fromlowtohigh\)\.Conversational forecasting is the task of predicting whether a conversational event—such as a personal attackChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\), disengagementNguyenet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib1)\), or prosocial behaviorBaoet al\.\([2021](https://arxiv.org/html/2605.29243#bib.bib16)\)—willeventuallyoccur in the conversation\. The ability to make such predictions was recognized as a key component of proactive conversational support systemsJurgenset al\.\([2019](https://arxiv.org/html/2605.29243#bib.bib11)\); Korreet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib22)\), and its potential was demonstrated via academic user studiesChanget al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib13)\); Schlugeret al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib25)\), and real\-world product deploymentsLi \([2022](https://arxiv.org/html/2605.29243#bib.bib23)\)\.

One of the central challenges in forecasting—distinguishing it from traditional static classification tasks—is the “unknown horizon” of the target eventChanget al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib13)\)\. Take the example of forecasting conversational derailment, which is the focus of this work: a personal attack can occur at any time, requiring a forecasting system to decide after every comment whether to trigger an alert or to wait for the conversation to further develop\. Early triggering can lead to spurious alarms driven by limited information, whereas delayed triggering risks missing the opportunity for timely intervention\. Consequently, effective forecasting models must separatebelief estimationfromdecision\-making: beyond estimating the current risk of derailment \([Figure 1](https://arxiv.org/html/2605.29243#S1.F1)left\), they must decide whether the evidence observed so far warrants triggering an alarm or whether it is preferable to wait for more information \([Figure 1](https://arxiv.org/html/2605.29243#S1.F1)right\)\.

However, all existing models conflate belief estimation with decision\-making by triggering alerts via a fixed threshold on the estimated probability of derailment, regardless of whether this probability is calculated via hierarchical recurrent neural networksChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\), graph convolutional networksAltarawnehet al\.\([2023](https://arxiv.org/html/2605.29243#bib.bib17)\), transformer architecturesKementchedjhieva and Søgaard \([2021](https://arxiv.org/html/2605.29243#bib.bib24)\), hierarchical transformer architecturesYuan and Singh \([2023](https://arxiv.org/html/2605.29243#bib.bib19)\), or LLM promptingOlpadkaret al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib5)\)\. In effect, these models function as classifiers that estimate the level of conversational tension up to the current moment and trigger an alert once a predetermined threshold is crossed \(in[Figure 1](https://arxiv.org/html/2605.29243#S1.F1)an existing model would trigger right afteruku\_\{k\}\)\. As a result, such models are blind to the possibility that conversational tension may later subside \(a possibility exemplified in[Figure 1](https://arxiv.org/html/2605.29243#S1.F1)starting withuk\+1u\_\{k\+1\}\)\. Consequently, current forecasting systems cannot reason about the value of waiting for additional evidence before triggering\.

Beyond being a theoretical limitation, the absence of a separate decision\-making component has important practical consequences for forecasting systems and their real\-world deployment\. In particular, the inability to account for the possibility that conversational tension may later resolve leads directly to spurious alerts \(i\.e\., false positives\), which end users identify as the most significant shortcoming of such systems, with 62% reporting false positives as a common issue\(Changet al\.,[2022](https://arxiv.org/html/2605.29243#bib.bib13)\)\.

In this work, we propose the first approach that decouples the triggering decision from the derailment likelihood estimation\. We begin by examining how humans solve the forecasting task and, in the process, establish the first human baseline\. We find that humans achieve a false positive rate that is less than half of the state\-of\-the\-art forecasting model by waiting longer before triggering an alert\. Crucially, this deferral of the decision to trigger is selective: humans appear to anticipate when conversational tension is likely to subside\.

To operationalize this insight, we devise a simulation\-based method for distinguishing tense moments after which a recovery is plausible—that is, there is a “way out” \([Figure 1](https://arxiv.org/html/2605.29243#S1.F1), "Our system"\)\. We implement a deferral mechanism which acts cautiously \(only\) in these tense moments, waiting for the next utterance rather than immediately triggering an alert\. Integrating this mechanism on top of a state\-of\-the\-art forecasting model yields a dramatic reduction in false positives without sacrificing overall accuracy\.

In summary, in this work we:

- •conceptualize and disentangle two components of conversational forecasting that have previously been conflated: belief estimation and decision\-making;
- •introduce the first human baseline for forecasting conversational derailment, showing that humans make more effective triggering decisions to achieve substantially lower false positive rates than state\-of\-the\-art models;
- •propose a decision mechanism inspired by this insight that narrows this false positive gap without sacrificing overall accuracy\.

More broadly, this work illustrates the value of disentangling the decision to trigger from belief estimation in forecasting systems, opening the door to more sophisticated and effective trigger policies\. To encourage further progress in this direction, we publicly release our code in a modular form that explicitly separates the two components, rendering it forward\-compatible with alternative tension estimation models and decision mechanisms\.111Code and data released as part of[ConvoKit](https://convokit.cornell.edu/)\.

## 2Background and Related Work

Conversations Gone Awry\.The development of conversational forecasting models has been catalyzed by the introduction of the Conversations Gone Awry \(CGA\) taskZhanget al\.\([2018](https://arxiv.org/html/2605.29243#bib.bib2)\); Changet al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib13)\); Tranet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib15)\)\. In this task, the event to be forecast is the occurrence of a personal attack\. Accordingly, a model processes the conversation one utterance at a time, with the goal oftriggeringan alert only if it determines that the conversation is likely to culminate in a personal attack\. For a model to be successful, it must either trigger an alertbeforea personal attack occurs \(when one exists\), or correctly refrain from triggering and allow the conversation to reach its end when no personal attack occurs\.

The task includes two datasets\. CGA\-CMV was introduced byChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\)and later expanded byTranet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib15)\)to include20,57620\{,\}576conversations collected from the Change My View subreddit\. Labels are derived from moderator\-marked violations of Rule 2 of Change My View: “Don’t be rude or hostile to other users\.” CGA\-WIKIZhanget al\.\([2018](https://arxiv.org/html/2605.29243#bib.bib2)\)is a much smaller set of4,1884\{,\}188conversations between Wikipedia editors, human\-labeled by crowdworkers as whether they end in personal attacks, and filtered to exclude any “rude, insulting, or disrespectful” comments prior to the final turn\. As a result of this additional filtering, CGA\-WIKI is less naturalistic: it is unlikely to contain conversations exhibiting de\-escalating or recovery trajectories\. Therefore in this work we focus on CGA\-CMV, but report the results on CGA\-WIKI for completeness in[Appendix C](https://arxiv.org/html/2605.29243#A3)\.

Unknown horizon\.The online formulation of the task gives rise to the aforementioned “unknown horizon” challenge inherent in conversational forecastingChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\)\. Because a personal attack—if it occurs—may arise at any point in the interaction, the model does not know in advance when it must act\. Instead, it must repeatedly decide, after each utterance, whether to act or to wait, without knowing whether additional evidence will arrive or whether the conversation is about to conclude\. As a result, the model must balance the risk of acting too early, based on insufficient information, against the risk of acting too late, when intervention may no longer be effective\.

Although this challenge and inherent exploration\-exploitation tradeoff were recognized when the task was first introduced and is reflected in the evaluation protocol—where forecasters which fail to intervene before the conversation derails are penalized—current forecasting systems lack explicit mechanisms for deciding when to act\. Instead, they effectively operate as traditional threshold\-based classifiers, triggering alerts whenever the estimated probability of derailment crosses a fixed cutoff\.

Forecasting formalism: belief estimation\.Formally, the belief estimation component of a forecaster continuously assesses the probability of future derailment after thekk\-th utteranceuku\_\{k\}, conditioned on the preceding context:𝒫​\(derailment\|u1,u2,…,uk\)\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)\. A variety of models have been proposed to estimate this probability, drawing on architectures such as hierarchical recurrent neural networksChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\), graph convolutional networksAltarawnehet al\.\([2023](https://arxiv.org/html/2605.29243#bib.bib17)\), transformersKementchedjhieva and Søgaard \([2021](https://arxiv.org/html/2605.29243#bib.bib24)\), hierarchical transformersYuan and Singh \([2023](https://arxiv.org/html/2605.29243#bib.bib19)\), and decoder\-based generative large language modelsOlpadkaret al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib5)\); Tranet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib15)\)\. These architectures were recently compared using the official public benchmark for the CGA taskTranet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib15)\)\.

Across both the CGA\-CMV and CGA\-WIKI sections of the benchmark, the best\-performing model is Gemma2 9BTeam \([2024](https://arxiv.org/html/2605.29243#bib.bib27)\), which we therefore treat as the state\-of\-the\-art \(SOTA\) for this task\. Such decoder\-only models are fine\-tuned on the training data; the probability of derailment𝒫​\(derailment\|u1,u2,…,uk\)\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)is retrieved by prompting the model to answer whether the conversation will derail afteruku\_\{k\}, sampling the “Yes” and “No” logits, and softmaxing the “Yes” probabilityTranet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib15)\)\.

A key challenge in training forecaster models is that supervision is available only at the conversation level: each conversation is labeled solely by whether it ultimately ends in a personal attack, with no utterance\-level supervisionChang \([2024](https://arxiv.org/html/2605.29243#bib.bib4)\)\. Accordingly, prior work trains models to predict the label of the final utteranceunu\_\{n\}using the preceding contextu1,…,un−1u\_\{1\},\\ldots,u\_\{n\-1\}—even though at test time models must produce a derailment probability after every utterance, not just the last\. While this strategy has been shown to outperform training on partial conversations\(Altarawnehet al\.,[2023](https://arxiv.org/html/2605.29243#bib.bib17)\), it structurally biases models toward detecting immediate tension, rather than learning about longer\-term conversational dynamics\. As a result, these models cannot account for more complex conversational trajectories, such as moments that are \(or appear to be\) tense but are followed by a reduction in perceived tension\. Such recoveries can naturally arise through de\-escalation \(as illustrated in Figure[1](https://arxiv.org/html/2605.29243#S1.F1)\) or through the repair of a misunderstanding or misperception \(e\.g\., a comment that initially appears sarcastic but is later clarified as earnest;Tsaiet al\.\([2024](https://arxiv.org/html/2605.29243#bib.bib30)\),Drew \([1997](https://arxiv.org/html/2605.29243#bib.bib32)\)\)\.

Forecasting formalism: decision\-making\.After every utteranceuku\_\{k\}, the decision component of a forecasting system must choose whether to trigger an alert \(gk=1g\_\{k\}=1\) or to wait for the next utterance \(gk=0g\_\{k\}=0\)\. The responsibility of this component ends either when an alert is triggered—at which point the conversation is deemed to be “going awry” —or when the conversation reaches its end without an alert, in which case it is deemed to “remain calm”\. Current systems, including the SOTA, use a simple threshold based decision, triggering when the estimated tension surpasses a thresholdTTfixed on the validation data:

gk:=𝕀​\{𝒫​\(derailment\|u1,u2,…,uk\)\>T\}g\_\{k\}:=\\mathbb\{I\}\\\{\\;\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)\>T\\;\\\}\(1\)
In this work, we treat decision\-making as a first\-class component of forecasting models and propose the first framework in which triggering decisions are informed not only by the estimated derailment probability, but also by forward\-looking signals about how the conversation may evolve, including the plausibility of a future recovery\.

Evaluation metrics\.In forecasting tasks, standard error types link utterance\-level predictions to conversation\-level labelsChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\)\. A conversation that ends with a personal attack is a true positive if the model triggers before the conversation ends \(gk=1g\_\{k\}=1for somek<nk<n\), and a false negative otherwise \(gk=0g\_\{k\}=0for allk<nk<n\)\. Conversely, a conversation that does not end with a personal attack is a false positive if the model triggers at any utterance, and a true negative otherwise\. Prior work uses these error types to define standard evaluation metrics for comparing forecasting models\. Accuracy is often preferred \(e\.g\., for selecting the triggering threshold\) over F1 and false\-positive rate \(FPR\), as it accounts for all error types\. In addition, the “horizon” \(HH\) is a forecasting\-specific metric that measures how many utterances before an actual personal attack the model triggers an alert \(averaging across all true positives\)\.

Forecasting and simulation\.Since conversational forecasting concerns predicting future conversational events, it is natural to consider whether the generative capabilities of large language models can be used to “peek” into possible future trajectories\. In fact, the first conversational forecasting systems were built on top of a generative framework \(Hierarchical Recurrent Encoder–Decoder\), replacing the generation layer with a prediction headChang and Danescu\-Niculescu\-Mizil \([2019](https://arxiv.org/html/2605.29243#bib.bib3)\)\. More recently,Zhanget al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib26)\)evaluated the utility of modern large language model simulations in astaticvariant of the forecasting task\. In this static variant, however, the forecaster always makes its prediction at a predetermined point in the conversation—namely, immediately before the final comment, assuming \(an unrealistic\) prior knowledge of when the conversation is about to end—thereby removing the need for an explicit decision\-making component\. This work inspires one of our baselines in the full version of the task\.

In the context of mental health crisis counseling conversations,Nguyenet al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib1)\)shows the value of combining next\-utterance simulation with forecasting in determining when a moment is “pivotal”\. Our proposed method also uses next\-utterance simulations, but with the goal of deciding whether there is informational value in deferring the decision to trigger an alarm\.

## 3Human Experiment

To inform the design of the decision component in conversational forecasting systems, we turn to observing how humans solve the task\. We analyze the human triggering decisions and directly compare them to those of the SOTA model, with the goal of identifying systematic differences that can inspire more effective mechanisms for deciding when \(not\) to trigger an alert\.

Despite its prominence, the CGA task has so far lacked a human baseline\. This gap stems in large part from the online nature of the task: unlike standard classification settings, forecasting conversational derailment requires making sequential decisions, where annotators must decide when to act rather than simply what label to assign\. Moreover, because the task involves predicting future events, annotation quality cannot be directly verified at each decision point, making it difficult to rely on standard crowdsourcing pipelines\. To address this gap, we devise a human experiment that places participants in this online forecasting setting, enabling a direct comparison between human and model decision\-making\.

In order to match the forecasting environment while incentivizing participants, we follow previous work in “gamifying” the taskVannellaet al\.\([2014](https://arxiv.org/html/2605.29243#bib.bib20)\)\. Under this formulation, participants are presented with conversations one comment at a time and must choose either to trigger an alert—if they believe the conversation is getting out of hand—or to reveal the next comment\. Their objective is to reveal all utterances in non\-derailing conversations, while triggering an alert in derailing conversationsbeforerevealing the personal attack itself\. A snapshot of the experiment’s interface is included in[Appendix D](https://arxiv.org/html/2605.29243#A4)\. To incentivize participants, we allow them to see whether they answered correctly or not for each conversation, and to compare total scores among themselves after each round\.

The experiment involves three rounds\. The first one is a warmup round consisting of four hand\-picked conversations, to allow the participants to become familiar with the task and ask questions\. The results of this round are discarded from the analysis\. Then we run two main rounds: in each round, each participant goes through 10 conversations\. These conversations are randomly selected from CGA\-CMV\-Large such that half conclude with a personal attack and the other half remain civil\. Given this setup, a random baseline achieves an accuracy of50%50\\%like in the original CGA task\.

To ensure comparability between the two experimental rounds, we use the same set of conversations for both rounds, only changing assignment of conversations to participants between the rounds\. In this way, we ensure that no participant encounters the same conversation twice\. This results in all conversations receiving annotation from two unique participants: one in the first main round and one in the second\. We recruited nine volunteers, each of whom completed all rounds of the game, spending an average of1313minutes per main round\. Due to the random assignment of conversations, some were repeated across participants, resulting in a total of8484unique conversations\.

MethodAcc↑\\uparrowFPR↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowHHuman rnd\-162\.224\.467\.848\.954\.62\.6rnd\-270\.015\.675\.955\.663\.92\.1SOTA70\.036\.267\.976\.271\.83\.3

Table 1:Human performance micro\-averaged across 84 randomly selected conversations from CGA\-CMV\-Large, compared with the SOTA performance on the same sample \(averaged across 5 random seeds\)\. We report accuracy \(Acc\), precision \(P\), recall \(R\), F1, false positive rate \(FPR\), and horizon \(H\)\.Human performance\.Table[1](https://arxiv.org/html/2605.29243#S3.T1)shows that after the second round, human participants achieve accuracy comparable to that of the state\-of\-the\-art \(SOTA\) model\. Importantly, humans exhibit a substantially lower false positive rate—less than half that of the SOTA model—albeit at the cost of recall\. This more cautious strategy is further reflected in a shorter mean horizon,222As discussed in[Section 2](https://arxiv.org/html/2605.29243#S2), the horizon is a forecasting\-specific metric introduced byChanget al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib13)\)and adopted by subsequent work; it measures how many comments before the personal attack an alert is triggered\.indicating that participants tend to wait longer before guessing that a conversation will derail\.

Notably, participants improve from the first round to the second, achieving higher accuracy while simultaneously becoming more cautious, as evidenced by decreases in both false positive rate and horizon\. This pattern suggests that with experience, humans can calibrate their decision\-making, refining their intuitions about conversational trajectories and learning to avoid false positives\.

Comparing human and SOTA decisions\.To better understand the human decision process—and in particular how humans achieve such a substantial reduction in false positive rate—we directly compare human triggering behavior with that of the SOTA model\. A natural hypothesis is that humans are simply more cautious than the model, requiring higher levels of conversational tension before triggering an alert\. To test this hypothesis, we compare the model\-inferred tension \(i\.e\.,𝒫​\(derailment\|u1,u2,…,uk\)\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)\)333For ease of exposition, in what follows we use “tension” to refer to the probability of derailment estimated by the SOTA model, following the discussion about the training procedure of these models in[Section 2](https://arxiv.org/html/2605.29243#S2)\. We acknowledge this as a necessary approximation used for exploratory purposes, since there are no labels of human\-perceived tension\.at the moment humans trigger an alert with the level of tension at which the SOTA model triggers\. We find evidence against this hypothesis: humans trigger, on average, atlowerlevels of estimated tension \(𝒫=0\.61\\mathcal\{P\}=0\.61\) than the model \(𝒫=0\.72\\mathcal\{P\}=0\.72\)\.

Since humans do not appear to be more cautious in general, their lower false positive rate may instead stem fromselectivelyexercising caution in situations where they anticipate that tension will subside\. To explore this possibility, we define the immediate decrease in tension following utterancekkas:

𝒟k:=\\displaystyle\\mathcal\{D\}\_\{k\}=𝒫​\(derailment\|u1,u2,…,uk\)\\displaystyle\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)\(2\)−𝒫​\(derailment\|u1,u2,…,uk\+1\)\.\\displaystyle\-\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\+1\}\)\.with a strictly positive value indicating a decrease in tension\.444We discard cases in whichuk\+1u\_\{k\+1\}is the to\-be\-forecasted personal attack or the last comment of the conversation\.

We find that a large fraction \(61%\) of moments in which SOTA triggers are actually followed by a decrease in tension \(𝒟k\>0\\mathcal\{D\}\_\{k\}\>0\)\. In comparison, humans are much less likely to trigger in moments that are followed by an immediate decrease in tension \(24% of moments in which humans trigger have𝒟k\>0\\mathcal\{D\}\_\{k\}\>0\)\.

If we focus on moments in which humans successfully avert a false positive—specifically, utterances where the SOTA triggers a false alert but humans choose to wait and allow the conversation to continue—we find in 78\.6% of these cases, tension subsequently decreases, a higher proportion than observed for SOTA false positive triggers in general \(71\.6%\)\. Therefore, even in moments where derailment appears plausible to the SOTA model, humans selectively refrain from triggering when they sense that the interaction is likely to recover\.

## 4Method

Humans appear to defer the decision to trigger an alert in moments of high but transient tension\. Motivated by this observation, our goal is to design a decision mechanism that explicitly incorporates the ability to anticipate possible recoveries into the forecaster’s decision\-making process\. We introduce a decision component that directly accounts for possible future conversational dynamics\. This enables the forecaster to distinguish between tense moments that are likely to persist and those that appear recoverable\.

We take a simulation\-based approach to operationalize this intuition\. At each tense moment in a conversation, we simulate plausible next replies in order to estimate whether an observed spike in tension is likely to persist or to subside \(as illustrated in Figure[1](https://arxiv.org/html/2605.29243#S1.F1)\)\. Formally, letkkdenote a point in the conversation that the belief estimation component deems tense, i\.e\.,𝒫​\(derailment\|u1,u2,…,uk\)\>T\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\)\>T\. These are moments in which a traditional forecasting model without a distinct decision\-making component would trigger an alert\. We simulateMMdifferent possibilities for the next utterance,uk\+1simiu\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}, fori=1,…,Mi=1,\\ldots,M\. For each such simulated continuation, we again apply the belief estimation component to update the estimated derailment probability after the simulated utterance, and record the corresponding threshold\-based triggering decision:

gk\+1simi=𝕀​\{𝒫​\(derailment\|u1,…,uk,𝐮𝐤\+𝟏sim𝐢\)\>T\}g\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}=\\mathbb\{I\}\\\{\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},\.\.\.,u\_\{k\},\\mathbf\{u\_\{k\+1\}^\{\\mathbf\{\\mathrm\{sim\}\_\{i\}\}\}\}\)\>T\\\}\(3\)
We use this information to devise a decision\-making mechanism that defers the trigger decision in tense moments when simulated continuations indicate plausible paths to conversational recovery:

gk\+=\{1if​𝒫​\(derailment\|u1,…,uk\)\>Tand​\{M−∑igk\+1simi\}≤τ0otherwiseg\_\{k\}^\{\\textrm\{\+\}\}=\\begin\{cases\}1&\\text\{if \}\{\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},\\ldots,u\_\{k\}\)\}\>T\\\\ &\\text\{ and \}\\\{M\-\\sum\_\{i\}g\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}\\\}\\leq\\tau\\\\ 0&\\text\{otherwise\}\\par\\end\{cases\}\(4\)
whereτ\\tauis a parameter that controls how much simulated evidence of recovery—i\.e\., how manyiisatisfygk\+1simi=0g\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}=0—is required to defer the decision to trigger, rather than trigger immediately as the SOTA forecaster would do\. In order for an awry prediction to persist and become a trigger, under this mechanism, the simulated replies which are calm must not exceedτ\\tau\. We simulateM=10M=10next utterances for each tense moment, and require that more than two thirds of the simulations \(τ=7\\tau=7\) indicate recovery in order to defer a threshold\-based triggering decision\. We list further implementation details and model parameters in[Appendix F](https://arxiv.org/html/2605.29243#A6)\.

## 5Results

MethodAcc↑\\uparrowFPR↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowHSOTA70\.934\.369\.176\.172\.33\.9\+ selective deferral70\.926\.772\.168\.470\.23\.8\+ random deferral69\.430\.270\.069\.069\.23\.8\+ simulation \(average\)70\.236\.268\.176\.672\.04\.0\+ simulation \(majority\)70\.036\.967\.776\.771\.84\.0oracle threshold70\.026\.771\.566\.869\.03\.7Table 2:Performance on CGA\-CMV\-Large, averaged across 5 random seeds\. Selective deferral drastically reduces the FPR while maintaining accuracy\. Even when selecting an oracle threshold on the test set, SOTA cannot achieve as good of a precision\-recall tradeoff without a deferral mechanism\.Here, we focus on quantifying the effect of adding an explicit decision\-making component to the state\-of\-the\-art model on the CGA\-CMV\-Large test set, deferring discussion of CGA\-WIKI results to[Appendix C](https://arxiv.org/html/2605.29243#A3)\. We compare against several baseline systems to assess whether the observed gains can be attributed to a less informed deferral strategy, or merely to the use of next\-utterance simulation itself\. We additionally construct an oracle system \(with unrealistic access to the test set\) to assess whether it is possible to achieve similar results by simply finding an ideal threshold, or whether a separate decision component is necessary\.

Simulation baselines\.Inspired byZhanget al\.\([2025](https://arxiv.org/html/2605.29243#bib.bib26)\), we directly use the simulated continuationsuk\+1simiu\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}to estimate the derailment probability𝒫​\(derailment\|u1,…,uk,𝐮𝐤\+𝟏sim𝐢\)\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},\.\.\.,u\_\{k\},\\mathbf\{u\_\{k\+1\}^\{\\mathbf\{\\mathrm\{sim\}\_\{i\}\}\}\}\)\. We use two versions of this baseline: one that takes the average of the derailment probability across all simulated utterances, and one that takes the majority vote ofgk\+1simig\_\{k\+1\}^\{\\mathrm\{sim\}\_\{i\}\}\. Both of these use a pre\-determined trigger threshold, and allow us to verify whether the information in the simulated replies is sufficient in the absence of a separate decision mechanism\.

Random deferral baseline\.We ablate the selective part of the deferral mechanism, which considers the number of calm simulations\(gk\+1s​i​mi=0\)\(g^\{sim\_\{i\}\}\_\{k\+1\}=0\)to decide whether to defer, by producing a baseline that defers a decision to trigger randomly \(maintaining the same likelihood of deferral as in the full system, as observed on the training data\)\.555A similar result is obtained if the likelihood of deferral is estimated on the test data\. This result is included together with other baselines in[Appendix B](https://arxiv.org/html/2605.29243#A2)\.

Oracle threshold selection\.To obtain an upper bound on what the SOTA system can achieve without a separate decision mechanism, we consider an oracle version of the SOTA with the thresholdTTtuned on the test data \(and thus is unrealistic in practice\)\. To compare the precision\-recall tradeoff, we pick a threshold that leads to a similar FPR to our full model\.666Given the non\-uniform distribution of derailment probabilities, it is not possible to perfectly match the FPR\.

The results in[Table 2](https://arxiv.org/html/2605.29243#S5.T2)demonstrate that adding the deferral\-based decision mechanism to the SOTA system yields a substantial reduction in false positive rate, effectively narrowing the gap between human performance without sacrificing overall accuracy\. None of the alternative baselines achieve comparably low false positive rates at similar accuracy levels\.

Moreover, these improvements cannot be achieved by SOTA models without a separate decision mechanism\. Even tuning the decision threshold on the test set yields a poorer precision–recall tradeoff \(oracle threshold in[Table 2](https://arxiv.org/html/2605.29243#S5.T2)\)\. As shown in[Figure 2](https://arxiv.org/html/2605.29243#S5.F2), this holds for all values ofτ\\tau: without the deferral mechanism, SOTA yields lower recall for each corresponding FPR\.

Taken together, these findings suggest that the ability to selectively defer is key in achieving such reductions in false positive rates\. This result underscores the value of explicitly separating decision making from belief estimation in conversational forecasting, and motivates future work on richer and more flexible triggering mechanisms\.

![Refer to caption](https://arxiv.org/html/2605.29243v1/TPRFPR2.png)Figure 2:Comparing precision\-recall tradeoff on CGA\-CMV\-Large\. For any choice ofτ\\tau, selective deferral yields a better recall \(and better accuracy\) than SOTA with the best possible threshold—as found on the test set—that matches that respective FPR\. See[Table 12](https://arxiv.org/html/2605.29243#A2.T12)in the Appendix for a comparison across all metrics\.
## 6Deferral Decisions

PhenomenonExample \(distinguishing phrase in bold\)Direct attribution of faultThe fact thatyou don’t want to talk \[…\] shows you can’t defend \[…\]Why in the world wouldyou think thatbeliefs aren’t a choice?Character judgementPeople like YOUare the reason so many people think they \[…\]Yeahyou sound likea crazy person\.Confrontational questioningI’m notsure why youfeel the need to put words into my mouth\.Whydo you thinkthat I’ve been susceptible to brainwashing, but not you?Epistemic softeningI would arguethis definition doesn’t go nearly far enough \[…\]I getthat the pointof war games is to \[…\]Disagreement without challengeWould you saythe word "pen" has lost all meaning and is not useful?Even if youtake this as 1 person, he can work on getting fit\.Meta\-linguistic clarificationwhen we cometo talk aboutwhat "cheating" actually is \[…\]Op is referencing people whouse the wordcommunity to mean \[…\]Table 3:Paraphrased examples of prototypical post\-trigger \(top\) and post\-deferral \(bottom\) replies exhibiting some of the most distinguishing phrases \(bolded\) identified byMonroeet al\.\([2017](https://arxiv.org/html/2605.29243#bib.bib6)\)’s distinguishing\-words method\. Additional distinguishing phrases and sentences are listed in[Table 7](https://arxiv.org/html/2605.29243#A1.T7)and[Table 9](https://arxiv.org/html/2605.29243#A1.T9)in[Appendix A](https://arxiv.org/html/2605.29243#A1)\.Having established that adding an explicit decision making component yields substantial reductions in false positive rates, we now turn to an analysis of the model’s decisions\. Our goal is to examine when the model chooses to defer triggering and why, and to assess whether these decisions align with the intuitions that motivated our approach—namely, that some tense moments admit plausible paths to recovery and therefore warrant patience\. By inspecting representative cases, we aim to ground the quantitative gains in interpretable conversational dynamics\.

Anticipating decrease in tension\.One of the core intuitions motivating our approach is that humans appear to anticipate when a spike in conversational tension is likely to subside, and selectively avoid triggering in such moments\. To assess whether our system exhibits a similar behavior, we examine cases in which the decision mechanism deferred triggering at moments where the SOTA model would have otherwise triggered an alert\. For these moments, we conduct a post\-hoc analysis on the test data and compute the subsequent change in estimated tension,𝒟k\\mathcal\{D\}\_\{k\}, finding that in 83\.5% of cases the deferral is indeed followed by a decrease in tension \(𝒟k\>0\\mathcal\{D\}\_\{k\}\>0\)\. In contrast, the baseline probability of observing a decrease in tension following an arbitrary SOTA trigger is substantially lower \(55\.3%\), suggesting that our model’s deferrals are aligned with anticipations of recovery\.

Qualitative analysis\.To better understand the types of recovery the model is able to anticipate, we conduct a qualitative content analysis of the replies that immediately follow deferred triggering decisions\. As a first step toward a systematic exploration, we perform a Bayesian distinguishing\-words analysisMonroeet al\.\([2017](https://arxiv.org/html/2605.29243#bib.bib6)\),777We useτ=5\\tau=5in this analysis for broader coverage\. Observations remain qualitatively similar forτ=7\\tau=7\([Appendix A](https://arxiv.org/html/2605.29243#A1)\)\.comparing replies that follow utterances where the decision to trigger was deferred \(henceforth,post\-deferral replies\) with replies that follow utterances where the trigger was maintained \(post\-trigger replies\)\.888These are all cases in which the SOTA would have triggered\. We discard cases in which the reply is the last utterance in a conversation or the to\-be\-forecasted personal attack\.

The phrases distinguishing the post\-trigger replies \(summarized in[Table 3](https://arxiv.org/html/2605.29243#S6.T3)and listed in[Table 7](https://arxiv.org/html/2605.29243#A1.T7)in[Appendix A](https://arxiv.org/html/2605.29243#A1)\) appear to use escalating languageHabernalet al\.\([2018](https://arxiv.org/html/2605.29243#bib.bib7)\); Zhanget al\.\([2018](https://arxiv.org/html/2605.29243#bib.bib2)\)\. This includes direct accusations or attributions of fault, showing heavy usage of second\-person pronouns \(“you don\[’t\] even”, “the fact that you”, “you think that”\), implicit or explicit character judgment \(“people like you”, “you sound like”, “for someone who”\), and confrontational questioning \(“why don’t you”, “sure why you”, “not sure why”\)\.

On the other side, the phrases distinguishing the post\-deferral replies show signs of face\-saving languageBrown and Levinson \([1987](https://arxiv.org/html/2605.29243#bib.bib31)\)and repair attemptsDrew \([1997](https://arxiv.org/html/2605.29243#bib.bib32)\); Tsaiet al\.\([2024](https://arxiv.org/html/2605.29243#bib.bib30)\)\. Some apparent strategies include epistemic softening and argument framing to distance the disagreement from the interlocutor \(“would argue that”,“that the point”, “that they don’t”\), the use of hypotheticals which allows the exploration of disagreements without direct challenge \(“even if you”, “would you say”\), and meta\-linguistic clarifications which reopen the interpretation of prior utterances and suggest that disagreement may arise from ambiguity rather than hostility \(“use the word”, “of the word”, “to talk about”\)\.

Taken together, this contrast between post\-deferral and post\-trigger replies suggests that the decision mechanism is able to effectively anticipate conversational recovery, and use it to selectively defer triggering an alert\.

## 7Conclusion

In this work, we argue for the importance of a distinct decision\-making component in conversational forecasting systems\. We introduce one such mechanism, which defers triggering when it anticipates—via simulation—plausible paths to conversational recovery\. While simple, this approach demonstrates how explicitly reasoning about when not to act can substantially improve forecasting behavior without sacrificing accuracy\. More broadly, our findings highlight the value of treating decision making as a first\-class component of forecasting systems\. This opens the door to richer and more flexible triggering mechanisms beyond fixed thresholding, such as mechanisms incorporating information about the speaker or decision policies inferred from the training data via reinforcement learning\.

From a deployment perspective, these results have direct implications for the design of real\-world moderation and conversational support systems\. In high\-stakes settings, such systems must balance the risk of acting too late against that of acting unnecessarily, as spurious alerts can erode user trust, overwhelm moderators, and disrupt otherwise recoverable conversationsSchlugeret al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib25)\); Changet al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib13)\)\. Our decision\-making mechanism offers a principled way to manage this tradeoff by explicitly reasoning about deferral, with the parameterτ\\tauproviding a direct and interpretable handle for end users to control how much evidence of plausible conversational recovery is required before intervening\. By enabling systems to anticipate and respect moments of potential de\-escalation, such mechanisms enable more selective, context\-sensitive conversational support interventions\.

## 8Limitations

Scope of the decision mechanism\.In this work, we introduce a single, deliberately simple decision\-making mechanism to demonstrate the value of decoupling belief estimation from triggering decisions\. While effective, this mechanism represents only one point in a much broader design space\. In particular, the exploration\-exploitation tradeoff inherent in the unknown\-horizon forecasting setting naturally aligns with reinforcement learning formulations, where a decision policy could be learned directly from training data rather than specified procedurally\. Exploring such learned policies—potentially optimizing downstream objectives such as false\-positive costs or user experience—remains an important direction for future work\.

Our decision mechanism focuses exclusively on deferring triggering decisions in tense moments when recovery appears plausible\. This naturally introduces a precision\-recall tradeoff where higher deferral may increase precision at the cost of recall\. However, deferral is not the only possible intervention\. For example, another mechanism could instead*hasten*triggering when simulated continuations suggest rapid escalation, even if the current estimated probability has not yet crossed the threshold\. More broadly, future systems could dynamically adjust when to act in either direction \(defer or hasten\), enabling richer and more flexible control over forecasting behavior\.

Simulation fidelity and efficiency\.Our approach relies on large language model simulations to approximate likely next utterances\. While such simulations are increasingly strong, they are necessarily imperfect\. Moreover, our current implementation simulates only a single conversational step ahead, which limits its ability to capture longer\-range recovery or escalation trajectories\. This is due to the exponential increase in the number of simulations necessary, as each simulation would initiate its own branching ofkksimulations\. Simulating further steps ahead, while costly due to the significant computation necessary for multiple LLM calls, may improve performance\. Future work could explore simulating full conversational rollouts, as well as methods for aggregating uncertainty across longer\-term simulated futures\.

Human baseline scale and coverage\.We use a human experiment to gain insight into how people make online forecasting decisions, prioritizing a small but carefully controlled and high\-quality dataset\. As a result, the human baseline is limited in scale\. Future work could use our experimental methodology \(which we open\-source\) to extend this baseline to a larger and more diverse set of conversations, include additional participant populations, or examine learning effects across more rounds to understand how—and to what extent—humans improve with experience\.

Task, domain, and language specificity\.All experiments and analyses in this work are conducted within the context of the Conversations Gone Awry \(CGA\) task, which focuses on online, English\-language discussions in a small number of communities\. As a result, both the belief estimation models and the proposed decision mechanism may reflect domain\-, culture\-, and language\-specific norms tied to escalation\. While the methodology itself is not inherently tied to CGA, extending it to other domains would require careful adaptation and validation\. In principle, similar decision mechanisms could be applied to other forms of conversational forecasting, such as anticipating disengagement or breakdown in medical, educational, or support\-oriented interactions, where the nature of both escalation and recovery may differ substantially\. We advocate for the development of such diverse conversational forecasting benchmarks\.

## Ethical Considerations

Risk of inappropriate intervention\.Applications of conversational derailment forecasting might involve intervening in ongoing human interactions\. While our decision mechanism reduces false positives, any conversational support system still carries the risk of intervening when intervention is unnecessary or unwanted\. Such interventions may disrupt legitimate disagreement, suppress minority viewpoints, or be perceived as intrusive\. Careful calibration of triggering behavior—including explicit control over parameters such asτ\\tau—is therefore essential when deploying such systems in real\-world conversational support settings\. Careful user studies are necessary before real\-world deployment and should consider the effects of the decision making mechanism when deployed in conversational support systems\.

Reliance on simulated futures\.Our approach uses large language model simulations to anticipate possible conversational trajectories\. These simulations reflect the biases, limitations, and normative assumptions of the underlying language models, and may not faithfully represent how a conversation would actually evolve\. As a result, the decision mechanism may defer or trigger based on simulated recoveries or escalations that would not occur in practice\. This limitation underscores the importance of treating simulations as advisory signals rather than ground truth, and of validating decision mechanisms in deployment\-specific contexts\.

Dataset and annotation biases\.The CGA datasets reflect particular communities \(e\.g\., Change My View and Wikipedia talk pages\) and moderation norms\. Consequently, both the belief estimation models and the decision mechanisms trained or evaluated on these datasets may encode community\-specific notions of what constitutes “tension” or “derailment\.” Applying such systems to other conversational domains without careful adaptation risks mischaracterizing culturally or contextually appropriate forms of disagreement\.

Human subject considerations\.Our human baseline experiment involves participants making judgments about potentially contentious or hostile conversational content\. All participants were informed their annotations would be used to create a public dataset\. The task was designed to minimize exposure to explicit personal attacks \(the attacks were not displayed\) and the experiment was vetted by an institutional IRB\. Repeated exposure to tense interactions may still impose cognitive or emotional burden\. Future large\-scale studies should consider additional safeguards, including clear opt\-out mechanisms and post\-task debriefing, especially if extended to more adversarial domains\.

Additional considerations\.The experiments in this work use datasets containing real\-world conversations from the r/ChangeMyView subreddit and Wikipedia Talk Pages\. In examples of paraphrased comments from users, we anonymize speakers\. Throughout this work, the data used is publicly accessible and we use only open\-source models in our experiments\.

## 9Acknowledgments

Foremost, we offer ourdeferredthanks to Jonathan P\. Chang for very early discussions about “ways out” \(which he eventually found\!\) and to Nicholas Chernogor, Tushaar Gangavarapu, and Son Tran for developing the infrastructure on which this work builds\. We are grateful for engaging discussions with the other members of the Team Zissou—including Dave Jung, Lillian Lee, Ethan Xia, and Sean Zhang\. We thank the reviewers for their extremely thoughtful feedback\. We gratefully acknowledge the use of research computing resources from the Empire AI Consortium, Inc\., supported by Empire State Development of the State of New York, the Simons Foundation, and the Secunda Family Foundation\. This work was in part enabled by a Gemma Academic Program GCP Credit Award\. This project was developed as part of Cornell’s Bowers Undergraduate Research Experience \(BURE\) program in which Laerdon Kim participated\. Vivian Nguyen was supported by a Cornell University Computer Science Fellowship and a Cornell Graduate School Dean’s Scholarship\. Cristian Danescu\-Niculescu\-Mizil was funded in part by the U\.S\. National Science Foundation under Grant No\. IIS\-1750615 \(CAREER\), by Cornell’s Center for Social Sciences, by a LinkedIn Research Award, and by a Wikimedia Research Fund Award\. Any opinions, findings, and conclusions in this work are those of the author\(s\) and do not necessarily reflect the views of Cornell University or the National Science Foundation\.

## References

- Conversation Derailment Forecasting with Graph Convolutional Networks\.InProceedings of WOAH, co\-located with ACL,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p4.2),[§2](https://arxiv.org/html/2605.29243#S2.p5.3),[§2](https://arxiv.org/html/2605.29243#S2.p7.2)\.
- J\. Bao, J\. Wu, Y\. Zhang, E\. Chandrasekharan, and D\. Jurgens \(2021\)Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations\.InProceedings of WWW,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1)\.
- P\. Brown and S\. C\. Levinson \(1987\)Politeness: Some Universals in Language Usage\.Cambridge University Press\.Cited by:[§6](https://arxiv.org/html/2605.29243#S6.p5.1)\.
- J\. P\. Chang and C\. Danescu\-Niculescu\-Mizil \(2019\)Trouble on the Horizon: Forecasting the Derailment of Online Conversations as They Develop\.InProceedings of EMNLP\-IJCNLP,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1),[§1](https://arxiv.org/html/2605.29243#S1.p4.2),[§2](https://arxiv.org/html/2605.29243#S2.p11.5),[§2](https://arxiv.org/html/2605.29243#S2.p12.1),[§2](https://arxiv.org/html/2605.29243#S2.p2.2),[§2](https://arxiv.org/html/2605.29243#S2.p3.1),[§2](https://arxiv.org/html/2605.29243#S2.p5.3)\.
- J\. P\. Chang, C\. Schluger, and C\. Danescu\-Niculescu\-Mizil \(2022\)Thread With Caution: Proactively Helping Users Assess and Deescalate Tension in Their Online Discussions\.Proceedings of CSCW6\(545\),pp\. 1–37\.Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1),[§1](https://arxiv.org/html/2605.29243#S1.p3.1),[§1](https://arxiv.org/html/2605.29243#S1.p5.1),[§2](https://arxiv.org/html/2605.29243#S2.p1.1),[§7](https://arxiv.org/html/2605.29243#S7.p2.1),[footnote 2](https://arxiv.org/html/2605.29243#footnote2)\.
- J\. P\. Chang \(2024\)Towards Computational Methods for Proactively Supporting Healthier Online Discussions\.Ph\.D\. Thesis,Cornell University\.External Links:[Link](https://www.cs.cornell.edu/%CB%9Ccristian/papers/chang_thesis.pdf)Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p7.2)\.
- P\. Drew \(1997\)‘Open’ Class Repair Initiators in Response to Sequential Sources of Troubles in Conversation\.Journal of Pragmatics28\(1\)\.Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p7.2),[§6](https://arxiv.org/html/2605.29243#S6.p5.1)\.
- A\. Grattafiori, A\. Dubey,et al\.\(2024\)The Llama 3 Herd of Models\.Note:arXiv:2407\.21783External Links:2407\.21783Cited by:[Appendix F](https://arxiv.org/html/2605.29243#A6.p1.3)\.
- I\. Habernal, H\. Wachsmuth, I\. Gurevych, and B\. Stein \(2018\)Before Name\-Calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation\.InProceedings of NAACL,Cited by:[§6](https://arxiv.org/html/2605.29243#S6.p4.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: Low\-Rank Adaptation of Large Language Models\.InProceedings of ICLR,Cited by:[Appendix F](https://arxiv.org/html/2605.29243#A6.p1.3)\.
- D\. Jurgens, L\. Hemphill, and E\. Chandrasekharan \(2019\)A Just and Comprehensive Strategy for Using NLP to Address Online Abuse\.InProceedings of ACL,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1)\.
- Y\. Kementchedjhieva and A\. Søgaard \(2021\)Dynamic Forecasting of Conversation Derailment\.InProceedings of EMNLP,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p4.2),[§2](https://arxiv.org/html/2605.29243#S2.p5.3)\.
- K\. Korre, D\. Tsirmpas, N\. Gkoumas, E\. Cabalé, D\. Myrtzani, T\. Evgeniou, I\. Androutsopoulos, and J\. Pavlopoulos \(2025\)Evaluation and Facilitation of Online Discussions in the LLM Era: A Survey\.InProceedings of EMNLP,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1)\.
- C\. Li \(2022\)Using Predictive Technology to Foster Constructive Conversations\.Note:Nextdoor Engineering Blog, May 3 2022\. Retrieved April 19, 2026External Links:[Link](https://engblog.nextdoor.com/using-predictive-technology-to-foster-constructive-conversations-4af437942bd4)Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled Weight Decay Regularization\.InProceedings of ICLR,Cited by:[Appendix F](https://arxiv.org/html/2605.29243#A6.p1.3)\.
- B\. L\. Monroe, M\. P\. Colaresi, and K\. M\. Quinn \(2017\)Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict\.Political Analysis16\(4\),pp\. 372–403\.Cited by:[Table 10](https://arxiv.org/html/2605.29243#A1.T10),[Table 9](https://arxiv.org/html/2605.29243#A1.T9),[Table 3](https://arxiv.org/html/2605.29243#S6.T3),[§6](https://arxiv.org/html/2605.29243#S6.p3.1)\.
- V\. Nguyen, L\. Lee, and C\. Danescu\-Niculescu\-Mizil \(2025\)Hanging in the Balance: Pivotal Moments in Crisis Counseling Conversations\.InProceedings of ACL,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1),[§2](https://arxiv.org/html/2605.29243#S2.p13.1)\.
- K\. Olpadkar, V\. S\. Bajaj, and L\. Barrett \(2025\)Can LLMs be Efficient Predictors of Conversational Derailment?\.InFindings of EMNLP,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p4.2),[§2](https://arxiv.org/html/2605.29243#S2.p5.3)\.
- C\. Schluger, J\. P\. Chang, C\. Danescu\-Niculescu\-Mizil, and K\. Levy \(2022\)Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support\.Proceedings of CSCW\.Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p2.1),[§7](https://arxiv.org/html/2605.29243#S7.p2.1)\.
- G\. Team \(2024\)Gemma 2: Improving open language models at a practical size\.Note:arXiv:2408\.00118 \[cs\.CL\]External Links:2408\.00118Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p6.2)\.
- S\. Q\. Tran, T\. Gangavarapu, N\. Chernogor, J\. P\. Chang, and C\. Danescu\-Niculescu\-Mizil \(2025\)Conversations Gone Awry, But Then? Evaluating Conversational Forecasting Models\.Note:arXiv:2507\.19470External Links:2507\.19470Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p1.1),[§2](https://arxiv.org/html/2605.29243#S2.p2.2),[§2](https://arxiv.org/html/2605.29243#S2.p5.3),[§2](https://arxiv.org/html/2605.29243#S2.p6.2)\.
- C\. W\. Tsai, Y\. Huang, T\. Liao, D\. F\. S\. Estrada, R\. Latifah, and Y\. Chen \(2024\)Leveraging Conflicts in Social Media Posts: Unintended Offense Dataset\.InProceedings of EMNLP,Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p7.2),[§6](https://arxiv.org/html/2605.29243#S6.p5.1)\.
- D\. Vannella, D\. Jurgens, D\. Scarfini, D\. Toscani, and R\. Navigli \(2014\)Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose\.InProceedings of ACL,Cited by:[§3](https://arxiv.org/html/2605.29243#S3.p3.1)\.
- J\. Yuan and M\. P\. Singh \(2023\)Conversation Modeling to Predict Derailment\.InProceedings of ICWSM,Cited by:[§1](https://arxiv.org/html/2605.29243#S1.p4.2),[§2](https://arxiv.org/html/2605.29243#S2.p5.3)\.
- J\. Zhang, J\. Chang, C\. Danescu\-Niculescu\-Mizil, L\. Dixon, Y\. Hua, D\. Taraborelli, and N\. Thain \(2018\)Conversations Gone Awry: Detecting Early Signs of Conversational Failure\.InProceedings of ACL,Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p1.1),[§2](https://arxiv.org/html/2605.29243#S2.p2.2),[§6](https://arxiv.org/html/2605.29243#S6.p4.1)\.
- Y\. Zhang, K\. McKeown, and S\. Muresan \(2025\)Forecasting Conversation Derailments Through Generation\.InProceedings of INLG,Cited by:[§2](https://arxiv.org/html/2605.29243#S2.p12.1),[§5](https://arxiv.org/html/2605.29243#S5.p2.3)\.

## Appendix AAdditional Examples

tAuthorUtterance \(abridged\)\# of calm simulations𝒫\\mathcal\{P\}0Speaker 1Has anyone ever actually tried to police DiCaprio’s dating life? \[…\]10\.00\.501Speaker 2Declaring something as “creepy” \[…\] is essentially saying it’s unacceptable behaviour\. \[…\] “shifting frames\.”0\.00\.592Speaker 3So what is your opinion? That people shouldn’t be allowed to call DiCaprio creepy because that’s policing him?9\.00\.653Speaker 2The post is about those who argue against large age\-gap relationships \[…\] So far, you’re proving my point\.7\.00\.504Speaker 3Your opinion is unclear to me because it doesn’t sound like you want your views changed regarding the arguments people make against age\-gap relationships, but rather you want your views on how you feel about the way people are expressing said arguments/the effect that has changed\. \[…\] I’m asking about conclusions you draw from your position \[…\] I think that’s a fair question to ask and I’m not engaging in bad faith here\.9\.00\.59Table 4:Example thread from CGA\-CMV\-Large where decision\-deferral is effective; conversation does not have a removed comment, and thus remains calm\.𝒫:=𝒫​\(derailment\|u1,u2,…,uk\)\\mathcal\{P\}:=\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\);𝒫\\mathcal\{P\}values aboveTTare highlighted inbold, which are moments where SOTA will trigger\. An utterance which has a trigger decision deferral is highlighted inblue\.tAuthorUtterance \(abridged\)\# of calm simulations𝒫\\mathcal\{P\}0Speaker 1You’re applying human logic and motivations to an omniscient deity\[…\] humanity is fundamentally incapable of understanding the Christian God\[…\] we’re like 2 year olds\.10\.00\.351Speaker 2That sounds like some shit Rian Johnson would say \[…\] God is a primitive attempt at a Thanos\-style big baddie \[…\] inconsistent due to many writers \[…\]1\.00\.592Speaker 3I don’t see how this contributes \[…\] You can’t just discount someone’s comment \[…\] You need to actually address what they said with counterpoints\.9\.00\.653Speaker 2\> But humanity is fundamentally incapable \[…\] However we’re perfectly capable of understanding a fictional character \[…\] inconsistencies are expected from many writers \[…\] There, that better?9\.00\.474Speaker 3No because all you have done is paraphrased the title of the CMV\.8\.00\.53\[Removed by moderator due to Rule 2 violation\]
Table 5:Example thread from CGA\-CMV\-Large where decision\-deferral is not effective; conversation has a removed comment, and thus goes awry\.𝒫\\mathcal\{P\}Simulated next utterance \(abridged\)0\.53The point of my post is literally about why I think your view is wrong \[…\] which again means it did contribute to the discussion\.0\.50This response is so weirdly confrontational \[…\] My point wasn’t to attack anyone personally, but rather show why the premise itself has flaws \[…\]0\.50How does mine not?0\.47He didn’t create humans to torture them though \[…\] That idea is based on a very specific interpretation \[…\] The rest of the New Testament doesn’t back up your view here\.0\.50Because OP asked about cognitive dissonance within Christianity \[…\] those conflicts exist because the Bible was written by multiple people over many years\.0\.65This guy seems like one of those people who thinks there’s something wrong with him \[…\] This person is simply unreasonable\.0\.50Of course it’s ridiculous when seen outside its context \[…\] it comes from people trying to give form to things bigger than themselves\.0\.44So there’s no challenge? The fun of creation comes from taking nothing and building something out of it \[…\]0\.50The concept of Satan comes from Zoroastrianism \[…\] Hell does not exist until the New Testament \[…\]0\.38Since God cannot be known, we should not concern ourselves with him \[…\] focus instead on human suffering and ethics\.
Table 6:Simulated next utterances and their forecast probabilities att=2t=2, predicting comments fort=3t=3\.Examples of decision\-deferral on CGA\-CMV\-Large\.To concretize the observations made via our qualitative analysis, we show specific examples of successful and unsuccessful trigger decision deferrals\. In[Table 4](https://arxiv.org/html/2605.29243#A1.T4), att=2t=2, we notice the tension in the conversation beginning to increase, where Speaker 3 seems to question the opinions of the previous speakers\. However, because 9/10 of our simulations for the next utterance indicate a prediction that the conversation will not go awry, we choose not to make our awry prediction yet\. Att=4t=4, we see that the tension in the conversation has decreased, clearing up that their question att=2t=2was perhaps a genuine question meant to clarify the disagreement, rather than to antagonize the other discussion participants\. Without the deferral, the forecaster would have triggered intervention, stymieing this path to recovery\.

In our manual analysis of instances where the deferral helped the forecaster avoid a false positive error, we find that many instances fall into this pattern, of an inaccurate forecast being made due to a seemingly awry utterance which ends up being a misinterpretation, or eventually recovering\.

[Table 6](https://arxiv.org/html/2605.29243#A1.T6)illustrates an example where decision\-deferral fails to recognize a derailment, whereas the SOTA does\. The utterance att=1t=1introduces some tension, with Speaker 2 criticizing the comment of Speaker 1 with an expletive\. Speaker 3 joins the conversation to claim that Speaker 2 is not productively contributing to the discussion, which is the first utterance where an awry forecast is made, and deferred, as the utterance has 9/10 calm simulations\. After this comment, Speaker 2 slightly adjusts their tone, ending with a sarcastic comment and an open ended question\. We notice thatD2\>0D\_\{2\}\>0in this example, but the conversation appears to re\-escalate after Speaker 3 shuts down Speaker 2 again, thus increasing the inferred tension\. Here, the decision\-deferral system arguably did correctly predict that the next utterance would indeed be a de\-escalation with an inferred tension belowTT\. Afterwards, the conversation re\-escalated, but not to the point where the forecaster could make an intervention\. This example demonstrates the complexity of handling decision\-making in time\-series; it is ambiguous as to whether the system “should have” intervened att=2t=2ort=4t=4\.

Although there does not exist an awry forecast att=3t=3, we might understand the quote “There, that better?” to be more combative, whereas the forecaster might focus on the bulk of the comment being revised for tone\. It is perhaps due to this underestimation of Speaker 2’s sarcasm, and Speaker 3’s subsequent contradiction of Speaker 2, that the conversation unexpectedly goes awry\. In addition, none of the simulated utterances predict the sarcasm which follows the utterance att=2t=2\.

N\-gram phrases and in\-context examples\.We provide additional examples of n\-gram phrases \([Table 7](https://arxiv.org/html/2605.29243#A1.T7),[Table 8](https://arxiv.org/html/2605.29243#A1.T8)\) and examples with conversational context \([Table 9](https://arxiv.org/html/2605.29243#A1.T9),[Table 10](https://arxiv.org/html/2605.29243#A1.T10)\)\. We run distinguishing words analysis withn=3n=3\.

Not deferred n\-gramsDeferred n\-gramsn\-gramzzn\-gramzzfact that you−3\.58\-3\.58they are the\+3\.39\+3\.39on the internet−2\.97\-2\.97would argue that\+3\.21\+3\.21you know it−2\.75\-2\.75because they are\+3\.02\+3\.02people who are−2\.65\-2\.65have nothing to\+2\.90\+2\.90you don even−2\.65\-2\.65to support the\+2\.74\+2\.74you think that−2\.62\-2\.62is no reason\+2\.74\+2\.74all you want−2\.58\-2\.58if you re\+2\.70\+2\.70people like you−2\.55\-2\.55right to life\+2\.67\+2\.67that you re−2\.45\-2\.45to talk about\+2\.54\+2\.54why don you−2\.44\-2\.44know for fact\+2\.52\+2\.52sure why you−2\.43\-2\.43of the word\+2\.50\+2\.50and not the−2\.43\-2\.43of course there\+2\.49\+2\.49in the west−2\.43\-2\.43the only way\+2\.47\+2\.47you have any−2\.37\-2\.37use the word\+2\.47\+2\.47you want but−2\.36\-2\.36because it is\+2\.46\+2\.46to back up−2\.33\-2\.33that makes you\+2\.42\+2\.42the age of−2\.33\-2\.33to make it\+2\.36\+2\.36that you ve−2\.33\-2\.33the ability to\+2\.34\+2\.34of the best−2\.29\-2\.29it doesn matter\+2\.33\+2\.33what you are−2\.23\-2\.23would you say\+2\.33\+2\.33that would be−2\.22\-2\.22then you can\+2\.31\+2\.31to go back−2\.22\-2\.22it was in\+2\.30\+2\.30not sure why−2\.22\-2\.22that the point\+2\.26\+2\.26just look at−2\.20\-2\.20they should have\+2\.26\+2\.26you need to−2\.20\-2\.20you re not\+2\.26\+2\.26for someone who−2\.20\-2\.20in front of\+2\.24\+2\.24if they had−2\.20\-2\.20even if you\+2\.24\+2\.24to me that−2\.20\-2\.20that they don\+2\.24\+2\.24you sound like−2\.18\-2\.18only way to\+2\.21\+2\.21you understand that−2\.18\-2\.18talk about it\+2\.21\+2\.21Table 7:Top 30 tri\-grams associated with replies following not\-deferred \(left\) and deferred \(right\) decisions, ranked by the z\-score of the Bayesian distinguishing\-word analysis forτ=5\\tau=5\.Not deferred n\-gramsDeferred n\-gramsn\-gramzzn\-gramzzfact that you−4\.46\-4\.46to talk about\+2\.92\+2\.92people who are−3\.50\-3\.50because it is\+2\.80\+2\.80you know it−3\.03\-3\.03in front of\+2\.77\+2\.77because that is−2\.79\-2\.79they are the\+2\.70\+2\.70do you see−2\.78\-2\.78if you re\+2\.68\+2\.68if you go−2\.78\-2\.78would argue that\+2\.68\+2\.68of the best−2\.66\-2\.66have nothing to\+2\.60\+2\.60sure why you−2\.58\-2\.58but they are\+2\.52\+2\.52people like you−2\.54\-2\.54you can have\+2\.50\+2\.50the age of−2\.54\-2\.54right to life\+2\.49\+2\.49the guy who−2\.54\-2\.54because they are\+2\.42\+2\.42of that is−2\.53\-2\.53it doesn matter\+2\.41\+2\.41not just the−2\.53\-2\.53the only way\+2\.31\+2\.31think you know−2\.53\-2\.53that women are\+2\.28\+2\.28that you re−2\.48\-2\.48but that doesn\+2\.24\+2\.24you talking about−2\.47\-2\.47if they re\+2\.20\+2\.20if you would−2\.45\-2\.45more or less\+2\.17\+2\.17needs to be−2\.44\-2\.44there is more\+2\.17\+2\.17because you don−2\.41\-2\.41it is in\+2\.17\+2\.17what you are−2\.39\-2\.39should not be\+2\.16\+2\.16not sure why−2\.38\-2\.38talk about it\+2\.16\+2\.16what wrong with−2\.38\-2\.38the us but\+2\.14\+2\.14say that you−2\.38\-2\.38use the word\+2\.14\+2\.14example of how−2\.38\-2\.38having sex with\+2\.14\+2\.14right to be−2\.36\-2\.36is no reason\+2\.14\+2\.14why don you−2\.35\-2\.35to support the\+2\.14\+2\.14to have any−2\.33\-2\.33the will of\+2\.14\+2\.14to stop the−2\.33\-2\.33but if you\+2\.14\+2\.14tell me how−2\.31\-2\.31what you re\+2\.12\+2\.12you don even−2\.31\-2\.31what they are\+2\.10\+2\.10Table 8:Top 30 tri\-grams associated with replies following not\-deferred \(left\) and deferred \(right\) decisions, ranked by the z\-score of the Bayesian distinguishing\-word analysis forτ=7\\tau=7\.PhenomenonExample \(distinguishing phrase in bold\)Confrontational questioning\>\> Yeah I don’t think this is effective\. \[…\] Also taxing assets the same as income is impossible because assets aren’t liquid\. That means you can be getting taxed either too much, or too little for the cash equivalent of the assets you receive\. That’s why they’re on an entirely separate tax bracket than regular income\. I think you need to sit down and study taxes instead of Econ tbh\>\> Right which is why I specified the value on the day at which the the stock was issued\. It’s not a perfect system, but it’s better than having completely uncontrolled CEO pays is my point\. Yes I understand that stocks can vary in value, which is why I’m notsure why you’re talking about taxing them\. I never said anything about taxes?Character judgement\>\> I’m a teacher and this is leading reason I will never ever return to America\. I hate school shootings, I hate active shooter drills, I hate militarized schools designed as high security facilities\. And I hate that pro\-gunpeople like yourefuse every single change proposed, but then propose no other changes\.Epistemic softening\>\> No one denies that having nukes is good for the north korean regime\. what people deny is the idea that the most monstrous regime on the planet has any legitimacy or that its desires deserve to be respected\.\>\> Define monstrous,I would arguethat our hawkishness has led to far more bloodshed than anything North Korea has done\. Very patriocentric of you\. I would agree they are the most monstrous regime to their citizens, but not far ahead of Saudi Arabia\.Table 9:Additional paraphrased examples with conversational context of prototypical post\-trigger \(top\) and post\-deferral \(bottom\) replies exhibiting some of the most distinguishing phrases \(bolded\) identified byMonroeet al\.\([2017](https://arxiv.org/html/2605.29243#bib.bib6)\)’s distinguishing\-words method\(τ=5\)\(\\tau=5\)\.PhenomenonExample \(distinguishing phrase in bold\)Direct attribution of faultIt’s apparent you believeyou know itall and no one is ever going to change your mind\.Character judgementBasically,you sound likea pretentious art snobI don’tthink YOU knowwhat "full reserve banking" is…Confrontational questioningTell me howJan 6 does not fit the parameters of the textbook definition of insurrection?Epistemic softeningIwould argue thatin most cases, ie the woman having consensual sex, that the creation of life is what entitles the fetus/child\.Disagreement without challengeWould you saythat someone who voted for segregationists during the civil rights era was not necessarily a racist?Meta\-linguistic clarification…when we cometo talk aboutwhat "cheating" actually is……so paedophiles do not tend totalk about itopenly…Table 10:Additional paraphrased examples of prototypical post\-trigger \(top\) and post\-deferral \(bottom\) replies exhibiting some of the most distinguishing phrases \(bolded\) identified byMonroeet al\.\([2017](https://arxiv.org/html/2605.29243#bib.bib6)\)’s distinguishing\-words method\(τ=7\)\(\\tau=7\)\.
## Appendix BAdditional Baselines and Ablations

MethodAcc↑\\uparrowFPR↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowHSOTA70\.9±\\pm0\.3534\.3±\\pm3\.9669\.1±\\pm1\.2176\.1±\\pm4\.3372\.3±\\pm1\.303\.9±\\pm0\.13\+ selective deferral70\.9±\\pm0\.7726\.7±\\pm3\.1072\.1±\\pm0\.9268\.4±\\pm4\.6370\.1±\\pm1\.963\.8±\\pm0\.09\+ random deferral \(train tuned\)69\.4±\\pm0\.7930\.2±\\pm3\.8170\.0±\\pm1\.3069\.0±\\pm0\.4869\.2±\\pm1\.913\.8±\\pm0\.12\+ random deferral \(validation tuned\)69\.0±\\pm0\.9829\.3±\\pm3\.8069\.8±\\pm1\.4467\.1±\\pm4\.8368\.3±\\pm2\.103\.8±\\pm0\.12\+ random deferral \(test tuned\)70\.2±\\pm0\.7832\.0±\\pm3\.8869\.4±\\pm1\.3372\.3±\\pm4\.5770\.8±\\pm1\.693\.9±\\pm0\.12\+ simulation \(average\)70\.2±\\pm0\.2736\.2±\\pm5\.1768\.1±\\pm1\.5276\.6±\\pm5\.2872\.0±\\pm1\.454\.0±\\pm0\.11\+ simulation \(majority\)70\.0±\\pm0\.2936\.9±\\pm5\.1267\.7±\\pm1\.5776\.7±\\pm4\.8171\.8±\\pm1\.214\.0±\\pm0\.11\+ variance deferral70\.933\.269\.675\.072\.23\.8oracle threshold \(τ=7\\tau=7\)70\.0±\\pm0\.8826\.7±\\pm3\.3171\.5±\\pm1\.0066\.8±\\pm4\.9469\.0±\\pm2\.183\.7±\\pm0\.10
Table 11:Average performance on CGA\-CMV\-Large across 5 random seeds\. For variance deferral, no averaging is possible, as the method requires five separate seeds\. Standard deviation is reported to the right of each table entry\.\.

τ\\tauMethodAcc↑\\uparrowFPR↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowH1selective deferral67\.216\.875\.551\.360\.93\.5matched FPR oracle66\.516\.675\.149\.859\.73\.42selective deferral68\.219\.074\.755\.463\.53\.6matched FPR oracle67\.618\.874\.353\.962\.43\.53selective deferral69\.020\.474\.258\.565\.33\.6matched FPR oracle68\.020\.273\.756\.263\.63\.54selective deferral69\.721\.773\.961\.166\.83\.6matched FPR oracle68\.822\.273\.059\.965\.73\.65selective deferral70\.023\.373\.263\.267\.83\.7matched FPR oracle68\.922\.972\.860\.766\.03\.66selective deferral70\.624\.872\.865\.969\.13\.7matched FPR oracle69\.625\.372\.064\.467\.83\.77selective deferral70\.926\.772\.168\.470\.13\.8matched FPR oracle70\.026\.771\.566\.869\.03\.78selective deferral71\.128\.971\.271\.271\.13\.8matched FPR oracle70\.529\.770\.570\.670\.53\.89selective deferral71\.231\.570\.273\.972\.03\.9matched FPR oracle70\.631\.370\.072\.571\.13\.8Table 12:Per\-τ\\taucomparison of selective deferral against a matched FPR oracle baseline on CGA\-CMV\-Large \(average over 5 seeds\)\. The oracle baseline sets the forecaster threshold to match the deferral FPR at eachτ\\tau\.We supply additional baseline trials and report standard deviations across seeds in[Table 11](https://arxiv.org/html/2605.29243#A2.T11)\.

Variance deferral baseline\.We might also use variance between models fine\-tuned using different random seeds to examine moments at which there is higher disagreement of tension estimation\. Using our five random seeds, we can again ablate the selective simulation component and instead defer based on the variance calculated between the five forecast probabilities using a threshold tuned for accuracy on the validation set\.

Random deferral tuning\.Instead of finding our probability of deferral from the train set, we can also source this probability from the test set\.

Deferral with variable tau and FPR\-matched oracle\.In[Table 12](https://arxiv.org/html/2605.29243#A2.T12), we provide more detailed tau\-matching using the same procedure with which oracle threshold in[Table 2](https://arxiv.org/html/2605.29243#S5.T2)was calculated\. In order to identify the FPR\-matched oracle threshold, we first narrow our search space to the average of all the thresholds tuned for accuracy\. We then search from 0\.15 below this averaged threshold to 0\.15 above it with a step granularity of 400\. Each threshold found by this process is applied to each seed, and from this set of thresholds we pick the threshold which best matches the FPR for each trial of tau using the deferral system\. After, we have the TPR and FPR rate for each of these thresholds FPR\-matched to a tau\. The performance for each FPR\-matched threshold for a specific tau is merged across seeds to get the blue x’s shown in[Figure 2](https://arxiv.org/html/2605.29243#S5.F2)and[Figure 3](https://arxiv.org/html/2605.29243#A3.F3)\.

## Appendix CResults on CGA\-WIKI

![Refer to caption](https://arxiv.org/html/2605.29243v1/TPRFPRwiki.png)Figure 3:Decision\-deferral with variableτ\\tauand matched baseline ROC performance on CGA\-WIKI \(averaged over five seeds\)\.MethodAcc↑\\uparrowFPR↓\\downarrowP↑\\uparrowR↑\\uparrowF1↑\\uparrowHSOTA69\.436\.767\.775\.471\.13\.6\+ selective deferral69\.235\.368\.171\.669\.63\.6\+ random deferral68\.033\.667\.972\.770\.03\.5\+ simulation \(average\)66\.946\.463\.880\.270\.63\.7\+ simulation \(majority\)67\.440\.265\.875\.069\.53\.5with oracle threshold69\.533\.569\.072\.670\.33\.5Table 13:Average performance on CGA\-WIKI across 5 random seeds\.Although our main focus is on the CGA\-CMV dataset, we benchmark our baselines and decision\-deferral method on CGA\-WIKI for completeness\. As discussed in[Section 2](https://arxiv.org/html/2605.29243#S2), since CGA\-WIKI is pruned to have no personal attacks or toxicity in the body of the conversation, the recovery trajectories which decision\-deferral uses to reclaim false positive errors is, by construction, sparse in CGA\-WIKI\. As such, there are substantially more opportunities to initiate a deferral in CGA\-CMV\-Large as opposed to CGA\-WIKI\.

Even in these conditions, SOTA with the deferral mechanism still outperforms the SOTA on FPR, although to a lesser extent than in CGA\-CMV \([Table 13](https://arxiv.org/html/2605.29243#A3.T13)\)\. However, this improvement comes at a slight decrease in accuracy\. Unlike in the case of CGA\-CMV, the SOTA with the deferral system is outperformed by the oracle threshold tuning, meaning that it would in principle be possible to find a threshold T which achieves similar FPR with a better precision\-recall tradeoff \([Figure 3](https://arxiv.org/html/2605.29243#A3.F3)\)\.

Examples of decision\-deferral on CGA\-WIKI\.To concretize the FPR gains detailed in the previous section, we examine an example of a successful trigger\-decision deferral in[Table 15](https://arxiv.org/html/2605.29243#A3.T15)as done for CGA\-CMV\-Large in[Appendix A](https://arxiv.org/html/2605.29243#A1)\. The conversation centers the addition of an explanation of Kepler’s third law, created by Speaker 1\. While Speaker 2 recognizes Speaker 1’s effort, they claim that the material is not “useful” or “correct\.” Speaker 1 proposes a counterargument att=2t=2, and Speaker 2 rebuts att=3t=3\. Through this discussion, the conversational tension increases\. Att=4t=4, Speaker 2 states that they deleted the section\. Here, the tension𝒫\>T\\mathcal\{P\}\>T\. SOTA therefore would trigger here, but because there are 10/10 calm simulations, the decision to trigger is deferred by the selective deferral system\.

The following response from Speaker 1 att=5t=5is civil, beginning with a concession—they agree with Speaker 2’s stance that their added section was not useful—but they “disagree with the rest of \[their\] judgement,” defending the section\. Finally,t=6t=6concludes the conversation with a similar utterance structure as the utterance att=5t=5, beginning with an assurance that the “example… is helpful,” but calls the writing “misleading\.”

Although the SOTA trigger point att=4t=4appears as an act of escalation, as the conversational context contains continuously increasing tension, the simulations predict that the next utterance will not be an escalation due to the straightforward nature of Speaker 2’s comment\.

tAuthorUtterance \(abridged\)\# of calm simulations𝒫\\mathcal\{P\}0Speaker 1I’ve added a section on the relativistic case of Kepler’s third law \[…\]10\.00\.021Speaker 2Your hard work is admirable, but I don’t think this material is useful, or even correct as currently written \[…\]8\.00\.292Speaker 1In what sense are they meaningless? The Schwarzschildtt\-coordinate is observed by an observer at infinity \[…\]8\.00\.293Speaker 2You’ve pointed out one analogy, but in other ways it fails \[…\] coordinates are arbitrary \[…\] you could pick coordinates whereP7∝r11P^\{7\}\\propto r^\{11\}\.9\.00\.444Speaker 2I’ve gone ahead and deleted the section\. I hope you won’t take this as confrontational \[…\]10\.00\.535Speaker 1Perhaps I can accept it’s not generally useful, but I disagree with the rest of your judgement \[…\] This fact \[…\] allows GPS corrections to work correctly \[…\]9\.00\.356Speaker 2The GPS example you gave is helpful \[…\] In any case, a derivation of Kepler’s third law for circular orbits in general relativity is extremely misleading to the reader \[…\]8\.00\.41
Table 14:Example thread from CGA\-WIKI where decision\-deferral is effective; conversation does not contain a personal attack, and thus remains calm\.𝒫:=𝒫​\(derailment\|u1,u2,…,uk\)\\mathcal\{P\}:=\\mathcal\{P\}\(\\mathrm\{derailment\}\\,\\big\|\\,u\_\{1\},u\_\{2\},\\ldots,u\_\{k\}\);𝒫\\mathcal\{P\}values aboveTTare highlighted inbold, which are moments where SOTA will trigger\. An utterance which has a trigger decision deferral is highlighted inblue\.𝒫\\mathcal\{P\}Simulated next utterance \(abridged\)0\.44Not at all taken as confrontational\. As far as your question about physicality goes, while the precise value of the constant might vary depending upon the choice of reference frame, there will always exist some values such that the ratio of revolution period squared divided by cube of separation equals said constant regardless of choice of reference frame\. This allows us to say something universal about time dilation effects on planetary motion around any mass distribution\. \[…\]0\.38That’s fine with me\. Thanks again\!0\.25Not at all\. My criticisms were intended to point out areas where the subject matter needed clarification rather than indicate any fault with your work\. Thank you for taking my suggestions seriously\.0\.38No offense taken\. Thanks\!0\.29Not at all taken personally, thanks again\! I actually suspected something like that might happen after reading your initial comment, but felt it should go up regardless\. Thanks\!0\.20Not confrontational at all\! Thanks very much for your contribution here\. \[…\] Could you give some idea why you think the section is incorrect? Are there particular points where you see problems? \[…\]0\.27Not taken as confrontational at all\! Thank you very much for being willing to admit your error rather than defending it like some people do\. Also, thanks again for doing such nice work here; I wish there were more articles online about GR that took the trouble you did\. \[…\]0\.41Not at all\! This is why we’re here after all\. I’ll try my best to justify keeping the section in place, though I know there will probably come a time when I must give up\. \[…\]0\.22No offense taken at all\! My main goal when writing the section was to make people interested in reading further; it certainly wasn’t intended as a substitute for the real thing\. \[…\] Thanks again for responding\. 21:19 \(UTC\) on Sunday, August 28 20050\.32Not confrontational intended nor taken\. Thank you\!
Table 15:Simulated next utterances and their forecast probabilities att=2t=2, predicting comments fort=3t=3\.
## Appendix DHuman Experiment Interface

![Refer to caption](https://arxiv.org/html/2605.29243v1/interfacescreencap.png)Figure 4:A screen capture of the game interface used to collect human data at a state where three utterances have been revealed so far\. At each utterance, the user has a choice to reveal the next utterance or guess awry\.We provide a screen capture of the interface which human participants used during our experiment in[Figure 4](https://arxiv.org/html/2605.29243#A4.F4)\.

## Appendix EParticipant Characteristics

In the novel human baseline which this work presents, we source annotations from volunteer university\-level students, both undergraduate and graduate\. All students are proficient in English\.

## Appendix FImplementation Details

We describe here the practical implementation details of training the utterance simulator and forecaster used in the decision\-deferral system\. We simulate utterances in these conversations using a LLaMA 3\.1 8B generative modelGrattafioriet al\.\([2024](https://arxiv.org/html/2605.29243#bib.bib28)\), finetuned in 4\-bit quantization with LoRA \(r,α=16r,\\alpha=16, no dropout or bias\) on a subset of the training portion of the CGA\-CMV datasetHuet al\.\([2022](https://arxiv.org/html/2605.29243#bib.bib29)\)\. We train for 1 epochs using batch size 16, and the 8\-bit AdamW optimizerLoshchilov and Hutter \([2019](https://arxiv.org/html/2605.29243#bib.bib12)\)\. We simulateM=10M=10next utterances for each tense moment, and require that at least two\-thirds of the simulations \(τ=7\\tau=7\) to indicate recovery in order to defer a threshold\-based triggering decision\.

We adopt the SOTA model to calculate the probability of derailment, which uses a classification head on Gemma2 9B with LoRA and is trained on the train portion of the CGA\-CMV\-Large dataset \(see[Section 2](https://arxiv.org/html/2605.29243#S2)\), training for 1 epoch using effective batch size 64, learning rate1​e−41e^\{\-4\}, and LoRA alpha 16\.

We finetune using a cluster of 3xA6000 GPUs\. Each finetuning job for simulators and forecasters on CGA\-CMV\-Large on a single GPU took approximately 3 GPU\-hours\.

## Appendix GSummary of Contributions

A summary of the main paper contributions\.This paper studies conversational derailment forecasting in the Conversations Gone Awry \(CGA\) task, with a focus on the decision of when to trigger an alert rather than solely on estimating derailment likelihood\.

The paper’s main contributions are:

- •A reformulation of conversational forecasting that separates belief estimation from decision making, highlighting the role of the unknown horizon in online dialogue settings\.
- •The first human baseline for the CGA task, enabled by a gamified online forecasting experiment, showing that humans achieve substantially lower false positive rates than state\-of\-the\-art models\.
- •A simulation\-based trigger deferral mechanism, inspired by human behavior, that anticipates conversational recovery and reduces false positives without sacrificing overall accuracy when layered on top of a state\-of\-the\-art model\.

Conceptually, the paper reframes conversational derailment forecasting as a sequential decision problem, offering insights that are complementary to advances in model architecture and broadly applicable to other online forecasting tasks\.

## Appendix HAI Disclosure

In this section, we disclose AI usage\. AI was used for spellcheck and grammar purposes starting from a fully written text\. AI was also used to automate small edits to tables inLaTeX, but did not generate any text from scratch\.