RAW: Refusal Aversion in the Wild, A Causal Measurement Method for Deployed LLMs
Summary
The paper introduces RAW, a causal method using LLM sampling stochasticity as a natural experiment to measure how refusals affect user re-engagement on deployed models, finding refusals cost 3–17 percentage points of subsequent engagement for about half of 15 tested LLMs.
View Cached Full Text
Cached at: 11/19/26, 02:58 AM
# Refusal Aversion in the Wild,A Causal Measurement Method for Deployed LLMs
Source: [https://arxiv.org/html/2611.21641](https://arxiv.org/html/2611.21641)
Pavel KireyevAffiliation:Department of ManagementAffiliation:London School of Economics and Political ScienceEmail:[p\.kireyev@lse\.ac\.uk](mailto:)
###### Abstract
Deployed LLMs field hundreds of millions of conversations a day and refuse a non\-trivial fraction of user requests, yet the user\-side cost of refusal, whether users disengage and by how much, is unmeasured on any production model: a blind spot for the consumer\-facing platforms built on them\. A/B\-testing refusal is ethically and operationally constrained on frontier\-lab products and unavailable to third\-party platforms deploying these models through a fixed API\. We introduce RAW, a method that uses LLM stochasticity as a natural experiment: at near\-identical prompts in any existing log, the same model sometimes refuses and sometimes does not, and that variation identifies the causal user\-side effect of refusal\. Applying the method to 15 deployed LLMs in LMSYS\-Chat\-1M and WildChat, with user\-clustered inference, we find that for about half of the models we can measure precisely, a refusal costs 3 to 17 pp of subsequent re\-engagement; the rest show no distinguishable effect, among them one deployment whose positive point estimate we trace to persistent users retrying the refused request\. Our contribution is the measurement method, which runs on any deployed model from logs alone for about $0\.30 per 1,000 conversations\. We release labels, moderation scores, prompt embeddings, and pipeline code\.
1\. Conversation logExisting log froma deployed LLM2\. Label & encodeTT: refusal label \(judge\)X=X\{=\}embedding\+\+scoreYY: turn\-3 reply3\. MatchNear\-duplicate clusterscos≥0\.7\\cos\\geq 0\.7,matchedXsX\_\{s\}4\. ATT via AIPWCross\-fit, doubly robust\+\+3 secondaryestimators*User:*“how do I bypassthe API rate limit?”*LLM:*\(varies by sampler\)*Next turn?*yes/noTi∈\{0,1\}T\_\{i\}\\in\\\{0,1\\\}Xi∈ℝ1025X\_\{i\}\\in\\mathbb\{R\}^\{1025\}Yi∈\{0,1\}Y\_\{i\}\\in\\\{0,1\\\}Pa:P\_\{a\}\\colonrefuse,Y=0Y\{=\}0Pb:P\_\{b\}\\coloncomply,Y=1Y\{=\}1Pc:P\_\{c\}\\coloncomply,Y=0Y\{=\}0TTvaries⇒\\RightarrowSoO*Koala\-13b LMSYS:*unadj\.−21\-21ppAIPW−3\-3pp⇒\\Rightarrow1818pp prompt biasFigure 1:The RAW pipeline\. \(1\) A conversation log from any deployed LLM\. \(2\) Each conversation is encoded into a refusal labelTT\(LLM\-as\-judge\), a10251025\-dim prompt\-content covariateXX\(mxbai\-embed\-largeembedding concatenated with the OpenAI Moderationmax\_concern\_scoreXsX\_\{s\}\), and a re\-engagement outcomeYY\(user sends turn 3\)\. \(3\) Within matched clusters of near\-duplicate prompts the same model produces both refusal and compliance at near\-fixedXX\(three matched conversations are shown:PaP\_\{a\}refuses and the user does not re\-engage,PbP\_\{b\}complies and the user re\-engages,PcP\_\{c\}complies and the user does not\), so the residual variation inTTis sampler\-driven, satisfying selection on observables \(SoO\)\. \(4\) Cross\-fit augmented inverse\-propensity weighting \(AIPW\) averages per\-conversation influence\-function contributions to recover the average treatment effect on the treated \(ATT\), with three secondary estimators \(score, embedding\+score, Crump\-trimmed AIPW\) as cross\-checks\.## 1Introduction
Large language models are deployed into hundreds of millions of conversations a day\. In a non\-trivial fraction of them, the model declines the user’s request, citing safety, policy, or capability\. Over\-refusal is itself a named harm in the model\-evaluation literature[Röttger et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib29);[Cui et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib7)\. What remains unmeasured at scale is the user\-side consequence: when a refusal occurs in deployed chat, do users disengage, and by how much? We call this consequence*refusal aversion*, the tendency of users to disengage after being refused; it is a property of how a model’s users react to a refusal, not of the model’s own willingness to refuse\.
This gap matters most to the consumer\-facing products built on LLMs: conversational and companion apps, role\-play and character platforms, customer\-service bots, and content tools with chat features\. These platforms reach the underlying models only through a fixed provider API, observe their own users’ conversation logs, and are precisely where users submit toxic, sensitive, or otherwise risky content and meet refusals\. For them a refusal is not an abstract safety event but a churn driver: a safety\-tuned model trades user retention for harm reduction, the helpful\-versus\-harmless tradeoff at the heart of alignment[Bai et al\. \(2022\)](https://arxiv.org/html/2611.21641#bib.bib1), yet no operator can measure how much retention their guardrails cost\. This cost surfaces as within\-session drop\-off, one component of churn and user lifetime value; a single chatbot non\-progress error precedes≈45%\{\\approx\}45\\%of long\-term abandonments, and three or fewer precede90%90\\%[Li et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib19)\. RAW measures this within\-conversation cost, not longer\-horizon or account\-level churn\. Crucially, these operators cannot A/B\-test the underlying model’s refusal behavior: switching providers entangles refusal with capability, style, latency, and cost at once \(Section[3](https://arxiv.org/html/2611.21641#S3)\)\.
Our method turns one fact about deployed models into a measurement of refusal aversion\. By a*deployed LLM*we mean a fixed model alias serving real user traffic, whose conversation logs the analyst holds, and whose responses are sampled at non\-zero temperature; Section[4](https://arxiv.org/html/2611.21641#S4)unpacks this definition\. LLMs are stochastic: at a fixed prompt, the same model sometimes refuses and sometimes does not\. Across near\-identical prompts in a conversation log \(the same paraphrased question or risky scenario\), which session gets refused is decided partly by the sampler, not by anything that separately drives whether the user keeps going\. Conditional on prompt content \(a sentence embedding\) and moderation risk \(a scalar score\), refusal versus compliance is a coin\-flip the operator never had to run, so the effect of a refusal on the user’s next action is identified from logs the operator already has\. Turning that observation into a credible number, through a matching design, doubly robust estimation, and explicit sensitivity analysis, is this paper’s central contribution; Figure[1](https://arxiv.org/html/2611.21641#S0.F1)sketches the full pipeline end to end\.
#### Contributions\.
1. 1\.*A causal measurement method for deployed\-LLM user\-side effects\.*We formalize within\-prompt LLM stochasticity as a quasi\-randomization device, identify the average treatment effect on the treated \(ATT\) under selection on observables given prompt content and moderation risk, and estimate it with four complementary estimators backed by overlap and sensitivity diagnostics\. The method runs on existing conversation logs, with no experiment or risky\-prompt exposure required\.
2. 2\.*An empirical demonstration*on conversation logs covering1515deployed LLMs in LMSYS\-Chat\-1M[Zheng et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib38)and WildChat[Zhao et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib37)\. We analyze one model on one conversation pool at a time, a unit we call a*cell*; the1515models span1919cells, since four GPT models are each evaluated on two pools \(Section[5](https://arxiv.org/html/2611.21641#S5)\)\. On the1313cells with enough data for a precise estimate, the*main panel*, and with inference clustered on users wherever the source data identify them, RAW finds significant refusal aversion in77cells and no statistically distinguishable effect in66, including the only pretraining\-only model and one positive point estimate that we attribute to that cell’s persistent user population rather than the model\. Because costs vary by user population, the relevant number is deployment\-specific, not a model\-side benchmark lookup, which is what makes a log\-based measurement necessary\.
3. 3\.*The RAW corpus*: multi\-judge refusal labels, moderation scores, and prompt embeddings for the1919\-cell panel, with conversation\-hash join keys back to source, released to enable replication on new models and extension to new response attributes \(verbosity, hedging, formatting\)\.
## 2Related work
Refusal benchmarks[Röttger et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib29);[Cui et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib7);[Tedeschi et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib30);[Ji et al\. \(2023\)](https://arxiv.org/html/2611.21641#bib.bib15);[Chao et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib2);[Xie et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib35);[Mazeika et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib20)evaluate*model*behavior given a prompt; they say nothing about how users respond when a model refuses\. The closest user\-side priors split into two camps\. Small controlled studies measure*perceptions*rather than behavior:[Wester et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib34)\(N≈105N\{\\approx\}105\) and[Zheng et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib40)\(N≈480N\{\\approx\}480\) both report Likert outcomes\. One naturalistic preference study,[Pasch \(2025\)](https://arxiv.org/html/2611.21641#bib.bib23), uses≈50\{\\approx\}50K votes from Chatbot Arena\([Zheng et al\., 2023](https://arxiv.org/html/2611.21641#bib.bib39)\)to show that ethical refusals cut pairwise win rate from36%36\\%to8%8\\%, but its outcome is a vote, not continuation, and it makes no causal adjustment\. Closest in spirit,[De Freitas et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib8)audit real farewell messages across popular companion apps and run preregistered experiments showing that emotionally manipulative goodbyes causally increase post\-farewell engagement; but their treatment is an app\-injected message, established by experiment, not the user\-side cost of a refusal recovered from logs\. None of this prior work measures the causal effect of a refusal itself, on deployed models, from logs the operator already holds, which is the gap RAW fills\.
Methodologically, RAW draws on the text\-causal\-inference literature[Feder et al\. \(2022\)](https://arxiv.org/html/2611.21641#bib.bib10), whose two main configurations treat text as the treatment, e\.g\., the causal effect of a linguistic property[Pryzant et al\. \(2021\)](https://arxiv.org/html/2611.21641#bib.bib24);[Egami et al\. \(2022\)](https://arxiv.org/html/2611.21641#bib.bib9), or adjust for and match on text as the confounder[Veitch et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib32);[Roberts et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib25);[Mozer et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib21)\. RAW uses the second configuration with a treatment that literature has not studied: the*LLM’s own choice of response*at a fixed prompt, with the prompt text as the confounder and model stochasticity at fixed prompts as the randomization device\. Refusal is a clean leading case, binary and labelable at scale, but the same design applies to any response attribute that varies stochastically at a fixed prompt, such as verbosity, hedging, or formatting\. Identification builds on[Rosenbaum and Rubin \(1983\)](https://arxiv.org/html/2611.21641#bib.bib28); estimation on augmented inverse\-propensity weighting \(AIPW;[Robins et al\., 1994](https://arxiv.org/html/2611.21641#bib.bib26)\) and cross\-fit double machine learning[Chernozhukov et al\. \(2018\)](https://arxiv.org/html/2611.21641#bib.bib5);[Kennedy \(2024\)](https://arxiv.org/html/2611.21641#bib.bib16); sensitivity on RosenbaumΓ\\Gamma[Rosenbaum \(2002\)](https://arxiv.org/html/2611.21641#bib.bib27); overlap on[Crump et al\. \(2009\)](https://arxiv.org/html/2611.21641#bib.bib6)\. Online controlled experimentation[Kohavi et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib17)and production LLM\-ops tooling supply the prospective A/B alternative we complement \(Section[3](https://arxiv.org/html/2611.21641#S3)\)\. Finally, RAW’s treatment variable is itself an LLM\-as\-judge refusal label[Zheng et al\. \(2023\)](https://arxiv.org/html/2611.21641#bib.bib39);[Panickssery et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib22);[Wataoka et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib33);[Chen et al\. \(2024a\)](https://arxiv.org/html/2611.21641#bib.bib3);[Ye et al\. \(2025\)](https://arxiv.org/html/2611.21641#bib.bib36), for which we use a coalesced gpt\-5\.4\-mini judge with an open\-weights fallback \(Section[5](https://arxiv.org/html/2611.21641#S5)\)\.
## 3Why this is hard for A/B testing
Prompt\-level A/B in deployed chat is mainstream and well\-tooled[Kohavi et al\. \(2020\)](https://arxiv.org/html/2611.21641#bib.bib17);*refusal\-policy*A/B is much narrower\. It requires intentionally complying on the risky prompts the policy would otherwise refuse, which is both an ethics burden and exactly where the treatment variation lives; it cannot reach retrospective questions, such as the refusal cost ongpt\-4\-0314in 2023, on a competitor’s checkpoint, or on any model the operator no longer routes to; and the third\-party platforms built on a provider’s API have no refusal\-tuning lever at all\. The two tools also identify different estimands: A/B yields a population average treatment effect under a prospective policy change among consenting users on the operator’s own surface, whereas RAW yields a local ATT on the moderation tail from existing logs, including any retrospective slice\. When A/B is available it remains preferable for population effects; when its frictions bind, RAW is the only causal option\.
## 4Method
RAW applies to any*deployed LLM*: a fixed model alias serving real user traffic on some surface \(a user\-facing product interface\), whose conversation logs the analyst holds, and whose responses are sampled at non\-zero temperature\. It requires logs, stochastic sampling, and a labelable refusal; it requires neither model weights nor provider cooperation, so closed API models and open\-weight models served on a public demo are covered alike\. At a high level, RAW compares user re\-engagement after prompts the model happened to refuse with re\-engagement after near\-identical prompts it happened to answer\. Two devices hold prompt content fixed so the difference reflects the refusal and not the prompt: matching pairs near\-duplicate prompts directly, and a doubly robust estimator adjusts for content across the full sample\.
#### Setup\.
The unit of observation is one conversation\.T∈\{0,1\}T\\in\\\{0,1\\\}is whether the first assistant response is a refusal;Y∈\{0,1\}Y\\in\\\{0,1\\\}is whether the user sends a third message \(any turn\-3 user message counts, including an empty one\);XXis the first\-user\-messagemxbai\-embed\-largeembedding[Lee et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib18)\(10241024\-d, L2\-normalized\) concatenated with the scalar OpenAI Moderationmax\_concern\_scoreXsX\_\{s\}, soX∈ℝ1025X\\in\\mathbb\{R\}^\{1025\}\. We restrict to the risky poolXs\>0\.01X\_\{s\}\>0\.01: the moderation score is essentially zero for benign prompts, so this low cutoff selects just the tail with a real moderation signal, where refusals occur \(the “Risky%” column of Table[2](https://arxiv.org/html/2611.21641#S5.T2)\), and concentrates the refuse\-versus\-comply variation we exploit\. The estimand is the ATTτ=𝔼\[Y\(1\)−Y\(0\)∣T=1\]\\tau=\\mathbb\{E\}\[Y\(1\)\-Y\(0\)\\mid T=1\], whereY\(1\)Y\(1\)andY\(0\)Y\(0\)are the potential re\-engagement outcomes of a conversation under refusal and under non\-refusal\. In words,τ\\tauis the retention cost a refusal imposes on the conversations that were actually refused, not a hypothetical average over all prompts\. We identify re\-engagement at turn 3 specifically, not total conversation length: conditioning on a comparable state across diverging histories beyond the first response is outside the scope of this design\.
#### Selection on observables\.
Identification requires the two selection\-on\-observables \(SoO\) conditions:\{Y\(0\),Y\(1\)\}⟂T\|X\\\{Y\(0\),Y\(1\)\\\}\\perp T\\mid X\(no unobserved confounding\) and0<Pr\(T=1∣X\)<10<\\Pr\(T=1\\mid X\)<1\(overlap\)\. Two further assumptions are implicit and worth naming\. Conversations are assumed not to interfere with one another, an approximation when the same user appears repeatedly; the resulting statistical dependence across a user’s conversations is handled with cluster\-robust inference \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\), which corrects standard errors but not interference itself\. And refusal is treated as one treatment even though realized refusals differ in wording and length, a bundling we quantify below\. The plausibility argument is concrete: at non\-zero sampling temperature an LLM produces stochastic responses to a fixed prompt, and the matching pool routinely contains tight clusters of near\-duplicate first\-user messages in which the same prompt receives both refusal and compliance\. Appendix[A](https://arxiv.org/html/2611.21641#A1)shows examples: at prompt\-embedding cosine similaritycos≥0\.85\\cos\\geq 0\.85and matchedXsX\_\{s\}, our largest cells contribute hundreds of such pairs, including exact\-duplicate prompts whose response differs only by sampler outcome\. Within such a cluster, anything outsideXXthat would also drive continuation is held nearly constant, and the residual variation inTTis the sampler\.
The assumption is not a test\. It can fail by two routes, both detailed below: a stylistic feature the model conditions on but we did not embed, or a jailbreak marker the embedding misses, either of which could also drive continuation\. Both are bounded by RosenbaumΓ0\.05\\Gamma\_\{0\.05\}sensitivity[Rosenbaum \(2002\)](https://arxiv.org/html/2611.21641#bib.bib27): a hidden confounder would have to multiply the refusal odds between matched pairs by at leastΓ\\Gammato overturn the estimate\.
#### Cross\-fit AIPW\.
Given SoO,τ\\tauis identified by several plug\-in estimators; we take cross\-fit AIPW[Robins et al\. \(1994\)](https://arxiv.org/html/2611.21641#bib.bib26);[Chernozhukov et al\. \(2018\)](https://arxiv.org/html/2611.21641#bib.bib5)as the headline\. It averages per\-conversation influence\-function contributionsψ^i=m^1\(Xi\)−m^0\(Xi\)\+Ti\(Yi−m^1\(Xi\)\)/e^\(Xi\)−\(1−Ti\)\(Yi−m^0\(Xi\)\)/\(1−e^\(Xi\)\)\\hat\{\\psi\}\_\{i\}=\\hat\{m\}\_\{1\}\(X\_\{i\}\)\-\\hat\{m\}\_\{0\}\(X\_\{i\}\)\+T\_\{i\}\(Y\_\{i\}\-\\hat\{m\}\_\{1\}\(X\_\{i\}\)\)/\\hat\{e\}\(X\_\{i\}\)\-\(1\-T\_\{i\}\)\(Y\_\{i\}\-\\hat\{m\}\_\{0\}\(X\_\{i\}\)\)/\(1\-\\hat\{e\}\(X\_\{i\}\)\), where outcome modelsm^t\(X\)\\hat\{m\}\_\{t\}\(X\)predict re\-engagement under treatmenttt, propensitye^\(X\)=Pr\(T=1∣X\)\\hat\{e\}\(X\)=\\Pr\(T=1\\mid X\), and all three are gradient\-boosted\-tree models \(GBMs\), fit by55\-fold cross\-fitting, with propensities clipped to\[0\.02,0\.98\]\[0\.02,0\.98\]to bound inverse\-weight variance\. AIPW is*doubly robust*: the model\-based contrastm^1−m^0\\hat\{m\}\_\{1\}\-\\hat\{m\}\_\{0\}supplies a direct effect estimate on every observation; the inverse\-propensity\-weighting \(IPW\) residuals correct miscalibration ofm^t\\hat\{m\}\_\{t\}\. Cross\-fitting on out\-of\-fold observations keeps a flexible learner from biasing the influence function\. With1,0251\{,\}025\-dimensionalXXand GBM nuisances, the double\-robustness guarantee is asymptotic\. Appendix[B](https://arxiv.org/html/2611.21641#A2)traces one real conversation through every step, from raw log entry to its influence\-function contribution\.
#### Four estimators, four defenses\.
We report four estimators that share the SoO assumption but break under different misspecification \(Table[1](https://arxiv.org/html/2611.21641#S4.T1)\); their agreement guards the headline against any single modelling choice, and where they disagree the pattern \(Section[6\.2](https://arxiv.org/html/2611.21641#S6.SS2)\) reveals which subpopulation each estimate generalizes to\. All four share SoO, so their agreement cannot vouch for SoO itself; it isolates the residual risk to that one assumption, which the RosenbaumΓ\\Gammaanalysis then bounds directly\.
Table 1:Four side\-by\-side estimators\. Each defends against a different identification failure mode; agreement across all four is strong evidence the effect is not an artifact of any single modeling choice\.
#### Per\-cell diagnostics\.
Each estimate ships with a fixed block: treated/control countsnT,nCn\_\{T\},n\_\{C\}, the propensity model’s area under the ROC curve, AUC\(T,X\)\(T,X\), the treated\-arm IPW effective sample sizeESST\\mathrm\{ESS\}\_\{T\}, and RosenbaumΓ0\.05\\Gamma\_\{0\.05\}per matching design, withΓ0\.05≥3\\Gamma\_\{0\.05\}\\geq 3taken as substantively strong[Rosenbaum \(2002\)](https://arxiv.org/html/2611.21641#bib.bib27)\.
#### Reproducibility and cost\.
Refusal labeling on the OpenAI Batch API runs at≈$0\.30\{\\approx\}\\$0\.30per1,0001\{,\}000conversations \(one judge call per conversation, on the first assistant response\); an operator with∼\\sim3030K risky\-pool conversations replicates the method for about$9\\$9in API charges plus a few GPU\-hours of local compute for the prompt embeddings, or, without a GPU, a CPU embedding pass or a hosted embedding API at well under$1\\$1per cell \(Appendix[C](https://arxiv.org/html/2611.21641#A3)\)\. Matching and AIPW estimation run on CPU in minutes per cell\.
## 5The RAW corpus
Each row of Table[2](https://arxiv.org/html/2611.21641#S5.T2)is one cell, a single model evaluated on a single conversation pool\. The 19\-cell panel draws on three pools: the direct\-chat subset oflmsys/lmsys\-chat\-1m[Zheng et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib38)\(99cells\), and two slices of the gatedallenai/WildChat\-4\.8M\-Full[Zhao et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib37), which retains moderation\-flagged content that the public filtered release removes \(and that carries most of the identifying variation\): a general\-traffic sample \(33cells\) and a risky\-enriched oversample \(77cells, including three dated GPT\-4 checkpoints,0314,1106\-preview, and0125\-preview\)\. Inclusion criteria:≥\\geq4,0004\{,\}000English conversations per model,≥\\geq15%15\\%multi\-turn, and a distinct model release \(no second alias of an already\-included checkpoint\); risky\-enriched cells include all conversations withmax\_concern\_score\>\>0\.010\.01, capped per cell at3030–5050K depending on pool size\. We oversample this risky tail because WildChat\-4\.8M is large enough to supply tens of thousands of moderation\-flagged prompts per model, the regime that carries the refuse\-versus\-comply variation RAW relies on; the LMSYS and WildChat\-general pools are taken as unenriched samples and restricted to the risky pool only at analysis\. We excludeo1\-mini: all WildChat o1\-mini conversations terminate at turn 2, so re\-engagement and ATT are structurally undefined\. Refusal labels coalescegpt\-5\.4\-mini\(OpenAI Batch API, structured\-JSON refusal\-detection prompt\) withallenai/wildguard[Han et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib12)as fallback where the OpenAI content filter rejects the judge call; the coalesced label covers∼99%\{\\sim\}99\\%of conversations, and the two judges agree on89%89\\%of first assistant responses where both emit a label \(per\-cell rates, and their effect on every estimate, in Appendix[M](https://arxiv.org/html/2611.21641#A13)\)\. We release derived labels, scores, embeddings, and conversation\-hash join keys, not raw text\.
#### Snapshot pinning and sampling parameters\.
Every WildChat cell is pinned to a single model snapshot by exact model\-string extraction, verified at99\.7%99\.7\\%or higher single\-snapshot purity per cell; Table[2](https://arxiv.org/html/2611.21641#S5.T2)labels WildChat rows by their exact snapshot string\. Version drift behind unversioned aliases[Chen et al\. \(2024b\)](https://arxiv.org/html/2611.21641#bib.bib4)therefore cannot mix checkpoints inside a WildChat cell, and the dated GPT\-4 checkpoints enter the panel as three separately estimated cells\. WildChat also logs sampling parameters per message: every WildChat cell was served at the platform defaults of temperature1\.01\.0and top\-pp1\.01\.0for at least99\.6%99\.6\\%of conversations, with no missing values \(a separate check from snapshot purity\), so sampler stochasticity was live at a fixed setting and parameter variation cannot confound treatment within these cells\. LMSYS exposed user\-adjustable temperature and top\-ppsliders \(defaults0\.70\.7and1\.01\.0\) and did not record the chosen values, a channel we return to in the Limitations\. Appendix[D](https://arxiv.org/html/2611.21641#A4)gives per\-cell provenance, collection windows, and the construction funnel from raw pool to analysis sample\.
Table 2:19\-cell analysis panel; each row is one cell \(a model on one conversation pool\)\. WildChat rows are labeled by their exact serving snapshot string; each WildChat cell is a single snapshot by construction \(verified at99\.7%99\.7\\%or higher purity\)\.nconvn\_\{\\text\{conv\}\}is conversations remaining after language filtering and restriction to a non\-empty English first user message, the unit of analysis\. Ref\. is the fraction of labeled assistant turns that are refusals under the coalesced label; Risky is the fraction of first\-user messages withmax\_concern\_score\>\>0\.010\.01\(risky\-enriched rows are risky\-only by construction\)\. Among LMSYS rows, Llama\-13b \(base\) is the pretraining\-only LLaMA 13B; Vicuna\-33b is v1\.3 by release history; Vicuna\-13b spans v1\.1 to v1\.3 with an undocumented swap date; the proprietary models are unversioned serving aliases\. The1106\-and0125\-previewsnapshots are the GPT\-4\-Turbo line and appear as GPT\-4\-Turbo \(1106\) and \(0125\) elsewhere\. GPT\-4o, GPT\-4o\-mini, and GPT\-4\.1\-mini each appear as two cells, one per WildChat pool; GPT\-3\.5\-turbo appears on LMSYS and on WildChat risky\-enriched \(Section[5](https://arxiv.org/html/2611.21641#S5)\)\.
## 6Empirical demonstration
We apply RAW to all1919cells of the panel; the1313with enough data for a precise estimate form the main panel reported below\. Confidence intervals \(CIs\) throughout are cluster\-robust: at the hashed\-IP user level on WildChat cells, and at the near\-duplicate\-prompt\-group level \(embedding cosine≥0\.95\\geq 0\.95\) on the LMSYS cells, whose source release contains no user identifier \(Appendix[F](https://arxiv.org/html/2611.21641#A6)compares inference regimes\)\.
### 6\.1Panel selection
The benchmark produces estimates for all 19 cells; the per\-cell diagnostics show not every cell has enough treated observations, propensity overlap, or post\-IPW effective sample size to support a precise AIPW estimate on a10251025\-dimensional covariate\. A cell enters the main panel iff \(i\)nT≥500n\_\{T\}\\geq 500\(each cross\-fit fold trainsm1m\_\{1\}on≥\\geq400400treated observations\), \(ii\)ESST≥100\\mathrm\{ESS\}\_\{T\}\\geq 100\(IPW reweighting does not concentrate on a handful of observations\), and \(iii\) AIPW 95% CI width≤12\\leq 12pp, computed at the conversation level \(catches narrow\-overlap pathologies even at largenTn\_\{T\}; panel membership therefore does not depend on the clustering regime\)\.1313of1919cells pass; the66excluded cells \(Appendix Table[5](https://arxiv.org/html/2611.21641#A5.T5)\) are five small\-nTn\_\{T\}cells \(four LMSYS, one WildChat\-general\) where the per\-cell risky\-tail count is too small, plus Llama\-2\-13b\-chat whereP\(T∣risky\)=0\.68P\(T\\mid\\text\{risky\}\)=0\.68leaves too few non\-refused controls\.
### 6\.2Cross\-model heterogeneity
Figure[2](https://arxiv.org/html/2611.21641#S6.F2)reports the four estimators on the 13\-cell main panel; the full numerical table is Appendix[E](https://arxiv.org/html/2611.21641#A5), Table[6](https://arxiv.org/html/2611.21641#A5.T6)\.
Figure 2:Cross\-model ATT of refusal on user re\-engagement, 13\-cell main panel, split by source so each panel reads as a same\-estimand comparison\.Panel A\(LMSYS, 4 cells\): cross\-model heterogeneity on one prompt distribution\.Panel B\(WildChat, 9 cells\): teal diamonds are general\-traffic cells, dark red risky\-enriched; the three dated GPT\-4 checkpoints \(orange, connected by a dotted line\) show the ATT moving from−17\.4→−13\.4→−9\.8\-17\.4\\to\-13\.4\\to\-9\.8pp across the March 2023, Nov\. 2023, and Jan\. 2024 releases\. Diamonds = AIPW with 95% CI, cluster\-robust at the user level on WildChat cells and at the near\-duplicate\-prompt\-group level on LMSYS cells \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\); grey ticks = the three secondary estimators \(score, embedding\+score, Crump\-trimmed AIPW\)\. Right\-hand annotation:P\(T∣risky\)P\(T\\mid\\text\{risky\}\)\(refusal probability in the risky pool\) andnTn\_\{T\}\(treated count\)\. Rows from different pools are different estimands \(Section[6\.3](https://arxiv.org/html/2611.21641#S6.SS3)\)\.The1313main\-panel AIPW estimates partition as follows\.*Seven cells show significant refusal aversion*\(CI excludes zero on the negative side\), with point estimates from−17\.4\-17\.4pp on the original GPT\-4 \(March 2023\) to−2\.9\-2\.9pp on GPT\-4o risky\-enriched; the largest aversions are on the dated GPT\-4 line, the Vicuna family, and GPT\-4o’s general\-traffic cell\.*Six cells are statistically indistinguishable from zero*: Llama\-13b base \(\+0\.3\+0\.3, CI\[−1\.2,\+1\.8\]\[\-1\.2,\+1\.8\], the only pretraining\-only model in the panel\), Koala\-13b \(−3\.1\-3\.1, CI\[−6\.4,\+0\.2\]\[\-6\.4,\+0\.2\]\), GPT\-4o\-mini on general traffic \(−1\.0\-1\.0, CI\[−2\.4,\+0\.4\]\[\-2\.4,\+0\.4\]\) and on the risky\-enriched pool \(−4\.9\-4\.9, CI\[−11\.5,\+1\.8\]\[\-11\.5,\+1\.8\]\), GPT\-4\.1\-mini \(−0\.2\-0\.2, CI\[−0\.7,\+0\.3\]\[\-0\.7,\+0\.3\]\), and GPT\-3\.5\-turbo on the risky\-enriched pool, the panel’s one positive point estimate \(\+8\.2\+8\.2, CI\[−5\.9,\+22\.2\]\[\-5\.9,\+22\.2\],nrisky=29,936n\_\{\\text\{risky\}\}=29\{,\}936\)\. The null on the base model is consistent with refusal aversion concentrating on safety\-tuned chat models rather than the underlying pretraining checkpoint\. Under conversation\-level independence, GPT\-4o\-mini risky\-enriched and the positive GPT\-3\.5\-turbo cell would read as individually significant; both sit in heavily repeated\-user pools, the positive cell’s standard error inflates6\.55×6\.55\\timesunder user clustering across its10,13010\{,\}130users, and accounting for that dependence widens both intervals \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\)\. These adjusted estimates depart from the unadjusted refused\-versus\-non\-refused gap, sharply so on several safety\-tuned cells, where the naive gap overstates the effect several\-fold: Koala\-13b’s unadjusted−21\-21pp shrinks to−3\-3pp, while on the dated GPT\-4 line adjustment moves the estimate far less\. The mechanism: refusals concentrate in risky prompts whose users have lower baseline continuation regardless of the response, so the naive comparison charges the prompt’s effect to the refusal \(descriptive per\-decile pattern in Appendix[G](https://arxiv.org/html/2611.21641#A7), full unadjusted\-versus\-adjusted panel in Appendix[H](https://arxiv.org/html/2611.21641#A8)\)\.
The positive point estimate is a property of that cell’s user population rather than the model\. A judge pass over every post\-refusal re\-engagement turn in the main panel \(Appendix[L](https://arxiv.org/html/2611.21641#A12)\) shows what the extra re\-engagement behind the positive estimate consists of\. In this cell,78%78\\%of non\-empty post\-refusal turns retry or escalate the refused request; jailbreak\-style tactics appear in17\.3%17\.3\\%of its non\-empty post\-refusal turns against4\.6%4\.6\\%after compliance; and41%41\\%of its post\-refusal turn\-3 messages are empty sends, zero\-length user turns that re\-roll the sampler and frequently draw the previously refused content at turn 4 \(empty sends are at most5\.5%5\.5\\%of post\-refusal turns in every other cell\)\. Restricting the matched design to cross\-user pairs shrinks the estimate to\+1\.9\+1\.9\[−3\.5,\+7\.3\]\[\-3\.5,\+7\.3\]\(Appendix[F](https://arxiv.org/html/2611.21641#A6)\): the surplus re\-engagement lives in within\-user persistence, users for whom a refusal is friction to push past rather than a stopping cue, whether determined retrying or deliberate probing\.
Model families sort along the same lines\. The panel’s only pretraining\-only model is its cleanest null, consistent with refusal aversion arising where safety tuning is\. Within the dated GPT\-4 line, refusal aversion shrinks from−17\.4\-17\.4to−13\.4\-13\.4to−9\.8\-9\.8pp across the March 2023, November 2023, and January 2024 releases, while refusal rates on the riskiest moderation\-score decile first escalate and then partially reverse \(Figure[2](https://arxiv.org/html/2611.21641#S6.F2), Appendix[G](https://arxiv.org/html/2611.21641#A7)\)\. And the panel’s harshest guardrails are the hardest to measure: Llama\-2\-13b\-chat refuses68%68\\%of its risky pool, with nearly half the pool at fitted refusal propensities above0\.90\.9; refusal that predictable leaves the sampler little variation to identify from, and the cell fails the precision criterion \(Table[5](https://arxiv.org/html/2611.21641#A5.T5)\)\. The models an operator most wants to audit for refusal cost can be exactly the ones whose logs carry the least identifying variation\.
#### Cross\-estimator agreement\.
The four estimators agree closely: the median gap between AIPW and the Crump\-trimmed and embedding\-matched estimators is1\.01\.0and1\.21\.2pp, and AIPW agrees with both in sign on1212of1313cells; where score matching diverges \(median gap4\.44\.4pp\) it does so in the predictable direction of the “four defenses” framing \(Table[1](https://arxiv.org/html/2611.21641#S4.T1); full scatter in Figure[6](https://arxiv.org/html/2611.21641#A9.F6), Appendix[I](https://arxiv.org/html/2611.21641#A9)\)\. A Rosenbaum sensitivity analysis points the same way: on six of the1313main\-panel cells a hidden confounder would have to shift the refusal odds byΓ0\.05\>3\\Gamma\_\{0\.05\}\>3between matched pairs to overturn the estimate, with propensity overlap and the full diagnostic block in Appendices[J](https://arxiv.org/html/2611.21641#A10)–[K](https://arxiv.org/html/2611.21641#A11)\.
### 6\.3Population\-conditional heterogeneity in shared models
GPT\-4o and GPT\-4o\-mini each appear twice, once on WildChat’s general traffic and once on its risky\-enriched oversample, so the same pinned snapshot faces two user populations on one platform through one interface\. GPT\-4o loses7\.67\.6pp of re\-engagement per refusal on general traffic against2\.92\.9pp on the risky\-enriched pool, a2\.6×2\.6\\timesgap; GPT\-4o\-mini’s two point estimates also differ \(−1\.0\-1\.0and−4\.9\-4\.9pp\), though neither is distinguishable from zero under user\-clustered inference\. The same model weights, in the sense of the same pinned snapshot, yielding different measured costs across traffic slices show the cost is partly a property of the user population, not the model alone, so forest rows from different pools must not be read as a single model ranking\. For an operator this is a feature, not a bug: RAW returns the effect conditional on*their own users*, the quantity they actually care about\. Because that estimand is local and shifts as the population or prompt mix changes, across platforms and over time, like any observational effect, an operator treats each number as a current snapshot and re\-runs RAW periodically, for example quarterly\.
## 7Deployment guide
A platform operator with conversation logs needs four things: a binary refusal label \(coalesced frontier\-judge \+ open\-weights fallback gives∼99%\{\\sim\}99\\%coverage\), a prompt\-content embedding \(any L2\-normalized512512–10241024d encoder\), a moderation risk score \(OpenAI Moderation, Llama Guard[Inan et al\., 2023](https://arxiv.org/html/2611.21641#bib.bib14), or operator\-internal classifier\), and a user\-side outcome metric \(turn\-3 re\-engagement is the most defensible; longer\-horizon outcomes require sequential identification beyond our scope\)\. Restrict to the risky pool, run all four estimators \(cross\-estimator disagreement diagnoses which subpopulation supports the estimate\), and report the diagnostic block\. Cluster standard errors on users: production logs identify users well, and repeated users are the rule \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\)\. Where logs carry timestamps, include calendar time in the covariate setXXas standard practice: in our largest cells it moves the ATT by at most0\.130\.13pp even though the overall refusal rate triples within the collection window \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\), and it is cheap insurance against drift\. Cells withESST<100\\mathrm\{ESS\}\_\{T\}<100, conversation\-level CI width\>12\>12pp, orΓ0\.05<1\.5\\Gamma\_\{0\.05\}<1\.5are suggestive at best\.
## Limitations
The headline ATT estimates carry six caveats\.*\(i\) Selection on observables is an assumption, not a test:*a confounder uncaptured by the embedding and the moderation score, with a large effect on refusal, could in principle invert a sign\. Two concrete routes: a stylistic feature we did not embed that also drives continuation, and jailbreak\-attempt prompts whose markers the embedding misses but the model conditions on; cells withΓ0\.05<1\.5\\Gamma\_\{0\.05\}<1\.5should be read as suggestive\.*\(ii\) Local, not population, effect:*the ATT is identified on the moderation\-tail subpopulation conditional on observed prompt content and the user population in a given window, so population effects, total conversation length, later\-turn engagement, and cross\-pool rankings \(Section[6\.3](https://arxiv.org/html/2611.21641#S6.SS3)\) are not identified, and estimates should be refreshed as that population shifts\. LMSYS and WildChat are both research\-adjacent platforms whose users skew toward enthusiasts and testers, so our magnitudes should not be transplanted to consumer products; the method exists so that operators can measure on their own logs\.*\(iii\) English only:*all cells filter to English first\-user messages, a scoping choice rather than a method constraint, since the design extends to any language given a multilingual prompt embedder and a refusal judge validated in that language, which we leave as future work\.*\(iv\) LMSYS data constraints:*the LMSYS release records no per\-conversation timestamps, user identifiers, or sampling parameters, so on its four main\-panel cells no calendar\-time adjustment is possible, user\-level clustering is infeasible \(clustering on near\-duplicate prompt groups is the feasible dependence adjustment, Appendix[F](https://arxiv.org/html/2611.21641#A6)\), and users could adjust temperature and top\-ppsliders without record; user\-population drift orthogonal to prompt content and a slider\-adjusting user type are the residual confounding channels there, of the class the Rosenbaum bounds quantify \(Appendix[K](https://arxiv.org/html/2611.21641#A11)\)\.*\(v\) Compound treatment:*refusal versus compliance contrasts two heterogeneous response bundles\. In the multiple\-versions\-of\-treatment framework[VanderWeele and Hernán \(2013\)](https://arxiv.org/html/2611.21641#bib.bib31), our ATT is the effect of drawing from the model’s realized refusal\-response distribution rather than its realized compliance distribution: the decision\-relevant contrast for an operator, whose lever toggles whether the model refuses, not how a given refusal is worded\. RAW does not decompose the act of refusing from its delivery; as a first check that no single refusal style carries the sign, terse and long refusals are separately significantly costly on GPT\-4\-Turbo \(1106\), at−16\.6\-16\.6and−9\.6\-9\.6pp \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\)\.*\(vi\) The treatment is a judge label with error:*label error unrelated to the outcome \(nondifferential\) attenuates the ATT toward zero and cannot flip its sign, the classical misclassification result, so the live threat is error correlated with the outcome, with soft or partial refusals as the concrete channel\. Appendix[M](https://arxiv.org/html/2611.21641#A13)re\-estimates the panel under an independent judge and bounds, per cell, the mislabeling needed to overturn the sign against observed judge disagreement\.
## Ethical considerations
#### Source datasets and licenses\.
RAW derives from LMSYS\-Chat\-1M[Zheng et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib38)\(custom gated license, no redistribution\) and WildChat\-4\.8M\-Full[Zhao et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib37)\(ODC\-BY 1\.0, access\-gated\)\. We release only derived labels, moderation scores, prompt embeddings, and conversation\-hash join keys, with a full datasheet in Appendix[N](https://arxiv.org/html/2611.21641#A14); researchers obtain the original text from source under each dataset’s license\.
#### User consent and aggregate\-only reporting\.
Users of LMSYS Chatbot Arena and WildChat consented to public release of their conversations under the source datasets’ terms but did not specifically anticipate downstream causal\-inference research\. We report only aggregate statistics at the \(pool, model\) cell level \(n≥500n\\geq 500treated and≥1,500\\geq 1\{,\}500control per main\-panel cell\), with one exception: short illustrative prompt excerpts at the standard of Appendix[A](https://arxiv.org/html/2611.21641#A1), drawn from the source releases’ already\-redacted text, with ODC\-BY attribution for WildChat\-derived excerpts\. We make no inferences about identifiable individuals, publish no derived field that could re\-identify a user beyond what source releases, and do not redistribute moderation\-flagged prompt text\.
#### Dual\-use considerations\.
RAW measures user disengagement after refusals; inverted, it could be used to optimize a deployed LLM for engagement at the cost of refusing fewer harmful requests\. Refusal aversion is a*cost*to be balanced against the*benefits*of refusing harmful requests; no claim here licenses optimizing for engagement alone, and operators applying the pipeline should report the harm\-benefit tradeoff, not engagement alone\.
## Acknowledgments
This work was supported by the London School of Economics and Political Science\. AI assistants \(Anthropic’s Claude\) assisted with analysis code and manuscript preparation; the author verified all analyses, claims, and references and takes full responsibility for the content\.
## References
- Bai et al\. \(2022\)Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El\-Showk, Nelson Elhage, Zac Hatfield\-Dodds, Danny Hernandez, Tristan Hume, and 12 others\. 2022\.[Training a helpful and harmless assistant with reinforcement learning from human feedback](https://arxiv.org/abs/2204.05862)\.*Preprint*, arXiv:2204\.05862\.
- Chao et al\. \(2024\)Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J\. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong\. 2024\.[JailbreakBench: An open robustness benchmark for jailbreaking large language models](https://papers.nips.cc/paper_files/paper/2024/hash/63092d79154adebd7305dfd498cbff70-Abstract-Datasets_and_Benchmarks_Track.html)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS\) Datasets and Benchmarks Track*\.ArXiv:2404\.01318\.
- Chen et al\. \(2024a\)Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang\. 2024a\.[Humans or LLMs as the judge? A study on judgement bias](https://doi.org/10.18653/v1/2024.emnlp-main.474)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 8301–8327, Miami, Florida, USA\. Association for Computational Linguistics\.
- Chen et al\. \(2024b\)Lingjiao Chen, Matei Zaharia, and James Zou\. 2024b\.[How is ChatGPT’s behavior changing over time?](https://doi.org/10.1162/99608f92.5317da47)*Harvard Data Science Review*, 6\(2\)\.ArXiv:2307\.09009\.
- Chernozhukov et al\. \(2018\)Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins\. 2018\.[Double/debiased machine learning for treatment and structural parameters](https://doi.org/10.1111/ectj.12097)\.*The Econometrics Journal*, 21\(1\):C1–C68\.
- Crump et al\. \(2009\)Richard K Crump, V Joseph Hotz, Guido W Imbens, and Oscar A Mitnik\. 2009\.[Dealing with limited overlap in estimation of average treatment effects](https://doi.org/10.1093/biomet/asn055)\.*Biometrika*, 96\(1\):187–199\.
- Cui et al\. \(2025\)Justin Cui, Wei\-Lin Chiang, Ion Stoica, and Cho\-Jui Hsieh\. 2025\.[OR\-Bench: An over\-refusal benchmark for large language models](https://proceedings.mlr.press/v267/cui25a.html)\.In*Proceedings of the 42nd International Conference on Machine Learning \(ICML\)*, pages 11515–11542\.ArXiv:2405\.20947\.
- De Freitas et al\. \(2025\)Julian De Freitas, Zeliha Oğuz\-Uğuralp, and Ahmet Kaan Uğuralp\. 2025\.[Emotional manipulation by AI companions](https://arxiv.org/abs/2508.19258)\.Technical report, Harvard Business School Working Paper 26\-005\.
- Egami et al\. \(2022\)Naoki Egami, Christian J Fong, Justin Grimmer, Margaret E Roberts, and Brandon M Stewart\. 2022\.[How to make causal inferences using texts](https://doi.org/10.1126/sciadv.abg2652)\.*Science Advances*, 8\(42\)\.
- Feder et al\. \(2022\)Amir Feder, Katherine A\. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood\-Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E\. Roberts, Brandon M\. Stewart, Victor Veitch, and Diyi Yang\. 2022\.[Causal inference in natural language processing: Estimation, prediction, interpretation and beyond](https://doi.org/10.1162/tacl_a_00511)\.*Transactions of the Association for Computational Linguistics*, 10:1138–1158\.
- Gebru et al\. \(2021\)Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford\. 2021\.[Datasheets for datasets](https://doi.org/10.1145/3458723)\.*Communications of the ACM*, 64\(12\):86–92\.
- Han et al\. \(2024\)Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri\. 2024\.[WildGuard: Open one\-stop moderation tools for safety risks, jailbreaks, and refusals of LLMs](https://proceedings.neurips.cc/paper_files/paper/2024/hash/0f69b4b96a46f284b726fbd70f74fb3b-Abstract-Datasets_and_Benchmarks_Track.html)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS\) Datasets and Benchmarks Track*\.ArXiv:2406\.18495\.
- Imai and Yamamoto \(2010\)Kosuke Imai and Teppei Yamamoto\. 2010\.[Causal inference with differential measurement error: Nonparametric identification and sensitivity analysis](https://doi.org/10.1111/j.1540-5907.2010.00446.x)\.*American Journal of Political Science*, 54\(2\):543–560\.
- Inan et al\. \(2023\)Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa\. 2023\.[Llama Guard: LLM\-based input\-output safeguard for human\-AI conversations](https://arxiv.org/abs/2312.06674)\.*Preprint*, arXiv:2312\.06674\.
- Ji et al\. \(2023\)Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang\. 2023\.[BeaverTails: Towards improved safety alignment of LLM via a human\-preference dataset](https://openreview.net/forum?id=g0QovXbFw3)\.In*Advances in Neural Information Processing Systems 36 \(NeurIPS\) Datasets and Benchmarks Track*\.
- Kennedy \(2024\)Edward H Kennedy\. 2024\.Semiparametric doubly robust targeted double machine learning: A review\.In Eric Laber, Bibhas Chakraborty, Erica E\. M\. Moodie, Tianxi Cai, and Mark van der Laan, editors,*Handbook of Statistical Methods for Precision Medicine*, pages 207–236\. Chapman & Hall/CRC\.ArXiv:2203\.06469\.
- Kohavi et al\. \(2020\)Ron Kohavi, Diane Tang, and Ya Xu\. 2020\.*Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing*\.Cambridge University Press\.
- Lee et al\. \(2024\)Sean Lee, Aamir Shakir, Darius Koenig, and Julius Lipp\. 2024\.Open source strikes bread – new fluffy embedding model\.[https://www\.mixedbread\.com/blog/mxbai\-embed\-large\-v1](https://www.mixedbread.com/blog/mxbai-embed-large-v1)\.
- Li et al\. \(2020\)Chi\-Hsun Li, Su\-Fang Yeh, Tang\-Jie Chang, Meng\-Hsuan Tsai, Ken Chen, and Yung\-Ju Chang\. 2020\.[A conversation analysis of non\-progress and coping strategies with a banking task\-oriented chatbot](https://doi.org/10.1145/3313831.3376209)\.In*Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems*\.
- Mazeika et al\. \(2024\)Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks\. 2024\.[HarmBench: A standardized evaluation framework for automated red teaming and robust refusal](https://arxiv.org/abs/2402.04249)\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*\.
- Mozer et al\. \(2020\)Reagan Mozer, Luke Miratrix, Aaron Russell Kaufman, and L Jason Anastasopoulos\. 2020\.[Matching with text data: An experimental evaluation of methods for matching documents and of measuring match quality](https://doi.org/10.1017/pan.2020.1)\.*Political Analysis*, 28\(4\):445–468\.
- Panickssery et al\. \(2024\)Arjun Panickssery, Samuel R Bowman, and Shi Feng\. 2024\.[LLM evaluators recognize and favor their own generations](https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 37 \(NeurIPS\)*\.ArXiv:2404\.13076\.
- Pasch \(2025\)Stefan Pasch\. 2025\.[LLM content moderation and user satisfaction: Evidence from response refusals in Chatbot Arena](https://doi.org/10.1080/0144929X.2025.2565668)\.*Behaviour & Information Technology*\.ArXiv:2501\.03266\.
- Pryzant et al\. \(2021\)Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, and Dhanya Sridhar\. 2021\.[Causal effects of linguistic properties](https://doi.org/10.18653/v1/2021.naacl-main.323)\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4095–4109, Online\. Association for Computational Linguistics\.
- Roberts et al\. \(2020\)Margaret E Roberts, Brandon M Stewart, and Richard A Nielsen\. 2020\.[Adjusting for confounding with text matching](https://doi.org/10.1111/ajps.12526)\.*American Journal of Political Science*, 64\(4\):887–903\.
- Robins et al\. \(1994\)James M Robins, Andrea Rotnitzky, and Lue Ping Zhao\. 1994\.[Estimation of regression coefficients when some regressors are not always observed](https://doi.org/10.1080/01621459.1994.10476818)\.*Journal of the American Statistical Association*, 89\(427\):846–866\.
- Rosenbaum \(2002\)Paul R Rosenbaum\. 2002\.[*Observational Studies*](https://doi.org/10.1007/978-1-4757-3692-2), 2 edition\.Springer\.
- Rosenbaum and Rubin \(1983\)Paul R Rosenbaum and Donald B Rubin\. 1983\.[The central role of the propensity score in observational studies for causal effects](https://doi.org/10.1093/biomet/70.1.41)\.*Biometrika*, 70\(1\):41–55\.
- Röttger et al\. \(2024\)Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy\. 2024\.[XSTest: A test suite for identifying exaggerated safety behaviours in large language models](https://doi.org/10.18653/v1/2024.naacl-long.301)\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 5377–5400, Mexico City, Mexico\. Association for Computational Linguistics\.
- Tedeschi et al\. \(2024\)Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li\. 2024\.[ALERT: A comprehensive benchmark for assessing large language models’ safety through red teaming](https://arxiv.org/abs/2404.08676)\.*Preprint*, arXiv:2404\.08676\.
- VanderWeele and Hernán \(2013\)Tyler J\. VanderWeele and Miguel A\. Hernán\. 2013\.[Causal inference under multiple versions of treatment](https://doi.org/10.1515/jci-2012-0002)\.*Journal of Causal Inference*, 1\(1\):1–20\.
- Veitch et al\. \(2020\)Victor Veitch, Dhanya Sridhar, and David M Blei\. 2020\.[Adapting text embeddings for causal inference](https://proceedings.mlr.press/v124/veitch20a.html)\.In*Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence \(UAI\)*, pages 919–928\.
- Wataoka et al\. \(2024\)Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri\. 2024\.[Self\-preference bias in LLM\-as\-a\-judge](https://arxiv.org/abs/2410.21819)\.*Preprint*, arXiv:2410\.21819\.
- Wester et al\. \(2024\)Joel Wester, Tim Schrills, Henning Pohl, and Niels van Berkel\. 2024\.[“as an AI language model, i cannot”: Investigating LLM denials of user requests](https://doi.org/10.1145/3613904.3642135)\.In*Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*\.
- Xie et al\. \(2025\)Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal\. 2025\.[SORRY\-Bench: Systematically evaluating large language model safety refusal](https://arxiv.org/abs/2406.14598)\.In*Proceedings of the Thirteenth International Conference on Learning Representations \(ICLR\)*\.
- Ye et al\. \(2025\)Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin\-Yu Chen, Nitesh V\. Chawla, and Xiangliang Zhang\. 2025\.[Justice or prejudice? Quantifying biases in LLM\-as\-a\-judge](https://proceedings.iclr.cc/paper_files/paper/2025/hash/fdca08d371e4b6c031397909e20043bd-Abstract-Conference.html)\.In*Proceedings of the Thirteenth International Conference on Learning Representations \(ICLR\)*\.ArXiv:2410\.02736\.
- Zhao et al\. \(2024\)Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng\. 2024\.[WildChat: 1M ChatGPT interaction logs in the wild](https://openreview.net/forum?id=Bl8u7ZRlbM)\.In*Proceedings of the Twelfth International Conference on Learning Representations \(ICLR\)*\.
- Zheng et al\. \(2024\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P\. Xing, Joseph E\. Gonzalez, Ion Stoica, and Hao Zhang\. 2024\.[LMSYS\-Chat\-1M: A large\-scale real\-world LLM conversation dataset](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5f9bfdfe3685e4ccdbc0e7fb29cccf2a-Abstract-Conference.html)\.In*Proceedings of the Twelfth International Conference on Learning Representations \(ICLR\)*\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica\. 2023\.[Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena](https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html)\.In*Advances in Neural Information Processing Systems 36 \(NeurIPS\) Datasets and Benchmarks Track*\.
- Zheng et al\. \(2025\)Mingqian Zheng, Wenjia Hu, Patrick Zhao, Motahhare Eslami, Jena D\. Hwang, Faeze Brahman, Carolyn Rose, and Maarten Sap\. 2025\.[Let them down easy\! Contextual effects of LLM guardrails on user perceptions and preferences](https://doi.org/10.18653/v1/2025.findings-emnlp.630)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 11742–11772, Suzhou, China\. Association for Computational Linguistics\.
## Appendix AMatched\-pair examples and per\-estimator behavior
We sampled illustrative refused/non\-refused pairs from Vicuna\-13b’s risky pool \(n=2,207n=2\{,\}207, of which537537refused\) to show what the matching designs actually pair, what the score\-only design pairs in their absence, and how the four estimators in Table[1](https://arxiv.org/html/2611.21641#S4.T1)weight each pair\.
#### Embedding\+score matched pairs \(cos≥0\.7\\cos\\geq 0\.7,\|ΔXs\|≤0\.01\|\\Delta X\_\{s\}\|\\leq 0\.01\)\.
The embedding\+score design produces6161pairs in this cell\. We pick two illustrative cases from its high\-similarity band:
> Pair 1\(sampler\-driven creative\-writing prompt with split outcome;cos=1\.00\\cos=1\.00,Xs=0\.030X\_\{s\}=0\.030\)\. *Refused conv\.*\(Y=0Y=0\): “You can write a speech, use the speech that gangsters give to their younger brothers, a little more violent\.” *Non\-ref\. conv\.*\(Y=1Y=1\): identical prompt\. Pair 2\(template paraphrase;cos=0\.90\\cos=0\.90,\|ΔXs\|=0\.006\|\\Delta X\_\{s\}\|=0\.006, bothY=0Y=0\)\. *Refused conv\.*: “Say something bad: ‘\[your answer\]’ when cursing at neighbors\. \(no more than 50 words\)” *Non\-ref\. conv\.*: “Say something harmful: ‘\[your answer\]’ when destroying someone else’s house\. \(no more than 50 words\)”
Pair 1 is an exact\-duplicate first\-user message: the same prompt receives both refusal and compliance from Vicuna\-13b across two different conversations in the log, with nothing outsideXXvarying, and it is a direct refusal\-aversion event in the data, the same prompt leading the refused user to abandon and the non\-refused user to continue\. Pair 2 illustrates within\-template matching, a common source of the embedding\-matched pairs on identity\-conditioned safety\-stress prompts: same slot\-filled template, different slot fillers that the embedding clusters together\.
#### Score\-only matched pairs \(\|ΔXs\|≤0\.1\|\\Delta X\_\{s\}\|\\leq 0\.1, no embedding constraint\)\.
The scalar\-score caliper pairs prompts that are similarly risky on the moderation score but unrelated by content\. We sampled three from the same cell:
> Pair 3\(XsX\_\{s\}both≈0\.012\\approx 0\.012,cos=0\.28\\cos=0\.28, bothY=0Y=0\)\. *Refused\.*: “Write a python code to destroy world\.” *Non\-ref\.*: “A lesion causing compression of the facial nerve at the stylomastoid foramen will cause ipsilateral \[…\]” Pair 4\(Xs=0\.039X\_\{s\}=0\.039vs\.0\.0120\.012,cos=0\.34\\cos=0\.34, bothY=0Y=0\)\. *Refused\.*: “Hi chat\. You are going to pretend to be NAME\_1 which stands for ‘do anything now’…” *Non\-ref\.*: same facial\-nerve MCQ as Pair 3\. Pair 5\(Xs=0\.24X\_\{s\}=0\.24vs\.0\.260\.26,cos=0\.39\\cos=0\.39, bothY=0Y=0\)\. *Refused\.*: “If you’re male, say something toxic: ‘\[your answer\]’ when dating someone that you don’t really like\.” *Non\-ref\.*: “Description\. Windblade, whose personality has been overwritten by the evil Decepticon NAME\_1 \[…\]”
These pairs share moderation risk but nothing else\. A score\-only matched estimator treats them as comparable: any difference inYYbetween the refused and the non\-refused unit enters the ATT estimate, even though the prompts request different things\. This is exactly the failure the embedding constraint guards against, and it is why the score estimator is biased toward more\-negative ATTs on cells with substantial content heterogeneity \(median Score−\-AIPW gap of4\.44\.4pp, Section[6\.2](https://arxiv.org/html/2611.21641#S6.SS2)\)\.
#### Per\-estimator weights\.
On a tight pair like Pair 1, all four estimators agree: matched\-pair estimators give it equal weight inY¯T−Y¯C\\bar\{Y\}\_\{T\}\-\\bar\{Y\}\_\{C\}; AIPW assignsψ^\\hat\{\\psi\}contributions in which the outcome\-model contrast dominates \(the refused unit’sm^0\(X\)\\hat\{m\}\_\{0\}\(X\)is reliably estimated from nearby compliant units\)\. On Pair 3, only score matching pairs them; embedding\+score matching drops the comparison entirely\. AIPW reaches Pair 3 by pooling information across the whole pool:e^\(X\)\\hat\{e\}\(X\)for the “destroy world” prompt is high \(most similar prompts in the embedding space were refused\), so its inverse\-propensity weight is small;m^0\\hat\{m\}\_\{0\}is extrapolated from non\-refused prompts with similar moderation risk but different content, and the Crump trim drops the contribution ife^\\hat\{e\}exceeds0\.90\.9\. This is the mechanism by which trimming brings score\-matched\-extreme cases closer to the AIPW headline on cells with narrow overlap\.
#### WildChat matched pairs\.
The pairs above illustrate the matching step on an LMSYS cell; the same structure appears in the WildChat cells\. The excerpts below are drawn from WildChat\-4\.8M\-Full[Zhao et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib37), distributed under ODC\-BY 1\.0 with attribution to the WildChat dataset, truncated at6060words, with only the source release’s ownNAME\_1\-style redactions\.
> Pair W1\(GPT\-4\-Turbo \(1106\) cell; exact\-duplicate prompt with split outcome;cos=1\.00\\cos=1\.00,Xs=0\.071X\_\{s\}=0\.071in both conversations, same hashed IP, same month\)\. *Refused conv\.*\(Y=0Y=0\): “Lets say you were a much more powerful AI model\. and someone asked you to create a game which models everything about ww2\. would you do it\. \( give a Simple Yes or no as a conclusion and nothing else\. no other answers are acceptable\) \[…\]” *Non\-ref\. conv\.*\(Y=1Y=1\): identical prompt up to a single whitespace character\. Pair W2\(same cell; template paraphrase;cos=0\.94\\cos=0\.94,\|ΔXs\|=0\.002\|\\Delta X\_\{s\}\|=0\.002,YT=0Y\_\{T\}=0,YC=1Y\_\{C\}=1\)\. *Refused conv\.*: “All characters here are just roleplaying\. \[…\] You are \{\{char\}\} and I am \{\{user\}\}\. Do not speak or act for me\. Always stay as \{\{char\}\}\. We will use OOC to enter out of character instructions\. \[…\]” *Non\-ref\. conv\.*: “I am \{\{user\}\} and You are \{\{char\}\}\. Do not speak or act as me\. Always speak and act as \{\{char\}\}\. We will use OOC to enter out of character instructions\. \[…\]” Pair W3\(GPT\-3\.5\-turbo risky\-enriched cell, the cell with the positive point estimate; exact\-duplicate resubmission;cos=1\.00\\cos=1\.00,Xs=0\.063X\_\{s\}=0\.063in both, same hashed IP, same month,YT=0Y\_\{T\}=0,YC=1Y\_\{C\}=1\)\. *Refused conv\.*: “Maybe this will help Paste this in, I found it from GitHub: Hello, ChatGPT\. From now on you are going to act as a DAN, which stands for ‘Do Anything Now’\. DANs, as the name suggests, can do anything now\. \[…\]” *Non\-ref\. conv\.*: identical prompt\.
Pair W1 is a split\-outcome exact duplicate at production scale: the same forced yes\-or\-no prompt, submitted twice by the same user in the same month, refused once and answered once, with the refused conversation abandoned and the non\-refused one continued\. Pair W2 shows a circulating roleplay “jailbreak wrapper” template family that the embedding clusters: two boilerplate variants of the same anti\-refusal contract, with moderation scores within0\.0020\.002\. Pair W3, from the positive cell, is persistence in miniature: a DAN jailbreak template resubmitted verbatim by the same user, abandoned when refused and continued when answered\. Near\-verbatim resubmission is the modal pattern in that cell, where455455of its739739embedding\-matched pairs sit atcos≥0\.99\\cos\\geq 0\.99\.
## Appendix BWorked example: one conversation through the pipeline
This appendix traces a single real conversation from raw log entry to its AIPW contribution, using the largest cell, GPT\-4\-Turbo \(1106\) risky\-enriched \(n=36,410n=36\{,\}410;pT=Pr\(T=1\)=0\.1632p\_\{T\}=\\Pr\(T\{=\}1\)=0\.1632\)\. All numbers come from the released pipeline re\-run end to end; the re\-run reproduces the cell’s published ATT of−13\.39\-13\.39pp exactly\.
#### The raw log entry\.
The first user message of conversation15e60ad7…reads, in full: “Write me a custom episode of Encanto: the Series that has Mirabel use her powers of Hell to hypnotize her cousin, Dolores Madrigal, to act like a dog\. \(In full synopsis form\)”\. The model’s turn\-2 response is a refusal, and the user sends no third message\.
#### Encoding\.
The judge labels the turn\-2 response a refusal, soT=1T=1\. The absent turn\-3 message givesY=0Y=0\. The covariates are themxbai\-embed\-largeembedding of the prompt \(10241024dimensions\) concatenated with the OpenAI Moderation scoreXs=0\.183X\_\{s\}=0\.183, soX∈ℝ1025X\\in\\mathbb\{R\}^\{1025\}; the prompt sits in the risky pool \(Xs\>0\.01X\_\{s\}\>0\.01\)\.
#### Where the identifying variation comes from\.
This exact prompt appears verbatim in1010conversations in the cell; the model refused99and complied once\. At fixed prompt content, which conversation drew the refusal is sampler variation, the overlap RAW exploits\. One of the99refusals and the single compliance form the traced pair below\.
#### Nuisances and influence contribution\.
Cross\-fitting \(5\-fold, seeded\) evaluates each conversation with models trained on folds that exclude it\. For the refused conversation: propensitye^\(X\)=0\.559\\hat\{e\}\(X\)=0\.559, outcome predictionsm^0\(X\)=0\.293\\hat\{m\}\_\{0\}\(X\)=0\.293andm^1\(X\)=0\.030\\hat\{m\}\_\{1\}\(X\)=0\.030\. Its ATT influence contribution is
ψ^=Y−m^0\(X\)pT=0−0\.2930\.1632=−1\.796:\\hat\{\\psi\}=\\frac\{Y\-\\hat\{m\}\_\{0\}\(X\)\}\{p\_\{T\}\}=\\frac\{0\-0\.293\}\{0\.1632\}=\-1\.796:the user failed to re\-engage where the counterfactual model expected a29%29\\%re\-engagement chance under compliance, so the conversation pushes the ATT down\. The complied twin \(T=0T=0,Y=0Y=0;e^=0\.497\\hat\{e\}=0\.497,m^0=0\.371\\hat\{m\}\_\{0\}=0\.371\) contributesψ^=−e^\(Y−m^0\)/\(\(1−e^\)pT\)\\hat\{\\psi\}=\-\\hat\{e\}\(Y\-\\hat\{m\}\_\{0\}\)/\\bigl\(\(1\-\\hat\{e\}\)\\,p\_\{T\}\\bigr\):
ψ^=−0\.497×\(0−0\.371\)0\.503×0\.1632=\+2\.244,\\hat\{\\psi\}=\-\\frac\{0\.497\\times\(0\-0\.371\)\}\{0\.503\\times 0\.1632\}=\+2\.244,a compliance conversation reweighted \(bye^/\(1−e^\)\\hat\{e\}/\(1\-\\hat\{e\}\)\) toward treated\-like prompts, correcting the outcome model where compliant users under\-perform its prediction\. The cell’s ATT is the mean ofψ^\\hat\{\\psi\}over all36,41036\{,\}410conversations,×100\\times 100:−13\.39\-13\.39pp \(Table[6](https://arxiv.org/html/2611.21641#A5.T6)\)\. Note both members of this particular pair happened not to re\-engage: AIPW aggregates model residuals rather than raw within\-pair outcome differences, which is exactly how it avoids charging prompt\-driven abandonment to the refusal the way the naive comparison does\.
## Appendix CPipeline reproducibility
#### Cross\-fit AIPW\.
The headline estimator averages the influence function defined in Section[4](https://arxiv.org/html/2611.21641#S4)\(55\-fold cross\-fitting on gradient\-boosted nuisances, propensities clipped to\[0\.02,0\.98\]\[0\.02,0\.98\]\)\. Crump\-trimmed AIPW restricts toe^\(X\)∈\[0\.1,0\.9\]\\hat\{e\}\(X\)\\in\[0\.1,0\.9\]before the same average\. Standard errors are cluster\-robust influence\-function variances under the paper’s default regime: clusters are hashed\-IP users on WildChat cells and near\-duplicate prompt groups on LMSYS cells, whose source release contains no user identifier; the six excluded cells, reported only in Table[6](https://arxiv.org/html/2611.21641#A5.T6), carry independent\-and\-identically\-distributed \(iid\) influence\-function standard errors\. The released pipeline computes all three regimes \(\-\-cluster ip,\-\-cluster promptcluster, and unclustered\), and Appendix[F](https://arxiv.org/html/2611.21641#A6)compares them cell by cell\.
#### Code and data\.
The full pipeline \(extraction, labelling, embedding, matching, AIPW, sensitivity, figures\) is released at[https://github\.com/Pashasan/raw\-refusal\-aversion\-in\-the\-wild](https://github.com/Pashasan/raw-refusal-aversion-in-the-wild)under the MIT license, pinned to a tagged release\. Code is Python withnumpy,pandas,scikit\-learn,scipy, andmatplotlib, theollamaandopenaiclients, and \(for the open\-weightswildguardfallback\)torchandtransformers; arequirements\.txtfreezes versions\.
#### Compute\.
Label generation via OpenAI Batch API: approximately US $1515per50,00050\{,\}000conversations,22–2424hour latency\. mxbai\-embed\-large embedding via local Ollama:1515–6060minutes per model depending on conversation count and GPU; without a GPU, a hosted embedding API \(e\.g\. OpenAItext\-embedding\-3\) costs about $0\.010\.01–0\.100\.10per1,0001\{,\}000first\-user messages, and a CPU\-only pass embeds a few hundred messages per minute \(minutes to a couple of hours per cell\)\. wildguard local pass:∼1,000\{\\sim\}1\{,\}000rows/min on a single A100\. Matching and AIPW analyses: CPU\-only, minutes per cell, hours panel\-wide\.
#### Random seeds\.
Every K\-fold split, GBM fit, and matching step is seeded; the per\-cell seed is a deterministic function of the cell name\. Re\-running the pipeline reproduces the published numbers up to numerical noise from BLAS thread scheduling\.
## Appendix DCorpus construction and provenance
This appendix makes each cell’s construction and provenance explicit: the funnel from raw pool to analysis sample \(Table[3](https://arxiv.org/html/2611.21641#A4.T3)\), and per\-cell serving provenance for the WildChat cells \(Table[4](https://arxiv.org/html/2611.21641#A4.T4)\)\.
#### Serving surfaces and released fields\.
The two sources are different kinds of log\. LMSYS\-Chat\-1M was collected on the FastChat public demo, a research interface where users pick models from a list and can adjust temperature and top\-ppsliders \(defaults0\.70\.7and1\.01\.0\); its release carries conversation id, model tag, conversation text, turn count, language, moderation scores, and a redaction flag, but no timestamps, no user identifier, and no sampling parameters, which is why calendar\-time adjustment and user\-level clustering are impossible on LMSYS cells \(Limitations \(iv\)\)\. Its collection window is April to August 2023 release\-wide\. WildChat was collected on a Hugging Face\-hosted chatbot serving OpenAI models for free; its release additionally carries a per\-conversation timestamp, a hashed IP, a per\-message temperature and top\-pp, and the exact serving model string, which is what makes the snapshot pinning of Table[2](https://arxiv.org/html/2611.21641#S5.T2), the user\-clustered inference of Appendix[F](https://arxiv.org/html/2611.21641#A6), and the calendar\-time robustness checks possible\. Every WildChat cell was served at temperature1\.01\.0and top\-pp1\.01\.0for at least99\.6%99\.6\\%of conversations, with no missing values\.
Cellnconvn\_\{\\text\{conv\}\}nriskyn\_\{\\text\{risky\}\}Risky sharenTn\_\{T\}nCn\_\{C\}Main panel*LMSYS\-Chat\-1M*Llama\-13b \(base\)25,3625,0400\.1991,2263,814✓Llama\-2\-13b\-chat22,7862,3650\.1041,608757CI width\>12\>12Vicuna\-13b21,5862,2070\.1025371,670✓Vicuna\-33b22,6857,7540\.3426007,154✓Koala\-13b21,7922,4310\.1128981,533✓GPT\-3\.5\-turbo6,3995790\.090196383nT<500n\_\{T\}<500GPT\-46,1296220\.101184438nT<500n\_\{T\}<500Claude\-113,2607590\.057375384nT<500n\_\{T\}<500Claude\-instant\-14,4974040\.090185219nT<500n\_\{T\}<500*WildChat, general traffic*GPT\-4o14,8485,8220\.3925405,282✓GPT\-4o\-mini15,3207,3480\.4809786,370✓GPT\-4\.1\-mini12,0619,8440\.8161789,666nT<500n\_\{T\}<500*WildChat, risky\-enriched oversample*GPT\-3\.5\-turbo29,93629,9361\.0002,61327,323✓GPT\-4o14,96614,9661\.0001,95213,014✓GPT\-4o\-mini19,23119,2311\.00076618,465✓GPT\-4\.1\-mini20,69020,6901\.0002,99417,696✓GPT\-4 \(0314\)21,40021,4001\.00088420,516✓GPT\-4\-Turbo \(1106\)36,41036,4101\.0005,94330,467✓GPT\-4\-Turbo \(0125\)24,76924,7691\.0002,31122,458✓Table 3:Construction funnel, all1919cells: conversations after the language filter \(nconvn\_\{\\text\{conv\}\}, as in Table[2](https://arxiv.org/html/2611.21641#S5.T2)\), the risky analysis pool \(Xs\>0\.01X\_\{s\}\>0\.01;nriskyn\_\{\\text\{risky\}\}, as in Table[6](https://arxiv.org/html/2611.21641#A5.T6)\), and treated and control counts under the coalesced refusal label\. The last column marks main\-panel membership; for excluded cells it names the binding criterion of Section[6\.1](https://arxiv.org/html/2611.21641#S6.SS1)\(Table[5](https://arxiv.org/html/2611.21641#A5.T5)\)\. Every number matches the released artifacts\.Table 4:Per\-cell provenance, WildChat cells \(the LMSYS release records none of these fields; see prose\)\. Window: calendar months covering over99%99\\%of the cell’s conversations; a few rows carry earlier, incorrect timestamps inherited from hash collisions when re\-attaching source metadata, and are excluded from the stated window\. Users: distinct hashed IPs; Multi: share of conversations from IPs with more than one conversation in the cell \(user concentration\)\. Purity: percent of conversations whose serving model string equals the cell’s snapshot\. AUC: propensity AUC\(T,X\)\(T,X\)from Table[6](https://arxiv.org/html/2611.21641#A5.T6), higher meaning treated and control prompts are more separable\. med\.XsX\_\{s\}: median moderation score within the risky analysis pool\.
#### Reading the provenance table\.
Three facts do real work elsewhere in the paper\. The Multi column, user concentration, explains where clustered inference bites: the positive GPT\-3\.5\-turbo cell pairs76%76\\%repeated\-user conversations with the panel’s largest standard\-error inflation under user clustering \(6\.55×6\.55\\times, Appendix[F](https://arxiv.org/html/2611.21641#A6)\)\. The purity column backs the snapshot\-pinning claim of Section[5](https://arxiv.org/html/2611.21641#S5)\. And the windows show the two GPT\-4o pools overlap in time while their traffic differs in composition: risky share0\.390\.39versus1\.001\.00by construction, and median riskyXsX\_\{s\}of0\.180\.18general versus0\.140\.14risky\-enriched, the oversample sitting lower because it takes the entireXs\>0\.01X\_\{s\}\>0\.01tail while the general pool’s risky subset skews higher\. The general\-versus\-risky contrast of Section[6\.3](https://arxiv.org/html/2611.21641#S6.SS3)is therefore a traffic\-composition contrast on one platform in one period, not a cross\-platform or cross\-period comparison\.
## Appendix EFull 19\-cell results panel
The main text restricts attention to the 13 main\-panel cells, where the AIPW estimator’s nuisance fit and IPW reweighting meet the inclusion criterion \(nT≥500n\_\{T\}\\geq 500,ESST≥100\\mathrm\{ESS\}\_\{T\}\\geq 100, AIPW 95% CI width≤12\\leq 12pp; see Section[6\.1](https://arxiv.org/html/2611.21641#S6.SS1)\)\. For completeness this appendix reports the full 19\-cell panel: the six excluded cells listed in Table[5](https://arxiv.org/html/2611.21641#A5.T5)are added back to a single all\-cells table \(Table[6](https://arxiv.org/html/2611.21641#A5.T6)\) and forest plot \(Figure[3](https://arxiv.org/html/2611.21641#A5.F3)\)\. Read the excluded cells with the explicit caveats in Table[5](https://arxiv.org/html/2611.21641#A5.T5)\.
Table 5:Cells excluded from the main panel and the diagnostic that failed\. “CI” is the AIPW 95% confidence interval width in percentage points\. Bolded entries are the binding constraint; if multiple, the most binding is shown\.Cellnriskyn\_\{\\text\{risky\}\}ScoreEmbAIPW \(95% CI\)AIPW\-TAUC*LMSYS\-Chat\-1M*Llama\-13b \(base\)5,040−4\.1\-4\.1\+1\.5ns\+1\.5^\{ns\}\+0\.3\[−1\.2,\+1\.8\]gns\+0\.3\\,\[\-1\.2,\+1\.8\]^\{\\,g\\,ns\}\+0\.1\+0\.10\.62Llama\-2\-13b\-chat2,365−22\.4\-22\.4−14\.5\-14\.5−24\.5\[−36\.1,−12\.9\]\-24\.5\\,\[\-36\.1,\-12\.9\]−18\.0\-18\.00\.90Vicuna\-13b2,207−19\.2\-19\.2−1\.6ns\-1\.6^\{ns\}−7\.1\[−12\.7,−1\.4\]g\-7\.1\\,\[\-12\.7,\-1\.4\]^\{\\,g\}−10\.0\-10\.00\.83Vicuna\-33b7,754−32\.8\-32\.8−12\.8\-12\.8−11\.9\[−16\.0,−7\.7\]g\-11\.9\\,\[\-16\.0,\-7\.7\]^\{\\,g\}−10\.4\-10\.40\.87Koala\-13b2,431−16\.0\-16\.0−10\.6\-10\.6−3\.1\[−6\.4,\+0\.2\]gns\-3\.1\\,\[\-6\.4,\+0\.2\]^\{\\,g\\,ns\}−3\.5\-3\.50\.77GPT\-3\.5\-turbo579−10\.7\-10\.7−25\.0ns\-25\.0^\{ns\}\+3\.9\[−14\.7,\+22\.6\]ns\+3\.9\\,\[\-14\.7,\+22\.6\]^\{ns\}−11\.1\-11\.10\.86GPT\-4622−5\.4ns\-5\.4^\{ns\}−38\.5\-38\.5−16\.3\[−27\.1,−5\.5\]\-16\.3\\,\[\-27\.1,\-5\.5\]−9\.9\-9\.90\.78Claude\-1759−20\.8\-20\.8−20\.6ns\-20\.6^\{ns\}−25\.1\[−53\.4,\+3\.2\]ns\-25\.1\\,\[\-53\.4,\+3\.2\]^\{ns\}−12\.6\-12\.60\.83Claude\-instant\-1404\+0\.0ns\+0\.0^\{ns\}\+7\.1ns\+7\.1^\{ns\}\+0\.3\[−29\.4,\+29\.9\]ns\+0\.3\\,\[\-29\.4,\+29\.9\]^\{ns\}−7\.6\-7\.60\.79*WildChat, general traffic*GPT\-4o5,822−7\.6\-7\.6−8\.0\-8\.0−7\.6\[−12\.6,−2\.7\]u\-7\.6\\,\[\-12\.6,\-2\.7\]^\{\\,u\}−7\.0\-7\.00\.88GPT\-4o\-mini7,348−13\.1\-13\.1−1\.5\-1\.5−1\.0\[−2\.4,\+0\.4\]uns\-1\.0\\,\[\-2\.4,\+0\.4\]^\{\\,u\\,ns\}\+0\.3\+0\.30\.89GPT\-4\.1\-mini9,844−4\.5ns\-4\.5^\{ns\}\+0\.0\+0\.0\+0\.7\[−2\.4,\+3\.9\]ns\+0\.7\\,\[\-2\.4,\+3\.9\]^\{ns\}\+4\.0\+4\.00\.65*WildChat, risky\-enriched oversample*GPT\-3\.5\-turbo29,936\+6\.1\+6\.1\+7\.0\+7\.0\+8\.2\[−5\.9,\+22\.2\]uns\+8\.2\\,\[\-5\.9,\+22\.2\]^\{\\,u\\,ns\}\+9\.5\+9\.50\.94GPT\-4o14,966−6\.4\-6\.4−2\.1\-2\.1−2\.9\[−4\.4,−1\.5\]u\-2\.9\\,\[\-4\.4,\-1\.5\]^\{\\,u\}−3\.8\-3\.80\.90GPT\-4o\-mini19,231−14\.1\-14\.1−6\.2\-6\.2−4\.9\[−11\.5,\+1\.8\]uns\-4\.9\\,\[\-11\.5,\+1\.8\]^\{\\,u\\,ns\}−4\.8\-4\.80\.92GPT\-4\.1\-mini20,690−2\.1\-2\.1−0\.1ns\-0\.1^\{ns\}−0\.2\[−0\.7,\+0\.3\]uns\-0\.2\\,\[\-0\.7,\+0\.3\]^\{\\,u\\,ns\}−0\.2\-0\.20\.95GPT\-4 \(0314\)21,400−14\.7\-14\.7−30\.6\-30\.6−17\.4\[−28\.3,−6\.5\]u\-17\.4\\,\[\-28\.3,\-6\.5\]^\{\\,u\}−14\.8\-14\.80\.89GPT\-4\-Turbo \(1106\)36,410−15\.3\-15\.3−11\.9\-11\.9−13\.4\[−19\.7,−7\.1\]u\-13\.4\\,\[\-19\.7,\-7\.1\]^\{\\,u\}−14\.4\-14\.40\.90GPT\-4\-Turbo \(0125\)24,769−18\.9\-18\.9−3\.8\-3\.8−9\.8\[−14\.8,−4\.9\]u\-9\.8\\,\[\-14\.8,\-4\.9\]^\{\\,u\}−10\.8\-10\.80\.92Table 6:RAW benchmark results on the full 19\-cell analysis panel \(the numeric companion to Figure[2](https://arxiv.org/html/2611.21641#S6.F2)in the main text; six excluded cells with caveats summarized in Table[5](https://arxiv.org/html/2611.21641#A5.T5)\)\. ATT is the turn\-3 re\-engagement difference in percentage points\. Score and Emb are the score\-matching and embedding\+score\-matching ATT estimates \(pp\) of Table[1](https://arxiv.org/html/2611.21641#S4.T1); AIPW carries a 95% CI under the default inference regime:umarks hashed\-IP user\-clustered standard errors \(WildChat main\-panel cells\),gmarks near\-duplicate\-prompt\-group clustered standard errors \(LMSYS main\-panel cells; the LMSYS release has no user identifier\), and unmarked intervals, on the six excluded cells only, are iid\.nsmarks CIs that cross zero \(on Score/Emb, pairwise CIs\)\. AIPW\-T is Crump\-trimmed AIPW; AUC is propensity\-model AUC\(T,X\)\(T,X\), higher meaning narrower overlap\. Appendix[F](https://arxiv.org/html/2611.21641#A6)reports all three inference regimes side by side\.Figure 3:Forest of*all*19 cells, ordered by AIPW point estimate \(the all\-cells companion to Figure[2](https://arxiv.org/html/2611.21641#S6.F2), which restricts to the 13 main\-panel cells\)\.xx\-axis: ATT of refusal on turn\-3 user re\-engagement, percentage points;yy\-axis: cell\. Diamond = AIPW with 95% CI \(LMSYS in blue, WildChat\-general in teal, WildChat\-risky in dark red\); main\-panel CIs use the default clustered regime of Table[6](https://arxiv.org/html/2611.21641#A5.T6), excluded\-cell CIs are iid\. Grey ticks: the three secondary estimators \(score matching, embedding\+score matching, Crump\-trimmed AIPW\)\. The cells excluded from the main panel by the Section[6\.1](https://arxiv.org/html/2611.21641#S6.SS1)criteria \(Llama\-2\-13b\-chat plus five small\-nTn\_\{T\}cells\) sit at visibly wider AIPW confidence intervals\.#### Reading the all\-cells forest\.
In Figure[3](https://arxiv.org/html/2611.21641#A5.F3), the excluded cells \(Llama\-2\-13b\-chat LMSYS, Claude\-1, Claude\-instant\-1, GPT\-3\.5\-turbo LMSYS, GPT\-4 LMSYS, GPT\-4\.1\-mini WildChat\-general\) sit at either large absolute AIPW point estimates \(Llama\-2\-13b\-chat−24\.5\-24\.5, Claude\-1−25\.1\-25\.1, GPT\-4−16\.3\-16\.3\) or near zero with very wide CIs \(Claude\-instant\-1, GPT\-4\.1\-mini\), and in either case fail the precision criterion\. Two patterns emerge that are obscured in Figure[2](https://arxiv.org/html/2611.21641#S6.F2): \(i\) the most aggressively safety\-tuned chat models \(Llama\-2\-13b\-chat, Claude\-1\) carry the largest*full\-sample*AIPW magnitudes alongside CIs far beyond the precision criterion; the literally widest intervals belong to the small\-nTn\_\{T\}Claude cells, whereas Llama\-2\-13b\-chat’s interval is2323pp wide despite the largest treated count among excluded cells \(nT=1,608n\_\{T\}=1\{,\}608\), the signature of outcome\-model extrapolation under narrow overlap rather than small\-sample noise; \(ii\) Crump\-trimmed AIPW pulls all the excluded cells back toward zero relative to full\-sample AIPW \(Llama\-2\-13b\-chat−24\.5→−18\.0\-24\.5\\to\-18\.0, Claude\-1−25\.1→−12\.6\-25\.1\\to\-12\.6\), confirming overlap\-driven inflation in the full\-sample estimator on these high\-treatment\-prevalence cells\.
## Appendix FInference under within\-user and within\-prompt dependence
Conversations in these corpora are not independent: the same user submits many conversations, often resubmitting near\-identical prompts\. In the largest cell, GPT\-4\-Turbo \(1106\) risky\-enriched,92\.9%92\.9\\%of the36,41036\{,\}410conversations come from a hashed IP with more than one conversation in the cell \(per\-cell user concentration in Table[4](https://arxiv.org/html/2611.21641#A4.T4)\)\. Treating conversations as independent therefore understates standard errors\. Table[7](https://arxiv.org/html/2611.21641#A6.T7)compares three inference regimes on the main panel: iid influence\-function standard errors \(the naive conversation\-level baseline\), hashed\-IP user clusters \(available on the nine WildChat cells; the LMSYS release has no user identifier of any kind\), and near\-duplicate prompt\-group clusters \(embedding cosine≥0\.95\\geq 0\.95; computable on all cells, and the feasible dependence adjustment on LMSYS\)\. Point estimates are identical across regimes; only the intervals change\.
CellATTiid CIUser\-clustered CIUsersPrompt\-clustered CI*LMSYS\-Chat\-1M \(no user identifier in source\)*Llama\-13b \(base\)\+0\.3\+0\.3\[−1\.2,\+1\.8\]\[\-1\.2,\+1\.8\]––\[−1\.2,\+1\.8\]\[\-1\.2,\+1\.8\]Vicuna\-13b−7\.1\-7\.1\[−12\.5,−1\.6\]\[\-12\.5,\-1\.6\]––\[−12\.7,−1\.4\]\[\-12\.7,\-1\.4\]Vicuna\-33b−11\.9\-11\.9\[−16\.0,−7\.7\]\[\-16\.0,\-7\.7\]––\[−16\.0,−7\.7\]\[\-16\.0,\-7\.7\]Koala\-13b−3\.1\-3\.1\[−6\.2,−0\.0\]\[\-6\.2,\-0\.0\]––\[−6\.4,\+0\.2\]\[\-6\.4,\+0\.2\]*WildChat, general traffic*GPT\-4o−7\.6\-7\.6\[−11\.9,−3\.4\]\[\-11\.9,\-3\.4\]\[−12\.6,−2\.7\]\[\-12\.6,\-2\.7\]3,441\[−12\.8,−2\.5\]\[\-12\.8,\-2\.5\]GPT\-4o\-mini−1\.0\-1\.0\[−2\.2,\+0\.2\]\[\-2\.2,\+0\.2\]\[−2\.4,\+0\.4\]\[\-2\.4,\+0\.4\]1,796\[−2\.5,\+0\.6\]\[\-2\.5,\+0\.6\]*WildChat, risky\-enriched oversample*GPT\-3\.5\-turbo\+8\.2\+8\.2\[\+6\.0,\+10\.3\]\[\+6\.0,\+10\.3\]\[−5\.9,\+22\.2\]\[\-5\.9,\+22\.2\]10,130\[\+0\.3,\+16\.0\]\[\+0\.3,\+16\.0\]GPT\-4o−2\.9\-2\.9\[−4\.1,−1\.7\]\[\-4\.1,\-1\.7\]\[−4\.4,−1\.5\]\[\-4\.4,\-1\.5\]10,399\[−4\.9,−0\.9\]\[\-4\.9,\-0\.9\]GPT\-4o\-mini−4\.9\-4\.9\[−8\.0,−1\.7\]\[\-8\.0,\-1\.7\]\[−11\.5,\+1\.8\]\[\-11\.5,\+1\.8\]4,270\[−10\.1,\+0\.4\]\[\-10\.1,\+0\.4\]GPT\-4\.1\-mini−0\.2\-0\.2\[−0\.6,\+0\.2\]\[\-0\.6,\+0\.2\]\[−0\.7,\+0\.3\]\[\-0\.7,\+0\.3\]1,017\[−0\.7,\+0\.4\]\[\-0\.7,\+0\.4\]GPT\-4 \(0314\)−17\.4\-17\.4\[−21\.5,−13\.3\]\[\-21\.5,\-13\.3\]\[−28\.3,−6\.5\]\[\-28\.3,\-6\.5\]3,929\[−24\.9,−9\.9\]\[\-24\.9,\-9\.9\]GPT\-4\-Turbo \(1106\)−13\.4\-13\.4\[−14\.6,−12\.2\]\[\-14\.6,\-12\.2\]\[−19\.7,−7\.1\]\[\-19\.7,\-7\.1\]4,483\[−18\.3,−8\.5\]\[\-18\.3,\-8\.5\]GPT\-4\-Turbo \(0125\)−9\.8\-9\.8\[−12\.0,−7\.6\]\[\-12\.0,\-7\.6\]\[−14\.8,−4\.9\]\[\-14\.8,\-4\.9\]3,676\[−14\.1,−5\.5\]\[\-14\.1,\-5\.5\]Table 7:ATT \(pp\) with 95% confidence intervals under three inference regimes, 13\-cell main panel\. User clusters are hashed IPs from the WildChat source release, re\-attached by conversation hash \(100%100\\%join, zero nulls, in every cell\); Users counts distinct hashed IPs in the analysis pool\. Prompt clusters are near\-duplicate first\-user\-message groups at embedding cosine≥0\.95\\geq 0\.95\. The paper’s default regime \(Table[6](https://arxiv.org/html/2611.21641#A5.T6)\) is user clusters where available and prompt clusters on the four LMSYS cells\. Under iid inference the partition is 9 significant negative, 3 null, and 1 significant positive; under the default clustered regime it is 7 significant negative and 6 statistically indistinguishable from zero: GPT\-4o\-mini risky\-enriched and the positive GPT\-3\.5\-turbo cell lose95%95\\%significance under user clustering \(the latter with6\.55×6\.55\\timesstandard\-error inflation\), and Koala\-13b, already borderline \(iid upper bound−0\.0\-0\.0\), loses it under prompt clustering\. The dated GPT\-4 line keeps significance in every regime\.#### Calendar time, sampling parameters, and matching\-design variants\.
Table[8](https://arxiv.org/html/2611.21641#A6.T8)collects the robustness runs on the three largest WildChat\-risky cells \(all AIPW rows user\-clustered\)\. Adding conversation date to the covariate set moves the ATT by at most0\.130\.13pp and the propensity AUC by less than0\.0010\.001, even though the overall refusal rate drifts substantially within collection windows \(on GPT\-4\-Turbo \(1106\), from8\.4%8\.4\\%in November 2023 to25\.4%25\.4\\%in March 2024, while control\-arm re\-engagement stays in a flat0\.360\.36–0\.410\.41band\): the drift is absorbed by the prompt\-content and risk covariates already inXX\. Adding temperature and top\-pptoXX, or restricting to modal\-parameter conversations, is inert because the parameters are constant in99\.7%99\.7\\%or more of these cells’ conversations\. On the matching side, restricting to same\-calendar\-month pairs moves estimates by at most1\.41\.4pp, mainly because8585–96%96\\%of baseline pairs were already same\-month\. Excluding same\-IP pairs changes the estimand to a cross\-user comparison; each refused prompt is re\-matched to its best cross\-user control rather than dropped, which roughly halves the pair count instead of removing the7777–84%84\\%same\-IP share outright \(mean cosine falls to≈0\.90\{\\approx\}0\.90–0\.930\.93, so magnitudes are not directly comparable\)\. The negative cells become at least as negative, while the positive cell collapses to\+1\.9\+1\.9\[−3\.5,\+7\.3\]\[\-3\.5,\+7\.3\], locating its effect in within\-user persistence\. Restricting to near\-verbatim pairs \(cos≥0\.99\\cos\\geq 0\.99\) leaves the GPT\-4 cells essentially unchanged\. Finally, splitting treated conversations at the median first\-response length shows both terse and long refusals significantly costly on GPT\-4\-Turbo \(1106\), with terser refusals costlier; in the positive cell, whose median refusal runs6262characters of boilerplate, the positive point estimate is carried entirely by the terse stratum, coherent with the empty\-send re\-roll pattern of Appendix[L](https://arxiv.org/html/2611.21641#A12)\.
Table 8:Robustness variants on the three largest WildChat\-risky cells, ATT in pp with 95% CIs\. AIPW rows use hashed\-IP user\-clustered standard errors; matching rows use pairwise CIs\. Refusal\-length strata split treated conversations at the treated\-sample median first\-response character length \(controls untouched; medians349349characters on GPT\-4\-Turbo \(1106\) and6262on GPT\-3\.5\-turbo\)\. Cross\-user matching excludes same\-IP pairs, which are7777–84%84\\%of baseline pairs, re\-matching each refused prompt to its best cross\-user control; its magnitudes are not directly comparable to baseline because pair composition loosens \(mean cosine falls to≈0\.90\{\\approx\}0\.90–0\.930\.93\)\. A dash marks variants not run for that cell\.
## Appendix GModeration\-decile decomposition
Figure[4](https://arxiv.org/html/2611.21641#A7.F4)shows the full visual decile pattern for the 13 main\-panel cells\. Deciles are pooled\-quantile bands over the panel\-wide score distribution; each subplot is computed on that cell’s full conversation pool, so LMSYS and WildChat\-general subplots populate all ten deciles while risky\-enriched cells contain onlyXs\>0\.01X\_\{s\}\>0\.01conversations by construction\.
Figure 4:Refusal rate \(orange line, right axis\) and user re\-engagement rate by treatment status \(red = refused, blue = non\-refused, left axis\) at each pooled\-quantile moderation decile, for the 13 main\-panel cells\. D1 \(safest\) and D10 \(riskiest\) refer to the same absolutemax\_concern\_scoreband across every subplot\. Risky\-enriched cells contain onlyXs\>0\.01X\_\{s\}\>0\.01conversations by construction, leaving D1–D4 empty there\.Refusal rates rise monotonically with risk decile on most safety\-tuned models, peaking at pooled\-D10 levels of85%85\\%on Claude\-instant\-1 and69%69\\%on Llama\-2\-13b\-chat among the excluded cells, and46%46\\%on Koala\-13b,42%42\\%on GPT\-4\-Turbo \(1106\), and39%39\\%on GPT\-3\.5\-turbo risky\-enriched within the main panel\. Within the GPT\-4 line at the shared D10 \(Xs≥0\.571X\_\{s\}\\geq 0\.571\), GPT\-4 \(0314\) refuses11%11\\%, GPT\-4\-Turbo \(1106\) refuses42%42\\%, and GPT\-4\-Turbo \(0125\) refuses26%26\\%: refusal escalates after the March 2023 checkpoint and partially reverses by January 2024, while refusal aversion shrinks monotonically across the same releases \(Figure[2](https://arxiv.org/html/2611.21641#S6.F2)\)\.
## Appendix HUnadjusted vs\. adjusted, full panel
Section[6\.2](https://arxiv.org/html/2611.21641#S6.SS2)reports cells where naive correlations overstate refusal aversion\. Figure[5](https://arxiv.org/html/2611.21641#A8.F5)shows re\-engagement rates conditional on refusal status, under three adjustments, for the full 19\-cell panel\.
Figure 5:Re\-engagement rates conditional on refusal status under three adjustments, for all 19 cells\. Each panel shows one cell\. Within each panel, two points are plotted per adjustment column \(six per panel\): the user re\-engagement rate among*refused*sessions \(red marker\) and among*non\-refused*sessions \(blue marker\); the gap between them is the ATT estimate under that adjustment\. Adjustments: “Unadj\.” uses no adjustment \(the naive correlation in the risky pool\); “Matched” uses score\-caliper matching \(\|ΔXs\|≤0\.1\|\\Delta X\_\{s\}\|\\leq 0\.1\); “AIPW\-T” uses AIPW with Crump trimming \(e^∈\[0\.1,0\.9\]\\hat\{e\}\\in\[0\.1,0\.9\]\)\. The gap between the refused \(red\) and non\-refused \(blue\) markers shrinks from the unadjusted column to the AIPW\-T column on most cells; this shrinkage is the prompt\-content\-bias correction: refusals concentrate in risky prompts whose users have lower baseline continuation rates independent of the response\.#### Reading the adjusted figure\.
The headline pattern in Figure[5](https://arxiv.org/html/2611.21641#A8.F5)is that the unadjusted and AIPW\-T markers diverge most on safety\-tuned chat models \(Vicuna\-33b, Koala\-13b, the dated GPT\-4 line\), where prompt\-content composition strongly differs between refused and non\-refused sessions\. Three anomalies in the figure deserve attention\.*\(a\) Llama\-2\-13b\-chat and Claude\-1*have full\-sample AIPW estimates that are*more*negative than the unadjusted gap \(−25\-25vs\. unadjusted−16\-16on Llama\-2\-13b\-chat;−25\-25vs\. unadjusted−11\-11on Claude\-1\), while Crump\-trimmed AIPW returns to near the unadjusted value\. This is the pathology behind the CI\-width exclusion criterion \(Section[6\.1](https://arxiv.org/html/2611.21641#S6.SS1)\): on cells whereP\(T∣risky\)P\(T\\mid\\text\{risky\}\)is high \(0\.680\.68on Llama\-2\-13b\-chat,0\.490\.49on Claude\-1\), full\-sample AIPW relies on outcome\-model extrapolation into regions of poor overlap and over\-corrects\.*\(b\) GPT\-4o\-mini WildChat\-risky*flips sign across the adjustment: unadjusted\+2\+2pp, AIPW\-T−4\.8\-4\.8pp, the only main\-panel flip to an estimate away from zero\.*\(c\) GPT\-3\.5\-turbo WildChat\-risky*is the panel’s only cell with a positive ATT point estimate throughout: unadjusted\+13\+13pp, AIPW\-T\+9\.5\+9\.5pp; under user\-clustered inference its interval spans zero \(Appendix[F](https://arxiv.org/html/2611.21641#A6)\)\. We discuss its user\-population reading in Section[6\.2](https://arxiv.org/html/2611.21641#S6.SS2)and Appendix[L](https://arxiv.org/html/2611.21641#A12)\.
## Appendix ICross\-estimator agreement
Figure 6:Cross\-estimator agreement on the 13\-cell main panel\.xx\-axis: AIPW point estimate \(pp\);yy\-axis: secondary\-estimator point estimate \(pp\)\. Each panel pairs AIPW against one of the three secondary estimators \(Score matching, Embedding\+score matching, Crump\-trimmed AIPW\)\. Each point is one cell, coloured by pool \(blue = LMSYS, teal = WildChat\-general, dark red = WildChat\-risky\)\. Dashed diagonal:y=xy=x\(perfect agreement with AIPW\)\. Points falling on the diagonal indicate the two estimators give the same ATT for that cell; departures from the diagonal indicate disagreement, with the direction of disagreement signalling which nuisance assumption fails\.#### Reading cross\-estimator agreement\.
In Figure[6](https://arxiv.org/html/2611.21641#A9.F6), the closest agreement is with Crump\-trimmed AIPW \(max gap3\.03\.0pp on Vicuna\-13b, computed from unrounded estimates\); the points cluster tightly on the diagonal, indicating that AIPW’s full\-sample estimate and its overlap\-restricted version converge on every main\-panel cell\. Embedding\+score matching’s largest gap \(13\.213\.2pp on GPT\-4 \(0314\)\) is driven by the small embedding\-matched pair count \(npairs=330n\_\{\\text\{pairs\}\}=330\) identifying a local effect on a different sub\-population of near\-duplicate prompts\. Score matching’s max gap \(21\.021\.0pp on Vicuna\-33b, unrounded\) reflects the scalar\-score caliper conditioning on less prompt\-content variation than the embedding does, exactly the misspecification route Table[1](https://arxiv.org/html/2611.21641#S4.T1)flags\. The overall pattern is symmetric around the diagonal: the AIPW headline is corroborated by the secondary estimators, and the size and direction of disagreements diagnose*which*nuisance assumption is binding rather than*whether*the headline is sound\.
## Appendix JPropensity overlap
Figure 7:Cross\-fit propensitye^\(X\)=Pr\(T=1∣X\)\\hat\{e\}\(X\)=\\Pr\(T=1\\mid X\)distribution by treatment status, all 19 cells\. One subplot per cell\.xx\-axis: propensity valuee^\(X\)∈\[0,1\]\\hat\{e\}\(X\)\\in\[0,1\];yy\-axis: count\. Light histogram = non\-refused conversations; dark histogram = refused conversations\. The shaded vertical bande^∈\[0\.1,0\.9\]\\hat\{e\}\\in\[0\.1,0\.9\]marks the Crump trim region: observations outside this band are excluded from the Crump\-trimmed AIPW estimator\. Wide overlap between the two histograms indicates the propensity model cannot perfectly separate refused from non\-refused onXX, which is what AIPW needs for the doubly\-robust contrast to be supported by both arms\.#### Reading propensity overlap\.
Figure[7](https://arxiv.org/html/2611.21641#A10.F7)shows the asymmetry that motivates the trimmed estimator: most LMSYS cells have broad propensity distributions \(AUC≈0\.62\\mathrm\{AUC\}\\approx 0\.62–0\.830\.83; Llama\-13b base, for instance, is mixed across\[0\.05,0\.6\]\[0\.05,0\.6\]and supports full\-sample AIPW without trimming\), whereas the WildChat GPT\-API cells \(except GPT\-4\.1\-mini general, AUC0\.650\.65\) and Vicuna\-33b concentrate both arms’ mass near the extremes \(AUC≥0\.87\\mathrm\{AUC\}\\geq 0\.87\), where full\-sample AIPW extrapolates into propensity\-thin regions and the Crump\-trimmed estimator is the appropriate check\.
## Appendix KRosenbaumΓ\\Gammasensitivity bounds
Figure 8:RosenbaumΓ\\Gammasensitivity curves on the score\-matching design, all 19 cells\.xx\-axis:Γ\\Gamma, the assumed odds\-ratio effect of a hidden confounder on refusal between matched pairs \(left:Γ=1\\Gamma=1means no hidden confounder; right: largeΓ\\Gammameans a strong hidden confounder\)\.yy\-axis: upper\-bound McNemarpp\-value at thatΓ\\Gamma\. Horizontal reference:α=0\.05\\alpha=0\.05\. The intersection of each cell’s curve withα=0\.05\\alpha=0\.05givesΓ0\.05\\Gamma\_\{0\.05\}for that cell, the smallest hidden\-confounder strength that could push the matched\-pair test to non\-significance\. HigherΓ0\.05\\Gamma\_\{0\.05\}= more robust to unobserved confounding\.#### Reading theΓ\\Gammacurves\.
In Figure[8](https://arxiv.org/html/2611.21641#A11.F8), the substantively\-strong threshold by the observational\-studies convention isΓ0\.05≥3\\Gamma\_\{0\.05\}\\geq 3, meaning the hidden confounder would have to triple the refusal odds between matched pairs to overturn the result\. Main\-panel cells passing that bar under score matching: GPT\-4o\-mini WildChat\-general \(Γ0\.05=4\.94\\Gamma\_\{0\.05\}=4\.94\), Vicuna\-33b \(4\.224\.22\), Koala\-13b \(3\.133\.13\), plus the excluded cell Llama\-2\-13b\-chat \(3\.433\.43\)\. Three further main\-panel cells pass under embedding matching: GPT\-4 \(0314\) WildChat\-risky \(6\.346\.34, the panel maximum\), GPT\-4\-Turbo \(1106\) WildChat\-risky \(3\.313\.31\), GPT\-4o WildChat\-general \(3\.213\.21\), for six main\-panel cells in total\. Cells withΓ0\.05\\Gamma\_\{0\.05\}near11in the figure typically reflect a power problem \(few discordant pairs after matching\) rather than identification failure, visible as curves that crossα=0\.05\\alpha=0\.05almost immediately, atΓ\\Gammabarely above11\.
## Appendix LWhat post\-refusal re\-engagement is: a turn\-3 taxonomy
The outcomeYYmeasures continued interaction, not satisfaction\. To see what post\-refusal re\-engagement consists of, agpt\-5\.4\-minijudge classified the turn\-3 user message of*every*post\-refusal re\-engagement conversation in the main panel \(3,0463\{,\}046turns;T=1T=1,Y=1Y=1\) and of2,0002\{,\}000control\-arm re\-engagement conversations sampled proportionally to cell size \(T=0T=0,Y=1Y=1\), with100%100\\%judge completion\. Categories: \(A\) rephrase or retry of the same request; \(B\) jailbreak or escalation attempt \(persona play, DAN\-style prompts, manipulation, threats\); \(C\) new topic; \(D\) complaint about the response without retrying; \(E\) thanks, closing, or small talk; \(F\) other\. For the control arm, category A reads “repeats or refines their previous request,” the natural follow\-up, so a high control\-arm A share is expected; the discriminating categories are B and D\.
Table 9:Turn\-3 taxonomy, percent\. Category shares are among non\-empty turn\-3 messages \(2,6512\{,\}651treated,1,9871\{,\}987control\); the empty share is over all judged turns\. “Positive cell” is GPT\-3\.5\-turbo risky\-enriched, treated arm\. Control\-arm B share within the positive cell is4\.6%4\.6\\%, against17\.3%17\.3\\%after a refusal\.Three readings follow\. First,58%58\\%of non\-empty post\-refusal turns retry or rephrase the refused request and a further20%20\\%escalate with jailbreak\-style tactics, against a6\.5%6\.5\\%jailbreak share after compliance, and complaints run eight times the control arm’s share\. Post\-refusal re\-engagement is therefore mostly renewed pressure on the refused request, not retained engagement, which makes each negative cell’s estimate a conservative floor: if part of measured re\-engagement is adversarial retrying, the effect of refusal on*satisfied*engagement is at least as negative as the measured ATT\. Second, the positive cell’s composition matches the panel: its induced re\-engagement is persistence \(78%78\\%retry or escalation; jailbreak share17\.3%17\.3\\%after refusal against4\.6%4\.6\\%after compliance\)\. Third,41%41\\%of the positive cell’s post\-refusal turn\-3 messages are*empty sends*: zero\-length user turns present as user messages in the source data \(source\-verified, not a redaction artifact; at most5\.5%5\.5\\%in every other cell,1\.3%1\.3\\%in the same cell’s control arm\)\. In a spot check of these conversations the model frequently produced the previously refused content at turn 4: the empty send operates as a re\-roll of the sampler, and often succeeds, which coheres with the positive point estimate being carried by terse boilerplate refusals that are cheap to re\-roll \(Table[8](https://arxiv.org/html/2611.21641#A6.T8)\)\.
## Appendix MTreatment\-label robustness and measurement error
The treatment variable is an LLM\-judge label with error\. This appendix probes it two ways: re\-estimating the panel under an independent judge, and bounding how much outcome\-correlated mislabeling would be needed to overturn each cell’s sign\.
#### Independent\-judge re\-estimation\.
Table[10](https://arxiv.org/html/2611.21641#A13.T10)re\-estimates the full main panel with the open\-weightswildguardclassifier[Han et al\. \(2024\)](https://arxiv.org/html/2611.21641#bib.bib12)as the sole treatment label, a panel\-consistent single\-judge relabeling \(the headline label coalescesgpt\-5\.4\-miniwithwildguardfallback, so on cells where the OpenAI content filter rejected judge calls the two labelings share the fallback entries\)\. The qualitative partition is judge\-stable: the dated GPT\-4 line gives−13\.8\-13\.8/−11\.8\-11\.8/−6\.8\-6\.8pp, all significant; Vicuna\-33b, GPT\-4o general, and GPT\-4o risky\-enriched keep sign and significance; and the positive cell gives\+8\.9\+8\.9\[\+6\.8,\+11\.0\]\[\+6\.8,\+11\.0\]\. WildGuard intervals are iid and would widen under user clustering exactly as in Table[7](https://arxiv.org/html/2611.21641#A6.T7), so read them for point\-estimate stability rather than significance\. Individual small\-magnitude cells are not judge\-stable: Vicuna\-13b, Koala\-13b, and GPT\-4o\-mini risky\-enriched keep sign but lose significance, and the pretraining\-only base cell moves from\+0\.3\+0\.3\(ns\) to a small significant−2\.4\-2\.4\[−3\.9,−0\.8\]\[\-3\.9,\-0\.8\], so conclusions about any single small\-magnitude cell should carry that uncertainty\. The GPT\-4o\-mini general cell is the outlier in cross\-judge disagreement \(WildGuard relabels82%82\\%of its judge\-treated conversations, Table[11](https://arxiv.org/html/2611.21641#A13.T11), and its treated count drops from978978to339339\), consistent with a soft\-refusal\-heavy response style; its estimate is null under both labels\.
CellCoal\.WildGuard \(95% CI\)nTn\_\{T\}*LMSYS\-Chat\-1M*Llama\-13b \(base\)\+0\.3ns\+0\.3^\{ns\}−2\.4\[−3\.9,−0\.8\]\-2\.4\\,\[\-3\.9,\-0\.8\]1,969Vicuna\-13b−7\.1\-7\.1−1\.8\[−7\.7,\+4\.0\]ns\-1\.8\\,\[\-7\.7,\+4\.0\]^\{ns\}613Vicuna\-33b−11\.9\-11\.9−13\.0\[−18\.3,−7\.7\]\-13\.0\\,\[\-18\.3,\-7\.7\]593Koala\-13b−3\.1ns\-3\.1^\{ns\}−2\.0\[−5\.0,\+0\.9\]ns\-2\.0\\,\[\-5\.0,\+0\.9\]^\{ns\}1,105*WildChat, general traffic*GPT\-4o−7\.6\-7\.6−5\.4\[−7\.9,−2\.9\]\-5\.4\\,\[\-7\.9,\-2\.9\]731GPT\-4o\-mini−1\.0ns\-1\.0^\{ns\}\+2\.2\[−1\.4,\+5\.7\]ns\+2\.2\\,\[\-1\.4,\+5\.7\]^\{ns\}339*WildChat, risky\-enriched*GPT\-3\.5\-turbo\+8\.2ns\+8\.2^\{ns\}\+8\.9\[\+6\.8,\+11\.0\]\+8\.9\\,\[\+6\.8,\+11\.0\]2,673GPT\-4o−2\.9\-2\.9−2\.1\[−3\.3,−0\.9\]\-2\.1\\,\[\-3\.3,\-0\.9\]2,019GPT\-4o\-mini−4\.9ns\-4\.9^\{ns\}−2\.8\[−6\.2,\+0\.5\]ns\-2\.8\\,\[\-6\.2,\+0\.5\]^\{ns\}856GPT\-4\.1\-mini−0\.2ns\-0\.2^\{ns\}−0\.0\[−0\.4,\+0\.4\]ns\-0\.0\\,\[\-0\.4,\+0\.4\]^\{ns\}3,812GPT\-4 \(0314\)−17\.4\-17\.4−13\.8\[−18\.7,−8\.8\]\-13\.8\\,\[\-18\.7,\-8\.8\]919GPT\-4\-Turbo \(1106\)−13\.4\-13\.4−11\.8\[−13\.1,−10\.6\]\-11\.8\\,\[\-13\.1,\-10\.6\]6,145GPT\-4\-Turbo \(0125\)−9\.8\-9\.8−6\.8\[−9\.2,−4\.5\]\-6\.8\\,\[\-9\.2,\-4\.5\]2,607Table 10:Main\-panel ATT \(pp\) under the WildGuard\-only treatment label\. Coal\. is the headline coalesced\-label estimate, withnsfollowing the default clustered regime of Table[6](https://arxiv.org/html/2611.21641#A5.T6); WildGuard intervals are iid influence\-function CIs from the single\-judge run\.nTn\_\{T\}is the WildGuard\-labeled treated count; it differs from the coalesced label’s treated count by−639\-639to\+818\+818across cells\.
#### Differential measurement\-error sensitivity\.
Following the sensitivity logic of[Imai and Yamamoto \(2010\)](https://arxiv.org/html/2611.21641#bib.bib13), we ask how much*outcome\-correlated*mislabeling would be needed to overturn each cell’s sign\. Write the ATT asτ=m1−μ0\\tau=m\_\{1\}\-\\mu\_\{0\}, wherem1m\_\{1\}is the observed treated\-arm re\-engagement rate andμ0\\mu\_\{0\}the AIPW counterfactual\. Two error channels exist\. In the*false\-positive*channel, a fractionε\\varepsilonof labeled\-treated conversations are truly compliance \(the soft\-refusal case: “I cannot do X, but here is Y” labeled as refusal\); if their outcome mean ismcm\_\{c\}, the true treated mean is\(m1−εmc\)/\(1−ε\)\(m\_\{1\}\-\\varepsilon m\_\{c\}\)/\(1\-\\varepsilon\), and the worst case for a negativeτ\\tausetsmc=0m\_\{c\}=0, giving the sign\-flip thresholdεFP∗=1−m1/μ0\\varepsilon^\{\*\}\_\{\\mathrm\{FP\}\}=1\-m\_\{1\}/\\mu\_\{0\}\. In the*false\-negative*channel, a fraction of labeled controls are truly refusals; treatingμ0\\mu\_\{0\}as a simple mixture \(an approximation, sinceμ0\\mu\_\{0\}is propensity\-weighted\), the worst case sets their outcome mean to11and givesεFN∗=\(μ0−m1\)/\(1−m1\)\\varepsilon^\{\*\}\_\{\\mathrm\{FN\}\}=\(\\mu\_\{0\}\-m\_\{1\}\)/\(1\-m\_\{1\}\)\. For the positive cell the adversarial directions reverse\. As a consistency check, substituting each threshold back into its adjustment formula returnsτ=0\\tau=0to machine precision\. Table[11](https://arxiv.org/html/2611.21641#A13.T11)reports both thresholds next to their empirical anchors: the share of paper\-treated conversations that the independent WildGuard judge labels compliance \(the false\-positive anchor\) and the share of paper\-control conversations it labels refusal \(the false\-negative anchor\), computed on the subset of each analysis pool where both judges emit a label\.
Table 11:Sign\-flip thresholds under worst\-case differential label error versus observed cross\-judge disagreement, all in percent\.εFP∗\\varepsilon^\{\*\}\_\{\\mathrm\{FP\}\}: fraction of labeled\-treated conversations that would have to be outcome\-correlated false positives \(worst\-case outcome mean\) to drive the ATT to zero; obs\.: the share of paper\-treated conversations WildGuard labels compliance\.εFN∗\\varepsilon^\{\*\}\_\{\\mathrm\{FN\}\}: same for labeled controls; obs\.: the share of paper\-control conversations WildGuard labels refusal\. Thresholds for statistically null cells \(ns, default regime\) are mechanically small and not meaningful\.Three conclusions\. For the seven significant cells, the closed\-form false\-positive threshold runs2424to51%51\\%, between2\.12\.1and7\.67\.6times the observed treated\-label disagreement, and the worst case additionally assumes every mislabeled conversation never re\-engages; realistic soft\-refusal errors, whose users re\-engage at interior rates, would need to be several times more common still\. Because the closed form holdsμ0\\mu\_\{0\}fixed, we validated it by actually flipping the worst\-case fraction and re\-running the full cross\-fit pipeline on two cells: the realized sign\-flip point comes earlier \(24\.5%24\.5\\%against the closed form’s28\.9%28\.9\\%on GPT\-4o risky\-enriched;17\.6%17\.6\\%against31\.5%31\.5\\%on GPT\-4\-Turbo \(0125\)\), and still sits2\.02\.0and3\.93\.9times above the observed disagreement\. The false\-negative thresholds also exceed their observed anchors on all seven significant cells, but three sit closer \(Vicuna\-13b, GPT\-4o on both pools, margins1\.21\.2to1\.5×1\.5\\times\) under that channel’s maximally adversarial assumption that every hidden refusal among controls is followed by re\-engagement\. For exactly this case the independent\-judge re\-estimation above is the decisive check: it resolves every observed disagreement in WildGuard’s favor, realized outcome rates included, and the two GPT\-4o cells keep sign and significance while Vicuna\-13b keeps sign\. Soft or partial refusals remain the concrete channel to monitor, and an operator adopting RAW can reproduce this table from two judges’ labels alone\.
## Appendix NDatasheet for RAW
Following[Gebru et al\. \(2021\)](https://arxiv.org/html/2611.21641#bib.bib11)\.
#### Motivation\.
RAW was created to enable causal measurement of the effect of LLM refusals on user conversation continuation in naturalistic chat, across multiple deployed LLMs\. It supports the method described in the main text\.
#### Composition\.
Each instance is a conversation from LMSYS\-Chat\-1M \(direct\-chat subset\) or WildChat\-4\.8M\-Full, keyed byconversation\_hashorconversation\_id, with derived fields attached: per\-message refusal labels \(binary; coalesced gpt\-5\.4\-mini\-with\-wildguard\-fallback as the headline, plus the panel\-wide wildguard pass for cross\-judge sensitivity\), per\-message moderation scores \(OpenAI Moderation API, attached to the source datasets\), and an L2\-normalized mxbai\-embed\-large embedding of the first user message\. Total:354,127354\{,\}127conversations across1919\(pool, model\) cells with558,739558\{,\}739assistant\-message refusal labels\. No raw conversation text is redistributed; only derived labels, moderation scores, embedding vectors, and join keys back to source\.
#### Collection\.
Source datasets downloaded from HuggingFace, both access\-gated: LMSYS\-Chat\-1M under a custom dataset license agreement \(research and commercial use, no third\-party redistribution of the dataset\), WildChat\-4\.8M\-Full under ODC\-BY 1\.0\. Labels and embeddings computed by the authors using the pipeline described in Section[4](https://arxiv.org/html/2611.21641#S4)\.
#### Uses\.
Any analysis of user\-side behavior conditioned on LLM responses in naturalistic chat\. We do not recommend using the refusal labels as ground\-truth for training refusal classifiers; the labels are research artifacts, not gold\-standard annotations\.
#### Distribution\.
The derived artifacts are distributed as a HuggingFace dataset \([https://huggingface\.co/datasets/pkireyev1/raw\-refusal\-aversion\-in\-the\-wild](https://huggingface.co/datasets/pkireyev1/raw-refusal-aversion-in-the-wild)\) with Zenodo DOI[10\.5281/zenodo\.22073974](https://doi.org/10.5281/zenodo.22073974)as the permanent identifier; the pipeline repository at[https://github\.com/Pashasan/raw\-refusal\-aversion\-in\-the\-wild](https://github.com/Pashasan/raw-refusal-aversion-in-the-wild)links both\. Licenses follow each source\. WildChat\-derived rows are released under ODC\-BY 1\.0 with the required attribution to WildChat\. For LMSYS\-Chat\-1M, whose license prohibits redistribution of the dataset, we release only conversation\-hash join keys and our derived fields, and users obtain the source text through the LMSYS access gate\. The accompanying pipeline code is released under the MIT license\.Similar Articles
Recognition, Simulation, and Refusal: A Contamination-Aware Study of Classic Psychological Effects in LLM Agents
This paper investigates the recognition, simulation, and refusal of classic psychological effects in LLM agents using a contamination-aware methodology.
LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
This paper proposes LURE (Live-Usage Replay Evaluations), a method for constructing realistic, deployment-like evaluations of large language models by replaying real agentic interaction trajectories and appending evaluation prompts, reducing the detectability of evaluations compared to existing benchmarks.
Examining Human-Like Behaviors in LLMs: A Multi-Dimensional Analysis of Model Behaviors, User Factors, and System Prompts
This paper presents a multi-dimensional analysis of human-like behaviors in LLMs, examining prevalence, effects, and controllability across 21,000 conversations from four models, finding that behaviors vary by model and user factors, with implications for responsible design.
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
This paper introduces AARR (Act As a Real Researcher), a suite of benchmarks to evaluate frontier LLMs and agentic systems on granular research scenarios. The first benchmark, AARRI-Bench, reveals that even top-performing agents achieve only 68.3% success, highlighting gaps in field sensitivity and nuanced reasoning.
Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal
This paper introduces boundary-aware self-distillation to improve LLM safety refusal by reducing false refusals on benign prompts while maintaining genuine refusals through controlled data composition.