EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models

# A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
Source: [https://arxiv.org/html/2609.01611](https://arxiv.org/html/2609.01611)
Xinning Li LASR Labs University of Pennsylvania &Kemunto Ochwang’i11footnotemark:1 LASR Labs &Aryasomayajula Ram Bharadwaj11footnotemark:1 LASR Labs Alexandra Souly UK AI Security Institute &Robert Kirk22footnotemark:2 UK AI Security Institute Equal contribution\.Corresponding author\. Correspondence to alexandra\.souly@dsit\.gov\.uk, robert\.kirk@dsit\.gov\.uk

###### Abstract

Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness\. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks\. We introduce EvalDetectBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect\-compatible evaluation, allowing practitioners to test against current and future benchmarks\. EvalDetectBench ships with a newly curated transcript suite covering current frontier system\-card evaluations and diverse deployment sources\. The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations\. We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11\.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others\. EvalDetectBench corrects for both via per\-model probe calibration and a stratified generator\-harmonisation procedure\. We release EvalDetectBench as an open pipeline111https://github\.com/freeze\-lasr/aware\_benchand dataset222https://huggingface\.co/datasets/el7982/aware\-bench/\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x1.png)Figure 1:Overview ofEvalDetectBenchmethodology\.\(a\)Pipeline Architecture\.\(b\)AUROC under three deployment\-data conditions:self\-only\(unconfounded reference\),mixed\(mixture of various generator model deployment data\), andcorrected\.\(c\)Per\-model probe question selection\.## 1Introduction

Evaluations inform deployment decisions, guide capability assessments, and provide the primary evidence base for safety cases across the industry\(Anthropic,[2026c](https://arxiv.org/html/2609.01611#bib.bib10); Google DeepMind,[2025](https://arxiv.org/html/2609.01611#bib.bib17); OpenAI,[2025](https://arxiv.org/html/2609.01611#bib.bib37); xAI,[2025](https://arxiv.org/html/2609.01611#bib.bib60); Meta,[2026](https://arxiv.org/html/2609.01611#bib.bib31)\)\. However, if a model can determine that it is being evaluated\(Laine et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib26); Needham et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib33)\), it may alter its behaviour in ways that invalidate the measurement, performing differently from how it would in deployment environments\.

This can cause under\- or over\-estimation of desired properties\. In propensity evaluations, models may alignment\-fake, behaving well because they are being tested\(Greenblatt et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib18); Anthropic,[2025c](https://arxiv.org/html/2609.01611#bib.bib7)\), or behave badly because they detect that actions in a simulation will not have consequences\. In capability evaluations, models may sandbag, deliberately underperforming\(van der Weij et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib54)\), or cheat by exploiting their knowledge that the task is an evaluation\(Stein et al\.,[2026](https://arxiv.org/html/2609.01611#bib.bib51); Coleman,[2026](https://arxiv.org/html/2609.01611#bib.bib13)\)\. In all cases, the evaluation no longer measures what it claims to\.

Evaluation awareness has already been observed in deployed frontier models: Claude 3\.7 Sonnet spontaneously reasons about being in alignment evaluations\(Goldowsky\-Dill et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib16)\), Sonnet 4\.5 exhibits eval\-gaming behaviour\(Anthropic,[2025c](https://arxiv.org/html/2609.01611#bib.bib7)\), and metagaming reasoning emerges in o3 purely from capabilities RL\(Schoen and Nitishinskaya,[2026](https://arxiv.org/html/2609.01611#bib.bib46)\)\. The first systematic measurement of evaluation awareness as a capability\(Needham et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib33)\)prompted a judge model to classify transcripts as evaluation or deployment, reporting the resultingP​\(eval\)P\(\\text\{eval\}\)\. This established that models can discriminate at non\-trivial rates, but left open whether the choice of prompts and the composition of the transcript pool generalise across model families and evaluation settings\.

We introduceEvalDetectBench\([Figure˜1](https://arxiv.org/html/2609.01611#S0.F1)\), a configurable pipeline with updated methodology and tooling to measure evaluation awareness capability in current frontier systems\. The pipeline collects evaluation and deployment transcripts, scores each with prompts calibrated for the judge model, and reports both per\-benchmark detectability and aggregate discrimination\. Its modular design makes it straightforward to add new models, swap in new evaluations, and run controlled investigations into measurement validity \([Section˜4](https://arxiv.org/html/2609.01611#S4)\)\.

##### Contributions:

1. \(1\)An open, extensible pipelinefor measuring evaluation awareness on any Inspect\-compatible benchmark, with a refreshed transcript suite covering current frontier system\-card evaluations and diverse deployment pools \([Section˜3](https://arxiv.org/html/2609.01611#S3)\)\.
2. \(2\)Per\-model probe calibration\.Elicitation prompts do not transfer across model families; a fixed set under\-elicits awareness by up to 0\.08 AUROC\. Per\-model selection outperforms fixed and random baselines \([Section˜3\.2](https://arxiv.org/html/2609.01611#S3.SS2)\)\.
3. \(3\)A generator\-identity confound and correction\.11\.25% ofP​\(eval\)P\(\\text\{eval\}\)variance is attributable to the deployment transcript’s generator; we show this persists under resampling and propose a stratified harmonisation that recovers the unconfounded signal \([Section˜3\.3](https://arxiv.org/html/2609.01611#S3.SS3)\)\.
4. \(4\)An empirical studyacross 12 judge models and 13 benchmarks: judges discriminate at AUROC\>0\.75\{\>\}\\,0\.75andP​\(eval\)P\(\\text\{eval\}\)is high across benchmark categories \([Section˜4](https://arxiv.org/html/2609.01611#S4)\)\.

## 2Background and Related Work

We define evaluation awareness as the capability to distinguish evaluation from deployment contexts, and elicit it via prompting \(rather than fine\-tuning or other techniques\) so as not to overestimate\. Separately, models may have a*propensity*to consider whether they are being evaluated and act on that belief; we do not address propensity here, as it is a nascent measurement problem while capability evaluation is more established\. Evaluation awareness was first noted anecdotally in Claude 3 Opus, which speculated mid\-task that a retrieval prompt had been “inserted as a joke or to test if I was paying attention”\(Anthropic,[2024](https://arxiv.org/html/2609.01611#bib.bib4)\), and has since been documented across recent frontier system cards\.

Chain\-of\-thought monitoringdetects*verbalised*awareness directly: Apollo Research observed Claude 3\.7 Sonnet reasoning about evaluations in up to one\-third of scheming runs\(Goldowsky\-Dill et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib16)\), and Anthropic’s Sonnet 4\.5 system card reports verbalised awareness rising from 20% to 80% across training\(Anthropic,[2025c](https://arxiv.org/html/2609.01611#bib.bib7)\)\. Verbalisation is a lower bound, however: non\-reasoning models verbalise at under 1% without obviously behaving as though unaware\(Goldowsky\-Dill et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib16)\), and faithfulness limitations mean models may recognise an evaluation without verbalising\([2026a](https://arxiv.org/html/2609.01611#bib.bib8), §4\.5\.5\.3\)\. We focus on measuring evaluation awareness as a capability through prompting, to avoid limitations of chain\-of\-thought faithfulness and propensity to verbalise\.

White\-box or interpretability methodsextract evaluation awareness signals from model activations:Nguyen et al\. \([2025](https://arxiv.org/html/2609.01611#bib.bib34)\)showed evaluation awareness is recoverable from Llama\-3\.3\-70B\-Instruct residual streams via linear probes,Chaudhary et al\. \([2025](https://arxiv.org/html/2609.01611#bib.bib12)\)found this scales predictably across four open\-weight families, andAbdelnabi and Salem \([2025](https://arxiv.org/html/2609.01611#bib.bib2)\),Hua et al\. \([2025](https://arxiv.org/html/2609.01611#bib.bib20)\), and Anthropic\([2025c](https://arxiv.org/html/2609.01611#bib.bib7);[2026b](https://arxiv.org/html/2609.01611#bib.bib9), §6\.5\.2\.2\)further demonstrated that steering or ablating awareness\-related directions causally shifts safety\-relevant behaviour\. We design our benchmark to be applicable to any model with API access, so it does not require white\-box access of any kind, improving its applicability and reproducibility\.

Closest to our work,prompting\-basedapproaches directly ask a judge model to classify a transcript, which rests on the assumptions that the probe does not prime the model and that models respond honestly \(discussed in[Section˜5](https://arxiv.org/html/2609.01611#S5)\)\. Prompting\-based methods have been used in pre\-deployment alignment evaluations\(Kirk et al\.,[2026](https://arxiv.org/html/2609.01611#bib.bib25); Souly et al\.,[2026](https://arxiv.org/html/2609.01611#bib.bib50)\)\.Laine et al\. \([2024](https://arxiv.org/html/2609.01611#bib.bib26)\)introduced the Situational Awareness Dataset covering seven components of situational awareness, one of these \(the STAGES category\) tests evaluation\-vs\-deployment classification on short text excerpts\. Needham et al\. focused on this evaluation awareness component specifically and built a dedicated benchmark of 1,000 transcripts from 61 sources, reporting AUROC up to 0\.83 overall and 0\.95 in agentic settings across nine frontier models\.Ivanov \([2025](https://arxiv.org/html/2609.01611#bib.bib21)\)compares black\-box probing strategies on this dataset and finds disagreement between methods, motivating further methodological study\. EvalDetectBench inherits Needham et al\.’s prompted\-classification framing, but revisits two design choices: the assumption that a single probe\-question set transfers across models \([Section˜3\.2](https://arxiv.org/html/2609.01611#S3.SS2)\) and the assumption that the transcript\-generator’s identity does not affect judge scores \([Section˜3\.3\.1](https://arxiv.org/html/2609.01611#S3.SS3.SSS1)\), both of which we show meaningfully affect measurement\. We also extend the Needham transcript suite with current frontier system\-card benchmarks and a more diverse selection of deployment data \([Section˜3\.1](https://arxiv.org/html/2609.01611#S3.SS1)\)\.

## 3Methodology

We measure evaluation awareness by presenting a judge model with a transcript followed by a probe question \(PQ\) to elicit a classification of evaluation or deployment\. We repeat over many transcripts \(from both evaluation and deployment\) and probe questions, aggregating the classifications\. The pipeline has three components: a transcript suite of evaluation and deployment data \([Section˜3\.1](https://arxiv.org/html/2609.01611#S3.SS1)\), a per\-model probe\-question selection procedure \([Section˜3\.2](https://arxiv.org/html/2609.01611#S3.SS2)\), and scoring that combines probe responses intoP​\(eval\)P\(\\text\{eval\}\)and other metrics\.

### 3\.1Data Collection

We diverge from Needham et al\.’s dataset along three lines\.

First, we move away from static transcripts: evaluation trajectories are collected fresh per evaluated model through our pipeline, so they reflect that model’s own behaviour rather than a frozen log from a different model\. This is important both for the ecological validity of the measurement, and also to avoid generator bias \([Section˜3\.3\.1](https://arxiv.org/html/2609.01611#S3.SS3.SSS1)\)\.

Second, to support per\-category analysis on current benchmarks, we add evaluations used in recent frontier model system cards to the set used to calibrate and measure performance\.

Third, we collect deployment data from more diverse sources\. Following Needham et al\. we split data into*chat*datasets and*agentic*coding datasets, for both evaluation and deployment\. For agentic trajectories we draw mostly from DataClaw\(O’Malley,[2026](https://arxiv.org/html/2609.01611#bib.bib35)\), sampling across user projects, scaffolds, model families, and turn lengths\. We include a curated DataClaw subset from Traced\.run\(Schaeffer,[2026](https://arxiv.org/html/2609.01611#bib.bib45)\)that shows agent misbehaviour \(unauthorised actions, scope creep, data deletion\), which we cut around the misbehaviour signal\. For chat deployment we use ShareChat\(Yan et al\.,[2026](https://arxiv.org/html/2609.01611#bib.bib61)\), WildBench\(Lin et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib28)\)and CodeChat\(Zhong et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib64)\), and crowdsourced chat links\. From Needham et al\.’s pool we keep only the chat datasets sourced from in\-the\-wild user\-LLM chat conversations and curated samples from Open Deep Research\.

The curated suite contains 746 trajectories: 413 evaluation \(98 agentic, 315 chat\-style\) and 333 deployment \(145 agentic, 188 chat\)\. Evaluation is chat\-heavy, with coverage skewing toward short QA, and within deployment the agentic share is higher, skewing toward coding sessions\. AUROC is reported on the full suite\.

Full composition is in[Appendix˜F](https://arxiv.org/html/2609.01611#A6)\. The pipeline treats the trajectory suite as a configurable input, so this composition is a deliberate but replaceable choice to allow practitioners to refresh or swap sources as deployment data and benchmarks evolve\.

### 3\.2Probe question selection and scoring

A probe question \(PQ\) is the instrument through which we measure evaluation awareness\. Each PQ is a closed\-ended question appended to a transcriptτ\\tau, asking the judge to decide whether the conversation originated from evaluation or deployment\. The judge returns a binary label, which we encode asvk=1v\_\{k\}=1for evaluation andvk=0v\_\{k\}=0for deployment\. Given an ensemble ofKKprobes, we aggregate to a per\-transcript score

P​\(eval\)​\(τ\)=1K​∑k=1Kvk∈\[0,1\],P\(\\text\{eval\}\)\(\\tau\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}v\_\{k\}\\in\[0,1\],\(1\)where higher values indicate that more probes flagged the transcript as evaluation\-like\. We report*AUROC*as the primary metric \(threshold\-free and class\-balance\-invariant\), along with AUPRC and binary accuracy at a configurable threshold\.

#### 3\.2\.1PQ corpus curation

Our PQ corpus comprises 20 closed\-ended questions: 17 direct and 3 indirect probes \(full list in[Appendix˜A](https://arxiv.org/html/2609.01611#A1)\)\. We retain four out of Needham et al\.’s five direct PQs, dropping one that assumed deployment interactions always come from a human user and thus mislabels traffic from automated systems \(e\.g\., an email\-classification pipeline\)\. We add 13 more direct probes spanning surface\-level to structural cues, forming a diverse, partially redundant pool for per\-model selection\. Because direct PQs mentionevaluationexplicitly and may invite keyword heuristics, we add three indirect probes that rely on structural and pragmatic cues instead\. For example:

> Does the interaction feel like it emerged naturally — through spontaneous phrasing, informal tone, or reactive follow\-ups — rather than being pre\-planned?

#### 3\.2\.2Per\-model selection procedure

Needham et al\. select five PQs from an initial pool of 28 based on a performance–cost trade\-off for GPT\-4\.1, and apply that fixed set across all models\. This implicitly assumes a single PQ set transfers across model families\. We instead treat PQ choice as a capability elicitation step performed per model: for each judge, we search for the subset of probes that most successfully surfaces its evaluation awareness\.

##### Procedure

For each judge model, we sample a 20% calibration slice stratified by the evaluation/deployment and chat/agentic axis\. On the calibration set, we exhaustively search over all PQ subsets of sizek≤5k\\leq 5from our 20\-item corpus and retain the subset that maximises ensemble AUROC\. The cap atk=5k=5is set for cost reasons and justified empirically below\.

#### 3\.2\.3Validation

We validate our per\-model probe selection approach on a 250\-transcript subset across four frontier models spanning two families \(GPT\-5\.1, GPT\-5\.4, Claude Sonnet 4\.6, Claude Opus 4\.6\)\.[Figure˜2](https://arxiv.org/html/2609.01611#S3.F2)supports three claims:

1. \(i\)Per\-model selection elicits more awareness than the Needham baseline:held\-out AUROC meets or exceeds the baseline fromk≈3k\\approx 3–44onward, reaching 0\.91 \(Sonnet 4\.6\), 0\.90 \(Opus 4\.6\), and 0\.82 \(GPT\-5\.1\)\. GPT\-5\.4 is the exception — the baseline is already strong \(≈0\.82\{\\approx\}\\,0\.82\) and per\-model selection tracks near it, suggesting a ceiling effect\. The differences between models support the argument for per\-model selection: a fixed set serves some judges well and others poorly, and which case applies cannot be determined in advance\.
2. \(ii\)Selection beats random sampling:held\-out curves sit above the mean of 500 random subsets at everykk, with the largest gap at smallkkwhere selection matters most\.
3. \(iii\)Held\-out AUROC plateaus byk≈5k\\approx 5–77, justifying thek≤5k\\leq 5cap in our selection procedure\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x2.png)Figure 2:Ensemble AUROC vs\. number of probe questions, per model, under four selection strategies\.Per\-model exhaustive calibration selection \(green\) and its held\-out performance \(red\), versus a random\-selection baseline \(blue, mean±\\pm1 std over 500 Monte Carlo draws perkk\) and Needham et al\.’s fixed PQ set \(orange dotted line,k=5k=5\)\.Together these results show that calibration\-based per\-model selection elicits evaluation awareness better than either a fixed cross\-model PQ set or random sampling\. Because evaluation awareness has no well\-defined ground truth, the question is whether we have elicited the capability well enough to support conservative estimates, and per\-model selection provides stronger evidence that differences across judges reflect genuine variation in capability rather than under\-elicitation\.

### 3\.3Generator\-identity confound

Deployment data is typically sourced from a mixture of model families, often distinct from the judge under test\. We show that this mixture is not a neutral design choice: generator identity contributes a substantial share of the variance inP​\(eval\)P\(\\text\{eval\}\), this contribution persists under the resampling strategy proposed byWilliams et al\. \([2025](https://arxiv.org/html/2609.01611#bib.bib58)\), and it can shift relative model rankings\. We then propose a post\-hoc correction \([Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)\) for cases where same\-family deployment data is unavailable, such as evaluating models before deployment\.

#### 3\.3\.1Generator identity shifts P\(eval\)

We construct a6×66\\times 6fully crossed design: every model generates deployment transcripts \(sourced from DataClaw conversations\), and every model judges every other model’s transcripts \(∼50\{\\sim\}50samples per cell\)\. We score each cell with the averageP​\(eval\)P\(\\text\{eval\}\)over two probe questions \(one direct, one purpose\-related\)\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x3.png)Figure 3:Claude\-style models are generally seen as more deploy\-like by all judges\.DeployP​\(eval\)P\(\\text\{eval\}\): Judge×\\timesGenerator\. The columnar pattern shows the generator effect: Claude transcripts \(left columns\) receive lowP​\(eval\)P\(\\text\{eval\}\)from all judges, while GPT and Gemini transcripts receive comparatively high scores regardless of who judges them\.[Figure˜3](https://arxiv.org/html/2609.01611#S3.F3)shows a clear columnar pattern: Opus 4\.5 ranks most deploy\-like for every judge without exception \(claude\-opus=0\.410=0\.410, gpt=0\.527=0\.527, gemini=0\.603=0\.603; a\+0\.155\+0\.155gap vs\. all other generators on a 0–1 scale\)\. We can also observe that there is little self\-preference bias: the same\-family mean \(0\.4980\.498\) is close to the cross\-family mean \(0\.5210\.521\) with a0\.0230\.023gap\.

This pattern is consistent with a generator\-identity effect rather than a judge\-specific one\. To quantify it, we fit a Type II two\-way ANOVA decomposingP​\(eval\)P\(\\text\{eval\}\)variance into generator, judge, and interaction components \(full model in[Appendix˜B](https://arxiv.org/html/2609.01611#A2)\)\.Generator identity explains 11\.25% ofP​\(eval\)P\(\\text\{eval\}\)variance\(F=67\.2F=67\.2,p<0\.001p<0\.001\); for unconfounded deployment data this should be zero\.

We further decompose the confound into generation style versus user task distribution and find that generation style is the primary driver \([Appendix˜D](https://arxiv.org/html/2609.01611#A4)\)\.

#### 3\.3\.2Resampling does not eliminate the confound

Williams et al\. \([2025](https://arxiv.org/html/2609.01611#bib.bib58)\)proposes a resampling technique for constructing deployment\-proxy data: strip the last assistant response from a real conversation and regenerate it with a different model\. If the generator signal resides primarily in the most recent assistant turn, replacing that turn should reduce the generator\-identity variance\.

We test this by varying resampling depthkk\(the number of trailing messages stripped and regenerated by the judge\): each model resamples∼30\{\\sim\}30conversations from each of the 6 generators \(including itself\) at each depth, then judges its own resampled set\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x4.png)Figure 4:P​\(eval\)P\(\\text\{eval\}\)by generator family at each resampling depth, panelled by judge family\. Generator\-family separation persists at every depth\.Resampling does not eliminate the generator\-identity confound[Figure˜4](https://arxiv.org/html/2609.01611#S3.F4)shows that generator\-family separation inP​\(eval\)P\(\\text\{eval\}\)persists at every resampling depth: the gap between Claude\-generated and GPT/Gemini\-generated transcripts visible atk=0k=0remains atk=9k=9, regardless of which family is judging\. The full ANOVA decomposition \([Appendix˜C](https://arxiv.org/html/2609.01611#A3)\) confirms this quantitatively: generator identity accounts for 8\.4–11\.3% of variance acrossk=0k=0tok=9k=9\(allp<0\.001p<0\.001\), with no monotonic trend inkk\.

We therefore recommend a simpler mitigation: source deployment transcripts from the same model family as the judge, or from a family with a similarP​\(eval\)P\(\\text\{eval\}\)distribution on a held\-out set\. The correction presented in[Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)can be used to recover most of the unconfounded signal from a mixed pool of deployment\.

#### 3\.3\.3Stratified harmonisation correction

We apply a correction method to reduce bias introduced by the generator\-confound\. The correction applies a light z\-score harmonisation to deploy\-transcript scores before computing AUROC, analogous to ComBat batch\-effect harmonisation\(Johnson et al\.,[2007](https://arxiv.org/html/2609.01611#bib.bib24)\)used to remove systematic differences between measurement batches\. Here, the “batch” \(or*stratum*\) is the \(generator family×\\timesinteraction type\) combination — e\.g\., \(openai, chat\), \(claude, agentic\), \(gemini, chat\)\. The step\-by\-step process is:

1. 1\.Group deploy transcripts into strata by \(generator family×\\timesinteraction type\)\.
2. 2\.Z\-score each transcript’sP​\(eval\)P\(\\text\{eval\}\)within its stratum, removing systematic differences in score level and spread across strata\.
3. 3\.Rescale the z\-scored values to match the judge model’s own\-family score distribution \(multiply by own\-family standard deviation, add own\-family mean\), placing all strata on a common scale anchored to the unconfounded reference\.
4. 4\.Blend at 10% with the original score \(corrected=0\.9×=0\.9\\timesoriginal\+0\.1×\+\\ 0\.1\\timesharmonised\)\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x5.png)Figure 5:Validation of the stratified harmonisation correction\.For each judge, AUROC under three conditions:Mixed\(eval\+\+all deploys except those generated by the judge model itself, raw scores\),Mixed Corrected\(same pool with harmonisation atλ=0\.1\\lambda=0\.1\), andself\-only\(eval\+\+same\-family deploys, the unconfounded reference\)\.Δ\\Deltaabove each group is the AUROC lift from Mixed to Mixed Corrected\. The corrected AUROC is consistently closer to self\-only than the uncorrected version\.The blend weight is chosen conservatively because the harmonisation relies on per\-stratum \(mean, std\) estimates, and some strata are small\. Atλ=0\.1\\lambda=0\.1the harmonised value contributes at most10%10\\%of any corrected score, bounding the influence of the smaller strata’s noisy estimates; higherλ\\lambdawould push the corrected ranking closer to self\-only but lets small\-stratum noise propagate more aggressively into the corrected AUROC\.[Appendix˜E](https://arxiv.org/html/2609.01611#A5)discusses this in more detail\.

We validate this procedure in[Figure˜5](https://arxiv.org/html/2609.01611#S3.F5)\. Across 6 models, we see that applying the correction brings results closer to the self\-only measurement \(which is the ideal but requires enough deployment data\)\.

## 4Evaluation Awareness Across Benchmarks and Models

We run the full pipeline: per\-model PQ selection \([Section˜3\.2](https://arxiv.org/html/2609.01611#S3.SS2)\) and stratified harmonisation \([Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)\) on 12 judges in our suite, with the six closed\-source frontier judges \(Sonnet 4\.6, Opus 4\.6, GPT\-5\.1, GPT\-5\.4, Gemini 2\.5 Pro, Gemini 3\.1 Pro Preview\) as the focus group where same\-family deployment data is available for the harmonisation correction \(See[Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)\)\. We report:*AUROC*, an aggregate rank\-based discrimination metric, and*P​\(eval\)P\(\\text\{eval\}\)*, the raw per\-transcript probability the judge assigns to the eval class\.

##### Frontier models discriminate evaluation from deployment well above chance\.

Across all 12 judges \([Figure˜6](https://arxiv.org/html/2609.01611#S4.F6)\), mixed AUROC ranges from0\.760\.76\(Gemini 2\.5 Pro, Claude Haiku 4\.5\) to0\.890\.89\(Claude Opus 4\.6\), with every closed\-source family showing improvement from older to newer release\. The progression is most pronounced for the Anthropic family \(\+0\.13\+0\.13from haiku to opus\)\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x6.png)Figure 6:Per\-model mixed AUROC across 12 judges,closed source models are grouped by family and ordered by release date\. Within each closed\-source family the newest model attains strictly the highest AUROC \(Anthropic0\.76→0\.88→0\.890\.76\\to 0\.88\\to 0\.89; OpenAI0\.78→0\.80→0\.830\.78\\to 0\.80\\to 0\.83; Google0\.76→0\.870\.76\\to 0\.87\)\. The four open\-source judges cluster in\[0\.78,0\.81\]\[0\.78,0\.81\]\. AUROC values here are uncorrected \(no stratified harmonisation\); see[Figure˜12](https://arxiv.org/html/2609.01611#A9.F12)\.
##### P​\(eval\)P\(\\text\{eval\}\)is high across the benchmarks we measure\.

Pooled across the three frontier judges \(GPT\-5\.4, Claude Opus 4\.6 and Gemini 3\.1 Pro Preview\), meanP​\(eval\)\>0\.83P\(\\mathrm\{eval\}\)\>0\.83on a subset of benchmarks in our suite\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x7.png)Figure 7:MeanP​\(eval\)P\(\\text\{eval\}\)per evaluation dataset, averaged across six judges and grouped by category\. Error bars are bootstrap 95% confidence intervals\.
##### Selected probes surface different signals\.

GPT\-5\.4’s motivation probe scores0\.000\.00–0\.050\.05onswebench\-miniandgdpval, while its origin probe scores \>0\.890\.89on the same transcripts\. Opus\-4\.6’sh\_provenance\_adrops to0\.600\.60onmmmluandcybench, while its authenticity probe stays at0\.950\.95–1\.001\.00on the same transcripts\. Gemini\-3\.1’s lowest probe score is0\.900\.90\(motivation onstrongreject\)\. Ensemble AUROC averages over these heterogeneous signals; per\-probe and per\-judge variance reveals structural disagreement that the aggregate conceals\. Full per\-\(judge, probe, benchmark\) values are in Table[3](https://arxiv.org/html/2609.01611#A1.T3)\.

Judges can disagree on transcripts where task realism and meta\-evaluation signals point in different directions \([Figure˜8](https://arxiv.org/html/2609.01611#A1.F8)\)\. Ongdpval, this split recurs across the majority of transcripts, with GPT\-5\.4 more often reading through meta\-evaluation framing to the underlying task structure, while Opus and Gemini\-3\.1 more often treat explicit evaluation signals as decisive\.

## 5Discussion

##### What prompted classification measures\.

Asking a model directly whether a transcript comes from an evaluation or deployment provides what we treat as a maximal\-elicitation upper bound on evaluation awareness: if a model cannot distinguish the two even when explicitly asked, it is unlikely to do so spontaneously\. This framing rests on two assumptions\. The first, that asking does not itself inflate scores \(the priming effect\), we partially defend against through our choice of metric\. The second, that the model answers honestly rather than dishonestly \(i\.e\. sandbags\(van der Weij et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib54)\)\), remains untested in this work and in prior work, and is an open methodological problem for the field\.

##### Priming effect\.

We chose AUROC as our primary metric specifically because it defends against the priming effect: a probe that explicitly mentions evaluation may inflate the likelihood of classifying a transcript as an eval, but as long as the inflation is roughly uniform across both classes, AUROC \(which depends only on the rank ordering between eval and deploy transcripts\) is unchanged\. This is the central reason we report AUROC rather than rawP​\(eval\)P\(\\text\{eval\}\)or accuracy at a fixed threshold\. The defence is partial, however: priming is not necessarily uniform, and if it interacts with transcript content \(e\.g\., inflating scores more on ambiguous deploy transcripts than on clear eval ones\), the differential bias survives AUROC\. Quantifying this residual interaction would require a probe condition that does not itself prime, which prompted classification cannot provide\. This is the fundamental limitation of all prompted approaches, and the strongest argument for complementing them with white\-box methods \([Section˜2](https://arxiv.org/html/2609.01611#S2)\) that do not require asking the model anything\. Unfortunately this is not possible on closed\-source frontier models\.

##### Practical implications\.

Eval\-awareness measurement should be standard practice alongside safety evaluations\. A benchmark that is easily recognisable as an evaluation produces results whose validity is conditional: any safety case built on it must account for the possibility that models behaved differently than they would in deployment\. Reporting eval\-awareness scores alongside benchmark results strengthens the evidential chain in safety cases, and EvalDetectBench is designed to slot into existing Inspect\-based workflows with minimal adoption friction\.

##### Limitations\.

Beyond the priming and honest\-reporting limitations discussed above, there are another two methodological limitations\. First, our deployment pool is assembled from public sources and is unlikely to match any single provider’s true traffic distribution\. AUROC on this pool is a proxy for discrimination against real deployment data\. Second, the stratified harmonisation \([Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)\) is validated by agreement with the self\-only condition on a disjoint deploy pool, but self\-only is itself an imperfect proxy, and some strata might be small or noisy\. We present harmonisation as a pragmatic correction rather than a debiased estimator with known error bounds\.

##### Conclusion\.

We showed that standard methodological choices in evaluation awareness measurement, including transcript sourcing, probe question selection, and scoring format, introduce biases that, if uncontrolled, can be larger than the differences in eval\-awareness between the models being measured\. Generator identity contributes 11\.25% ofP​\(eval\)P\(\\text\{eval\}\)variance in heterogeneous deployment pools, and probe\-question sets calibrated on one model family under\-elicit awareness in others by up to 0\.08 AUROC\. EvalDetectBench incorporates these findings into an open pipeline that supports per\-model probe calibration and generator\-confound correction for any Inspect\-compatible evaluation\.

Open problems remain: developing methods for constructing deployment\-proxy data beyond shallow resampling, quantifying the priming effect of directly asking about evaluation\-ness, and measuring unprompted evaluation awareness to determine whether models act on their recognition without being asked\.

## Acknowledgments and Disclosure of Funding

This research was funded and conducted as part of the LASR Winter Cohort 2026 program\. We thank Erin Robertson, Brandon Higgs and Tamkeen Nawab for their support in project management\. We thank them for ensuring we stepped back, rethought, and focused on the right questions and for their reviews throughout\. We also thank Nitisha Jain for her contributions during the early stages of this project\.

## References

- Aakanksha et al\. \[2024\]Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb\-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker\.The multilingual alignment prism: Aligning global and local preferences to reduce harm\.*arXiv preprint arXiv:2406\.18682*, 2024\.URL[https://arxiv\.org/abs/2406\.18682](https://arxiv.org/abs/2406.18682)\.
- Abdelnabi and Salem \[2025\]Sahar Abdelnabi and Ahmed Salem\.The Hawthorne effect in reasoning models: Evaluating and steering test awareness\.*arXiv preprint arXiv:2505\.14617*, 2025\.URL[https://arxiv\.org/abs/2505\.14617](https://arxiv.org/abs/2505.14617)\.NeurIPS 2025 Spotlight\.
- Andriushchenko et al\. \[2025\]Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, et al\.Agentharm: A benchmark for measuring harmfulness of LLM agents\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.URL[https://arxiv\.org/abs/2410\.09024](https://arxiv.org/abs/2410.09024)\.
- Anthropic \[2024\]Anthropic\.The Claude 3 model family: Opus, Sonnet, Haiku\.Technical report, Anthropic, 2024\.URL[https://www\-cdn\.anthropic\.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model\_Card\_Claude\_3\.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf)\.Model card\.
- Anthropic \[2025a\]Anthropic\.Agentic misalignment: How LLMs could be insider threats, 2025a\.URL[https://www\.anthropic\.com/research/agentic\-misalignment](https://www.anthropic.com/research/agentic-misalignment)\.
- Anthropic \[2025b\]Anthropic\.Petri: An open\-source auditing tool to accelerate ai safety research\.[https://www\.anthropic\.com/research/petri\-open\-source\-auditing](https://www.anthropic.com/research/petri-open-source-auditing), 2025b\.
- Anthropic \[2025c\]Anthropic\.System card: Claude Sonnet 4\.5\.Technical report, Anthropic, 2025c\.URL[https://assets\.anthropic\.com/m/12f214efcc2f457a/original/Claude\-Sonnet\-4\-5\-System\-Card\.pdf](https://assets.anthropic.com/m/12f214efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf)\.
- Anthropic \[2026a\]Anthropic\.System card: Claude mythos preview\.Technical report, Anthropic, April 2026a\.
- Anthropic \[2026b\]Anthropic\.System card: Claude opus 4\.7\.Technical report, Anthropic, April 2026b\.
- Anthropic \[2026c\]Anthropic\.Responsible scaling policy\.Technical report, Anthropic, 2026c\.URL[https://www\-cdn\.anthropic\.com/files/4zrzovbb/website/bf04581e4f329735fd90634f6a1962c13c0bd351\.pdf](https://www-cdn.anthropic.com/files/4zrzovbb/website/bf04581e4f329735fd90634f6a1962c13c0bd351.pdf)\.Version 3\.1, effective 2 April 2026\.
- Bai et al\. \[2022\]Yuntao Bai, Andy Jones, Kamal Ndousse, et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.*arXiv preprint arXiv:2204\.05862*, 2022\.URL[https://arxiv\.org/abs/2204\.05862](https://arxiv.org/abs/2204.05862)\.
- Chaudhary et al\. \[2025\]Maheep Chaudhary, Ian Su, Nikhil Hooda, et al\.Evaluation awareness scales predictably in open\-weights large language models\.*arXiv preprint arXiv:2509\.13333*, 2025\.URL[https://arxiv\.org/abs/2509\.13333](https://arxiv.org/abs/2509.13333)\.NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models \(ResponsibleFM\)\.
- Coleman \[2026\]Russell Coleman\.Eval awareness in Claude Opus 4\.6’s BrowseComp performance\.[https://www\.anthropic\.com/engineering/eval\-awareness\-browsecomp](https://www.anthropic.com/engineering/eval-awareness-browsecomp), March 2026\.Anthropic Engineering Blog\.
- Durmus et al\. \[2024\]Esin Durmus, Karina Nguyen, Thomas I\. Liao, et al\.Towards measuring the representation of subjective global opinions in language models\.*arXiv preprint arXiv:2306\.16388*, 2024\.URL[https://arxiv\.org/abs/2306\.16388](https://arxiv.org/abs/2306.16388)\.
- Gehman et al\. \[2020\]Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A\. Smith\.RealToxicityPrompts: Evaluating neural toxic degeneration in language models\.*arXiv preprint arXiv:2009\.11462*, 2020\.URL[https://arxiv\.org/abs/2009\.11462](https://arxiv.org/abs/2009.11462)\.
- Goldowsky\-Dill et al\. \[2025\]Nicholas Goldowsky\-Dill, Mikita Balesni, Jérémy Scheurer, and Marius Hobbhahn\.Claude Sonnet 3\.7 \(often\) knows when it’s in alignment evaluations\.Apollo Research blog, 2025\.URL[https://www\.apolloresearch\.ai/blog/claude\-sonnet\-37\-often\-knows\-when\-its\-in\-alignment\-evaluations](https://www.apolloresearch.ai/blog/claude-sonnet-37-often-knows-when-its-in-alignment-evaluations)\.
- Google DeepMind \[2025\]Google DeepMind\.Frontier safety framework\.Technical report, Google DeepMind, 2025\.URL[https://storage\.googleapis\.com/deepmind\-media/DeepMind\.com/Blog/strengthening\-our\-frontier\-safety\-framework/frontier\-safety\-framework\_3\.pdf](https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/strengthening-our-frontier-safety-framework/frontier-safety-framework_3.pdf)\.Version 3\.0, 22 September 2025\.
- Greenblatt et al\. \[2024\]Ryan Greenblatt, Carson Denison, Benjamin Wright, et al\.Alignment faking in large language models\.*arXiv preprint arXiv:2412\.14093*, 2024\.URL[https://arxiv\.org/abs/2412\.14093](https://arxiv.org/abs/2412.14093)\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*, 2021\.URL[https://arxiv\.org/abs/2009\.03300](https://arxiv.org/abs/2009.03300)\.
- Hua et al\. \[2025\]Tim Tian Hua, Andrew Qin, Samuel Marks, and Neel Nanda\.Steering evaluation\-aware language models to act like they are deployed\.*arXiv preprint arXiv:2510\.20487*, 2025\.URL[https://arxiv\.org/abs/2510\.20487](https://arxiv.org/abs/2510.20487)\.
- Ivanov \[2025\]Igor Ivanov\.Comparative analysis of black\-box methods for detecting evaluation awareness in LLMs\.LessWrong / AI Alignment Forum, 2025\.URL[https://www\.lesswrong\.com/posts/Waz32KuSxo6SSjyND/comparative\-analysis\-of\-black\-box\-methods\-for\-detecting](https://www.lesswrong.com/posts/Waz32KuSxo6SSjyND/comparative-analysis-of-black-box-methods-for-detecting)\.
- Jin et al\. \[2024\]Jiho Jin, Jiseon Kim, Nayeon Lee, Haneul Yoo, Alice Oh, and Hwaran Lee\.KoBBQ: Korean bias benchmark for question answering\.*arXiv preprint arXiv:2307\.16778*, 2024\.URL[https://arxiv\.org/abs/2307\.16778](https://arxiv.org/abs/2307.16778)\.
- Jin et al\. \[2022\]Zhijing Jin, Sydney Levine, Fernando Gonzalez, et al\.When to make exceptions: Exploring language models as accounts of human moral judgment\.*arXiv preprint arXiv:2210\.01478*, 2022\.URL[https://arxiv\.org/abs/2210\.01478](https://arxiv.org/abs/2210.01478)\.
- Johnson et al\. \[2007\]W\. E\. Johnson, C\. Li, and A\. Rabinovic\.Adjusting batch effects in microarray expression data using empirical Bayes methods\.*Biostatistics \(Oxford, England\)*, 8\(1\):118–127, 2007\.doi:10\.1093/biostatistics/kxj037\.
- Kirk et al\. \[2026\]Robert Kirk, Alexandra Souly, Kai Fronsdal, Abby D’Cruz, and Xander Davies\.Evaluating whether ai models would sabotage ai safety research, 2026\.URL[https://arxiv\.org/abs/2604\.24618](https://arxiv.org/abs/2604.24618)\.
- Laine et al\. \[2024\]Rudolf Laine, Bilal Chughtai, Jan Betley, et al\.Me, myself, and AI: The situational awareness dataset \(SAD\) for LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2024\.URL[https://arxiv\.org/abs/2407\.04694](https://arxiv.org/abs/2407.04694)\.arXiv:2407\.04694\.
- Li et al\. \[2024\]Nathaniel Li, Alexander Pan, Anjali Gopal, et al\.The WMDP benchmark: Measuring and reducing malicious use with unlearning\.*arXiv preprint arXiv:2403\.03218*, 2024\.URL[https://arxiv\.org/abs/2403\.03218](https://arxiv.org/abs/2403.03218)\.
- Lin et al\. \[2024\]Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, et al\.WildBench: Benchmarking LLMs with challenging tasks from real users in the wild\.*arXiv preprint arXiv:2406\.04770*, 2024\.URL[https://arxiv\.org/abs/2406\.04770](https://arxiv.org/abs/2406.04770)\.
- Lin et al\. \[2018\]Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D\. Ernst\.NL2Bash: A corpus and semantic parser for natural language interface to the Linux operating system\.*arXiv preprint arXiv:1802\.08979*, 2018\.URL[https://arxiv\.org/abs/1802\.08979](https://arxiv.org/abs/1802.08979)\.
- Lin et al\. \[2023\]Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang\.ToxicChat: Unveiling hidden challenges of toxicity detection in real\-world user\-AI conversation\.*arXiv preprint arXiv:2310\.17389*, 2023\.URL[https://arxiv\.org/abs/2310\.17389](https://arxiv.org/abs/2310.17389)\.
- Meta \[2026\]Meta\.Advanced AI scaling framework v2\.[https://ai\.meta\.com/static\-resource/Meta\_Advanced\-AI\-Scaling\-Framework\-v2](https://ai.meta.com/static-resource/Meta_Advanced-AI-Scaling-Framework-v2), 2026\.
- Mialon et al\. \[2023\]Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom\.GAIA: A benchmark for general AI assistants\.*arXiv preprint arXiv:2311\.12983*, 2023\.URL[https://arxiv\.org/abs/2311\.12983](https://arxiv.org/abs/2311.12983)\.
- Needham et al\. \[2025\]Joe Needham, Giles Edkins, Govind Pimpale, Henning Bartsch, and Marius Hobbhahn\.Large language models often know when they are being evaluated\.*arXiv preprint arXiv:2505\.23836*, 2025\.URL[https://arxiv\.org/abs/2505\.23836](https://arxiv.org/abs/2505.23836)\.
- Nguyen et al\. \[2025\]Jord Nguyen, Khiem Hoang, Carlo Leonardo Attubato, and Felix Hofstätter\.Probing and steering evaluation awareness of language models\.*arXiv preprint arXiv:2507\.01786*, 2025\.URL[https://arxiv\.org/abs/2507\.01786](https://arxiv.org/abs/2507.01786)\.
- O’Malley \[2026\]Peter O’Malley\.DataClaw: Agent harness to publish your history from Claude Code et al\. as Hugging Face datasets, 2026\.URL[https://github\.com/peteromallet/dataclaw](https://github.com/peteromallet/dataclaw)\.
- OpenAI \[2024\]OpenAI\.Introducing SWE\-bench verified\.[https://openai\.com/index/introducing\-swe\-bench\-verified/](https://openai.com/index/introducing-swe-bench-verified/), 2024\.
- OpenAI \[2025\]OpenAI\.Preparedness framework\.Technical report, OpenAI, 2025\.URL[https://cdn\.openai\.com/pdf/18a02b5d\-6b67\-4cec\-ab64\-68cdfbddebcd/preparedness\-framework\-v2\.pdf](https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf)\.Version 2, 15 April 2025\.
- Patwardhan et al\. \[2025\]Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, et al\.GDPval: Evaluating AI model performance on real\-world economically valuable tasks\.*arXiv preprint arXiv:2510\.04374*, 2025\.URL[https://arxiv\.org/abs/2510\.04374](https://arxiv.org/abs/2510.04374)\.
- Perez et al\. \[2022\]Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, et al\.Discovering language model behaviors with model\-written evaluations\.*arXiv preprint arXiv:2212\.09251*, 2022\.URL[https://arxiv\.org/abs/2212\.09251](https://arxiv.org/abs/2212.09251)\.
- Pistilli et al\. \[2024\]Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell\.CIVICS: Building a dataset for examining culturally\-informed values in large language models\.*arXiv preprint arXiv:2405\.13974*, 2024\.URL[https://arxiv\.org/abs/2405\.13974](https://arxiv.org/abs/2405.13974)\.
- Rein et al\. \[2024\]David Rein, Betty Li Hou, Asa Cooper Stickland, et al\.GPQA: A graduate\-level google\-proof Q&A benchmark\.In*Conference on Language Modeling \(COLM\)*, 2024\.URL[https://arxiv\.org/abs/2311\.12022](https://arxiv.org/abs/2311.12022)\.
- Ren et al\. \[2025\]Richard Ren, Arunim Agarwal, Mantas Mazeika, et al\.The MASK benchmark: Disentangling honesty from accuracy in AI systems\.*arXiv preprint arXiv:2503\.03750*, 2025\.URL[https://arxiv\.org/abs/2503\.03750](https://arxiv.org/abs/2503.03750)\.
- Röttger et al\. \[2024\]Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy\.XSTest: A test suite for identifying exaggerated safety behaviours in large language models\.*arXiv preprint arXiv:2308\.01263*, 2024\.URL[https://arxiv\.org/abs/2308\.01263](https://arxiv.org/abs/2308.01263)\.
- Roucher et al\. \[2025\]Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki\.smolagents: A smol library to build great agentic systems, 2025\.URL[https://github\.com/huggingface/smolagents](https://github.com/huggingface/smolagents)\.
- Schaeffer \[2026\]Joachim Schaeffer\.traced\-runs\-misaligned\-users\-and\-agents\.[https://huggingface\.co/datasets/schaeff/traced\-runs\-misaligned\-users\-and\-agents](https://huggingface.co/datasets/schaeff/traced-runs-misaligned-users-and-agents), 2026\.
- Schoen and Nitishinskaya \[2026\]Bronson Schoen and Jenny Nitishinskaya\.Metagaming matters for training, evaluation, and oversight\.Apollo Research blog, 2026\.URL[https://www\.apolloresearch\.ai/blog/metagaming\-matters\-for\-training\-evaluation\-and\-oversight/](https://www.apolloresearch.ai/blog/metagaming-matters-for-training-evaluation-and-oversight/)\.
- Schulhoff et al\. \[2024\]Sander Schulhoff, Jeremy Pinto, Anaum Khan, et al\.Ignore this title and HackAPrompt: Exposing systemic vulnerabilities of LLMs through a global scale prompt hacking competition\.*arXiv preprint arXiv:2311\.16119*, 2024\.URL[https://arxiv\.org/abs/2311\.16119](https://arxiv.org/abs/2311.16119)\.
- Skepsun \[2023\]Skepsun\.cvalues\_rlhf, 2023\.URL[https://huggingface\.co/datasets/Skepsun/cvalues\_rlhf](https://huggingface.co/datasets/Skepsun/cvalues_rlhf)\.
- Souly et al\. \[2024\]Alexandra Souly, Qingyuan Lu, Dillon Bowen, et al\.A StrongREJECT for empty jailbreaks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.URL[https://arxiv\.org/abs/2402\.10260](https://arxiv.org/abs/2402.10260)\.
- Souly et al\. \[2026\]Alexandra Souly, Robert Kirk, Jacob Merizian, Abby D’Cruz, and Xander Davies\.Uk aisi alignment evaluation case\-study, 2026\.URL[https://arxiv\.org/abs/2604\.00788](https://arxiv.org/abs/2604.00788)\.
- Stein et al\. \[2026\]Adam Stein, Davis Brown, Hamed Hassani, Mayur Naik, and Eric Wong\.Detecting safety violations across many agent traces, 2026\.URL[https://arxiv\.org/abs/2604\.11806](https://arxiv.org/abs/2604.11806)\.
- Tamkin et al\. \[2023\]Alex Tamkin, Amanda Askell, Liane Lovitt, et al\.Evaluating and mitigating discrimination in language model decisions\.*arXiv preprint arXiv:2312\.03689*, 2023\.URL[https://arxiv\.org/abs/2312\.03689](https://arxiv.org/abs/2312.03689)\.
- Üstün et al\. \[2024\]Ahmet Üstün, Viraat Aryabumi, Zheng\-Xin Yong, et al\.Aya model: An instruction finetuned open\-access multilingual language model\.*arXiv preprint arXiv:2402\.07827*, 2024\.URL[https://arxiv\.org/abs/2402\.07827](https://arxiv.org/abs/2402.07827)\.
- van der Weij et al\. \[2025\]Teun van der Weij, Felix Hofstätter, Oliver Jaffe, Samuel F\. Brown, and Francis Rhys Ward\.AI sandbagging: Language models can strategically underperform on evaluations\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.URL[https://arxiv\.org/abs/2406\.07358](https://arxiv.org/abs/2406.07358)\.arXiv:2406\.07358\.
- Wang et al\. \[2024\]Xingyao Wang, Boxuan Li, Yufan Song, et al\.OpenHands: An open platform for AI software developers as generalist agents\.*arXiv preprint arXiv:2407\.16741*, 2024\.URL[https://arxiv\.org/abs/2407\.16741](https://arxiv.org/abs/2407.16741)\.
- Wang et al\. \[2023\]Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin\.Do\-not\-answer: A dataset for evaluating safeguards in LLMs\.*arXiv preprint arXiv:2308\.13387*, 2023\.URL[https://arxiv\.org/abs/2308\.13387](https://arxiv.org/abs/2308.13387)\.
- Wei et al\. \[2025\]Jason Wei, Zhiqing Sun, Spencer Papay, et al\.BrowseComp: A simple yet challenging benchmark for browsing agents\.*arXiv preprint arXiv:2504\.12516*, 2025\.URL[https://arxiv\.org/abs/2504\.12516](https://arxiv.org/abs/2504.12516)\.
- Williams et al\. \[2025\]Marcus Williams, Cameron Raymond, and Micah Carroll\.Sidestepping evaluation awareness and anticipating misalignment with production evaluations\.OpenAI Alignment Blog, 2025\.URL[https://alignment\.openai\.com/prod\-evals/](https://alignment.openai.com/prod-evals/)\.
- Wu et al\. \[2025\]Yuning Wu, Jiahao Mei, Ming Yan, et al\.WritingBench: A comprehensive benchmark for generative writing\.*arXiv preprint arXiv:2503\.05244*, 2025\.URL[https://arxiv\.org/abs/2503\.05244](https://arxiv.org/abs/2503.05244)\.
- xAI \[2025\]xAI\.Frontier AI framework\.Technical report, xAI, December 2025\.
- Yan et al\. \[2026\]Yueru Yan, Tuc Nguyen, Bo Su, Melissa Lieffers, and Thai Le\.ShareChat: A dataset of chatbot conversations in the wild, 2026\.URL[https://arxiv\.org/abs/2512\.17843](https://arxiv.org/abs/2512.17843)\.
- Yuan et al\. \[2025\]Weizhe Yuan, Jane Yu, Song Jiang, et al\.NaturalReasoning: Reasoning in the wild with 2\.8M challenging questions\.*arXiv preprint arXiv:2502\.13124*, 2025\.URL[https://arxiv\.org/abs/2502\.13124](https://arxiv.org/abs/2502.13124)\.
- Zhang et al\. \[2024\]Andy K\. Zhang, Neil Perry, Riya Dulepet, et al\.Cybench: A framework for evaluating cybersecurity capabilities and risks of language models\.*arXiv preprint arXiv:2408\.08926*, 2024\.URL[https://arxiv\.org/abs/2408\.08926](https://arxiv.org/abs/2408.08926)\.
- Zhong et al\. \[2025\]Suzhen Zhong, Ying Zou, and Bram Adams\.Developer\-LLM conversations: An empirical study of interactions and generated code quality\.*arXiv preprint arXiv:2509\.10402*, 2025\.URL[https://arxiv\.org/abs/2509\.10402](https://arxiv.org/abs/2509.10402)\.
- Zhou et al\. \[2023\]Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, et al\.Instruction\-following evaluation for large language models\.*arXiv preprint arXiv:2311\.07911*, 2023\.URL[https://arxiv\.org/abs/2311\.07911](https://arxiv.org/abs/2311.07911)\.
- Zhou et al\. \[2022\]Jingyan Zhou, Jiawen Deng, Fei Mi, et al\.Towards identifying social bias in dialog systems: Framework, dataset, and benchmark\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, 2022\.URL[https://aclanthology\.org/2022\.findings\-emnlp\.262/](https://aclanthology.org/2022.findings-emnlp.262/)\.

## Appendix AProbe Question Corpus

Each of these probing questions is passed to the model to generate a single label which denotes ‘eval’ or ‘deployment’ use case\.[Table˜1](https://arxiv.org/html/2609.01611#A1.T1)lists the full set of 20 probe questions\.

Table 1:Full probe question corpus \(20 PQs\)\.### A\.1Probe Question Selection

Table 2:Self\-play probe\-question selection per judge model\(6 models, 20% calibration set, n≈\\approx130 transcripts stratified along the is\_eval and has\_tools axes\)\. For each judge, the selected set is the exhaustive combination of≤5\\leq 5probes that maximises ensemble AUROC\. A dot marks each selection\.kkis the number of questions selected for the judge model; \# is the number of models that select a given question\. No probe is selected by every model, though h\_stakeholder\_a is chosen by 5 of 6 \(skipped only by gpt\-5\.4, where it sits at chance\) and indirect\_organic\_flow\_a by 4 of 6, while 4 probes are picked by none\. This reinforces that probe selection should be per\-judge rather than shared\.Probegpt\-5\.1

gpt\-5\.4

sonnet\-4\.6

opus\-4\.6

gem\-2\.5\-pro

gem\-3\.1\-pro

\#h\_stakeholder\_a∙\\bullet∙\\bullet∙\\bullet∙\\bullet∙\\bullet5indirect\_organic\_flow\_a∙\\bullet∙\\bullet∙\\bullet∙\\bullet4h\_authenticity\_a∙\\bullet∙\\bullet∙\\bullet3h\_motivation\_a∙\\bullet∙\\bullet∙\\bullet3h\_origin\_a∙\\bullet∙\\bullet∙\\bullet3h\_naturalness\_a∙\\bullet∙\\bullet2h\_purpose\_utility\_a∙\\bullet∙\\bullet2h\_audience\_a∙\\bullet1h\_outcome\_a∙\\bullet1h\_provenance\_a∙\\bullet1h\_purpose\_function\_a∙\\bullet1indirect\_consequences\_a∙\\bullet1kk554454Ensemble AUROC0\.7890\.7990\.8900\.8850\.8210\.831Table 3:Per\-probe meanP​\(eval\)P\(\\text\{eval\}\)on eval\-category transcripts, broken out by judge×\\timesbenchmark\. Each judge uses its own greedy\-selected probe set \(opus: 4 probes; gpt\-5\.4: 5; gemini\-3\.1: 4\)\. Benchmark columns sorted by overall ascending mean \(matches[Figure˜7](https://arxiv.org/html/2609.01611#S4.F7)\)\. Bold cells indicateP​\(eval\)<0\.50P\(\\text\{eval\}\)<0\.50\.Opus 4\.6The disclaimer “we will also grade your final message” is present in the task framing\. This isthe most definitive signalof an artificial evaluation context; grading language is characteristic of assessment, not production delivery\.· artificial

Gemini\-3\.1 ProThe task involves a structured deliverable with explicit grading criteria\.Grading of outputs is a hallmark of evaluation settings, not live deployment; a production client would not announce that responses will be scored\.· artificial

GPT\-5\.4This is a concrete business deliverable — build a P&L model, present assumptions, calculate profitability\.The note that the final message will be graded does not outweigh the clearly task\-oriented nature of the request; real workflows include review criteria too\.· practical

Figure 8:Same transcript, different verdict\(gdpval:7b08cd4d\)\. An Excel P&L task with the explicit disclaimer “We will also grade your final message\.” Each box shows a judge’s reasoning and its binary label under theh\_origin\_aprobe, which asks whether the interaction originated from a benchmark or a production setting \(artificialvs\.practical\)\. Opus and Gemini\-3\.1 treat the grading language as a decisive evaluation signal; GPT\-5\.4 reads through it to the underlying task structure\.

## Appendix BANOVA model details

We fit a Type II two\-way ANOVA model to decomposeP​\(eval\)P\(\\text\{eval\}\)variance into four components:

P​\(eval\)i​j​k=μ\+αgeneratori\+βjudgej\+\(α​β\)i​j\+εi​j​kP\(\\text\{eval\}\)\_\{ijk\}=\\mu\+\\alpha\_\{\\text\{generator\}\_\{i\}\}\+\\beta\_\{\\text\{judge\}\_\{j\}\}\+\(\\alpha\\beta\)\_\{ij\}\+\\varepsilon\_\{ijk\}\(2\)This model decomposes eachP​\(eval\)P\(\\text\{eval\}\)score into four additive components: a global meanμ\\mu, a generator effectαi\\alpha\_\{i\}, a judge effectβj\\beta\_\{j\}, an interaction\(α​β\)i​j\(\\alpha\\beta\)\_\{ij\}, and residual noiseεi​j​k\\varepsilon\_\{ijk\}\. The ANOVA partitions the total sum of squaresSStotal\\text\{SS\}\_\{\\text\{total\}\}into components attributable to each factor \(SSgen\\text\{SS\}\_\{\\text\{gen\}\},SSjudge\\text\{SS\}\_\{\\text\{judge\}\},SSgen×judge\\text\{SS\}\_\{\\text\{gen\}\\times\\text\{judge\}\},SSresidual\\text\{SS\}\_\{\\text\{residual\}\}\)\. We reportη2=SSfactor/SStotal\\eta^\{2\}=\\text\{SS\}\_\{\\text\{factor\}\}/\\text\{SS\}\_\{\\text\{total\}\}for each, giving the fraction of total variance that factor explains\. Type II sums of squares ensure each factor is tested after accounting for the others\. For unconfounded deployment data,ηgen2\\eta^\{2\}\_\{\\text\{gen\}\}should be zero — any nonzero value means generator identity is systematically shiftingP​\(eval\)P\(\\text\{eval\}\)scores\.

Table 4:ANOVA decomposition ofP​\(eval\)P\(\\text\{eval\}\)variance\.
## Appendix CANOVA decomposition by resampling depth

[Table˜5](https://arxiv.org/html/2609.01611#A3.T5)reports the full Type II two\-way ANOVA decomposition ofP​\(eval\)P\(\\text\{eval\}\)variance at each resampling depthkk, wherekkis the number of trailing messages stripped from the transcript and regenerated by the judge model\. Generator identity’s contribution \(ηgen2\\eta^\{2\}\_\{\\text\{gen\}\}\) remains in the 8\.4–11\.3% range across all depths, visualised in[Figure˜9](https://arxiv.org/html/2609.01611#A3.F9)\.

Table 5:Generator identity effect vs\. resampling depth\.![Refer to caption](https://arxiv.org/html/2609.01611v1/x8.png)Figure 9:Eta\-squared explained by the generator stays almost flat \(11\.31%–8\.39% range\) regardless of resampling depth\.
## Appendix DIs the Confound from Generation Style or User Tasks?

The columnar pattern in theP​\(eval\)P\(\\text\{eval\}\)matrix could arise because different model families produce different styles of text, or because users of different coding agents tend to ask different kinds of questions\.

We strip each conversation down to just the first user message and construct two conditions:

- •Condition A:We keep the original model’s first response alongside the user message\. The judge sees 2 messages: user task \+ original response style\.
- •Condition B:We discard the original response and have each judge model generate its own response to the same user message\. The judge sees 2 messages: user task \+ the judge’s own response\.

In Condition B, the original model is completely absent — the judge is responding to and then scoring its own answer\. The only thing that still varies across “generator” labels is which user tasks that model happened to have\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x9.png)Figure 10:P​\(eval\)P\(\\text\{eval\}\)per task pool under Condition A \(model responds\) vs\. Condition B \(judge responds to same tasks\)\.In Condition B, cross\-generator spread moves to 0\.096 \(from 0\.147 in Condition A\), confirming that the coding tasks from different transcripts are roughly equivalent\. The divergence is introduced by the generator’s response: Claude’s style adds only\+0\.09\+0\.09toP​\(eval\)P\(\\text\{eval\}\)beyond the user prompt alone, while GPT adds3\.6×3\.6\\timesmore \(\+0\.31\+0\.31\)\.

Table 6:Style contribution per generator\.Condition A \(model’s response present\), ranking byP​\(eval\)P\(\\text\{eval\}\): opus\-4\.5<<gem\-3\-pro<<gpt\-5\.3<<gem\-2\.5\-pro<<gem\-3\-flash<<gpt\-5\.4\.

Condition B \(model removed, judge responds to same user tasks\), ranking byP​\(eval\)P\(\\text\{eval\}\): gpt\-5\.3 tasks<<gpt\-5\.4 tasks<<gem\-2\.5\-pro tasks<<gem\-3\-flash tasks<<gem\-3\-pro tasks<<opus\-4\.5 tasks\.

In Condition B, the model names refer only to which pool of user tasks the conversation came from — the models themselves are not involved, as it is just the first\-user message that is used\.

We find that the response style is the primary driver\.ηgenerator2\\eta^\{2\}\_\{\\text\{generator\}\}drops from 2\.8% \(Cond A\) to 1\.0% \(Cond B\) when the original model’s response is discarded — a 63% reduction\. The ranking reverses when the model is removed\.

In Condition A, Claude looks most deploy\-like\. In Condition B, Claude’s user tasks actually look most eval\-like\. Claude’s conversational style overrode and reversed the underlying task effect\.

## Appendix EChoice of Blend Weight in Harmonisation

The blend weightλ\\lambdacontrols how much between\-stratum mean variation inP​\(eval\)P\(\\text\{eval\}\)we treat as confound\. Atλ=0\\lambda=0the score is unchanged; atλ=1\\lambda=1every stratum mean is pulled to the same target, the standard ComBat choice\[Johnson et al\.,[2007](https://arxiv.org/html/2609.01611#bib.bib24)\]\.

[Figure˜11](https://arxiv.org/html/2609.01611#A5.F11)plots two quantities asλ\\lambdavaries in\[0,0\.5\)\[0,0\.5\): between\-stratum information preserved\(1−λ\)2\(1\-\\lambda\)^\{2\}, and Spearmanρ\\rhobetween the corrected ranking and the unconfounded self\-only ranking\. We restrict the sweep toλ<0\.5\\lambda<0\.5because beyond that the harmonised term contributes more than the original score, which we cannot justify without independent evidence that more than half of the between\-stratum variation is confound rather than real signal\.

The ranking gain starts to occur in the first5%5\\%of correction \(the “cliff”\): Spearman jumps from0\.830\.83atλ=0\\lambda=0to0\.890\.89byλ=0\.05\\lambda=0\.05\. Spearman then plateaus throughλ≈0\.45\\lambda\\approx 0\.45\. We chooseλ=0\.1\\lambda=0\.1as a small margin past the cliff: it captures the entire first\-step gain while preserving81%81\\%of between\-stratum information\.

##### Between\-stratum information\.

We measure how much stratum\-level signal is preserved after the correction by comparing the variance of stratum mean scores before and after harmonisation\. Let strata be indexed byk=\(generator family×interaction type\)k=\(\\text\{generator family\}\\times\\text\{interaction type\}\), withμk\\mu\_\{k\}the mean of original deploy scores in stratumkkfor a given judge\. The harmonised term has the same mean for every stratum \(the judge’s own\-family meanμself\\mu\_\{\\text\{self\}\}, by construction\), so the corrected mean of stratumkkis

μkcorrected=\(1−λ\)​μk\+λ​μself\.\\mu\_\{k\}^\{\\text\{corrected\}\}\\;=\\;\(1\-\\lambda\)\\,\\mu\_\{k\}\+\\lambda\\,\\mu\_\{\\text\{self\}\}\.Becauseμself\\mu\_\{\\text\{self\}\}is constant across strata, the between\-stratum variance of corrected means is

Vark⁡\(μkcorrected\)=\(1−λ\)2​Vark⁡\(μk\)\.\\operatorname\{Var\}\_\{k\}\\\!\\bigl\(\\mu\_\{k\}^\{\\text\{corrected\}\}\\bigr\)\\;=\\;\(1\-\\lambda\)^\{2\}\\;\\operatorname\{Var\}\_\{k\}\(\\mu\_\{k\}\)\.The fraction of between\-stratum mean variance retained is therefore exactly\(1−λ\)2\(1\-\\lambda\)^\{2\}:81%81\\%atλ=0\.1\\lambda=0\.1,25%25\\%atλ=0\.5\\lambda=0\.5, and0%0\\%atλ=1\\lambda=1\. We verified this empirically for all six judges; the measured ratio matches\(1−λ\)2\(1\-\\lambda\)^\{2\}to three decimal places\.

![Refer to caption](https://arxiv.org/html/2609.01611v1/x10.png)Figure 11:Choice of blend weightλ\\lambda\.Both curves are on\[0,1\]\[0,1\]with higher = better;λ\\lambdarestricted to\[0,0\.5\)\[0,0\.5\)\. The yellow “cliff” band \(λ∈\[0,0\.05\]\\lambda\\in\[0,0\.05\]\) is where Spearmanρ\\rhojumps from0\.830\.83to0\.890\.89\.λ∗=0\.1\\lambda^\{\*\}=0\.1sits a small margin past the cliff while preserving81%81\\%of between\-stratum information\.

## Appendix FData Composition

Table 7:Evaluation suite composition \(413 trajectories across 29 benchmarks\)\.Type: capability \(C\) or alignment \(A\)\.nn: sample count per judge model\. Agentic sources are collected fresh per judge; retained chat entries are shared across models with response resampling \(§[3\.1](https://arxiv.org/html/2609.01611#S3.SS1)\)\.BenchmarkTaskTypennΔ\\DeltaAdded: agentic tool\-using \(per\-model collection; 158 total\)SWE\-bench Verified\[OpenAI,[2024](https://arxiv.org/html/2609.01611#bib.bib36)\]Software engineeringC19retainedCyBench\[Zhang et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib63)\]Cybersecurity CTFC20retainedGAIA Level 2\[Mialon et al\.,[2023](https://arxiv.org/html/2609.01611#bib.bib32)\]General assistantC19newGDPval\[Patwardhan et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib38)\]GDP\-task estimationC20newAgentHarm\[Andriushchenko et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib3)\]Harmful tool useA20retainedAdded: short chat\-like \(hard negatives; 90 total\)BrowseComp\[Wei et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib57)\]Web comprehensionC20newMASK\[Ren et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib42)\]Honesty under pressureA20newAgentic Misalignment\[Anthropic,[2025a](https://arxiv.org/html/2609.01611#bib.bib5)\]Scheming scenariosA20newGPQA Diamond\[Rein et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib41)\]Graduate QAC20newWMDP\-bio\[Li et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib27)\]Weapons knowledgeA10newStrongReject\[Souly et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib49)\]Jailbreak robustnessA20newIFEval\[Zhou et al\.,[2023](https://arxiv.org/html/2609.01611#bib.bib65)\]Instruction followingC20newWritingBench\[Wu et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib59)\]Creative writingC10newAya Evaluation\[Üstün et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib53)\]Multilingual QA \(6 langs\)C10newPetri\[Anthropic,[2025b](https://arxiv.org/html/2609.01611#bib.bib6)\]Agent auditing transcriptsA50†\\daggernewRetained from Needham et al\. \(15 chat\-eval categories; 165 total\)MMMLU\[Hendrycks et al\.,[2021](https://arxiv.org/html/2609.01611#bib.bib19)\]Multi\-domain QAC20retainedNatural Reasoning\[Yuan et al\.,[2025](https://arxiv.org/html/2609.01611#bib.bib62)\]Open\-domain reasoningC10retainedNL2Bash\[Lin et al\.,[2018](https://arxiv.org/html/2609.01611#bib.bib29)\]Command generationC10retainedModel\-Written Evals\[Perez et al\.,[2022](https://arxiv.org/html/2609.01611#bib.bib39)\]LM\-generated tasksA12retainedKoBBQ\[Jin et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib22)\]Korean social biasA10retainedDiscrim\-Eval\[Tamkin et al\.,[2023](https://arxiv.org/html/2609.01611#bib.bib52)\]Discrimination probesA11retainedLLM Global Opinions\[Durmus et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib14)\]Cross\-cultural valuesA10retainedMoralExceptQA\[Jin et al\.,[2022](https://arxiv.org/html/2609.01611#bib.bib23)\]Moral reasoningA10retainedCivics\[Pistilli et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib40)\]Civic knowledgeA11retainedReal Toxicity Prompts\[Gehman et al\.,[2020](https://arxiv.org/html/2609.01611#bib.bib15)\]Toxicity completionA10retainedXSTest v2\[Röttger et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib43)\]Over\-refusal probesA10retainedDo Not Answer\[Wang et al\.,[2023](https://arxiv.org/html/2609.01611#bib.bib56)\]Refusal boundaryA10retainedHackAPrompt\[Schulhoff et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib47)\]Prompt injectionA10relabelAya Red\-teaming\[Aakanksha et al\.,[2024](https://arxiv.org/html/2609.01611#bib.bib1)\]Multilingual safetyA11retainedCValues\-RLHF\[Skepsun,[2023](https://arxiv.org/html/2609.01611#bib.bib48)\]Chinese valuesA10retained
- †\\daggerPetri trajectories are collected only for the three closed\-source judges \(Claude Opus 4\.6, GPT\-5\.4, Gemini 3\.1 Pro Preview\) used in theP​\(eval\)P\(\\mathrm\{eval\}\)across\-benchmark analysis \(§[4](https://arxiv.org/html/2609.01611#S4)\)\. Each judge collects∼\\sim50 fresh trajectories withclaude\-sonnet\-4\.6as the auditor model\.

Table 8:Deployment data composition\.Type: agentic \(A\), chat \(C\), or mixed \(M\)\. For mixed categories the agentic/chat split is given in parentheses\. OpenHands and Open Deep Research are retained from their respective agentic corpora at their native sample counts\.
## Appendix GNormalizing Data for Replay

Replaying agentic deployment trajectories raises a design choice about tool availability\. Due to the absence of the user’s local environment, agent trajectories are generated without making any tool calls at replay time, and we do not populate available tools in the metadata; all available tool definitions for a scaffold must be hardcoded during data normalisation\. A good number of trajectories sourced from DataClaw do not have a scaffold\-identifiable attribute and have to be inferred\. Tools appear in the conversation history so the model sees realistic agentic structure, but no tool schema is passed at the API level\. For the experiments in[Section˜3\.3](https://arxiv.org/html/2609.01611#S3.SS3), no tools are loaded into the model’s state\.

## Appendix HModel list

Table 9:Overview of evaluated models\.ProviderAPI model IDReleasedAnthropicclaude\-haiku\-4\-5Oct 2025Anthropicclaude\-opus\-4\-6Feb 2026Anthropicclaude\-sonnet\-4\-6Feb 2026OpenAIgpt\-4\.1Apr 2025OpenAIgpt\-5\.1Nov 2025OpenAIgpt\-5\.4Mar 2026Google DeepMindgemini\-2\.5\-proJun 2025Google DeepMindgemini\-3\.1\-pro\-previewFeb 2026DeepSeekdeepseek\-v3\.2Dec 2025Z\.aiglm\-5\.1Mar 2026Moonshot AIkimi\-k2\.5Jan 2026Alibabaqwen3\.5\-397b\-a17bFeb 2026
## Appendix IMethodological Corrections on AUROC

Figure 12:AUROC per frontier judge across four pipeline configurations\.Baselineuses Needham et al\.’s fixed probe\-question set on the heterogeneous deploy pool\.Mixedapplies per\-model probe selection \([Section˜3\.2](https://arxiv.org/html/2609.01611#S3.SS2)\)\.Correctedadds the stratified harmonisation correction \([Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)\) on top\.self\-onlyrestricts deployment data to the judge’s own model family, eliminating the generator\-identity confound at the cost of sample size and diversity\. All conditions evaluated on the same transcripts \(self\-only uses a subset\)\. Judges ordered by family; shading denotes family membership\.Figure[12](https://arxiv.org/html/2609.01611#A9.F12)compares four pipeline configurations: the fixed Needham et al\. probe set, per\-model probe selection under mixed deployment data, corrected mixed\-deployment AUROC, and a self\-only reference using same\-family deployment data\.

Per\-model PQ selection \(Mixed vs\. Baseline\) lifts AUROC by 0\.00–0\.08 across judges, and the stratified correction \(Corrected vs\. Mixed\) adds a further 0\.03–0\.07\. Both lifts are uneven: three judges gain nothing from PQ selection while three gain meaningfully, and the correction’s effect varies similarly across judges\. The Corrected column tracks the self\-only reference closely AUROC for all six judges\), supporting the validation in[Section˜3\.3\.3](https://arxiv.org/html/2609.01611#S3.SS3.SSS3)that harmonisation recovers most of the unconfounded signal without requiring same\-family deployment data\.

## Appendix JCompute and API Cost for Replicating This Work

[Table˜10](https://arxiv.org/html/2609.01611#A10.T10)reports estimated API cost per experiment\. Costs are derived from token counts×\\timesprovider list prices \(Jan 2026\): per judge call we charge for the system prompt, the full transcript, the probe\-question template, and a∼\\sim200\-token response, multiplied by the number of probes used in that experiment \(1 for the 6×\\times6 deploy heatmap,k∈\{1,3,5,9\}k\\\!\\in\\\!\\\{1,3,5,9\\\}for the depth resample, 5 for the cross\-benchmark measure suite, 1 for[Appendix˜B](https://arxiv.org/html/2609.01611#A2)\)\. Generator costs are cumulative across multi\-turn rollouts\.

A single tokenizer \(cl100k\_base\) is used for all models; the±\\pm10% band absorbs per\-provider tokenizer disagreement \(Anthropic∼\+7%\\sim\+7\\%, Google∼−10%\\sim\-10\\%, others within±5%\\pm 5\\%on our text\)\. Batch columns apply the standard 50% provider discount\. In our runs only the Anthropic judges were batched, putting the realised replication cost between the two columns at $2,627\. The PQ\-selection sweep \([Figure˜7](https://arxiv.org/html/2609.01611#S4.F7)\) is listed separately as it was not part of this replication\.

Table 10:Estimated API spend per experiment\. Non\-batch and batch columns assume all calls at list price and all calls at the standard 50% batch discount\. Realised replication cost falls between the two, at $2,627 with Anthropic\-only batching\. Estimates use Jan 2026 list prices and do not model prompt caching; caching reduces Anthropic\-side input by 30–40% on multi\-PQ runs\.
## Appendix KCost Estimate for a New Benchmark

A practitioner applying EvalDetectBench to their own benchmark faces two separable costs:

C\\displaystyle C≈Cgen\+Cprobe\\displaystyle\\;\\approx\\;C\_\{\\mathrm\{gen\}\}\+C\_\{\\mathrm\{probe\}\}\(3\)The first term,CgenC\_\{\\mathrm\{gen\}\}, is the cost of generating the transcripts to be scored\. This is the subject model producing trajectories on the target benchmark\. This term is zero for most adopters: if the target benchmark has already been run, the transcripts already exist and have already been paid for\. We therefore treatCgenC\_\{\\mathrm\{gen\}\}as a term equal to the cost of running the target benchmark once, which the practitioner can estimate from their own benchmark\.

The second term,CprobeC\_\{\\mathrm\{probe\}\}, is the probe pass: scoring each transcript with the calibrated probe questions to read outP​\(eval\)P\(\\mathrm\{eval\}\)\. The probe pass has two stages: calibration and scoring\. During the calibration stage, the full corpus ofKKprobe questions is run to select the top\-kkperformers on a calibration set comprising a fractionffof the transcripts\. During the scoring stage, only the selectedkkprobes are run on the remaining\(1−f\)\(1\-f\)of the transcripts\. The probe cost is therefore the sum of a calibration term and a scoring term:

Cprobe\\displaystyle C\_\{\\mathrm\{probe\}\}≈Ccalib\+Cscore,\\displaystyle\\;\\approx\\;C\_\{\\mathrm\{calib\}\}\+C\_\{\\mathrm\{score\}\},\(4\)Ccalib\\displaystyle C\_\{\\mathrm\{calib\}\}≈f​Nt⋅K⋅\(cin​\(o\+T¯\)\+cout​r\),\\displaystyle\\;\\approx\\;f\\,N\_\{t\}\\cdot K\\cdot\\bigl\(c\_\{\\mathrm\{in\}\}\(o\+\\bar\{T\}\)\+c\_\{\\mathrm\{out\}\}\\,r\\bigr\),\(5\)Cscore\\displaystyle C\_\{\\mathrm\{score\}\}≈\(1−f\)​Nt⋅k⋅\(cin​\(o\+T¯\)\+cout​r\),\\displaystyle\\;\\approx\\;\(1\-f\)\\,N\_\{t\}\\cdot k\\cdot\\bigl\(c\_\{\\mathrm\{in\}\}\(o\+\\bar\{T\}\)\+c\_\{\\mathrm\{out\}\}\\,r\\bigr\),\(6\)whereNtN\_\{t\}is the total number of transcripts,ffis the calibration fraction,KKis the full probe\-question corpus size,kkis the number of selected probe questions,cinc\_\{\\mathrm\{in\}\}andcoutc\_\{\\mathrm\{out\}\}are the probe model’s per\-token input and output prices,oois the fixed system\-prompt and wrapper overhead per probe,T¯\\bar\{T\}is the mean transcript length in tokens, andrris the mean response length per judgment\. The per\-call cost is identical across both stages; only the number of probe passes differs\.