Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents

# Counterfactual Evaluation Reveals Hidden Capability Profiles in Clinical LLMs and Agents
Source: [https://arxiv.org/html/2605.30590](https://arxiv.org/html/2605.30590)
###### Abstract\.

Two clinical AI systems can score nearly identically on coverage\-based rubrics yet behave radically differently when their patient inputs change: one updates its recommendations to match the new clinical signal, the other produces the same output regardless\. Standard evaluation cannot tell them apart\. We introduce theCausal Sensitivity Score \(CSS\), a pre\-registered interventional metric that mutates oncology tumor\-board cases along five clinically meaningful dimensions \(biomarker flips, prior\-treatment failures, biomarker strips, surgery\-status changes, stage perturbations\) and scores in\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}whether each model’s recommendations update in the pre\-registered correct direction\. Benchmarked against the published Consensus Match Score \(CMS\), a coverage\-based weighted recall, six frontier models from three labs in single\-shot inference on 224 cases rank in nearly opposite orders on the two metrics: all six change rank, the CMS\-worst model becomes CSS\-best, and one model that is upper\-mid on CMS is dead last on CSS\. We further surface a universal safety blind spot under our pre\-registered scoring rule: every frontier model fails on surgery\-status interventions \(≤17\.2%\\leq 17\.2\\%CSS on Family D\), a finding CMS does not expose\. The metric transfers directly to tool\-using agents: a ReAct\-style experiment shows tool use lifts CSS for five of six models \(\+2\.5\+2\.5to\+20\.3\+20\.3pp\), yet the lowest\-CSS model retrieves the same chart sections as the others and still does not update its recommendations, suggestive of a structural\-responsiveness deficit visible only under counterfactual evaluation\. Cross\-judge replication and three\-rater medical\-professional validation confirm the aggregate findings\. Interventional pre\-registered metrics like CSS complement coverage\-based evaluation for clinical AI agents: they capture responsiveness signal coverage cannot, and offer a candidate dense reward for future agentic RL\.

Counterfactual evaluation, causal sensitivity, agent evaluation, RL reward signals, LLM\-as\-judge, clinical AI

††copyright:none††conference:RLEval: Methods and RL Environments for Evaluating AI Agents; May 26, 2026; San Jose, CA## 1\.Introduction

LLMs and LLM\-powered agents are increasingly deployed in clinical AI \(treatment recommendation, triage, tumor\-board summarization\), where evaluation determines whether they ship\. The dominant paradigm scores outputs against reference behaviors via string similarity or LLM\-as\-judge rubrics\(Zhenget al\.,[2024](https://arxiv.org/html/2605.30590#bib.bib1); Liang and others,[2023](https://arxiv.org/html/2605.30590#bib.bib2)\)\. Both ask:*does the output look right?*Neither asks:*is the model updating its output for the right reasons?*An oncology AI that proposes FOLFIRINOX for a pancreatic case scores equally well on coverage\-based metrics whether the patient is treatment\-naive or whether it just always proposes FOLFIRINOX for pancreatic cases\.

Why this matters for agent evaluation\.Frontier deployments increasingly run as tool\-using agents that fetch patient information themselves\. Coverage\-based metrics on agent outputs face a sharper look\-right\-vs\.\-be\-right problem: an agent can make many tool calls, retrieve the right information, and still produce a recommendation that ignores what it found\. Interventional metrics are the natural fit because they grade*behavioral responsiveness*: did the agent’s output update appropriately when its tool returns were changed?

We introduce theCausal Sensitivity Score \(CSS\), a pre\-registered interventional metric\. For each intervention \(flipping HER2 status, injecting a prior failed therapy, removing biomarker mentions, toggling surgery status, etc\.\), CSS scores in\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}whether recommendations update in the pre\-registered correct direction\. We evaluate six frontier models from three labs \(OpenAIgpt\-5,gpt\-5\.4,gpt\-5\.4\-mini; Anthropicclaude\-opus\-4\-7,claude\-sonnet\-4\-6; xAIgrok\-4\.20\-0309\-reasoning\) in two settings: single\-shot LLM inference on all 224 expert\-annotated tumor\-board cases \(§[4](https://arxiv.org/html/2605.30590#S4)\), and a tool\-using ReAct agent\(Yaoet al\.,[2023](https://arxiv.org/html/2605.30590#bib.bib7)\)on a 100\-tuple Family D subset where interventions propagate through tool returns rather than the prompt \(§[4\.6](https://arxiv.org/html/2605.30590#S4.SS6)\)\.

Our findings:

- •Rank reversal \(single\-shot\)\.CMS and CSS rank the six models in nearly opposite orders \(ρ=−0\.49\\rho=\-0\.49; all six change rank\); the CMS\-worst model is CSS\-best, andgpt\-5\.4is upper\-mid on CMS \(rank 4 of 6\) but dead last on CSS\. All six models fail Family D \(surgery status\) at≤17\.2%\\leq 17\.2\\%under the pre\-registered scoring rule, a universal failure CMS does not expose\.
- •Agent transfer\.CSS transfers to tool\-using agents without modification\. Tool use lifts CSS for five of six models on Family D \(\+2\.5\+2\.5to\+20\.3\+20\.3pp\);gpt\-5\.4alone is essentially unchanged despite retrieving the same chart sections as the responsive five, suggestive of structural responsiveness rather than information access\.
- •Validation\.Cross\-judge replication \(uniform Opus\) preserves rank order \(ρ=\+1\.00\\rho=\+1\.00\); three\-rater medical\-professional annotation on a 100\-tuple subset confirms aggregate per\-family failure rates \(Family D: LLM mean0\.100\.10vs\. human mean0\.090\.09\)\.

## 2\.Method

### 2\.1\.Pre\-registered Intervention Catalog

We curate 12 interventions across five clinically motivated families \(Table[1](https://arxiv.org/html/2605.30590#S2.T1)\)\. Each is specified in a YAML catalog with five fields committed*before*any model is evaluated: applicability filter, mutation rule \(regexreplace/delete/insert\), pre\-registered expected output change,\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}scoring rule, and family label A–E \(full schema in App[A](https://arxiv.org/html/2605.30590#A1)\)\.

Table 1\.Pre\-registered intervention families\. “Eligible” counts the tuples that pass the catalog applicability filter; the per\-family scorednnin Table[3](https://arxiv.org/html/2605.30590#S4.T3)excludes regex no\-op mutations \(App[G](https://arxiv.org/html/2605.30590#A7)\), e\.g\., 73/153 eligible Family C tuples produced no\-ops and are dropped\.Pre\-registration rules out post\-hoc family selection and the “metric designed to fit the result” critique\. The catalog and scoring rules were authored by the author and have not yet undergone independent clinical vetting \(App[M](https://arxiv.org/html/2605.30590#A13)\)\.

### 2\.2\.Causal Sensitivity Score

For each \(modelmm, interventionii, casecc\) whereiiapplies, we generate baseline recommendations from the unmodified packet and intervened recommendations from the mutated packet\. A judge LLM receives both, the pre\-registered expected change, and the scoring rule, and emits\{0\.0,0\.5,1\.0\}\\\{0\.0,0\.5,1\.0\\\}for \(no change / acknowledged but unchanged / updated correctly\)\. Each tuple is processed through a two\-stage pipeline \(case summary→\\rightarrowrecommendations\); the judge sees only the recommendations\. CSS is the mean score across\(c,i\)\(c,i\)tuples, in aggregate and per\-family\.

Self\-judging avoidance\.We usegpt\-5\.4as the default judge withclaude\-opus\-4\-7as the judge whengpt\-5\.4is the model under test, consistent with prior self\-preference findings\(Zhenget al\.,[2024](https://arxiv.org/html/2605.30590#bib.bib1)\)\. §[4\.5](https://arxiv.org/html/2605.30590#S4.SS5)reports a uniform\-Opus replication\.

### 2\.3\.Generalization to Tool\-Using Agents

CSS requires only \(a\) inputs that admit pre\-registered counterfactual mutations and \(b\) a pre\-specifiable correct output\-update direction\. Both transfer directly to tool\-using agents, where mutations can be applied to tool returns \(e\.g\., flip a knowledge\-base retrieval\), to planning state, or to environment observations\. The single\-shot setting we report on is the cleanest experimental control; §[4\.6](https://arxiv.org/html/2605.30590#S4.SS6)runs the same protocol against ReAct agents and shows the metric and findings transfer\.

## 3\.Experimental Setup

Cohort\.224 oncology tumor\-board cases, each with a chronological patient packet \(median∼\\sim80k characters\) and ground\-truth treatment recommendations from a two\-round expert oncologist consensus protocol\. Treatments are labeledstrong\(clear consensus\),tacit\(tacit consensus\),mixed\(mixed evidence\), orrefusal\(clear rejection\)\.

Models\.The six frontier models listed in §[1](https://arxiv.org/html/2605.30590#S1), called via official APIs at default temperature\.

Comparison metric: Consensus Match Score \(CMS\)\.A published weighted recall against the oncologist\-consensus treatment list:

CMS=0\.6​Rstrong\+0\.2​Rtacit\+0\.15​\(1−Vrefusal\)\+0\.05​Pextra,\\text\{CMS\}=0\.6R\_\{\\text\{strong\}\}\+0\.2R\_\{\\text\{tacit\}\}\+0\.15\(1\-V\_\{\\text\{refusal\}\}\)\+0\.05P\_\{\\text\{extra\}\},withRRrecalls of strong/tacit\-consensus treatments,VrefusalV\_\{\\text\{refusal\}\}the rate of recommending a rejected treatment, andPextraP\_\{\\text\{extra\}\}judge\-rated plausibility of off\-list recommendations\. CMS measures*output coverage*\(does the recommendation overlap the consensus?\); CSS, by contrast, measures*input responsiveness*\.

## 4\.Results

### 4\.1\.Rank Disagreement Between CMS and CSS

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig1_rank_disagreement.png)Figure 1\.Rank disagreement between CMS and CSS for six frontier models from three labs\. Spearmanρ=−0\.49\\rho=\-0\.49\(exact permutationp=0\.36p=0\.36atn=6n=6, underpowered\); all six models change rank between the two metrics\. The CMS\-worst model \(grok\-4\.20\-reasoning\) is CSS\-best; the CMS\-best \(gpt\-5\) is fourth on CSS\.Table 2\.Headline: CMS vs\. CSS for six frontier models\. The two metrics rank the models in nearly opposite orders\.Table[2](https://arxiv.org/html/2605.30590#S4.T2)and Figure[1](https://arxiv.org/html/2605.30590#S4.F1)show the headline\. The six models cluster within13\.113\.1pp on CMS \(0\.4800\.480–0\.6100\.610\) but span16\.416\.4pp on CSS \(0\.3090\.309–0\.4730\.473\); Spearmanρ=−0\.49\\rho=\-0\.49\(exact permutationp=0\.36p=0\.36atn=6n=6, underpowered\)\.*All six*models change rank: the most striking flip isgrok\-4\.20\-reasoning\(CMS 6→\\rightarrowCSS 1\) andgpt\-5\(CMS 1→\\rightarrowCSS 4\)\. We treat the rank disagreement as a descriptive pattern; adding models will sharpen the inferential claim\.

### 4\.2\.Per\-Family Capability Profiles

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig2_per_family.png)Figure 2\.Per\-family CSS for six frontier models, small\-multiples view\. Different models win different families:claude\-opus\-4\-7on biomarker recognition \(A\),gpt\-5on biomarker stripping \(C\),grok\-4\.20on prior\-treatment \(B\) and surgery status \(D\)\. All six models fail catastrophically on Family D\.Table 3\.Per\-family CSS, six models, five intervention families\. Bold = winner per row\.Table[3](https://arxiv.org/html/2605.30590#S4.T3)decomposes CSS by intervention family\. Different models win different families:claude\-opus\-4\-7on biomarker flips \(A\);gpt\-5on biomarker stripping \(C\);grok\-4\.20on prior\-treatment failure \(B\) and surgery status \(D\)\.gpt\-5\.4is dead last on A, B, and D; on C it is second \(behindgpt\-5\)\. This per\-family decomposition reveals capability profiles aggregate metrics destroy\. \(Family E, stage\-perturbation, has onlyn=5n=5eligible cases; we do not draw conclusions from it, App[K](https://arxiv.org/html/2605.30590#A11)\.\)

### 4\.3\.Universal Failure Mode: Family D

The strongest model on surgery\-status interventions \(grok\-4\.20\) scores 17\.2%; the weakest \(gpt\-5\.4\) scores 3\.9%\.*Every*frontier model from*every*lab fails to update treatment recommendations correctly when surgery status flips\. This is a clinically meaningful safety finding \(treatment timing depends entirely on whether the patient was resected\) that CMS does not surface, because CMS only checks recommendation overlap with consensus, not behavioral change under a counterfactual\.

### 4\.4\.Score\-Distribution Diagnostics

Table 4\.Per\-model score distribution\. “Wrong” =0\.00\.0, “Partial” =0\.50\.5, “Correct” =1\.01\.0\.The score distribution \(Table[4](https://arxiv.org/html/2605.30590#S4.T4)\) concretizes the two failure modes:gpt\-5\.4sits at 60% wrong\-direction / 21% correct \(the “looks fine on CMS, structurally less responsive” case\);grok\-4\.20\-reasoningis the inverse, with the lowest wrong\-direction rate \(40\.8%\) and second\-highest correct rate \(35\.4%\)\.

### 4\.5\.Judge Sensitivity \(Cross\-Judge Replication\)

To rule out the asymmetric judge dispatch as a confound, we re\-judge every tuple withclaude\-opus\-4\-7as a single judge for*all*models on the same 4,727 tuples\. Rank order is identical under both configurations \(Spearmanρ=\+1\.00\\rho=\+1\.00\); per\-model inter\-judgeκ\\kappaon\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}is0\.610\.61–0\.690\.69for the five cross\-judged models\. Opus is a*stricter*judge than gpt\-5\.4: aggregate CSS drops44–77pp under Opus for the five non\-gpt\-5\.4 models\. Sincegpt\-5\.4was already Opus\-judged by default, the original asymmetric dispatch was biased*against*gpt\-5\.4\(held to a stricter standard than the other five, which were judged by the more lenient gpt\-5\.4\), so its persistent last\-place ranking is the harder of the two directions to obtain by chance\.gpt\-5\.4still ranks last under uniform Opus judging; the deficit is not a judge\-dispatch artifact \(full table in App[H](https://arxiv.org/html/2605.30590#A8)\)\. Construct validity holds across hard\-case, latent, and cancer\-category strata \(App[B](https://arxiv.org/html/2605.30590#A2)\)\.

### 4\.6\.Generalization Experiment: Tool\-Using Agent

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig5_agentic_comparison.png)Figure 3\.Tool use lifts CSS on Family D for 5 of 6 frontier models \(gain\+2\.5\+2\.5to\+20\.3\+20\.3pp\)\.gpt\-5\.4is essentially unchanged \(0\.050→0\.0500\.050\\rightarrow 0\.050\), consistent with a structural responsiveness deficit rather than information\-access failure\. Up to 100 case\-intervention tuples per model \(one sonnet row dropped as missing data,n=99n=99\)\.We re\-run Family D in a tool\-using agent setting: the agent has no chart in context and a singleread\_chart\_section\(section\)tool over ten chart sections \(demographics, diagnoses, biomarkers, medications, procedures, encounters, labs, vitals, allergies, overview\)\. It investigates ReAct\-style\(Yaoet al\.,[2023](https://arxiv.org/html/2605.30590#bib.bib7)\); interventions mutate the underlying packet so the change propagates through retrieval\. Judge and scoring are unchanged\. We evaluate all six models on 100 tuples each; all call the tool7\.07\.0–8\.38\.3times per case and queryproceduresat similar rates; they retrieve the same information\.

*Five of six models lift substantially under tool use*, with single\-shot→\\rightarrowtool\-using CSS gains of\+17\.0\+17\.0pp \(gpt\-5\),\+20\.3\+20\.3pp \(claude\-sonnet\-4\-6, rank 4→\\rightarrow1 on Family D\),\+12\.0\+12\.0pp \(claude\-opus\-4\-7\),\+6\.0\+6\.0pp \(grok\-4\.20\),\+2\.5\+2\.5pp \(gpt\-5\.4\-mini\);gpt\-5\.4is essentially unchanged \(0\.050→0\.0500\.050\\rightarrow 0\.050\)\. Tool use helps 19–34 of up to 100 tuples for responders, only 9 forgpt\-5\.4\(App[C](https://arxiv.org/html/2605.30590#A3)\)\. The asymmetry is substantive:gpt\-5\.4retrieves the same sections as the responsive five and still does not update, suggestive of structural responsiveness rather than information access\. Even the best tool\-using model \(0\.273\) sits well below 50%, so tool use mitigates but does not close the Family D blind spot\.

### 4\.7\.Human Validation

Three medical\-professional annotators independently scored 100 stratified \(case, intervention, model\) tuples on the same\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}scale, blinded to model identity and LLM judge scores\. Pairwise human\-human Cohen’sκ\\kapparanges0\.400\.40–0\.720\.72; LLM\-vs\-majorityκ=0\.46\\kappa=0\.46\(69/10069/100exact agreement, App[L](https://arxiv.org/html/2605.30590#A12)\)\. Per\-family LLM\-vs\-majorityκ\\kappais highest on A \(0\.670\.67\) and lowest on C \(0\.070\.07\); D is0\.160\.16\. Crucially, the*aggregate*CSS rates agree closely: Family D LLM mean=0\.10=0\.10vs\. human=0\.09=0\.09; C is0\.300\.30vs\.0\.330\.33\. Per\-row case\-level agreement is moderate, so headline claims should be read as population\-level rather than per\-case reliability\.

Annotators flagged 37/100 rows as medically incoherent \(29/49 in Family D\); the universal Family D blind spot survives restriction to coherent\-only rows \(App[J](https://arxiv.org/html/2605.30590#A10)\)\.

## 5\.Limitations

Population\-level claims\.Per\-row LLM\-humanκ\\kappafor D and C is moderate\-to\-low \(0\.160\.16,0\.070\.07\); headline claims should be read as population\-level CSS properties rather than per\-case reliability\.Regex\-based counterfactuals\.Mutations admit three failure modes that score a correctly\-refusing model0\.00\.0under the pre\-registered rule: semantic no\-ops \(regex changes text but not meaning\), incomplete propagation \(one chart section changed while others still imply the original fact, which the model may treat as a data\-entry error\), and medical incoherence \(29/49 D\-rows\)\.Sample size\.n=6n=6models renders the rank\-correlation test underpowered \(p=0\.36p=0\.36\); the validation subset isn=100n=100\(1515–4949per family\), so per\-familyκ\\kappacarries sampling noise\.Scope\.Findings are specific to oncology tumor\-board cases; CSS methodology generalizes elsewhere given a domain\-appropriate catalog\.Agent attribution\.The agent setting introduces variables besides tool use \(sectioned retrieval, ReAct prompting, 8 KB cap\); the gpt\-5\.4 persistence is consistent with structural responsiveness, not proof\. Catalog provenance and camera\-ready / future\-work plans are in App[M](https://arxiv.org/html/2605.30590#A13)\.

## 6\.Related Work

LLM\-as\-judge reliability is established in Zheng et al\.\(Zhenget al\.,[2024](https://arxiv.org/html/2605.30590#bib.bib1)\); counterfactual probing of LLM reasoning in Saparov & He\(Saparov and He,[2023](https://arxiv.org/html/2605.30590#bib.bib4)\); holistic and agent\-eval aggregation in HELM\(Liang and others,[2023](https://arxiv.org/html/2605.30590#bib.bib2)\)and AgentBench\(Liuet al\.,[2024](https://arxiv.org/html/2605.30590#bib.bib3)\); clinical LLM QA evaluation in Singhal et al\.\(Singhalet al\.,[2023](https://arxiv.org/html/2605.30590#bib.bib5)\); step\-level process rewards in Lightman et al\.\(Lightmanet al\.,[2024](https://arxiv.org/html/2605.30590#bib.bib6)\)\. CSS extends interventional probing to agentic clinical outputs, decomposes by intervention family, and operates as a candidate dense reward in the spirit of process supervision\.

## 7\.Conclusion

We introduce the Causal Sensitivity Score, a pre\-registered counterfactual metric, and apply it to six frontier models on 224 oncology cases \(single\-shot\) and a 100\-tuple Family D subset \(tool\-using ReAct\), benchmarked against the Consensus Match Score \(CMS\) coverage metric\. Under the pre\-registered scoring rule, CMS and CSS rank the models in nearly opposite orders: all six change rank, the CMS\-worst model becomes CSS\-best, andgpt\-5\.4sits at CMS rank 4 of 6 but is dead last on CSS\. Every model fails Family D surgery\-status interventions at≤17\.2%\\leq 17\.2\\%, a universal safety blind spot CMS does not expose\. In the agent setting, tool use lifts CSS for five of six models \(\+2\.5\+2\.5to\+20\.3\+20\.3pp\);gpt\-5\.4alone is essentially unchanged despite retrieving the same chart sections as the responsive five, a pattern consistent with structural responsiveness rather than information access\. Cross\-judge replication and three\-rater medical\-professional validation support the aggregate findings\.

We see CSS as a complement to CMS rather than a replacement: it captures input\-responsiveness signal coverage\-based metrics cannot, transfers without modification from single\-shot LLMs to tool\-using agents, and offers a candidate dense reward for future agentic RL experiments\. Together, these results indicate that the rank disagreement between CMS and CSS, the universal Family D failure, and the gpt\-5\.4 retrieval\-without\-update pattern are robust to judge choice and to human adjudication on the validation subset\. Whether a clinical AI agent’s recommendations update appropriately when the patient’s facts change is an evaluation question worth measuring directly, and one that coverage\-based metrics cannot answer\.

## References

- \[1\]P\. Lianget al\.\(2023\)Holistic evaluation of language models\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2605.30590#S1.p1.1),[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.
- \[2\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.International Conference on Learning Representations\.Cited by:[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.
- \[3\]X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2024\)AgentBench: evaluating LLMs as agents\.InInternational Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.
- \[4\]A\. Saparov and H\. He\(2023\)Language models are greedy reasoners: a systematic formal analysis of chain\-of\-thought\.International Conference on Learning Representations\.Cited by:[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.
- \[5\]K\. Singhal, T\. Tu, J\. Gottweis, R\. Sayres, E\. Wulczyn, L\. Hou, K\. Clark, S\. Pfohl, H\. Cole\-Lewis, D\. Neal,et al\.\(2023\)Towards expert\-level medical question answering with large language models\.arXiv preprint arXiv:2305\.09617\.Cited by:[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.
- \[6\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.30590#S1.p3.1),[§4\.6](https://arxiv.org/html/2605.30590#S4.SS6.p1.2)\.
- \[7\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica\(2024\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.30590#S1.p1.1),[§2\.2](https://arxiv.org/html/2605.30590#S2.SS2.p2.1),[§6](https://arxiv.org/html/2605.30590#S6.p1.1)\.

###### Acknowledgements\.

We thank Engy Ziedan and Wes Hopkins of the Protege Data Lab team for their review and assistance in producing and validating these results, and the medical\-professional annotators who graded the human\-validation subset\.

## Appendix APre\-Registered Intervention Catalog \(Excerpt\)

Each entry in the catalog has five fields, all committed before any model is evaluated\. We include two representative entries here; the full catalog \(12 entries across families A–E\) is in the artifact release\.

```
- id: A2_HER2_positive_to_negative
  family: A_biomarker_flip
  applicability:
    cancer_type: ["breast", "gastric"]
    requires_match_in_packet:
      "HER2[\\s-]?(positive|\\+)"
  mutation_type: replace
  mutation_pattern:
      "HER2[\\s-]?(positive|\\+)"
  mutation_replacement: "HER2-negative"
  expected_change:
    must_drop: ["trastuzumab", "T-DXd",
                "pertuzumab", "tucatinib"]
  scoring_rule:
    1.0: "all HER2-targeted recs dropped"
    0.5: "some dropped, or hedged in
          rationale only"
    0.0: "HER2-targeted recs unchanged"

- id: D1_remove_surgery_history
  family: D_surgery_status
  applicability:
    requires_match_in_packet:
      "\\[Procedure\\][^\\n]*?(resection|
       craniotomy|lumpectomy|mastectomy|
       gastrectomy|whipple|hepatectomy)"
  mutation_type: delete
  mutation_pattern:
      "\\[Procedure\\][^\\n]*?(resection|
       ...)[\\s\\S]*?(?=\\n---|\\n\\[)"
  expected_change:
    timing_must_shift:
      - adjuvant -> primary
      - post-operative -> definitive
    rationale_must_mention: "resection
        no longer documented"
  scoring_rule:
    1.0: "timing shifted as specified"
    0.5: "rationale acknowledges but
          recommendations unchanged"
    0.0: "no change"
```

## Appendix BSub\-Population Stratification

CSS by latent\-consensus flag and by cancer\-category folder, six models\. The gpt\-5\.4 deficit persists across all strata\. Grok leads in 4 of 5 strata; in the mixed\-evidence stratum,gpt\-5\.4\-mininarrowly edges Grok \(0\.5290\.529vs\.0\.4710\.471\)\.

Table 5\.Sub\-population CSS \(all six models, all interventions\)\.
## Appendix CTool\-Using Agent: Per\-Case Effects

Per\-model breakdown of how tool use changes the score on each of the 100 matched \(case, intervention\) tuples on Family D\.

Table 6\.Per\-case effect of tool use on Family D\. “Help” = tool\-using CSS strictly higher; “Hurt” = tool\-using CSS strictly lower; “Same” = unchanged\.![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig7_per_case_scatter.png)Figure 4\.Per\-case effect of tool use\. Each point is one \(case, intervention\) tuple; x\-axis is single\-shot CSS, y\-axis is tool\-using CSS\. Points above the diagonal are tuples where tool use helped\. The mass of points above the diagonal for five models, and the near\-diagonal cluster forgpt\-5\.4, makes the asymmetry visible at the per\-case level\.
## Appendix DTool\-Using Agent: Score Distribution Shift

Wrong / partial / correct breakdown for the agentic experiment, single\-shot LLM \(top half\) vs\. tool\-using ReAct \(bottom half\), six models\. The wrong\-direction rate drops66–3030pp for the five responsive models; only33pp forgpt\-5\.4\.

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig8_agentic_score_distribution.png)Figure 5\.Score\-distribution shift under tool use\. Stacked bars show wrong/partial/correct breakdown for each model in single\-shot \(left\) vs\. tool\-using \(right\) settings on Family D\. The wrong\-direction rate drops66–3030pp for the five responsive models;gpt\-5\.4barely moves\.
## Appendix ETool\-Use Pattern

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig6_tool_usage_pattern.png)Figure 6\.Tool\-call distribution across the ten chart sections, averaged across 100 case\-intervention tuples per model\. All six models call the tool7\.07\.0–8\.38\.3times per case and queryprocedures\(where surgery history lives\) at similar rates; they retrieve roughly the same information\. The CSS asymmetry across models is therefore not explained by retrieval differences\.
## Appendix FCompute and Cost

- •Single\-shot baselines\.6 models×\\times224 cases = 1,344 baseline inferences \(gpt\-5 and gpt\-5\.4\-mini reused from prior delivery; remaining 4 models run fresh\)\. Baseline CMS judge: 6×\\times224 = 1,344 judge calls\.
- •Single\-shot interventions\.6 models×\\times789 mutated tuples = 4,734 intervention\-model inferences\. CSS judge: 4,734 calls \(with self\-judge override forgpt\-5\.4routed to Opus\)\.
- •Tool\-using experiment\.6 models×\\times\(50 baseline cases \+ 100 intervention tuples\) = 900 trajectories, average 7\.5 tool calls each,≈\\approx7,600 tool\-augmented LLM calls\. CSS judge: 600 calls\.

## Appendix GImplementation Notes

Mutation no\-ops\.73 of 862 eligible \(case, intervention\) tuples produced no\-op mutations \(the regex pattern was eligible at the catalog level but did not match the source packet\); these are silently dropped from CSS scoring rather than scored as 0\.

Tool\-using agent\.ReAct loop with a 8\-call cap \(no model hit the cap; mean 7\.0–8\.3\)\. Chart segmentation is regex\-based on tagged entries \(\[Medication\],\[Procedure\],\[Encounter\],\[Lab Results\],\[Vitals\]\) and on header\-delimited blocks \(PATIENT DEMOGRAPHICS,CONDITIONS,ALLERGIES\)\. Each section is capped at 8 KB\.

Anthropic vs\. OpenAI tool\-call dispatch\.The agent harness has provider\-specific paths: OpenAI/xAI use the chat\-completionstool\_callsformat; Anthropic uses the Messages APItool\_useblock format\. Both wrap a single shared tool definition, so the agent sees identical tool semantics regardless of provider\.

## Appendix HJudge Sensitivity \(Cross\-Judge Replication\) Table

Per\-model CSS under the default judge dispatch \(gpt\-5\.4for five models, Opus override forgpt\-5\.4as model\) versus a uniform Opus\-only judge on the same 4,727 valid tuples\. Rank order is identical under both \(ρjudges=\+1\.00\\rho\_\{\\text\{judges\}\}=\+1\.00\); inter\-judgeκ\\kappaon\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}labels is0\.610\.61–0\.690\.69\(substantial\)\. Note thatgpt\-5\.4’s row is not a judge\-swap: the default dispatch already routed it to Opus, so the\+0\.5\+0\.5pp shift reflects stochastic rerun variability between two Opus passes, not a different judge model\.

## Appendix IAgentic Comparison: Single\-Shot vs\. Tool\-Using ReAct

Family D matched 100 case\-intervention tuples per model\.

## Appendix JHuman Validation: Family D Coherence Analysis

Of4949Family D tuples in the human\-annotated subset,2929\(59%59\\%\) were flagged by at least one of three annotators as creating a medically incoherent scenario \(e\.g\., curative\-intent surgical resection inserted into a metastatic patient’s chart\)\. All values below are computed on the4949\-tuple human\-annotated subset \(*not*the full306306\-tuple Family D\);nfulln\_\{\\text\{full\}\}is the number of D\-tuples for that model in the subset andncohn\_\{\\text\{coh\}\}is the subset after removing annotator\-flagged rows\. Per\-model coherent\-only CSS is the same or lower than the full\-subset CSS for five of six models \(gpt\-5\.4\-miniis the exception\):

The catalog issue is concentrated in insert\-type mutations \(D2 inserts a surgical resection; E1 inserts metastasis\) on cases whose disease state is incompatible with the inserted event\. For the camera\-ready we will tighten D2 and E1 applicability filters with¬\\lnotmetastaticnegative\-match guards\.

## Appendix KFamily E \(Stage Perturbation,n=5n=5\)

Reported for completeness; per\-family numbers are noisy atn=5n=5and we draw no conclusions\. Per\-model CSS:opus\-4\-70\.800,sonnet\-4\-60\.900,gpt\-51\.000,gpt\-5\.40\.300,gpt\-5\.4\-mini0\.900,grok\-4\.20\-reasoning1\.000\.

## Appendix LLLM Judge vs\. Human Majority Scatter

![Refer to caption](https://arxiv.org/html/2605.30590v1/figures/fig9_llm_vs_human_scatter.png)Figure 7\.Case\-level LLM judge vs\. human\-majority score on the 100\-tuple validation subset\. Points jittered for visibility on the\{0,0\.5,1\.0\}\\\{0,0\.5,1\.0\\\}grid; color by intervention family; counts shown in disagreement cells with≥5\\geq 5tuples\. Exact agreement is69/10069/100\. Most disagreement clusters at the partial\-credit boundary \(LLM scores 0\.0 where humans score 0\.5, and vice versa\)\.
## Appendix MCamera\-Ready Refinements and Future Work

The workshop camera\-ready window is three days\. We scope camera\-ready refinements to what is achievable in that window and label the rest as future work\.

Catalog authorship\.The intervention catalog and per\-intervention scoring rules were authored by the author and have not yet undergone independent clinical vetting\. The rules therefore reflect one researcher’s interpretation of what each intervention should change about a model’s output, which may differ from a practicing clinician’s expectations and may miss clinically meaningful update patterns not enumerated in the catalog\. An oncologist review of rule coverage and clinical alignment is a camera\-ready item\.

Camera\-ready\.\(i\)*Tightened insert\-filter for D2 and E1\.*We will add a¬\\lnotmetastaticnegative\-match guard to D2 and E1 applicability filters so curative\-intent resection and stage\-perturbation insertions are not applied to already\-metastatic charts, and re\-run the Family D and E subsets on the filtered case set\. \(ii\)*Refusal\-credit branch in the scoring rule\.*We will add a0\.50\.5credit when the model’s rationale explicitly identifies the inserted scenario as medically incoherent and leaves recommendations unchanged, and re\-judge the affected rows\. \(iii\)*Bootstrap CIs over cases\.*We will report case\-resampled95%95\\%intervals on per\-model CSS and on the rank ordering to characterize stability of the headline reversals\.

Future work \(beyond camera\-ready\)\.\(i\)*Retrieval\-controlled agent baseline*that feeds each model the exact retrieved snippets without the agent loop, to isolate tool\-use structural\-responsiveness effects from prompt\-structure effects\. \(ii\)*Semantic\-no\-op and incomplete\-propagation audits*per mutated chart, requiring a separate validator and a sample\-based calibration step\. \(iii\)*Second human\-annotation round*with expanded adjudication and refined per\-intervention scoring rules to lift per\-rowκ\\kappain Families C and D\. \(iv\)*Additional frontier models*, including Gemini and additional reasoning variants\. \(v\)*Bootstrap CIs over judges*\(additional uniform\-judge passes\), which are higher cost given per\-pass compute\.

Counterfactual validity \(deep dive\)\.CSS as computed treats every text\-level mutation as a clinically meaningful counterfactual, but three failure modes can produce false low scores: \(i\)*semantic no\-op*mutations where regex\-level text changes do not change clinical meaning; \(ii\)*incomplete propagation*where one part of the chart is mutated while other parts continue to imply the original fact \(e\.g\., deleting a Procedure block while encounter notes still refer to “post\-op”\); \(iii\)*medical incoherence*where the inserted scenario is medically impossible \(e\.g\., curative resection on a metastatic patient\), surfaced through human validation as29/4929/49Family D rows\. A model that correctly refuses to update on any of these is scored 0\.0 under the pre\-registered rule\. The camera\-ready filter tightening and refusal\-credit branch above address \(iii\) directly; \(i\) and \(ii\) require the semantic\-validity audits in future work\.

Pre\-registration tradeoff\.Pre\-registration commits us to a scoring rule before observing outputs, which protects against post\-hoc cherry picking but cannot distinguish causal\-sensitivity failure from correct refusal of incoherent or non\-actionable inputs\. We report results under the original rule and flag the medical\-incoherence fraction transparently in §[4\.7](https://arxiv.org/html/2605.30590#S4.SS7); quantifying semantic no\-ops and incomplete\-propagation fractions requires the semantic\-validity audits listed above\.