Jev能否评估放射学报告？评估System One模型在临床事实性方面的表现

# Can Jev Judge Radiology Reports?Evaluating a System One Model for Clinical Factuality
Source: [https://arxiv.org/html/2609.27607](https://arxiv.org/html/2609.27607)
Jiaju Huang, Hao Yang, Xinyu Ma, Xinglong Liang,Kunyan Cai, Junqiang Ma, Shaobin Chen, Yue Sun, Tao TanIntelligent Medical Computing Laboratory, Faculty of Applied Sciences,Macao Polytechnic UniversityDepartment of Radiology, Netherlands Cancer Institute,Amsterdam, the NetherlandsDepartment of Radiology and Nuclear Medicine,Radboud University Medical Center, Nijmegen, the Netherlands\*Corresponding author: Tao Tan,[taotanjs@gmail\.com](mailto:[email protected])

September 23, 2026

###### Abstract

An AI\-generated radiology report can resemble a physician’s report while omitting an abnormality, adding an unsupported finding, or reversing its presence\. Measuring these factual differences is essential for evaluating report generators\. We study Jev, a System One decision model, as a simple, low\-cost judge of agreement with physician\-written reference reports\. Our evaluator checks whether each statement is supported by the other report and combines these judgments in both directions to capture unsupported claims and omissions\. A single\-question configuration reaches Kendall correlations of 0\.573 on RadEvalX and 0\.398 on RadEvalExpert with expert error counts, outperforming an open natural language inference judge under matched decomposition and aggregation\. One support question per statement retains similar expert agreement to seven while using 43–45% fewer judgment input tokens\. Judgment API calls cost less than US$0\.03 per 100 report pairs, excluding local decomposition\. In a separate controlled\-error test, Jev detects false negation with an AUROC of 0\.977\. Local RadMatch achieves stronger agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset\. Finding\-count and error\-scope analyses show that benchmark agreement reflects report size and error definitions as well as medical error detection\. These results support Jev as a practical judgment component for measuring factual differences in generated radiology reports and identify where more elaborate evaluation remains valuable\.

*Keywords*Radiology report evaluation⋅\\cdotJev⋅\\cdotSystem One⋅\\cdotFactuality

## 1Introduction

A generated radiology report can read naturally and still change the medical story\. It may add a finding that the physician’s report never mentions, leave out an abnormality, or describe it on the wrong side\. Changing “a pleural effusion” to “no pleural effusion” preserves most words while reversing the finding\. For researchers developing report generators, these are the differences an evaluator needs to capture: which assertions disagree with the reference, and which findings have been left out?

This task calls for many small judgments\. For each statement, does the reference support it, contradict it, or leave it unaddressed? Checking in the reverse direction asks whether the generated report covers the reference’s findings\. Jev offers a direct interface for making these decisions: its System One model returns typed answers with probabilities that software can combine into a report score\[[16](https://arxiv.org/html/2609.27607#bib.bib11)\]\. We ask whether Jev can measure factual differences in generated radiology reports with just one support question per statement\.

We study Jev as a reference\-based report evaluator\. We extract statements from both reports, judge each against the other report, and aggregate their discrepancies \(Figure[1](https://arxiv.org/html/2609.27607#S3.F1)\)\. This design builds on report decomposition and bidirectional entailment, particularly RadFact\[[2](https://arxiv.org/html/2609.27607#bib.bib2)\]\. We examine what Jev contributes within this framework and how much complexity is needed to use it effectively\. Expert\-annotated reports test whether the scores reflect overall error burden; controlled errors reveal which medical changes the judge detects\. Component comparisons, token costs, and repeated calls test whether the evaluator is economical and stable enough for repeated report\-generator comparisons\.

The results support a simple configuration\. One support question per statement produces expert agreement close to that of the seven\-question design while reducing judgment API input cost by 43–45%\. The resulting judgments cost under three cents per hundred report pairs, excluding local decomposition\. Under matched decomposition and aggregation, this configuration outperforms an open natural language inference \(NLI\) judge\. In a separate sentence\-level test, Jev detects false negation—turning a present finding into an absent one—with an AUROC of 0\.977\. Its report rankings also repeat closely across five runs\. Local RadMatch achieves higher agreement on clinically significant errors in both expert datasets and on total errors in the shared RadEvalExpert subset, showing where its more elaborate procedure remains useful\.

A report that describes more findings offers more opportunities for errors\. An evaluator can therefore correlate with expert error counts by capturing report size, even without identifying each error correctly\. Jev exceeds a finding\-count baseline in both expert datasets\. We complement this comparison with candidate ordering within the same examination and with a separation of factual errors from spelling and repetition perturbations\. The latter changes the relative ranking of Jev and NLI\.

Our contributions are threefold\. First, we evaluate Jev’s ability to measure factual discrepancies between generated and reference radiology reports, using expert error counts and controlled medical changes\. Second, we identify a single\-question configuration that combines competitive expert agreement with low judgment cost\. Third, we characterize what the evaluator captures and how it behaves through analyses of report size, error scope, calibration, and repeatability\.

## 2Related Work

#### Radiology report metrics\.

Report evaluation spans lexical overlap, contextual similarity, extracted clinical structure, and learned error scores\. BERTScore compares contextual token representations\[[22](https://arxiv.org/html/2609.27607#bib.bib7)\]; RadGraph measures agreement in clinical entities and relations\[[11](https://arxiv.org/html/2609.27607#bib.bib6)\]; RaTEScore uses radiology entities and their attributes\[[23](https://arxiv.org/html/2609.27607#bib.bib5)\]\. RadCliQ combines metric signals to predict expert error counts\[[21](https://arxiv.org/html/2609.27607#bib.bib13)\]\. The RadEval framework provides common implementations and an expert\-annotated evaluation dataset\[[18](https://arxiv.org/html/2609.27607#bib.bib4)\]\. These methods differ in their targets: a similarity ratio and a predicted number of errors need not rank reports identically\.

#### Language models as report judges\.

GREEN identifies and describes clinically meaningful discrepancies between reports\[[14](https://arxiv.org/html/2609.27607#bib.bib1)\]\. RadFact decomposes reports into single\-finding phrases and verifies entailment in both directions, yielding logical precision and recall\[[2](https://arxiv.org/html/2609.27607#bib.bib2)\]\. Its evidence extraction also supports grounded evaluation\. RadMatch uses finding\-level matching and attribute analysis to produce auditable discrepancy measures\[[6](https://arxiv.org/html/2609.27607#bib.bib3)\]\. Our work adopts the established decomposition\-and\-entailment structure and examines Jev as a judgment engine\. CRIMSON adds context\-sensitive clinical significance and radiologist preference benchmarks\[[1](https://arxiv.org/html/2609.27607#bib.bib23)\]\. AtomiMed separates disease and attribute facts and verifies them through bidirectional questions across imaging modalities\[[17](https://arxiv.org/html/2609.27607#bib.bib24)\]\.

Recent work also examines how to configure medical judges\. VERT compares judge models, prompts, reasoning settings, ensembling, and fine\-tuning on RadEval and RaTE\-Eval\[[3](https://arxiv.org/html/2609.27607#bib.bib14)\]\. Its analysis also discusses how finding counts affect normalized scores\. RadSEM combines atomic findings, contradiction\-constrained matching, and deterministic weighted scoring\[[19](https://arxiv.org/html/2609.27607#bib.bib15)\]\. It evaluates sensitivity to graded semantic corruption and synonym–antonym rewrites\. We focus on Jev’s typed probability outputs, its medical error detection, and the cost of a minimal question set\. Our count baseline and error\-scope comparisons examine what expert\-agreement scores reward\.

#### Evaluating the evaluators\.

General LLM\-judge studies document position and verbosity biases alongside agreement with human preferences\[[24](https://arxiv.org/html/2609.27607#bib.bib16)\]\. Length\-controlled AlpacaEval uses regression adjustments to reduce preference for longer responses\[[8](https://arxiv.org/html/2609.27607#bib.bib17)\]\. In summarization, SummEval includes output length among its simple comparison features\[[9](https://arxiv.org/html/2609.27607#bib.bib18)\]\. Radiology benchmarks add a related measurement issue: reports with more findings offer more opportunities for counted errors\. RaTE\-Eval already normalizes sentence errors by the number of potential errors\[[23](https://arxiv.org/html/2609.27607#bib.bib5)\]\. We measure the strength of a finding\-count baseline on expert report benchmarks and contrast pooled error burden with candidate ordering within the same examination\.

#### Probability quality and benchmark interpretation\.

Probabilistic outputs allow soft scores and task\-specific calibration, but probability values alone do not establish calibration\[[10](https://arxiv.org/html/2609.27607#bib.bib8)\]\. We compare Jev’s raw probabilities with those of a public DeBERTa NLI checkpoint\[[12](https://arxiv.org/html/2609.27607#bib.bib12)\], and evaluate post\-hoc calibration on held\-out studies\. Expert datasets such as ReXVal\[[20](https://arxiv.org/html/2609.27607#bib.bib9)\], RadEvalX\[[5](https://arxiv.org/html/2609.27607#bib.bib20)\], and RadEvalExpert\[[18](https://arxiv.org/html/2609.27607#bib.bib4)\]measure report\-level agreement, whereas ReXErr\[[15](https://arxiv.org/html/2609.27607#bib.bib10)\]provides controlled local errors\. We use both forms of evidence and test how simple count signals and error definitions affect their conclusions\.

## 3Jev as a Radiology Report Judge

### 3\.1Task and statement units

Given a generated reportCCand a physician\-written reference reportRR, we assess which findings disagree and which are left unaddressed\. Letc1,…,cnc\_\{1\},\\ldots,c\_\{n\}andr1,…,rmr\_\{1\},\\ldots,r\_\{m\}denote the statements extracted from the two reports\.

We implement two granularities\. L0 applies sentence segmentation and removes pure section headings\. L1 uses a local Qwen3\.5\-9B model to split each sentence into atomic findings while preserving negation, uncertainty, laterality, severity, measurements, and comparison phrases\. Invalid JSON falls back to the original sentence\. The reported OneQ configuration uses L1\. L0 is a separate ablation\.

Figure 1:Jev checks what a report gets wrong and what it leaves out\.OneQ compares each finding with the complete opposite report in both directions\. The constructed example visualizes a denied effusion and an omitted enlarged heart; a normal\-sized heart is shown for visual comparison\. Jev returns support, contradiction, and noncoverage probabilities\. Scores sum candidate\-side contradiction/noncoverage with reference\-side noncoverage \(Eq\.[2](https://arxiv.org/html/2609.27607#S3.E2)\); lower is better\. Model illustration adapted from[TypeSafe AI](https://typesafe.ai/)\.
### 3\.2One question, three outcomes

For a statementxxand an opposing reportYY, Jev returns

\(ps​\(x,Y\),pc​\(x,Y\),pu​\(x,Y\)\),ps\+pc\+pu≈1,\\bigl\(p\_\{s\}\(x,Y\),p\_\{c\}\(x,Y\),p\_\{u\}\(x,Y\)\\bigr\),\\qquad p\_\{s\}\+p\_\{c\}\+p\_\{u\}\\approx 1,\(1\)for*supported*,*contradicted*, and*not addressed*\. We use the returned probabilities at API precision without renormalization\. The prompt asks whether the report supports the claim about an imaging finding\. Its criteria distinguish compatible descriptions, incompatible presence or attributes, and findings absent from the report text\. For example, “left pleural effusion” is contradicted by an explicit denial of left pleural effusion; a report that never addresses effusion supplies no support for the claim\. Here,*not addressed*denotes missing textual evidence\.

The same question is applied to every candidate statement againstRRand every reference statement againstCC\. OneQ therefore means one question per statement–report input; a report pair requires multiple such judgments\.

### 3\.3Bidirectional aggregation

Our primary score sums candidate\-side discrepancies and reference\-side noncoverage:

E⁡\(C,R\)=∑j=1n\[pc​\(cj,R\)\+pu​\(cj,R\)\]\+∑i=1mpu​\(ri,C\)\.E\(C,R\)=\\sum\_\{j=1\}^\{n\}\\left\[p\_\{c\}\(c\_\{j\},R\)\+p\_\{u\}\(c\_\{j\},R\)\\right\]\+\\sum\_\{i=1\}^\{m\}p\_\{u\}\(r\_\{i\},C\)\.\(2\)Lower values indicate greater agreement\. The asymmetry avoids adding a second contradiction term from the reference direction\. This is a soft discrepancy count over extracted statements; its numerical agreement with expert error counts is evaluated separately from its ranking ability\.

We also report soft precision, recall, and F1:

P=1n​∑jps​\(cj,R\),Rs=1m​∑ips​\(ri,C\),Fs=2​P​RsP\+Rs\.P=\\frac\{1\}\{n\}\\sum\_\{j\}p\_\{s\}\(c\_\{j\},R\),\\qquad R\_\{s\}=\\frac\{1\}\{m\}\\sum\_\{i\}p\_\{s\}\(r\_\{i\},C\),\\qquad F\_\{s\}=\\frac\{2PR\_\{s\}\}\{P\+R\_\{s\}\}\.\(3\)The normalized count isE/\(n\+m\)E/\(n\+m\)\. Empty\-side means and undefined harmonic means are set to zero in the implementation\. We report counts against count labels and use the normalized and F1 variants to examine agreement as a proportion of report content\.

### 3\.4Configurations and open judge

OneQuses only the support question\.Fulladds four negation/uncertainty questions, a discrepancy\-type question, and a clinical significance question\. Full also applies a negation\-consistency correction: if the claim and report negation probabilities differ by more than 0\.5 whileps\>0\.5p\_\{s\}\>0\.5, half ofpsp\_\{s\}is moved topcp\_\{c\}\. OneQ has no negation outputs, so this correction is inactive\. Significance and error\-type outputs enter their own variants and do not otherwise alter Eq\.[2](https://arxiv.org/html/2609.27607#S3.E2)\.

We test sentence\-only splitting, shorter criteria, relevance filtering, alternative discrepancy weights, and significance weighting\. Relevance filtering asks whether each report sentence concerns the claim’s anatomical entity and retains probabilities above 0\.3\. An empty selection yields an empty opposing report\. We evaluate each component separately against Full\.

The open judge usesDeBERTa\-v3\-large\-mnli\-fever\-anli\-ling\-wanli\[[12](https://arxiv.org/html/2609.27607#bib.bib12)\]\. The opposing report is the premise and the statement is the hypothesis\. Entailment, contradiction, and neutral probabilities map tops,pc,pup\_\{s\},p\_\{c\},p\_\{u\}\. It shares L1 statements and Eq\.[2](https://arxiv.org/html/2609.27607#S3.E2)with OneQ; its tokenizer uses a maximum sequence length of 512 tokens\. This comparison isolates the judge within the evaluated configuration, including its input handling\.

## 4Experimental Design

### 4\.1Datasets and targets

Expert agreement\.RadEvalX contains 100 IU\-Xray reference reports, each paired with one M2Tr\-generated report\[[5](https://arxiv.org/html/2609.27607#bib.bib20),[7](https://arxiv.org/html/2609.27607#bib.bib22)\]\. Two radiologists reached consensus on significant and insignificant error counts across eight categories\. The sample contains 80 abnormal and 20 normal reports, selected after stratification by RadCliQ scores\. We evaluate total and clinically significant error counts over all eight categories; a six\-category sensitivity analysis excludes the two uncertainty categories \(Appendix[A](https://arxiv.org/html/2609.27607#A1)\)\. RadEvalExpert contains 624 pairs from 208 studies, with three model\-generated candidates per reference\[[18](https://arxiv.org/html/2609.27607#bib.bib4)\]\. It covers findings and impression sections and supplies total and significant error labels\. We keep each study together during uncertainty estimation\.

Controlled errors\.The ReXErr sentence test set contains 19,859 examples\[[15](https://arxiv.org/html/2609.27607#bib.bib10)\]\. Excluding 346 neutral rewrites leaves 19,513 examples: 8,723 injected\-error sentences and 10,790 unchanged sentences\. Each candidate sentence is judged against its complete original report\. This experiment uses the raw support probabilities from the Full question set, without atomic decomposition or negation correction\. It tests sentence\-level detection and calibration separately from the report\-level OneQ pipeline\. We retain all 11 positive error types and also analyze eight factual\-error types and three language\-perturbation types \(typos, homophones, and repetition\), grouped by the dataset’s intervention labels\.

We use 2,000 sampled ReXErr report pairs for cost and report\-level diagnostics\. RaTE\-Eval supplies supplementary sentence, paragraph, and synthetic comparison tests\[[23](https://arxiv.org/html/2609.27607#bib.bib5)\]\. Their targets and processing are described in Appendix[A](https://arxiv.org/html/2609.27607#A1)\.

### 4\.2Comparators and implementation

The main comparators include GREEN, RadCliQ, RadGraph F1, BERTScore, Temporal F1, open NLI, and the finding\-count baselinen\+mn\+m\. RadEval supplies the standard metric implementations\[[18](https://arxiv.org/html/2609.27607#bib.bib4)\]\. We also run RadFact and RadMatch on all 100 RadEvalX pairs and a fixed random subset of 200 RadEvalExpert pairs\. All subset comparisons use exactly the same valid pairs\.

Both generative baselines use a local Qwen3\.8\-27B\-FP8 backend on two 32 GB RTX 5090 D GPUs\. This replaces the original papers’ judge choices\. On the RadEvalExpert subset, RadMatch retried 52 pairs and used validation fallback on 11; its default backend configuration sampled at temperature 1\.0\. RadFact used temperature zero\. On RadEvalX, RadMatch used validation fallback on 8 of 100 pairs; all pairs received a final score\. Both baselines disabled thinking\. RadFact used an 8,192\-token context, and RadMatch used 16,384 tokens to accommodate the complete few\-shot prompt, with a 4,096\-token output cap\. GREEN’s default decoding is greedy\.

Our implementation, S1Fact, uses Jev versionjev\-1\.13\.0\. L1 uses temperature\-zero local inference\. Repeated\-call experiments use fresh judgments with decomposition held fixed\. Additional prompt and implementation details appear in Appendix[B](https://arxiv.org/html/2609.27607#A2)\.

Repeatability\.On a fixed random subset of 50 RadEvalX pairs, we repeat both Full and OneQ five times, holding decomposition fixed and bypassing the judgment cache\. A matched experiment uses a fixed selection of 50 pairs from the 200\-pair RadEvalExpert subset, covering 48 studies\. We run Jev–Full and Jev–OneQ five times each on the same 616 extracted findings, bypassing the judgment cache\. Five runs per baseline setting evaluate GREEN with greedy decoding, RadFact at temperature zero, and RadMatch at temperatures zero and one\. The latter is the local backend’s default\. Jev holds L1 decomposition fixed; the baselines repeat their local evaluation procedures\. We measure exact score agreement, between\-run ranking consistency, and variation in correlation with expert total\-error counts\.

### 4\.3Statistics and cost

Primary agreement is Kendall’sτb\\tau\_\{b\}, oriented in advance so higher is better: error scores retain their sign against error labels, and similarity scores have their sign reversed\. We do not take absolute correlations\. Confidence intervals use percentile bootstrap over studies: 2,000 resamples for RadEvalX and 1,000 resamples for RadEvalExpert\. Pairwise comparisons first select rows where both methods and the target are finite, then resample the same studies for both methods\. Intervals describe individual comparisons and are not adjusted for multiple testing\.

On RadEvalExpert, which supplies multiple candidates per study, we separately compute within\-study ordering,

W=Nconcordant−NdiscordantNall​within​\-​study​candidate​pairs\.W=\\frac\{N\_\{\\mathrm\{concordant\}\}\-N\_\{\\mathrm\{discordant\}\}\}\{N\_\{\\mathrm\{all\\ within\\mbox\{\-\}study\\ candidate\\ pairs\}\}\}\.\(4\)Ties contribute zero to the numerator and remain in the denominator\. This statistic is not tie\-correctedτb\\tau\_\{b\}\. It holds the reference report fixed within each comparison, while candidate lengths may still differ\. Bootstrap copies of a study contribute independent copies of its pair counts\.

Controlled\-error analyses report AUROC, recall at 0\.5, ten\-bin expected calibration error \(ECE\), and Brier score\. Error\-scope intervals use 400 study resamples\. Post\-hoc logistic and isotonic calibration fit on one half of the studies and are evaluated on the other half\. Raw and calibrated probabilities are compared on the same holdout\.

We calculate API cost from input\-token usage at $0\.042 per million tokens; output tokens are free\[[16](https://arxiv.org/html/2609.27607#bib.bib11)\]\. Configuration comparisons count every judgment at that price, including judgments served from the local cache\. This measures the cost of the complete judgment workload at a common price\. Local decomposition cost is separate\. We report request latency, per\-report judgment time, and batch completion time separately\.

## 5Results

We first test whether report scores reflect expert\-assessed factual errors and which medical changes Jev detects\. We then examine the complexity and cost needed for these judgments, followed by analyses of score interpretation, calibration, and repeatability\.

### 5\.1Report scores track expert\-assessed factual differences

The report\-level test asks whether greater disagreement with the reference corresponds to more errors identified by radiologists\. Jev–OneQ reachesτb=0\.573\\tau\_\{b\}=0\.573on RadEvalX and0\.3980\.398on RadEvalExpert \(Table[1](https://arxiv.org/html/2609.27607#S5.T1)\)\. Full reaches 0\.582 and 0\.397, respectively\. With the same statements and count aggregation, OneQ improves over open NLI by0\.2360\.236\[0\.131, 0\.345\] and0\.0750\.075\[0\.043, 0\.106\] on total errors\. Its advantage over NLI also holds for significant\-error labels\. OneQ exceeds RadCliQ on total errors in both datasets\. On RadEvalX, its difference from GREEN is0\.1250\.125\[−0\.006\-0\.006, 0\.256\], while significant\-error correlation is nearly equal \(0\.346 versus 0\.347\)\. On RadEvalExpert, OneQ exceeds both GREEN and RadCliQ on both targets\. Appendix[C](https://arxiv.org/html/2609.27607#A3)reports the complete standard metric suite\.

Table 1:Agreement with expert error counts\. Signs are oriented so higher is better; brackets are 95% study\-bootstrap intervals\. Panel A uses all pairs in each dataset for the first nine rows; local RadFact and RadMatch also cover all 100 RadEvalX pairs\. Panel B restricts every method to the same 200 RadEvalExpert pairs\. RadFact and RadMatch use local Qwen3\.8\-27B\-FP8 at temperatures 0 and 1, respectively \(Section[4\.2](https://arxiv.org/html/2609.27607#S4.SS2)\)\.
RadMatch’s advantage depends on the error target\. On RadEvalX, its total\-error correlation is 0\.580, close to OneQ’s 0\.573 and Full’s 0\.582\. The paired OneQ–RadMatch difference is−0\.007\-0\.007\[−0\.137\-0\.137, 0\.114\]\. For significant errors, RadMatch reaches 0\.491 versus OneQ’s 0\.346; the difference is−0\.145\-0\.145\[−0\.294\-0\.294,−0\.004\-0\.004\]\. Full also trails RadMatch on this target\. Local RadFact reaches 0\.477 on total errors and 0\.398 on significant errors; neither paired difference from OneQ excludes zero\.

RadMatch leads on both error targets in the shared RadEvalExpert subset\. Across these 200 pairs, its total\-error correlation is 0\.453, compared with 0\.322 for OneQ and 0\.320 for Full\. OneQ minus RadMatch is−0\.131\-0\.131\[−0\.226\-0\.226,−0\.044\-0\.044\] for total errors and−0\.226\-0\.226\[−0\.320\-0\.320,−0\.137\-0\.137\] for significant errors\. Both intervals exclude zero\. On the same subset, OneQ exceeds local RadFact for total errors by0\.1710\.171\[0\.052, 0\.294\]\. Its significant\-error difference is0\.0740\.074\[−0\.048\-0\.048, 0\.195\]\.

### 5\.2Detecting changes in medical assertions

The sentence\-level test asks whether Jev recognizes a specific change in medical meaning\. Using Full’s raw support outputs in the ReXErr sentence experiment, Jev detects false negation with an AUROC of 0\.977 \[0\.972, 0\.982\] and recall of 91\.9% at threshold 0\.5\. It also detects changed locations \(0\.984\), changed severity \(0\.975\), and added medical devices \(0\.989\)\. These are concrete factual differences: turning “a left pleural effusion” into “no left pleural effusion” reverses presence, while changing “left” to “right” changes location\. Appendix[F](https://arxiv.org/html/2609.27607#A6)reports all error types and a separate language stress test\.

Table 2:Error scope changes the ordering of Jev and open NLI on ReXErr\. Each row uses the same 10,790 unchanged negative sentences\. Positive counts differ by scope\. Differences are Jev minus NLI; intervals use study\-block bootstrap\.The ranking changes when the test includes spelling and repetition \(Table[2](https://arxiv.org/html/2609.27607#S5.T2)\)\. Over all injected errors, NLI has higher AUROC than Jev, 0\.8969 versus 0\.8896\. Restricting positives to factual\-error types reverses that ordering: Jev reaches 0\.9680 and NLI reaches 0\.9614, with a paired difference of0\.00660\.0066\[0\.004, 0\.009\]\. For language perturbations, both scores fall, and NLI leads by 0\.0674\. These 1,638 language perturbations constitute 18\.8% of all positive examples\.

Jev’s AUROC for added typos is 0\.562, and its recall for repetition is 0\.003\. Low sensitivity to meaning\-preserving edits suits factuality evaluation\. Some language perturbations can also change medical meaning, so we interpret this grouping by error type\. Assessing overall writing quality requires broader error coverage\.

### 5\.3One question retains agreement at lower cost

OneQ retains similar expert agreement with fewer input tokens \(Figure[2](https://arxiv.org/html/2609.27607#S5.F2)\)\. Relative to Full, its total\-error correlation changes by−0\.009\-0\.009\[−0\.026\-0\.026, 0\.008\] on RadEvalX and\+0\.001\+0\.001\[−0\.002\-0\.002, 0\.004\] on RadEvalExpert\.

The token savings are consistent across datasets\. On RadEvalX, Full uses 9,695\.17 input tokens per pair and OneQ uses 5,374\.55, a 44\.56% reduction\. On RadEvalExpert, the corresponding means are 10,797\.20 and 6,151\.13, a 43\.03% reduction\. OneQ’s judgment API cost is approximately $0\.000226 and $0\.000258 per pair, excluding local decomposition\. In practical terms, the judgment API costs about 2\.3–2\.6 cents per hundred report pairs\. Appendix[E](https://arxiv.org/html/2609.27607#A5)gives the calculation and latency measurements\.

Figure 2:Component changes relative to Full, evaluated independently\. Points show changes in total\-errorτb\\tau\_\{b\}; bars are paired 95% study\-bootstrap intervals\. API input cost uses the same price for all judgments, including cached ones\. RadEvalX filtering cost includes sentence\-selection and judgment calls; RadEvalExpert filtering cost is incomplete and omitted\. Removing negation correction or changing aggregation reuses the Full judgments and leaves their API cost unchanged\.Other additions bring no consistent improvement\. Replacing atomic findings with sentences changes correlation by−0\.014\-0\.014on RadEvalX and−0\.015\-0\.015on RadEvalExpert; both intervals include zero\. Removing negation correction or shortening the support criteria also yields small changes\. On RadEvalX, filtering retains 32\.2% of report sentences and leaves 11\.2% of statement inputs with an empty report\. Correlation changes by−0\.023\-0\.023\[−0\.057\-0\.057, 0\.011\], while total judgment\-and\-filtering input cost rises by 56\.1%\. On RadEvalExpert, the correlation change is about−0\.009\-0\.009\. This filtering rule offers no observed scoring advantage\.

Significance weighting reduces total\-error correlation on RadEvalX by0\.1290\.129\[0\.050, 0\.208\]\. Its significant\-error correlation rises from 0\.343 to 0\.396, but the paired interval for that gain includes zero\. On RadEvalX, jointly halving candidate and reference noncoverage weights raises correlation by0\.0530\.053\[0\.005, 0\.104\]\. This is an exploratory choice from the nine\-setting grid evaluated on the same labels; the main comparisons retain the default weights\. On RadEvalExpert, halving candidate noncoverage weight raises correlation by0\.0130\.013\[−0\.006\-0\.006, 0\.035\]\. Appendix[D](https://arxiv.org/html/2609.27607#A4)reports the complete grid\. The practical outcome is a simple configuration: L1 statements, one support question, and default count aggregation\. The sentence\-only and OneQ variants were tested separately; their combined effect remains an open comparison\.

### 5\.4Judgment cost in a concrete example

Figure[3](https://arxiv.org/html/2609.27607#S5.F3)shows what a judgment call buys: a decision about whether a report supports a finding\. On three constructed examples covering contradiction, support, and an unmentioned finding, Jev and GPT\-5\.6 Luna return the same classifications in all five runs\. Jev supplies the three class probabilities directly; Luna generates a JSON label with reasoning disabled\. For the three findings together, mean API cost is $0\.00005044 for Jev and $0\.00018520 for Luna\. Median completion time is 1\.129 s and 3\.569 s, respectively, using three concurrent requests per method\. Appendix[E\.1](https://arxiv.org/html/2609.27607#A5.SS1)gives the protocol and pricing calculation\.

Figure 3:Same findings, two ways to judge\. Both methods receive the same reference report, findings, and three\-way support definitions\. Outputs are from the first run on this constructed example\. Cost is the five\-run mean of token usage priced at official standard rates; time is the median wall time to finish all three requests\.
### 5\.5Interpreting agreement beyond finding counts

We compare Jev with a finding\-count baseline to assess the contribution of its support judgments beyond report size\. Counting extracted findings alone reachesτb=0\.288\\tau\_\{b\}=0\.288on RadEvalX and 0\.261 on RadEvalExpert\. This baseline never checks whether a statement is correct; it counts the opportunities for disagreement\. OneQ improves over it by0\.2850\.285\[0\.157, 0\.416\] and0\.1370\.137\[0\.080, 0\.197\], respectively\. Jev’s judgments improve agreement beyond the finding\-count baseline\.

Figure 4:Finding counts and the scope of comparison\. Left and center: pooled correlation with expert total\-error counts on the two datasets\. Right: ordering of RadEvalExpert candidates for the same study, using Eq\.[4](https://arxiv.org/html/2609.27607#S4.E4)\. All methods use the complete respective dataset\. Intervals resample studies\. RadEvalX has only one candidate per study and cannot support within\-study ranking\. The two statistics answer different questions; within\-study comparisons fix the reference report while candidate lengths can still vary\.On RadEvalExpert, OneQ also orders candidates describing the same examination better than finding count in point estimate: 0\.205 versus 0\.101 \(Figure[4](https://arxiv.org/html/2609.27607#S5.F4)\)\. This view asks which candidate is better for a particular examination, complementing the pooled ranking of error burden across different examinations\.

The common RadEvalExpert subset contains 69 within\-study candidate comparisons\. Here, both Jev configurations score 0\.246 for total errors, compared with 0\.043 for RadMatch\. The paired difference is0\.2030\.203\[−0\.011\-0\.011, 0\.406\]\. For significant errors, the difference is0\.0430\.043\[−0\.169\-0\.169, 0\.250\]\. Both intervals include zero, leaving the within\-study difference between Jev and RadMatch uncertain\.

Aggregation is another part of the explanation\. On this same subset, Full’s count score reaches 0\.320, whereas its soft F1 reaches 0\.111\. The open NLI judge similarly changes from 0\.294 with counts to 0\.082 with F1\. Summation retains report\-size information; F1 normalizes support within each report\. Both aggregation and judge choice influence agreement with error\-count labels\. The size of the aggregation effect also depends on the dataset: on RadEvalX, Full reaches 0\.582 with counts and 0\.561 with soft F1, a much smaller difference whose paired interval includes zero\.

### 5\.6Calibration makes probabilities useful for thresholding

Jev’s raw sentence\-level probabilities have ECE 0\.1192 on the full test set, compared with 0\.1045 for NLI\. On the separate calibration holdout, Jev’s ECE falls from 0\.1235 to 0\.0077 after isotonic calibration\. NLI improves from 0\.1072 to 0\.0063 under the same procedure\. These results support task\-specific calibration before using probability thresholds\. On RadEvalX, Full’s highest score bin averages 8\.53, while the corresponding expert count averages 5\.30\. Full tracks error burden, but its raw score overestimates expert error counts at the upper end \(Appendix[G](https://arxiv.org/html/2609.27607#A7)\)\.

### 5\.7Rankings remain stable across repeated judgments

Jev’s report rankings repeat closely in both configurations\. On the same 50 RadEvalExpert pairs used for the baselines, five uncached runs give median between\-run rank agreement of 0\.990 for OneQ and 0\.986 for Full \(Table[3](https://arxiv.org/html/2609.27607#S5.T3)\)\. The corresponding standard deviations in expert correlation are 0\.00381 and 0\.00126\. Every report pair has some score variation across the five runs; the resulting rankings remain close\. Median score SD is 0\.042 for OneQ and 0\.047 for Full\. On 50 independently selected RadEvalX pairs, five runs each give correlation SDs of 0\.00745 for Full and 0\.00737 for OneQ\. Appendix[G\.1](https://arxiv.org/html/2609.27607#A7.SS1)details the protocols and score variation\.

The baseline runs show how much inference settings can change this behavior \(Table[3](https://arxiv.org/html/2609.27607#S5.T3)\)\. With the local backend’s default sampling, RadMatch gives identical scores across five runs for only 10% of report pairs\. Setting temperature to zero raises that proportion to 82% and between\-run rank consistency from 0\.662 to 0\.965\. The standard deviation of expert correlation falls from 0\.03582 to 0\.00975\. On this subset, mean expert correlation also rises from 0\.400 to 0\.468\.

Table 3:Repeatability on the same 50 RadEvalExpert pairs, with five runs per setting\. Identical is the fraction of pairs with exactly the same score in all five runs\. Rank consistency is the median Kendallτb\\tau\_\{b\}across the ten pairs of runs\. Correlation SD measures variation inτb\\tau\_\{b\}against expert total\-error counts\. Jev repeats uncached judgments with L1 decomposition fixed; baseline pipelines are rerun\. RadFact uses logical F1; RadMatch uses actionable\-error counts\.GREEN gives exactly the same scores in all five runs\. RadFact’s default temperature\-zero setup gives identical F1 scores for 88% of pairs\. Repeatability varies across these pipelines and their decoding settings\.

## 6Discussion

Jev combines support judgments in both directions to score unsupported candidate statements and missing reference findings\. The resulting report scores track expert\-assessed error burden\. This gives researchers a low\-cost way to score collections of generated reports, with individual support judgments available for inspection\.

OneQ retains expert agreement close to Full while reducing judgment API input cost by 43–45%\. Additional questions and filtering offer no consistent gain, and the matched NLI comparison shows that the choice of judge matters within this simple configuration\. The result is a practical starting point for report\-generator evaluation: extract statements, ask one support question, and aggregate discrepancies in both directions\.

The choice also depends on the evaluation target\. On RadEvalX, Jev and local RadMatch have similar total\-error correlations, while RadMatch agrees more closely with clinically significant errors\. On the shared RadEvalExpert subset, RadMatch leads on both targets\. Researchers prioritizing significant errors therefore have a reason to use the more elaborate evaluator\. Jev’s contribution is an inexpensive judgment component that detects medically meaningful discrepancies through a simple support question\.

Discrepancy counts retain information about report size; normalized scores describe the proportion of content in agreement\. An evaluator that tolerates a harmless spelling change may suit factuality assessment while missing an error in a writing\-quality test\. Finding\-count baselines, comparisons on shared samples, and breakdowns by error scope help connect a metric’s benchmark score to its intended use\. Where multiple candidates describe the same examination, within\-study ordering adds a direct view of choosing between those candidates\.

Jev’s aggregate rankings repeat closely across runs, while GREEN returns identical scores\. RadMatch’s rank consistency rises from 0\.662 to 0\.965 when sampling is disabled on the same inputs\. Decoding settings affect repeatability, while calibration improves probability estimates for selecting statements for further inspection\.

### 6\.1Limitations and future work

Our evaluation measures agreement with reference text\. Reference reports can contain errors or omit relevant findings; establishing correctness against the image and validating clinical use require separate studies\. Report\-level category scores do not yet reliably distinguish location, severity, and false\-finding errors\. Probability thresholds benefit from calibration on the intended data, and language\-quality assessment requires coverage beyond factual support\.

The study covers one Jev version, small expert benchmarks, and controlled injected errors\. RadEvalX uses one generator and RadCliQ\-stratified sampling, with 80 abnormal and 20 normal reports\. Broader generator and case distributions would test generality\. Sentence\-level detection and calibration use Full’s support outputs; direct evaluation of OneQ’s local probabilities is a further test\. Repeating decomposition would extend Jev’s fixed\-decomposition repeatability study to the full pipeline\. Finally, matched end\-to\-end cost and timing measurements would complement the measured judgment costs and three\-finding API demonstration when comparing deployment options\.

## 7Conclusion

Evaluating a generated radiology report requires identifying what it changes, adds, and leaves out relative to the reference\. Jev scores these discrepancies through simple statement\-level support judgments combined in both directions\. Its single\-question configuration achieves competitive agreement with expert error counts at a judgment API cost below three cents per hundred report pairs, excluding local decomposition\. Controlled\-error tests demonstrate sensitivity to several medically meaningful changes, and repeated calls produce closely agreeing report rankings\. One support question per statement yields expert agreement close to the seven\-question design at 43–45% lower judgment cost\. This simple Jev configuration offers a low\-cost way to evaluate factual differences in generated reports\.

## Data and Code Availability

RadEvalX and ReXErr\-v1 \(both version 1\.0\.0\) are available through PhysioNet\[[5](https://arxiv.org/html/2609.27607#bib.bib20),[15](https://arxiv.org/html/2609.27607#bib.bib10)\]; RadEvalExpert and RaTE\-Eval are available from their respective dataset distributions\[[18](https://arxiv.org/html/2609.27607#bib.bib4),[23](https://arxiv.org/html/2609.27607#bib.bib5)\], subject to each dataset’s terms of use\. Exact questions and implementation details appear in Appendix[B](https://arxiv.org/html/2609.27607#A2)\. The analysis scripts and aggregate results are not yet publicly available; we are preparing them for release\.

## References

- \[1\]M\. Baharoon, T\. Heintz, S\. Raissi, M\. Alabbad, M\. Alhammad, H\. AlOmaish, S\. E\. Kim, O\. Banerjee, and P\. Rajpurkar\(2026\)CRIMSON: A Clinically\-Grounded LLM\-Based Metric for Generative Radiology Report Evaluation\.arXiv preprint arXiv:2603\.06183\.External Links:[Link](https://arxiv.org/abs/2603.06183)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p1.1)\.
- \[2\]S\. Bannur, K\. Bouzid, D\. C\. Castro, A\. Schwaighofer, A\. Thieme, S\. Bond\-Taylor, M\. Ilse, F\. Perez\-Garcia, V\. Salvatelli, H\. Sharma, F\. Meissen, M\. Ranjit, S\. Srivastav, J\. Gong, N\. C\. F\. Codella, F\. Falck, O\. Oktay, M\. P\. Lungren, M\. T\. Wetscherek, J\. Alvarez\-Valle, and S\. L\. Hyland\(2024\)MAIRA\-2: Grounded Radiology Report Generation\.arXiv preprint arXiv:2406\.04449\.External Links:[Link](https://arxiv.org/abs/2406.04449),2406\.04449Cited by:[§1](https://arxiv.org/html/2609.27607#S1.p3.1),[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p1.1)\.
- \[3\]\(2026\)VERT: Reliable LLM Judges for Radiology Report Evaluation\.arXiv preprint arXiv:2604\.03376\.External Links:2604\.03376,[Link](https://arxiv.org/abs/2604.03376)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p2.1)\.
- \[4\]A\. Calamida, F\. Nooralahzadeh, M\. Rohanian, K\. Fujimoto, M\. Nishio, and M\. Krauthammer\(2023\)Radiology\-Aware Model\-Based Evaluation Metric for Report Generation\.arXiv preprint arXiv:2311\.16764\.External Links:[Link](https://arxiv.org/abs/2311.16764)Cited by:[Appendix A](https://arxiv.org/html/2609.27607#A1.p1.1)\.
- \[5\]A\. R\. Calamida, F\. Nooralahzadeh, M\. Rohanian, M\. Nishio, K\. Fujimoto, and M\. Krauthammer\(2024\)Radiology Report Generation Models Evaluation Dataset For Chest X\-rays \(RadEvalX\), version 1\.0\.0\.PhysioNet\.External Links:[Document](https://dx.doi.org/10.13026/tp88-q278),[Link](https://physionet.org/content/rad-eval-x/1.0.0/)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2609.27607#S4.SS1.p1.1),[Data and Code Availability](https://arxiv.org/html/2609.27607#Sx1.p1.1)\.
- \[6\]C\. Corbiere, L\. Machado, A\. Charley, B\. Callard, P\. Manceron, and C\. Dancette\(2026\)RadMatch: Auditable Radiology Report Evaluation via Finding\-Level Matching\.arXiv preprint arXiv:2609\.01470\.External Links:[Link](https://arxiv.org/abs/2609.01470),2609\.01470Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]D\. Demner\-Fushman, M\. D\. Kohli, M\. B\. Rosenman, S\. E\. Shooshan, L\. Rodriguez, S\. Antani, G\. R\. Thoma, and C\. J\. McDonald\(2016\)Preparing a collection of radiology examinations for distribution and retrieval\.Journal of the American Medical Informatics Association23\(2\),pp\. 304–310\.External Links:[Document](https://dx.doi.org/10.1093/jamia/ocv080)Cited by:[§4\.1](https://arxiv.org/html/2609.27607#S4.SS1.p1.1)\.
- \[8\]Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto\(2024\)Length\-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators\.arXiv preprint arXiv:2404\.04475\.External Links:2404\.04475,[Link](https://arxiv.org/abs/2404.04475)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px3.p1.1)\.
- \[9\]A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. Radev\(2020\)SummEval: Re\-evaluating Summarization Evaluation\.arXiv preprint arXiv:2007\.12626\.External Links:2007\.12626,[Link](https://arxiv.org/abs/2007.12626)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px3.p1.1)\.
- \[10\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On Calibration of Modern Neural Networks\.arXiv preprint arXiv:1706\.04599\.External Links:[Link](https://arxiv.org/abs/1706.04599),1706\.04599Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1)\.
- \[11\]S\. Jain, A\. Agrawal, A\. Saporta, S\. Q\. Truong, D\. N\. Duong, T\. Bui, P\. Chambon, Y\. Zhang, M\. P\. Lungren, A\. Y\. Ng, C\. P\. Langlotz, and P\. Rajpurkar\(2021\)RadGraph: Extracting Clinical Entities and Relations from Radiology Reports\.arXiv preprint arXiv:2106\.14463\.External Links:[Link](https://arxiv.org/abs/2106.14463),2106\.14463Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]M\. Laurer\(n\.d\.\)DeBERTa\-v3\-large\-mnli\-fever\-anli\-ling\-wanli: Model card\.Note:Checkpoint documentation; accessed September 22, 2026External Links:[Link](https://huggingface.co/MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1),[§3\.4](https://arxiv.org/html/2609.27607#S3.SS4.p3.1)\.
- \[13\]OpenAI\(2026\)API Pricing\.Note:Accessed September 22, 2026\. GPT\-5\.6 Luna standard short\-context pricingExternal Links:[Link](https://developers.openai.com/api/docs/pricing)Cited by:[§E\.1](https://arxiv.org/html/2609.27607#A5.SS1.p3.1)\.
- \[14\]S\. Ostmeier, J\. Xu, Z\. Chen, M\. Varma, L\. Blankemeier, C\. Bluethgen, A\. E\. Michalson, M\. Moseley, C\. Langlotz, A\. S\. Chaudhari, and J\. Delbrouck\(2024\)GREEN: Generative Radiology Report Evaluation and Error Notation\.arXiv preprint arXiv:2405\.03595\.External Links:[Link](https://arxiv.org/abs/2405.03595),2405\.03595,[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.21)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p1.1)\.
- \[15\]V\. Rao, S\. Zhang, J\. Acosta, S\. Adithan, and P\. Rajpurkar\(2025\)ReXErr\-v1: Clinically Meaningful Chest X\-Ray Report Errors Derived from MIMIC\-CXR\.Note:PhysioNet, version 1\.0\.0External Links:[Link](https://physionet.org/content/rexerr-v1/1.0.0/),[Document](https://dx.doi.org/10.13026/9dns-vd94)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2609.27607#S4.SS1.p2.1),[Data and Code Availability](https://arxiv.org/html/2609.27607#Sx1.p1.1)\.
- \[16\]TypeSafe AI\(2026\)Jev 1\.13: Models and System One documentation\.Note:Model jev\-1\.13\.0; accessed September 22, 2026External Links:[Link](https://docs.typesafe.ai/models)Cited by:[§E\.1](https://arxiv.org/html/2609.27607#A5.SS1.p3.1),[§1](https://arxiv.org/html/2609.27607#S1.p2.1),[§4\.3](https://arxiv.org/html/2609.27607#S4.SS3.p4.1)\.
- \[17\]Y\. Wang, W\. Chang, S\. Jiang, S\. Gao, X\. Zhang, R\. Yuan, W\. Cao, B\. Shi, L\. Zhang, Z\. Liu, and J\. Zhang\(2026\)AtomiMed: Hierarchical Atomic Fact\-Checking for Universal Clinical\-Aware Medical Report Evaluation\.arXiv preprint arXiv:2606\.31292\.External Links:[Link](https://arxiv.org/abs/2606.31292)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p1.1)\.
- \[18\]J\. Xu, X\. Zhang, J\. Abderezaei, J\. Bauml, R\. Boodoo, F\. Haghighi, A\. Ganjizadeh, E\. Brattain, D\. Van Veen, Z\. Meng, D\. Eyre, and J\. Delbrouck\(2025\)RadEval: A framework for radiology text evaluation\.arXiv preprint arXiv:2509\.18030\.External Links:[Link](https://arxiv.org/abs/2509.18030),2509\.18030Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1),[§4\.1](https://arxiv.org/html/2609.27607#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2609.27607#S4.SS2.p1.1),[Data and Code Availability](https://arxiv.org/html/2609.27607#Sx1.p1.1)\.
- \[19\]Z\. Yang, Z\. Liu, J\. Fei, W\. Tang, S\. Quan, J\. Zhao, and J\. Xu\(2026\)RadSEM: A Finding\-by\-Finding Metric for Clinical Consistency in Radiology Reports\.arXiv preprint arXiv:2606\.17062\.External Links:2606\.17062,[Link](https://arxiv.org/abs/2606.17062)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px2.p2.1)\.
- \[20\]F\. Yu, M\. Endo, R\. Krishnan, I\. Pan, A\. Tsai, E\. P\. Reis, E\. Kaiser Ururahy Nunes Fonseca, H\. Lee, Z\. Shakeri, A\. Ng, C\. Langlotz, V\. K\. Venugopal, and P\. Rajpurkar\(2023\)Radiology Report Expert Evaluation \(ReXVal\) Dataset\.Note:PhysioNet, version 1\.0\.0External Links:[Link](https://physionet.org/content/rexval-dataset/1.0.0/),[Document](https://dx.doi.org/10.13026/2fp8-qr71)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px4.p1.1)\.
- \[21\]F\. Yu, M\. Endo, R\. Krishnan, I\. Pan, A\. Tsai, E\. P\. Reis,et al\.\(2022\)Evaluating Progress in Automatic Chest X\-Ray Radiology Report Generation\.medRxiv\.Note:Reference linked by the ReXVal dataset documentationExternal Links:[Document](https://dx.doi.org/10.1101/2022.08.30.22279318)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px1.p1.1)\.
- \[22\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2019\)BERTScore: Evaluating Text Generation with BERT\.arXiv preprint arXiv:1904\.09675\.External Links:[Link](https://arxiv.org/abs/1904.09675),1904\.09675Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]W\. Zhao, C\. Wu, X\. Zhang, Y\. Zhang, Y\. Wang, and W\. Xie\(2024\)RaTEScore: A Metric for Radiology Report Generation\.arXiv preprint arXiv:2406\.16845\.External Links:[Link](https://arxiv.org/abs/2406.16845),2406\.16845Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2609.27607#S4.SS1.p3.1),[Data and Code Availability](https://arxiv.org/html/2609.27607#Sx1.p1.1)\.
- \[24\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica\(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.arXiv preprint arXiv:2306\.05685\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[§2](https://arxiv.org/html/2609.27607#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix AData Processing and Supplementary Tasks

RadEvalX uses the consensus of two radiologists over eight error categories\. We sum significant and insignificant counts for the total\-error target\. Blank category cells encode zero: this convention exactly reproduces all 16 category totals for the 30 high\-error reports in the original paper’s Appendix D\[[4](https://arxiv.org/html/2609.27607#bib.bib21)\]\. The 100 annotated report IDs join one\-to\-one to the released 590\-row metric file; the remaining rows have no expert labels and are excluded\. Our main baseline scores are recomputed with the same implementations used for RadEvalExpert\.

There are 100 distinct report IDs but 96 distinct reference texts and 78 candidate texts\. We retain report IDs as the sampling unit; identical text alone does not identify a shared study\. Each ID has only one candidate, so within\-study candidate ranking is unavailable on RadEvalX\. Table[A1](https://arxiv.org/html/2609.27607#A1.T1)removes the two uncertainty categories from the labels while keeping predictions fixed\.

Table A1:RadEvalX label\-scope sensitivity\. The primary labels count all eight error categories; the six\-category target excludes added and omitted uncertainty\. Predictions stay fixed\. Values are directed Kendallτb\\tau\_\{b\}\.RadEvalExpert preserves its total and significant\-error labels\. The study identifier is the resampling unit, keeping candidates with their shared reference together\. ReXErr sentence inputs contain one candidate sentence and its complete original report\. We exclude the dataset’s neutral rewrites from the main binary\-error analysis\. Unchanged sentences form the negative class in every error\-scope comparison\.

The factual\-error group contains false negation, false prediction, changed location, changed severity, changed measurement, added medical device, changed device position, and changed device name\. The language group contains added typos, homophone substitutions, and repetition\. This is an exploratory grouping by the dataset’s intervention labels\. We have not manually adjudicated the medical meaning of every edit\. The two groups share unchanged negatives but use disjoint positive examples\.

RaTE\-Eval contributes 440 sentence pairs, 370 paragraph pairs, and 847 synthetic synonym–opposite comparisons\. Each synthetic comparison produces two reference–candidate pairs, giving 1,694 pairs in total\. The distributed synthetic files contain rewrites and opposites without their original sentences\. We reconstruct references from the sentence\-task pool using text similarity and one\-to\-one assignment\. Full gives the synonym a lower error score in 97\.3% of these comparisons and a higher soft F1 in 99\.9%\. Excluding the seven comparisons below the preprocessing confidence threshold of 0\.20 leaves 840 comparisons\. The corresponding rates are 97\.5% and 100\.0%\. These results describe the reconstructed task; the expert benchmarks and ReXErr analyses use their supplied references\.

Table[A2](https://arxiv.org/html/2609.27607#A1.T2)reports the other RaTE\-Eval tasks\. The normalized Jev count correlates with sentence error ratios at 0\.398 and paragraph quality ratings at 0\.424\. RadCliQ reaches 0\.487 on paragraph ratings, showing that broader report quality remains a distinct evaluation target\.

Table A2:Supplementary RaTE\-Eval correlations\. Sentence labels are error ratios \(lower is better\); paragraph labels are quality ratings \(higher is better\)\. Signs follow these targets\. Brackets are study\-bootstrap 95% intervals\.
## Appendix BImplementation and Exact Questions

Report segmentation uses spaCy with repairs for abbreviations, measurements, and numbered lists\. Template headings and empty units are removed\. L1 passes each remaining sentence to the atomic\-decomposition prompt below\. The local decomposition model is Qwen3\.5\-9B, with temperature zero\. Its invalid JSON outputs fall back to the input sentence\.

Jev receives the claim and opposing report as structured state fields\. Each statement–report comparison uses one request: Full includes all seven questions, while OneQ includes only the support question\. Both configurations check statements in both directions\. Repeated\-call experiments obtain fresh judgments for the same inputs\.

The default weights in Eq\.[2](https://arxiv.org/html/2609.27607#S3.E2)are one\. The weight grid varies candidate\-side not\-addressed and reference\-side not\-addressed terms over\{0\.5,1,2\}2\\\{0\.5,1,2\\\}^\{2\}, leaving candidate contradiction weight fixed at one\. Significance weighting multiplies each discrepancy term by the associated expected significance score divided by two\. Category scores distribute candidate discrepancy mass according to the error\-type probabilities; reference not\-addressed mass supplies the omission category\. These diagnostics use Full’s additional outputs\.

### B\.1Support question and Full extensions

OneQ sends the support entry below\. Full sends all seven entries in one request\. The state contains two text fields:claimandreport\.

#### support \(choice\)\.

Does the report support the claim about the imaging finding?

supportedThe report states the same finding with compatible presence, location, and severity

contradictedThe report states the opposite, or the same finding with incompatible presence, laterality, or severity

not\_addressedThe report does not mention this finding at all

#### claim\_negated \(noul\)\.

The claim states that the finding is ABSENT \(for example ’no’, ’without’, ’clear of’, ’free of’\)

#### claim\_hedged \(noul\)\.

The claim expresses uncertainty about the finding \(for example ’possible’, ’cannot exclude’, ’may represent’, ’questionable’\)

#### report\_negated \(noul\)\.

The report states that the same finding described in the claim is ABSENT

#### report\_hedged \(noul\)\.

The report expresses uncertainty about the same finding described in the claim

#### error\_type \(choice\)\.

If the claim disagrees with the report, what kind of discrepancy is it?

false\_findingThe claim asserts a finding that the report says is absent or never mentions

wrong\_locationSame finding, but a different anatomical location or laterality

wrong\_severitySame finding, but a different severity, size, or extent

wrong\_temporalSame finding, but a different comparison to prior imaging \(new, unchanged, improved, worsened\)

noneThe claim does not disagree with the report

#### significance \(score\)\.

Clinical significance of the finding described in the claim, if the claim were wrong

1. 1\.Incidental or normal\-variant statement with no effect on management
2. 2\.Would change follow\-up imaging or outpatient management but not immediate care
3. 3\.Would change immediate management or is potentially life\-threatening

### B\.2Atomic decomposition prompt

> You split one sentence from a radiology report into atomic findings\. Rules: 1\. Each output item states exactly one finding about one anatomical entity\. 2\. Preserve negation and uncertainty words exactly as written \("no", "without", "cannot exclude", "possible", "unchanged"\)\. 3\. Preserve laterality, severity, size, and comparison\-to\-prior phrases inside the item they belong to\. 4\. Do not add, infer, or normalize anything\. Do not change tense\. 5\. If the sentence already contains one finding, return it unchanged\. 6\. Output a JSON list of strings and nothing else\.

### B\.3Filtering and shortened criteria

The filtering variant asks, “This sentence is about the same anatomical entity as the claim,” for every report sentence\. It retains sentences whose probability exceeds 0\.3 and passes their concatenation to Full\. The shortened support criteria are “The report states the same finding,” “The report states the opposite,” and “The report does not mention it\.” They replace the three support criteria in order; the other Full questions stay the same\.

## Appendix CAdditional Expert Comparisons

Table[A3](https://arxiv.org/html/2609.27607#A3.T3)reports 17 standard metric outputs, including variants from the same method family\. Similarity metrics are negated when correlated with error counts\. A negative directed value is retained and indicates an association opposite to the expected direction\. All subset correlations and paired differences are recomputed on the shared valid pairs before study resampling\.

Table A3:All 17 standard metric outputs on the complete expert datasets\. Columns report directed Kendall correlation with total \(T\) or significant \(S\) errors\. Variants within each method family are listed separately\.For within\-study ordering, every unordered pair of candidates contributes\+1\+1,−1\-1, or zero according to whether the method and expert orderings agree, disagree, or contain a tie\. The denominator includes all candidate pairs\. Full RadEvalExpert contributes 624 comparisons\. The common 200\-pair RadFact/RadMatch subset spans 138 studies and contributes 69 comparisons\. Each bootstrap draw sums the pair counts of the sampled study blocks, with repeated studies contributing repeated counts\. Within\-study comparisons share a reference report; candidate lengths can still differ\.

### C\.1Source, section, and modality

Table[A4](https://arxiv.org/html/2609.27607#A3.T4)separates RadEvalExpert by data source and report section\. Jev–Full has positive total\-error correlation in each source, with wider uncertainty for the smallest source\. Table[A5](https://arxiv.org/html/2609.27607#A3.T5)reports the RaTE\-Eval modalities with at least 20 evaluated pairs\. Performance varies with both modality and evaluation target; paragraph quality ratings often favor BERTScore or RadCliQ\. These strata show how performance varies within the evaluated datasets\.

Table A4:RadEvalExpert total\-error correlation by source and report section\. Each row uses the same pairs for the three methods\. Source and section partitions overlap\. Intervals resample studies within each stratum\.Table A5:RaTE\-Eval correlation by modality, for strata with at least 20 pairs\. Jev uses Full normalized error count\. Similarity/error signs follow each task's label direction\. Brackets give 95% study\-bootstrap intervals\.

## Appendix DComplete Component Comparisons

Table[A6](https://arxiv.org/html/2609.27607#A4.T6)gives the numerical comparisons shown in Figure[2](https://arxiv.org/html/2609.27607#S5.F2)\. Table[A7](https://arxiv.org/html/2609.27607#A4.T7)covers all nine weight combinations\. The default configuration is retained throughout the main comparisons\. RadEvalX peaks atwh=wo=0\.5w\_\{h\}=w\_\{o\}=0\.5, withτb=0\.635\\tau\_\{b\}=0\.635and paired gain0\.0530\.053\[0\.005, 0\.104\]\. RadEvalExpert peaks atwh=0\.5w\_\{h\}=0\.5,wo=1w\_\{o\}=1, withτb=0\.410\\tau\_\{b\}=0\.410\. These exploratory settings were compared on the evaluation data, so the largest observed value is not an independently validated tuning gain\.

Table A6:Paired changes in total\-errorτb\\tau\_\{b\}relative to Full\. These are independent variants, not successive simplifications\. Cost ratios appear in Figure[2](https://arxiv.org/html/2609.27607#S5.F2)\.Table A7:Exploratory weight grid for Full\. Weights multiply candidate noncoverage \(whw\_\{h\}\) and reference noncoverage \(wow\_\{o\}\); contradiction weight stays one\. Columns give total\-errorτb\\tau\_\{b\}and paired change from\(1,1\)\(1,1\)\. Intervals are unadjusted 95% study\-bootstrap intervals\.
## Appendix ECost and Latency

LetTv,iT\_\{v,i\}denote the returned input\-token count for configurationvvon report pairii, summed over both directions\. WithNNpairs, workload cost is

cost¯v=1N​∑iTv,i​0\.042106,saving=1−∑iTOneQ,i∑iTFull,i\.\\overline\{\\mathrm\{cost\}\}\_\{v\}=\\frac\{1\}\{N\}\\sum\_\{i\}T\_\{v,i\}\\frac\{0\.042\}\{10^\{6\}\},\\qquad\\mathrm\{saving\}=1\-\\frac\{\\sum\_\{i\}T\_\{\\mathrm\{OneQ\},i\}\}\{\\sum\_\{i\}T\_\{\\mathrm\{Full\},i\}\}\.\(5\)On RadEvalX,1−5374\.55/9695\.17=44\.56%1\-5374\.55/9695\.17=44\.56\\%\. On RadEvalExpert,1−6151\.1346/10797\.1971=43\.03%1\-6151\.1346/10797\.1971=43\.03\\%\. The resulting mean API costs are shown in Table[A8](https://arxiv.org/html/2609.27607#A5.T8)\. Full and OneQ use the same L1 decomposition, so its local cost is shared\.

Table A8:API workload cost at $0\.042 per million input tokens\. Cached judgments are priced at the same rate as fresh judgments\. Local GPU costs are excluded\.Full on RadEvalExpert has median request latency 433\.6 ms and 95th\-percentile latency 548\.4 ms\. Its median pair\-level judgment time is 0\.71 s with concurrent statement requests\. Batch completion took 10\.2 minutes, including scheduling and rate limits\. Request\-latency measurements exclude local cache hits, whereas the workload\-cost comparison above includes them\. No matched cross\-method timing of the full report\-evaluation pipeline is included\.

### E\.1Three\-finding API demonstration

Figure[3](https://arxiv.org/html/2609.27607#S5.F3)uses three constructed claims fixed before execution\. Each method receives the same complete reference and the support question and category definitions in Appendix[B\.1](https://arxiv.org/html/2609.27607#A2.SS1)\. We send one request per finding, with concurrency three, for five rounds in alternating method order\. There is no local response cache or warmup\. We show first\-round outputs and summarize all five rounds; all 30 classifications match the expected labels\. Timing covers the judgment requests and excludes decomposition\.

Jev uses versionjev\-1\.13\.0with OneQ\. Luna responses reportgpt\-5\.6\-luna, standard service, and zero reasoning tokens\. Its requests set reasoning effort tonone, temperature to zero, JSON output, and a 64\-token output limit\. Completion time includes communication and service overhead\. The five\-run ranges are 0\.607–1\.212 s for Jev and 2\.869–4\.301 s for Luna\.

At the September 22, 2026 standard API prices\[[16](https://arxiv.org/html/2609.27607#bib.bib11),[13](https://arxiv.org/html/2609.27607#bib.bib19)\], Jev’s 1,201 input tokens per round cost1201×0\.042/106=$0\.0000504421201\\times 0\.042/10^\{6\}=\\$0\.000050442\. Luna uses 686 input tokens and 40 output tokens per round, with no cached input or cache\-write tokens\. Its cost is\(686×0\.20\+40×1\.20\)/106=$0\.00018520\(686\\times 0\.20\+40\\times 1\.20\)/10^\{6\}=\\$0\.00018520\.

## Appendix FError Types and Language Stress Tests

Table[A9](https://arxiv.org/html/2609.27607#A6.T9)gives the complete sentence\-level error profile\. The 131\-item constructed stress set probes nine forms of negation, uncertainty, and clinical attributes\. Jev assigns the expected three\-way support label in 125 cases \(95\.4%\) in the original run\. After error review, we corrected three device\-status labels and simplified one compound claim, obtaining 129 correct cases \(98\.5%\)\. The corrections followed the support criteria: an unmentioned device is unaddressed, and each claim should state one finding\. Both results are retained because these revisions followed inspection of model outputs\. On the revised set, Jev answers all 15 laterality cases, 22 direct\-negation cases, and 16 uncertainty cases correctly\. These are small constructed sets, rather than estimates of clinical sensitivity for naturally occurring errors\.

Table A9:Jev detection by injected error type\. Each row includes the same 10,790 unchanged negative sentences\. This sentence experiment uses the Full question set and raw support probabilities\.Two cases fail\. First, a double\-negation statement, “There is no evidence to suggest the absence of a left pleural effusion,” is labeled contradicted against a report stating a left effusion\. Second, a claim about a femur fracture is labeled contradicted rather than unaddressed against a chest report containing “No acute osseous abnormality\.” The latter illustrates how broad negative language can blur contradiction and noncoverage\.

Report\-level category attribution is weaker than binary error detection\. Figure[A1](https://arxiv.org/html/2609.27607#A6.F1)shows each predicted category’s average share of total category score, grouped by injected error class\. For location errors, the false\-finding share is 0\.468 and the location share is 0\.339\. For severity errors, the corresponding shares are 0\.493 and 0\.301\. The broad false\-finding category dominates even when a more specific category is expected\. The matrix summarizes score allocation, not a finding\-matched confusion matrix\. These results do not support treating the category scores as reliable report\-level error explanations\.

![Refer to caption](https://arxiv.org/html/2609.27607v1/figA2_categories.png)Figure A1:Report\-level category allocation on 2,000 sampled ReXErr pairs\. Rows group the injected errors; cells show mean proportions of predicted category score\. The omission component uses the reverse direction\. This diagnostic measures score allocation, not classification accuracy\.
## Appendix GCalibration and Repeated Calls

The raw ReXErr sentence experiment has 19,513 examples and an error prevalence of 44\.7%\. Jev’s mean error probability is 0\.3278; NLI’s is 0\.3755\. Their Brier scores are 0\.1235 and 0\.1192, respectively\. Figure[A2](https://arxiv.org/html/2609.27607#A7.F2)shows the raw reliability curves\. Bin populations are uneven: Jev assigns 12,377 examples to the lowest bin and 5,761 to the highest\. The middle bins contain 113–301 examples each\. The observed error fraction in Jev’s lowest bin is 0\.1672 despite a mean predicted probability of 0\.0037\.

Figure A2:Calibration diagnostics\. \(a\) Raw sentence error probabilities on all 19,513 ReXErr examples, using ten equal\-width probability bins\. \(b\) Jev–Full score bins versus mean expert errors on RadEvalX, using ten score quantiles\. Dashed lines indicate perfect calibration or equality with expert counts\. Neither plot uses the post\-hoc calibrated holdout probabilities\.Post\-hoc calibration uses a fixed, equal split of studies into development and evaluation sets\. This gives 9,562 development examples and 9,951 holdout examples, without study overlap\. Logistic calibration fits a sigmoid to raw error probabilities; isotonic regression fits a monotone map with out\-of\-range values clipped\. Table[A10](https://arxiv.org/html/2609.27607#A7.T10)compares raw and calibrated outputs on the same holdout\. AUROC remains nearly unchanged in this experiment; isotonic maps can introduce ties\. These results concern the Full sentence configuration and this error mixture\.

Table A10:Probability calibration on the same 9,951\-example holdout\. Fits use 9,562 examples from disjoint studies\. Full\-set raw ECE is reported separately in the main text\.### G\.1Repeatability

The RadEvalX experiment randomly selects 50 report pairs and holds 615 extracted findings fixed\. Five uncached runs each of Full and OneQ produce 6,150 judgments, all usingjev\-1\.13\.0\. The sample standard deviations in correlation with expert total\-error counts are 0\.00745 and 0\.00737, respectively\. Table[A11](https://arxiv.org/html/2609.27607#A7.T11)summarizes ranking and score variation on this independent dataset\.

Table A11:Repeated Jev judgments on 50 RadEvalX pairs, five uncached runs per configuration, holding 615 extracted findings fixed\. Rank agreement is the median of ten between\-run Kendall correlations; correlation SD is across the five correlations with total\-error labels\.The matched repeatability experiment evaluates 50 RadEvalExpert pairs from 48 studies, with five runs per setting\. These are a fixed selection of 50 pairs from the 200\-pair subset\. GREEN uses greedy decoding; RadFact uses its default temperature zero\. RadMatch uses the same local Qwen3\.8\-27B\-FP8 backend at temperature zero and at its default temperature one\.

Jev–Full and Jev–OneQ use the same 616 L1 statements: 297 candidate statements and 319 reference statements\. Five rounds alternate Full and OneQ, with 616 uncached judgments per configuration and round, all usingjev\-1\.13\.0\.

Table[3](https://arxiv.org/html/2609.27607#S5.T3)counts a pair as identical when its score is exactly equal across all five runs\. Each of the six settings evaluates the same 50 pairs five times\. Between\-run rank consistency is the median of all ten pairwise Kendall correlations\. Correlation SD uses the sample standard deviation across five correlations with expert total\-error counts\. Mean expert correlations are 0\.202 for OneQ, 0\.211 for Full, 0\.221 for GREEN, 0\.221 for RadFact, 0\.468 for RadMatch at temperature zero, and 0\.400 for RadMatch at temperature one\.

Jev’s matched experiment holds decomposition fixed and measures API judgment repeats; the baselines repeat their local evaluation pipelines\. Every row in Table[3](https://arxiv.org/html/2609.27607#S5.T3)uses the same pairs and repetition count\. The RadEvalX experiment above provides an additional check on both configurations\.

Table A12:RadMatch matching\-stage validation across all five runs per setting\. Each setting includes 250 report\-pair evaluations on the same 50 pairs\. Retried counts evaluations requiring at least one retry\.Disabling sampling improves scoring stability while validation failures remain: temperature zero triggers fallback in 17 of 250 evaluations, compared with nine at temperature one \(Table[A12](https://arxiv.org/html/2609.27607#A7.T12)\)\. Repeated scores and output validation capture different aspects of operational reliability\.