PathReportEval: A Systematic Benchmark for Pathology Report Generation

# PathReportEval: A Systematic Benchmark for Pathology Report Generation
Source: [https://arxiv.org/html/2607.18448](https://arxiv.org/html/2607.18448)
Suryakant Singh Department of Biomedical Informatics Stony Brook University Stony Brook, NY, USA suryakant\.singh@stonybrook\.edu &Sejuti Majumder Department of Biomedical Informatics Stony Brook University Stony Brook, NY, USA sejuti\.majumder@stonybrook\.edu &Beatrice Knudsen Department of Pathology University of Utah Salt Lake City, UT, USA beatrice\.knudsen@path\.utah\.edu &Joel Saltz Department of Biomedical Informatics Stony Brook University Stony Brook, NY, USA joel\.saltz@stonybrook\.edu &Prateek Prasanna Department of Biomedical Informatics Stony Brook University Stony Brook, NY, USA prateek\.prasanna@stonybrook\.edu

###### Abstract

Pathology report generation from whole\-slide images \(WSIs\) is a rapidly growing multimodal learning problem, yet progress is difficult to measure because existing studies use heterogeneous datasets, model settings, visual encoders, and evaluation protocols\. Moreover, commonly used natural language generation metrics, including BLEU, ROUGE, and METEOR, primarily reward lexical similarity and can fail to detect clinically consequential errors such as omitted diagnoses, hallucinated findings, or discordant tumor attributes\.

We present a standardized benchmark and evaluation framework for pathology report generation\. The benchmark evaluates four representative generation methods across three datasets, TCGA, HistAI, and REG 2025, using three pathology foundation encoders, CONCHv1\.5, UNI2\-h, and H\-Optimus\-1\. Our framework standardizes the full experimental pipeline, including preprocessing, tokenization, feature extraction, training, decoding, and evaluation, enabling fair comparison across model architectures and visual representations\. The framework is modular and extensible, allowing new methods, datasets, and encoders to be incorporated under a shared protocol\.

A central contribution of this work is the Clinical Report Quality Score \(CRQS\), a clinically grounded metric for evaluating factual correctness in generated pathology reports\. CRQS maps reference and generated reports into structured clinical attributes and measures four complementary dimensions of report quality: clinical fact coverage, key information recall, hallucination rate, and clinical discordance\. This decomposition provides both an aggregate score and interpretable sub\-scores that expose clinically meaningful failure modes\.

Our experiments show that conventional language\-generation metrics are weakly aligned with clinical correctness and can overestimate report quality\. CRQS, in contrast, identifies substantial differences between models and encoders that are not captured by lexical metrics\. By providing a standardized benchmark, a public plug\-and\-play framework, and a structured clinical evaluation metric, this work establishes a reproducible foundation for measuring progress in pathology report generation\. Evaluation framework:https://github\.com/surykntsingh/PathBench

## 1Introduction

Pathology report generation from whole\-slide images \(WSIs\) is an emerging multimodal learning task that requires models to convert gigapixel histopathology images into clinically meaningful textual reportsLucassenet al\.\([2025](https://arxiv.org/html/2607.18448#bib.bib14)\); Huet al\.\([2025](https://arxiv.org/html/2607.18448#bib.bib15)\); Jinet al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib16)\); Gaoet al\.\([2025](https://arxiv.org/html/2607.18448#bib.bib24)\)\. Recent advances in vision–language modeling and pathology foundation models have made this task increasingly feasibleLuet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib7)\); Chenet al\.\([2024b](https://arxiv.org/html/2607.18448#bib.bib9)\); Bioptimus \([2025](https://arxiv.org/html/2607.18448#bib.bib25)\), enabling systems that can integrate visual evidence across large tissue regions and generate diagnostic descriptions\. However, progress in this area remains difficult to measure\. Existing studiesGuoet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib10)\); Chenet al\.\([2024a](https://arxiv.org/html/2607.18448#bib.bib12)\); Zhanget al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib11)\); Singhet al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib13)\)often use different datasets, preprocessing pipelines, visual encoders, model implementations, decoding strategies, and evaluation protocols, making direct comparison across methods unreliable\. As a result, it is often unclear whether reported performance gains reflect improvements in model design or differences and biases in experimental setup\.

A second challenge is evaluation\. Most pathology report generation studies rely on standard natural language generation \(NLG\) metrics, such as BLEUPapineniet al\.\([2002](https://arxiv.org/html/2607.18448#bib.bib28)\), ROUGELin \([2004](https://arxiv.org/html/2607.18448#bib.bib30)\), and METEORBanerjee and Lavie \([2005](https://arxiv.org/html/2607.18448#bib.bib29)\)\. These metrics measure lexical overlap between generated and reference reports, but they do not directly assess whether clinically important information is preservedYuet al\.\([2023](https://arxiv.org/html/2607.18448#bib.bib31)\); Baharoonet al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib6)\)\. In pathology, this distinction is critical: a generated report may be lexically similar to a reference report while omitting a key diagnosis, misstating tumor grade or stage, hallucinating lymph\-node involvement, or contradicting margin status\. Such errors are not merely linguistic failures; they reflect failures of clinical fidelity\. Thus, evaluating pathology report generation requires metrics that move beyond surface\-form similarity and explicitly measure the correctness of clinically meaningful content\.

![Refer to caption](https://arxiv.org/html/2607.18448v1/x1.png)Figure 1:Overview of the proposed pathology report generation benchmark and evaluation framework\. The framework standardizes data splits, feature extraction, model training, and evaluation across datasets, encoders, and report generation methods, while also incorporating the clinically grounded CRQS pipeline for structured assessment of report quality\.In this work, we introduce a standardized benchmark and evaluation framework for pathology report generation from WSIs\. We evaluate four state\-of\-the\-art report generation methods across three datasets with distinct reporting characteristics and three pathology visual encoders representing different foundation\-model families\. By evaluating all methods under a shared experimental protocol, our benchmark enables systematic comparison across model architectures, datasets, and visual representations\. To support reproducibility and future extension, we further provide a modular plug\-and\-play framework that standardizes preprocessing, tokenization, feature extraction, training, decoding, and evaluation\. The framework is designed to allow new report generation models, datasets, and visual encoders to be incorporated with minimal changes while preserving a common evaluation pipeline\.

To address the limitations of lexical evaluation, we propose the*Clinical Report Quality Score*\(CRQS\), a pathologist\-verified structured metric for assessing clinical fidelity in generated pathology reports\. CRQS is motivated by synoptic reporting practices, such as those recommended by the College of American PathologistsCollege of American Pathologists \([2023a](https://arxiv.org/html/2607.18448#bib.bib32),[b](https://arxiv.org/html/2607.18448#bib.bib33)\), in which key diagnostic and prognostic attributes are documented using standardized fields\. Rather than comparing reports only as free text, CRQS maps both generated and reference reports into structured clinical attributes, such as diagnosis, histologic type, tumor grade, stage, margin status, lymph\-node involvement, biomarker status, and other dataset\-specific pathological findings\. It then evaluates report quality using four interpretable components: clinical fact coverage, key information recall, hallucination rate, and clinical discordance\. This decomposition allows CRQS to capture distinct failure modes, including omitted findings, missing high\-impact diagnostic fields, unsupported generated facts, and direct contradictions with the reference report\.

Across datasets, methods, and encoders, our analysis shows that conventional NLG metrics can obscure clinically meaningful differences between systems\. Models with similar lexical scores may differ substantially in their ability to recover key diagnostic information, avoid hallucinations, and maintain consistency with reference clinical attributes\. CRQS provides a complementary evaluation signal that is more directly aligned with clinical content and supports fine\-grained error analysis\. Together, the benchmark, public framework, and CRQS provide a reproducible foundation for measuring progress in pathology report generation\. Our contributions are threefold:

1. 1\.We introduce astandardized benchmark for pathology report generationfrom WSIs, evaluating representative methods across multiple datasets and pathology visual encoders under a shared experimental protocol\.
2. 2\.We provide amodular plug\-and\-play benchmarking frameworkthat standardizes preprocessing, feature extraction, training, decoding, and evaluation, enabling fair comparison and rapid extension to new models, datasets, and encoders\.
3. 3\.Wepropose CRQS, a clinically grounded evaluation metricthat assesses generated reports using structured clinical attributes and decomposes report quality into fact coverage, key information recall, hallucination, and clinical discordance\.

Figure[1](https://arxiv.org/html/2607.18448#S1.F1)presents an overview of this benchmark and evaluation pipeline, highlighting how datasets, visual encoders, report generation models, and clinical evaluation components are organized within a single standardized framework\. All code related to this evaluation framework is available athttps://github\.com/surykntsingh/PathBench

## 2Related Work

##### Pathology report generation from WSIs\.

Automated pathology report generation has recently emerged as a challenging multimodal learning problem, requiring models to translate gigapixel whole\-slide images \(WSIs\) into clinically meaningful textGamper and Rajpoot \([2021](https://arxiv.org/html/2607.18448#bib.bib34)\); Huanget al\.\([2023](https://arxiv.org/html/2607.18448#bib.bib35)\); Luet al\.\([2023](https://arxiv.org/html/2607.18448#bib.bib36)\)\. Early approaches extended image captioning frameworks to WSIs using multiple instance learning \(MIL\), where patch\-level features are aggregated to produce slide\-level reports\. WSI\-Caption \(MI\-Gen\)Chenet al\.\([2024a](https://arxiv.org/html/2607.18448#bib.bib12)\)demonstrated the feasibility of transformer\-based report generation at gigapixel scale\. Subsequent methods have introduced more structured modeling strategies: HistGenGuoet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib10)\)uses local–global hierarchical encoding and cross\-modal context interaction to better align multi\-scale visual evidence with textual descriptions, while BiGenZhanget al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib11)\)incorporates retrieval\-based knowledge to enrich visual representations with semantically relevant information\. These works represent important progress, but they are typically evaluated under heterogeneous settings, including different datasets, preprocessing pipelines, visual encoders, decoding strategies, and evaluation protocols\. As a result, it remains difficult to determine whether observed performance differences arise from model design or from inconsistent experimental conditions\.

##### Vision–language models and pathology foundation encoders\.

Recent advances in vision–language learning have produced powerful pretrained encoders for histopathology, including CONCHLuet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib7)\), UNIChenet al\.\([2024b](https://arxiv.org/html/2607.18448#bib.bib9)\), and H\-OptimusBioptimus \([2025](https://arxiv.org/html/2607.18448#bib.bib25)\)\. These models provide strong visual representations and have improved performance across a range of pathology tasks\. However, most work using such encoders focuses on representation learning or downstream discriminative tasks, rather than systematic evaluation of generative pathology reporting\. Moreover, report generation requires not only visual recognition but also clinically faithful selection, organization, and expression of diagnostic findings\. Thus, there is a need to evaluate how different visual encoders influence clinical correctness in generated reports under a common benchmark\.

##### Evaluation of medical report generation\.

Evaluation remains a major bottleneck for medical report generation\. Most existing studies rely on standard natural language generation metrics such as BLEUPapineniet al\.\([2002](https://arxiv.org/html/2607.18448#bib.bib28)\), ROUGELin \([2004](https://arxiv.org/html/2607.18448#bib.bib30)\), and METEORBanerjee and Lavie \([2005](https://arxiv.org/html/2607.18448#bib.bib29)\), which measure lexical overlap between generated and reference reports\. While these metrics are easy to compute, they do not directly assess factual correctness or clinical relevance\. A generated report can achieve high lexical similarity while omitting key diagnostic findings, introducing unsupported clinical attributes, or contradicting the reference diagnosis\. Prior work in radiology has explored structured entity\-based or graph\-based evaluationJainet al\.\([2021](https://arxiv.org/html/2607.18448#bib.bib40)\); Yuet al\.\([2023](https://arxiv.org/html/2607.18448#bib.bib31)\); Baharoonet al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib6)\), and recent pathology report generation work has introduced fact\-based metrics such as FACTentMiuraet al\.\([2021](https://arxiv.org/html/2607.18448#bib.bib39)\); Chenet al\.\([2024a](https://arxiv.org/html/2607.18448#bib.bib12)\)\. These efforts move beyond surface\-form similarity, but they typically treat facts as unstructured entities and do not explicitly model the structured nature of pathology reporting\.

##### Need for standardized and clinically grounded benchmarking\.

Benchmarking has been central to progress in computational pathology, particularly for classification, segmentation, retrieval, and survival prediction\. However, existing pathology benchmarks largely focus on visual prediction tasks and do not address multimodal report generation\. Current report generation methods are therefore evaluated in isolation, limiting reproducibility and fair comparison\. In this work, we address this gap by introducing a standardized benchmark across models, datasets, and visual encoders, together with a modular plug\-and\-play framework for reproducible evaluation\. We further propose the Clinical Report Quality Score \(CRQS\), a structured metric that evaluates generated reports using clinically meaningful fields aligned with synoptic reporting practice\. Unlike lexical metrics or unstructured fact\-overlap scores, CRQS decomposes report quality into clinical fact coverage, key information recall, hallucination rate, and clinical discordance, enabling both aggregate comparison and interpretable error analysis\.

## 3Benchmark and Evaluation Framework

Table 1:Characteristics of datasets used for pathology report generation\.The datasets differ substantially in size, institutional diversity, annotation noise, and report structure, enabling a comprehensive evaluation of model robustness and generalization\.### 3\.1Task Definition

We study pathology report generation from whole\-slide images \(WSIs\)\. Given a WSIXX, a model generates a reportR^=\(r1,r2,…,rT\)\\hat\{R\}=\(r\_\{1\},r\_\{2\},\\ldots,r\_\{T\}\), where each tokenrtr\_\{t\}is drawn from a fixed vocabulary\. The corresponding reference report is denoted byRR\. Unlike generic image captioning, this task requires models to recover diagnostically relevant visual evidence, express it in clinically appropriate language, and preserve structured attributes such as diagnosis, grade, stage, margin status, and other pathology\-specific findings\.

Our goal is not to introduce a new report generation architecture, but to evaluate existing systems under a controlled and reproducible benchmark\. We therefore define a standardized protocol for comparing models across datasets and visual encoders, using both conventional language\-generation metrics and the proposed clinically grounded CRQS metric\.

![Refer to caption](https://arxiv.org/html/2607.18448v1/figures/report_homogeneity_histogram_overlay.png)

![Refer to caption](https://arxiv.org/html/2607.18448v1/figures/cancer_distribution_horizontal.png)

Figure 2:Dataset characteristics across the three pathology report generation benchmarks\. Left: distribution of report lengths \(word count\) across datasets\. TCGA exhibits substantially longer and more variable reports, while REG contains shorter and more consistent reports\. HistAI shows intermediate characteristics\. Dashed lines indicate mean word counts\. Right: cancer\-type distribution across datasets, highlighting differences in class composition\.
### 3\.2Datasets

We evaluate on three pathology report generation datasets with complementary characteristics, summarized in Table[1](https://arxiv.org/html/2607.18448#S3.T1)\. The datasets differ in scale, institutional diversity, report structure, and annotation noise, allowing us to test model behavior across heterogeneous reporting conditions\. As shown in Figure[2](https://arxiv.org/html/2607.18448#S3.F2), the datasets also exhibit distinct report length distributions, reflecting differences in structure and variability\.The same figure also shows marked differences in cancer\-type composition across datasets, which may influence task difficulty and cross\-dataset generalization\. Taken together, these distributions highlight that the benchmark captures variation in both report style and disease mix\.TCGA \(PathText\)Chenet al\.\([2024a](https://arxiv.org/html/2607.18448#bib.bib12)\):TCGA consists of WSIs paired with pathology reports derived from The Cancer Genome Atlas and used in prior WSI\-Caption work\. Its reports are relatively long and heterogeneous, with higher annotation noise due to variation in reporting style\.HistAIBulteet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib26)\):HistAI is a large\-scale, multi\-institutional dataset containing diverse tissue types and moderately structured reports\. It captures substantial variability in both image appearance and report language, making it useful for evaluating robustness under real\-world reporting heterogeneity\.REG 2025MICCAI COMPAY Workshop Organizers \([2025](https://arxiv.org/html/2607.18448#bib.bib27)\):REG 2025 is a benchmark dataset designed for pathology report generation\. It contains shorter, semi\-structured reports with higher report consistency and lower annotation noise\.

For reproducibility, we release the exact train, validation, and test splits used for all datasets\. These fixed splits ensure that future methods can be evaluated under the same data protocol rather than under independently constructed partitions\.

### 3\.3Models and Visual Encoders

We benchmark four representative pathology report generation methods spanning distinct modeling strategies: WSI\-Caption \(MI\-Gen\)Chenet al\.\([2024a](https://arxiv.org/html/2607.18448#bib.bib12)\), a multiple\-instance generative model; HistGenGuoet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib10)\), a hierarchical local\-global encoder with cross\-modal context interaction; BiGenZhanget al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib11)\), a retrieval\-augmented model that incorporates external textual knowledge; and SCOUTSinghet al\.\([2026](https://arxiv.org/html/2607.18448#bib.bib13)\), a context\-aware multimodal transformer with concept\-conditioned feature refinement\.

To isolate the role of visual representation, each report generation method is evaluated with three pretrained pathology encoders: CONCHv1\.5Luet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib7)\), UNI2\-hChenet al\.\([2024b](https://arxiv.org/html/2607.18448#bib.bib9)\), and H\-Optimus\-1Bioptimus \([2025](https://arxiv.org/html/2607.18448#bib.bib25)\)\. All methods are trained and evaluated using the same data splits, preprocessing pipeline, tokenizer, feature extraction protocol, and decoding strategy\. We reimplement the compared methods in a shared PyTorch Lightning codebase, which reduces implementation\-specific variation across originally separate repositories and enables controlled comparison of model architecture and encoder choice\. All visual features are extracted with TRIDENTZhanget al\.\([2025](https://arxiv.org/html/2607.18448#bib.bib4)\); Mahmood Lab \([2025](https://arxiv.org/html/2607.18448#bib.bib5)\)as explained in section[A\.1](https://arxiv.org/html/2607.18448#A1.SS1)\.

### 3\.4Clinical Report Quality Score \(CRQS\)

Clinical motivation\.Existing report\-generation benchmarks in pathology commonly rely on NLG metrics such as BLEU, ROUGE, and METEOR\. While useful for measuring lexical similarity, these metrics are poorly suited to evaluating whether a generated report preserves clinically actionable information\. In pathology, small factual errors can be consequential: an omitted margin status, an incorrect tumor grade, or a hallucinated lymph\-node finding can materially alter downstream clinical interpretation\. This motivates an evaluation metric that measures structured clinical fidelity rather than surface\-level textual overlap\.

We therefore propose the*Clinical Report Quality Score*\(CRQS\), a structured and clinically grounded metric for evaluating pathology report generation\. CRQS is motivated by synoptic reporting practices, such as those recommended by the College of American Pathologists, in which reports are organized around standardized diagnostic fields\. Instead of comparing generated and reference reports as unstructured text, CRQS organizes extracted clinical fields into clinically meaningful groups, including diagnosis hierarchy, tumor behavior, grade and aggressiveness, invasion patterns, and associated morphologic findings\. The hierarchy was designed in consultation with a practicing pathologist and prioritizes findings that are both clinically actionable and inferable from histomorphologic appearance in the whole\-slide image \(WSI\)\.

LetRRdenote the reference report andR^\\hat\{R\}denote the generated report\. We define a structured extraction function

ℰ​\(R\)=\{\(fi,vi\)\}i=1n,\\mathcal\{E\}\(R\)=\\\{\(f\_\{i\},v\_\{i\}\)\\\}\_\{i=1\}^\{n\},\(1\)wherefif\_\{i\}denotes a clinical field andviv\_\{i\}denotes its extracted value\. The same extraction function is applied toR^\\hat\{R\}\. In practice,ℰ\\mathcal\{E\}is implemented using a structured language\-model pipeline that maps free\-text reports into a predefined schema of pathology\-relevant attributes\. Extracted fields are normalized to canonical terminology where possible, including synonym resolution, negation handling, and value standardization\. A subset of fields, denoted𝒦\\mathcal\{K\}, is designated as clinically critical because errors in these fields are more likely to affect diagnostic interpretation or downstream clinical management\. In particular, these key fields emphasize visually grounded diagnostic attributes over metadata heavy fields that may not be directly observable from a single histopathology image\. The complete dataset\-specific list of extracted clinical fields and key fields is provided in the Appendix \(A\.7\)\.

For each fieldff, letvfv\_\{f\}andv^f\\hat\{v\}\_\{f\}denote the reference and generated values, respectively, when available\. A field is considered correctly matched if the generated value is clinically equivalent to the reference value under the normalization and matching rules of the schema\. This allows CRQS to account for clinically equivalent phrasings while penalizing omissions, unsupported findings, and discordant clinical values\.

CRQS is composed of four interpretable components\.

Clinical Fact Coverage\.Clinical Fact Coverage \(CFC\) measures the fraction of reference clinical fields that are correctly recovered in the generated report:

CFC=∑f∈ℱR𝟏​\[v^f≡vf\]\|ℱR\|,\\mathrm\{CFC\}=\\frac\{\\sum\_\{f\\in\\mathcal\{F\}\_\{R\}\}\\mathbf\{1\}\[\\hat\{v\}\_\{f\}\\equiv v\_\{f\}\]\}\{\|\\mathcal\{F\}\_\{R\}\|\},\(2\)whereℱR\\mathcal\{F\}\_\{R\}is the set of clinical fields present in the reference report, and≡\\equivdenotes clinical equivalence after normalization\. CFC captures overall completeness with respect to the reference report\.

Key Information Recall\.Key Information Recall \(KIR\) emphasizes recovery of high\-impact diagnostic attributes:

KIR=∑f∈𝒦R𝟏​\[v^f≡vf\]\|𝒦R\|,\\mathrm\{KIR\}=\\frac\{\\sum\_\{f\\in\\mathcal\{K\}\_\{R\}\}\\mathbf\{1\}\[\\hat\{v\}\_\{f\}\\equiv v\_\{f\}\]\}\{\|\\mathcal\{K\}\_\{R\}\|\},\(3\)where𝒦R=𝒦∩ℱR\\mathcal\{K\}\_\{R\}=\\mathcal\{K\}\\cap\\mathcal\{F\}\_\{R\}\. This term gives additional weight to clinically critical fields such as diagnosis, grade, stage, margin status, and lymph\-node involvement\.

Hallucination Rate\.Hallucination Rate \(HR\) measures the proportion of generated clinical facts that are not supported by the reference report:

HR=∑f∈ℱR^𝟏​\[f∉ℱR​or​v^f≢vf\]\|ℱR^\|,\\mathrm\{HR\}=\\frac\{\\sum\_\{f\\in\\mathcal\{F\}\_\{\\hat\{R\}\}\}\\mathbf\{1\}\[f\\notin\\mathcal\{F\}\_\{R\}\\ \\text\{or\}\\ \\hat\{v\}\_\{f\}\\not\\equiv v\_\{f\}\]\}\{\|\\mathcal\{F\}\_\{\\hat\{R\}\}\|\},\(4\)whereℱR^\\mathcal\{F\}\_\{\\hat\{R\}\}is the set of clinical fields asserted in the generated report\. HR penalizes unsupported or fabricated findings, which are particularly important in clinical report generation because fluent but incorrect statements may appear plausible to non\-expert evaluators\.

Clinical Discordance Score\.Clinical Discordance Score \(CDS\) measures disagreement on fields that are present in both the reference and generated reports:

CDS=∑f∈ℱR∩ℱR^𝟏​\[v^f≢vf\]\|ℱR∩ℱR^\|\.\\mathrm\{CDS\}=\\frac\{\\sum\_\{f\\in\\mathcal\{F\}\_\{R\}\\cap\\mathcal\{F\}\_\{\\hat\{R\}\}\}\\mathbf\{1\}\[\\hat\{v\}\_\{f\}\\not\\equiv v\_\{f\}\]\}\{\|\\mathcal\{F\}\_\{R\}\\cap\\mathcal\{F\}\_\{\\hat\{R\}\}\|\}\.\(5\)Unlike hallucination rate, which penalizes unsupported generated facts, CDS specifically captures direct contradictions between the generated and reference reports\. Examples include assigning the wrong tumor grade, incorrect stage, or discordant biomarker status\.

The final Clinical Report Quality Score combines completeness, critical information recall, hallucination avoidance, and clinical consistency:

CRQSraw=w1​CFC\+w2​KIR\+w3​\(1−HR\)\+w4​\(1−CDS\),\\mathrm\{CRQS\}\_\{\\mathrm\{raw\}\}=w\_\{1\}\\mathrm\{CFC\}\+w\_\{2\}\\mathrm\{KIR\}\+w\_\{3\}\(1\-\\mathrm\{HR\}\)\+w\_\{4\}\(1\-\\mathrm\{CDS\}\),\(6\)wherew1,w2,w3,w4≥0w\_\{1\},w\_\{2\},w\_\{3\},w\_\{4\}\\geq 0\. Unless otherwise specified, we use

w1=0\.30,w2=0\.40,w3=0\.20,w4=0\.40\.w\_\{1\}=0\.30,\\quad w\_\{2\}=0\.40,\\quad w\_\{3\}=0\.20,\\quad w\_\{4\}=0\.40\.\(7\)
These weights place greater emphasis on recovery of clinically critical information while penalizing hallucinated and discordant findings\. In particular, CRQS penalizes clinical discordance more heavily than hallucination because contradictory or clinically incorrect findings are typically more dangerous than unsupported but non\-conflicting additions\. Under this weighting scheme, the theoretical maximum raw score is0\.70\.7\. We therefore normalize the final score as

CRQS=CRQSraw0\.7,\\mathrm\{CRQS\}=\\frac\{\\mathrm\{CRQS\}\_\{\\mathrm\{raw\}\}\}\{0\.7\},\(8\)so thatCRQS∈\[0,1\]\\mathrm\{CRQS\}\\in\[0,1\], where higher values indicate better clinical report quality\.

This decomposition provides two advantages over conventional text\-overlap metrics\. First, it evaluates generated reports at the level of clinically meaningful facts rather than lexical similarity\. Second, it yields interpretable sub\-scores that distinguish between different failure modes: omission of relevant information, failure to recover key diagnostic fields, hallucination of unsupported findings, and direct clinical contradiction\. As a result, CRQS enables more clinically faithful comparison of pathology report generation systems and supports error analysis beyond aggregate language\-generation scores\.

### 3\.5Benchmarking Framework

To improve reproducibility and extensibility, we develop a unified benchmarking framework that standardizes data preprocessing, feature extraction, model training, and evaluation across all experiments\. As shown in Figure[1](https://arxiv.org/html/2607.18448#S1.F1), the framework is modular and plug\-and\-play, enabling researchers to integrate new report generation models, evaluate multiple datasets and visual encoders, and compare results under consistent experimental settings\. This unified setup reduces implementation variability and, in several cases, produces stronger baseline performance than originally reported, suggesting that consistent preprocessing and training contribute substantially to performance gains\. In addition to CRQS, we evaluate models using standard NLG metrics, including BLEU \(1–4\)Papineniet al\.\([2002](https://arxiv.org/html/2607.18448#bib.bib28)\), ROUGE\-LLin \([2004](https://arxiv.org/html/2607.18448#bib.bib30)\), and METEOR, and analyze their relationship with CRQS to highlight differences between lexical similarity and clinical correctness\.

## 4Experiments and Results

Experimental Setup\.All models are trained using identical preprocessing, feature extraction, and decoding settings, as described in Sec\.[3](https://arxiv.org/html/2607.18448#S3)\. All models are evaluated on held\-out test sets\. To ensure fairness, identical preprocessing and tokenization are used across methods, visual features are extracted using the same encoder for each experimental setting, and decoding strategies are fixed across models\. This standardized protocol ensures that performance differences reflect model capabilities rather than experimental inconsistencies\. Further details in Appendix[A\.2](https://arxiv.org/html/2607.18448#A1.SS2)\.

Main Results\.Tables[2\(a\)](https://arxiv.org/html/2607.18448#S4.T2.st1),[2\(c\)](https://arxiv.org/html/2607.18448#S4.T2.st3), and[2\(e\)](https://arxiv.org/html/2607.18448#S4.T2.st5)summarize method\-level performance for each dataset after averaging across the three encoders\. Complementarily, Tables[2\(b\)](https://arxiv.org/html/2607.18448#S4.T2.st2),[2\(d\)](https://arxiv.org/html/2607.18448#S4.T2.st4), and[2\(f\)](https://arxiv.org/html/2607.18448#S4.T2.st6)summarize encoder\-level performance for each dataset after averaging across the four methods\. This two\-view presentation keeps the tables compact while still exposing both model\-centric and encoder\-centric trends\.

Table 2:Dataset\-wise results averaged across encoders and methods\. Best results are in bold\(a\)Method\-wise results on TCGA, averaged across encoders\.
\(b\)Encoder\-wise results on TCGA, averaged across methods\.
\(c\)Method\-wise results on HistAI, averaged across encoders\.
\(d\)Encoder\-wise results on HistAI, averaged across methods\.
\(e\)Method\-wise results on REG 2025, averaged across encoders\.
\(f\)Encoder\-wise results on REG 2025, averaged across methods\.

Table 3:Qualitative results highlighting disagreement between METEOR and CRQS\.\[4pt\]

![Refer to caption](https://arxiv.org/html/2607.18448v1/figures/clean_meteor_vs_crqs_by_dataset.png)Figure 3:METEOR vs CRQS across datasets\.Across datasets, metric rankings vary by both method and encoder\. On REG 2025, SCOUT is the strongest method on the averaged NLG metrics, while BiGen attains the highest averaged CRQS\. On TCGA, the rankings are much closer and depend on the metric, with WSI\-Caption leading on BLEU\-4, SCOUT leading on BLEU\-1 and CRQS, and BiGen remaining competitive on ROUGE\-L\. On HistAI, SCOUT provides the strongest averaged BLEU and METEOR scores, while BiGen attains the highest averaged ROUGE\-L and CRQS\.

![Refer to caption](https://arxiv.org/html/2607.18448v1/figures/encoder_impact_crqs_all_datasets_grouped_barplot.png)Figure 4:Vision encoder sensitivity across TCGA, REG 2025, and HistAI\. The grouped bars show Clinical Report Quality Score \(CRQS\) for four pathology report generation methods using three pathology visual encoders: CONCHv1\.5, UNI2\-h, and H\-Optimus\-1\. Encoder choice produces method\- and dataset\-dependent differences in clinically grounded report quality\.Encoder Sensitivity\.Figure[4](https://arxiv.org/html/2607.18448#S4.F4)and the encoder\-wise tables show that visual representation choice has a measurable effect on clinically grounded report quality\. No single encoder dominates uniformly across every dataset and metric, indicating that encoder sensitivity depends on both the dataset distribution and the downstream generation architecture\. On TCGA, H\-Optimus\-1 yields the strongest averaged BLEU\-1 through BLEU\-3 and METEOR, while UNI2\-h achieves the highest averaged BLEU\-4, ROUGE\-L, and CRQS\. On HistAI, H\-Optimus\-1 provides the strongest averaged BLEU\-3, BLEU\-4, METEOR, ROUGE\-L, and CRQS, whereas CONCHv1\.5 is slightly stronger on BLEU\-1 and BLEU\-2\. On REG 2025, UNI2\-h delivers the best averaged performance across all reported NLG metrics and also attains the highest averaged CRQS\. These trends reinforce the importance of reporting encoder effects explicitly rather than collapsing them into a single global average\.

Qualitative Analysis\.As shown in Table[3](https://arxiv.org/html/2607.18448#S4.T3), the qualitative examples from the REG and HISTAI datasets respectively, reveal important limitations of traditional NLG metrics for clinical report evaluation\. In the REG example, METEOR assigns a moderately high score despite a clinically significant diagnostic error where the prediction incorrectly downgrades the lesion from HSIL/CIN2 to LSIL/CIN1, whereas CRQS correctly assigns a very low score\. Conversely, the HISTAI example demonstrates that METEOR can produce an extremely low score even when the generated report preserves the correct benign pathological interpretation using different clinical wording \(“reactive lymph node hyperplasia”\)\. In this case, CRQS successfully captures the preserved clinical meaning and assigns a high score\. These examples illustrate that lexical overlap metrics primarily evaluate surface text similarity, while CRQS more effectively measures clinically relevant correctness and semantic fidelity\.

Effect of Report Structure on Clinical Fidelity\.Performance varies substantially across datasets, demonstrating the impact of report structure and clinical heterogeneity on pathology report generation\. TCGA, which contains longer and less standardized free\-text reports, achieves lower CRQS and substantially higher hallucination rates \(HR∼\\sim0\.5\), indicating that lexical metrics such as BLEU, ROUGE, and METEOR can overestimate clinical quality by failing to capture clinically meaningful errors\. In contrast, REG shows stronger agreement between lexical metrics and CRQS, along with lower hallucination rates, likely due to its shorter and semi\-structured reports\. HistAI exhibits intermediate behavior\. As illustrated in Figure[3](https://arxiv.org/html/2607.18448#S4.F3), the relationship between METEOR and CRQS is inconsistent across datasets, highlighting the need for clinically grounded evaluation metrics such as CRQS and benchmarking across datasets with varying reporting styles and noise levels\.

## 5Conclusion

Our findings carry direct implications for how progress in pathology report generation should be measured\. Standard NLG metrics are insufficient proxies for clinical correctness; models with comparable lexical scores can differ substantially in diagnostic accuracy, making such metrics unreliable for guiding development\. Clinically grounded evaluation, as instantiated by CRQS, is essential: by decomposing report quality into factual coverage, hallucination rate, and clinical discordance, it surfaces failure modes that surface\-level metrics obscure\. Equally important is the practice of benchmarking across diverse datasets, encoders, and architectures — single\-setting evaluations risk overstating generalization and masking dataset\-specific confounds\. Taken together, these findings argue for a shift in evaluation norms: clinical correctness, not lexical similarity, should serve as the primary criterion for assessing pathology report generation systems\. Overall, the PathReportEval benchmark, public framework, and evaluation metric provide a foundation for more reliable comparison and future development of clinically faithful pathology report generation models\.

## Acknowledgments and Disclosure of Funding

This research was partially supported by NIH grants 1R01CA297843\-01, 1R03DE033489\-01A1, and NSF grant 2442053\. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH\.

## References

- M\. Baharoon, T\. Heintz, S\. Raissi, M\. Alabbad, M\. Alhammad, H\. AlOmaish, S\. E\. Kim, O\. Banerjee, and P\. Rajpurkar \(2026\)CRIMSON: a clinically\-grounded LLM\-based metric for generative radiology report evaluation\.arXiv preprint arXiv:2603\.06183\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p2.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1)\.
- METEOR: an automatic metric for mt evaluation with improved correlation with human judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,pp\. 65–72\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p2.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1)\.
- Bioptimus \(2025\)H\-optimus\-1External Links:[Link](https://huggingface.co/bioptimus/H-optimus-1)Cited by:[§A\.1](https://arxiv.org/html/2607.18448#A1.SS1.p1.3),[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p2.1)\.
- J\. Bulte, A\. Hering, M\. Schmitt, M\. Veta, N\. Brieu, M\. A\. Kimm, J\. van der Laak, and G\. Litjens \(2024\)HISTAI: an efficient and robust whole\-slide imaging repository for computational pathology challenges\.Scientific Data11\(1\),pp\. 543\.Cited by:[§3\.2](https://arxiv.org/html/2607.18448#S3.SS2.p1.1.2)\.
- P\. Chen, H\. Li, C\. Zhu, S\. Zheng, Z\. Shui, and L\. Yang \(2024a\)WsiCaption: multiple instance generation of pathology reports for gigapixel whole\-slide images\.External Links:2311\.16480,[Link](https://arxiv.org/abs/2311.16480)Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2607.18448#S3.SS2.p1.1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p1.1)\.
- R\. J\. Chen, T\. Ding, M\. Y\. Lu, D\. F\. Williamson, G\. Jaume, A\. H\. Song, B\. Chen, A\. Zhang, D\. Shao, M\. Shaban,et al\.\(2024b\)Towards a general\-purpose foundation model for computational pathology\.Nature Medicine30\(3\),pp\. 850–862\.Cited by:[§A\.1](https://arxiv.org/html/2607.18448#A1.SS1.p1.3),[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p2.1)\.
- College of American Pathologists \(2023a\)Cancer reporting tools\.Note:https://www\.cap\.org/protocols\-and\-guidelines/cancer\-reporting\-tools/cancer\-protocolsCited by:[§1](https://arxiv.org/html/2607.18448#S1.p4.1)\.
- College of American Pathologists \(2023b\)Protocol for the examination of specimens from patients with invasive carcinoma of the breast\.College of American Pathologists,Northfield, IL\.Note:Version 4\.5\.0\.0External Links:[Link](https://www.cap.org/protocols-and-guidelines/cancer-reporting-tools/cancer-protocol-templates)Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p4.1)\.
- T\. Ding, S\. J\. Wagner, A\. H\. Song, R\. J\. Chen, M\. Y\. Lu, A\. Zhang, A\. J\. Vaidya, G\. Jaume, M\. Shaban, A\. Kim,et al\.\(2025\)A multimodal whole\-slide foundation model for pathology\.Nature medicine,pp\. 1–13\.Cited by:[§A\.1](https://arxiv.org/html/2607.18448#A1.SS1.p1.3)\.
- J\. Gamper and N\. Rajpoot \(2021\)Multiple instance captioning: learning representations from histopathology textbooks and articles\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16549–16559\.Cited by:[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Gao, C\. Liu, and Y\. Li \(2025\)S2D\-align: shallow\-to\-deep auxiliary learning for anatomically\-grounded radiology report generation\.arXiv preprint arXiv:2511\.11066\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1)\.
- Z\. Guo, J\. Ma, Y\. Xu, Y\. Wang, L\. Wang, and H\. Chen \(2024\)HistGen: histopathology report generation via local\-global feature encoding and cross\-modal context interaction\.External Links:2403\.05396,[Link](https://arxiv.org/abs/2403.05396)Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p1.1)\.
- D\. Hu, Z\. Jiang, J\. Shi, F\. Xie, K\. Wu, K\. Tang, M\. Cao, J\. Huai, and Y\. Zheng \(2025\)Pathology report generation from whole slide images with knowledge retrieval and multi\-level regional feature selection\.Computer Methods and Programs in Biomedicine263,pp\. 108677\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1)\.
- Z\. Huang, F\. Bianchi, M\. Yuksekgonul, T\. J\. Montine, and J\. Zou \(2023\)A visual–language foundation model for pathology image analysis using medical twitter\.Nat\. Med\.29\(9\),pp\. 2307–2316\.Cited by:[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Jain, A\. Agrawal, A\. Saporta, S\. Q\. Truong, D\. N\. Duong, T\. Bui, P\. Chambon, Y\. Zhang, M\. P\. Lungren, A\. Y\. Ng, C\. P\. Langlotz, and P\. Rajpurkar \(2021\)RadGraph: extracting clinical entities and relations from radiology reports\.External Links:2106\.14463,[Link](https://arxiv.org/abs/2106.14463)Cited by:[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Jin, Q\. Sun, D\. Kang, Z\. Luo, T\. Yu, W\. Han, Y\. Zhang, M\. Wang, D\. Shi, and A\. Grzybowski \(2026\)Grounded report generation for enhancing ophthalmic ultrasound interpretation using vision\-language segmentation models\.npj Digital Medicine\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p2.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1),[§3\.5](https://arxiv.org/html/2607.18448#S3.SS5.p1.1)\.
- M\. Y\. Lu, B\. Chen, D\. F\. Williamson, R\. J\. Chen, I\. Liang, T\. Ding, G\. Jaume, I\. Odintsov, L\. P\. Le, G\. Gerber,et al\.\(2024\)A visual\-language foundation model for computational pathology\.Nature medicine30\(3\),pp\. 863–874\.Cited by:[§A\.1](https://arxiv.org/html/2607.18448#A1.SS1.p1.3),[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px2.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p2.1)\.
- M\. Y\. Lu, B\. Chen, A\. Zhang, D\. F\. Williamson, R\. J\. Chen, T\. Ding, L\. P\. Le, Y\. Chuang, and F\. Mahmood \(2023\)Visual language pretrained multiple instance zero\-shot transfer for histopathology images\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19764–19775\.Cited by:[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1)\.
- R\. T\. Lucassen, S\. P\. Moonemans, T\. van de Luijtgaarden, G\. E\. Breimer, W\. A\. Blokx, and M\. Veta \(2025\)Pathology report generation and multimodal representation learning for cutaneous melanocytic lesions\.InInternational Conference on Medical Image Computing and Computer\-Assisted Intervention,pp\. 502–511\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1)\.
- Mahmood Lab \(2025\)TRIDENT: gigapath slide processing and feature extraction toolkit\.Note:GitHub repositoryCited by:[§A\.1](https://arxiv.org/html/2607.18448#A1.SS1.p1.3),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p2.1)\.
- MICCAI COMPAY Workshop Organizers \(2025\)MICCAI 2025 workshop on computational pathology: report generation challenge\.Note:Challenge website and dataset descriptionExternal Links:[Link](https://reg2025.grand-challenge.org/reg2025)Cited by:[§3\.2](https://arxiv.org/html/2607.18448#S3.SS2.p1.1.3)\.
- Y\. Miura, Y\. Zhang, T\. Chen, H\. Langlotz, and C\. Langlotz \(2021\)Improving factual completeness and consistency of image\-to\-text radiology report generation\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4488–4504\.External Links:[Link](https://aclanthology.org/)Cited by:[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Papineni, S\. Roukos, T\. Ward, and W\. Zhu \(2002\)BLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p2.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1),[§3\.5](https://arxiv.org/html/2607.18448#S3.SS5.p1.1)\.
- S\. Singh, S\. Kapse, J\. Saltz, and P\. Prasanna \(2026\)Semantic context\-aware modality fusion transformer \(scout\): a context\-aware multimodal transformer for concept\-grounded pathology report generation\.External Links:2605\.01144,[Link](https://arxiv.org/abs/2605.01144)Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p1.1)\.
- F\. Yu, M\. Endo, R\. Krishnan, I\. Pan, A\. Tsai, E\. P\. Reis, E\. K\. U\. N\. Fonseca, H\. M\. H\. Lee, Z\. S\. H\. Abad, A\. Y\. Ng, C\. P\. Langlotz, V\. K\. Venugopal, and P\. Rajpurkar \(2023\)Evaluating progress in automatic chest x\-ray radiology report generation\.Patterns4\(9\),pp\. 100802\.External Links:[Document](https://dx.doi.org/10.1016/j.patter.2023.100802)Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p2.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Zhang, G\. Jaume, A\. Vaidya, T\. Ding, and F\. Mahmood \(2025\)Accelerating data processing and benchmarking of ai models for pathology\.External Links:2502\.06750,[Link](https://arxiv.org/abs/2502.06750)Cited by:[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p2.1)\.
- L\. Zhang, B\. Yun, Q\. Li, and Y\. Wang \(2026\)Historical report guided bi\-modal concurrent learning for pathology report generation\.InMedical Image Computing and Computer Assisted Intervention – MICCAI 2025,J\. C\. Gee, D\. C\. Alexander, J\. Hong, J\. E\. Iglesias, C\. H\. Sudre, A\. Venkataraman, P\. Golland, J\. H\. Kim, and J\. Park \(Eds\.\),Cham,pp\. 343–352\.External Links:ISBN 978\-3\-032\-04978\-0Cited by:[§1](https://arxiv.org/html/2607.18448#S1.p1.1),[§2](https://arxiv.org/html/2607.18448#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2607.18448#S3.SS3.p1.1)\.

## Appendix ATechnical appendices and supplementary material

### A\.1Benchmark and Evaluation Framework

TRIDENT provides a standardized pipeline that performs tissue segmentation, patch coordinate extraction at a target magnification, and encoder\-specific feature computation from the resulting tissue regions\. In our workflow, WSIs are first segmented to remove background regions, after which tissue patches are generated at 20×\\timesmagnification before feature encoding\. We use TRIDENT to produce the encoder inputs for CONCHv1\.5Luet al\.\([2024](https://arxiv.org/html/2607.18448#bib.bib7)\), TITANDinget al\.\([2025](https://arxiv.org/html/2607.18448#bib.bib8)\), H\-Optimus\-1Bioptimus \([2025](https://arxiv.org/html/2607.18448#bib.bib25)\), and UNI2\-hChenet al\.\([2024b](https://arxiv.org/html/2607.18448#bib.bib9)\)under a common processing stack, which keeps segmentation, patch extraction, and feature serialization consistent across experiments\. Following the TRIDENT model registry, CONCHv1\.5 uses 512×\\times512 patches at 20×\\times, whereas other encoders use their corresponding encoder\-specific patch configurations within the same frameworkMahmood Lab \([2025](https://arxiv.org/html/2607.18448#bib.bib5)\)\. This standardized extraction stage reduces variation introduced by heterogeneous preprocessing pipelines and makes the downstream comparison of report generation models more controlled\.

### A\.2Experimental Setup

Visual features are extracted with the TRIDENT whole\-slide processing framework using a shared segmentation and 20×\\timespatch\-extraction pipeline before encoder\-specific embedding computation\. To ensure controlled comparison, all methods are reimplemented within a common PyTorch Lightning framework using the same tokenizer and standardized data\-cleaning pipeline\. Experiments are conducted on NVIDIA H100 and H200 GPUs with either 4 or 8 nodes\. The available memory per gpu is 32 GB and and a batch size of 1 is used for all methods\. For reproducibility, we make the dataset splits, benchmarking pipeline, and evaluation code publicly available through our GitHub repository\. This unified setup reduces implementation\-specific variance and likely contributes to the stronger performance we observe for WSI\-Caption, HistGen, and BiGen relative to their originally reported results\.

Performance is evaluated using both standard natural language generation \(NLG\) metrics \(BLEU\-1, BLEU\-2, BLEU\-3, BLEU\-4, ROUGE\-L, and METEOR\) and the proposed clinically grounded metric \(CRQS\)\.

### A\.3Experiments

This appendix reports the full per\-method and per\-encoder results from the final experimental runs for each dataset\. Each table lists the individual model–encoder combinations before averaging and includes both standard NLG metrics and CRQS\. Within each dataset table, the best score in each metric column is shown in bold\.

#### A\.3\.1TCGA Detailed Results

Table 4:Detailed results on TCGA for each method–encoder combination\.
#### A\.3\.2HistAI Detailed Results

Table 5:Detailed results on HistAI for each method–encoder combination\.
#### A\.3\.3REG 2025 Detailed Results

Table 6:Detailed results on REG 2025 for each method–encoder combination\.

### A\.4Vision Encoder Sensitivity

![Refer to caption](https://arxiv.org/html/2607.18448v1/figures/encoder_impact_crqs_all_datasets_grouped_barplot.png)Figure 5:Impact of different vision encoders on Clinical Report Quality Score \(CRQS\) across the TCGA, REG, and HistAI datasets\. The comparison evaluates four pathology report generation methods \(BiGen, WSI\-Caption, HistGen, and SCOUT\) using three visual encoders: CONCH, UNI, and Hoptimus\. Results show that encoder selection comparably affects clinically grounded report quality, with dataset\-specific performance variations observed across methods\.
### A\.5Limitations

Despite its contributions, this study has several limitations\. First, the proposed CRQS metric relies on structured extraction of clinical fields using a language model\-based pipeline\. Although these extracted fields are verified by expert pathologists, the extraction process may still introduce errors or inconsistencies, particularly for ambiguous or complex cases\.

Second, CRQS focuses on predefined clinical attributes and may not capture all aspects of report quality, such as nuanced descriptive language or rare pathological findings\. Extending the metric to incorporate a broader range of clinical concepts remains an important direction for future work\.

Third, our benchmark is limited to three datasets and a fixed set of models and encoders\. While these choices cover a range of scenarios, additional datasets and emerging architectures may further enrich the evaluation landscape\.

### A\.6Broader Impacts

Automated pathology report generation has the potential to improve clinical workflow efficiency and reduce documentation burden\. However, inaccurate or misleading reports could have serious consequences in medical settings\. By emphasizing clinically grounded evaluation, this work aims to promote the development of more reliable and trustworthy systems\.

At the same time, the use of automated extraction methods and machine learning models raises concerns about bias, generalization, and interpretability\. Ensuring that such systems are rigorously validated and used as decision\-support tools, rather than replacements for expert judgment, is critical for safe deployment\.

### A\.7Clinical Fields Extraction for each dataset

The following prompt was used to guide the large language model \(LLM\) in generating dataset\-specific clinical fields and key fields for the Clinical Report Quality Score \(CRQS\) framework\. The prompt was designed to enforce clinically meaningful, pathology\-oriented extraction schemas rather than using a universal reporting template\.

> ``` I am building a dataset-specific CRQS pipeline for pathology report evaluation. CRQS compares extracted clinical facts from: 1. ground-truth pathology report 2. predicted/generated pathology report Do NOT use a universal schema. Build a dataset-specific schema based on the dataset structure and the pathology content. Important clinical constraint: The fields should prioritize facts that are clinically important AND inferable from the histology slide/WSI whenever possible. Do not over-prioritize metadata fields such as organ site, specimen type, laterality, procedure type, margin status, lymph node status, or tumor stage unless they are clearly present and visually inferable in this dataset. Use the pathologist-informed hierarchy: 1. Diagnosis hierarchy - benign_vs_malignant - diagnosis - primary_vs_metastatic - lineage - histologic_type 2. Tumor behavior - in_situ_vs_invasive - in_situ_component 3. Grade / aggressiveness - tumor_grade - differentiation - dataset-specific grades such as Gleason score, WHO grade, nuclear grade, mitotic score, tubule score 4. Invasion hierarchy - invasion_status - invasion_depth - organ-specific invasion - lymphovascular_invasion - perineural_invasion 5. Premalignant / epithelial change - dysplasia_grade - epithelial_atypia 6. Associated morphology - necrosis - inflammation_status - calcification - tumor_volume or tumor_extent if visually meaningful For this dataset, do the following step by step: Step 1: Read and summarize the dataset. Explain: - file structure - keys/columns - whether it has target only or target + prediction - number of cases if visible - report style - organ/disease scope - examples of report text Step 2: Identify all clinically relevant fields for this dataset. Create a single hierarchical flat list: CLINICAL_FIELDS = [...] The list should be ordered so the hierarchy is obvious when reading top to bottom. Step 3: Select top 10 KEY_FIELDS. These must be: - clinically high-impact - commonly present in this dataset - inferable from WSI/histology image when possible - not mostly metadata Step 4: Explain why each key field was selected. Step 5: Write/update config.py. Include: - CLINICAL_FIELDS - KEY_FIELDS - CRQS_WEIGHTS - STRICT_FIELDS - NUMERIC_FIELDS Use raw CRQS weights: CRQS = 0.3*CFC + 0.4*KIR - 0.2*HR - 0.4*CDS Also include normalized CRQS: CRQS_norm = CRQS_raw / 0.7 Step 6: Write extract_fields.py. Make diagnosis/histology extraction data-driven when possible: - learn vocabulary from dataset reports - use stable regex for structured fields such as grade, score, invasion, necrosis, tumor size/volume - avoid manually hard-coding every diagnosis unless needed for normalization - include smoke-test examples Step 7: Write test_extract_fields.py. The smoke test should: - load vocabulary - run several representative reports - check expected extracted fields - print PASS/FAIL Step 8: Write compute_metrics.py. Implement: - CFC = correct target clinical fields / target clinical fields present - KIR = correct key fields / target key fields present - HR = predicted unsupported fields / predicted fields - CDS = discordant comparable fields / comparable fields - CRQS_raw - CRQS_norm Step 9: Write run_crqs.py. It should: - load dataset - learn/save/load vocabulary - extract target and prediction facts - compute metrics per case - save per-case CSV/JSON - save summary CSV/JSON - print summary Step 10: Add a short explanation of how to run: python src/extract_fields.py python src/test_extract_fields.py python src/compute_metrics.py python src/run_crqs.py --input data/<dataset>.json --output_dir outputs Important: Use dataset-specific fields only. Do not assume fields that are not present in the dataset. ```

Table 7:Clinical Fields for REG DatasetTable 8:Key Fields for REG DatasetTable 9:Clinical Fields for HistAI DatasetFieldDescription1\. Diagnosis Hierarchybenign\_vs\_malignantBenign versus malignant disease classificationdiagnosisPrimary pathologic diagnosisprimary\_vs\_metastaticDistinguishes primary tumor from metastatic diseaselineageCellular/tissue lineage of tumorhistologic\_typeHistologic subtype of lesion/tumor2\. Tumor Behaviorin\_situ\_vs\_invasiveWhether lesion is in\-situ or invasivein\_situ\_componentPresence of in\-situ neoplastic component3\. Grade / Aggressivenesstumor\_gradeOverall tumor grade/aggressivenessdifferentiationDegree of differentiationgleason\_scoreProstate Gleason grading scoregrade\_groupISUP prostate grade groupnuclear\_gradeCytologic/nuclear atypia grademitotic\_scoreMitotic activity score/count4\. Invasion Hierarchyinvasion\_statusPresence or absence of invasioninvasion\_depthExtent/depth of tissue invasionlymphovascular\_invasionTumor invasion into lymphatic or vascular spacesperineural\_invasionTumor invasion involving nervesextraprostatic\_extensionTumor extension beyond prostate capsule5\. Premalignant / Epithelial Changedysplasia\_gradeSeverity of epithelial dysplasiaepithelial\_atypiaPresence of epithelial atypia6\. Associated MorphologynecrosisPresence of tumor necrosisinflammation\_statusPresence of inflammatory changesinflammation\_activityDegree/activity of inflammationcalcificationPresence of calcificationstumor\_extentExtent of tumor involvementtumor\_volumeTumor burden or percentage involvement7\. GI\-Specific Mucosal PathologyatrophyMucosal or glandular atrophyintestinal\_metaplasiaPresence of intestinal metaplasiametaplasia\_typeType/subtype of metaplasiahpylori\_statusHelicobacter pylori infection statusgastritis\_typeHistologic subtype of gastritisolga\_stageOLGA gastritis stagingolgim\_stageOLGIM intestinal metaplasia staging8\. Polyp / Adenoma Pathologypolyp\_typeHistologic type of polypserrated\_lesionPresence of serrated lesion morphologyadenoma\_typeHistologic subtype of adenoma9\. Multifocal / Bilateral DiseasemultifocalityPresence of multiple tumor focibilateral\_involvementBilateral organ involvementTable 10:Key Fields for HistAI DatasetTable 11:Clinical Fields for TCGA DatasetFieldDescription1\. Diagnosis Hierarchybenign\_vs\_malignantBenign versus malignant disease classificationdiagnosisPrimary pathologic diagnosisprimary\_vs\_metastaticDistinguishes primary tumor from metastatic diseaselineageCellular or tissue lineage of tumorhistologic\_typeHistologic subtype of tumor or lesion2\. Tumor Behaviorin\_situ\_vs\_invasiveWhether lesion is in\-situ or invasivein\_situ\_componentPresence of an in\-situ neoplastic componenttumor\_focalityUnifocal versus multifocal tumor distribution3\. Grade / Aggressivenesstumor\_gradeOverall tumor grade or aggressivenessdifferentiationDegree of tumor differentiation4\. Invasion Hierarchyinvasion\_statusPresence or absence of invasionlymphovascular\_invasionTumor invasion into lymphatic or vascular spacesperineural\_invasionTumor invasion involving nervescapsular\_invasionTumor invasion through or into capsuleorgan\_specific\_invasionDataset\-specific invasion into adjacent organ structures5\. Spread / Involvementmetastatic\_involvementTumor involvement of distant or secondary siteslymph\_node\_involvementPresence of tumor involvement in lymph nodesmultifocal\_involvementMultiple sites or foci involved by disease6\. Premalignant / Epithelial Changedysplasia\_gradeSeverity of epithelial dysplasiaepithelial\_atypiaPresence of epithelial atypiacarcinoma\_in\_situPresence of carcinoma in situintratubular\_germ\_cell\_neoplasiaPresence of intratubular germ cell neoplasia7\. Associated MorphologynecrosisPresence of tumor necrosisinflammation\_statusPresence of inflammatory changescalcificationPresence of calcificationspsammoma\_bodiesPresence of psammoma bodieshemorrhagePresence of hemorrhagepapillary\_featuresPresence of papillary architectural featurescystic\_changePresence of cystic change8\. Tumor Extent / Burdentumor\_sizeTumor dimension or size when reported9\. Contextual / Lower\-Priority Metadataorgan\_siteAnatomical organ or tissue sitespecimen\_typeType of specimen or sampling proceduremargin\_statusPresence or absence of tumor at surgical marginsTable 12:Key Fields for TCGA Dataset