Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation

# Multi-Agent AI System for Radiology Report Structuring and Quality Assurance with Independent Radiologist Evaluation
Source: [https://arxiv.org/html/2608.18072](https://arxiv.org/html/2608.18072)
\\correspondance

\\extraAuth

Iryna Hartsock1,∗, Cesar Lam2, Christopher Otteni2, Aliya Qayyum2, Robert Gatenby2, Cyrillo Araujo2, and Ghulam Rasool1,2,3,4

###### Abstract

Purpose: To develop and evaluate a locally deployed multi\-agent AI system for radiology report structuring and quality assurance\. Materials and Methods: This retrospective study included 638 radiology reports from CT examinations of the chest, abdomen, and pelvis dictated by 15 board\-certified radiologists in 2023 and 2024\. A multi\-agent AI pipeline was developed to perform report structuring and quality assurance \(QA\)\. The system structured the report into standardized anatomical sections at the sentence level using regex rules and local large language models\. It also detected mismatches between the Findings and Impression sections, or within sections; gender–anatomy conflicts; and undocumented communication of critical findings\. Two board\-certified radiologists independently evaluated a 45\-report subset\. Results: The multi\-agent system structured the Findings sections of all reports \(22,270 sentences\) into a predefined anatomical format while retaining the original report content\. The system flagged 90 \(14\.1%\) reports, most commonly for section mismatches \(80 reports, 12\.5%\)\. In the radiologist evaluation, both reviewers agreed that 31 \(69%\) were correctly restructured, 2 reports \(4%\) were incorrectly restructured, and disagreed on the remaining 12 reports \(27%\)\. Both reviewers agreed that no clinically important information was omitted and no fabricated content was introduced\. Overall QA performance was rated as “excellent” or “good” in 84% of the evaluated reports, with the remaining reports rated as “fair”\. Conclusion: A locally deployed multi\-agent AI system combined radiology report structuring and quality assurance within a single workflow\. The system demonstrated favorable performance in radiologist evaluation\. Such systems may support standardization of reporting and quality assurance in radiology practice\.

\\helveticabold

## 1Keywords:

multi\-agent artificial intelligence, radiology reports, structured reporting, quality assurance, large language models

††firstpage:1## 2Introduction

A challenge in radiology reporting is inconsistent or absent structure, which can lead referring physicians to overlook critical clinical information or spend time searching for it\([17](https://arxiv.org/html/2608.18072#bib.bib1)\)\. Radiology reports include sections such as findings and impressions, with findings sometimes described by anatomic regions or organs\. However, the organization and placement of findings within reports can vary across radiologists, which may complicate communication between radiologists and referring clinicians\([2](https://arxiv.org/html/2608.18072#bib.bib2),[15](https://arxiv.org/html/2608.18072#bib.bib3)\)\. In addition, radiology reports without a consistent structure can limit their secondary use for research, quality monitoring, and clinical decision support systems\([13](https://arxiv.org/html/2608.18072#bib.bib4)\)\.

Several approaches have been proposed to address these limitations\. Structured reporting templates have been developed to standardize radiology reporting and improve consistency, although adoption has been variable due to concerns regarding workflow burden and reporting flexibility\([17](https://arxiv.org/html/2608.18072#bib.bib1),[12](https://arxiv.org/html/2608.18072#bib.bib16)\)\. Recently, natural language processing \(NLP\) and large language model \(LLM\) approaches have been explored to extract structured information from narrative radiology reports or convert free\-text reports into structured formats\([1](https://arxiv.org/html/2608.18072#bib.bib5),[7](https://arxiv.org/html/2608.18072#bib.bib6),[18](https://arxiv.org/html/2608.18072#bib.bib15)\)\. These methods typically focus on extracting specific findings, generating labels, or summarizing report content and do not preserve the full content of the original report\.

Another challenge is the presence of reporting discrepancies or communication errors that can lead to misinterpretation, delayed treatment, or inappropriate clinical management\([3](https://arxiv.org/html/2608.18072#bib.bib7)\)\. In some cases, inconsistencies within a report may require clinicians to re\-examine imaging studies for clarification\. The widespread adoption of voice recognition systems further highlights the need for quality control, as speech\-to\-text reporting may introduce transcription errors or unintended wording changes\([11](https://arxiv.org/html/2608.18072#bib.bib8)\)\. Automated methods can more efficiently detect such inconsistencies than manual review, particularly in high\-volume reporting environments\. Recent studies have explored the use of LLMs for automated error detection in radiology reports, demonstrating that LLMs can identify certain types of inconsistencies within report text\([16](https://arxiv.org/html/2608.18072#bib.bib9),[4](https://arxiv.org/html/2608.18072#bib.bib10),[14](https://arxiv.org/html/2608.18072#bib.bib11),[9](https://arxiv.org/html/2608.18072#bib.bib12),[10](https://arxiv.org/html/2608.18072#bib.bib13)\), representing an emerging and rapidly evolving direction in clinical AI research\([8](https://arxiv.org/html/2608.18072#bib.bib14)\)\.

In this work, we propose a multi\-agent AI system that standardizes radiology report organization and performs automated quality assurance \(QA\) while preserving the original report text\. The system assigns sentences from the report to predefined anatomical sections without rewriting the content\. It also identifies inconsistencies, including mismatches between findings and impressions, discrepancies within sections, gender–anatomy conflicts, and critical findings without documented communication\. This approach enables structured reporting and automated quality checks while allowing radiologists to maintain their reporting style\. A subset of reports was evaluated by radiologists to assess the quality of the multi\-agent AI system\.

## 3Materials and Methods

### 3\.1Radiology Reports

In this retrospective study, we collected 638 complete radiology reports based on CT scans of the chest, abdomen, and pelvis\. Reports were dictated by 15 board\-certified radiologists using a voice\-recognition system between 2023 and 2024\. The reports ranged from 146 to 1,594 words, with an average of 438 words\. Fourteen radiologists organized the Findings section by organ\-based sections using different styles, while reports from one radiologist had largely unstructured Findings sections\. Radiology reports were processed behind the institution’s firewall using locally deployed AI models\. The institutional review board \(IRB\) approved the study and waived the requirement for informed consent from participating radiologists and patients\.

### 3\.2Report Preprocessing for the Structuring Task

For the structuring task, the Findings section was extracted using rule\-based pattern matching to identify section boundaries\. A custom sentence segmentation algorithm was implemented to handle radiology\-specific formatting, including numbered lists, multi\-line sentences, and punctuation inconsistencies\. Short follow\-up fragments \(e\.g\., “Stable since prior”\) were conditionally merged with the preceding sentence using rule\-based pattern matching to preserve contextual meaning\. These steps produced sentence\-level inputs for downstream organ\-based structuring of the report\.

![Refer to caption](https://arxiv.org/html/2608.18072v1/AI-agentic-rad-system.png)Figure 1:Overview of the proposed multi\-agent AI pipeline for radiology report structuring and quality assurance\. The structuring task operates at the sentence level and reorganizes Findings section content into predefined anatomical sections\. Each sentence is first evaluated by the rule\-based agent \(Agent 1\)\. Device\-related sentences and duplicate sentences not classified by Agent 1 are routed directly to the DeepSeek\-R1 reasoning model \(Agent 3\)\. Other unclassified sentences are processed by the LLaMA\-3 model \(Agent 2\), and sentences that remain unclassified or are assigned to nonspecific “Other” categories are subsequently reviewed by Agent 3\. Quality assurance is performed at the whole\-report level by a separate DeepSeek\-R1 agent \(Agent 4\), which flags potential reporting inconsistencies and assigns a severity level to the flag\. The upper\-right panel shows an example output for a Findings–Impression mismatch; similar outputs are generated for gender–anatomy discrepancies and undocumented communication of critical findings\.
### 3\.3Multi\-agent AI system pipeline

We developed a four\-agent pipeline for radiology report structuring and QA, shown in Figure[1](https://arxiv.org/html/2608.18072#S3.F1)\. The first three agents organized findings into the predefined anatomical format shown in Figure[2](https://arxiv.org/html/2608.18072#S3.F2), while the fourth agent performed report\-level QA\. The first agent performed deterministic sentence classification using a domain\-specific dictionary of approximately 600 regular expression \(regex\) patterns covering common anatomical regions and radiological findings\. The classification rules and anatomical assignments were developed with input from a board\-certified radiologist to classify common findings into predefined anatomical sections\. The second agent applied an LLM \(LLaMA\-3–8B, quantized to q8\)\([5](https://arxiv.org/html/2608.18072#bib.bib17)\)to resolve cases not captured by rules, using the full Findings text and local sentence context to infer organ\-level associations\. The third agent employed a reasoning LLM \(DeepSeek\-R1\-Distill\-Llama\-70B, quantized to q4\)\([6](https://arxiv.org/html/2608.18072#bib.bib18)\)for selected sentences requiring additional contextual interpretation\. Device\-related sentences were identified using device\-specific regex patterns and routed directly to DeepSeek\-R1 because they often required assignment to both the “Devices” section and the relevant anatomical section describing the device location or associated finding\. Duplicate sentences that Agent 1 could not classify were also bypassed by the LLaMA agent and routed directly to DeepSeek\-R1\. In addition, DeepSeek\-R1 reviewed sentences that remained unclassified or were assigned to “Other” categories by the LLaMA agent\. In all cases, the sentence being classified was provided together with the full Findings section\.

The fourth agent, also based on DeepSeek\-R1\-Distill\-Llama\-70B \(quantized to q4\), operated at the whole\-report level to identify clinically meaningful inconsistencies, including cross\-section and within\-section laterality discrepancies, negation reversals, polarity conflicts, organ or segment mismatches, size or unit discrepancies, gender–anatomy inconsistencies, and undocumented communication of potentially critical findings, while suppressing clinically acceptable summarization differences between Findings and Impressions\. The QA agent also assigned an overall severity level \(none, minor, major, or critical\) to each report\.

All LLM\-based agents used structured prompts tailored to their respective tasks and generated predefined JSON outputs for automated downstream processing\. Structuring agents performed single\-pass inference without retry loops, whereas the QA agent used a retry mechanism of up to five attempts when strict JSON parsing failed\. The pipeline was implemented in Python and executed on a computing cluster using a single NVIDIA H100 GPU and 12 CPU cores\. Local LLM inference was performed through the Ollama111[https://ollama\.com/](https://ollama.com/)\(accessed June 4, 2026\)platform, which served the quantized LLaMA\-3\-8B and DeepSeek\-R1\-Distill\-Llama\-70B models\. The multi\-agent pipeline is available at:[https://github\.com/lab\-rasool/Multi\-agent\-AI\-system\-for\-Radiology\-Report\-Structuring\-and\-Quality\-Assurance](https://github.com/lab-rasool/Multi-agent-AI-system-for-Radiology-Report-Structuring-and-Quality-Assurance)\.

### 3\.4Expert Evaluation Process

Clinical evaluation was conducted through independent review by 2 board\-certified radiologists using a standardized evaluation form\. The evaluating radiologists were distinct from the group that authored the original reports\. A fixed set of 45 reports was sampled and provided to all reviewers, consisting of three reports per radiologist\-writer \(with author identities redacted\)\. For each radiologist\-writer, up to three reports flagged by the agentic system were randomly selected; if fewer than three were available, all flagged reports were included, and the remaining cases were randomly sampled from unflagged reports to maintain three reports per radiologist\-writer\. This sampling procedure yielded 41 flagged and 4 unflagged reports\. The review focused on both the agentic system’s structuring and QA components\. Reviewers assessed whether sentence\-level classifications were correctly assigned to the appropriate organ\-based sections, whether any clinically relevant information was omitted, or any fabricated content was introduced during structuring, and whether the structured report was preferable to the original report for clinical review\. Reviewers also evaluated whether system\-flagged errors were correct and complete, whether severity assignments were appropriate, whether the reasoning generated by the QA LLM agent was logically consistent and clinically sound, and the overall performance of the QA agent\. Unless otherwise specified, reported evaluation outcomes in the Results section represent agreement between both reviewers\.

![Refer to caption](https://arxiv.org/html/2608.18072v1/example-fig.png)Figure 2:Example of automated structuring of the radiology report Findings section by the proposed multi\-agent AI system\. The left panel shows the original Findings section from a synthetic radiology report\. The right panel shows the system’s corresponding structured output, with findings reorganized into pre\-defined organ\-based sections\. All Findings sections of the radiology reports processed by the pipeline were converted to this structured format\.

## 4Results

### 4\.1Radiology Report Structuring

Sentence\-level classification of Findings section content across 638 radiology reports \(22,270 sentences\) processed by the multi\-agent AI system is summarized in Table[1](https://arxiv.org/html/2608.18072#Sx3.T1)\. Most sentences were handled by the rule\-based stage, with Agent 1 \(regex\) classifying 18,493 sentences \(83\.0%\)\. Agent 2 \(LLaMA 3\) accounted for 2,729 sentences \(12\.3%\), primarily capturing cases where simple pattern matching was insufficient, while Agent 3 \(DeepSeek\-R1\) classified 1,041 sentences \(4\.7%\) that required additional reasoning or disambiguation\. Only 7 sentences \(<<0\.04%\) remained unclassified; these were not true misses but report section headers\. Because reports did not consistently use punctuation, sentence boundaries were determined using a combination of delimiters \(e\.g\., line breaks and spacing\), which occasionally isolated headers as standalone sentences\.

Performance was generally consistent across radiologists, with some variability in how often later agents were required\. Reports written by the radiologists 3, 6, and 13 had the highest proportion of sentences classified by Agent 1 \(93\.3%, 90\.0%, and 90\.3%, respectively\), indicating strong alignment with the rule\-based patterns\. In contrast, reports from radiologists 2 and 4 showed the greatest usage of LLM\-based agents \(26\.7% and 27\.2% combined for Agents 2 and 3\), reflecting increased reliance on contextual classification\. Reports from radiologist 11 showed the highest use of Agent 3 \(12\.7%\), suggesting a greater proportion of sentences requiring deeper reasoning or disambiguation compared with other radiologists\.

### 4\.2Radiology Report Quality Assurance

The counts of various QA flags across 638 reports are shown in Table[2](https://arxiv.org/html/2608.18072#Sx3.T2)\. A total of 90 reports \(14\.1%\) were flagged by Agent 4 \(DeepSeek\-R1\) for potential inconsistencies or errors\. Most flags were non\-critical: 40 reports \(6\.3%\) contained major severity findings, 44 \(6\.9%\) contained minor findings, and only 6 \(0\.9%\) were classified as critical\. The most common error type was section mismatches, identified in 80 reports \(12\.5%\), indicating discrepancies between Findings and Impression sections or within either section\. Gender\-anatomy discrepancies were rare \(4 reports, 0\.6%\), as were missing critical findings communication flags \(9 reports, 1\.4%\)\. Across radiologists, the proportion of flagged reports varied, ranging from 4\.8% \(radiologist 3\) to 27\.5% \(radiologist 1\)\. Radiologists 1, 6, and 9 showed higher flagging rates \(27\.5%, 22\.2%, and 24\.1%, respectively\), while others such as radiologists 3 and 14 had relatively low rates \(4\.8% and 5\.0%\)\.

### 4\.3Processing Time

The average processing time per report for each stage of the pipeline is shown in Table[3](https://arxiv.org/html/2608.18072#Sx3.T3)\. The overall average total processing time was 55\.6 seconds per report\. The regex\-based stage \(Agent 1\) was the fastest component \(0\.54 seconds per report\), followed by Agent 2 \(1\.31 seconds\)\. In contrast, the majority of processing time was attributed to the DeepSeek\-R1 agents, with Agent 3 averaging 28\.62 seconds and Agent 4 averaging 25\.13 seconds per report\. Processing time varied across radiologists, with the longest average runtime observed for radiologist 15 \(265\.5 seconds\)\. This was due to increased Agent 3 processing time, as radiologist 15’s reports contained poorly organized findings by anatomical sections, requiring more contextual interpretation by the reasoning model\.

### 4\.4Radiologist Evaluation of the Performance of the Multi\-Agent AI System

Independent evaluations performed by two radiologists across 45 \(7% of all reports\) reports for both the structuring and QA tasks are summarized in Table[4](https://arxiv.org/html/2608.18072#Sx3.T4)\. For report restructuring, both reviewers agreed that 31 reports \(69%\) were correctly restructured, while 2 reports \(4%\) were incorrectly restructured, meaning at least 1 sentence was not correctly classified\. Reviewer disagreement occurred in 12 reports \(27%\), primarily involving borderline section assignment or organizational preferences\. Both reviewers agreed that no clinically important information was omitted and no fabricated information was introduced in any evaluated report\. For clinical usability, both reviewers rated the restructured report as “about the same” as the original report in 23 cases \(51%\), “better” in 1 case \(2%\), and “worse” in 3 cases \(7%\), with disagreement in 18 cases \(40%\), mostly between better and “about the same”\. The only report that both radiologists agreed was improved by restructuring was the report whose Findings section was not organized by organ system in the original report\.

For QA\-related tasks, mismatches between the Findings and Impression sections, or within either section, were correctly identified in 27 reports, while the absence of a mismatch was correctly identified in 10 reports; overall, 37 reports \(82%\) had correctly assessed mismatches\. False\-positive mismatch flags occurred in 5 reports, with evaluator disagreement regarding mismatch assessment in an additional 3 reports\. Gender–anatomy discrepancy detection produced 1 incorrect discrepancy flag across all evaluated reports, caused by misidentification of the research coordinator’s name in the indication section as the patient’s name, resulting in an apparent gender mismatch\. Detection of undocumented communication of potentially critical findings correctly identified missing communication in 5 reports and correctly identified no communication issue in 38 reports, resulting in correct assessment in 43 reports \(96%\)\. One false\-positive flag and one evaluator disagreement were also identified\. Severity assignment was considered correct in 28 reports \(22 minor and 6 major\) and incorrect in 2 reports, with evaluator disagreement in 15 reports\. Overall QA performance was rated as “excellent” in 21 reports, “good” in 2 reports, and “fair” in 7 reports, with disagreement between “excellent” and “good” assessments in 15 reports\.

## 5Discussion

In this study, we developed and evaluated a multi\-agent AI system for radiology report structuring and quality assurance, combining rule\-based methods with locally deployed LLMs\. The system organized radiology reports written in varying styles into a standardized anatomical structure, preserving the original report text while simultaneously performing automated QA analysis for clinically relevant inconsistencies\. Independent radiologist evaluation of the 45\-report subset demonstrated favorable performance for both structuring and QA tasks\.

By combining deterministic rule\-based classification with selective LLM\-based reasoning for more ambiguous findings, the pipeline achieved consistent report organization while limiting reliance on computationally intensive models\. Previous studies have demonstrated that LLMs can transform free\-text radiology reports into structured formats, including with locally deployed open\-source models\([1](https://arxiv.org/html/2608.18072#bib.bib5),[7](https://arxiv.org/html/2608.18072#bib.bib6),[18](https://arxiv.org/html/2608.18072#bib.bib15)\)\. In contrast to approaches that generate or rewrite structured report content, the present system assigned existing report sentences to anatomical sections without modifying the source text, thereby limiting opportunities for unsupported content generation\. Approximately 95% of sentences were classified by the regex\-based and LLaMA\-based agents without requiring escalation to the DeepSeek\-R1 reasoning model\. Both reviewers agreed that 69% of reports were correctly restructured, while only 4% were considered incorrectly restructured\. Disagreement occurred in 27% of reports, most often involving findings that could reasonably be assigned to more than one anatomical section because of overlapping anatomy or differences in report organization preferences\. Importantly, both reviewers agreed that no clinically important findings were omitted and no fabricated findings were introduced, indicating that the original report content was preserved despite occasional differences in preferred organization\. Reviewers most commonly rated the restructured reports as comparable to the original reports rather than superior\. This was largely attributable to occasional duplication of findings across multiple anatomical sections when a statement was considered relevant to more than one category\. Although this approach reduced the risk of omitting information, duplicated statements sometimes reduced the report’s perceived readability\.

The QA component identified several types of reporting inconsistencies, predominantly mismatches between Findings and Impression sections, with relatively few gender–anatomy discrepancies or cases of potentially critical findings without documented communication\. These findings are consistent with recent studies demonstrating the potential of LLMs to identify errors and internal inconsistencies in radiology reports\([16](https://arxiv.org/html/2608.18072#bib.bib9),[4](https://arxiv.org/html/2608.18072#bib.bib10),[14](https://arxiv.org/html/2608.18072#bib.bib11),[9](https://arxiv.org/html/2608.18072#bib.bib12),[10](https://arxiv.org/html/2608.18072#bib.bib13)\)\. Previous LLM\-based radiology QA studies have evaluated errors such as laterality, negation, transcription, omission, and discrepancies between Findings and Impression sections\. The present QA agent extended this scope by assessing inconsistencies both across and within report sections and by evaluating gender–anatomy inconsistencies and undocumented communication of potentially critical findings\. To our knowledge, these latter error categories have not been systematically evaluated in prior LLM\-based radiology report QA studies\.

In radiologist review, all but one identified mismatch were detected by the QA agent\. In one report, one reviewer identified an additional discrepancy involving conflicting references to abdominoperineal resection \(APR\) and low anterior resection \(LAR\) that was not flagged by the QA agent or identified by the second reviewer, illustrating that subtle inconsistencies may also be overlooked during manual review\. False\-positive QA flags occurred in 7 of the 45 evaluated reports\. Both reviewers rated overall QA performance as either “excellent” or “good” in 38 of 45 reports \(84%\), although they differed between these two favorable categories in 15 reports; the remaining 7 reports \(16%\) were rated as “fair” by both reviewers\. No reports received a “poor” rating\. Reviewer assessment indicated that the QA agent’s reasoning was clinically reasonable, with disagreements on distinctions between fully correct and mostly correct reasoning rather than fundamentally incorrect conclusions\. Despite requiring approximately 25 seconds per report, the QA agent demonstrated good performance across multiple error categories\.

A potential advantage of the proposed system is the integration of report structuring and quality assurance within a single workflow\. Inconsistent organization of radiology reports can make it more difficult for referring physicians to quickly identify and interpret clinically relevant findings\. By reorganizing findings into a consistent anatomical framework, the system provides a standardized report structure while preserving the original report content\. It also achieved consistent structuring across 15 different radiologist writing styles, underscoring its ability to handle the degree of variability commonly encountered in real\-world reporting environments\. The QA component further enables automated identification of reporting inconsistencies and communication issues that may otherwise require manual review\. In addition, all models were deployed locally behind the institution’s firewall\.

This study has several limitations\. Clinical evaluation was limited to a review of 45 reports by two radiologists\. While a larger evaluation cohort would strengthen the findings, a comprehensive radiologist review of multi\-agent outputs would require substantial time and effort\. The expert evaluation subset was intentionally enriched for QA\-flagged reports \(41 flagged and 4 unflagged\) and therefore was not intended to provide unbiased estimates of overall QA performance across the full dataset\. Some aspects of report organization are also inherently subjective, particularly when findings are relevant to multiple anatomical sections and may reasonably be categorized differently by different radiologists\. In a small number of cases, the restructuring process duplicated findings across multiple sections, improving information retention but occasionally reducing readability\. Similarly, the QA component was designed to prioritize identifying potential inconsistencies and therefore produced a small number of false\-positive flags\. Future work should focus on validation across additional institutions, imaging modalities, and reporting styles, as well as further refinement of the structuring and QA components\.

In conclusion, the proposed system demonstrated that automated report structuring and quality assurance can be integrated within a single workflow\. The combination of deterministic and reasoning\-based agents enabled consistent organization of report content and the identification of clinically relevant reporting discrepancies across a heterogeneous dataset\. Although further validation is needed, these findings suggest that agent\-based AI approaches may help improve reporting consistency and support quality assurance processes in radiology practice\.

## Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest\.

## Author Contributions

Conceptualization, I\.H\., C\.L\., C\.A\., and G\.R\.; software development and formal analysis, I\.H\.; LLM prompt refinement and keyword selection, I\.H\. and C\.A\.; radiology report evaluation, C\.L\. and C\.O\.; writing—original draft, I\.H\.; funding acquisition, A\.Q\., R\.G\., and G\.R\.;writing—review and editing, all authors\.

## Data Availability Statement

The radiology report data analyzed in this study are not publicly available because they contain protected health information\.

## References

- \[1\]L\. C\. Adams, D\. Truhn, F\. Busch, A\. Kader, S\. M\. Niehues, M\. R\. Makowski,et al\.\(2023\)Leveraging GPT\-4 for post hoc transformation of free\-text radiology reports into structured reporting: a multilingual feasibility study\.Radiology307\(4\),pp\. e230725\.External Links:[Document](https://dx.doi.org/10.1148/radiol.230725)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p2.1),[§5](https://arxiv.org/html/2608.18072#S5.p2.1)\.
- \[2\]J\. M\. L\. Bosmans, J\. J\. Weyler, A\. M\. De Schepper, and P\. M\. Parizel\(2011\)The radiology report as seen by radiologists and referring clinicians: results of the COVER and ROVER surveys\.Radiology259\(1\),pp\. 184–195\.External Links:[Document](https://dx.doi.org/10.1148/radiol.10101045)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p1.1)\.
- \[3\]A\. P\. Brady\(2017\)Error and discrepancy in radiology: inevitable or avoidable?\.Insights into Imaging8\(1\),pp\. 171–182\.External Links:[Document](https://dx.doi.org/10.1007/s13244-016-0534-1)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1)\.
- \[4\]R\. J\. Gertz, T\. Dratsch, A\. C\. Bunck, S\. Lennartz, A\. I\. Iuga, M\. G\. Hellmich,et al\.\(2024\)Potential of GPT\-4 for detecting errors in radiology reports: implications for reporting accuracy\.Radiology311\(1\),pp\. e232714\.External Links:[Document](https://dx.doi.org/10.1148/radiol.232714)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1),[§5](https://arxiv.org/html/2608.18072#S5.p3.1)\.
- \[5\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle,et al\.\(2024\)The Llama 3 herd of models\.Note:Accessed 2026\-04\-24External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3\.3](https://arxiv.org/html/2608.18072#S3.SS3.p1.1)\.
- \[6\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Wang, R\. Zhu,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§3\.3](https://arxiv.org/html/2608.18072#S3.SS3.p1.1)\.
- \[7\]I\. Hartsock, C\. Araujo, L\. Folio, and G\. Rasool\(2026\)Improving radiology report conciseness and structure via local large language models\.Journal of Imaging Informatics in Medicine39\(1\),pp\. 1005–1016\.External Links:[Document](https://dx.doi.org/10.1007/s10278-025-01510-w)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p2.1),[§5](https://arxiv.org/html/2608.18072#S5.p2.1)\.
- \[8\]I\. Hartsock, N\. Koutsoubis, S\. Ahmed, N\. Parker, M\. B\. Schabath, C\. Araujo,et al\.\(2026\)Clinical AI in radiology: foundations, trends, applications, and emerging directions\.Cancers18\(6\),pp\. 942\.External Links:[Document](https://dx.doi.org/10.3390/cancers18060942)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1)\.
- \[9\]S\. Kim, D\. Kim, H\. J\. Shin, S\. H\. Lee, Y\. Kang, S\. Jeong,et al\.\(2025\)Large\-scale validation of the feasibility of GPT\-4 as a proofreading tool for head CT reports\.Radiology314\(1\),pp\. e240701\.External Links:[Document](https://dx.doi.org/10.1148/radiol.240701)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1),[§5](https://arxiv.org/html/2608.18072#S5.p3.1)\.
- \[10\]S\. Kim, S\. Lee, S\. Y\. Lee, J\. Kim, K\. Kan, H\. Lee, and D\. Yoon\(2026\)Improving radiology report error detection using a multipass large language model: framework development and validation\.JMIR Medical Informatics14,pp\. e87368\.External Links:[Document](https://dx.doi.org/10.2196/87368),[Link](https://doi.org/10.2196/87368)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1),[§5](https://arxiv.org/html/2608.18072#S5.p3.1)\.
- \[11\]S\. McGurk, K\. Brauer, T\. V\. Macfarlane, and K\. A\. Duncan\(2008\)The effect of voice recognition software on comparative error rates in radiology reports\.British Journal of Radiology81\(970\),pp\. 767–770\.External Links:[Document](https://dx.doi.org/10.1259/bjr/20698753)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1)\.
- \[12\]J\. M\. Nobel, K\. van Geel, and S\. G\. F\. Robben\(2022\)Structured reporting in radiology: a systematic review to explore its potential\.European Radiology32\(4\),pp\. 2837–2854\.External Links:[Document](https://dx.doi.org/10.1007/s00330-021-08327-5),[Link](https://doi.org/10.1007/s00330-021-08327-5)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p2.1)\.
- \[13\]E\. Pons, L\. M\. M\. Braun, M\. G\. M\. Hunink, and J\. A\. Kors\(2016\)Natural language processing in radiology: a systematic review\.Radiology279\(2\),pp\. 329–343\.External Links:[Document](https://dx.doi.org/10.1148/radiol.16142770)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p1.1)\.
- \[14\]B\. Salam, C\. Stüwe, S\. Nowak, A\. M\. Sprinkart, M\. Theis, D\. Kravchenko,et al\.\(2025\)Large language models for error detection in radiology reports: a comparative analysis between closed\-source and privacy\-compliant open\-source models\.European Radiology35\(8\),pp\. 4549–4557\.External Links:[Document](https://dx.doi.org/10.1007/s00330-025-11438-y)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1),[§5](https://arxiv.org/html/2608.18072#S5.p3.1)\.
- \[15\]C\. L\. Sistrom and C\. P\. Langlotz\(2005\)A framework for improving radiology reporting\.Journal of the American College of Radiology2\(2\),pp\. 159–167\.External Links:[Document](https://dx.doi.org/10.1016/j.jacr.2004.06.015)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p1.1)\.
- \[16\]C\. Sun, K\. Teichman, Y\. Zhou, B\. Critelli, D\. Nauheim, G\. Keir,et al\.\(2025\)Generative large language models trained for detecting errors in radiology reports\.Radiology315\(2\),pp\. e242575\.External Links:[Document](https://dx.doi.org/10.1148/radiol.242575)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p3.1),[§5](https://arxiv.org/html/2608.18072#S5.p3.1)\.
- \[17\]D\. L\. Weiss and C\. P\. Langlotz\(2008\)Structured reporting: patient care enhancement or productivity nightmare?\.Radiology249\(3\),pp\. 739–747\.External Links:[Document](https://dx.doi.org/10.1148/radiol.2493080988)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p1.1),[§2](https://arxiv.org/html/2608.18072#S2.p2.1)\.
- \[18\]P\. Woźnicki, C\. Laqua, I\. Fiku, A\. Hekalo, D\. Truhn, S\. Engelhardt, J\. Kather, S\. Foersch, T\. Akinci D’Antonoli, D\. Pinto dos Santos, B\. Baeßler, and F\. C\. Laqua\(2025\)Automatic structuring of radiology reports with on\-premise open\-source large language models\.European Radiology35\(4\),pp\. 2018–2029\.External Links:[Document](https://dx.doi.org/10.1007/s00330-024-11074-y),[Link](https://doi.org/10.1007/s00330-024-11074-y)Cited by:[§2](https://arxiv.org/html/2608.18072#S2.p2.1),[§5](https://arxiv.org/html/2608.18072#S5.p2.1)\.

Table 1:Number of reports and total number of sentences in the Findings sections for each radiologist, with sentence\-level classification distribution across the three\-agent report structuring pipeline, including sentences classified by Agent 1 \(regex\), Agent 2 \(LLaMA 3\), Agent 3 \(DeepSeek\-R1\), and remaining unclassified sentences\.RadiologistReports, nSentences, nSentences classified by, n \(%\)Unclassifiedsentences, n \(%\)Agent 1\(regex\)Agent 2\(LLaMA 3\)Agent 3\(DeepSeek\-R1\)Radiologist 1401,4031,151187632\(82%\)\(13\.3%\)\(4\.5%\)\(<<0\.2%\)Radiologist 219767562141640\(73\.3%\)\(18\.4%\)\(8\.3%\)Radiologist 3421,3821,28930612\(93\.3%\)\(2\.2%\)\(4\.4%\)\(<<0\.2%\)Radiologist 4452,2111,6094401620\(72\.8%\)\(19\.9%\)\(7\.3%\)Radiologist 5622,1481,769344350\(82\.4%\)\(16%\)\(1\.6%\)Radiologist 6271,4871,338112370\(90%\)\(7\.5%\)\(2\.5%\)Radiologist 7391,021851132371\(83\.3%\)\(12\.9%\)\(3\.6%\)\(<<0\.2%\)Radiologist 8211,192984152560\(82\.6%\)\(12\.8%\)\(4\.7%\)Radiologist 92994878286800\(82\.5%\)\(9\.1%\)\(8\.4%\)Radiologist 101159948980300\(81\.6%\)\(13\.4%\)\(5%\)Radiologist 11361,1208611171420\(76\.9%\)\(10\.4%\)\(12\.7%\)Radiologist 121193,1862,766320982\(86\.8%\)\(10%\)\(3\.1%\)\(0\.1%\)Radiologist 13712,2021,988147670\(90\.3%\)\(6\.7%\)\(3%\)Radiologist 14602,1171,685362700\(79\.6%\)\(17\.1%\)\(3\.3%\)Radiologist 151748736979390\(75\.8%\)\(16\.2%\)\(8%\)Total63822,27018,4932,7291,0417\(83\.04%\)\(12\.25%\)\(4\.67%\)\(<<0\.04%\)

Percentages may not total 100% because of rounding\.

Table 2:Report\-level QA flags identified by Agent 4 \(DeepSeek\-R1\) across 638 radiology reports, showing the total number of reports reviewed for each radiologist, the number of reports flagged by the QA agent, the severity of flagged errors \(critical, major, minor\), and the frequency of specific error categories, including section mismatches, gender–anatomy discrepancies, and missing communication of critical findings\.RadiologistReports, nFlaggedreports,n \(%\)Flag severity, n \(%\)Error type, n \(%\)CriticalMajorMinorSectionmismatchGender\-anatomydiscrepancyMissing criticalcommunicationRadiologist 140110651001\(27\.5%\)\(15%\)\(12\.5%\)\(25%\)\(2\.5%\)Radiologist 2192020200\(10\.5%\)\(10\.5%\)\(10\.5%\)Radiologist 3422002210\(4\.8%\)\(4\.8%\)\(4\.8%\)\(2\.4%\)Radiologist 4454112321\(8\.9%\)\(2\.2%\)\(2\.2%\)\(4\.4%\)\(6\.7%\)\(4\.4%\)\(2\.2%\)Radiologist 5626042600\(9\.7%\)\(6\.5%\)\(3\.2%\)\(9\.7%\)Radiologist 6276123501\(22\.2%\)\(3\.7%\)\(7\.4%\)\(11\.1%\)\(18\.5%\)\(3\.7%\)Radiologist 7394004400\(10\.3%\)\(10\.3%\)\(10\.3%\)Radiologist 8213111201\(14\.3%\)\(4\.8%\)\(4\.8%\)\(4\.8%\)\(9\.5%\)\(4\.8%\)Radiologist 9297133701\(24\.1%\)\(3\.4%\)\(10\.3%\)\(10\.3%\)\(24\.1%\)\(3\.4%\)Radiologist 10111001100\(9\.1%\)\(9\.1%\)\(9\.1%\)Radiologist 11365023300\(13\.9%\)\(5\.6%\)\(8\.3%\)\(8\.3%\)Radiologist 1211922012102211\(18\.5%\)\(10\.1%\)\(8\.4%\)\(18\.5%\)\(0\.8%\)\(0\.8%\)Radiologist 137111155901\(15\.5%\)\(1\.4%\)\(7%\)\(7%\)\(12\.7%\)\(1\.4%\)Radiologist 14603012300\(5%\)\(1\.7%\)\(3\.3%\)\(5%\)Radiologist 15173111102\(17\.6%\)\(5\.9%\)\(5\.9%\)\(5\.9%\)\(5\.9%\)\(11\.8%\)Total63890640448049\(14\.1%\)\(0\.9%\)\(6\.3%\)\(6\.9%\)\(12\.5%\)\(0\.6%\)\(1\.4%\)

Error categories are not mutually exclusive; a single report may contain multiple error types\.

Table 3:Average processing time per report across 15 radiologists for radiology report structuring and quality assurance tasks performed by the multi\-agent AI pipeline\. Columns represent average processing times for each agent, while the rightmost column shows the overall average total processing time per report for each radiologist\.Table 4:Independent radiologist evaluation of report structuring and QA performance in 45 reports\.TaskEvaluation categoryAnswerReports, n \(%\)StructuringNewly restructured report is correctly structuredYes31 \(69%\)No2 \(4%\)Evaluator disagreement12 \(27%\)Clinically important information was omitted from the newly structured reportYes0No45 \(100%\)Fabricated information was introduced in the newly structured reportYes0No45 \(100%\)Newly structured report is better for clinical review than the original reportYes1 \(2%\)About the same23 \(51%\)No3 \(7%\)Evaluator disagreement18 \(40%\)QADetection of mismatches between the Findings and Impression sections or within either sectionMismatch detected27 \(60%\)No mismatch detected10 \(22%\)False\-positive flag5 \(11%\)Evaluator disagreement3 \(7%\)Detection of gender–anatomy discrepanciesFalse\-positive flag1 \(2%\)No discrepancy detected44 \(98%\)Detection of undocumented communication of critical findingsIdentified missing communication5 \(11%\)Identified no communication issue38 \(84%\)False\-positive flag1 \(2%\)Evaluator disagreement1 \(2%\)Flag severityCorrectly assigned28 \(62%\)Incorrectly assigned2 \(4%\)Evaluator disagreement15 \(33%\)QA\-agent reasoning is correct and clinically reasonableYes29 \(64%\)Mostly6 \(13%\)Evaluator disagreement10 \(22%\)Overall QA performanceExcellent21 \(47%\)Good2 \(4%\)Fair7 \(16%\)Evaluator disagreement15 \(33%\)
Values are numbers of reports, with percentages calculated using the 45\-report expert evaluation subset\. Evaluator disagreement indicates discordant assessments between the two reviewers\. Percentages may not total 100% because of rounding\.