From ML Predictions to Informed Diagnostic Assistance Using the Toulmin Model of Argumentation

# From ML Predictions to Informed Diagnostic Assistance using the Toulmin Model of Argumentation
Source: [https://arxiv.org/html/2607.09664](https://arxiv.org/html/2607.09664)
Anca Marginean[ORCID: 0000\-0001\-8426\-588X](https://orcid.org/0000-0001-8426-588X)Adrian Groza[ORCID: 0000\-0003\-0143\-5631](https://orcid.org/0000-0003-0143-5631)Artificial Intelligence Research Institute AIRi@UTCN, Technical University of Cluj\-Napoca, Romania

###### Abstract

To provide a structured and interpretable assessment, we decompose the image\-based diagnosis into components following the Toulmin model of argumentation\. This model consists of a claim, grounds, warrant, qualifier, rebuttal, and backing\. Consider a claim generated by a machine learning \(ML\) model for retinal diagnosis\. Rather than accepting this claim at face value, one could either apply explainable AI \(XAI\) methods or adopt an argumentation\-based approach\. In our framework, a model specialized in biomarker extraction from images provides the grounds\. The warrant—linking the grounds to the claim—is analyzed by an agent equipped with medical knowledge; in our architecture, this role is fulfilled by a MedGemma agent\. The qualifier is determined based on the overall quantitative evaluation of both the warrant and grounds models\. Finally, a rebuttal is constructed using image similarity measures computed with MedSigLip\. All these components are presented to the human expert, enabling a more informed and critical assessment of the ML\-generated diagnosis\.

Keywords:Toulmin model of argumentation; MedGemma; retinal diagnosis; object detection

## 1Introduction

There is increasing interest in AI approaches that provide transparent and evidence\-based reasoning\. Argumentation frameworks are well\-suited to this objective because they explicitly connect observations, assumptions, and conclusions while also allowing uncertainty and counter\-evidence to be represented\. The Toulmin model of argumentation is particularly relevant, as it structures reasoning through components such as claim, grounds, warrant, qualifier, rebuttal, and backing\.

In this work, we explore the use of the Toulmin model for a multimodal retinal image assessment system\. Our approach combines automated biomarker extraction from OCT scans, retrieval of visually similar reference cases, and reasoning performed by a medical large language model\. By organizing these heterogeneous sources of information into a coherent argumentative structure, the system aims to support clinicians with a more interpretable assessment process\. We evaluated the proposed system on two tasks: T1\) given an OCT image, determine whether the patient suffers from age\-related macular degeneration \(AMD\); T2\) given an OCT image from a patient with AMD, identify the stage of the disease\.

## 2Reference cases

In order to build our set of reference cases, we used cases from the OCT Training Manual provided by Heidelberg, the manufacturer of the OCT acquisition device\.

Each case includes a fundus image, an OCT image, a list of findings corresponding to a structured OCT assessment, a clinical interpretation, and a visually annotated image highlighting the detected alterations\. These alterations may be located in different areas of retina: sub\-RPE, subretinal, intraretinal, and epi\-/preretinal\. For each identified alteration, a textual description is provided\. Figure[1](https://arxiv.org/html/2607.09664#S2.F1)presents an example of such a case \(Case 3\)\.

We extracted all the cases and we built the*RefC\_\\\_dataset*with9393cases\. We selected the OCT training manual because it serves as a reference guide for OCT assessment and is not focused on any single retinal condition\. In addition, unlike most publicly available OCT datasets, these cases provide detailed information on the appearance of the alterations together with clear visual annotations indicating their localization and an interpretation\. Together, this information can be highly informative for the human expert\. The purpose of this dataset is to support the diagnostic process in two respects: first, by helping to validate or refine biomarker identification, and second, by supporting the reasoning that links these biomarkers to a diagnosis or staging information\.

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/manual_sample5.png)

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/manual_sample3.png)

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/manual_sample4.png)

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/manual_sample2.png)

Figure 1:Case 3 included in the OCTTraining document: the raw OCT image, the structuredF​i​n​d​i​n​g​sFindings, visualisation of the alterations andI​n​t​e​r​p​r​e​t​a​t​i​o​nInterpretation
## 3Biomarker extraction models

Table 1:Subset of evaluations for YOLOE models extracted fromArdeleanet al\.\([2025](https://arxiv.org/html/2607.09664#bib.bib7)\)We use two publicly available models for biomarker extractionArdeleanet al\.\([2025](https://arxiv.org/html/2607.09664#bib.bib7)\)\. Both models are based on YOLOE and were trained on two different OCT datasets: AROIMelinščaket al\.\([2021](https://arxiv.org/html/2607.09664#bib.bib4)\)and OCT5KArikanet al\.\([2025](https://arxiv.org/html/2607.09664#bib.bib6)\)\.

The AROI dataset contains OCT scans from patients with AMD, whereas OCT5k covers a broader range of retinal pathologies, including AMD and DME \(diabetic macular edema\)\. The YOLOE trained on AROI detects three biomarkers: SRF\(subretinal fluid\), IRF\(intraretinal fluid\), and PED \(pigment epithelial detachment\)\. The model trained on OCT5k targets nine biomarkers, including soft drusen, hard drusen, retinal fluid, geographic atrophy, and choroidal folds\. According to the evaluations reported inArdeleanet al\.\([2025](https://arxiv.org/html/2607.09664#bib.bib7)\)\(Table[1](https://arxiv.org/html/2607.09664#S3.T1)\), YOLOE trained on AROI achieves good performances for PED, SRF and IRF detection, although very small IRF regions may occasionally be missed\. In contrast, the model trained on OCT5k shows lower overall performance\.

An important observation is the relation between PED and drusen: once the size of a drusen increases drastically, or multiple drusens merge, the resulting alteration is called drusenoid PED\. There are two other types of PED: fibrovascular \(F\-PED\) and serous PED \(S\-PED\)\. The AROI dataset, consequently, the AROI model, does not differentiate between PED types and tends to consider drusens of any size as PED\. This observation is very important since the staging of AMD depends on the presence or absence of PED\. Figure[2](https://arxiv.org/html/2607.09664#S3.F2)shows the detections of YOLOE AROI and OCT5k model on the case included in Figure[1](https://arxiv.org/html/2607.09664#S2.F1)\. It can be observed that the OCT5k model correctly identifies drusens \(hard or soft\), while the AROI model considers all the drusens to be PED\.

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/case_manual_yoloe.png)Figure 2:YOLOE models \(left: AROI, right: OCT5k\) detections for the Case 3 \(Figure[1](https://arxiv.org/html/2607.09664#S2.F1)\)\. For AROI model, Red = PED\. For the OCT5k model, Blue = Hard Drusen, and Red = SoftDrusen\.We evaluated YOLOE AROI model on the reference cases\. In our reference cases, no annotated bounding boxes are available\. Therefore, we consider PED, IRF, SRF to be present whenever the corresponding biomarker is mentioned in given findings\. Image\-level precision and recall are reported in Table[3](https://arxiv.org/html/2607.09664#S3)\. As expected, PED shows low precision but high recall\. The model frequently confuses drusen or other RPE changes with PED, which is anatomically plausible since both involve RPE elevation, and the AROI dataset does not explicitly emphasize the distinction between drusen and PED\. We also observe that the number of PED false negatives at a threshold of0\.50\.5increases substantially compared to a threshold of0\.250\.25\. A manual review of these FN samples showed that PED was consistently present together with other alterations located near the RPE\. These observations are later incorporated into the construction of our qualifier\.

Table 2:Image level evaluation of YOLOE AROI model on reference cases*RefC\_\\\_Dataset*![[Uncaptioned image]](https://arxiv.org/html/2607.09664v1/fig/case28_1.png)

![[Uncaptioned image]](https://arxiv.org/html/2607.09664v1/fig/case13_1.png)

Figure 3:The closest cases to Case 3: Case 28 \(left\)\.955\.955similarity\. Case 13 \(right\)\.954\.954similarity\.

## 4System architecture

We propose a system \(Figure[4](https://arxiv.org/html/2607.09664#S4.F4)\) that \(i\) detects evidence in the input image and summarizes the detections \(grounds\), \(ii\) retrieves visually similar cases from a reference case dataset \(rebuttal\), and \(iii\) revives a textual qualifier based on the overall evaluation of the detection and retrieval modules\. This information is then provided to MedGemma for reasoning\. Both the extracted evidence and the generated reasoning are presented to the human expert\.

MedGemma is a medically adapted version of the Gemma LLM\. The multimodal variants include a SigLIP image encoder pretrained on medical data, including ophthalmology images \(mostly eye fundus\)\. The LLM component was trained not only on medical question\-answer pairs but also on medical text, health records, and medical record comprehension tasks\. We use the 4\-bit quantization of the multimodal variant 27B from Huggingface, without any additional finetuning\.

![Refer to caption](https://arxiv.org/html/2607.09664v1/fig/eut_medgemma1.png)Figure 4:System architecture### 4\.1Grounds \- Biomarker detection models

The grounds for our reasoning are extracted with two YOLOE detection models\. Raw YOLOE detections \(BBoxes, confidence, class, and computed area\) are translated into concise, physician\-friendly language using Claude Opus 4\.7, based on number, relative position, confidence interpretation\. For example, the AROI model detections from Fig\.[2](https://arxiv.org/html/2607.09664#S3.F2)are summarized as*Nine PEDs \(one large, several medium, few small\) across mid\-retina, mostly high\-confidence centrally, lower\-confidence peripherally\.*The MedGemma provided text include both the raw detections and the summarized ones\.

### 4\.2Rebutals \- Similarity Case Retrieval \(MedSigLIP\)

MedSigLIP is a medical\-domain adaptation of the SigLIP vision–language model designed to align medical images and text within a shared embedding space\. It is not particularly trained on OCT images\. We use MedSigLIP to retrieve OCT reference cases that are visually similar to a query scan\. For each reference case, the MedSigLIP image embeddings, theF​i​n​d​i​n​g​sFindings, and theI​n​t​e​r​p​r​e​t​a​t​i​o​nInterpretationare all stored in OpenSearch\. The advantage of using OpenSearch is that three retrieval models are possible: vector\-only, text\-only, and hybrid\.

For example, when we use vector\-only retrieval for Case 3 \(Fig\.[1](https://arxiv.org/html/2607.09664#S2.F1)\) from our*RefC\_\\\_dataset*, the closest 2 images correspond to Case 28 and Case 13 \(Fig\.[3](https://arxiv.org/html/2607.09664#S3.F3)\)\. We can observe that the Findings are similar\. However, when analyzing the Interpretations, it becomes clear that small differences in the images correspond to significant differences in interpretation\. All three cases have AMD as a diagnosis, but the stage of the disease is probably*Late*for Case 3 and Case 13, while for Case 28 it is*Intermediate*\.

### 4\.3Qualifier \- Overall performance evaluations

As a qualifier component of our argument, we consider the following observations\.O1O\_\{1\}:*The AROI model does not make a distinction between SRF and SRHM \(subretinal hyperreflective material\), consequently they were jointly annotated as SRF\.*O2O\_\{2\}:*The AROI model has the following performance*…\(the values from Table[3](https://arxiv.org/html/2607.09664#S3)\)\.*Consequently, the model confuses the drusen/RPE changes with PED\. The predictions for IRF are of very good quality\.*O3O\_\{3\}:*The OCT5k model has very good precision on Softdrusen and SoftdrusenPED but a smaller recall\.*O4O\_\{4\}:*The given similar cases can be wrong, since the set of cases is small\. Use the given similar cases mainly to correct the list of extracted biomarkers; do not rely heavily on the interpretation of similar cases\.*

## 5Experiments and Evaluation

For the automated quantitative evaluation of our framework, five experiments were performed on200200cases extracted from the OCTDL datasetKulyabinet al\.\([2024](https://arxiv.org/html/2607.09664#bib.bib5)\)\. We used100100images from the AMD class, and100100from the DME class\. For the AMD diagnosis, OCTDL also includes annotation on staging: early, intermediate, and late\.

The first experiment \(E​x​p0Exp\_\{0\}\) measured the ability of MedGemma to solve T1 and T2 without additional information besides the OCT scan\. The next four experiments differ in terms of rebuttal and qualifier\. InE​x​p1Exp\_\{1\}, the rebuttal includes the top\-3 cases retrieved but only if the similarity is above0\.930\.93\. The qualifier is not included\. InE​x​p2Exp\_\{2\}, the rebuttal is similar toE​x​p1Exp\_\{1\}, but we include the first three observations for the qualifier\. InE​x​p3Exp\_\{3\}, the qualifier is the same as inE​x​p2Exp\_\{2\}, but no similar cases are provided \(therefore no rebuttal\)\. InE​x​p4Exp\_\{4\}, the rebuttal includes the top\-3 cases without the threshold\-based condition\. The qualifier includes all four observations\.

Table[3](https://arxiv.org/html/2607.09664#S5.T3)reports the precision and recall for T1 and T2\. It can be observed that MedGemma alone achieves the weakest overall performance on both T0 and T1, indicating that providing externally extracted information is necessary\. The contribution of similar cases retrieved suggests room for improvement\. For T1,E​x​p1Exp\_\{1\}andE​x​p3Exp\_\{3\}\- where similar cases are restricted or absent\- achieve the best balance between precision and recall\. For T2, however, rebuttals combined with a cautious qualifier appear to provide benefit particularly for late\-stage AMD\. Early AMD obtains the weakest performance due to the fact that AROI model does not differentiate between drusens and PED, and in AMD, PED is a significant finding that moves the diagnosis beyond early AMD\.

Table 3:Quantitative evaluations of the system on T1 and T2
## 6Discussion and Conclusions

The notions of argumentation, justification, and explanation are related, but not the same\. Argumentation is the broader process of defending a position in the presence of possible disagreement\. Justification is narrower: it provides support for accepting a position, while explanation increases understanding of a position, process, or event\.

The complementarity between argumentation and explanation is characterised by the fact that clinicians base their decisions on both evidence and understandingLetia and Groza \([2012](https://arxiv.org/html/2607.09664#bib.bib2)\)\. For example, image\-derived biomarkers provide evidence supporting a diagnostic claim, but this evidence is more meaningful when accompanied by an explanation of how these findings relate to the underlying retinal pathology\. Conversely, an explanation of the disease mechanism alone is insufficient without plausible visual evidence\. Zhan et al\.Zhanet al\.\([2026](https://arxiv.org/html/2607.09664#bib.bib1)\)propose a training pipeline in which Curriculum Goal\-Conditioned Learning \(CGCL\) trains LLMs to generate diagnostic arguments that explicitly follow this Toulmin structure\. However, it focuses primarily on text\-based clinical reasoning and LLM training\.

In contrast, we consider that not only the output reasoning, but also the information provided to the LLM for a given case should be organized according to the Toulmin structure\. By adopting such a framework, external models can be integrated as complementary sources of evidence, extending the LLM’s internal knowledge with image\-based findings and other specialized analyses\. In our case, the multimodal LLM \(MedGemma\) was not specifically trained on OCT scans, but it has knowledge of the anatomy of the retina, the normal structure of the retina, and pathological alterations\.

Our work is closely aligned with the growing view that AI in healthcare should expose reasoning processes, not only final answers\. YOLOE models extract evidence \(grounds\) in the form of detectable biomarkers that can be directly inspected\. Retrieved similar cases \(rebuttal\) may support or challenge these findings\. The qualifier is derived from the overall performance and confidence of the detection and retrieval modules\. As a result, MedGemma receives a structured and interpretable input that can also be reviewed by the human expert\.

The proposed framework is general and not limited to diagnosis based on OCT imaging\. The main limitations of our current work are the small number of reference cases, the limited evaluation, and the absence of subjective assessment by a human expert\.

Acknowledgment\.This work was supported in part by the project ”Romanian Hub for Artificial Intelligence\-HRIA” Smart Growth, Digitization and Financial Instruments Program, MySMIS no\. 334906\.

## References

- A\. Ardelean, E\. Ardelean, and A\. Marginean \(2025\)Can YOLO Detect Retinal Pathologies? A Step Towards Automated OCT Analysis\.Diagnostics15\(14\),pp\. 1823\.External Links:ISSN 2075\-4418,[Link](http://dx.doi.org/10.3390/diagnostics15141823),[Document](https://dx.doi.org/10.3390/diagnostics15141823)Cited by:[Table 1](https://arxiv.org/html/2607.09664#S3.T1),[§3](https://arxiv.org/html/2607.09664#S3.p1.1),[§3](https://arxiv.org/html/2607.09664#S3.p2.1)\.
- M\. Arikan, J\. Willoughby, and e\. Ongun \(2025\)OCT5k: A dataset of multi\-disease and multi\-graded annotations for retinal layers\.Scientific Data12\(1\)\.External Links:ISSN 2052\-4463,[Link](http://dx.doi.org/10.1038/s41597-024-04259-z),[Document](https://dx.doi.org/10.1038/s41597-024-04259-z)Cited by:[§3](https://arxiv.org/html/2607.09664#S3.p1.1)\.
- M\. Kulyabin, A\. Zhdanov, A\. Nikiforova, A\. Stepichev, A\. Kuznetsova, M\. Ronkin, V\. Borisov, A\. Bogachev, S\. Korotkich, P\. A\. Constable, and A\. Maier \(2024\)OCTDL: Optical Coherence Tomography Dataset for Image\-Based Deep Learning Methods\.Scientific Data11\(1\)\.External Links:ISSN 2052\-4463,[Link](http://dx.doi.org/10.1038/s41597-024-03182-7),[Document](https://dx.doi.org/10.1038/s41597-024-03182-7)Cited by:[§5](https://arxiv.org/html/2607.09664#S5.p1.3)\.
- I\. A\. Letia and A\. Groza \(2012\)Interleaved argumentation and explanation in dialog\.InThe 12th workshop on Computational Models of Natural Argument,Vol\.44\.Cited by:[§6](https://arxiv.org/html/2607.09664#S6.p2.1)\.
- M\. Melinščak, M\. Radmilovič, Z\. Vatavuk, and S\. Lončarić \(2021\)AROI: Annotated Retinal OCT Images Database\.In2021 44th International Convention on Information, Communication and Electronic Technology \(MIPRO\),Vol\.,pp\. 371–376\.External Links:[Document](https://dx.doi.org/10.23919/MIPRO52101.2021.9596934)Cited by:[§3](https://arxiv.org/html/2607.09664#S3.p1.1)\.
- C\. Zhan, X\. Tan, G\. Ma, Y\. Xiong, X\. Jiang, and X\. Qiu \(2026\)From Answers to Arguments: Toward Trustworthy Clinical Diagnostic Reasoning with Toulmin\-Guided Curriculum Goal\-Conditioned Learning\.External Links:2604\.11137,[Link](https://arxiv.org/abs/2604.11137)Cited by:[§6](https://arxiv.org/html/2607.09664#S6.p2.1)\.