Can Multimodal Large Language Models Understand OCT?

Hugging Face Daily Papers Papers
multimodal large-language-models oct benchmark medical-image-analysis

Summary

This paper introduces OCT-Bench, a comprehensive benchmark for evaluating multimodal large language models on OCT image understanding, comprising 10,076 questions across 20 fine-grained tasks in perception, cognition, and reasoning. Evaluation of 20 models shows current MLLMs fall short of reliable OCT understanding.

Optical coherence tomography (OCT) imaging is essential for the diagnosis and treatment of retinal diseases. Although multimodal large language models (MLLMs) have demonstrated considerable potential in medical image analysis, existing benchmarks largely reduce OCT understanding to coarse-grained disease classification or isolated visual question answering, leaving the complete cognitive process from visual perception to clinical reasoning insufficiently evaluated. To address this limitation, we introduce OCT-Bench, a comprehensive benchmark dedicated to OCT image understanding. OCT-Bench comprises 10,076 high-quality multiple-choice questions constructed from 4,137 OCT images across seven public datasets. Following the real-world clinical interpretation workflow, we establish a hierarchical capability taxonomy consisting of 20 fine-grained tasks across three dimensions: Perception, Cognition, and Reasoning. These tasks cover a broad range of capabilities, including imaging attributes, retinal anatomy, lesion characteristics, spatial relationships, disease assessment, therapeutic decision-making, and prognostic management. We systematically evaluate 20 representative MLLMs, including proprietary models, open-source general-purpose models, and medical-domain models. Experimental results demonstrate that current models remain substantially short of reliable OCT understanding. Moreover, neither medical-domain adaptation nor increased model scale consistently improves performance across capability levels. OCT-Bench enables comprehensive and fine-grained evaluation of MLLMs, providing a foundation for identifying capability bottlenecks and advancing clinically grounded OCT understanding.
Original Article
View Cached Full Text

Cached at: 07/21/26, 02:37 PM

Paper page - Can Multimodal Large Language Models Understand OCT?

Source: https://huggingface.co/papers/2607.16609

Abstract

Opticalcoherencetomography(OCT)imagingisessentialforthediagnosisandtreatmentofretinaldiseases.Althoughmultimodallargelanguagemodels(MLLMs)havedemonstratedconsiderablepotentialinmedicalimageanalysis,existingbenchmarkslargelyreduceOCTunderstandingtocoarse-graineddiseaseclassificationorisolatedvisualquestionanswering,leavingthecompletecognitiveprocessfromvisualperceptiontoclinicalreasoninginsufficientlyevaluated.Toaddressthislimitation,weintroduceOCT-Bench,acomprehensivebenchmarkdedicatedtoOCTimageunderstanding.OCT-Benchcomprises10,076high-qualitymultiple-choicequestionsconstructedfrom4,137OCTimagesacrosssevenpublicdatasets.Followingthereal-worldclinicalinterpretationworkflow,weestablishahierarchicalcapabilitytaxonomyconsistingof20fine-grainedtasksacrossthreedimensions:Perception,Cognition,andReasoning.Thesetaskscoverabroadrangeofcapabilities,includingimagingattributes,retinalanatomy,lesioncharacteristics,spatialrelationships,diseaseassessment,therapeuticdecision-making,andprognosticmanagement.Wesystematicallyevaluate20representativeMLLMs,includingproprietarymodels,open-sourcegeneral-purposemodels,andmedical-domainmodels.ExperimentalresultsdemonstratethatcurrentmodelsremainsubstantiallyshortofreliableOCTunderstanding.Moreover,neithermedical-domainadaptationnorincreasedmodelscaleconsistentlyimprovesperformanceacrosscapabilitylevels.OCT-Benchenablescomprehensiveandfine-grainedevaluationofMLLMs,providingafoundationforidentifyingcapabilitybottlenecksandadvancingclinicallygroundedOCTunderstanding.

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2607\.16609

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.16609 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.16609 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.16609 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Beyond APIs: Probing the Limits of MLLMs in Physical Tool Use

arXiv cs.CL

This paper introduces PhysTool-Bench, a benchmark for evaluating multimodal large language models' ability to recognize and plan the use of physical tools in real-world scenes. The authors find that even the best model identifies only 58.7% of tools and completes just 21.0% of queries end-to-end, revealing a two-level deficit in perception and functional commonsense.

What We are Missing in Multimodal LLM Evaluation?

arXiv cs.AI

This paper reviews current multimodal LLM evaluation benchmarks and identifies key gaps such as temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention, arguing that existing isolated-task benchmarks fail to measure true cross-modal integration.