Can Multimodal Large Language Models Understand OCT?
Summary
This paper introduces OCT-Bench, a comprehensive benchmark for evaluating multimodal large language models on OCT image understanding, comprising 10,076 questions across 20 fine-grained tasks in perception, cognition, and reasoning. Evaluation of 20 models shows current MLLMs fall short of reliable OCT understanding.
View Cached Full Text
Cached at: 07/21/26, 02:37 PM
Paper page - Can Multimodal Large Language Models Understand OCT?
Source: https://huggingface.co/papers/2607.16609
Abstract
Opticalcoherencetomography(OCT)imagingisessentialforthediagnosisandtreatmentofretinaldiseases.Althoughmultimodallargelanguagemodels(MLLMs)havedemonstratedconsiderablepotentialinmedicalimageanalysis,existingbenchmarkslargelyreduceOCTunderstandingtocoarse-graineddiseaseclassificationorisolatedvisualquestionanswering,leavingthecompletecognitiveprocessfromvisualperceptiontoclinicalreasoninginsufficientlyevaluated.Toaddressthislimitation,weintroduceOCT-Bench,acomprehensivebenchmarkdedicatedtoOCTimageunderstanding.OCT-Benchcomprises10,076high-qualitymultiple-choicequestionsconstructedfrom4,137OCTimagesacrosssevenpublicdatasets.Followingthereal-worldclinicalinterpretationworkflow,weestablishahierarchicalcapabilitytaxonomyconsistingof20fine-grainedtasksacrossthreedimensions:Perception,Cognition,andReasoning.Thesetaskscoverabroadrangeofcapabilities,includingimagingattributes,retinalanatomy,lesioncharacteristics,spatialrelationships,diseaseassessment,therapeuticdecision-making,andprognosticmanagement.Wesystematicallyevaluate20representativeMLLMs,includingproprietarymodels,open-sourcegeneral-purposemodels,andmedical-domainmodels.ExperimentalresultsdemonstratethatcurrentmodelsremainsubstantiallyshortofreliableOCTunderstanding.Moreover,neithermedical-domainadaptationnorincreasedmodelscaleconsistentlyimprovesperformanceacrosscapabilitylevels.OCT-Benchenablescomprehensiveandfine-grainedevaluationofMLLMs,providingafoundationforidentifyingcapabilitybottlenecksandadvancingclinicallygroundedOCTunderstanding.
View arXiv pageView PDFGitHub3Add to collection
Get this paper in your agent:
hf papers read 2607\.16609
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.16609 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.16609 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.16609 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models
MMOOC is a large-scale benchmark with over 41K image-question pairs that evaluates how well multimodal large language models refuse out-of-context questions while answering shifted in-context questions, revealing current models struggle to balance these abilities.
Beyond Text-Dominance: Understanding Modality Preference of Omni-modal Large Language Models
This paper investigates modality preference in omni-modal large language models (OLLMs), revealing a paradigm shift from text-dominance to visual preference. The authors introduce a conflict-based benchmark and layer-wise probing to diagnose cross-modal hallucinations using internal model signals.
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
PerceptionBench is a benchmark designed to evaluate atomic visual perception capabilities of Multimodal Large Language Models (MLLMs), using a bottom-up taxonomy of ten atomic perceptual capabilities. Results across 16 frontier MLLMs show no model reaches 60% accuracy, indicating visual perception remains largely unsolved.
Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
This paper investigates why multimodal LLMs fail on vision-centric tasks when visual evidence conflicts with language priors, introducing the WhatIfVis benchmark and showing that models often encode visual evidence but cannot reliably control their reliance on it.
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
MCBench is a new benchmark for assessing the safety of omnimodal large language models across vision, audio, and text modalities. It includes 1196 scenarios and finds current models struggle with cross-modal safety reasoning.