MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models
Summary
MMOOC is a large-scale benchmark with over 41K image-question pairs that evaluates how well multimodal large language models refuse out-of-context questions while answering shifted in-context questions, revealing current models struggle to balance these abilities.
View Cached Full Text
Cached at: 08/12/26, 08:23 AM
Paper page - MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models
Source: https://huggingface.co/papers/2607.27637
Abstract
MMOOC is a large-scale benchmark assessing whether multimodal language models can correctly refuse out-of-context questions while answering shifted in-context questions, revealing that current models struggle to balance these abilities.
Multimodal Large Language Models(MLLMs) have achieved strong performance on a wide range of vision-language tasks, but often fail under imperfect or shifted contexts. A reliable MLLM should refuse trulyout-of-context(OOC) questions with subject-level context shifts while still answering shifted in-context (Shifted IC) questions with non-subject context shifts. Existing benchmarks mainly target OOC or visually unanswerable questions, but overlook answerableShifted ICcases and cover limited OOC shifts. To fill this gap, we present MMOOC, a large-scale benchmark for evaluating refusal and robust answering abilities of MLLMs. MMOOC contains over 41K image-question pairs, including answerableShifted ICcases and unanswerable OOC cases, spanning three question formats, eight shift types and six visual scenarios, with data quality ensured through MLLM-based filtering and human verification. We evaluate model responses using Accuracy andRefusal Rate, and further introduce anLLM-as-a-Judgemetric to assess the correctness of model reasoning. Experiments on diverse MLLMs show that current models still struggle to balance answer-ability and refusal under shifted contexts. We further analyze key failure patterns and show that post-training can improve robustness. MMOOC will be made publicly available.
View arXiv pageView PDFProject pageGitHub11Add to collection
Get this paper in your agent:
hf papers read 2607\.27637
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.27637 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.27637 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.27637 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Can Multimodal Large Language Models Understand OCT?
This paper introduces OCT-Bench, a comprehensive benchmark for evaluating multimodal large language models on OCT image understanding, comprising 10,076 questions across 20 fine-grained tasks in perception, cognition, and reasoning. Evaluation of 20 models shows current MLLMs fall short of reliable OCT understanding.
MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models
MCBench is a new benchmark for assessing the safety of omnimodal large language models across vision, audio, and text modalities. It includes 1196 scenarios and finds current models struggle with cross-modal safety reasoning.
KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context
KMMMU is a native Korean benchmark for evaluating multimodal understanding with 3,466 questions across nine disciplines and visual modality categories, addressing the gap of English-centric benchmarks by testing performance on Korean-specific cultural and institutional contexts.
MedLoCoMo: A Long-Context Multi-Session Medical Dialogue Benchmark for Large Language Models
MedLoCoMo is a new benchmark for evaluating LLMs on long-context, multi-session medical dialogue reasoning, constructed from MIMIC-IV data. It tests single-admission, cross-admission, and adversarial unanswerable questions, revealing that cross-admission reasoning remains challenging even for models with long context windows.
Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games
This paper introduces RNG-Bench, a benchmark suite for evaluating multimodal foundation models' ability to reconstruct past observations and use them for decision-making in multi-step interactions, featuring two games (Matching Pairs and 3D Maze) with controlled difficulty parameters and a memory gap metric to distinguish forgetting from poor decision-making.