Tag
论文提出 MM-FinEval，一个针对真实世界金融预测的多任务多模态基准，涵盖2019-2022年S&P 500公司的2,045次财报电话会议，包含文本转录、演示幻灯片和音频三种模态，并设有12项金融任务标签，用于评估19个多模态基线模型。
This paper introduces ReaLMem, the first benchmark for evaluating long-term multimodal memory in AI using authentic personal archives, and proposes ChronoProfiler for temporal weighting to improve personalization in AI companions.
Sci-MMR is a benchmark for evaluating multi-step evidence-grounded scientific reasoning in multimodal agents, revealing gaps where answer accuracy exceeds evidence recovery by over 20%.
This paper introduces CulturalMenuBench, a benchmark with 4,870 items across 10 languages and 18 regions, to test multimodal models' ability to apply culinary knowledge beyond mere recognition. Evaluations reveal a knowledge-application gap where models drop in performance when requiring cultural attribution despite high recognition scores.
PUMA is a Polish-language benchmark with 900 tasks designed to evaluate multimodal AI models on cultural understanding and practical skills across images, audio, and documents.
StateSight is a new benchmark for evaluating spatial-state reconstruction in vision-language models, showing that models like GPT-5.5 and Claude Sonnet 5 struggle with spatial reasoning tasks compared to human performance.
Holtercare-Bench is a multimodal benchmark introduced to evaluate long-term dynamic ECG analysis using the Holtercare-23K dataset, revealing performance gaps in current MLLMs and providing improvements through fine-tuning for clinical applications.
MCBench is a new benchmark for assessing the safety of omnimodal large language models across vision, audio, and text modalities. It includes 1196 scenarios and finds current models struggle with cross-modal safety reasoning.
BloomBench is a cognitively grounded bilingual (English-Arabic) multimodal benchmark for Vision-Language Models, systematically evaluating six cognitive levels based on Bloom's Taxonomy. Experiments reveal significant cognitive asymmetries and cross-lingual performance gaps in current models.
KMMMU is a native Korean benchmark for evaluating multimodal understanding with 3,466 questions across nine disciplines and visual modality categories, addressing the gap of English-centric benchmarks by testing performance on Korean-specific cultural and institutional contexts.