MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

Hugging Face Daily Papers Papers
benchmark multimodal large-language-models artistic-understanding creative-intent zero-shot evaluation

Summary

MuseBench is a comprehensive benchmark introduced to evaluate multimodal large language models on nuanced, intent-level understanding of audiovisual arts, revealing that even the best model achieves only 48.29% accuracy compared to 87.18% for human experts.

Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). True artistic understanding extends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress of multimodal large language models (MLLMs), this critical aspect of artistic understanding remains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning about creative intent. To address this gap, we introduce Musebench, a comprehensive benchmark designed to evaluate MLLMs on nuanced artistic understanding. It comprises 4,016 questions spanning cinematic arts, static visual arts, stage performing arts, and game arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, and expert validation. Comprehensive zero-shot evaluation of 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models' creative domain expertise.
Original Article
View Cached Full Text

Cached at: 07/08/26, 06:48 AM

Paper page - MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

Source: https://huggingface.co/papers/2606.30026

Abstract

A comprehensive benchmark called Musebench is introduced to evaluate multimodal large language models on nuanced artistic understanding, revealing a significant gap between current models and human expert performance in creative domain expertise.

Audiovisual artsencompass diverse creative disciplines, including cinema,visual arts, stage performance, and game design, where artistic meaning arises from deliberate combinations of visual, auditory, and narrative elements (e.g., fear amplified through claustrophobic framing, or grief conveyed through silence and lingering close-ups). Trueartistic understandingextends beyond recognizing what is depicted to reasoning about why it is expressed through particular creative choices. Despite the strong progress ofmultimodal large language models(MLLMs), this critical aspect ofartistic understandingremains underexplored, as existing benchmarks largely measure perceptual recognition while overlooking reasoning aboutcreative intent. To address this gap, we introduceMusebench, a comprehensive benchmark designed to evaluate MLLMs on nuancedartistic understanding. It comprises 4,016 questions spanningcinematic arts, staticvisual arts,stage performing arts, andgame arts, distilled from over 10K candidate video essays that pair professional commentary with visual demonstration. To capture the open-ended nature of artistic analysis at scale, the benchmark combines single-select and variable-option multi-select questions. All questions are generated and refined through a four-phase iterative pipeline combining shortcut filtering, adversarial distractors, andexpert validation. Comprehensivezero-shot evaluationof 28 state-of-the-art MLLMs reveals that even the best-performing model achieves only 48.29% accuracy, substantially below human expert performance of 87.18%, exposing a significant gap in current models’ creative domain expertise.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2606\.30026

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.30026 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.30026 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.30026 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles