Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
Summary
Artifact-Bench is a comprehensive benchmark that evaluates multimodal large language models on detecting and analyzing artifacts in AI-generated videos, revealing significant limitations and misalignment with human perception.
View Cached Full Text
Cached at: 05/20/26, 02:36 AM
Paper page - Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
Source: https://huggingface.co/papers/2605.18984 Published on May 18
#1 Paper of the day Authors:
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Artifact-Bench evaluates multimodal large language models’ capability to detect and analyze artifacts in AI-generated videos, revealing significant limitations in artifact perception and reasoning.
Recent video generative models have greatly improved the realism ofAI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. WhileMultimodal Large Language Models(MLLMs) show strongvisual understandingcapabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-graineddiagnostic reasoning, especially across diverseAI-generated videodomains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs onAI-generated videoartifact detectionand analysis. We first establish a three-level hierarchical taxonomy ofrealism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs.AI-generated videoclassification, pairwise realism comparison, andfine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators forAI-generated videorealism.
View arXiv pageView PDFGitHub1Add to collection
Get this paper in your agent:
hf papers read 2605\.18984
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.18984 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.18984 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.18984 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs
MuseBench is a comprehensive benchmark introduced to evaluate multimodal large language models on nuanced, intent-level understanding of audiovisual arts, revealing that even the best model achieves only 48.29% accuracy compared to 87.18% for human experts.
MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI
This paper introduces MLS-Bench, a benchmark designed to assess whether AI systems can invent generalizable and scalable machine learning methods rather than just performing engineering tuning.
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
PerceptionBench is a benchmark designed to evaluate atomic visual perception capabilities of Multimodal Large Language Models (MLLMs), using a bottom-up taxonomy of ten atomic perceptual capabilities. Results across 16 frontier MLLMs show no model reaches 60% accuracy, indicating visual perception remains largely unsolved.
Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents
MissionBench is a new benchmark for evaluating multimodal large language models (MLLMs) on long-horizon embodied tasks in aerial 3D environments, revealing that even the best models succeed on fewer than 35% of missions compared to 84.4% human performance.
MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge
Researchers introduce MM-JudgeBias, a benchmark that exposes systematic compositional biases in multimodal large language models when used as automatic judges, testing 26 SOTA MLLMs across 1,800 samples.