OmniHallu: Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models

# Unified Hallucination Detection for Cross-Modal Comprehension and Generation in Multimodal Large Language Models
Source: [https://arxiv.org/html/2609.11244](https://arxiv.org/html/2609.11244)
Peihang LiAffiliation:The University of Hong KongShanqing XuAffiliation:Huazhong University of Science and TechnologyMengchen QianAffiliation:Huazhong University of Science and TechnologyLu ZhangAffiliation:Shanghai Academy of Educational SciencesMeng Luo111Corresponding authors: Meng Luo \(mluo@u\.nus\.edu\)\.Affiliation:National University of Singapore

###### Abstract

While Multimodal Large Language Models \(MLLMs\) have achieved remarkable progress across diverse tasks, they suffer from hallucinations where generated outputs contradict or misrepresent input semantics\. Existing research typically addresses hallucination detection within a single modality or task type, limiting generalizability\. We introduceOmniHallu, a unified hallucination detection framework spanning both comprehension and generation tasks across image, video, and audio modalities\. We contributeOmniHallu\-Bench, a 10,000\-sample benchmark with claim\-level human annotations covering six cross\-modal tasks: image\-to\-text \(I2T\), video\-to\-text \(V2T\), audio\-to\-text \(A2T\), text\-to\-image \(T2I\), text\-to\-video \(T2V\), and text\-to\-audio \(T2A\)\. Our multi\-agent architecture decomposes model outputs into atomic claims, verifies them through modality\-specific experts, and aggregates evidence via structured reasoning\. We further propose a preference\-optimized trainable verifier that approximates the multi\-agent decision boundary, reducing expert calls by 66% with minimal performance loss\. Extensive experiments reveal a consistent modality\-dependent performance gradient and provide fine\-grained insights into cross\-modal hallucination patterns\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.11244v1/example.png)Figure 1:MLLMs produce hallucinations in both comprehension and generation across modalities, spanning object, attribute, relation, and event hallucination types\.Multimodal Large Language Models \(MLLMs\)\([Li et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib28);[Chen et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib9);[Li et al\., 2025b](https://arxiv.org/html/2609.11244#bib.bib31);[Li et al\., 2026](https://arxiv.org/html/2609.11244#bib.bib30);[Luo et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib37)\)have achieved remarkable progress across vision, audio, and language tasks\. However, these models frequently hallucinate: generating outputs that contradict or misrepresent the input\([Bai et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib4);[Huang et al\., 2025b](https://arxiv.org/html/2609.11244#bib.bib22);[Lin et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib33);[Luo et al\., 2026a](https://arxiv.org/html/2609.11244#bib.bib38);[Luo et al\., 2026b](https://arxiv.org/html/2609.11244#bib.bib39)\)\. Hallucinations pose a fundamental barrier to deploying MLLMs in safety\-critical applications\.

Existing hallucination detection methods predominantly target a single modality or task type\. Image\-focused benchmarks\([Li et al\., 2023b](https://arxiv.org/html/2609.11244#bib.bib32);[Wang et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib57)\)do not cover video or audio; video\-focused\([Liu and Wan, 2023](https://arxiv.org/html/2609.11244#bib.bib35);[Wang et al\., 2024b](https://arxiv.org/html/2609.11244#bib.bib59)\)and audio\-focused\([Nishimura et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib43)\)evaluations similarly operate in isolation\. Moreover, most work addresses only comprehension while neglecting generation tasks, despite both sharing common hallucination patterns rooted in insufficient perception and reasoning\.

We study a unified claim\-level detection protocol that enables side\-by\-side comparison across modalities and task directions, while also revealing which components transfer and where modality\-specific verification remains necessary\.

To this end, we introduceOmniHallu, a unified hallucination detection framework, andOmniHallu\-Bench, a benchmark of 10,000 human\-verified samples\. Our method adapts the established decompose–verify–aggregate paradigm\([Chen et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib8)\)to six bidirectional tasks: video verification emphasizes temporal and causal evidence, whereas audio verification relies on acoustic cues and a less mature tool ecosystem\. Our contributions are:

- •OmniHallu\-Bench: A 10,000\-sample benchmark with claim\-level human annotations spanning six cross\-modal tasks \(I2T, V2T, A2T, T2I, T2V, T2A\) across four modalities\.
- •Cross\-modal systematization: A modality\-aware implementation of claim decomposition, specialized verification, and evidence aggregation, together with controlled analyses of component contributions, performance variations, and failure modes\.
- •Preference\-optimized verifier: A compact trainable verifier aligned via GRPO that reduces expensive expert calls by 66% with minimal performance loss\.

Table 1:Comparison of hallucination benchmarks\. “Function” indicates whether the benchmark supports fact\-Checking or hallucinationDetection\. “Granularity” denotes evaluation at theResponse,Segment, orClaim level\. “Rationale” indicates whether the benchmark provides explanatory justifications\.
## 2Related Work

#### Hallucination in MLLMs\.

Hallucinations manifest across all MLLM modalities\. In vision\-language models, generated descriptions may mention objects absent from the image\([Li et al\., 2023b](https://arxiv.org/html/2609.11244#bib.bib32)\)\. Video\-language models exhibit intrinsic and extrinsic hallucinations\([Wang et al\., 2024b](https://arxiv.org/html/2609.11244#bib.bib59);[Huang et al\., 2026](https://arxiv.org/html/2609.11244#bib.bib20)\), while audio\-video language models may ignore acoustic content and describe audio primarily from visual evidence\([Nishimura et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib43)\)\. In generation tasks, text\-to\-image models often fail on compositional prompt alignment\([Bakr et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib5);[Huang et al\., 2025a](https://arxiv.org/html/2609.11244#bib.bib21)\), and text\-to\-video models lack temporal coherence\([Chu et al\., 2024b](https://arxiv.org/html/2609.11244#bib.bib13);[Rawte et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib47)\)\. Despite this breadth of modality\-specific work, most prior research has studied each modality and task type independently, limiting insight into cross\-modal regularities\.

#### Hallucination Detection and Evaluation\.

Detection methods have evolved from simple self\-consistency checks\([Manakul et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib40);[Miao et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib41)\)to structured multi\-step pipelines\. UNIHD\([Chen et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib8)\)introduced claim decomposition and verification for image–text tasks, while FactVC\([Liu and Wan, 2023](https://arxiv.org/html/2609.11244#bib.bib35)\)proposed factuality metrics for video captioning\. CrossCheckGPT\([Sun et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib52)\)ranks systems through reference\-free cross\-system consistency across text, image, and audio\-visual domains, and video\-SALMONN 2\([Tang et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib54)\)uses preference optimization to mitigate errors in audio\-visual captioning\. These methods target system ranking, claim\-level evaluation, or hallucination mitigation, while benchmarks such as POPE\([Li et al\., 2023b](https://arxiv.org/html/2609.11244#bib.bib32)\), AMBER\([Wang et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib57)\), and MHaluBench\([Chen et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib8)\)remain restricted to a single modality or modality pair\. Our work studies claim\-level detection across six bidirectional tasks; Table[1](https://arxiv.org/html/2609.11244#S1.T1)summarizes the corresponding benchmark coverage\.

#### Tool\-Augmented and Multi\-Agent LLM Systems\.

Toolformer\([Schick et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib49)\)trains language models to decide which external tools to call and how to incorporate their outputs\. Grounding DINO\([Ren et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib48)\)enables zero\-shot object verification, and DoraemonGPT\([Yang et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib63)\)reformulates video understanding into tool invocations\. Our framework extends this paradigm to hallucination detection across modalities with unified reasoning\-based aggregation\.

## 3Task Formulation and Hallucination Taxonomy

#### Unified Formulation\.

Let𝒯\\mathcal\{T\},ℐ\\mathcal\{I\},𝒱\\mathcal\{V\},𝒜\\mathcal\{A\}denote textual, image, video, and audio data\. An MLLM maps input𝐱∈\{𝒯,ℐ,𝒱,𝒜\}\\mathbf\{x\}\\in\\\{\\mathcal\{T\},\\mathcal\{I\},\\mathcal\{V\},\\mathcal\{A\}\\\}to outputy^∈\{𝒯,ℐ,𝒱,𝒜\}\\hat\{y\}\\in\\\{\\mathcal\{T\},\\mathcal\{I\},\\mathcal\{V\},\\mathcal\{A\}\\\}\. We address*comprehension*tasks \(𝐱∈\{ℐ,𝒱,𝒜\}→y^∈𝒯\\mathbf\{x\}\\in\\\{\\mathcal\{I\},\\mathcal\{V\},\\mathcal\{A\}\\\}\\to\\hat\{y\}\\in\\mathcal\{T\}\), and*generation*tasks \(𝐱∈𝒯→y^∈\{ℐ,𝒱,𝒜\}\\mathbf\{x\}\\in\\mathcal\{T\}\\to\\hat\{y\}\\in\\\{\\mathcal\{I\},\\mathcal\{V\},\\mathcal\{A\}\\\}\)\. Comprehension hallucinations reside in the generated text, while generation hallucinations manifest as semantic discrepancies between the produced media and the prompt\.

#### Definition\.

An outputy^\\hat\{y\}is hallucinated if it contains any semantic claim that is unsupported by or contradicts the input𝐱\\mathbf\{x\}\. Formally, let𝒢\\mathcal\{G\}denote the set of ground\-truth semantic elements derivable from𝐱\\mathbf\{x\}:

Hallucinate⁡\(y^∣𝐱\)=\{1if​∃ϕ⁡\(y^\)∉𝒢0otherwise,\\Hallucinate\(\\hat\{y\}\\mid\\mathbf\{x\}\)=\\begin\{cases\}1&\\text\{if \}\\exists\\;\\phi\(\\hat\{y\}\)\\notin\\mathcal\{G\}\\\\ 0&\\text\{otherwise,\}\\end\{cases\}whereϕ⁡\(y^\)\\phi\(\\hat\{y\}\)denotes any semantic claim extractable fromy^\\hat\{y\}\.

#### Hallucination Taxonomy\.

We define four types applicable across all modalities: \(1\)Object: non\-existent entities introduced; \(2\)Attribute: misrepresented properties such as color, size, or timbre; \(3\)Relation: incorrectly stated spatial, temporal, or causal relationships; \(4\)Event: misrepresented event\-level details, including temporal ordering errors or fabricated actions\. Each type manifests across all six tasks, enabling direct cross\-modal comparison \(§[6](https://arxiv.org/html/2609.11244#S6)\)\.

## 4OmniHallu\-Bench

Figure 2:Statistics of OmniHallu\-Bench: distribution across modalities, tasks, and hallucination types\.#### Design Principles\.

OmniHallu\-Bench comprises 10,000 samples with stratified coverage across modalities and tasks\. Comprehension tasks account for 60% and generation tasks for 40%\. Image, video, and audio samples follow a 5:3:2 ratio\. Hallucination types are distributed as: object \(35%\), attribute \(25%\), event \(25%\), and relation \(15%\)\.

#### Comprehension Tasks\.

For I2T, we draw from COCO Captions\([Chen et al\., 2015](https://arxiv.org/html/2609.11244#bib.bib10)\), Nocaps\([Agrawal et al\., 2019](https://arxiv.org/html/2609.11244#bib.bib1)\), and Flickr30k\([Plummer et al\., 2015](https://arxiv.org/html/2609.11244#bib.bib46)\), with model outputs generated by InternVL2\.5\-78B\([Chen et al\., 2024b](https://arxiv.org/html/2609.11244#bib.bib11)\), Qwen2\.5\-VL\-72B\([Bai et al\., 2025b](https://arxiv.org/html/2609.11244#bib.bib3)\), GPT\-4\.1\([OpenAI, 2025a](https://arxiv.org/html/2609.11244#bib.bib44)\), and Gemini\-2\.5\-Pro\([Gemini Team, 2025](https://arxiv.org/html/2609.11244#bib.bib18)\)\. For V2T, we sample from MSVD\([Chen et al\., 2022](https://arxiv.org/html/2609.11244#bib.bib7)\), MSRVTT\([Xu et al\., 2016](https://arxiv.org/html/2609.11244#bib.bib61)\), and VATEX\([Wang et al\., 2019](https://arxiv.org/html/2609.11244#bib.bib58)\), using InternVL2\.5\-78B, Qwen2\.5\-VL\-72B, VideoLLaMA3\([Zhang et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib64)\), and LLaVA\-OneVision\([Li et al\., 2025a](https://arxiv.org/html/2609.11244#bib.bib27)\)\. For A2T, we use AudioCaps\([Kim et al\., 2019](https://arxiv.org/html/2609.11244#bib.bib25)\), ClothoV2\([Drossos et al\., 2020](https://arxiv.org/html/2609.11244#bib.bib16)\), and AudioSetCaps\([Bai et al\., 2025a](https://arxiv.org/html/2609.11244#bib.bib2)\), with outputs from Qwen2\-Audio\-7B\-Instruct\([Chu et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib12)\), GAMA\([Ghosh et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib19)\), Pengi\([Deshmukh et al\., 2023a](https://arxiv.org/html/2609.11244#bib.bib14)\), and SALMONN\([Tang et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib55)\)\.

#### Generation Tasks\.

For T2I, prompts from T2I\-CompBench\+\+\([Huang et al\., 2025a](https://arxiv.org/html/2609.11244#bib.bib21)\)and HRS\-Bench\([Bakr et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib5)\)are used to generate images via DALL\-E 3, Stable Diffusion 3\.5 Large, and Midjourney v6\([Betker et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib6);[Stability AI, 2024](https://arxiv.org/html/2609.11244#bib.bib51);[Midjourney, 2023](https://arxiv.org/html/2609.11244#bib.bib42)\)\. For T2V, prompts from T2V\-CompBench\([Sun et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib53)\)and FETV\([Liu et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib36)\)drive generation via Open\-Sora 1\.2\([Zheng et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib65)\)and CogVideoX\-5B\([Yang et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib62)\)\. For T2A, prompts from WavText5K\([Deshmukh et al\., 2023b](https://arxiv.org/html/2609.11244#bib.bib15)\), FSD50K\([Fonseca et al\., 2022](https://arxiv.org/html/2609.11244#bib.bib17)\), and SoundDescs\([Koepke et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib26)\)generate audio via Make\-an\-Audio\([Huang et al\., 2023](https://arxiv.org/html/2609.11244#bib.bib23)\), AudioGPT\([Huang et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib24)\), and AudioLCM\([Liu et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib34)\)\. Multiple generators are used for each task to reduce dependence on model\-specific artifacts\.

#### Annotation and Quality Control\.

Samples undergo a structured atomic claim decomposition using Chain\-of\-Thought prompting\([Wei et al\., 2022](https://arxiv.org/html/2609.11244#bib.bib60)\)with self\-reflection verification\. Three trained annotators independently review each sample; a sample is retained only upon full consensus\. Annotators first validate the decomposed claims for semantic fidelity, then classify each as hallucinatory or non\-hallucinatory\. Approximately 24\.3% of initial samples were removed due to disagreement\. Before consensus filtering, Fleiss’κ\\kappais 0\.89 \(image\), 0\.86 \(video\), and 0\.83 \(audio\)\. The final dataset contains 4,000 human\-curated and 6,000 model\-generated \(human\-audited\) samples\.

#### Dataset Statistics\.

The average number of atomic claims per sample is 4\.2 \(I2T\), 3\.8 \(T2I\), 5\.6 \(V2T\), 4\.1 \(T2V\), 3\.5 \(A2T\), and 3\.1 \(T2A\)\. The overall hallucination rate is 42\.8%, compared with 48\.1% for generation tasks and 39\.2% for comprehension tasks\.

## 5Multi\-Agent Hallucination Detection

Our framework consists of three stages: atomic claim decomposition, modality\-aware expert verification, and reasoning\-based aggregation \(Figure[3](https://arxiv.org/html/2609.11244#S5.F3)\)\.

![Refer to caption](https://arxiv.org/html/2609.11244v1/figure/framework.png)Figure 3:Overview of the OmniHallu multi\-agent framework\. Model outputs are decomposed into atomic claims, verified by modality\-specific experts, and aggregated through reasoning\-based decision making\.#### Atomic Claim Decomposition \(ACD\)\.

We decompose the target text \(model\-generated captions for comprehension, input prompts for generation\) into atomic claims using GPT\-4\.1\. Each sample\(y,\{c1,…,cny\}\)\(y,\\\{c\_\{1\},\\ldots,c\_\{n\_\{y\}\}\\\}\)consists of textyyand corresponding claims, where eachcic\_\{i\}is a semantically discrete, grammatically self\-contained, verifiable statement\. For comprehension tasks,yyis the model\-generated caption; for generation tasks,yyis the input prompt whose claims must be verified against the generated media\. We compare GPT\-4\.1 claims against human reference claims on a 300\-sample subset \(Table[2](https://arxiv.org/html/2609.11244#S5.T2)\); mean coverage is 0\.92 and mean redundancy is 1\.15\.

Table 2:Atomic claim decomposition quality against human references, measured by coverage, redundancy, and agreement\.
#### Modality\-Aware Expert Verification\.

Different modalities and hallucination types require specialized verification:

Image tasks \(I2T, T2I\)\.Object hallucinations are verified using Grounding DINO 1\.5 Pro\([Ren et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib48)\)for open\-set detection\. Attribute, relation, and event hallucinations are assessed by an ensemble of MLLMs \(Qwen2\.5\-VL\-72B, InternVL2\.5\-78B, GPT\-4\.1\), where each model independently evaluates the claim against visual evidence\. For T2I tasks, the text prompt serves as ground truth and experts verify whether the generated image faithfully reflects each prompted claim\.

Video tasks \(V2T, T2V\)\.Following DoraemonGPT\([Yang et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib63)\), each atomic claim is reformulated into a targeted QA query via GPT\-4\.1, enabling temporal decomposition and frame\-level evidence extraction\. The default three\-expert configuration uses Qwen2\.5\-VL\-72B, InternVL2\.5\-78B, and VideoLLaMA3\([Zhang et al\., 2025](https://arxiv.org/html/2609.11244#bib.bib64)\), which can attend to specific temporal segments\.

Audio tasks \(A2T, T2A\)\.The default three\-expert configuration uses Qwen2\-Audio\-7B\-Instruct, GAMA\([Ghosh et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib19)\), and SALMONN\([Tang et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib55)\)\.

Within each modality, expert judgments are summarized by equal\-weight majority voting, chosen for transparency and because no reliable cross\-model confidence calibration metric is available\. The vote summary and the individual evidence traces are then passed to the final reasoning\-based aggregator described below \(see Appendix[A](https://arxiv.org/html/2609.11244#A1)\)\.

#### Reasoning\-Based Aggregation\.

Expert verification results and atomic claims are consolidated by GPT\-5\.2 in the main experiments; the controlled comparison in Table[6](https://arxiv.org/html/2609.11244#S6.T6)replaces it with GPT\-4\.1\([OpenAI, 2025a](https://arxiv.org/html/2609.11244#bib.bib44);[OpenAI, 2025b](https://arxiv.org/html/2609.11244#bib.bib45)\)\. The aggregator receives each expert’s judgment, supporting evidence, and confidence signal, and outputs a final label with an explanatory rationale\.

#### Trainable Verifier via Preference Optimization\.

To reduce dependence on expensive expert ensembles, we train a compact claim\-level verifierπψ\\pi\_\{\\psi\}\(initialized from Qwen2\.5\-VL\-7B\) that outputs a judgment conditioned on the task input and an atomic claim:

πψ​\(ℓ∣𝐱,ci\),ℓ∈\{SUP,UNSUP,ABS\}\.\\pi\_\{\\psi\}\(\\ell\\mid\\mathbf\{x\},c\_\{i\}\),\\;\\;\\ell\\in\\\{\\texttt\{SUP\},\\texttt\{UNSUP\},\\texttt\{ABS\}\\\}\.\(1\)UNSUPmaps to hallucinated,SUPto non\-hallucinated, andABS\(abstain\) triggers full expert verification\.

We define a rewardR=λlab​Rlab\+λev​Rev\+λcal​RcalR=\\lambda\_\{\\text\{lab\}\}R\_\{\\text\{lab\}\}\+\\lambda\_\{\\text\{ev\}\}R\_\{\\text\{ev\}\}\+\\lambda\_\{\\text\{cal\}\}R\_\{\\text\{cal\}\}, using\(λlab,λev,λcal\)=\(1\.0,0\.7,0\.3\)\(\\lambda\_\{\\text\{lab\}\},\\lambda\_\{\\text\{ev\}\},\\lambda\_\{\\text\{cal\}\}\)=\(1\.0,0\.7,0\.3\)\. Here,RlabR\_\{\\text\{lab\}\}rewards ground\-truth match,RevR\_\{\\text\{ev\}\}rewards consistency with multi\-agent consensus, andRcalR\_\{\\text\{cal\}\}penalizes overconfidence when expert signals conflict\. We train via Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib50)\):

ℒGRPO=−𝔼⁡\[1K​∑k=1KA^k​log⁡πψ​\(yk∣𝐱,ci\)\],\\mathcal\{L\}\_\{\\text\{GRPO\}\}=\-\\mathbb\{E\}\\Bigg\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\hat\{A\}\_\{k\}\\,\\log\\pi\_\{\\psi\}\(y\_\{k\}\\mid\\mathbf\{x\},c\_\{i\}\)\\Bigg\],\(2\)whereA^k=\(Rk−μ⁡\(R\)\)/\(σ⁡\(R\)\+ϵ\)\\hat\{A\}\_\{k\}=\(R\_\{k\}\-\\mu\(R\)\)/\(\\sigma\(R\)\+\\epsilon\)is the normalized advantage overKKsampled judgments\. GRPO uses group\-level relative advantages instead of explicit preference pairs\. The trained verifier integrates into the pipeline as a low\-cost filter: confident predictions \(probability\>0\.85\>0\.85\) skip expert calls, while uncertain samples \(ABS\) fall through to the full ensemble\.

## 6Experiments

#### Setup\.

We follow the evaluation protocol of UNIHD\([Chen et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib8)\), computing precision \(P\), recall \(R\), and F1 for both hallucinatory and non\-hallucinatory categories at claim level, along with accuracy and macro\-averaged F1 \(Mac\.F1\)\. Mac\.F1 is our primary metric as it balances detection of both hallucinatory and non\-hallucinatory claims\. Baselines include: \(1\)Self\-Check\([Miao et al\., 2024](https://arxiv.org/html/2609.11244#bib.bib41)\), which uses a single MLLM’s chain\-of\-thought self\-verification without external tools; and \(2\)UNIHD\([Chen et al\., 2024a](https://arxiv.org/html/2609.11244#bib.bib8)\), a multi\-step pipeline applicable only to image tasks\. For video and audio tasks where UNIHD is inapplicable, we compare against Self\-Check with the strongest available MLLMs per modality\. Our full framework uses GPT\-5\.2\([OpenAI, 2025b](https://arxiv.org/html/2609.11244#bib.bib45)\)as the reasoning model; we additionally report GPT\-4\.1 results \(Table[6](https://arxiv.org/html/2609.11244#S6.T6)\) to enable direct comparison at equal model capacity\. For the verifier, we use a disjoint 6,000/1,000/3,000\-sample train/dev/test split with no source\-media overlap\. The trainable verifier is initialized from Qwen2\.5\-VL\-7B and trained for 3 epochs with learning rate1×10−51\\times 10^\{\-5\}and GRPO group sizeK=8K=8\.

Table 3:Hallucination detection results across six tasks\. P/R/F1 are reported for hallucinatory \(H\) and non\-hallucinatory \(NH\) categories; Mac\.F1 is the macro\-average\. All values are claim\-level percentages\. Our framework uses GPT\-5\.2 as the reasoning model; baselines use their respective models as noted\. Best results inbold\.
### 6\.1Main Results

Table[3](https://arxiv.org/html/2609.11244#S6.T3)presents claim\-level results across all six tasks\. Our multi\-agent framework consistently outperforms all baselines, with Mac\.F1 improvements ranging from \+3\.4 to \+8\.1 points over the strongest baseline per task\. The improvements are statistically significant across all tasks \(p<0\.01p<0\.01, bootstrap test with 10,000 resamples\)\. We discuss the results along three dimensions: modality, task type, and hallucination category\.

Image tasksexhibit the highest performance overall \(Mac\.F1: 81\.8–83\.0\)\. With the UNIHD pipeline available, baselines are relatively strong, yet our framework still achieves \+4\.1 \(I2T\) and \+3\.4 \(T2I\) points over GPT\-4\.1 UNIHD\.

Video tasksshow moderate performance \(Mac\.F1: 75\.6–77\.1\), reflecting the added complexity of temporal reasoning\. Our framework achieves \+7\.0 \(V2T\) and \+7\.2 \(T2V\) points over the best Self\-Check baseline, demonstrating the value of multi\-agent verification for temporal content\.

Audio tasksare the most challenging \(Mac\.F1: 73\.4–74\.1\), consistent with limited expert tool availability for auditory understanding\. Our framework still provides \+7\.8 \(A2T\) and \+8\.1 \(T2A\) points improvement, the largest absolute gains across all tasks, demonstrating that the multi\-agent architecture is particularly valuable when individual models are weaker\.

Within each modality, comprehension tasks consistently outperform their generation counterparts \(e\.g\., I2T 82\.95 vs\. T2I 81\.81; V2T 77\.14 vs\. T2V 75\.63\), as comprehension verification can directly ground claims against the source media, whereas generation verification must assess whether the produced media faithfully reflects the textual prompt\. This modality\-dependent performance gradient \(image\>\>video\>\>audio\) results from two interacting factors: the intrinsic complexity of each modality and the varying maturity of available expert models, where vision experts \(70B\+ parameters\) far exceed current audio experts\.

### 6\.2Ablation Studies

Table 4:Module ablation results in Mac\.F1\. ACD and MV denote Atomic Claim Decomposition and Multi\-expert Voting, respectively\.#### Module Ablation\.

Table[4](https://arxiv.org/html/2609.11244#S6.T4)shows that removing ACD causes the largest degradation \(\-6\.6 to \-7\.9 points\), confirming the importance of structured claim decomposition for precise hallucination localization\. Without ACD, the reasoning model must assess hallucination at the response level, losing fine\-grained pinpointing\. Removing multi\-expert voting \(MV\) reduces performance by 4\.5–5\.3 points uniformly across modalities, validating the ensemble verification strategy\. The consistent gap between ACD and MV degradations across all six tasks indicates that precise claim\-level granularity contributes more to detection accuracy than ensemble diversity, underscoring that decomposition quality is the primary bottleneck in the pipeline\.

Table 5:Reward ablation for the GRPO\-trained verifier\.
#### Reward Ablation\.

Table[5](https://arxiv.org/html/2609.11244#S6.T5)shows that removingRevR\_\{\\text\{ev\}\}\(expert evidence consistency\) causes the largest Mac\.F1 drop \(\-7\.4\), underscoring the importance of aligning the verifier with multi\-agent consensus\. RemovingRcalR\_\{\\text\{cal\}\}\(calibration penalty\) yields the largest ECE increase \(0\.042→\\to0\.154\), indicating that the verifier becomes overconfident without calibration pressure\. All three components serve complementary roles:RlabR\_\{\\text\{lab\}\}removal yields a moderate decline \(\-4\.7 Mac\.F1\) with ECE rising to 0\.085, confirming that ground\-truth supervision contributes to both accuracy and calibration, though its calibration effect is secondary to the dedicatedRcalR\_\{\\text\{cal\}\}penalty\.

### 6\.3Reasoning Model and Cost Analysis

Table 6:Impact of reasoning model choice on Mac\.F1\.#### Reasoning Model\.

Table[6](https://arxiv.org/html/2609.11244#S6.T6)shows that GPT\-5\.2 consistently outperforms GPT\-4\.1 as the reasoning backbone by 4\.5–6\.6 points, with the gap more pronounced on video and audio tasks where evidence synthesis across temporal and acoustic dimensions is harder\. We acknowledge that the main results \(Table[3](https://arxiv.org/html/2609.11244#S6.T3)\) use GPT\-5\.2, giving our framework a stronger backbone than baselines\. Even with GPT\-4\.1, however, our framework outperforms baselines on video and audio tasks by 1\.3–1\.9 points, demonstrating that the multi\-agent architecture provides genuine value beyond model strength alone\. Notably, the gap widens from 4\.5–5\.0 points on image tasks to 6\.2–6\.6 points on audio tasks, suggesting that stronger reasoning models compensate more effectively when expert tools provide weaker or more ambiguous perceptual evidence\.

Table 7:Computational cost analysis per sample, averaged across tasks\.
#### Computational Cost\.

Table[7](https://arxiv.org/html/2609.11244#S6.T7)reports computational overhead\. The full pipeline requires 5–8 API calls at∼\\sim$0\.12 per sample\. The verifier filter reduces API calls to 1\.7–2\.8 \(66% reduction\) and cost to∼\\sim$0\.05 per sample\. For the full OmniHallu\-Bench, the estimated total cost is∼\\sim$1,200 for the full pipeline or∼\\sim$500 with the verifier filter, offering a practical trade\-off for large\-scale deployment\. The latency reduction from∼\\sim15s to∼\\sim6s per sample further enables near\-real\-time hallucination feedback in interactive applications\.

### 6\.4Trainable Verifier Integration

Table 8:Trainable verifier integration modes\. “Expert Calls” indicates the fraction of samples requiring full ensemble invocation\.Table[8](https://arxiv.org/html/2609.11244#S6.T8)compares integration strategies\. The GRPO\-trained verifier alone approaches the full pipeline without any expert calls, providing a practical option when latency or cost precludes expert invocation\. The filter mode achieves 80\.52 Mac\.F1 while requiring expert calls for only 34\.2% of samples: the verifier handles easy cases \(clear hallucinations or clearly supported claims\) and defers ambiguous ones to the full ensemble\. The aggregator mode usesπψ\\pi\_\{\\psi\}to re\-weight expert votes rather than replace them, achieving the highest performance \(83\.15\) by improving robustness under expert disagreement\. GRPO training consistently outperforms SFT \(\+7\.5\) and DPO \(\+2\.1\), validating the advantage of group\-relative optimization for calibrated hallucination judgment\. The three integration modes form a practical cost\-performance spectrum, enabling practitioners to select an operating point based on deployment constraints: the standalone verifier for latency\-sensitive applications, the filter mode for balanced cost\-accuracy trade\-offs, and the aggregator for high\-stakes scenarios requiring maximum reliability\.

### 6\.5Fine\-Grained Analysis

Figure 4:Detection performance by hallucination type\. Object hallucinations are easiest to detect; relation hallucinations are the most challenging across all modalities\.#### Hallucination Type Analysis\.

Figure[4](https://arxiv.org/html/2609.11244#S6.F4)reveals a consistent difficulty hierarchy across modalities on OmniHallu\-Bench: object\>\>attribute\>\>event\>\>relation\. Object hallucinations benefit from direct grounding tools, while attribute hallucinations require finer\-grained perception\. Event hallucinations demand temporal awareness, and relation hallucinations require compositional reasoning over multiple entities and their interactions, representing the most demanding verification task\. This consistent ordering suggests that hallucination difficulty is primarily governed by the compositional complexity of claims rather than by modality\-specific perceptual challenges alone\. Our framework shows the largest improvements on relation hallucinations \(\+9\.2 points on average\), where multi\-expert aggregation is most beneficial as individual experts often capture complementary relational evidence\.

#### Cross\-Modal Failure Patterns\.

Manual error analysis on 50 misclassified samples per task reveals distinct failure modes\.Image: small or heavily occluded objects and subtle attribute differences account for the majority of errors; object detectors frequently miss small\-scale instances\.Video: temporal misalignment, including event ordering errors, omitted intermediate actions, and incorrect causal attributions, is the primary failure mode, particularly when evidence is distributed across non\-adjacent frames requiring long\-range temporal reasoning\.Audio: weak source separation and ambiguous acoustic cues are the main error sources; the ensemble voting strategy is vulnerable when all models share similar perceptual limitations\. These patterns highlight modality\-specific bottlenecks: image detection is primarily perception\-limited, video detection is reasoning\-limited, and audio detection is tool\-limited\. While image and video failures share perception\-related origins rooted in visual understanding, audio failures arise from fundamentally different acoustic modeling limitations, suggesting that future improvements in each modality may require targeted architectural innovations\. Addressing these modality\-specific bottlenecks represents the most promising avenue for advancing cross\-modal hallucination detection\.

## 7Conclusion

We presented OmniHallu and OmniHallu\-Bench as a unified framework and claim\-level benchmark for hallucination detection across cross\-modal comprehension and generation\. By placing diverse tasks and modalities under a common evaluation protocol, our work enables systematic comparison beyond modality\-specific settings\. Our results show that claim decomposition, modality\-aware evidence, and structured reasoning improve hallucination detection across heterogeneous scenarios\. They also reveal persistent weaknesses in temporal and audio\-grounded verification, and a clear difficulty shift from object and attribute errors to event and relation errors\. These findings suggest that future progress will require stronger temporal, auditory, and compositional verification capabilities rather than generic detectors alone\.

## Limitations

#### Taxonomy granularity and scope\.

Our four\-type taxonomy provides a common cross\-modal foundation but compresses modality\-specific phenomena and compound errors\. For example, video events could be divided into temporal ordering, action omission, and causal errors, while audio errors could distinguish source separation from timbre\. Our text\-centric formulation also excludes non\-text pairings such as image\-to\-audio and video\-to\-image, which may require different decomposition strategies\.

#### Model and expert dependence\.

Performance depends on the reasoning model \(Table[6](https://arxiv.org/html/2609.11244#S6.T6)\) and the available experts\. Vision experts are substantially larger than current audio experts, so cross\-modal differences partly reflect tool capability rather than modality alone\. Majority voting also cannot correct errors shared by all experts\.

## Ethical Considerations

All benchmark sources are publicly available, and model\-generated content is disclosed and human\-audited\. We recommend transparent documentation and human review before high\-stakes use\.

## References

- Agrawal et al\. \(2019\)Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson\. 2019\.[nocaps: novel object captioning at scale](https://doi.org/10.1109/iccv.2019.00904)\.In*2019 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 8947–8956\. IEEE\.
- Bai et al\. \(2025a\)Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D\. Plumbley, Woon\-Seng Gan, and Jianfeng Chen\. 2025a\.[AudioSetCaps: An enriched audio\-caption dataset using automated generation pipeline with large audio and language models](https://doi.org/10.1109/TASLPRO.2025.3583354)\.*IEEE Transactions on Audio, Speech and Language Processing*, 33:2817–2829\.
- Bai et al\. \(2025b\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others\. 2025b\.[Qwen2\.5\-VL technical report](https://doi.org/10.48550/arXiv.2502.13923)\.*arXiv preprint arXiv:2502\.13923*\.
- Bai et al\. \(2024\)Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou\. 2024\.[Hallucination of multimodal large language models: A survey](https://doi.org/10.48550/arXiv.2404.18930)\.*arXiv preprint arXiv:2404\.18930*\.
- Bakr et al\. \(2023\)Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny\. 2023\.[HRS\-Bench: Holistic, reliable and scalable benchmark for text\-to\-image models](https://doi.org/10.1109/ICCV51070.2023.01834)\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 19984–19996\.
- Betker et al\. \(2023\)James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh\. 2023\.[Improving image generation with better captions](https://cdn.openai.com/papers/dall-e-3.pdf)\.
- Chen et al\. \(2022\)Haoran Chen, Jianmin Li, Simone Frintrop, and Xiaolin Hu\. 2022\.[The MSR\-Video to text dataset with clean annotations](https://doi.org/10.1016/j.cviu.2022.103581)\.*Computer Vision and Image Understanding*, 225:103581\.
- Chen et al\. \(2024a\)Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen\. 2024a\.[Unified hallucination detection for multimodal large language models](https://doi.org/10.18653/v1/2024.acl-long.178)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3235–3252\. Association for Computational Linguistics\.
- Chen et al\. \(2025\)Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan\. 2025\.[Janus\-Pro: Unified multimodal understanding and generation with data and model scaling](https://arxiv.org/abs/2501.17811)\.*Preprint*, arXiv:2501\.17811\.
- Chen et al\. \(2015\)Xinlei Chen, Hao Fang, Tsung\-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C\. Lawrence Zitnick\. 2015\.[Microsoft COCO captions: Data collection and evaluation server](https://arxiv.org/abs/1504.00325)\.*Preprint*, arXiv:1504\.00325\.
- Chen et al\. \(2024b\)Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yiming Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others\. 2024b\.[Expanding performance boundaries of open\-source multimodal models with model, data, and test\-time scaling](https://arxiv.org/abs/2412.05271)\.*arXiv preprint arXiv:2412\.05271*\.
- Chu et al\. \(2024a\)Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou\. 2024a\.[Qwen2\-Audio technical report](https://doi.org/10.48550/arXiv.2407.10759)\.*arXiv preprint arXiv:2407\.10759*\.
- Chu et al\. \(2024b\)Zhixuan Chu, Lei Zhang, Yichen Sun, Siqiao Xue, Zhibo Wang, Zhan Qin, and Kui Ren\. 2024b\.[Sora detector: A unified hallucination detection for large text\-to\-video models](https://arxiv.org/abs/2405.04180)\.*Preprint*, arXiv:2405\.04180\.
- Deshmukh et al\. \(2023a\)Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang\. 2023a\.[Pengi: An audio language model for audio tasks](https://proceedings.neurips.cc/paper_files/paper/2023/hash/3a2e5889b4bbef997ddb13b55d5acf77-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 18090–18108\.
- Deshmukh et al\. \(2023b\)Soham Deshmukh, Benjamin Elizalde, and Huaming Wang\. 2023b\.[Audio retrieval with WavText5K and CLAP training](https://doi.org/10.21437/Interspeech.2023-1136)\.In*Proceedings of Interspeech 2023*, pages 2948–2952\.
- Drossos et al\. \(2020\)Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen\. 2020\.[Clotho: An audio captioning dataset](https://doi.org/10.1109/ICASSP40776.2020.9052990)\.In*2020 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, pages 736–740\.
- Fonseca et al\. \(2022\)Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra\. 2022\.[FSD50K: An open dataset of human\-labeled sound events](https://doi.org/10.1109/TASLP.2021.3133208)\.*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:829–852\.
- Gemini Team \(2025\)Gemini Team\. 2025\.[Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://arxiv.org/abs/2507.06261)\.*arXiv preprint arXiv:2507\.06261*\.
- Ghosh et al\. \(2024\)Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha\. 2024\.[GAMA: A large audio\-language model with advanced audio understanding and complex reasoning abilities](https://doi.org/10.18653/v1/2024.emnlp-main.361)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 6288–6313\. Association for Computational Linguistics\.
- Huang et al\. \(2026\)Haojian Huang, Harold Haodong Chen, Meng Luo, Junjia Du, Shanqing Xu, Ziheng Chen, Yanxiang Huang, Yinchuan Li, and Ying\-Cong Chen\. 2026\.[No place to hide: Benchmarking video hallucination with background\-controlled pairs](https://doi.org/10.48550/arXiv.2606.31933)\.*arXiv preprint arXiv:2606\.31933*\.
- Huang et al\. \(2025a\)Kaiyi Huang, Chengqi Duan, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu\. 2025a\.[T2I\-CompBench\+\+: An enhanced and comprehensive benchmark for compositional text\-to\-image generation](https://doi.org/10.1109/TPAMI.2025.3531907)\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 47\(5\):3563–3579\.
- Huang et al\. \(2025b\)Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu\. 2025b\.[A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://doi.org/10.1145/3703155)\.*ACM Transactions on Information Systems*, 43\(2\):1–55\.
- Huang et al\. \(2023\)Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao\. 2023\.[Make\-an\-audio: Text\-to\-audio generation with prompt\-enhanced diffusion models](https://proceedings.mlr.press/v202/huang23i.html)\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 13916–13932\. PMLR\.
- Huang et al\. \(2024\)Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Yuexian Zou, Zhou Zhao, and Shinji Watanabe\. 2024\.[AudioGPT: Understanding and generating speech, music, sound, and talking head](https://doi.org/10.1609/aaai.v38i21.30570)\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 38\(21\):23802–23804\.
- Kim et al\. \(2019\)Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim\. 2019\.[AudioCaps: Generating captions for audios in the wild](https://doi.org/10.18653/v1/N19-1011)\.In*Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\)*, pages 119–132, Minneapolis, Minnesota\. Association for Computational Linguistics\.
- Koepke et al\. \(2023\)A\. Sophia Koepke, Andreea\-Maria Oncescu, João F\. Henriques, Zeynep Akata, and Samuel Albanie\. 2023\.[Audio retrieval with natural language queries: A benchmark study](https://doi.org/10.1109/tmm.2022.3149712)\.*IEEE Transactions on Multimedia*, 25:2675–2685\.
- Li et al\. \(2025a\)Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li\. 2025a\.[LLaVA\-OneVision: Easy visual task transfer](https://openreview.net/forum?id=zKv8qULV6n)\.*Transactions on Machine Learning Research*\.
- Li et al\. \(2024\)Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, Ying Tai, Wankou Yang, Yabiao Wang, and Chengjie Wang\. 2024\.[A survey on benchmarks of multimodal large language models](https://arxiv.org/abs/2408.08632)\.*arXiv preprint arXiv:2408\.08632*\.
- Li et al\. \(2023a\)Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian\-Yun Nie, and Ji\-Rong Wen\. 2023a\.[HaluEval: A large\-scale hallucination evaluation benchmark for large language models](https://doi.org/10.18653/v1/2023.emnlp-main.397)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 6449–6464\. Association for Computational Linguistics\.
- Li et al\. \(2026\)Yanlin Li, Minghui Guo, Kaiwen Zhang, Shize Zhang, Yiran Zhao, Haodong Li, Congyue Zhou, Weijie Zheng, Yushen Yan, Shengqiong Wu, Wei Ji, Lei Cui, Furu Wei, Hao Fei, Mong\-Li Lee, and Wynne Hsu\. 2026\.[UniM: A unified any\-to\-any interleaved multimodal benchmark](https://openaccess.thecvf.com/content/CVPR2026/html/Li_UniM_A_Unified_Any-to-Any_Interleaved_Multimodal_Benchmark_CVPR_2026_paper.html)\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 15902–15911\.
- Li et al\. \(2025b\)Yanlin Li, Hao Liu, Huimin Liu, Kun Wang, Yinwei Wei, and Yupeng Hu\. 2025b\.[MIST: Towards multi\-dimensional implicit BiaS evaluation of LLMs for theory of mind](https://doi.org/10.48550/arXiv.2506.14161)\.*arXiv preprint arXiv:2506\.14161*\.
- Li et al\. \(2023b\)Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji\-Rong Wen\. 2023b\.[Evaluating object hallucination in large vision\-language models](https://doi.org/10.18653/v1/2023.emnlp-main.20)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 292–305\. Association for Computational Linguistics\.
- Lin et al\. \(2025\)Hongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang, Jing Ma, See Kiong Ng, and Tat\-Seng Chua\. 2025\.Fact\-audit: An adaptive multi\-agent framework for dynamic fact\-checking evaluation of large language models\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 360–381\.
- Liu et al\. \(2024\)Huadai Liu, Rongjie Huang, Yang Liu, Hengyuan Cao, Jialei Wang, Xize Cheng, Siqi Zheng, and Zhou Zhao\. 2024\.[AudioLCM: Efficient and high\-quality text\-to\-audio generation with minimal inference steps](https://doi.org/10.1145/3664647.3681072)\.In*Proceedings of the 32nd ACM International Conference on Multimedia*, pages 7008–7017\. ACM\.
- Liu and Wan \(2023\)Hui Liu and Xiaojun Wan\. 2023\.[Models see hallucinations: Evaluating the factuality in video captioning](https://doi.org/10.18653/v1/2023.emnlp-main.723)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 11807–11823\. Association for Computational Linguistics\.
- Liu et al\. \(2023\)Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou\. 2023\.[FETV: A benchmark for fine\-grained evaluation of open\-domain text\-to\-video generation](https://doi.org/10.52202/075280-2723)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 62352–62387\.
- Luo et al\. \(2024\)Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong\-Li Lee, and Wynne Hsu\. 2024\.[PanoSent: A panoptic sextuple extraction benchmark for multimodal conversational aspect\-based sentiment analysis](https://doi.org/10.1145/3664647.3680705)\.In*Proceedings of the 32nd ACM International Conference on Multimedia*, pages 7667–7676\. ACM\.
- Luo et al\. \(2026a\)Meng Luo, Bobo Li, Shanqing Xu, Shize Zhang, Qiuchan Chen, Menglu Han, Wenhao Chen, Yanxiang Huang, Hao \(Scofield\) Fei, Mong\-Li Lee, and Wynne Hsu\. 2026a\.[Unveiling the cognitive compass: Theory\-of\-mind–guided multimodal emotion reasoning](https://proceedings.iclr.cc/paper_files/paper/2026/file/89ab43b626b480347dbaccda7be8aedb-Paper-Conference.pdf)\.In*International Conference on Learning Representations*, volume 2026, pages 85428–85498\.
- Luo et al\. \(2026b\)Meng Luo, Shengqiong Wu, Liqiang Jing, Tianjie Ju, Li Zheng, Jinxiang Lai, Tianlong Wu, Xinya Du, Jian Li, Siyuan Yan, Jiebo Luo, William Yang Wang, Hao Fei, Mong\-Li Lee, and Wynne Hsu\. 2026b\.[Dr\.V: A hierarchical perception\-temporal\-cognition framework to diagnose video hallucination by fine\-grained spatial\-temporal grounding](https://doi.org/10.1007/s11263-026-02831-1)\.*International Journal of Computer Vision*, 134\(6\):278\.
- Manakul et al\. \(2023\)Potsawee Manakul, Adian Liusie, and Mark Gales\. 2023\.[SelfCheckGPT: Zero\-resource black\-box hallucination detection for generative large language models](https://doi.org/10.18653/v1/2023.emnlp-main.557)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 9004–9017\. Association for Computational Linguistics\.
- Miao et al\. \(2024\)Ning Miao, Yee Whye Teh, and Tom Rainforth\. 2024\.[SelfCheck: Using LLMs to zero\-shot check their own step\-by\-step reasoning](https://proceedings.iclr.cc/paper_files/paper/2024/hash/fa1edbc195336e29893cdae0939e495f-Abstract-Conference.html)\.In*The Twelfth International Conference on Learning Representations*\.
- Midjourney \(2023\)Midjourney\. 2023\.[Version: Midjourney V6](https://docs.midjourney.com/hc/en-us/articles/32199405667853-Version)\.V6 released 2023\-12\-20; accessed 2026\-08\-23\.
- Nishimura et al\. \(2024\)Taichi Nishimura, Shota Nakada, and Masayoshi Kondo\. 2024\.[On the audio hallucinations in large audio\-video language models](https://doi.org/10.48550/arXiv.2401.09774)\.*arXiv preprint arXiv:2401\.09774*\.
- OpenAI \(2025a\)OpenAI\. 2025a\.[GPT\-4\.1 model](https://developers.openai.com/api/docs/models/gpt-4.1)\.Accessed 2026\-08\-23\.
- OpenAI \(2025b\)OpenAI\. 2025b\.[Introducing GPT\-5\.2](https://openai.com/index/introducing-gpt-5-2/)\.Accessed 2026\-08\-22\.
- Plummer et al\. \(2015\)Bryan A\. Plummer, Liwei Wang, Chris M\. Cervantes, Juan C\. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik\. 2015\.[Flickr30k entities: Collecting region\-to\-phrase correspondences for richer image\-to\-sentence models](https://doi.org/10.1109/ICCV.2015.303)\.In*Proceedings of the IEEE International Conference on Computer Vision \(ICCV\)*, pages 2641–2649\.
- Rawte et al\. \(2025\)Vipula Rawte, Sarthak Jain, Aarush Sinha, Garv Kaushik, Aman Bansal, Prathiksha Rumale Vishwanath, Samyak Rajesh Jain, Aishwarya Naresh Reganti, Vinija Jain, Aman Chadha, Amit Sheth, and Amitava Das\. 2025\.[ViBe: A text\-to\-video benchmark for evaluating hallucination in large multimodal models](https://doi.org/10.18653/v1/2025.trustnlp-main.15)\.In*Proceedings of the 5th Workshop on Trustworthy NLP \(TrustNLP 2025\)*, pages 232–246\. Association for Computational Linguistics\.
- Ren et al\. \(2024\)Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang\. 2024\.[Grounding DINO 1\.5: Advance the "edge" of open\-set object detection](https://arxiv.org/abs/2405.10300)\.*Preprint*, arXiv:2405\.10300\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\. 2023\.[Toolformer: Language models can teach themselves to use tools](https://doi.org/10.52202/075280-2997)\.In*Advances in Neural Information Processing Systems*, volume 36, pages 68539–68551\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y\. K\. Li, Y\. Wu, and Daya Guo\. 2024\.[DeepSeekMath: Pushing the limits of mathematical reasoning in open language models](https://doi.org/10.48550/arXiv.2402.03300)\.*arXiv preprint arXiv:2402\.03300*\.
- Stability AI \(2024\)Stability AI\. 2024\.[Introducing Stable Diffusion 3\.5](https://stability.ai/news-updates/introducing-stable-diffusion-3-5)\.Accessed 2026\-08\-23\.
- Sun et al\. \(2024\)Guangzhi Sun, Potsawee Manakul, Adian Liusie, Kunat Pipatanakul, Chao Zhang, Phil Woodland, and Mark Gales\. 2024\.[CrossCheckGPT: Universal hallucination ranking for multimodal foundation models](https://arxiv.org/abs/2405.13684)\.*arXiv preprint arXiv:2405\.13684*\.
- Sun et al\. \(2025\)Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu\. 2025\.[T2V\-CompBench: A comprehensive benchmark for compositional text\-to\-video generation](https://doi.org/10.1109/CVPR52734.2025.00787)\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 8406–8416\.
- Tang et al\. \(2025\)Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang\. 2025\.[video\-SALMONN 2: Caption\-enhanced audio\-visual large language models](https://arxiv.org/abs/2506.15220)\.*arXiv preprint arXiv:2506\.15220*\.
- Tang et al\. \(2024\)Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang\. 2024\.[SALMONN: Towards generic hearing abilities for large language models](https://openreview.net/forum?id=14rn7HpKVk)\.In*The Twelfth International Conference on Learning Representations*\.
- Wang et al\. \(2020\)Alex Wang, Kyunghyun Cho, and Mike Lewis\. 2020\.[Asking and answering questions to evaluate the factual consistency of summaries](https://doi.org/10.18653/v1/2020.acl-main.450)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 5008–5020\. Association for Computational Linguistics\.
- Wang et al\. \(2024a\)Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang\. 2024a\.[AMBER: An LLM\-free multi\-dimensional benchmark for MLLMs hallucination evaluation](https://arxiv.org/abs/2311.07397)\.*Preprint*, arXiv:2311\.07397\.
- Wang et al\. \(2019\)Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan\-Fang Wang, and William Yang Wang\. 2019\.[VaTeX: A large\-scale, high\-quality multilingual dataset for video\-and\-language research](https://doi.org/10.1109/ICCV.2019.00468)\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 4581–4591\.
- Wang et al\. \(2024b\)Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng\. 2024b\.[VideoHallucer: Evaluating intrinsic and extrinsic hallucinations in large video\-language models](https://doi.org/10.48550/arXiv.2406.16338)\.*arXiv preprint arXiv:2406\.16338*\.
- Wei et al\. \(2022\)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H\. Chi, Quoc V\. Le, and Denny Zhou\. 2022\.[Chain\-of\-thought prompting elicits reasoning in large language models](https://doi.org/10.52202/068431-1800)\.In*Advances in Neural Information Processing Systems*, volume 35, pages 24824–24837\. Curran Associates, Inc\.
- Xu et al\. \(2016\)Jun Xu, Tao Mei, Ting Yao, and Yong Rui\. 2016\.[MSR\-VTT: A large video description dataset for bridging video and language](https://doi.org/10.1109/CVPR.2016.571)\.In*2016 IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 5288–5296\.
- Yang et al\. \(2025\)Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, and Jie Tang\. 2025\.[CogVideoX: Text\-to\-video diffusion models with an expert transformer](https://openreview.net/forum?id=LQzN6TRFg9)\.In*The Thirteenth International Conference on Learning Representations*\.
- Yang et al\. \(2024\)Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang\. 2024\.[DoraemonGPT: Toward understanding dynamic scenes with large language models \(exemplified as a video agent\)](https://proceedings.mlr.press/v235/yang24d.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 55976–55997\. PMLR\.
- Zhang et al\. \(2025\)Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao\. 2025\.[VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding](https://arxiv.org/abs/2501.13106)\.*Preprint*, arXiv:2501\.13106\.
- Zheng et al\. \(2024\)Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You\. 2024\.[Open\-Sora: Democratizing efficient video production for all](https://arxiv.org/abs/2412.20404)\.*Preprint*, arXiv:2412\.20404\.

## Appendix AVoting Mechanism Design

We use equal\-weight voting because the selected experts \(e\.g\., GPT\-4\.1, Qwen2\.5\-VL, InternVL2\.5\) have comparable performance in their respective domains\. Unequal weighting would require reliable cross\-model calibration, whereas equal voting is transparent and requires no task\-specific tuning\.

## Appendix BAdditional Ablation Studies

Table 9:Ablation on the number of expert agents \(Mac\.F1\)\.#### Model Capacity\.

Replacing large experts \(72B/78B\) with small counterparts \(7B/8B\) on video tasks reduces Mac\.F1 from 76\.58/74\.78 to 62\.54/59\.25\.

#### Content Complexity\.

On T2V, hallucination prevalence increases with prompt complexity: 23\.2% \(low density,<<10s, 1 action\), 28\.7% \(medium, 10–20s, 2–3 actions\), and 40\.3% \(high,\>\>20s,≥\\geq4 actions\)\.