Paper page - IntReal: A Real-World Dataset and Benchmark for Multimodal Interleaved Generation

Source: https://huggingface.co/papers/2610.38425

Abstract

IntReal introduces a real-world interleaved multimodal dataset and evaluation framework to diagnose unified model strengths and improve generation through targeted training.

Humans routinely consume multimodal content, where textual and visual information is naturally interleaved. AlthoughUnified Multimodal Models(UMMs) can jointly generate text and images, multimodalinterleaved generationremains largely underexplored. Progress in this area is hindered by two key limitations: (1) the lack of well-definedinterleaved generationtasks and corresponding training data, and (2) the absence of fine-grained evaluation protocols that can reliably diagnose model performance across different capability dimensions. To this end, we present IntReal, comprising two complementary components:IntReal-100K, a real-world dataset of multimodal interleaved samples spanning four major categories, and theIntReal Evaluation Suite, which evaluates model performance across seven complementary dimensions using specializedexpert models, enabling coarse-to-fine diagnosis. Extensive experiments validate our evaluation suite through low inter-metric redundancy (max pairwise absolute Spearman’s ρleq 0.42 ), high agreement with human judgments (83.3% accuracy), and strong consistency across different expert evaluators (Spearman’s ρ> 0.9 ), supporting the complementarity and reliability of our metrics. By benchmarking four families of UMMs, we show thatIntReal Evaluation Suitereveals distinct strengths and weaknesses across models. Post-training experiments demonstrate the utility ofIntReal-100Kfor improvinginterleaved generationand further show that training unimodal generation can also benefitinterleaved generation. These findings suggest thatinterleaved generationcan benefit from modality-specific optimization, rather than requiring joint text-image training at every stage.

View arXiv pageView PDFProject pageGitHub2Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.38425 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.38425 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.38425 in a Space README.md to link it from this page.

Collections including this paper1