Tag
MobileForge is a benchmark for project-level multi-screen mobile app generation, evaluating multimodal LLMs on build success, cross-page navigation, visual fidelity, maintainability, and efficiency. Experiments on six frontier multimodal LLMs show current models can compile and reach pages but still struggle with interactive navigation and visual quality.
This paper introduces TokenSwap, a method to convert text-only benchmarks into image-interleaved counterparts, and TokenSwap-Bench to measure the modality gap across 42 multimodal LLMs. It finds reasoning models have smaller gaps and shows that TokenSwap-based training can reduce the gap.
This paper introduces Symphony-Bias, a multimodal dataset for evaluating gender bias in LLMs' associations with musical instruments across text, vision, and audio. It finds that 92% of instrument-level outcomes align with prior social-science findings, with gender biases strongest in text and weakest in audio.
This paper investigates why multimodal LLMs fail on vision-centric tasks when visual evidence conflicts with language priors, introducing the WhatIfVis benchmark and showing that models often encode visual evidence but cannot reliably control their reliance on it.
A comprehensive survey on multimodal humor understanding using large language models, covering methods, datasets, evaluation protocols, and challenges in interpreting humor in memes, cartoons, and comics.
This paper introduces ActiveVision, a benchmark to evaluate active observation in multimodal large language models. Frontier models like GPT-5.5 and Claude Fable 5 perform poorly, solving only 10.6% and 3.5% of tasks respectively, compared to human 96.1%, highlighting a lack of iterative visual perception.
This paper evaluates the zero-shot capability of multimodal large language models (MLLMs) for localized concept naming in images, proposing a reproducible evaluation protocol that achieves 62-88% object-level accuracy without training.
NormAct is a benchmark that evaluates embodied planning agents on hidden social norm compliance, revealing that state-of-the-art MLLMs achieve 67.3% goal achievement but only 26.4% norm compliance, and proposes NormPerceptor to improve task success from 24.2% to 46.7%.
This paper identifies a specialized subset of attention heads called CoRe heads in multimodal LLMs that exhibit functional sparsity in cross-modal retrieval. Causal interventions show these heads are crucial for multimodal reasoning, and leveraging this sparsity can accelerate inference.
A survey presenting a human-view perspective on video understanding with multimodal large language models, organized around watching, remembering, and reasoning abilities, covering challenges, methods, and applications.
This paper introduces ReasonMatch-Bench, a benchmark for wide-baseline matching in multimodal LLMs, and proposes Dynamic Correspondence Reinforcement Learning (DCRL) to improve spatial reasoning. Experiments show significant gains on the benchmark while maintaining general performance.
LatentOmni proposes a cross-modal reasoning framework that interleaves textual reasoning with audio-visual latent states, outperforming explicit text-based chain-of-thought methods in audio-visual reasoning tasks.
This paper introduces a distribution-aware reinforcement learning framework that enhances MLLM performance in long-tailed numerical regression tasks using batch-level comparison-based supervision.
This paper proposes a causal framework for probing internal visual representations in Multimodal Large Language Models, revealing differences in how entities and abstract concepts are encoded. The study highlights that increasing model depth is crucial for encoding abstract concepts and uncovers a disconnect between perception and reasoning in current MLLMs.
The paper introduces BalCapRL, a balanced reinforcement learning framework for multimodal large language models that jointly optimizes correctness, coverage, and linguistic quality in image captioning. It demonstrates improved performance over existing methods by addressing trade-offs between utility and fluency through reward decoupling and length-conditional masking.
This paper investigates the arithmetic limitations of multimodal LLMs on multi-digit multiplication across text, image, and audio modalities, introducing a controlled benchmark and a novel 'arithmetic load' metric (C) that better predicts model accuracy than traditional step-counting methods. Results show accuracy collapses as C grows, and that performance degradation is primarily computational rather than perceptual.
Researchers introduce Mind’s Eye, a benchmark of eight visual-cognitive tasks that reveals top multimodal LLMs score under 50% while humans reach 80%, exposing major gaps in visual abstraction, relation mapping and mental transformation.
Research shows Chain-of-Thought prompting harms visual-spatial reasoning in multimodal LLMs due to shortcut learning and hallucinating visual details from text alone.