multimodal-llms

Event-STVG 提出一个面向长视频事件推理的时空视频定位基准，包含跨足球等8个领域的4,016个标注事件；现有最强MLLM在10分钟时长上仅取得5.4% m-vIoU，凸显长上下文事件推理与多目标定位的显著挑战。

The paper proposes Spatial Memory Intelligence (SMI), the first framework to use an understanding model (MLLM) to systematically manage spatial memory in long-video world models via four atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Experiments show improvements in memory sparsity, generation stability, and spatial consistency across multiple world-model backbones.

Where-OPD proposes spatially guided on-policy self-distillation for MLLMs, where a teacher with privileged spatial text guidance is distilled into a student using procedurally generated synthetic scenes, yielding a 3.23-point average gain on real-world perception benchmarks via synthetic-to-real transfer.

This paper identifies a 'latent evidence-credit gap' in latent visual reasoning for multimodal LLMs and proposes ReaLVR, which supplies visual-evidence supervision to free-running latent trajectories via contrastive correct/wrong answers and relevant/mismatched images. ReaLVR outperforms LVR baselines across three model families (63.7% five-task average on Qwen2.5-VL-7B) and scales robustly to 235B-parameter frontier models.

BenchBench is a benchmark that challenges multimodal LLMs to compete in assembling Ikea furniture, testing their practical AI capabilities.

This paper introduces SCOPE, a joint training method for computer-use agents that improves both task completion and safety, using a synthesized dataset and achieving strong performance on benchmarks OSWorld and OS-BLIND.

A systematic study benchmarking training-free uncertainty quantification strategies for multimodal Large Language Models, categorizing methods into token-level, verbalized, and semantic approaches and finding optimal strategies depend on response length.

PolyBridgeBench is a new executable benchmark for evaluating multimodal LLMs on physics-grounded bridge design tasks, revealing gaps between deterministic validity and dynamic success in structure synthesis and repair.

This paper presents AREA, a training-free inference-time method that adaptively allocates evidence highlighting in multimodal large language models, improving performance on knowledge-based visual question answering and standard multimodal benchmarks.

OmniHallu introduces a unified hallucination detection framework for multimodal large language models, covering comprehension and generation tasks across image, video, and audio modalities, with a benchmark and multi-agent architecture.

ReactHuman is a physics-grounded benchmark that evaluates multimodal large language models for human-like reactive decision-making in simulated humanoid robots facing household hazards, revealing persistent safety failures and offering a scalable diagnostic tool.

The paper proposes HEAL, a method to mitigate hallucinations in Multimodal Large Language Models by analyzing and calibrating information distribution in synergy heads through causal interventions and counterfactual analysis.

The paper introduces TA-SPA, a black-box jailbreak attack framework for multimodal large language models that uses text-anchored semantic perturbations to achieve effective and transferable attacks against safety alignments.

Video-IFBench introduces a comprehensive benchmark for evaluating how well multimodal large language models follow diverse instructions in video understanding tasks, covering various constraints and task types.

The paper proposes Length-Aware Contrastive Learning for GUI Agents (LACL-GUI), a contrastive reinforcement learning framework that incorporates trajectory-level quality signals to improve agent performance by addressing reward-gradient misalignment.

OraRL improves reinforcement learning post-training for video multimodal language models by integrating annotations as oracle rollouts, achieving higher sample efficiency and performance without chain-of-thought generation.

The tweet points out that multimodal large language models capable of processing smells do not yet exist, highlighting a current limitation in AI technology.

MobileForge is a benchmark for project-level multi-screen mobile app generation, evaluating multimodal LLMs on build success, cross-page navigation, visual fidelity, maintainability, and efficiency. Experiments on six frontier multimodal LLMs show current models can compile and reach pages but still struggle with interactive navigation and visual quality.

This paper introduces TokenSwap, a method to convert text-only benchmarks into image-interleaved counterparts, and TokenSwap-Bench to measure the modality gap across 42 multimodal LLMs. It finds reasoning models have smaller gaps and shows that TokenSwap-based training can reduce the gap.

This paper introduces Symphony-Bias, a multimodal dataset for evaluating gender bias in LLMs' associations with musical instruments across text, vision, and audio. It finds that 92% of instrument-level outcomes align with prior social-science findings, with gender biases strongest in text and weakest in audio.