VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
Summary
This paper proposes VibeWorlding, a framework for benchmarking and training multimodal agents to construct 3D open worlds from user queries, showing that reinforcement learning improves open-source models to compete with closed-source frontiers.
View Cached Full Text
Cached at: 08/18/26, 03:53 AM
Paper page - VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
Source: https://huggingface.co/papers/2608.15265
Abstract
A unified framework benchmarks and trains multimodal agents that infer intent, plan 3D scenes, invoke tools, and reflect on feedback, revealing that reinforcement learning improves open-source models beyond closed-source frontiers.
Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare howmultimodal agentsunderstand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we proposeVibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first buildVWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we developVibeWorlding-Gym, a jointmultimodal RLpost-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering asMCP tools, and (2) arubric-based verifierthat combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalablemultimodal RLreward service. Our experiments show that current frontierMLLMsare far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-sourceMLLMsto even surpass closed-source frontiers: ourVibeWorlder-8B is comparable to frontierMLLMs, while our flagshipVibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.
View arXiv pageView PDFProject pageGitHub24Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.15265 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.15265 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.15265 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Infinite Worlds with Versatile Interactions
This paper introduces LingBot-World 2.0, an advanced world modeling system with unbounded interaction horizons, real-time 720p 60fps video generation, diverse interactive elements (e.g., attacking, spell-casting), and novel multi-agent behavior control via pilot and director agents, along with a shared multi-player interface.
SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
SpatialWorld is a unified benchmark for evaluating interactive spatial reasoning in multimodal agents across diverse real-world tasks, revealing that even the strongest models achieve low task success rates.
Thinking with Imagination: Agentic Visual Spatial Reasoning with World Simulators
The paper proposes Astra, an agentic spatial reasoning framework that couples a reinforcement learning-trained VLM policy with a world simulator to generate novel-view observations for improved spatial reasoning in Vision-Language Models.
MultiWorld: Scalable Multi-Agent Multi-View Video World Models
MultiWorld is a unified framework for multi-agent multi-view video world modeling that achieves accurate control of multiple agents while maintaining multi-view consistency through a Multi-Agent Condition Module and Global State Encoder.
MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
The MineExplorer benchmark evaluates multimodal large language model agents' open-world exploration abilities in Minecraft using atomic and multi-hop tasks designed through multi-agent synthesis. Experiments show that open-world exploration remains challenging, with strong models degrading sharply over longer trajectories.