InteractiveWorld

A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

paper: https://t.co/rNQOBFqi81 https://t.co/CZHca4zuwW

Paper page - InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants

Source: https://huggingface.co/papers/2610.33966 Published on Oct 25

·

Submitted byhttps://huggingface.co/Qinsi1

Wangon Oct 27

Abstract

InteractiveWorld provides a verifiable simulation environment and benchmark to evaluate whether multimodal language models can reliably guide humans through physical tasks.

People have long dreamed of an assistant that is always there to guide them and remind what they forget, from new dishes to lost keys. Powered bymultimodal large language models(MLLMs),egocentric AI assistantson wearable devices are bringing this dream within reach: they see what the user sees and give instructions for user’s physical work when necessary. However, testing such assistance is challenging and usually requires real people. The expensive and rare real interaction data leaves a key question unanswered: Can today’s MLLMs truly help people in physical world? To answer this question, we proposeInteractiveWorld, the first verifiable environment forclosed-loop interactionbetween human and AI assistant. Unlike existing robot-centric simulators,InteractiveWorldlets the assistant guide a human agent who observes the scene, understands instructions, and acts independently. It thus realistically simulates how AI assistant collaborates with human: the assistant instructs, the human acts, and the environment executes actions and updates what they see next. Building on it, we introduceInteractiveBench, the first dynamic benchmark foregocentric AI assistants, with 120 multi-turn tasks across six categories. In each task, the assistant guides the human in real time, and every instruction shapes the outcome, which is scored by verifiable,state-based rewards. Experiments with 12frontier MLLMsshow they are still far from reliable assistants. Even the strongest model, Claude Fable 5.1, achieves at most 20% and 25% on long-horizon Guidance and Planning tasks, respectively. Overall, our work offers a verifiable platform and a rigorous evaluation set, laying a solid foundation for research on interactive MLLM assistants that truly help people in the physical world.

View arXiv pageView PDFProject pageGitHub27Add to collection

Get this paper in your agent:

hf papers read 2610\.33966

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.33966 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.33966 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.33966 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.