Paper page - ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

Source: https://huggingface.co/papers/2606.17730 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

ActWorld extends navigation-centric interactive world models to support object interaction through a chunk-autoregressive framework with hierarchical action-aware memory and persistent memory banks.

Interactive world modelsaim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends priornavigation-centric generatorsto support mid-rolloutobject interactionwithin achunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interactiondata with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards theevent-transition framesthat causally determine subsequent object states, leading to anaction-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated withper-chunk captionsviachain-of-thought reasoning. On the model side, we introduce a hierarchicalaction-aware memorydesign that routes history compression by interaction importance, complemented by apersistent memory bankthat maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and richobject interactionwithin a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.17730

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.17730 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.17730 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.17730 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.