Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
Summary
Incantation presents an interactive video world model that uses natural language as the action interface for fine-grained multi-entity control and cross-entity generalization, achieving high performance and real-time streaming through novel attention and distillation techniques.
View Cached Full Text
Cached at: 05/19/26, 06:30 AM
Paper page - Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
Source: https://huggingface.co/papers/2605.18601 Authors:
,
,
,
,
,
,
,
,
,
,
,
,
Abstract
Incantation enables interactive video world modeling with natural language conditioning for fine-grained multi-entity control and cross-entity generalization through novel video backbone and attention mechanisms.
Moderninteractive video world modelshave achieved impressive visual fidelity, yet lack fine-grainedmulti-entity controland cross-entity, cross-world generalization. We trace this gap to the action interface: standard control protocols (e.g. animation IDs, device inputs, scene-level captions) bind action semantics to specific entities or engines at design time. We propose natural language as the interface to unlock expressiveness that no prior interface can achieve, and we present Incantation, the first interactive video world model withper-latent-frame(0.25 s) natural-language conditioning that supports simultaneousmulti-entity controland concept-levelcross-entity transferbeyond any fixed rendering pipeline. We pair a pretrainedbidirectional video backbonewithframe-local text cross-attention, and enable real-time long-horizon streaming throughODE-initialized Self-Forcing distillationwith aRoPE-decoupled sliding KV-cache. We surpass the Action-Index baseline oncross-entity transfer(89% vs. 43%) and out-of-vocabulary prompts (90% vs. 0%), and our 2-step student sustains 19.7 FPS at 480p with stableFVDover 2-hour rollouts. We further apply the same architecture and training recipe to The King of Fighters, changing only the per-entityaction vocabulary slots. We have released a preview subset of the Incantation dataset at https://huggingface.co/datasets/zhush/incantation-elden-ring-scenes, containing manually collected Elden Ring player-boss combat clips with structured action-oriented metadata. Larger-scale Elden Ring and KOF data will be released with the full project.
View arXiv pageView PDFProject pageGitHub2Add to collection
Get this paper in your agent:
hf papers read 2605\.18601
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.18601 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.18601 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.18601 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Wonder: Video World Model Done Better
Wonder is a general-purpose video world model that enables real-time, camera-controllable world exploration from an image or conditional video. It introduces camera conditioning via dense coordinate fields, a sparse attention memory mechanism, and techniques to improve distillation, allowing minute-scale video generation at 16 FPS.
Infinite Worlds with Versatile Interactions
This paper introduces LingBot-World 2.0, an advanced world modeling system with unbounded interaction horizons, real-time 720p 60fps video generation, diverse interactive elements (e.g., attacking, spell-casting), and novel multi-agent behavior control via pilot and director agents, along with a shared multi-player interface.
InternVideo3: Agentify Foundation Models with Multimodal Contextual Reasoning
InternVideo3 introduces Multimodal Contextual Reasoning (MCR) and efficient attention mechanisms to enhance long-horizon multimodal tasks, achieving strong results on video understanding benchmarks and demonstrating video agent capabilities.
INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models
INTACT is an end-to-end unified JEPA that learns the intent-to-action mapping directly, enabling search-free world model control. It achieves 95.33% direct macro success rate across four visual-control tasks with zero test-time search and ~300x lower planning latency.
MultiWorld: Scalable Multi-Agent Multi-View Video World Models
MultiWorld is a unified framework for multi-agent multi-view video world modeling that achieves accurate control of multiple agents while maintaining multi-view consistency through a Multi-Agent Condition Module and Global State Encoder.