WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
Summary
WorldToken presents a time-first sequence modeling method for robotic imitation learning, using causal Transformers and diffusion action heads to fuse heterogeneous observations, with performance evaluations on RoboCasa and RMBench benchmarks.
View Cached Full Text
Cached at: 08/25/26, 12:35 PM
Paper page - WorldToken: Time-First Sequence Modeling for Robotic Imitation Learning
Source: https://huggingface.co/papers/2608.22591
Abstract
WorldToken fuses heterogeneous robot observations into per-timestep world tokens processed by a causal Transformer and diffusion action head, with scaling and temporal-context analyses on RoboCasa and RMBench.
Robot policies receive heterogeneous observations at each decision step, yet sequence models differ in how they organize these inputs over time. We introduceWorldToken, a time-first policy instantiation that fusesmultiview images,proprioception, andtask conditioningwithin each policy timestep into oneworld token. Acausal temporal Transformermodels the resulting world-token sequence, and adiffusion action headgenerates action chunks. On 23 RoboCasa tasks, an 85.3M-parameter policy trained from scratch apart from a frozen pretrained CLIP text encoder achieves 59.45% meanclosed-loop successusing 2,900 generated demonstrations per task. A complete factorial sweep over five dataset sizes, five model sizes, and two training seeds shows consistent gains from additional target-domain data and diminishing returns beyond moderate model size. Under same-checkpoint history truncation, reducing visible history to one or two policy timesteps lowersclosed-loop successfor all 50 RoboCasa policies. On RMBench Blocks Ranking, reducing visible history from 146 to 8 seconds lowers evaluator success from 95% to 28%, while an exploratory extended rollout sustains the reference swap sequence for over 850 seconds. These results establish the empirical feasibility of the completeWorldTokeninstantiation and characterize its data-scaling and temporal-context behavior under the tested recipes. They do not establish superiority over alternative sequence organizations or isolate which components of the complete implementation drive the observed performance.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.22591
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.22591 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.22591 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.22591 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Identifiable Token Correspondence for World Models
This paper introduces Identifiable Token Correspondence, a method that models token correspondence across time frames to improve temporal consistency in transformer-based world models for visual reinforcement learning, achieving state-of-the-art results on multiple benchmarks.
World Machine: Towards Generative World Modeling for Time-Series
World Machine proposes a transformer-based generative world modeling architecture for time series that uses latent states to adapt to varying context lengths, addressing the quadratic memory cost of traditional transformers. Experiments on a synthetic dataset validate its feasibility and show improvements over conventional transformers.
τ_0-WM: A Unified Video-Action World Model for Robotic Manipulation
τ_0-WM is a unified video-action world model for robotic manipulation that integrates policy learning, video prediction, and action evaluation using a shared video diffusion backbone. It shows superior performance on challenging long-horizon and fine-grained tasks.
RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
RepWAM introduces a world action modeling approach using representation visual-action tokenizers, aiming to learn unified visual and action representations for planning and control.
Learning Transferable Dynamics Priors from Action to World Modeling
This paper introduces A2World, a diffusion-based world model pretrained on large-scale robot manipulation data to learn transferable dynamics priors. The model can be adapted into a real-world simulator (A2World-sim) for policy evaluation or a video-action prediction model (A2World-policy) for action prediction, demonstrating benefits for both simulator-centric and policy-centric robot learning.