Paper page - WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Source: https://huggingface.co/papers/2610.39278 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

WorldFlow is a generative world model that jointly predicts future video and 3D trajectories using continuous displacement tokens and a Mixture-of-Transformers backbone, improving physical dynamics prediction across diverse benchmarks.

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce , agenerative world modelthat jointly predicts future video and query-based3D trajectories. By parameterizing 3D motion ascontinuous displacement tokensaligned with the video grid, a unifiedMixture-of-Transformers(MoT) backbone jointly denoises both modalities under arectified-flowobjective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduceWorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowestaverage displacement error(ADE) andfinal displacement error(FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

View arXiv pageView PDFProject pageGitHub17Add to collection

Get this paper in your agent:

hf papers read 2610\.39278

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.39278 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.39278 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.39278 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.