Orca: The World is in Your Mind

Hugging Face Daily Papers Papers
world-foundation-model multimodal next-state-prediction video-pretraining embodied-ai unified-latent-space

Summary

This paper introduces Orca, a world foundation model that learns a unified latent space from multimodal data using next-state-prediction, outperforming specialized baselines on downstream tasks like text generation, image prediction, and embodied action generation.

We introduce Orca, an initial instantiation of a general world foundation model. Orca learns a unified world latent space from multimodal world signals and exposes it through multimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered on Next-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms: unconscious learning captures dense natural state transitions from continuous videos, and conscious learning models sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scale world-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unified world latent space. To examine whether the learned latent supports downstream, we evaluate it by three representative downstream readouts: text generation, image prediction, and embodied action generation. Orca's backbone is frozen, and only the lightweight modality-specific decoders are trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables stronger downstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a general world foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.
Original Article
View Cached Full Text

Cached at: 06/30/26, 03:33 AM

Paper page - Orca: The World is in Your Mind

Source: https://huggingface.co/papers/2606.30534 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Orca establishes a unified world latent space through next-state-prediction modeling using multimodal data and demonstrates superior performance in downstream tasks compared to specialized baselines.

We introduce Orca, an initial instantiation of a generalworld foundation model. Orca learns a unifiedworld latent spacefrom multimodal world signals and exposes it throughmultimodal readout interfaces. Rather than optimizing isolated next-token, next-frame, or next-action prediction, we are centered onNext-State-Prediction modeling, offering a unified state-transition modeling route toward understanding, predicting, and acting upon the world. Orca learns through two complementary paradigms:unconscious learningcaptures dense natural state transitions from continuous videos, andconscious learningmodels sparse meaningful state transitions by language-described events and VQA supervision. For pre-training, we construct a large-scaleworld-learning inventory data, including 125K hours of video data and 160M event annotations. After pre-training, Orca learns a unifiedworld latent space. To examine whether the learned latent supports downstream, we evaluate it by three representativedownstream readouts: text generation, image prediction, andembodied action generation. Orca’s backbone is frozen, and only the lightweightmodality-specific decodersare trainable. Experiments show the scalability of the proposed paradigm and verify that stronger world latent enables strongerdownstream readouts. Orca outperforms similar-sized specialized baselines. These results show that Orca, as a generalworld foundation model, presents a promising approach to understanding, predicting, and acting upon the world. Finally, we discuss the current limitations, aiming to provide useful insights and inspiration for the community.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.30534

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.30534 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.30534 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.30534 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles