ALPS: Visual State Encoders for World Models
Summary
ALPS is a pretrained ViT-B/16 visual state encoder for world models that unifies physical-state prediction, language alignment, and pixel reconstruction in a single patch-token representation, achieving state-probe performance matching the 3.5x larger V-JEPA 2-L and 85.5% mean policy success on LIBERO when paired with its native text tower.
View Cached Full Text
Cached at: 10/22/26, 03:45 PM
Paper page - ALPS: Visual State Encoders for World Models
Source: https://huggingface.co/papers/2610.30120
Abstract
ALPS is a pretrained visual encoder that unifies physical-state prediction, language alignment, and pixel reconstruction to serve as a shared foundation for world models.
World modelsturn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrainedvisual state encoderthat unifies all three in a single set ofpatch tokens. Using only still images and text, we pretrain aViT-B/16from scratch with language, reconstruction, andspatial correspondenceobjectives, and then refine it viacross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregateLIBEROstate-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success onLIBERO(85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action inworld models.
View arXiv pageView PDFProject pageGitHub6Add to collection
Get this paper in your agent:
hf papers read 2610\.30120
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### UCSC-VLAA/ALPS Image Feature Extraction• Updatedabout 14 hours ago • 1
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2610.30120 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.30120 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Stateful Visual Encoders for Vision-Language Models
This paper introduces a stateful visual encoder for vision-language models that conditions visual representations on prior features, enabling better visual comparison in multi-image and agentic settings. The method shows consistent improvements across tasks such as cross-image spatial aggregation and longitudinal radiology.
AtlasVLA: Persistent World-Ego State Modeling for Vision-Language-Action Models
AtlasVLA introduces a dual-memory architecture with persistent world-ego state modeling to overcome perception and task-progress forgetting in vision-language-action models, achieving state-of-the-art long-horizon manipulation from a single wrist camera.
LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
LeWorldModel introduces a stable, end-to-end Joint-Embedding Predictive Architecture that trains directly from pixels with minimal hyperparameters and provable anti-collapse guarantees. It achieves significant speedups in planning compared to foundation models while maintaining competitive performance on robotic manipulation tasks.
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
The paper proposes δ-Vision, a method to reduce computational overhead in multimodal language models by using lightweight low-rank adapters to reconstruct visual states efficiently without discarding visual tokens.
StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
StateSight is a new benchmark for evaluating spatial-state reconstruction in vision-language models, showing that models like GPT-5.5 and Claude Sonnet 5 struggle with spatial reasoning tasks compared to human performance.