GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Hugging Face Daily Papers Papers
benchmark world-models text-simulation llm-evaluation reinforcement-learning

Summary

GlobeBench introduces a standardized benchmark of 1,632 tasks across five domains for evaluating text-based world models, finding that current LLMs — led by GPT-6 Astra at 49.57% — struggle badly with propagating action effects across dependent objects.

Text-based world models have recently shown potential to simulate complex RL environment state transitions for a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by building GlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment's response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with both deterministic gates and a semantic judge. Among 20 general-purpose LLMs and specialized text world models, GPT-6 Astra achieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent's changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as a world model.
Original Article
View Cached Full Text

Cached at: 10/19/26, 07:30 PM

Paper page - GlobeBench: Evaluating Text-Based World Model Capabilities for Scalable Environment Simulations

Source: https://huggingface.co/papers/2610.22708

Abstract

GlobeBench introduces a standardized benchmark of 1,632 tasks across five domains to evaluate text-based world models, revealing that current LLMs struggle with propagating action effects across dependent objects.

Text-based world modelshave recently shown potential to simulate complexRL environment state transitionsfor a diverse set of use cases. However, no standardized benchmark exists to compare, study and improve these models. Through this paper, we attempt to solve this problem by buildingGlobeBench, a novel benchmark with 1,632 tasks across five separate domains and nineteen simulation skills. The tasks check whether models can predict the environment’s response to a specific action given the full agent trajectory by understanding earlier state changes, following environment rules, and predicting the effects of actions on both related and unrelated objects. We compare the predicted environment response with an execution-derived reference, using a hybrid architecture with bothdeterministic gatesand asemantic judge. Among 20 general-purpose LLMs and specialized textworld models,GPT-6 Astraachieves the highest task success rate at 49.57%. We find that model accuracy decreases as actions affect a larger portion of the environment. Specifically, when the agent’s changes must propagate to dependent objects, task success drops to an average of 5.1% across all 20 models. Although model strengths differ across tasks, 631 tasks remain unsolved by any model, suggesting a significant gap in general LLM capabilities as aworld model.

View arXiv pageView PDFProject pageGitHub0Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.22708 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.22708 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.22708 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

VibeWorlding benchmark: open 30B model tops GPT-5.5 on 3D worlds

Reddit r/ArtificialInteligence

A new benchmark for evaluating multimodal agents in building 3D open worlds from natural language shows GPT-5.5 and Qwen3.8-Max scoring below 60%, while an open 30B model achieves the best performance through reinforcement learning.